Mirror of
Merge pull request #5 from trueberryless-org/thesis
create thesis content
authored by
trueberryless
and committed by
GitHub
177256c2
2f20e590
+126
-3
+5
.changeset/stupid-lizards-travel.md
+5
.changeset/stupid-lizards-travel.md
+121
-3
starlight/src/content/docs/videos/thesis.mdx
+121
-3
starlight/src/content/docs/videos/thesis.mdx
···
1
1
---
2
-
title: Diploma Thesis
2
+
title: "Course: Diploma Thesis"
3
3
description: A SOA solution with Siemens AG to analyze power grid anomalies, featuring a Kafka pipeline, PostgreSQL, GraphQL API, and an Angular dashboard for visualisation.
4
4
video:
5
5
type: video
6
6
link: https://www.youtube.com/watch?v=p5xtVAwQdmE
7
-
duration: 146
7
+
duration: 500
8
8
date: 2024-03-27
9
9
transcript: ./src/assets/transcripts/thesis.srt
10
10
difficulty: Intermediate
11
11
actions:
12
-
- text: Read more
12
+
- text: Read the full story
13
13
link: https://www.trueberryless.org/work/thesis/
14
14
icon: external
15
15
---
16
+
17
+
import { Quiz, QuizOption } from 'starlight-videos/components';
18
+
19
+
## Course: Building a Data Pipeline and Visualization System
20
+
21
+
**Objective**
22
+
23
+
Learn the components and workflow of a service-oriented architecture (SOA) to handle and visualize power grid anomalies. This includes the integration of Apache Kafka, PostgreSQL, GraphQL, and Angular.
24
+
25
+
### Lesson 1: Introduction to the Architecture
26
+
27
+
**Key Components:**
28
+
29
+
1. **Apache Kafka** : A messaging system for real-time data ingestion.
30
+
31
+
2. **PostgreSQL** : A relational database to persist anomaly data.
32
+
33
+
3. **GraphQL API** : Serves as the interface between backend and frontend, enabling queries and filtering.
34
+
35
+
4. **Angular** : Frontend framework used to create an interactive and user-friendly dashboard.
36
+
37
+
**Workflow:**
38
+
39
+
1. Anomaly data is generated by Siemens' Global Network Analysis (GNA) system.
40
+
41
+
2. Data flows through Apache Kafka to PostgreSQL for structured storage.
42
+
43
+
3. A GraphQL API exposes this data for frontend use.
44
+
45
+
4. Angular-based dashboards visualize the data in tables, graphs, and key performance indicators (KPIs).
46
+
47
+
### Interactive Question 1: Kafka’s Role
48
+
49
+
<Quiz title="What is Apache Kafka used for in this project?">
50
+
<QuizOption>Database management</QuizOption>
51
+
<QuizOption correct>Real-time data ingestion</QuizOption>
52
+
<QuizOption>Frontend visualization</QuizOption>
53
+
<QuizOption>API implementation</QuizOption>
54
+
</Quiz>
55
+
56
+
### Lesson 2: Backend Details
57
+
58
+
**Apache Kafka** :
59
+
60
+
- Provides a publish-subscribe messaging model.
61
+
62
+
- Guarantees high throughput and fault tolerance.
63
+
64
+
- Messages persist for later consumption if needed.
65
+
**PostgreSQL** :
66
+
- Relational storage optimized for anomaly data.
67
+
68
+
- Queries can be filtered, sorted, and paginated.
69
+
**GraphQL API** :
70
+
- Flexible querying capabilities, reducing over-fetching.
71
+
72
+
- Facilitates frontend data access with structured responses.
73
+
**Key Consideration** : The decoupling of the backend from data sources and consumers ensures scalability and reliability.
74
+
75
+
### Interactive Question 2: Backend Technologies
76
+
77
+
<Quiz title="Which technologies form the backend of the application?" multiple>
78
+
<QuizOption correct>Kafka</QuizOption>
79
+
<QuizOption correct>PostgreSQL</QuizOption>
80
+
<QuizOption correct>GraphQL API</QuizOption>
81
+
<QuizOption>MySQL</QuizOption>
82
+
</Quiz>
83
+
84
+
### Lesson 3: Frontend Visualization
85
+
86
+
**Angular Framework** :
87
+
88
+
- Enables dynamic and responsive UI.
89
+
90
+
- Used to design key components:
91
+
92
+
- **Anomalies Table** : Searchable, paginated, and filterable table for anomaly data.
93
+
94
+
- **Graph View** : Visual representation of the power grid, pinpointing sources of issues.
95
+
96
+
- **Dashboard Overview** : Displays aggregated metrics for quick insights.
97
+
**Best Practices for Visualization** :
98
+
99
+
- Intuitive design with interactive elements.
100
+
101
+
- Use of frameworks like Cytoscape.js for graph rendering.
102
+
103
+
### Interactive Question 3: Frontend Framework
104
+
105
+
<Quiz title="Why was Angular chosen for this project?">
106
+
<QuizOption>Its support for backend integration</QuizOption>
107
+
<QuizOption>Inherent support for SQL queries</QuizOption>
108
+
<QuizOption>It is a messaging framework</QuizOption>
109
+
<QuizOption correct>Responsive and dynamic UI capabilities</QuizOption>
110
+
</Quiz>
111
+
112
+
### Lesson 4: Deployment and Integration
113
+
114
+
**Deployment** :
115
+
116
+
- Docker containers ensure consistency across environments.
117
+
118
+
- The SOA allows for modular development and maintenance.
119
+
**Integration with Siemens Infrastructure** :
120
+
- Kafka seamlessly integrates for data transmission.
121
+
122
+
- PostgreSQL supports Siemens' existing data workflows.
123
+
**Outcome** :
124
+
The solution enhances anomaly analysis by transforming unstructured logs into actionable insights, significantly reducing resolution time for Siemens engineers.
125
+
126
+
### Interactive Question 4: Integration
127
+
128
+
<Quiz title="Which feature enhances the system's modularity and scalability?">
129
+
<QuizOption correct>Service-Oriented Architecture</QuizOption>
130
+
<QuizOption>Direct database access</QuizOption>
131
+
<QuizOption>Monolithic architecture</QuizOption>
132
+
<QuizOption>Hardcoded endpoints</QuizOption>
133
+
</Quiz>