my prefect server setup
prefect-metrics.waow.tech
python
orchestration
docs: clean up pipeline diagrams
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+46
-45
+23
-22
README.md
+23
-22
README.md
···
3
3
[hub](https://hub.waow.tech) · [grafana](https://prefect-metrics.waow.tech/d/executive-overview/executive-overview?orgId=1&from=now-6h&to=now&timezone=browser)
4
4
5
5
```
6
-
github API ──┐
7
-
├──► ingest ──► raw_github_issues ──┐
8
-
tangled PDS ─┤ (hourly) raw_tangled_items ─┤
9
-
bluesky PDS ─┤ raw_likes + raw_liked_posts │
10
-
phi (tpuf) ─┘ raw_phi_observations ─┘
11
-
│
12
-
transform (dbt) ◄──────┘
13
-
[on ingest ✓]
14
-
│
15
-
┌───────────────┼───────────────┐
16
-
▼ ▼ ▼
17
-
brief compact hub UI
18
-
[on transform ✓] [on transform ✓]
19
-
│ │
20
-
▼ ▼
21
-
briefing.json TurboPuffer
22
-
(phi-users-*)
6
+
data sources
7
+
─────────────────────────────────────────────
8
+
github API ──┐
9
+
tangled PDS ──┤
10
+
bluesky likes ──┼──► ingest (hourly) ──► DuckDB
11
+
phi memory (tpuf) ──┘
12
+
│
13
+
▼
14
+
transform (dbt)
15
+
[on ingest ✓]
16
+
│
17
+
┌─────────────────────────┼──────────┐
18
+
▼ ▼ ▼
19
+
brief compact hub UI
20
+
[on transform ✓] [on transform ✓]
21
+
│ │
22
+
▼ ▼
23
+
briefing.json TurboPuffer
24
+
(phi-users-*)
23
25
24
-
morning ──► TurboPuffer + semble
25
-
(daily 8am CT)
26
-
27
-
rebuild-atlas ──► Cloudflare Pages
28
-
(every 6h)
26
+
standalone flows
27
+
─────────────────────────────────────────────
28
+
morning (daily 8am CT) ──► TurboPuffer + semble
29
+
rebuild-atlas (every 6h) ──► Cloudflare Pages
29
30
```
30
31
31
32
see [docs/hub.md](docs/hub.md) for the full pipeline breakdown.
+23
-23
docs/hub.md
+23
-23
docs/hub.md
···
17
17
## pipeline
18
18
19
19
```
20
-
github API ──┐
21
-
├──► ingest ──► raw_github_issues ──┐
22
-
tangled PDS ─┤ (hourly) raw_tangled_items ─┤
23
-
bluesky PDS ─┤ raw_likes + raw_liked_posts │
24
-
phi (tpuf) ─┘ raw_phi_observations ─┘
25
-
│
26
-
transform (dbt) ◄──────┘
27
-
[on ingest ✓]
28
-
│
29
-
┌───────────────┼───────────────┐
30
-
▼ ▼ ▼
31
-
brief compact hub UI
32
-
[on transform ✓] [on transform ✓] /api/cards.json
33
-
│ │ +page.svelte
34
-
▼ ▼
35
-
briefing.json TurboPuffer
36
-
/api/briefing (phi-users-*)
37
-
```
38
-
39
-
two additional flows run independently:
20
+
data sources
21
+
─────────────────────────────────────────────
22
+
github API ──┐
23
+
tangled PDS ──┤
24
+
bluesky likes ──┼──► ingest (hourly) ──► DuckDB
25
+
phi memory (tpuf) ──┘
26
+
│
27
+
▼
28
+
transform (dbt)
29
+
[on ingest ✓]
30
+
│
31
+
┌─────────────────────────┼──────────┐
32
+
▼ ▼ ▼
33
+
brief compact hub UI
34
+
[on transform ✓] [on transform ✓]
35
+
│ │
36
+
▼ ▼
37
+
briefing.json TurboPuffer
38
+
/api/briefing (phi-users-*)
40
39
41
-
```
42
-
morning (daily 8am CT) ──► TurboPuffer (tag maintenance) + semble (curation)
43
-
rebuild-atlas (every 6h) ──► Cloudflare Pages (leaflet-search atlas)
40
+
standalone flows
41
+
─────────────────────────────────────────────
42
+
morning (daily 8am CT) ──► TurboPuffer + semble
43
+
rebuild-atlas (every 6h) ──► Cloudflare Pages
44
44
```
45
45
46
46
## flows