my prefect server setup
prefect-metrics.waow.tech
python
orchestration
re-stagger tangled-items to :02 — DuckDB is single-writer-process
tested: two separate processes writing to the same DuckDB file fails
with a lock error. DuckDB only supports one writer process at a time.
gh-notifications and tangled-items must not overlap.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+4
-4
+3
-3
docs/hub.md
+3
-3
docs/hub.md
···
4
4
5
5
## data sources
6
6
7
-
two ingestion flows run hourly at :00, writing to separate DuckDB tables:
7
+
two ingestion flows run hourly, staggered because DuckDB only allows one writer process at a time:
8
8
9
9
**gh-notifications** (`flows/gh_notifications.py`) — fetches github notifications (issues + PRs) and open items authored by `zzstoatzz` via the search API. each issue is cached by repo+number for 24h. persists to `raw_github_issues`.
10
10
···
21
21
└────────────────────┘
22
22
│
23
23
tangled PDS ──► tangled-items ───► raw_tangled_items ────┘
24
-
(hourly :00) │
24
+
(hourly :02) │
25
25
▼
26
26
hub_action_items
27
27
(mart, top 200)
···
41
41
|---|---|---|
42
42
| `diagnostics` | `*/5 * * * *` | prints system info — canary for worker health |
43
43
| `gh-notifications` | `0 * * * *` | github notifications + authored open issues/PRs → `raw_github_issues` |
44
-
| `tangled-items` | `0 * * * *` | tangled.org issues/PRs/comments → `raw_tangled_items` |
44
+
| `tangled-items` | `2 * * * *` | tangled.org issues/PRs/comments → `raw_tangled_items` |
45
45
| `enrich` | `5 * * * *` | dbt build: staging → enrichment → mart. concurrency limit 1. runs under python 3.13 (dbt-core compat) |
46
46
| `curate` | `10 * * * *` | loads top 200 scored items, sends to claude haiku 4.5 via pydantic-ai, writes `briefing.json` |
47
47
| `cleanup` | `0 2 * * 0` | deletes old terminal flow runs (completed, failed, cancelled, crashed) older than 30 days |
+1
-1
prefect.yaml
+1
-1
prefect.yaml