my prefect server setup
prefect-metrics.waow.tech
python
orchestration
revert tangled-items schedule stagger
DuckDB supports concurrent writers through WAL — the contention was
from Grafana's exclusive lock, not between the two ingestion flows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+4
-4
+3
-3
docs/hub.md
+3
-3
docs/hub.md
···
4
4
5
5
## data sources
6
6
7
-
two ingestion flows run hourly, staggered to avoid DuckDB write contention:
7
+
two ingestion flows run hourly at :00, writing to separate DuckDB tables:
8
8
9
9
**gh-notifications** (`flows/gh_notifications.py`) — fetches github notifications (issues + PRs) and open items authored by `zzstoatzz` via the search API. each issue is cached by repo+number for 24h. persists to `raw_github_issues`.
10
10
···
21
21
└────────────────────┘
22
22
│
23
23
tangled PDS ──► tangled-items ───► raw_tangled_items ────┘
24
-
(hourly :02) │
24
+
(hourly :00) │
25
25
▼
26
26
hub_action_items
27
27
(mart, top 200)
···
41
41
|---|---|---|
42
42
| `diagnostics` | `*/5 * * * *` | prints system info — canary for worker health |
43
43
| `gh-notifications` | `0 * * * *` | github notifications + authored open issues/PRs → `raw_github_issues` |
44
-
| `tangled-items` | `2 * * * *` | tangled.org issues/PRs/comments → `raw_tangled_items` |
44
+
| `tangled-items` | `0 * * * *` | tangled.org issues/PRs/comments → `raw_tangled_items` |
45
45
| `enrich` | `5 * * * *` | dbt build: staging → enrichment → mart. concurrency limit 1. runs under python 3.13 (dbt-core compat) |
46
46
| `curate` | `10 * * * *` | loads top 200 scored items, sends to claude haiku 4.5 via pydantic-ai, writes `briefing.json` |
47
47
| `cleanup` | `0 2 * * 0` | deletes old terminal flow runs (completed, failed, cancelled, crashed) older than 30 days |
+1
-1
prefect.yaml
+1
-1
prefect.yaml