my prefect server setup prefect-metrics.waow.tech
python orchestration
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

revert tangled-items schedule stagger

DuckDB supports concurrent writers through WAL — the contention was
from Grafana's exclusive lock, not between the two ingestion flows.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

zzstoatzz 928c0710 b1ddddb2

+4 -4
+3 -3
docs/hub.md
··· 4 4 5 5 ## data sources 6 6 7 - two ingestion flows run hourly, staggered to avoid DuckDB write contention: 7 + two ingestion flows run hourly at :00, writing to separate DuckDB tables: 8 8 9 9 **gh-notifications** (`flows/gh_notifications.py`) — fetches github notifications (issues + PRs) and open items authored by `zzstoatzz` via the search API. each issue is cached by repo+number for 24h. persists to `raw_github_issues`. 10 10 ··· 21 21 └────────────────────┘ 22 22 23 23 tangled PDS ──► tangled-items ───► raw_tangled_items ────┘ 24 - (hourly :02) │ 24 + (hourly :00) │ 25 25 26 26 hub_action_items 27 27 (mart, top 200) ··· 41 41 |---|---|---| 42 42 | `diagnostics` | `*/5 * * * *` | prints system info — canary for worker health | 43 43 | `gh-notifications` | `0 * * * *` | github notifications + authored open issues/PRs → `raw_github_issues` | 44 - | `tangled-items` | `2 * * * *` | tangled.org issues/PRs/comments → `raw_tangled_items` | 44 + | `tangled-items` | `0 * * * *` | tangled.org issues/PRs/comments → `raw_tangled_items` | 45 45 | `enrich` | `5 * * * *` | dbt build: staging → enrichment → mart. concurrency limit 1. runs under python 3.13 (dbt-core compat) | 46 46 | `curate` | `10 * * * *` | loads top 200 scored items, sends to claude haiku 4.5 via pydantic-ai, writes `briefing.json` | 47 47 | `cleanup` | `0 2 * * 0` | deletes old terminal flow runs (completed, failed, cancelled, crashed) older than 30 days |
+1 -1
prefect.yaml
··· 36 36 entrypoint: flows/tangled_items.py:tangled_items 37 37 work_pool: *k8s 38 38 schedules: 39 - - cron: "2 * * * *" # staggered from gh-notifications to avoid DuckDB write contention 39 + - cron: "0 * * * *" # hourly 40 40 41 41 - name: enrich 42 42 entrypoint: flows/enrich.py:enrich