my prefect server setup
prefect-metrics.waow.tech
python
orchestration
fix enrich: PER_NODE + ThreadPoolTaskRunner for UI visibility without DuckDB locks
PER_WAVE produces no Prefect tasks (dbt runs as opaque subprocesses).
PER_NODE creates one task per dbt node (visible in UI) but defaults to
ProcessPoolTaskRunner which spawns separate processes that conflict on
the DuckDB file lock.
ThreadPoolTaskRunner runs all node tasks in the same process — DuckDB
handles concurrent connections within one process cleanly. Also restores
CacheConfig which is only supported in PER_NODE mode.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
+6
-1
+6
-1
flows/enrich.py
+6
-1
flows/enrich.py
···
1
1
import subprocess
2
+
from datetime import timedelta
2
3
from pathlib import Path
3
4
4
5
from prefect import flow, get_run_logger
6
+
from prefect.task_runners import ThreadPoolTaskRunner
5
7
from prefect_dbt.core._orchestrator import (
8
+
CacheConfig,
6
9
ExecutionMode,
7
10
PrefectDbtOrchestrator,
8
11
TestStrategy,
···
34
37
)
35
38
orchestrator = PrefectDbtOrchestrator(
36
39
settings=settings,
37
-
execution_mode=ExecutionMode.PER_WAVE,
40
+
execution_mode=ExecutionMode.PER_NODE,
41
+
task_runner_type=ThreadPoolTaskRunner,
42
+
cache=CacheConfig(expiration=timedelta(hours=1)),
38
43
test_strategy=TestStrategy.DEFERRED,
39
44
create_summary_artifact=True,
40
45
)