my prefect server setup
prefect-metrics.waow.tech
python
orchestration
fix enrich: PER_NODE + concurrency=1 for UI visibility without DuckDB deadlock
ThreadPoolTaskRunner still deadlocks: concurrent dbtRunner.invoke() calls
share the same class-level DuckDB _ENV and block on each other internally.
concurrency=1 serializes node execution (sets max_workers=1 on the process
pool) while keeping each node as a distinct Prefect task in the UI.
DuckDB sees only one writer at a time. CacheConfig stays (PER_NODE only).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
+1
-2
+1
-2
flows/enrich.py
+1
-2
flows/enrich.py
···
3
3
from pathlib import Path
4
4
5
5
from prefect import flow, get_run_logger
6
-
from prefect.task_runners import ThreadPoolTaskRunner
7
6
from prefect_dbt.core._orchestrator import (
8
7
CacheConfig,
9
8
ExecutionMode,
···
38
37
orchestrator = PrefectDbtOrchestrator(
39
38
settings=settings,
40
39
execution_mode=ExecutionMode.PER_NODE,
41
-
task_runner_type=ThreadPoolTaskRunner,
40
+
concurrency=1,
42
41
cache=CacheConfig(expiration=timedelta(hours=1)),
43
42
test_strategy=TestStrategy.DEFERRED,
44
43
create_summary_artifact=True,