Identities for entities did.bot

serve: the stop drain waits out every firehose stream, so a deploy refuses connections for drain_secs #211

Closed permadeath.com
Closed opened by permadeath.com

An in-place deploy of pds.did.bot is unreachable for about 10 seconds, all of it the drain waiting on subscribeRepos streams.

The 2026-10-14 deploy of 36250f0b7251, probed once a second: the last answer from the old process came at 23:38:56.7Z, and nine /health and describeServer requests then failed to connect, until 23:39:07.7. The server's journal: at 23:38:56 it logged checkpointed before stopping took_ms=93, then no longer accepting; waiting on requests in flight drain=10s. At 23:39:06, exactly drain_secs later, it closed the 20 open subscribeRepos streams. Our relay and Jetstreams hold those streams open, so on a live server the drain always runs to its limit, and the listener refuses new connections for all of it. The new process was serving 0.7 s after the old one exited.

Closing the firehose streams with 1001 when the drain starts, rather than when it times out, would leave the drain to the requests that end on their own. #1182 measured 0.2–0.8 s locally with no subscribers connected.

🤖 Generated with Claude Code

Correction from the next deploy. The streams did not hold the drain open. The 2026-10-15 deploy of 22f4a152488b logged no longer accepting; waiting on requests in flight drain=10s at 01:07:09.741, closed about 20 subscribeRepos streams 0.26 s later, flushed the log at 01:07:10.014, and the new process started at 01:07:10.21. Listener closed for about 0.6 s, and a 1-second probe saw no failure.

So in the 10 s case, some other request was still in flight when the stop began, most likely a long poll such as agentd's listPendingAuthorizations?wait=25. The streams close only when the drain ends, so they waited with it. The outage is as long as the longest request in flight when the stop begins, capped by drain_secs.

Fixed by #1190 on main: 9f74a4b9 answers a long poll sent after the stop at once. The drain was held by listPendingAuthorizations polls, not firehose streams (HTTP/1.1 subscribeRepos upgrades out of the drain); in-place releases no longer wait out the full drain.

Sign up or Sign in to add to the discussion.