On a stop, serve_tls_inner stops accepting and calls graceful_shutdown(Some(drain)). A subscribeRepos stream is a request in flight that never finishes on its own: it ends only when the firehose closes, after the drain runs out. With relays connected, a release can be down for the whole [server] drain_secs (default 10 s), not just the restart.
Seen deploying 4471df89 to pds.did.bot on 2026-10-19: the log shows no longer accepting; waiting on requests in flight drain=10s at 14:06:21, then 22 subscribeRepos consumer disconnected ... the server is closing the stream at 14:06:31. /health and describeServer failed for those 10 s. The bc4f5dea swap on 2026-10-17 ran the same drain code and missed one 1 s probe; what differed then is not known. The relays reconnected within 0.9 s of the new server starting, so ending each stream at the start of the drain would cost them nothing.
Expected: on a stop, the firehose streams end first, with the close frame they get now, and the drain waits only on ordinary requests.
The firehose streams were not what held the drain. Once hyper upgrades a connection to a WebSocket it hands the socket off, so axum-server no longer counts it, and
stop_window.rsalready showed 20 subscribers ending within a second. What holds a stop is a connection still in its TLS handshake. axum-server counts it from accept, its graceful shutdown cannot reach it until the handshake is done, and the handshake timeout is 10 s, the same asdrain_secs. On the public internet scanners leave such connections all the time, which would also explain why one deploy missed only one probe.Reproduced locally: a single TCP connection that sends half a ClientHello held the stop for 9.5 s. Fixed in #1236, which ends handshakes still in progress when a stop begins.
stop_window.rsnow includes a stalled handshake, and the process exits about 100 ms after SIGTERM.