What happens. com.atproto.sync.getBlob needs no credential and has no per-address limit. Every read takes one of 32 permits shared by all callers and keeps it until the whole body is sent. A client that reads the response headers and then stops reading keeps its permit indefinitely. Nothing times out a connection once its headers are read. With 32 such connections open, every other getBlob waits and gets a 503 after 30 s. Each waiting request also holds one of the 512 server-wide request slots for those 30 s. About 17 getBlob calls a second then fill all 512, and every route waits, /health included.
How to reproduce.
crates/didbot-serve/src/blobs.rs:62sets oneSemaphore::const_new(32)for every caller.blobs.rs:283-286takes a permit.blobs.rs:485keeps it for the reader task's whole life. That task parks inblocking_send(blobs.rs:504) once the 4-chunk readahead is full and the client reads nothing more.get_blob(blobs.rs:256) isPublic(crates/didbot-serve/src/routes/mod.rs:548) and never callsrequire_sync_budget.list_blobsdoes (blobs.rs:355).- The only connection timer is
header_read_timeout(crates/didbot-serve/src/lib.rs:432).TimeoutLayer(routes/mod.rs:890) stops when the response future resolves, not when the body finishes. - A local test against b8bab73cf uploaded a 1 MiB blob, opened 32 getBlob responses and never read them, then made a 33rd call with a 3 s timeout. It printed
still waiting (blocked)until the 32 connections were dropped.
Why it matters. One address holding a few dozen TCP connections stops this server serving blobs to every app view, so every account's avatars and images break. Adding queued requests takes the whole server down.
Fixed in #1248, stacked on #1243. One address may hold
[capacity] blob_reads_in_flight_per_addressreads (default 8 of the 32); one more is refused at once with a 429 rather than queued. A body whose client takes no chunk forblob_send_timeout_secs(default 30) ends short and gives its read back.