the problem #
The backlink webhook stream (unfiltered Jetstream) fell ~10.6 h behind on 2026-10-14 (2.6 h at 12:33Z, 10.6 h at 23:45Z) on sharkgirl. Its handler is fast, but the client spends most of its time reconnecting: a full intake queue (8192) closes the socket, waits for the queue to drain completely, then reconnects after a jittered 1-1000 ms backoff and a fresh TLS + Jetstream seek. Prod logs (last 10 min, 188 cycles): socket open p50 0.31 s, closed p50 1.48 s per 8192 events, open only 45% of the time, ~11 reconnects/min, ~2.6k events/s.
the solution #
Queue pressure is flow control, not failure. When the queue fills, the client still closes the socket, but it reconnects as soon as half the queue is free, immediately (no backoff), from the newest event still queued (every event up to it is acked or held in memory). Fetching then overlaps the drain. Handler failure and protocol quarantine are unchanged: they drop the queue, wait for a full drain and replay from the last durable ack with backoff. Playback is inclusive, so the boundary event arrives twice, which every replay already does and the durable delivery state deduplicates; dedup by time_us was rejected because time_us is not guaranteed unique. Not done here: keeping the socket open and pausing reads (TCP backpressure) would remove reconnects entirely, but Bun's WebSocket client cannot pause; that needs a separate experiment.
before #
NODE_ENV=development WEBHOOK_ALLOW_INSECURE_DEV=1 bun bench/jetstream-catchup.ts 10 45 <us/event> (new): unfiltered replay from 10 h back, wss://jetstream2.us-west, prod queue sizes (8192/4096), handler cost simulated per event, 45 s per run, runs alternated old/new from one laptop.
- 100 us/event: 6,482 events/s, 12.3x realtime, 13 connects
- 200 us/event: 3,263 events/s, 6.2x realtime, 14 connects
after #
- 100 us/event: 6,965 events/s, 13.2x realtime (+7%), 22 connects
- 200 us/event: 4,365 events/s, 8.3x realtime (+34%), 25 connects
Gains grow with handler cost and reconnect cost (sharkgirl reconnects to us-east with a seek ~10 h back, which this laptop bench does not reproduce). With the handler at ~7.5 us/event the queue never fills and both clients are network-bound (28.7x). Tests: jetstream 12/12 (two new tests fail on the old client), webhook-service 81/81, biome clean, tsc only the known blog/astro error.