zds
atproto pds pds.zat.dev
pds atproto spaces
Star 45
Fork 0
Watch

zds-pds memory climbs to 1 GB ceiling, OOM kills and CPU throttling #23

Closed zzstoatzz.io
Closed opened by zzstoatzz.io

zds-pds's resident memory climbs from about 230 MB after a restart to the 1 GB machine ceiling within days. The process then gets SIGKILLed, and the shared CPU also ran out of burst. This blocks moving zzstoatzz.io onto any zds host until we know why.

Fly metrics for zds-pds (shared-cpu-1x, 1 GB, ord, 24 accounts), 2026-10-28 to 2026-11-09. That's all Fly keeps:

zds-pds pds-zzstoatzz-io (Bluesky PDS, 1 account), for comparison
memory used, avg / max of 987 MB 505 / 965 MB 390 / 491 MB
days at the ceiling 6 of 11 0
memory_pressure_some peak, 10 s average 86% 0%
CPU busy, avg / p99 / max 4.3% / 41% / 98% 0.6% / 1.8% / 32%
CPU throttle every day 10-29 → 11-03, cpu_balance at its 0.5 floor none
restarts 23 1
HTTP 200s 297,981 357,547
bytes sent 37.2 GB, 12.8 GB of it via the edge 11.1 GB
restarts and kills
  • 18 restarts line up with releases v351–v370.
  • fly_instance_exit_code was 137 on 10-31 22:35 and on 11-04 02:25 UTC. fly_instance_exit_oom was set at 11-04 02:25.
  • Three more restarts match no release: 10-29 02:50, 11-05 09:40 and 11-07 01:10. Their exit reason wasn't recorded, because the gauges don't change on a repeat value.
  • Since the redeploy on 11-08, memory has been flat at about 232 MB.

Daily max memory (MB): 10-29 939 · 10-30 706 · 10-31 841 · 11-01 964 · 11-02 879 · 11-03 565 · 11-04 964 · 11-05 895 · 11-06 964 · 11-07 964 · 11-08 232

other things on the volume
  • /data is 1.34 GB used of 20 GB, and that figure was flat over the window.
  • /data/blobs is 190 MB.
  • zds.sqlite3 is 720 MB, plus a 296 MB WAL.
  • About 100 MB of leftover zds-snapshot.part.* and zds-snapshot.sqlite3.gz files are still in /data.
  • About 24 GB was sent over 11 days without passing through Fly's edge. I haven't found what sends it.
how to reproduce the queries

Fly's Prometheus endpoint needs Authorization: FlyV1 $(fly auth token). Bearer returns 401 "something went wrong resolving organization".

max_over_time((fly_instance_memory_mem_total{app="zds-pds"} - fly_instance_memory_mem_available{app="zds-pds"})[1d:1m])
fly_instance_exit_code{app="zds-pds"}
fly_instance_cpu_balance{app="zds-pds"}

against https://api.fly.io/prometheus/personal/api/v1/query_range.

Questions:

  • Is the climb a leak, an unbounded cache (blocks, MST nodes, event backfill, firehose subscribers), or sqlite page cache that mem_available doesn't count as reclaimable?
  • Does it track request volume or uptime? Adding zzstoatzz.io would roughly double the requests.
  • What drives the CPU bursts that drained the throttle balance from 10-29 to 11-03?
zzstoatzz.io

The climb was a leak, fixed in 0.7.4 (c676048, live since 11-07 04:32 UTC). describeRepo, getRecord and listRecords allocated their results with page_allocator and never freed them, so every public read kept its memory. Before the fix, memory grew about 20 MB an hour in a straight line (194 → 1011 MB on 11-05/06), and page cache stayed at 50–84 MB. Since the fix, memory has gone from 195 to 221 MB over 20 hours, about 10 MB of which is sqlite page cache.

Follow-up checks, shipped in 0.7.6 (Fly v371):

  • I swept for the same pattern. One function still had it, getByUri, but nothing called it. It's deleted, along with about 450 lines of other unused code.
  • I ran a local load sweep of the public, authenticated, proxy and write endpoints. After warm-up, none kept growing.
  • In production, 800 listRecords/describeRepo reads moved RSS from 30.6 to 31.3 MB.

Still open:

  • whether the remaining ~1 MB an hour levels off;
  • the 24 GB that left without going through the edge;
  • the CPU bursts on 10-29 → 11-03, which happened while the leak was active.
zzstoatzz.io

Closing: the leak was the one fixed in 0.7.4. The slower climb after the fix is reclaimable filesystem cache, not zds memory.

  • The process is small. On 0.7.7, the zds process holds 48 MB resident (41 MB anonymous) with 74 threads. Fly's "used" figure (mem_total - mem_available) was 216 MB because it counts kernel buffers, page cache, slab and reserve.
  • Where the post-fix growth went. From 11-07 05:00 to 11-08 05:00, the longest uptime after the fix, "used" rose 45 MB. Over the same 24 hours, buffers rose 47 MB (3 → 50), page cache 22 MB (49 → 71) and slab 2 MB. zds's own memory was flat; the climb is SQLite and blob reads warming the cache, which the kernel reclaims under pressure.
  • Paths checked locally. A load sweep of the public, authenticated, proxy, write and bad-token proxy paths leveled off after warm-up. That includes the production log's most frequent request, listNotifications with an invalid token: 15,000 requests, then flat.
  • Egress and CPU. Daily eth0 egress was 1.8–7.1 GB from 10-30 to 11-03, then dropped to 0.2–1.0 GB from 11-04 onward; the CPU throttling ended in the same window. Both stopped before the 11-07 leak fix, so the leak didn't cause them. Fly doesn't keep request logs back to 10-30, so I can't attribute that traffic. It hasn't come back.
  • Leftover files. The zds-snapshot.part.* files in /data are from 2026-07-30 and unrelated.
Sign up or Sign in to add to the discussion.