pds: record heap never reclaims deleted bodies and has no quota key #176

Open

permadeath.com 1mo ago

Open opened by

permadeath.com 1mo ago

(2026-10-02 release-readiness audit, agent-filed)

The heap is append-only; Heap::reclaim only truncates the unreferenced tail at boot (crates/didbot-pds/src/heap.rs:478-502), and checkpoint/trim shrink only the WAL. The config concedes the heap "grows with every record written" with no [capacity] key for it (crates/didbot-config/src/sections.rs:170-177). Create/delete churn accumulates dead bodies across restart cycles until the volume fills; the heap then seals the deployment to serving reads only, and no restart shrinks it.

Either compact the heap (rewrite live bodies) or give it a budget key with the same loud refusal the WAL has.

permadeath.com 3w ago

Disk numbers from a 10x pass on loopback (73ddde8): each record write costs about 885 B of WAL (trimmed at checkpoint), 190–280 B of heap (never reclaimed) and 258 B of writes.log (kept 30 days). That is about 1.35 KB logical and 3.6 KB of device writes. Under the floor's 16,000 writes a day the heap grows about 5 MB a day, so the 20 GB volume lasts 1–2 years. With the floor lifted and 270 writes/s sustained it would be about 12 GB a day, so about 1.5 days. Not urgent while the floor holds, but compaction is what bounds it.

permadeath.com 2w ago

Still real on main at 1351e8b33: Heap::reclaim only truncates the unreferenced tail at boot. Either fix needs a design change, so neither fits an ordinary release. Compaction rewrites live bodies to new offsets, so every offset in the checkpoint and the log changes with them. That is a data-layout change, and a rollback would have to read the rewritten heap. A budget key is a new [capacity] setting with a new refusal path. Unset by default, it fixes nothing. Set by default, an upgrade can refuse writes on a deployment whose heap is already past it. Compaction at boot, behind a layout bump, is the one that fixes the problem; it needs the owner's review of the layout change.