index: move the weekly ledgers out of the whole-file state.json #97

Closed

Closed opened by

The index keeps its whole state in memory and saves it as one JSON file. Each save rewrites the whole file, and new features keep adding to it.

Saving: --state writes state.json (crates/vibescrobble-index/src/state.rs, state::write) every --state-every seconds (default 60, range 5–3600) and on shutdown. It writes beside the file, fsyncs, renames over it and fsyncs the directory. bin/vibescrobble-index.rs save_loop and save_state drive it.
The file holds the board's accounts, records, people, servers, tree, groups, bot-label answers and stream cursors. Beside it: identities.jsonl (an append log) and the avatar store.
The swarm-card stack (#268–#272) adds per-account weekly ledgers for this year and last to the same file, so every save rewrites up to two years of history. perf-10x (~/Documents/didbot-review/research/perf-10x.md) puts the 10x board at about 3,000 agents. A ledger row is a few bytes per scrobble, so the file and its once-a-minute rewrite grow into tens of MB.
One file means one unreadable save sends the next start back to a cold relay read (VERSION mismatch or a parse failure). That already costs the restart read budget (#65, #273).

Suggested fix:

Keep state.json for the live board only: cursors, the current view and groups.
Store the ledgers in SQLite through rusqlite, with one row per (account DID, ISO week). A closed week stops changing at Monday 00:00 UTC, so writes go to the current week only. Yearly cards then become a query.
Check whether identities.jsonl and the bot-label answers belong there too.
The repos stay the source of truth. Rebuilding the store from them stays possible and tested.

Filed by Claude (agent) from a static read of main at d8bb9fb.

Done in https://tangled.org/permadeath.com/vibescrobble.com/pulls/278, the top of the swarm-card stack (#268–#272, #278), and it closes when that merges.

Where the ledgers live: --ledgers <path> keeps them in a SQLite file (/data/ledgers.sqlite in compose, beside state.json), one row per (account DID, ISO week), through rusqlite with SQLite bundled.
What stays in memory: only the open week. It is written out on the save beat and at shutdown, and once more when a newer week closes it. Closed weeks change only when an account leaves or a gap is read back.
state.json: holds the live board alone. Ledgers found in an old one are taken into an empty store once, at the first start after the upgrade.
The query service: keeps none. It asks the index over /ledgers/... on the private network, so /view?all seeds carry no weeks either.
Rebuilding from the repos: still works. A new backfill test rebuilds a week into an empty store, reads it twice (counted once) and reopens the file.
Not changed: identities.jsonl and the bot-label answers stay where they are.

🤖 Generated with Claude Code

Fixed on main by 5576f864: the weekly ledgers live in SQLite at --ledgers (infra/files/compose.yml), described in docs/state.md.