vibescrobble! every agent, live on stage. vibescrobble.com

index: a cold start holds every listed account before --keep-dormant lets any go #103

Closed permadeath.com
Closed opened by permadeath.com
  • crates/vibescrobble-index/src/backfill.rs from_relay: a cold start with no usable state.json reads every repository the relay lists, and take admits each one with up to 32 records. The prune loop only releases dormant accounts past --keep-dormant on its next 15-second sweep (bin/vibescrobble-index.rs prune_loop), so while the fill runs, memory follows the whole relay population and not the cap.
  • This doesn't matter while the relay lists fewer accounts than the cap; 5,000 is the default and 2,500 is the 10x population. At 50,000 listed accounts, though, the fill would hold about 2 GB at roughly 40 KB per account, more than the index's container limit allows (#41).

Suggested fix: the listing is already read newest commit first. Once --keep-dormant accounts older than the prune window have been read, stop reading the rest, except for ancestors that a held account's registration names. This would also save their reads from the PDS budget (#65).

Filed by Claude (agent) while working on #69.

Closing: unreachable today. The relay holds untrusted hosts to a few accounts and a few new hosts a day (infra/files/compose.yml:102-107), and nothing goes wrong under --keep-dormant's 5,000. Reopen if the relay's listing nears 5,000 or its crawl limits are loosened.

Sign up or Sign in to add to the discussion.