Clone this repository
For self-hosted knots, clone URLs may differ based on your setup.
Download tar.gz
Replace buffered res.arrayBuffer() with incremental stream parsing.
Each block is .slice()d into its own Uint8Array, eliminating the single
large external ArrayBuffer that V8 can't GC (213MB → 64MB external).
Also adds diff-based backfill via `since` parameter with fallback to
full import when the PDS has compacted history.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Retry timers in triggerAutoBackfill bypassed the concurrency check in
processMessage, allowing unbounded concurrent CAR downloads (23 observed
in production, 2GB RSS). Move the concurrency gate into
triggerAutoBackfill itself and use config.backfill.parallelism instead of
a hardcoded constant. Add dedup set to prevent re-schedule timer
accumulation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace eager block Map with LazyBlockMap that stores byte offsets
into the CAR buffer instead of copying block data
- Convert walkMst to a generator so entries are yielded and processed
one at a time instead of collecting 193k+ entries into an array
- Delete block offsets as they're consumed to allow mid-processing GC
- Add iterator support to LazyBlockMap for indexer compatibility
- Reduce default backfill parallelism from 5 to 3
- Bump max-old-space-size from 256 to 512 in Dockerfile template
- Add build/publish/release scripts to root package.json
Benchmarked against a 71MB CAR (244k blocks, 193k records):
peak heap dropped from OOM at 512MB to ~415MB with recovery to 109MB.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>