lightweight
com.atproto.sync.listReposByCollection
update
+4
-1
+2
-1
hacking.md
+2
-1
hacking.md
···
133
133
- [x] config: db mem limit `--fjall-cache-mb`
134
134
- [x] config: per-host request rate self-throttling `--crawl-qps` (name from collectiondir)
135
135
- [x] resync: estimate CAR size from `getRecord` mst height; `getRepo` if it's likely very small
136
+
- [x] admin view of backfill state etc
136
137
- [ ] special did:web ident cache behaviour to keep reusing a stale resolution on failure
137
-
- [ ] admin view of backfill state etc
138
138
- [ ] vanity stats for optimizations, like how many in-flight repos were saved from resync due to high-water-mark firehose cursor persistence
139
139
- [ ] if the upstream is a PDS (check with describeServer?) then make only accept events for DIDs that have it as their PDS
140
140
- [ ] use `since` on getRepo for resync to get a smaller partial export in many cases (and then more-carefully do the actual resync)
···
143
143
- [ ] check response headers and adjust self-throttling rate limits per-host if present
144
144
- [ ] make backfill go _really fast_
145
145
- [ ] clean up commit validation (eg we're checking signatures twice, lenient handling is weird)
146
+
- [ ] metrics for db size
146
147
147
148
going to be annoying but doable
148
149
- [ ] multi-relay subscriber
+2
readme.md
+2
readme.md
···
2
2
3
3
**status: almost working well but _not stable yet!!_**
4
4
5
+
**...full-network backfill mostly works but needs tuning and tweaking. firehose collection extraction mostly works but needs more verification**.
6
+
5
7
Lightrail uses the _adjacent keys_ in firehose commit CAR slices to detect first-record-added-to and last-record-removed-from collections in atproto repos, _statelessly_. Since most commits don't change repos' collection lists, this eliminates most of the work to maintain an accurate repos-by-collection index.
6
8
7
9
Compared to Bluesky's [`collectiondir`](https://github.com/bluesky-social/indigo/tree/main/cmd/collectiondir) service, lightrail: