A diet Tap option for handling atproto sync

Add /resync/:did and /resync-all endpoints #20

Merged pds.dad
targeting tab from deleted fork:feature/resync-endpoints
Merged opened by and assigned to pds.dad
targeting tab from deleted fork:feature/resync-endpoints

Adds two routes for re-running backfill on repos tab already tracks:

  • POST /resync/:did queues one tracked repo for resync and returns its resulting state, or 404 if tab does not track it
  • POST /resync-all queues every tracked repo and returns how many were queued

These are the routes @dholms proposed for upstream Tap in bluesky-social/indigo#1261.

Why #

Adding a collection to TAB_COLLECTION_FILTERS only affects new events. Repos that are already tracked were backfilled through the old filters, and /repos/add ignores DIDs that are already tracked, so their existing records in the new collection never arrive.

The only workaround today is /repos/remove followed by /repos/add. That deletes the repo's stored record state, so the backfill that follows emits every record in every filtered collection again, and the repo's live events are skipped while it waits as pending.

How #

Both routes move repos in active or error state to desynchronized and clear any retry backoff. The resync workers already claim desynchronized repos, after pending ones. The repo's RepoRecord rows are kept, so the existing CID comparison in the resync skips everything already delivered and only emits records that are newly matched, such as those of an added collection, with live: false.

Repos already pending, desynchronized or resyncing are left alone, as are accounts that cannot be fetched (takendown, suspended, deactivated). /resync/:did never adds an untracked repo, since /repos/add does that.

/resync-all can pause live processing for a long time on a large deployment, since each repo's live events wait until its resync runs. The README says so, and callers that need to slow-roll can use /resync/:did.

Testing #

  • TestResyncRepo and TestResyncAll cover the state transitions, the cleared backoff, kept RepoRecord rows, the 404 for untracked DIDs, and the states that are left alone. Both fail without the handlers.
  • go test ./cmd/tab/ and go vet ./cmd/tab/ pass.
  • The same change was run against a real database of about 18,000 tracked repos, after adding collections to the filters. Across 11 repos in two runs, the resyncs emitted exactly the records of the new collections (134 in total, each repo's /info record count rising by its count in them) and none of the 904 records in collections that were already filtered. Each run finished in about 6 seconds.
Commits 0