very fast at protocol indexer with flexible filtering, xrpc queries, cursor-backed event stream, and more, built on fjall
rust
fjall
at-protocol
atproto
indexer
[docs] clearer readme explanations, remove unused conf var
+5
-3
+5
-3
README.md
+5
-3
README.md
···
18
18
| :--- | :--- | :--- |
19
19
| distribution | sharded work queue: events are load-balanced across connected clients. If 5 clients connect, each receives ~20% of events. | broadcast: every connected client receives a full copy of the event stream. if 5 clients connect, all 5 receive 100% of events. |
20
20
| cursors | server-managed: clients ACK messages. the server tracks progress and redelivers unacked messages. | client-managed: client provides `?cursor=123`. the server streams from that point. |
21
-
| persistence | events are stored in an outbox and sent to the consumer, removing them, so they can't be replayed once they are acked. | `record` events are replayable. `identity`/`account` are ephemeral. use `GET /repos/:did` to query identity / account info (handle, pds, signing key, etc.). |
21
+
| persistence | events are stored in an outbox and sent to the consumer, and removed from the outbox when acked. nothing is replayable. | `record` events are replayable. `identity`/`account` are ephemeral. use `GET /repos/:did` to query identity / account info (handle, pds, signing key, etc.). |
22
22
| backfill | backfill events are mixed into the live queue and prioritized (per-repo, acting as synchronization barrier) by the server. | backfill simply inserts historical events (`live: false`) into the global event log. streaming is just reading this log sequentially. synchronization is the same as tap, `live: true` vs `live: false`. |
23
-
| event types | `record`, `identity` (includes status) | `record`, `identity` (handle), `account` (status) |
23
+
| event types | `record`, `identity` (includes status) | `record`, `identity` (handle, cache-buster), `account` (status) |
24
24
25
25
### multiple relay support
26
26
···
31
31
- each relay maintains its own firehose / crawler cursor state
32
32
- all ingestion loops and crawlers share the same worker pool and database
33
33
- all crawlers share the same pending queue for backfill
34
+
35
+
commit events are de-duplicated according to the repo `rev`. account / identity events are de-duplicated using the `time` field.
36
+
todo: decide what to do on relay-side account takedowns or if relays set the `time` field.
34
37
35
38
## configuration
36
39
···
58
61
| `API_PORT` | `3000` | port for the API server. |
59
62
| `ENABLE_DEBUG` | `false` | enable debug endpoints. |
60
63
| `DEBUG_PORT` | `API_PORT + 1` | port for debug endpoints (if enabled). |
61
-
| `NO_LZ4_COMPRESSION` | `false` | disable lz4 compression for storage. |
62
64
| `ENABLE_FIREHOSE` | `true` | whether to ingest relay subscriptions. |
63
65
| `ENABLE_BACKFILL` | `true` | whether to backfill from PDS instances. |
64
66
| `ENABLE_CRAWLER` | `false` (if Filter), `true` (if Full) | whether to actively query the network for unknown repositories. |