very fast at protocol indexer with flexible filtering, xrpc queries, cursor-backed event stream, and more, built on fjall
rust
fjall
at-protocol
atproto
indexer
[docs] fix filter management section order
+37
-37
+37
-37
README.md
+37
-37
README.md
···
111
111
112
112
## REST api
113
113
114
-
### management
114
+
### filter management
115
115
116
116
- `GET /filter`: get the current filter configuration.
117
117
- `PATCH /filter`: update the filter configuration.
118
118
119
-
#### ingestion control
119
+
#### filter mode
120
+
121
+
the `mode` field controls what gets indexed:
122
+
123
+
| mode | behaviour |
124
+
| :--- | :--- |
125
+
| `filter` | auto-discovers and backfills any account whose firehose commit touches a collection matching one of the `signals` patterns. you can also explicitly track individual repositories via the `/repos` endpoint regardless of matching signals. |
126
+
| `full` | index the entire network. `signals` are ignored for discovery, but `excludes` and `collections` still apply. |
127
+
128
+
#### fields
129
+
130
+
| field | type | description |
131
+
| :--- | :--- | :--- |
132
+
| `mode` | `"filter"` \| `"full"` | indexing mode (see above). |
133
+
| `signals` | set update | NSID patterns (e.g. `app.bsky.feed.post` or `app.bsky.*`) that trigger auto-discovery in `filter` mode. |
134
+
| `collections` | set update | NSID patterns used to filter which records are stored. if empty, all collections are stored. applies in all modes. |
135
+
| `excludes` | set update | set of DIDs to always skip, regardless of mode. checked before any other filter logic. |
136
+
137
+
#### set updates
138
+
139
+
each set field accepts one of two forms:
140
+
141
+
- **replace**: an array replaces the entire set, eg. `["did:plc:abc", "did:web:example.org"]`
142
+
- **patch**: an object maps items to `true` (add) or `false` (remove), eg. `{"did:plc:abc": true, "did:web:example.org": false}`
143
+
144
+
#### NSID patterns
145
+
146
+
`signals` and `collections` support an optional `.*` suffix to match an entire namespace:
147
+
148
+
- `app.bsky.feed.post`: exact match only
149
+
- `app.bsky.feed.*`: matches any collection under `app.bsky.feed`
150
+
151
+
### ingestion control
120
152
121
153
- `GET /ingestion`: get the current ingestion status.
122
154
- returns `{ "crawler": bool, "firehose": bool, "backfill": bool }`.
···
128
160
finishes processing the current message). they resume immediately when
129
161
re-enabled.
130
162
131
-
#### crawler source management
163
+
### crawler source management
132
164
133
165
- `GET /crawler/sources`: list all currently active crawler sources.
134
166
- returns a JSON array of `{ "url": string, "mode": "relay" | "by_collection", "persisted": bool }`.
···
155
187
the source to restart from the beginning when re-added.
156
188
- returns `200 OK` if the source was found and removed, `404 Not Found` otherwise.
157
189
158
-
#### firehose source management
190
+
### firehose source management
159
191
160
192
- `GET /firehose/sources`: list all currently active firehose relay sources.
161
193
- returns a JSON array of `{ "url": string, "persisted": bool }`.
···
180
212
the relay to restart from the beginning when re-added.
181
213
- returns `200 OK` if the relay was found and removed, `404 Not Found` otherwise.
182
214
183
-
#### database operations
215
+
### database operations
184
216
185
217
- `POST /db/train`: train zstd compression dictionaries for the `repos`,
186
218
`blocks`, and `events` keyspaces. dictionaries are written to disk; a restart
···
193
225
where key is a URL. clears both the firehose cursor and the relay crawler cursor,
194
226
as well as any by-collection cursors associated with that URL. causes the next
195
227
firehose connection and crawler pass to restart from the beginning.
196
-
197
-
#### filter mode
198
-
199
-
the `mode` field controls what gets indexed:
200
-
201
-
| mode | behaviour |
202
-
| :--- | :--- |
203
-
| `filter` | auto-discovers and backfills any account whose firehose commit touches a collection matching one of the `signals` patterns. you can also explicitly track individual repositories via the `/repos` endpoint regardless of matching signals. |
204
-
| `full` | index the entire network. `signals` are ignored for discovery, but `excludes` and `collections` still apply. |
205
-
206
-
#### fields
207
-
208
-
| field | type | description |
209
-
| :--- | :--- | :--- |
210
-
| `mode` | `"filter"` \| `"full"` | indexing mode (see above). |
211
-
| `signals` | set update | NSID patterns (e.g. `app.bsky.feed.post` or `app.bsky.*`) that trigger auto-discovery in `filter` mode. |
212
-
| `collections` | set update | NSID patterns used to filter which records are stored. if empty, all collections are stored. applies in all modes. |
213
-
| `excludes` | set update | set of DIDs to always skip, regardless of mode. checked before any other filter logic. |
214
-
215
-
#### set updates
216
-
217
-
each set field accepts one of two forms:
218
-
219
-
- **replace**: an array replaces the entire set, eg. `["did:plc:abc", "did:web:example.org"]`
220
-
- **patch**: an object maps items to `true` (add) or `false` (remove), eg. `{"did:plc:abc": true, "did:web:example.org": false}`
221
-
222
-
#### NSID patterns
223
-
224
-
`signals` and `collections` support an optional `.*` suffix to match an entire namespace:
225
-
226
-
- `app.bsky.feed.post`: exact match only
227
-
- `app.bsky.feed.*`: matches any collection under `app.bsky.feed`
228
228
229
229
### repository management
230
230