very fast at protocol indexer with flexible filtering, xrpc queries, cursor-backed event stream, and more, built on fjall
rust
fjall
at-protocol
atproto
indexer
[docs] better instructions for hosting in the readme
+68
-1
+68
-1
README.md
+68
-1
README.md
···
2
2
3
3
-> [hydrant](#hydrant)</br>
4
4
-> [vs tap](#vs-tap) | [stream](#stream-behavior) | [multi-relay](#multiple-relay-support) | [seeding](#firehose-seeding) | [crawler sources](#crawler-sources)</br>
5
-
-> [configuration](#configuration) | [build features](#build-features)</br>
5
+
-> [building](#building-and-running) | [proxying](#reverse-proxying) | [configuration](#configuration) | [build features](#build-features)</br>
6
6
-> [rest api](#rest-api) | [filter](#filter-management) | [ingestion](#ingestion-control) | [crawler](#crawler-management) | [firehose](#firehose-management) | [pds](#pds-management) | [repos](#repository-management)</br>
7
7
-> [xrpc api](#data-access-xrpc) | [atproto](#comatproto) | [backlinks](#bluemicrocosmlinks) | [identity](#bluemicrocosmidentity) | [custom](#systemsgazehydrant)
8
8
···
132
132
database and survive restarts. `CRAWLER_URLS` sources are startup-only: they are
133
133
not written to the database and will always reappear after a restart regardless of
134
134
runtime changes (unless you change the config of course).
135
+
136
+
## building and running
137
+
138
+
<small>[<- back to toc](#table-of-contents)</small>
139
+
140
+
hydrant is written in rust and requires the rust toolchain (including `cargo`), `make`, `cmake`
141
+
for some dependencies. you will also need the clang toolchain and the [wild linker](https://github.com/wild-linker/wild).
142
+
143
+
### from source
144
+
145
+
to build a production binary:
146
+
147
+
```bash
148
+
cargo build --release
149
+
```
150
+
151
+
the binary will be located at `target/release/hydrant`.
152
+
153
+
#### build features
154
+
155
+
see [build features](#build-features) for optional features (like `relay` or `backlinks`). to build with a specific feature:
156
+
157
+
```bash
158
+
cargo build --release --features backlinks
159
+
```
160
+
161
+
### running
162
+
163
+
you can run hydrant by executing the binary. make sure to provide the necessary
164
+
environment variables (see [configuration](#configuration)).
165
+
166
+
```bash
167
+
export HYDRANT_DATABASE_PATH=./hydrant.db
168
+
./target/release/hydrant
169
+
```
170
+
171
+
### reverse proxying
172
+
173
+
<small>[<- back to toc](#table-of-contents)</small>
174
+
175
+
it is **highly recommended** to run hydrant behind a reverse proxy (like nginx or
176
+
caddy) if you intend to expose the XRPC or event stream APIs to the public. hydrant's
177
+
API includes several management endpoints that do not require or support authentication.
178
+
**you MUST NOT expose these management endpoints to the public internet.**
179
+
180
+
#### public endpoints (safe to proxy)
181
+
182
+
you should only expose the following paths:
183
+
184
+
- `GET /xrpc/*`: XRPC endpoints.
185
+
- `GET /stream`: hydrant's ordered event stream.
186
+
- `GET /stats`: general database statistics.
187
+
- `GET /health`: simple health check.
188
+
189
+
#### management endpoints (keep private)
190
+
191
+
the following endpoints allow modifying the indexer state and should be kept internal:
192
+
193
+
- `/repos`: explicit repository tracking/resyncing/untracking.
194
+
- `/filter`: management of NSID filter patterns.
195
+
- `/ingestion`: manual control over component lifecycle (crawler, firehose, etc.).
196
+
- `/crawler/sources`: management of crawler relays.
197
+
- `/firehose/sources`: management of firehose relays.
198
+
- `/pds/tiers`: rate-limit tier assignments.
199
+
- `/db/train` / `/db/compact`: database maintenance tasks.
200
+
- `*/cursors`: cursor management.
201
+
- `/debug/*`: introspection and testing endpoints (only when enabled).
135
202
136
203
## configuration
137
204