this repo has no description
simplify README
+12
-168
+12
-168
cmd/relay/README.md
+12
-168
cmd/relay/README.md
···
35
35
36
36
You can re-build and run the command directly to get a list of configuration flags and env vars; env vars will be loaded from `.env` if that file exists:
37
37
38
-
RELAY_ADMIN_KEY=localdev go run ./cmd/relay/ --help
38
+
RELAY_ADMIN_PASSWORD=dummy go run ./cmd/relay/ --help
39
39
40
40
By default, the daemon will use sqlite for databases (in the directory `./data/relay/`) and the HTTP API will be bound to localhost port 2470.
41
41
···
50
50
# careful! double-check this destructive command
51
51
rm -rf ./data/relay/*
52
52
53
-
There is a basic web dashboard, though it will not be included unless built and copied to a local directory `./public/`. Run `make build-relay-ui`, and then when running the daemon the dashboard will be available at: <http://localhost:2470/dash/>. Paste in the admin key, eg `localdev`.
53
+
There is a basic web dashboard, though it will not be included unless built and copied to a local directory `./public/`. Run `make build-relay-ui`, and then when running the daemon the dashboard will be available at: <http://localhost:2470/dash/>. Paste in the admin key, eg `dummy`.
54
54
55
-
The local admin routes can also be accessed by passing the admin key as a bearer token, for example:
55
+
The local admin routes can also be accessed by passing the admin password using HTTP Basic auth (with username `admin`), for example:
56
56
57
-
http get :2470/admin/pds/list Authorization:"Bearer localdev"
57
+
http get :2470/admin/pds/list -a admin:dummy
58
58
59
59
Request crawl of an individual PDS instance like:
60
60
61
-
http post :2470/admin/pds/requestCrawl Authorization:"Bearer localdev" hostname=pds.example.com
61
+
http post :2470/admin/pds/requestCrawl -a admin:dummy hostname=pds.example.com
62
62
63
63
64
64
## Docker Containers
···
93
93
94
94
Some notable configuration env vars to set:
95
95
96
-
- `ENVIRONMENT`: eg, `production`
97
-
- `DATABASE_URL`: see section below
98
-
- `GOLOG_LOG_LEVEL`: log verbosity
96
+
- `RELAY_ADMIN_PASSWORD`
97
+
- `DATABASE_URL`: eg, `postgres://relay:CHANGEME@localhost:5432/relay`
98
+
- `RELAY_PERSIST_DIR`: storage location for "backfill" events, eg `/data/relay/persist`
99
+
- `RELAY_REPLAY_WINDOW`: the duration of output "backfill window", eg `24h`
100
+
- `RELAY_LENIENT_SYNC_VALIDATION`: if `true`, allow legacy upstreams which don't implement atproto sync v1.1
101
+
- `RELAY_TRUSTED_DOMAINS`: patterns of PDS hosts which get larger quotas by default, eg `*.host.bsky.network`
99
102
100
-
There is a health check endpoint at `/xrpc/_health`. Prometheus metrics are exposed by default on port 2471, path `/metrics`. The service logs fairly verbosely to stderr; use `GOLOG_LOG_LEVEL` to control log volume.
103
+
There is a health check endpoint at `/xrpc/_health`. Prometheus metrics are exposed by default on port 2471, path `/metrics`. The service logs fairly verbosely to stdout; use `LOG_LEVEL` to control log volume (`warn`, `info`, etc).
101
104
102
105
Be sure to double-check bandwidth usage and pricing if running a public relay! Bandwidth prices can vary widely between providers, and popular cloud services (AWS, Google Cloud, Azure) are very expensive compared to alternatives like OVH or Hetzner.
103
-
104
-
105
-
## Bootstrapping the Network
106
-
107
-
To bootstrap the entire network, you'll want to start with a list of large PDS instances to backfill from. You could pull from a public dashboard of instances (like [mackuba's](https://blue.mackuba.eu/directory/pdses)), or scrape the full DID PLC directory, parse out all PDS service declarations, and sort by count.
108
-
109
-
Once you have a set of PDS hosts, you can put the bare hostnames (not URLs: no `https://` prefix, port, or path suffix) in a `hosts.txt` file, and then use the `crawl_pds.sh` script to backfill and configure limits for all of them:
110
-
111
-
export RELAY_HOST=your.pds.hostname.tld
112
-
export RELAY_ADMIN_KEY=your-secret-key
113
-
114
-
# both request crawl, and set generous crawl limits for each
115
-
cat hosts.txt | parallel -j1 ./crawl_pds.sh {}
116
-
117
-
Just consuming from the firehose for a few hours will only backfill accounts with activity during that period. This is fine to get the backfill process started, but eventually you'll want to do full "resync" of all the repositories on the PDS host to the most recent repo rev version. To enqueue that for all the PDS instances:
118
-
119
-
# start sync/backfill of all accounts
120
-
cat hosts.txt | parallel -j1 ./sync_pds.sh {}
121
-
122
-
Lastly, can monitor progress of any ongoing re-syncs:
123
-
124
-
# check sync progress for all hosts
125
-
cat hosts.txt | parallel -j1 ./sync_pds.sh {}
126
-
127
-
128
-
## Admin API
129
-
130
-
The relay has a number of admin HTTP API endpoints. Given a relay setup listening on port 2470 and with a reasonably secure admin secret:
131
-
132
-
```
133
-
RELAY_ADMIN_PASSWORD=$(openssl rand --hex 16)
134
-
relay --api-listen :2470 --admin-key ${RELAY_ADMIN_PASSWORD} ...
135
-
```
136
-
137
-
One can, for example, begin compaction of all repos
138
-
139
-
```
140
-
curl -H 'Authorization: Bearer '${RELAY_ADMIN_PASSWORD} -H 'Content-Type: application/x-www-form-urlencoded' --data '' http://127.0.0.1:2470/admin/repo/compactAll
141
-
```
142
-
143
-
### /admin/subs/getUpstreamConns
144
-
145
-
Return list of PDS host names in json array of strings: ["host", ...]
146
-
147
-
### /admin/subs/perDayLimit
148
-
149
-
Return `{"limit": int}` for the number of new PDS subscriptions that the relay may start in a rolling 24 hour window.
150
-
151
-
### /admin/subs/setPerDayLimit
152
-
153
-
POST with `?limit={int}` to set the number of new PDS subscriptions that the relay may start in a rolling 24 hour window.
154
-
155
-
### /admin/subs/setEnabled
156
-
157
-
POST with param `?enabled=true` or `?enabled=false` to enable or disable PDS-requested new-PDS crawling.
158
-
159
-
### /admin/subs/getEnabled
160
-
161
-
Return `{"enabled": bool}` if non-admin new PDS crawl requests are enabled
162
-
163
-
### /admin/subs/killUpstream
164
-
165
-
POST with `?host={pds host name}` to disconnect from their firehose.
166
-
167
-
Optionally add `&block=true` to prevent connecting to them in the future.
168
-
169
-
### /admin/subs/listDomainBans
170
-
171
-
Return `{"banned_domains": ["host name", ...]}`
172
-
173
-
### /admin/subs/banDomain
174
-
175
-
POST `{"Domain": "host name"}` to ban a domain
176
-
177
-
### /admin/subs/unbanDomain
178
-
179
-
POST `{"Domain": "host name"}` to un-ban a domain
180
-
181
-
### /admin/repo/takeDown
182
-
183
-
POST `{"did": "did:..."}` to take-down a bad repo; deletes all local data for the repo
184
-
185
-
### /admin/repo/reverseTakedown
186
-
187
-
POST `?did={did:...}` to reverse a repo take-down
188
-
189
-
### /admin/pds/requestCrawl
190
-
191
-
POST `{"hostname":"pds host"}` to start crawling a PDS
192
-
193
-
### /admin/pds/list
194
-
195
-
GET returns JSON list of records
196
-
```json
197
-
[{
198
-
"Host": string,
199
-
"Did": string,
200
-
"SSL": bool,
201
-
"Cursor": int,
202
-
"Registered": bool,
203
-
"Blocked": bool,
204
-
"RateLimit": float,
205
-
"CrawlRateLimit": float,
206
-
"RepoCount": int,
207
-
"RepoLimit": int,
208
-
"HourlyEventLimit": int,
209
-
"DailyEventLimit": int,
210
-
211
-
"HasActiveConnection": bool,
212
-
"EventsSeenSinceStartup": int,
213
-
"PerSecondEventRate": {"Max": float, "Window": float seconds},
214
-
"PerHourEventRate": {"Max": float, "Window": float seconds},
215
-
"PerDayEventRate": {"Max": float, "Window": float seconds},
216
-
"CrawlRate": {"Max": float, "Window": float seconds},
217
-
"UserCount": int,
218
-
}, ...]
219
-
```
220
-
221
-
### /admin/pds/changeLimits
222
-
223
-
POST to set the limits for a PDS. body:
224
-
225
-
```json
226
-
{
227
-
"host": string,
228
-
"per_second": int,
229
-
"per_hour": int,
230
-
"per_day": int,
231
-
"crawl_rate": int,
232
-
"repo_limit": int,
233
-
}
234
-
```
235
-
236
-
### /admin/pds/block
237
-
238
-
POST `?host={host}` to block a PDS
239
-
240
-
### /admin/pds/unblock
241
-
242
-
POST `?host={host}` to un-block a PDS
243
-
244
-
245
-
### /admin/pds/addTrustedDomain
246
-
247
-
POST `?domain={}` to make a domain trusted
248
-
249
-
### /admin/consumers/list
250
-
251
-
GET returns list json of clients currently reading from the relay firehose
252
-
253
-
```json
254
-
[{
255
-
"id": int,
256
-
"remote_addr": string,
257
-
"user_agent": string,
258
-
"events_consumed": int,
259
-
"connected_at": time,
260
-
}, ...]
261
-
```