declarative relay deployment on hetzner
relay-eval.waow.tech
atproto
relay
add collectiondir sidecar for listReposByCollection
built from indigo, pushed to atcr.io/zzstoatzz.io/collectiondir.
subscribes to relay firehose, indexes (DID, collection) pairs in
pebble DB, serves com.atproto.sync.listReposByCollection — the
endpoint TAP crawlers need to enumerate the network.
ingress routes the endpoint + /v1/listCollections to collectiondir:2510,
relay endpoints unaffected. grafana dashboard gets collectiondir panels.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+184
-1
+6
-1
README.md
+6
-1
README.md
···
1
1
# relay.waow.tech
2
2
3
-
a full-network [ATProto](https://atproto.com) relay running on a single Hetzner Cloud node with k3s. a [jetstream](https://github.com/bluesky-social/jetstream) instance runs alongside it, re-encoding the relay's CBOR firehose into plain JSON over websockets — easier to consume if you don't need the full atproto SDK.
3
+
a full-network [ATProto](https://atproto.com) relay running on a single Hetzner Cloud node with k3s. a [jetstream](https://github.com/bluesky-social/jetstream) instance runs alongside it, re-encoding the relay's CBOR firehose into plain JSON over websockets — easier to consume if you don't need the full atproto SDK. a [collectiondir](https://github.com/bluesky-social/indigo/tree/main/cmd/collectiondir) sidecar indexes `(DID, collection)` pairs from the firehose and serves `com.atproto.sync.listReposByCollection` — the endpoint TAP crawlers need to enumerate the network.
4
4
5
5
**relay endpoint:** `wss://relay.waow.tech` — raw CBOR firehose ([`com.atproto.sync.subscribeRepos`](https://docs.bsky.app/docs/advanced-guides/firehose))
6
6
7
7
**jetstream endpoint:** `wss://jetstream.waow.tech/subscribe` — same data, JSON over websockets
8
+
9
+
**collection directory:** `https://relay.waow.tech/xrpc/com.atproto.sync.listReposByCollection` — paginated DID lists per collection
8
10
9
11
**health check:** [`https://relay.waow.tech/xrpc/_health`](https://relay.waow.tech/xrpc/_health)
10
12
···
67
69
└── deploy/ # helm values + k8s manifests
68
70
├── relay-values.yaml
69
71
├── jetstream-values.yaml
72
+
├── collectiondir-values.yaml
70
73
├── postgres-values.yaml
71
74
├── monitoring-values.yaml
72
75
├── relay-dashboard.json
73
76
├── relay-servicemonitor.yaml
74
77
├── jetstream-servicemonitor.yaml
78
+
├── collectiondir-servicemonitor.yaml
75
79
├── ingress.yaml
76
80
├── jetstream-ingress.yaml
77
81
├── grafana-ingress.yaml
···
156
160
157
161
- **relay** — [`ghcr.io/bluesky-social/indigo`](https://github.com/bluesky-social/indigo/pkgs/container/indigo) (tagged per-commit, e.g. `relay-bf41e2ee...`), deployed via [bjw-s/app-template](https://github.com/bjw-s-labs/helm-charts) helm chart with `hostNetwork: true` for lower-overhead networking
158
162
- **jetstream** — [`ghcr.io/bluesky-social/jetstream`](https://github.com/bluesky-social/jetstream) subscribes to the relay's firehose over localhost (`ws://relay:2470`) and re-serves it as JSON WebSocket events at [`jetstream.waow.tech/subscribe`](https://jetstream.waow.tech/subscribe). lightweight alternative for consumers that don't need CBOR/CAR decoding
163
+
- **collectiondir** — [`atcr.io/zzstoatzz.io/collectiondir`](https://github.com/bluesky-social/indigo/tree/main/cmd/collectiondir) subscribes to the relay firehose, indexes `(DID, collection)` pairs in a pebble DB, and serves `com.atproto.sync.listReposByCollection` at `relay.waow.tech`. routed via ingress path matching so the relay's existing endpoints are unaffected
159
164
- **postgresql** — relay's backing database, deployed via [bitnami/postgresql](https://github.com/bitnami/charts/tree/main/bitnami/postgresql) helm chart
160
165
- **prometheus + grafana** — metrics collection and dashboards via [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), public read-only access at [`relay-metrics.waow.tech`](https://relay-metrics.waow.tech)
161
166
+15
deploy/collectiondir-servicemonitor.yaml
+15
deploy/collectiondir-servicemonitor.yaml
···
1
+
apiVersion: monitoring.coreos.com/v1
2
+
kind: ServiceMonitor
3
+
metadata:
4
+
name: collectiondir
5
+
namespace: monitoring
6
+
spec:
7
+
selector:
8
+
matchLabels:
9
+
app.kubernetes.io/name: collectiondir
10
+
namespaceSelector:
11
+
matchNames:
12
+
- relay
13
+
endpoints:
14
+
- port: metrics
15
+
interval: 30s
+60
deploy/collectiondir-values.yaml
+60
deploy/collectiondir-values.yaml
···
1
+
# bjw-s/app-template helm values for collectiondir
2
+
# indexes (DID, collection) pairs from the relay firehose and serves
3
+
# com.atproto.sync.listReposByCollection for TAP crawlers
4
+
# schema: https://github.com/bjw-s-labs/helm-charts/tree/main/charts/other/app-template
5
+
6
+
controllers:
7
+
collectiondir:
8
+
containers:
9
+
main:
10
+
image:
11
+
repository: atcr.io/zzstoatzz.io/collectiondir
12
+
tag: latest
13
+
args:
14
+
- /usr/bin/collectiondir
15
+
- serve
16
+
- --pebble
17
+
- /data/collectiondir
18
+
- --dau-directory
19
+
- /data/dau
20
+
- --upstream
21
+
- ws://relay:2470
22
+
probes:
23
+
liveness: &probes
24
+
enabled: true
25
+
custom: true
26
+
spec:
27
+
httpGet:
28
+
path: /_health
29
+
port: &apiport 2510
30
+
initialDelaySeconds: 10
31
+
periodSeconds: 10
32
+
timeoutSeconds: 3
33
+
failureThreshold: 5
34
+
readiness: *probes
35
+
resources:
36
+
requests:
37
+
memory: 128Mi
38
+
cpu: 50m
39
+
limits:
40
+
memory: 512Mi
41
+
42
+
defaultPodOptions:
43
+
imagePullSecrets:
44
+
- name: atcr-creds
45
+
46
+
service:
47
+
collectiondir:
48
+
controller: collectiondir
49
+
ports:
50
+
http:
51
+
port: *apiport
52
+
metrics:
53
+
port: 2511
54
+
55
+
persistence:
56
+
data:
57
+
enabled: true
58
+
type: persistentVolumeClaim
59
+
accessMode: ReadWriteOnce
60
+
size: 10Gi
+14
deploy/ingress.yaml
+14
deploy/ingress.yaml
···
15
15
- host: RELAY_DOMAIN_PLACEHOLDER
16
16
http:
17
17
paths:
18
+
- path: /xrpc/com.atproto.sync.listReposByCollection
19
+
pathType: Exact
20
+
backend:
21
+
service:
22
+
name: collectiondir
23
+
port:
24
+
number: 2510
25
+
- path: /v1/listCollections
26
+
pathType: Prefix
27
+
backend:
28
+
service:
29
+
name: collectiondir
30
+
port:
31
+
number: 2510
18
32
- path: /
19
33
pathType: Prefix
20
34
backend:
+82
deploy/relay-dashboard.json
+82
deploy/relay-dashboard.json
···
288
288
"refId": "B"
289
289
}
290
290
]
291
+
},
292
+
{
293
+
"title": "collectiondir",
294
+
"type": "row",
295
+
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 33 },
296
+
"collapsed": false,
297
+
"panels": []
298
+
},
299
+
{
300
+
"title": "collectiondir firehose events/sec",
301
+
"type": "timeseries",
302
+
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 34 },
303
+
"datasource": { "type": "prometheus", "uid": "prometheus" },
304
+
"fieldConfig": {
305
+
"defaults": {
306
+
"unit": "ops",
307
+
"color": { "mode": "palette-classic" },
308
+
"custom": {
309
+
"fillOpacity": 15,
310
+
"lineWidth": 2,
311
+
"spanNulls": false
312
+
}
313
+
},
314
+
"overrides": []
315
+
},
316
+
"targets": [
317
+
{
318
+
"expr": "sum(rate(collectiondir_firehose_received_total[5m]))",
319
+
"legendFormat": "received",
320
+
"refId": "A"
321
+
}
322
+
]
323
+
},
324
+
{
325
+
"title": "collectiondir commits/sec",
326
+
"type": "timeseries",
327
+
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 34 },
328
+
"datasource": { "type": "prometheus", "uid": "prometheus" },
329
+
"fieldConfig": {
330
+
"defaults": {
331
+
"unit": "ops",
332
+
"color": { "mode": "palette-classic" },
333
+
"custom": {
334
+
"fillOpacity": 15,
335
+
"lineWidth": 2,
336
+
"spanNulls": false
337
+
}
338
+
},
339
+
"overrides": []
340
+
},
341
+
"targets": [
342
+
{
343
+
"expr": "sum(rate(collectiondir_firehose_commits[5m]))",
344
+
"legendFormat": "commits",
345
+
"refId": "A"
346
+
}
347
+
]
348
+
},
349
+
{
350
+
"title": "collectiondir new pairs indexed/sec",
351
+
"type": "timeseries",
352
+
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 34 },
353
+
"datasource": { "type": "prometheus", "uid": "prometheus" },
354
+
"fieldConfig": {
355
+
"defaults": {
356
+
"unit": "ops",
357
+
"color": { "mode": "palette-classic" },
358
+
"custom": {
359
+
"fillOpacity": 15,
360
+
"lineWidth": 2,
361
+
"spanNulls": false
362
+
}
363
+
},
364
+
"overrides": []
365
+
},
366
+
"targets": [
367
+
{
368
+
"expr": "sum(rate(collectiondir_pebble_new_total[5m]))",
369
+
"legendFormat": "new pairs",
370
+
"refId": "A"
371
+
}
372
+
]
291
373
}
292
374
],
293
375
"schemaVersion": 39,
+7
justfile
+7
justfile
···
132
132
sed "s|GRAFANA_DOMAIN_PLACEHOLDER|$GRAFANA_DOMAIN|g" deploy/grafana-ingress.yaml \
133
133
| kubectl apply -f -
134
134
135
+
echo "==> installing collectiondir"
136
+
helm upgrade --install collectiondir bjw-s/app-template \
137
+
--namespace relay \
138
+
--values deploy/collectiondir-values.yaml \
139
+
--wait --timeout 5m
140
+
kubectl apply -f deploy/collectiondir-servicemonitor.yaml
141
+
135
142
echo "==> installing jetstream"
136
143
JETSTREAM_DOMAIN="${JETSTREAM_DOMAIN:-jetstream.waow.tech}"
137
144
helm upgrade --install jetstream bjw-s/app-template \