declarative relay deployment on hetzner
relay-eval.waow.tech
atproto
relay
add coverage comparison doc, relay tuning, zlay build improvements
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+163
-7
+5
-3
deploy/relay-values.yaml
+5
-3
deploy/relay-values.yaml
···
12
12
# DATABASE_URL injected from secret via envFrom
13
13
RELAY_PERSIST_DIR: /data
14
14
RELAY_REPLAY_WINDOW: "2h"
15
-
RELAY_IDENT_CACHE_SIZE: "500000"
15
+
RELAY_IDENT_CACHE_SIZE: "2000000"
16
16
LOG_LEVEL: "info"
17
-
GOMEMLIMIT: "6GiB"
17
+
GOMEMLIMIT: "3GiB"
18
+
GOMAXPROCS: "4"
18
19
envFrom:
19
20
- secretRef:
20
21
name: relay-secret
···
36
37
memory: 1Gi
37
38
cpu: 500m
38
39
limits:
39
-
memory: 8Gi
40
+
memory: 4Gi
41
+
cpu: "4"
40
42
41
43
defaultPodOptions:
42
44
# hostNetwork recommended for full-network relays (high packet volume).
+152
docs/coverage-comparison-2026-03-04.md
+152
docs/coverage-comparison-2026-03-04.md
···
1
+
# relay coverage comparison — 2026-03-04
2
+
3
+
## relays tested
4
+
5
+
| relay | operator | firehose | collectiondir | notes |
6
+
|---|---|---|---|---|
7
+
| **relay.waow.tech** | nate (Go/indigo) | yes | yes | Ashburn VA |
8
+
| **zlay.waow.tech** | nate (Zig) | yes | yes | Hillsboro OR |
9
+
| **bsky.network** | Bluesky | yes | yes | reference relay |
10
+
| **europe.firehose.network** | sri | yes | **no** | firehose only |
11
+
| **atproto.africa** | BlackSky | yes | **no** | firehose only |
12
+
13
+
sri and blacksky return empty/404 for `listReposByCollection` — pure firehose relays.
14
+
15
+
## tools used
16
+
17
+
- **Pulsar** ([mackuba](https://tangled.org/mackuba.eu/pulsar)) — subscribes to multiple relay firehoses simultaneously, counts events + unique DIDs over a time window
18
+
- **relay-compare** ([evan](https://tangled.org/evan.jarrett.net/at-container-registry)) — compares `listReposByCollection` across relays for `io.atcr.*` collections, with `--verify` to check PDS for ghosts/deactivated accounts
19
+
- **tap** ([bluesky-social/indigo/cmd/tap](https://github.com/bluesky-social/indigo/tree/main/cmd/tap)) — atproto sync tool that uses `listReposByCollection` for repo discovery, then backfills repos from PDS and subscribes to the firehose
20
+
21
+
---
22
+
23
+
## 1. firehose coverage (Pulsar, 2 min window)
24
+
25
+
| relay | events | unique DIDs |
26
+
|---|---|---|
27
+
| europe.firehose.network | 45,254 | 17,649 |
28
+
| zlay.waow.tech | 44,999 | 17,476 |
29
+
| relay.waow.tech | 44,871 | 17,452 |
30
+
31
+
All three within ~1% of each other. sri edges on raw events, but DID coverage is essentially tied.
32
+
33
+
---
34
+
35
+
## 2. collection directory — io.atcr.* (relay-compare with --verify)
36
+
37
+
| collection | relay | zlay | bsky | winner |
38
+
|---|---|---|---|---|
39
+
| io.atcr.manifest | 19 | **25** | 25 | zlay = bsky |
40
+
| io.atcr.tag | 19 | **25** | 25 | zlay = bsky |
41
+
| io.atcr.sailor.profile | 82 | **90** | 88 | **zlay** (+2) |
42
+
| io.atcr.sailor.star | 13 | 14 | 14 | tied |
43
+
| io.atcr.repo.page | 17 | 19 | 20 | bsky (+1) |
44
+
| io.atcr.hold.* | in sync | in sync | in sync | tied |
45
+
46
+
relay's deficit is ghosts (4) + deactivated accounts (3) it hasn't pruned. zlay and bsky are nearly identical, with zlay ahead on sailor.profile.
47
+
48
+
---
49
+
50
+
## 3. collection directory — all indie NSIDs (lexicon garden)
51
+
52
+
Paginated counts where initial query hit 1000 cap.
53
+
54
+
### zlay leads
55
+
56
+
| collection | relay | zlay | bsky | notes |
57
+
|---|---|---|---|---|
58
+
| com.whtwnd.blog.entry | 427 | **698** | 530 | zlay +32% over bsky |
59
+
| blue.linkat.board | 303 | **447** | 303 | zlay +47% |
60
+
| app.bsky.labeler.service | 502 | **667** | 416 | zlay +60% over bsky |
61
+
| com.shinolabs.pinksea.oekaki | 311 | **481** | 313 | zlay +54% |
62
+
| place.stream.broadcast.origin | 330 | **518** | 517 | zlay = bsky |
63
+
| blue.trilinesat.like | 54 | **100** | 100 | zlay = bsky |
64
+
| blue.trilinesat.diary | 83 | **141** | 141 | zlay = bsky |
65
+
| site.standard.graph.subscription | 723 | **1,347** | 1,346 | zlay = bsky |
66
+
| site.standard.publication | 1,155 | **2,000** | 2,069 | bsky slightly ahead |
67
+
| social.drydown.* | 7 | **12-13** | 12-13 | zlay = bsky |
68
+
| app.fitsky.* | 1-5 | **2-6** | 2-6 | zlay = bsky |
69
+
70
+
### tied / close
71
+
72
+
| collection | relay | zlay | bsky |
73
+
|---|---|---|---|
74
+
| fyi.unravel.frontpage.post | **75** | 75 | 73 |
75
+
| fyi.unravel.frontpage.vote | 189 | 189 | 189 |
76
+
| tech.manos.twmirror.post | 5 | 6 | 6 |
77
+
| cx.vmx.matadisco | 1 | 1 | 1 |
78
+
| za.co.ciaran.cumulus.* | 1 | 1 | 1 |
79
+
80
+
### relay leads
81
+
82
+
| collection | relay | zlay | bsky |
83
+
|---|---|---|---|
84
+
| xyz.statusphere.status | **847** | 815 | 815 |
85
+
86
+
### zlay gaps (0 results where others have data)
87
+
88
+
| collection | relay | zlay | bsky |
89
+
|---|---|---|---|
90
+
| fyi.unravel.frontpage.comment | 78 | **0** | 43 |
91
+
| social.psky.feed.post | 42 | **0** | 0 |
92
+
| studio.voyager.account.autonomy | 11 | **0** | 13 |
93
+
| xyz.atpoke.feed.poke | 1 | **0** | 2 |
94
+
| com.crabdance.nandi.post | 1 | **0** | 2 |
95
+
96
+
These are long-tail collections that zlay's backfill hasn't picked up yet — either not in lexicon garden's llms.txt or not on bsky.network either (psky).
97
+
98
+
### bsky leads
99
+
100
+
| collection | relay | zlay | bsky |
101
+
|---|---|---|---|
102
+
| fm.plyr.actor.profile | 92 | 93 | **146** |
103
+
| site.standard.document | 667 | 6,049 | **6,078** |
104
+
105
+
---
106
+
107
+
## 4. tap — sync correctness
108
+
109
+
### io.atcr.manifest (small collection)
110
+
111
+
| relay | repos discovered | records synced | resync buffer | firehose |
112
+
|---|---|---|---|---|
113
+
| relay | 19 | 99,997 | 0 | streaming |
114
+
| zlay | **25** | **100,075** | 0 | streaming |
115
+
| bsky | 25 | 100,017 | 0 | streaming |
116
+
117
+
### fm.plyr.actor.profile (larger collection)
118
+
119
+
| relay | repos discovered | records synced | resync buffer |
120
+
|---|---|---|---|
121
+
| relay | 92 | 19,448 | 0 |
122
+
| zlay | 93 | 25,401 | 0 |
123
+
| bsky | **146** | **70,945** | 0 |
124
+
125
+
All three had zero resync buffer (fully caught up), zero out-of-order sequence errors on both collections. The monotonic sequence fix is holding.
126
+
127
+
---
128
+
129
+
## summary
130
+
131
+
### zlay wins
132
+
133
+
- collection directory coverage on most indie NSIDs — often 30-60% more DIDs than relay, matching or beating bsky.network
134
+
- firehose parity with all tested relays (within 1%)
135
+
- zero sequence ordering errors (monotonic fix confirmed)
136
+
- half the memory of the Go relay
137
+
138
+
### zlay gaps
139
+
140
+
- 5 long-tail collections with zero results (frontpage.comment, psky, voyager, atpoke, crabdance)
141
+
- fm.plyr.actor.profile trails bsky by ~36% (backfill coverage)
142
+
- site.standard.document: 6,049 vs bsky's 6,078 (nearly identical)
143
+
144
+
### relay (Go/indigo) gaps
145
+
146
+
- trails on most collection directory counts due to uncleared ghosts/deactivated accounts
147
+
- statusphere is the one collection where relay leads (847 vs 815)
148
+
149
+
### sri & blacksky
150
+
151
+
- firehose-only relays, no collection directory support
152
+
- firehose coverage comparable to everyone else
+6
-4
justfile
+6
-4
justfile
···
365
365
echo "==> pushed ${IMAGE}"
366
366
367
367
# build zlay on the server and import into k3s containerd (fast — native x86_64 build)
368
-
zlay-publish-remote:
368
+
# usage: just zlay-publish-remote (debug build)
369
+
# just zlay-publish-remote ReleaseSafe (optimized — needs 8 MiB stacks)
370
+
zlay-publish-remote optimize="":
369
371
#!/usr/bin/env bash
370
372
set -euo pipefail
371
373
ssh root@$(just zlay-server-ip) <<'DEPLOY'
···
374
376
git pull --ff-only
375
377
376
378
TAG=$(git rev-parse --short HEAD)
377
-
IMAGE="atcr.io/zzstoatzz.io/zlay:${TAG}"
379
+
IMAGE="atcr.io/zzstoatzz.io/zlay:{{ if optimize != "" { optimize + "-" } else { "debug-" } }}${TAG}"
378
380
379
-
echo "==> building binary (${TAG})"
380
-
zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux-gnu
381
+
echo "==> building binary (${TAG}{{ if optimize != "" { ", " + optimize } else { ", debug" } }})"
382
+
zig build {{ if optimize != "" { "-Doptimize=" + optimize + " " } else { "" } }}-Dtarget=x86_64-linux-gnu -Dcpu=baseline
381
383
382
384
echo "==> building container image (${IMAGE})"
383
385
buildah bud -t "${IMAGE}" -f Dockerfile.runtime .