atproto relay implementation in zig
zlay.waow.tech
fix: reduce thread stacks to 4 MiB to prevent OOM
8 MiB × 2,500 threads touches too many pages over time — RSS grew
monotonically to 8 GiB (3 OOM kills in 6 hours). 4 MiB is 2x the
proven debug floor. also updates incident doc to reflect current state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+15
-31
+12
-28
docs/incident-2026-03-04.md
+12
-28
docs/incident-2026-03-04.md
···
78
78
2. **exact breakdown** of the 3.9 MiB per thread — how much is stack pages vs thread-local heap
79
79
3. **whether `-fno-inline` or `noinline` annotations** could reduce per-thread RSS enough
80
80
81
-
## current state
81
+
## current state (2026-03-05)
82
82
83
-
- **running image**: `debug-8a0fb1f` (debug build, commit 8a0fb1f)
84
-
- **image digest**: `sha256:fd21eae467ac99772719b2d4119f6115cd4b4e66041f6911b4450e2fdc6beb3b`
85
83
- **optimize**: Debug (no optimization)
86
-
- **probes**: liveness and readiness both on `/_health:3000` (old helm values still active since deploy failed)
87
-
- **connected hosts**: ~2,654
84
+
- **thread stacks**: 4 MiB (reduced from 8 MiB — 8 MiB caused OOM at ~2,500 threads)
85
+
- **probes**: liveness `/_healthz:3000`, readiness `/_readyz:3000` (helm synced)
86
+
- **helm**: values applied via `helm upgrade`, no longer drifted from kubectl patches
87
+
- **reconnect cronjob**: deployed — reconciles PDS host list every 4 hours
88
+
- **grafana dashboard**: configmap synced from `deploy/zlay-dashboard.json`
88
89
- **metrics**: working — RSS, threads, disk, counters, caches all emitting
89
-
- **missing metrics**: `VmHWM`, `RssAnon`, `mallinfo` — proc parsing silently fails (not debugged yet)
90
-
- **grafana dashboard**: partially populated
91
-
92
-
## commits on zlay main
93
90
94
-
```
95
-
6925fa0 fix: stabilize probes, consolidate stacks, add build_info metric ← NOT DEPLOYED
96
-
8a0fb1f feat: restore missing prometheus metrics ← DEPLOYED (debug build)
97
-
df0e53a fix: 8 MiB thread stacks
98
-
952de0c fix: use 4 MiB stacks for all spawned threads
99
-
```
91
+
## notable commits
100
92
101
-
## commits on relay main
102
-
103
-
```
104
-
e111e47 fix: split liveness/readiness probes, raise tolerances ← NOT DEPLOYED (helm timed out)
105
-
dc29553 fix: update dashboard — remove nonexistent metrics panels
106
-
```
93
+
see `git log` for full history. key milestones:
94
+
- `952de0c` — first 4 MiB stack attempt (ReleaseSafe, overflowed)
95
+
- `8a0fb1f` — prometheus metrics restored
96
+
- `6925fa0` — probe consolidation, build_info metric
97
+
- `1d80d84` — metrics server tryLock fix (HEAD at time of 4 MiB stack reduction)
107
98
108
99
## what to do next
109
-
110
-
### immediate (before any more deploys)
111
-
1. wait for `debug-6925fa0` build to finish on server
112
-
2. deploy it with `kubectl set image` (NOT helm) to avoid probe mismatch
113
-
3. verify `/_healthz` and `build_info` metric work
114
-
4. THEN apply helm values (probe paths will match)
115
-
5. update grafana dashboard configmap
116
100
117
101
### investigate (not in prod)
118
102
1. figure out why `VmHWM`/`RssAnon`/`mallinfo` metrics silently fail
+3
-3
src/main.zig
+3
-3
src/main.zig
···
37
37
const log = std.log.scoped(.relay);
38
38
39
39
/// zig's default thread stack is 16 MB. with ~2,750 subscriber threads that's
40
-
/// 44 GB of virtual memory. 8 MB accommodates ReleaseSafe's inlined TLS/crypto
41
-
/// stack frames which overflow at 4 MB.
42
-
pub const default_stack_size = 8 * 1024 * 1024;
40
+
/// 44 GB of virtual memory. 4 MB is 2x the proven debug floor (2 MB worked,
41
+
/// 1 MB overflowed). ReleaseSafe needs 8 MB — but we only run debug builds.
42
+
pub const default_stack_size = 4 * 1024 * 1024;
43
43
44
44
var shutdown_flag: std.atomic.Value(bool) = .{ .raw = false };
45
45