atproto relay implementation in zig zlay.waow.tech
9
fork

Configure Feed

Select the types of activity you want to include in your feed.

add docs/stdlib-patches.md: catalog of zig stdlib patches and workarounds for Evented backend

covers the Uring networking patch, DNS fallback, ReleaseSafe GPF,
debug_io override, Io.Event single-waiter, and cross-Io Mutex issues.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

+204 -40
+12 -40
NOTES.md
··· 197 197 198 198 ## stdlib patches 199 199 200 - zlay patches `lib/std/Io/Uring.zig` at build time (see `patches/uring-networking.patch`, 201 - applied in `Dockerfile`). this is necessary because the upstream zig 0.16 stdlib ships 202 - these networking operations as `*Unavailable` stubs that return `error.NetworkDown`. 200 + six stdlib issues required patches or workarounds to run on the Evented 201 + (io_uring) backend. see **[docs/stdlib-patches.md](docs/stdlib-patches.md)** 202 + for the full writeup: what's patched, what's worked around, and what upstream 203 + changes would let us drop each one. 203 204 204 - ### what's patched 205 + summary: 205 206 206 - | function | opcode | why stubbed upstream | 207 + | # | issue | fix type | 207 208 |---|---|---| 208 - | `netListenIp` | sync `bind()` + `listen()` | IORING_OP_BIND/LISTEN require kernel 6.11+ | 209 - | `netAccept` | `IORING_OP_ACCEPT` | was unimplemented | 210 - | `netConnectIp` | `IORING_OP_CONNECT` | was unimplemented | 211 - | `netSend` | `IORING_OP_SENDMSG` | was unimplemented | 212 - | `netRead` | `IORING_OP_READV` / `IORING_OP_READ` | was unimplemented | 213 - | `netWrite` | `IORING_OP_SENDMSG` | was unimplemented | 214 - 215 - also adds a `connect()` helper and `netSendOne()` for individual message sending. 216 - 217 - ### why not upstream yet 218 - 219 - the Uring networking layer is under active development (see zig issue #31723). 220 - the patch uses sync syscalls for `bind`/`listen` (not io_uring opcodes) because 221 - those opcodes require kernel 6.11+ and production runs on older kernels. this is 222 - a pragmatic choice that may not match upstream's desired API shape. 223 - 224 - ### how it's applied 225 - 226 - ```dockerfile 227 - # Dockerfile, line 11-13 228 - COPY patches/ patches/ 229 - RUN patch /opt/zig-.../lib/std/Io/Uring.zig < patches/uring-networking.patch 230 - ``` 231 - 232 - pinned to zig `0.16.0-dev.3059+42e33db9d`. any zig version bump requires 233 - regenerating the patch. 234 - 235 - ### other stdlib issues hit (not patched, worked around) 236 - 237 - | issue | workaround | 238 - |---|---| 239 - | `Io.Event.reset()` assumes no pending waiters — panics under contention | replaced with futex counter in pg.zig fork (crash 2) | 240 - | `Io.Uring` GPFs under `ReleaseSafe` (aggressive inlining + fiber context) | build with `ReleaseFast` only (`Dockerfile` line 21) | 241 - | `Io.Mutex` / futex cannot cross Io types | run Threaded workloads on plain `std.Thread` (crashes 1, 6, 8) | 242 - | `std_options.debug_io` is single-threaded by default | override in root source file for multi-threaded contexts | 209 + | 1 | Uring networking stubs (`error.NetworkDown`) | patch (`patches/uring-networking.patch`) | 210 + | 2 | DNS resolution (`netLookup`) missing | Threaded fallback in subscriber | 211 + | 3 | ReleaseSafe GPF in fiber context | build with `ReleaseFast` | 212 + | 4 | `debug_io` single-threaded default | `std_options_debug_threaded_io` override | 213 + | 5 | `Io.Event.reset()` single-waiter panic | futex counter in pg.zig fork | 214 + | 6 | cross-Io Mutex/futex incompatibility | Io segregation (plain threads for Threaded work) | 243 215 244 216 ## where things live 245 217
+192
docs/stdlib-patches.md
··· 1 + # zig stdlib patches for the Evented backend 2 + 3 + zlay runs on `Io.Evented` (io_uring fiber scheduler) for network I/O. the 4 + upstream zig 0.16-dev stdlib (`0.16.0-dev.3059+42e33db9d`) ships several 5 + Uring networking operations as stubs that return `error.NetworkDown`. zlay 6 + patches these at build time and works around other stdlib limitations. 7 + 8 + this document tracks what we had to change, why, and what upstream work 9 + would let us drop each workaround. 10 + 11 + ## patch 1: Uring networking (`patches/uring-networking.patch`) 12 + 13 + **applied in**: `Dockerfile` line 13, patches `lib/std/Io/Uring.zig` 14 + 15 + the upstream stdlib has these functions stubbed as `*Unavailable`: 16 + 17 + ``` 18 + netListenIpUnavailable → return error.NetworkDown 19 + netAcceptUnavailable → return error.NetworkDown 20 + netConnectIpUnavailable → return error.NetworkDown 21 + netSendUnavailable → return error.NetworkDown 22 + netReadUnavailable → return error.NetworkDown 23 + netWriteUnavailable → return error.NetworkDown 24 + ``` 25 + 26 + without these, `Io.Evented` can init but any TCP operation fails immediately. 27 + the patch replaces all six with working implementations: 28 + 29 + | function | io_uring opcode | notes | 30 + |---|---|---| 31 + | `netListenIp` | sync `bind()` + `listen()` | IORING_OP_BIND/LISTEN need kernel 6.11+, so we use sync syscalls | 32 + | `netAccept` | `IORING_OP_ACCEPT` | fiber yields until connection arrives | 33 + | `netConnectIp` | `IORING_OP_CONNECT` | + socket creation via existing `ev.socket()` | 34 + | `netSend` | `IORING_OP_SENDMSG` | iterates message array, one SENDMSG per message | 35 + | `netRead` | `IORING_OP_READV` or `IORING_OP_READ` | scatter read; single-buffer fast path | 36 + | `netWrite` | `IORING_OP_SENDMSG` | gather write with iovec assembly + splat pattern handling | 37 + 38 + the patch also adds two helpers: 39 + - `connect()` — submits `IORING_OP_CONNECT` SQE, handles retry on `EINTR`/`ECANCELED` 40 + - `netSendOne()` — sends a single `OutgoingMessage` via `IORING_OP_SENDMSG` 41 + 42 + ### why sync bind/listen 43 + 44 + `IORING_OP_BIND` and `IORING_OP_LISTEN` were added in linux 6.11. production 45 + runs on bookworm (kernel 6.1). `bind()` and `listen()` are fast synchronous 46 + calls anyway — no benefit from async submission. the rest of the networking 47 + stack (accept, connect, read, write) uses proper io_uring async ops. 48 + 49 + ### why not upstream yet 50 + 51 + tracked as zig issue #31723. the Uring networking layer is under active 52 + development. our patch makes pragmatic choices (sync bind/listen, specific 53 + error mappings) that may not match upstream's desired API shape. we'd want 54 + to align with whatever design decisions the zig team makes before submitting. 55 + 56 + ### regenerating the patch 57 + 58 + the patch is pinned to zig `0.16.0-dev.3059+42e33db9d`. any zig version 59 + bump requires checking if the Uring.zig source changed and regenerating. 60 + 61 + ```bash 62 + # to regenerate after a zig update: 63 + diff -u /path/to/old-zig/lib/std/Io/Uring.zig /path/to/patched/Uring.zig > patches/uring-networking.patch 64 + ``` 65 + 66 + ## workaround 2: DNS resolution via Threaded fallback 67 + 68 + **not patched** — worked around in application code. 69 + 70 + `Io.Uring` does not implement `netLookup` (DNS resolution). instead of 71 + patching it, subscribers route DNS through `pool_io` (Threaded): 72 + 73 + ```zig 74 + // subscriber.zig:326-330 75 + // DNS + TCP connect through pool_io (Threaded — has working netLookup). 76 + const dns_io = self.pool_io orelse self.io; 77 + const net_stream = try host_name.connect(dns_io, 443, .{ .mode = .stream }); 78 + ``` 79 + 80 + this works because `Io.Threaded.netLookup` uses `getaddrinfo` on a worker 81 + thread. the resulting socket handle is then used with Evented I/O for 82 + the actual data transfer (reads/writes go through the patched Uring ops). 83 + 84 + **upstream fix**: implement `netLookup` in Uring.zig, probably by submitting 85 + the blocking `getaddrinfo` call on an io_uring worker thread 86 + (`IORING_OP_POLL_ADD` + thread pool, or the newer `IORING_OP_GETXATTR` 87 + pattern). not blocking — the Threaded fallback is fine. 88 + 89 + ## workaround 3: ReleaseSafe GPF in Uring fiber context 90 + 91 + **not patched** — worked around by building with `ReleaseFast`. 92 + 93 + ```dockerfile 94 + # Dockerfile line 21-23 95 + # ReleaseFast (not ReleaseSafe): Io.Uring fiber context-switch GPFs under ReleaseSafe 96 + RUN zig build -Doptimize=ReleaseFast ... 97 + ``` 98 + 99 + under `ReleaseSafe`, the optimizer's inlining interacts badly with Uring's 100 + fiber context-switch machinery. the result is a general protection fault 101 + during normal fiber yield/resume. `ReleaseFast` and `Debug` both work. 102 + `repro_evented.zig` in the repo root reproduces this — three simple fiber 103 + tests (no-sleep, yield, sleep) that pass under Debug and ReleaseFast but 104 + GPF under ReleaseSafe. 105 + 106 + this is likely a zig codegen/optimizer bug. we haven't filed it yet because 107 + the reproduction is minimal but the root cause analysis is incomplete — 108 + could be a safety check reading stale fiber stack, or an inlining decision 109 + that breaks the stack-swap assumptions. 110 + 111 + **upstream fix**: file a bug with the repro. probably a zig compiler issue, 112 + not an Uring.zig issue. 113 + 114 + ## workaround 4: `std_options.debug_io` single-threaded default 115 + 116 + **not patched** — worked around in `src/main.zig`. 117 + 118 + ```zig 119 + // main.zig:62-64 120 + var debug_threaded_io: Io.Threaded = undefined; 121 + pub const std_options_debug_threaded_io: ?*Io.Threaded = &debug_threaded_io; 122 + ``` 123 + 124 + `std.debug.print` internally uses an `Io`-managed lock for output 125 + serialization. the default (`debug_io = null`) assumes single-threaded 126 + execution. zlay has multiple OS threads (frame worker pool, GC thread, 127 + resyncer thread) that all call `std.debug.print` / `log.*`. without this 128 + override, concurrent debug prints corrupt each other or deadlock. 129 + 130 + **upstream fix**: arguably the default should be safe for multi-threaded 131 + programs. but explicit opt-in is reasonable — it requires initializing an 132 + `Io.Threaded` instance at startup which has a cost. 133 + 134 + ## workaround 5: `Io.Event.reset()` single-waiter assumption 135 + 136 + **not patched** — worked around in pg.zig fork. 137 + 138 + `Io.Event` has a `reset()` method with a stdlib invariant (Io.zig:1857): 139 + it assumes no pending call to `wait`. when multiple threads contend for 140 + a pooled resource (pg.Pool connections), `set()` wakes all waiters, one 141 + calls `reset()`, and the others hit `unreachable`. 142 + 143 + the pg.zig fork (`5ce2355`, dev branch) replaced `Io.Event` with a 144 + monotonic `u32` futex counter: 145 + - `release()` increments counter + `futexWake(1)` (wake one) 146 + - `acquire()` snapshots counter under mutex + `futexWaitTimeout()` with snapshot 147 + - no `reset()`, no single-waiter constraint 148 + 149 + **upstream fix**: `Io.Event` could support multi-waiter reset, or provide a 150 + semaphore/condvar primitive. the futex counter pattern is well-known and 151 + could be upstreamed to pg.zig proper. 152 + 153 + ## workaround 6: cross-Io Mutex/futex incompatibility 154 + 155 + **not patched** — worked around by careful Io segregation. 156 + 157 + `Io.Mutex` and `Io.Condition` use futex operations that are tied to their 158 + Io backend. calling `mutex.lockUncancelable(threaded_io)` from an Evented 159 + fiber dereferences `Thread.current()` — a threadlocal only set on Uring- 160 + managed threads. on Evented fibers it's NULL → SIGSEGV or heap corruption. 161 + 162 + this caused three separate crashes during the migration (crashes 1, 6, 8 in 163 + NOTES.md). the fix pattern is always the same: components that use Threaded 164 + resources (mutexes initialized with `pool_io`, pg.Pool) must run on plain 165 + `std.Thread`, not as Evented `io.concurrent()` fibers. 166 + 167 + current segregation: 168 + 169 + | component | runs on | why | 170 + |---|---|---| 171 + | GC loop | `std.Thread` + `pool_io` | uses DiskPersist mutex + pg.Pool | 172 + | resyncer | `std.Thread` + `pool_io` | uses DiskPersist + HTTP client | 173 + | frame workers | `std.Thread` + `pool_io` | uses Io.Mutex/Condition for queue sync | 174 + | subscribers | `io.concurrent` (Evented) | pure network I/O, no shared mutexes | 175 + | broadcast loop | `io.concurrent` (Evented) | lock-free ring buffer + atomics | 176 + | health checks | Evented handlers | use atomic `last_db_success`, not pg.Pool | 177 + 178 + **upstream fix**: there's no obvious stdlib fix here — this is architectural. 179 + either Mutex/Condition need to detect and handle cross-Io calls, or the docs 180 + need to clearly state the constraint. a `pg.Pool` that accepts an `Io` 181 + parameter per-call (rather than at init) would also help. 182 + 183 + ## summary table 184 + 185 + | # | issue | fix type | status | drops when | 186 + |---|---|---|---|---| 187 + | 1 | Uring networking stubs | patch | `patches/uring-networking.patch` | upstream implements (zig#31723) | 188 + | 2 | DNS resolution missing | app workaround | Threaded fallback in subscriber | upstream implements netLookup | 189 + | 3 | ReleaseSafe GPF | build flag | `-Doptimize=ReleaseFast` | upstream fixes codegen bug | 190 + | 4 | debug_io single-threaded | app workaround | `std_options_debug_threaded_io` | upstream changes default or n/a | 191 + | 5 | Io.Event single-waiter | dep fork | pg.zig futex counter | upstream adds multi-waiter Event | 192 + | 6 | cross-Io Mutex | app architecture | Io segregation | upstream makes Mutex cross-Io safe |