zzstoatzz.io / atproto-bench
this repo has no description
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

feat: corpus-based firehose benchmarks across four SDKs

capture ~10s of live firehose traffic into length-prefixed binary corpus,
then decode with zig (zat), rust (jacquard-style), go (indigo), and
python (atproto SDK). each SDK calls its real consumer API — no synthetic
shortcuts. reports frames/sec and MB/s.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

zzstoatzz 03809cac

+2001
+18
.gitignore
··· 1 + # zig 2 + zig/.zig-cache/ 3 + zig/zig-out/ 4 + 5 + # rust 6 + rust/target/ 7 + 8 + # go 9 + go/atproto-bench 10 + 11 + # python 12 + python/.venv/ 13 + python/uv.lock 14 + 15 + # fixtures (captured from live network, not committed) 16 + fixtures/*.bin 17 + fixtures/*.car 18 + fixtures/*.json
+91
README.md
··· 1 + # atproto-bench 2 + 3 + SDK-level firehose benchmarks for AT Protocol. 4 + 5 + decodes a corpus of real firehose frames using each language's AT Protocol SDK. 6 + same corpus, same work, measured side by side. 7 + 8 + ## what this measures 9 + 10 + each benchmark calls the SDK's real consumer API: raw firehose frame bytes in, typed commit with decoded records out. every SDK does the full decode path — frame splitting, CBOR header, typed commit payload, CAR block parsing, and record extraction. 11 + 12 + ## results 13 + 14 + _corpus of ~N firehose frames, 5 measured passes over full corpus, macOS arm64_ 15 + 16 + | SDK | frames/sec | MB/s | 17 + |-----|--------:|-----:| 18 + | zig ([zat](https://tangled.sh/@zzstoatzz.io/zat), arena reuse) | — | — | 19 + | zig (zat, alloc per frame) | — | — | 20 + | rust (jacquard-style) | — | — | 21 + | python ([atproto](https://github.com/MarshalX/atproto)) | — | — | 22 + | go ([indigo](https://github.com/bluesky-social/indigo)) | — | — | 23 + 24 + > run `just capture && just bench` to fill in these numbers on your machine. 25 + 26 + ## what each SDK does 27 + 28 + every SDK takes the same raw binary frame and produces typed output with decoded records: 29 + 30 + | SDK | code path | 31 + |-----|-----------| 32 + | zig | `firehose.decodeFrame(allocator, data)` → typed `CommitEvent` with `ops[]`, decoded records from CAR blocks | 33 + | rust | `ciborium::from_reader` → header, `serde_ipld_dagcbor::from_slice` → typed `Commit`, `iroh_car::CarReader` + `serde_ipld_dagcbor` per block | 34 + | go | `evt.Deserialize(reader)` → typed `RepoCommit` via code-gen CBOR, `car.NewBlockReader` + `cbornode.DecodeInto` per block | 35 + | python | `Frame.from_bytes` + `parse_subscribe_repos_message` → typed `Commit`, `CAR.from_bytes(commit.blocks)` | 36 + 37 + ## fairness notes 38 + 39 + - **zig** uses arena allocation (1 malloc/free per frame); rust/go/python use standard per-object allocators. the "alloc per frame" variant is the fair cross-language comparison; "arena reuse" shows the production pattern 40 + - **zig** returns zero-copy slices into the input buffer for strings and byte data; the other SDKs copy into owned allocations. this is a real architectural advantage, not a benchmark trick 41 + - **go** indigo uses code-generated CBOR unmarshal (no reflection, no serde) — very fast for known schemas 42 + - **python**'s CBOR/CAR work happens in compiled Rust (libipld via PyO3); the Python layer wraps results into typed models 43 + - **rust** pays async overhead (tokio runtime + iroh-car's async `CarReader`) even though the I/O is an in-memory buffer. there's no sync alternative in the library — this overhead applies equally in production 44 + 45 + ## corpus format 46 + 47 + the fixture file (`fixtures/firehose-frames.bin`) uses a simple length-prefixed binary format: 48 + 49 + ``` 50 + [u32 BE frame_count] 51 + [u32 BE frame_1_len][frame_1 bytes] 52 + [u32 BE frame_2_len][frame_2 bytes] 53 + ... 54 + ``` 55 + 56 + frames are captured from ~10 seconds of live firehose traffic, pre-filtered to commits with ops. this gives a realistic mix of frame sizes and record types. 57 + 58 + ## when this matters 59 + 60 + **for live firehose consumption: usually no.** at ~500-1000 events/sec (full bluesky network), any of these SDKs decode fast enough. the bottleneck is network I/O, database writes, and business logic. 61 + 62 + **where it matters:** 63 + - **backfill / replay** — processing months of historical data. decode throughput determines catch-up speed. 64 + - **relays at scale** — routing events to many downstream consumers. every microsecond counts when multiplied. 65 + - **memory** — smaller value types mean less memory per in-flight frame. 66 + 67 + ## SDKs tested 68 + 69 + | lang | SDK | version | CBOR engine | CAR engine | 70 + |------|-----|---------|-------------|------------| 71 + | zig | [zat](https://tangled.sh/@zzstoatzz.io/zat) | 0.1.7 | hand-rolled | hand-rolled | 72 + | rust | jacquard-style | — | [ciborium](https://crates.io/crates/ciborium) (header) + [serde_ipld_dagcbor](https://crates.io/crates/serde_ipld_dagcbor) (body) | [iroh-car](https://crates.io/crates/iroh-car) | 73 + | go | [indigo](https://github.com/bluesky-social/indigo) | latest | [cbor-gen](https://github.com/whyrusleeping/cbor-gen) (code-generated) | [go-car/v2](https://github.com/ipld/go-car) | 74 + | python | [atproto](https://github.com/MarshalX/atproto) | 0.0.65 | [libipld](https://github.com/MarshalX/atproto) (Rust via PyO3) | libipld | 75 + 76 + ## usage 77 + 78 + ```sh 79 + just capture # capture ~10s of firehose traffic 80 + just bench # run all benchmarks 81 + just bench-zig # run a single language 82 + ``` 83 + 84 + ## methodology 85 + 86 + - `just capture` connects to the live firehose for ~10 seconds, filters for commits with ops, writes a length-prefixed corpus 87 + - each benchmark calls the SDK's real consumer API on every frame in the corpus (no synthetic shortcuts) 88 + - 2 warmup passes, 5 measured passes over the full corpus 89 + - zig builds with `-Doptimize=ReleaseFast`, rust with `opt-level=3 lto=true` 90 + - go and python use their standard release toolchains 91 + - reported numbers: frames/sec = total_frames / elapsed, MB/s = total_corpus_bytes / elapsed
fixtures/.gitkeep

This is a binary file and will not be displayed.

+75
go/go.mod
··· 1 + module atproto-bench 2 + 3 + go 1.25.1 4 + 5 + require ( 6 + github.com/bluesky-social/indigo v0.0.0-20260220055544-bf41e2ee75ab 7 + github.com/ipfs/go-ipld-cbor v0.2.1 8 + github.com/ipld/go-car/v2 v2.16.0 9 + ) 10 + 11 + require ( 12 + github.com/RussellLuo/slidingwindow v0.0.0-20200528002341-535bb99d338b // indirect 13 + github.com/beorn7/perks v1.0.1 // indirect 14 + github.com/cespare/xxhash/v2 v2.3.0 // indirect 15 + github.com/earthboundkid/versioninfo/v2 v2.24.1 // indirect 16 + github.com/felixge/httpsnoop v1.0.4 // indirect 17 + github.com/go-logr/logr v1.4.3 // indirect 18 + github.com/go-logr/stdr v1.2.2 // indirect 19 + github.com/gogo/protobuf v1.3.2 // indirect 20 + github.com/google/uuid v1.6.0 // indirect 21 + github.com/gorilla/websocket v1.5.3 // indirect 22 + github.com/hashicorp/go-cleanhttp v0.5.2 // indirect 23 + github.com/hashicorp/go-retryablehttp v0.7.5 // indirect 24 + github.com/hashicorp/golang-lru v1.0.2 // indirect 25 + github.com/ipfs/bbloom v0.0.4 // indirect 26 + github.com/ipfs/boxo v0.34.0 // indirect 27 + github.com/ipfs/go-block-format v0.2.3 // indirect 28 + github.com/ipfs/go-cid v0.5.0 // indirect 29 + github.com/ipfs/go-datastore v0.8.3 // indirect 30 + github.com/ipfs/go-ipfs-blockstore v1.3.1 // indirect 31 + github.com/ipfs/go-ipfs-ds-help v1.1.1 // indirect 32 + github.com/ipfs/go-ipld-format v0.6.3 // indirect 33 + github.com/ipfs/go-log v1.0.5 // indirect 34 + github.com/ipfs/go-log/v2 v2.9.1 // indirect 35 + github.com/ipfs/go-metrics-interface v0.3.0 // indirect 36 + github.com/ipld/go-ipld-prime v0.21.0 // indirect 37 + github.com/jinzhu/inflection v1.0.0 // indirect 38 + github.com/jinzhu/now v1.1.5 // indirect 39 + github.com/klauspost/cpuid/v2 v2.3.0 // indirect 40 + github.com/mattn/go-isatty v0.0.20 // indirect 41 + github.com/minio/sha256-simd v1.0.1 // indirect 42 + github.com/mr-tron/base58 v1.2.0 // indirect 43 + github.com/multiformats/go-base32 v0.1.0 // indirect 44 + github.com/multiformats/go-base36 v0.2.0 // indirect 45 + github.com/multiformats/go-multibase v0.2.0 // indirect 46 + github.com/multiformats/go-multicodec v0.9.2 // indirect 47 + github.com/multiformats/go-multihash v0.2.3 // indirect 48 + github.com/multiformats/go-varint v0.1.0 // indirect 49 + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect 50 + github.com/opentracing/opentracing-go v1.2.0 // indirect 51 + github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9 // indirect 52 + github.com/polydawn/refmt v0.89.1-0.20221221234430-40501e09de1f // indirect 53 + github.com/prometheus/client_golang v1.23.0 // indirect 54 + github.com/prometheus/client_model v0.6.2 // indirect 55 + github.com/prometheus/common v0.65.0 // indirect 56 + github.com/prometheus/procfs v0.17.0 // indirect 57 + github.com/spaolacci/murmur3 v1.1.0 // indirect 58 + github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 // indirect 59 + github.com/whyrusleeping/cbor-gen v0.3.1 // indirect 60 + go.opentelemetry.io/auto/sdk v1.1.0 // indirect 61 + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 // indirect 62 + go.opentelemetry.io/otel v1.37.0 // indirect 63 + go.opentelemetry.io/otel/metric v1.37.0 // indirect 64 + go.opentelemetry.io/otel/trace v1.37.0 // indirect 65 + go.uber.org/atomic v1.11.0 // indirect 66 + go.uber.org/multierr v1.11.0 // indirect 67 + go.uber.org/zap v1.27.1 // indirect 68 + golang.org/x/crypto v0.41.0 // indirect 69 + golang.org/x/exp v0.0.0-20250813145105-42675adae3e6 // indirect 70 + golang.org/x/sys v0.35.0 // indirect 71 + golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect 72 + google.golang.org/protobuf v1.36.9 // indirect 73 + gorm.io/gorm v1.25.9 // indirect 74 + lukechampine.com/blake3 v1.4.1 // indirect 75 + )
+274
go/go.sum
··· 1 + github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= 2 + github.com/Jorropo/jsync v1.0.1 h1:6HgRolFZnsdfzRUj+ImB9og1JYOxQoReSywkHOGSaUU= 3 + github.com/Jorropo/jsync v1.0.1/go.mod h1:jCOZj3vrBCri3bSU3ErUYvevKlnbssrXeCivybS5ABQ= 4 + github.com/RussellLuo/slidingwindow v0.0.0-20200528002341-535bb99d338b h1:5/++qT1/z812ZqBvqQt6ToRswSuPZ/B33m6xVHRzADU= 5 + github.com/RussellLuo/slidingwindow v0.0.0-20200528002341-535bb99d338b/go.mod h1:4+EPqMRApwwE/6yo6CxiHoSnBzjRr3jsqer7frxP8y4= 6 + github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= 7 + github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= 8 + github.com/bluesky-social/indigo v0.0.0-20260220055544-bf41e2ee75ab h1:Cs35T2tAN3Q6mMH5mBaY09nmCNOn/GkZS1F7jfMxlR8= 9 + github.com/bluesky-social/indigo v0.0.0-20260220055544-bf41e2ee75ab/go.mod h1:VG/LeqLGNI3Ew7lsYixajnZGFfWPv144qbUddh+Oyag= 10 + github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= 11 + github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= 12 + github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= 13 + github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 14 + github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 15 + github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 16 + github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0 h1:NMZiJj8QnKe1LgsbDayM4UoHwbvwDRwnI3hwNaAHRnc= 17 + github.com/decred/dcrd/dcrec/secp256k1/v4 v4.4.0/go.mod h1:ZXNYxsqcloTdSy/rNShjYzMhyjf0LaoftYK0p+A3h40= 18 + github.com/earthboundkid/versioninfo/v2 v2.24.1 h1:SJTMHaoUx3GzjjnUO1QzP3ZXK6Ee/nbWyCm58eY3oUg= 19 + github.com/earthboundkid/versioninfo/v2 v2.24.1/go.mod h1:VcWEooDEuyUJnMfbdTh0uFN4cfEIg+kHMuWB2CDCLjw= 20 + github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= 21 + github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= 22 + github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8= 23 + github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= 24 + github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= 25 + github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= 26 + github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= 27 + github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= 28 + github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= 29 + github.com/go-redis/redis v6.15.9+incompatible h1:K0pv1D7EQUjfyoMql+r/jZqCLizCGKFlFgcHWWmHQjg= 30 + github.com/go-redis/redis v6.15.9+incompatible/go.mod h1:NAIEuMOZ/fxfXJIrKDQDz8wamY7mA7PouImQ2Jvg6kA= 31 + github.com/go-yaml/yaml v2.1.0+incompatible/go.mod h1:w2MrLa16VYP0jy6N7M5kHaCkaLENm+P+Tv+MfurjSw0= 32 + github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= 33 + github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= 34 + github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= 35 + github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= 36 + github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= 37 + github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= 38 + github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 39 + github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= 40 + github.com/gopherjs/gopherjs v0.0.0-20190430165422-3e4dfb77656c h1:7lF+Vz0LqiRidnzC1Oq86fpX1q/iEv2KJdrCtttYjT4= 41 + github.com/gopherjs/gopherjs v0.0.0-20190430165422-3e4dfb77656c/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= 42 + github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg= 43 + github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= 44 + github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= 45 + github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= 46 + github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI= 47 + github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ= 48 + github.com/hashicorp/go-retryablehttp v0.7.5 h1:bJj+Pj19UZMIweq/iie+1u5YCdGrnxCT9yvm0e+Nd5M= 49 + github.com/hashicorp/go-retryablehttp v0.7.5/go.mod h1:Jy/gPYAdjqffZ/yFGCFV2doI5wjtH1ewM9u8iYVjtX8= 50 + github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iPY6p1c= 51 + github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= 52 + github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= 53 + github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= 54 + github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs= 55 + github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0= 56 + github.com/ipfs/boxo v0.34.0 h1:pMP9bAsTs4xVh8R0ZmxIWviV7kjDa60U24QrlGgHb1g= 57 + github.com/ipfs/boxo v0.34.0/go.mod h1:kzdH/ewDybtO3+M8MCVkpwnIIc/d2VISX95DFrY4vQA= 58 + github.com/ipfs/go-bitfield v1.1.0 h1:fh7FIo8bSwaJEh6DdTWbCeZ1eqOaOkKFI74SCnsWbGA= 59 + github.com/ipfs/go-bitfield v1.1.0/go.mod h1:paqf1wjq/D2BBmzfTVFlJQ9IlFOZpg422HL0HqsGWHU= 60 + github.com/ipfs/go-block-format v0.2.3 h1:mpCuDaNXJ4wrBJLrtEaGFGXkferrw5eqVvzaHhtFKQk= 61 + github.com/ipfs/go-block-format v0.2.3/go.mod h1:WJaQmPAKhD3LspLixqlqNFxiZ3BZ3xgqxxoSR/76pnA= 62 + github.com/ipfs/go-cid v0.5.0 h1:goEKKhaGm0ul11IHA7I6p1GmKz8kEYniqFopaB5Otwg= 63 + github.com/ipfs/go-cid v0.5.0/go.mod h1:0L7vmeNXpQpUS9vt+yEARkJ8rOg43DF3iPgn4GIN0mk= 64 + github.com/ipfs/go-datastore v0.8.3 h1:z391GsQyGKUIUof2tPoaZVeDknbt7fNHs6Gqjcw5Jo4= 65 + github.com/ipfs/go-datastore v0.8.3/go.mod h1:raxQ/CreIy9L6MxT71ItfMX12/ASN6EhXJoUFjICQ2M= 66 + github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= 67 + github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= 68 + github.com/ipfs/go-ipfs-blockstore v1.3.1 h1:cEI9ci7V0sRNivqaOr0elDsamxXFxJMMMy7PTTDQNsQ= 69 + github.com/ipfs/go-ipfs-blockstore v1.3.1/go.mod h1:KgtZyc9fq+P2xJUiCAzbRdhhqJHvsw8u2Dlqy2MyRTE= 70 + github.com/ipfs/go-ipfs-ds-help v1.1.1 h1:B5UJOH52IbcfS56+Ul+sv8jnIV10lbjLF5eOO0C66Nw= 71 + github.com/ipfs/go-ipfs-ds-help v1.1.1/go.mod h1:75vrVCkSdSFidJscs8n4W+77AtTpCIAdDGAwjitJMIo= 72 + github.com/ipfs/go-ipfs-util v0.0.3 h1:2RFdGez6bu2ZlZdI+rWfIdbQb1KudQp3VGwPtdNCmE0= 73 + github.com/ipfs/go-ipfs-util v0.0.3/go.mod h1:LHzG1a0Ig4G+iZ26UUOMjHd+lfM84LZCrn17xAKWBvs= 74 + github.com/ipfs/go-ipld-cbor v0.2.1 h1:H05yEJbK/hxg0uf2AJhyerBDbjOuHX4yi+1U/ogRa7E= 75 + github.com/ipfs/go-ipld-cbor v0.2.1/go.mod h1:x9Zbeq8CoE5R2WicYgBMcr/9mnkQ0lHddYWJP2sMV3A= 76 + github.com/ipfs/go-ipld-format v0.6.3 h1:9/lurLDTotJpZSuL++gh3sTdmcFhVkCwsgx2+rAh4j8= 77 + github.com/ipfs/go-ipld-format v0.6.3/go.mod h1:74ilVN12NXVMIV+SrBAyC05UJRk0jVvGqdmrcYZvCBk= 78 + github.com/ipfs/go-log v1.0.5 h1:2dOuUCB1Z7uoczMWgAyDck5JLb72zHzrMnGnCNNbvY8= 79 + github.com/ipfs/go-log v1.0.5/go.mod h1:j0b8ZoR+7+R99LD9jZ6+AJsrzkPbSXbZfGakb5JPtIo= 80 + github.com/ipfs/go-log/v2 v2.1.3/go.mod h1:/8d0SH3Su5Ooc31QlL1WysJhvyOTDCjcCZ9Axpmri6g= 81 + github.com/ipfs/go-log/v2 v2.9.1 h1:3JXwHWU31dsCpvQ+7asz6/QsFJHqFr4gLgQ0FWteujk= 82 + github.com/ipfs/go-log/v2 v2.9.1/go.mod h1:evFx7sBiohUN3AG12mXlZBw5hacBQld3ZPHrowlJYoo= 83 + github.com/ipfs/go-metrics-interface v0.3.0 h1:YwG7/Cy4R94mYDUuwsBfeziJCVm9pBMJ6q/JR9V40TU= 84 + github.com/ipfs/go-metrics-interface v0.3.0/go.mod h1:OxxQjZDGocXVdyTPocns6cOLwHieqej/jos7H4POwoY= 85 + github.com/ipfs/go-unixfsnode v1.10.2 h1:TREegX1J4X+k1w4AhoDuxxFvVcS9SegMRvrmxF6Tca8= 86 + github.com/ipfs/go-unixfsnode v1.10.2/go.mod h1:ImDPTSiKZ+2h4UVdkSDITJHk87bUAp7kX/lgifjRicg= 87 + github.com/ipld/go-car/v2 v2.16.0 h1:LWe0vmN/QcQmUU4tr34W5Nv5mNraW+G6jfN2s+ndBco= 88 + github.com/ipld/go-car/v2 v2.16.0/go.mod h1:RqFGWN9ifcXVmCrTAVnfnxiWZk1+jIx67SYhenlmL34= 89 + github.com/ipld/go-codec-dagpb v1.7.0 h1:hpuvQjCSVSLnTnHXn+QAMR0mLmb1gA6wl10LExo2Ts0= 90 + github.com/ipld/go-codec-dagpb v1.7.0/go.mod h1:rD3Zg+zub9ZnxcLwfol/OTQRVjaLzXypgy4UqHQvilM= 91 + github.com/ipld/go-ipld-prime v0.21.0 h1:n4JmcpOlPDIxBcY037SVfpd1G+Sj1nKZah0m6QH9C2E= 92 + github.com/ipld/go-ipld-prime v0.21.0/go.mod h1:3RLqy//ERg/y5oShXXdx5YIp50cFGOanyMctpPjsvxQ= 93 + github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20250821084354-a425e60cd714 h1:cqNk8PEwHnK0vqWln+U/YZhQc9h2NB3KjUjDPZo5Q2s= 94 + github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20250821084354-a425e60cd714/go.mod h1:ZEUdra3CoqRVRYgAX/jAJO9aZGz6SKtKEG628fHHktY= 95 + github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E= 96 + github.com/jinzhu/inflection v1.0.0/go.mod h1:h+uFLlag+Qp1Va5pdKtLDYj+kHp5pxUVkryuEj+Srlc= 97 + github.com/jinzhu/now v1.1.5 h1:/o9tlHleP7gOFmsnYNz3RGnqzefHA47wQpKrrdTIwXQ= 98 + github.com/jinzhu/now v1.1.5/go.mod h1:d3SSVoowX0Lcu0IBviAWJpolVfI5UJVZZ7cO71lE/z8= 99 + github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo= 100 + github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= 101 + github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= 102 + github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= 103 + github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y= 104 + github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= 105 + github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= 106 + github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= 107 + github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= 108 + github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 109 + github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 110 + github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= 111 + github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= 112 + github.com/libp2p/go-buffer-pool v0.1.0 h1:oK4mSFcQz7cTQIfqbe4MIj9gLW+mnanjyFtc6cdF0Y8= 113 + github.com/libp2p/go-buffer-pool v0.1.0/go.mod h1:N+vh8gMqimBzdKkSMVuydVDq+UV5QTWy5HSiZacSbPg= 114 + github.com/libp2p/go-libp2p v0.43.0 h1:b2bg2cRNmY4HpLK8VHYQXLX2d3iND95OjodLFymvqXU= 115 + github.com/libp2p/go-libp2p v0.43.0/go.mod h1:IiSqAXDyP2sWH+J2gs43pNmB/y4FOi2XQPbsb+8qvzc= 116 + github.com/libp2p/go-libp2p-record v0.3.1 h1:cly48Xi5GjNw5Wq+7gmjfBiG9HCzQVkiZOUZ8kUl+Fg= 117 + github.com/libp2p/go-libp2p-record v0.3.1/go.mod h1:T8itUkLcWQLCYMqtX7Th6r7SexyUJpIyPgks757td/E= 118 + github.com/libp2p/go-libp2p-routing-helpers v0.7.5 h1:HdwZj9NKovMx0vqq6YNPTh6aaNzey5zHD7HeLJtq6fI= 119 + github.com/libp2p/go-libp2p-routing-helpers v0.7.5/go.mod h1:3YaxrwP0OBPDD7my3D0KxfR89FlcX/IEbxDEDfAmj98= 120 + github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= 121 + github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= 122 + github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= 123 + github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8= 124 + github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o= 125 + github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= 126 + github.com/multiformats/go-base32 v0.1.0 h1:pVx9xoSPqEIQG8o+UbAe7DNi51oej1NtK+aGkbLYxPE= 127 + github.com/multiformats/go-base32 v0.1.0/go.mod h1:Kj3tFY6zNr+ABYMqeUNeGvkIC/UYgtWibDcT0rExnbI= 128 + github.com/multiformats/go-base36 v0.2.0 h1:lFsAbNOGeKtuKozrtBsAkSVhv1p9D0/qedU9rQyccr0= 129 + github.com/multiformats/go-base36 v0.2.0/go.mod h1:qvnKE++v+2MWCfePClUEjE78Z7P2a1UV0xHgWc0hkp4= 130 + github.com/multiformats/go-multiaddr v0.16.1 h1:fgJ0Pitow+wWXzN9do+1b8Pyjmo8m5WhGfzpL82MpCw= 131 + github.com/multiformats/go-multiaddr v0.16.1/go.mod h1:JSVUmXDjsVFiW7RjIFMP7+Ev+h1DTbiJgVeTV/tcmP0= 132 + github.com/multiformats/go-multibase v0.2.0 h1:isdYCVLvksgWlMW9OZRYJEa9pZETFivncJHmHnnd87g= 133 + github.com/multiformats/go-multibase v0.2.0/go.mod h1:bFBZX4lKCA/2lyOFSAoKH5SS6oPyjtnzK/XTFDPkNuk= 134 + github.com/multiformats/go-multicodec v0.9.2 h1:YrlXCuqxjqm3bXl+vBq5LKz5pz4mvAsugdqy78k0pXQ= 135 + github.com/multiformats/go-multicodec v0.9.2/go.mod h1:LLWNMtyV5ithSBUo3vFIMaeDy+h3EbkMTek1m+Fybbo= 136 + github.com/multiformats/go-multihash v0.2.3 h1:7Lyc8XfX/IY2jWb/gI7JP+o7JEq9hOa7BFvVU9RSh+U= 137 + github.com/multiformats/go-multihash v0.2.3/go.mod h1:dXgKXCXjBzdscBLk9JkjINiEsCKRVch90MdaGiKsvSM= 138 + github.com/multiformats/go-varint v0.1.0 h1:i2wqFp4sdl3IcIxfAonHQV9qU5OsZ4Ts9IOoETFs5dI= 139 + github.com/multiformats/go-varint v0.1.0/go.mod h1:5KVAVXegtfmNQQm/lCY+ATvDzvJJhSkUlGQV9wgObdI= 140 + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA= 141 + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= 142 + github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs= 143 + github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc= 144 + github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9 h1:1/WtZae0yGtPq+TI6+Tv1WTxkukpXeMlviSxvL7SRgk= 145 + github.com/petar/GoLLRB v0.0.0-20210522233825-ae3b015fd3e9/go.mod h1:x3N5drFsm2uilKKuuYo6LdyD8vZAW55sH/9w+pbo1sw= 146 + github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 147 + github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 148 + github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 149 + github.com/polydawn/refmt v0.89.1-0.20221221234430-40501e09de1f h1:VXTQfuJj9vKR4TCkEuWIckKvdHFeJH/huIFJ9/cXOB0= 150 + github.com/polydawn/refmt v0.89.1-0.20221221234430-40501e09de1f/go.mod h1:/zvteZs/GwLtCgZ4BL6CBsk9IKIlexP43ObX9AxTqTw= 151 + github.com/prometheus/client_golang v1.23.0 h1:ust4zpdl9r4trLY/gSjlm07PuiBq2ynaXXlptpfy8Uc= 152 + github.com/prometheus/client_golang v1.23.0/go.mod h1:i/o0R9ByOnHX0McrTMTyhYvKE4haaf2mW08I+jGAjEE= 153 + github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= 154 + github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE= 155 + github.com/prometheus/common v0.65.0 h1:QDwzd+G1twt//Kwj/Ww6E9FQq1iVMmODnILtW1t2VzE= 156 + github.com/prometheus/common v0.65.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8= 157 + github.com/prometheus/procfs v0.17.0 h1:FuLQ+05u4ZI+SS/w9+BWEM2TXiHKsUQ9TADiRH7DuK0= 158 + github.com/prometheus/procfs v0.17.0/go.mod h1:oPQLaDAMRbA+u8H5Pbfq+dl3VDAvHxMUOVhe0wYB2zw= 159 + github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= 160 + github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= 161 + github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= 162 + github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= 163 + github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= 164 + github.com/smartystreets/assertions v1.2.0 h1:42S6lae5dvLc7BrLu/0ugRtcFVjoJNMC/N3yZFZkDFs= 165 + github.com/smartystreets/assertions v1.2.0/go.mod h1:tcbTF8ujkAEcZ8TElKY+i30BzYlVhC/LOxJk7iOWnoo= 166 + github.com/smartystreets/goconvey v1.7.2 h1:9RBaZCeXEQ3UselpuwUQHltGVXvdwm6cv1hgR6gDIPg= 167 + github.com/smartystreets/goconvey v1.7.2/go.mod h1:Vw0tHAZW6lzCRk3xgdin6fKYcG+G3Pg9vgXWeJpQFMM= 168 + github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= 169 + github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= 170 + github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 171 + github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 172 + github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 173 + github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= 174 + github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= 175 + github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= 176 + github.com/urfave/cli v1.22.10/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0= 177 + github.com/warpfork/go-testmark v0.12.1 h1:rMgCpJfwy1sJ50x0M0NgyphxYYPMOODIJHhsXyEHU0s= 178 + github.com/warpfork/go-testmark v0.12.1/go.mod h1:kHwy7wfvGSPh1rQJYKayD4AbtNaeyZdcGi9tNJTaa5Y= 179 + github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0 h1:GDDkbFiaK8jsSDJfjId/PEGEShv6ugrt4kYsC5UIDaQ= 180 + github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw= 181 + github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11 h1:5HZfQkwe0mIfyDmc1Em5GqlNRzcdtlv4HTNmdpt7XH0= 182 + github.com/whyrusleeping/cbor v0.0.0-20171005072247-63513f603b11/go.mod h1:Wlo/SzPmxVp6vXpGt/zaXhHH0fn4IxgqZc82aKg6bpQ= 183 + github.com/whyrusleeping/cbor-gen v0.3.1 h1:82ioxmhEYut7LBVGhGq8xoRkXPLElVuh5mV67AFfdv0= 184 + github.com/whyrusleeping/cbor-gen v0.3.1/go.mod h1:pM99HXyEbSQHcosHc0iW7YFmwnscr+t9Te4ibko05so= 185 + github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f h1:jQa4QT2UP9WYv2nzyawpKMOCl+Z/jW7djv2/J50lj9E= 186 + github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f/go.mod h1:p9UJB6dDgdPgMJZs7UjUOdulKyRr9fqkS+6JKAInPy8= 187 + github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 188 + github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 189 + go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA= 190 + go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A= 191 + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0 h1:Hf9xI/XLML9ElpiHVDNwvqI0hIFlzV8dgIr35kV1kRU= 192 + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0/go.mod h1:NfchwuyNoMcZ5MLHwPrODwUF1HWCXWrL31s8gSAdIKY= 193 + go.opentelemetry.io/otel v1.37.0 h1:9zhNfelUvx0KBfu/gb+ZgeAfAgtWrfHJZcAqFC228wQ= 194 + go.opentelemetry.io/otel v1.37.0/go.mod h1:ehE/umFRLnuLa/vSccNq9oS1ErUlkkK71gMcN34UG8I= 195 + go.opentelemetry.io/otel/metric v1.37.0 h1:mvwbQS5m0tbmqML4NqK+e3aDiO02vsf/WgbsdpcPoZE= 196 + go.opentelemetry.io/otel/metric v1.37.0/go.mod h1:04wGrZurHYKOc+RKeye86GwKiTb9FKm1WHtO+4EVr2E= 197 + go.opentelemetry.io/otel/sdk v1.37.0 h1:ItB0QUqnjesGRvNcmAcU0LyvkVyGJ2xftD29bWdDvKI= 198 + go.opentelemetry.io/otel/sdk v1.37.0/go.mod h1:VredYzxUvuo2q3WRcDnKDjbdvmO0sCzOvVAiY+yUkAg= 199 + go.opentelemetry.io/otel/sdk/metric v1.37.0 h1:90lI228XrB9jCMuSdA0673aubgRobVZFhbjxHHspCPc= 200 + go.opentelemetry.io/otel/sdk/metric v1.37.0/go.mod h1:cNen4ZWfiD37l5NhS+Keb5RXVWZWpRE+9WyVCpbo5ps= 201 + go.opentelemetry.io/otel/trace v1.37.0 h1:HLdcFNbRQBE2imdSEgm/kwqmQj1Or1l/7bW6mxVK7z4= 202 + go.opentelemetry.io/otel/trace v1.37.0/go.mod h1:TlgrlQ+PtQO5XFerSPUYG0JSgGyryXewPGyayAWSBS0= 203 + go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= 204 + go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= 205 + go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= 206 + go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= 207 + go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= 208 + go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= 209 + go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= 210 + go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= 211 + go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= 212 + go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= 213 + go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= 214 + go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ= 215 + go.uber.org/zap v1.27.1 h1:08RqriUEv8+ArZRYSTXy1LeBScaMpVSTBhCeaZYfMYc= 216 + go.uber.org/zap v1.27.1/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E= 217 + golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 218 + golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 219 + golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 220 + golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 221 + golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4= 222 + golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc= 223 + golang.org/x/exp v0.0.0-20250813145105-42675adae3e6 h1:SbTAbRFnd5kjQXbczszQ0hdk3ctwYf3qBNH9jIsGclE= 224 + golang.org/x/exp v0.0.0-20250813145105-42675adae3e6/go.mod h1:4QTo5u+SEIbbKW1RacMZq1YEfOBqeXa19JeshGi+zc4= 225 + golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 226 + golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= 227 + golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 228 + golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 229 + golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 230 + golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 231 + golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 232 + golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 233 + golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 234 + golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 235 + golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 236 + golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 237 + golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 238 + golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 239 + golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 240 + golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 241 + golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI= 242 + golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= 243 + golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 244 + golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 245 + golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 246 + golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 247 + golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 248 + golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= 249 + golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 250 + golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 251 + golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 252 + golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= 253 + golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= 254 + golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 255 + golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 256 + golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 257 + golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 258 + golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da h1:noIWHXmPHxILtqtCOPIhSt0ABwskkZKjD3bXGnZGpNY= 259 + golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90= 260 + google.golang.org/protobuf v1.36.9 h1:w2gp2mA27hUeUzj9Ex9FBjsBm40zfaDtEWow293U7Iw= 261 + google.golang.org/protobuf v1.36.9/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU= 262 + gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 263 + gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 264 + gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= 265 + gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 266 + gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= 267 + gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= 268 + gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 269 + gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 270 + gorm.io/gorm v1.25.9 h1:wct0gxZIELDk8+ZqF/MVnHLkA1rvYlBWUMv2EdsK1g8= 271 + gorm.io/gorm v1.25.9/go.mod h1:hbnx/Oo0ChWMn1BIhpy1oYozzpM15i4YPuHDmfYtwg8= 272 + honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= 273 + lukechampine.com/blake3 v1.4.1 h1:I3Smz7gso8w4/TunLKec6K2fn+kyKtDxr/xcQEN84Wg= 274 + lukechampine.com/blake3 v1.4.1/go.mod h1:QFosUxmjB8mnrWFSNwKmvxHpfY72bmD2tQ0kBMM3kwo=
+177
go/main.go
··· 1 + // atproto firehose benchmarks — go (indigo) 2 + // 3 + // decodes a corpus of real firehose frames using indigo's code-generated 4 + // CBOR types + go-car for block extraction. same decode path as indigo's relay. 5 + package main 6 + 7 + import ( 8 + "bytes" 9 + "encoding/binary" 10 + "fmt" 11 + "os" 12 + "path/filepath" 13 + "time" 14 + 15 + comatproto "github.com/bluesky-social/indigo/api/atproto" 16 + "github.com/bluesky-social/indigo/events" 17 + car "github.com/ipld/go-car/v2" 18 + 19 + cbornode "github.com/ipfs/go-ipld-cbor" 20 + ) 21 + 22 + const ( 23 + warmupPasses = 2 24 + measuredPasses = 5 25 + fixturesDir = "../fixtures" 26 + ) 27 + 28 + type corpusInfo struct { 29 + frames [][]byte 30 + totalBytes int 31 + minFrame int 32 + maxFrame int 33 + } 34 + 35 + func main() { 36 + fmt.Println("\n=== go benchmarks ===") 37 + fmt.Println() 38 + 39 + corpus, err := loadCorpus("firehose-frames.bin") 40 + if err != nil { 41 + fmt.Printf("firehose-frames.bin: SKIP (%v)\n", err) 42 + return 43 + } 44 + 45 + fmt.Printf("corpus: %d frames, %d bytes total\n", len(corpus.frames), corpus.totalBytes) 46 + fmt.Printf(" frame sizes: %d..%d bytes\n", corpus.minFrame, corpus.maxFrame) 47 + fmt.Printf(" passes: %d warmup, %d measured\n\n", warmupPasses, measuredPasses) 48 + 49 + // verify first frame 50 + if len(corpus.frames) > 0 { 51 + evt, blockCount, err := decodeFull(corpus.frames[0]) 52 + if err != nil { 53 + fmt.Printf("first frame verify failed: %v\n", err) 54 + return 55 + } 56 + if evt.RepoCommit != nil { 57 + c := evt.RepoCommit 58 + fmt.Printf("first frame: repo=%s ops=%d blocks=%d\n", c.Repo, len(c.Ops), blockCount) 59 + } 60 + } 61 + fmt.Println() 62 + 63 + benchDecode(corpus) 64 + 65 + fmt.Println() 66 + } 67 + 68 + func decodeFull(data []byte) (*events.XRPCStreamEvent, int, error) { 69 + var evt events.XRPCStreamEvent 70 + if err := evt.Deserialize(bytes.NewReader(data)); err != nil { 71 + return nil, 0, err 72 + } 73 + 74 + blockCount := 0 75 + commit := evt.RepoCommit 76 + if commit != nil && len(commit.Blocks) > 0 { 77 + reader, err := car.NewBlockReader(bytes.NewReader([]byte(commit.Blocks))) 78 + if err != nil { 79 + return &evt, 0, err 80 + } 81 + 82 + for { 83 + block, err := reader.Next() 84 + if err != nil { 85 + break 86 + } 87 + 88 + var v interface{} 89 + cbornode.DecodeInto(block.RawData(), &v) 90 + blockCount++ 91 + } 92 + } 93 + 94 + return &evt, blockCount, nil 95 + } 96 + 97 + func benchDecode(corpus *corpusInfo) { 98 + for i := 0; i < warmupPasses; i++ { 99 + for _, frame := range corpus.frames { 100 + decodeFull(frame) 101 + } 102 + } 103 + 104 + totalFrames := 0 105 + start := time.Now() 106 + for i := 0; i < measuredPasses; i++ { 107 + for _, frame := range corpus.frames { 108 + decodeFull(frame) 109 + totalFrames++ 110 + } 111 + } 112 + elapsed := time.Since(start) 113 + 114 + reportCorpusResult("decode", corpus, totalFrames, elapsed) 115 + } 116 + 117 + func reportCorpusResult(name string, corpus *corpusInfo, totalFrames int, elapsed time.Duration) { 118 + elapsedS := elapsed.Seconds() 119 + elapsedMs := elapsedS * 1000.0 120 + framesPerSec := float64(totalFrames) / elapsedS 121 + totalBytes := float64(corpus.totalBytes) * float64(measuredPasses) 122 + throughputMb := totalBytes / (1024 * 1024) / elapsedS 123 + 124 + fmt.Printf("%-14s %10.0f frames/sec %8.1f ms (%.1f MB/s)\n", 125 + name, framesPerSec, elapsedMs, throughputMb) 126 + } 127 + 128 + func loadCorpus(name string) (*corpusInfo, error) { 129 + path := filepath.Join(fixturesDir, name) 130 + data, err := os.ReadFile(path) 131 + if err != nil { 132 + fmt.Printf("cannot open %s: %v\n", path, err) 133 + fmt.Println("run `just capture` first to generate fixtures") 134 + return nil, err 135 + } 136 + 137 + if len(data) < 4 { 138 + return nil, fmt.Errorf("corpus file too small") 139 + } 140 + 141 + frameCount := int(binary.BigEndian.Uint32(data[0:4])) 142 + frames := make([][]byte, 0, frameCount) 143 + pos := 4 144 + totalBytes := 0 145 + minFrame := int(^uint(0) >> 1) 146 + maxFrame := 0 147 + 148 + for i := 0; i < frameCount; i++ { 149 + if pos+4 > len(data) { 150 + return nil, fmt.Errorf("truncated corpus") 151 + } 152 + frameLen := int(binary.BigEndian.Uint32(data[pos : pos+4])) 153 + pos += 4 154 + if pos+frameLen > len(data) { 155 + return nil, fmt.Errorf("truncated corpus") 156 + } 157 + frames = append(frames, data[pos:pos+frameLen]) 158 + pos += frameLen 159 + totalBytes += frameLen 160 + if frameLen < minFrame { 161 + minFrame = frameLen 162 + } 163 + if frameLen > maxFrame { 164 + maxFrame = frameLen 165 + } 166 + } 167 + 168 + return &corpusInfo{ 169 + frames: frames, 170 + totalBytes: totalBytes, 171 + minFrame: minFrame, 172 + maxFrame: maxFrame, 173 + }, nil 174 + } 175 + 176 + // ensure SyncSubscribeRepos_Commit is used 177 + var _ *comatproto.SyncSubscribeRepos_Commit
+53
justfile
··· 1 + # atproto firehose benchmarks 2 + 3 + # capture ~10s of firehose traffic + jetstream event 4 + capture: 5 + cd zig && zig build run-capture 6 + 7 + # run all benchmarks 8 + bench: _ensure-fixtures 9 + @echo "============================================" 10 + @echo " atproto firehose benchmarks" 11 + @echo "============================================" 12 + @echo "" 13 + @echo "corpus: firehose-frames.bin ($(wc -c < fixtures/firehose-frames.bin | tr -d ' ') bytes)" 14 + @echo "" 15 + @echo "--------------------------------------------" 16 + cd zig && zig build run-bench -Doptimize=ReleaseFast 17 + @echo "--------------------------------------------" 18 + cd rust && cargo run --release 2>&1 19 + @echo "--------------------------------------------" 20 + cd go && go run . 21 + @echo "--------------------------------------------" 22 + cd python && uv run bench.py 23 + @echo "============================================" 24 + 25 + # run a single language's bench 26 + bench-zig: _ensure-fixtures 27 + cd zig && zig build run-bench -Doptimize=ReleaseFast 28 + 29 + bench-rust: _ensure-fixtures 30 + cd rust && cargo run --release 31 + 32 + bench-go: _ensure-fixtures 33 + cd go && go run . 34 + 35 + bench-python: _ensure-fixtures 36 + cd python && uv run bench.py 37 + 38 + # build all (no run) 39 + build: 40 + cd zig && zig build 41 + cd rust && cargo build --release 42 + cd go && go build . 43 + 44 + # clean build artifacts 45 + clean: 46 + rm -rf zig/.zig-cache zig/zig-out 47 + cd rust && cargo clean 48 + rm -f go/atproto-bench 49 + rm -rf python/.venv 50 + 51 + _ensure-fixtures: 52 + @test -f fixtures/firehose-frames.bin \ 53 + || (echo "fixtures not found — run 'just capture' first" && exit 1)
+138
python/bench.py
··· 1 + """atproto firehose benchmarks — python (atproto SDK) 2 + 3 + decodes a corpus of real firehose frames using the atproto SDK's public API: 4 + Frame.from_bytes + parse_subscribe_repos_message + CAR.from_bytes. 5 + under the hood: libipld (Rust extension via PyO3) for CBOR/CAR, 6 + Python layer for typed model wrapping. 7 + """ 8 + 9 + import struct 10 + import time 11 + from dataclasses import dataclass 12 + from pathlib import Path 13 + 14 + from atproto import CAR, firehose_models, parse_subscribe_repos_message 15 + 16 + WARMUP_PASSES = 2 17 + MEASURED_PASSES = 5 18 + FIXTURES_DIR = Path(__file__).parent.parent / "fixtures" 19 + 20 + 21 + @dataclass 22 + class CorpusInfo: 23 + frames: list[bytes] 24 + total_bytes: int 25 + min_frame: int 26 + max_frame: int 27 + 28 + 29 + def load_corpus(name: str) -> CorpusInfo: 30 + path = FIXTURES_DIR / name 31 + if not path.exists(): 32 + print(f"cannot open {path}") 33 + print("run `just capture` first to generate fixtures") 34 + raise FileNotFoundError(path) 35 + 36 + data = path.read_bytes() 37 + if len(data) < 4: 38 + raise ValueError("corpus file too small") 39 + 40 + (frame_count,) = struct.unpack(">I", data[0:4]) 41 + frames: list[bytes] = [] 42 + pos = 4 43 + total_bytes = 0 44 + min_frame = float("inf") 45 + max_frame = 0 46 + 47 + for _ in range(frame_count): 48 + if pos + 4 > len(data): 49 + raise ValueError("truncated corpus") 50 + (frame_len,) = struct.unpack(">I", data[pos : pos + 4]) 51 + pos += 4 52 + if pos + frame_len > len(data): 53 + raise ValueError("truncated corpus") 54 + frames.append(data[pos : pos + frame_len]) 55 + pos += frame_len 56 + total_bytes += frame_len 57 + min_frame = min(min_frame, frame_len) 58 + max_frame = max(max_frame, frame_len) 59 + 60 + return CorpusInfo( 61 + frames=frames, 62 + total_bytes=total_bytes, 63 + min_frame=int(min_frame), 64 + max_frame=max_frame, 65 + ) 66 + 67 + 68 + def decode_full(frame_data: bytes): 69 + """Full decode: frame parse + typed commit + CAR block extraction.""" 70 + frame = firehose_models.Frame.from_bytes(frame_data) 71 + msg = parse_subscribe_repos_message(frame) 72 + if msg.blocks: 73 + CAR.from_bytes(msg.blocks) 74 + return msg 75 + 76 + 77 + def bench_decode(corpus: CorpusInfo) -> None: 78 + """Full pipeline for each frame: the real API a consumer calls.""" 79 + for _ in range(WARMUP_PASSES): 80 + for frame_data in corpus.frames: 81 + decode_full(frame_data) 82 + 83 + total_frames = 0 84 + start = time.perf_counter_ns() 85 + for _ in range(MEASURED_PASSES): 86 + for frame_data in corpus.frames: 87 + decode_full(frame_data) 88 + total_frames += 1 89 + elapsed_ns = time.perf_counter_ns() - start 90 + 91 + report_corpus_result("decode", corpus, total_frames, elapsed_ns) 92 + 93 + 94 + def report_corpus_result( 95 + name: str, corpus: CorpusInfo, total_frames: int, elapsed_ns: int 96 + ) -> None: 97 + elapsed_s = elapsed_ns / 1_000_000_000 98 + elapsed_ms = elapsed_s * 1000 99 + frames_per_sec = total_frames / elapsed_s 100 + total_bytes = corpus.total_bytes * MEASURED_PASSES 101 + throughput_mb = total_bytes / (1024 * 1024) / elapsed_s 102 + print( 103 + f"{name:<14} {frames_per_sec:>10.0f} frames/sec {elapsed_ms:>8.1f} ms ({throughput_mb:.1f} MB/s)" 104 + ) 105 + 106 + 107 + def main() -> None: 108 + print("\n=== python benchmarks ===\n") 109 + 110 + try: 111 + corpus = load_corpus("firehose-frames.bin") 112 + except Exception as e: 113 + print(f"firehose-frames.bin: SKIP ({e})") 114 + return 115 + 116 + print(f"corpus: {len(corpus.frames)} frames, {corpus.total_bytes} bytes total") 117 + print(f" frame sizes: {corpus.min_frame}..{corpus.max_frame} bytes") 118 + print(f" passes: {WARMUP_PASSES} warmup, {MEASURED_PASSES} measured") 119 + print() 120 + 121 + # verify first frame 122 + try: 123 + msg = decode_full(corpus.frames[0]) 124 + print(f"first frame: repo={msg.repo} ops={len(msg.ops)}") 125 + print() 126 + except Exception as e: 127 + print(f"first frame verify failed: {e}") 128 + 129 + try: 130 + bench_decode(corpus) 131 + except Exception as e: 132 + print(f"decode: SKIP ({e})") 133 + 134 + print() 135 + 136 + 137 + if __name__ == "__main__": 138 + main()
+5
python/pyproject.toml
··· 1 + [project] 2 + name = "atproto-bench" 3 + version = "0.1.0" 4 + requires-python = ">=3.11" 5 + dependencies = ["atproto>=0.0.55"]
+492
rust/Cargo.lock
··· 1 + # This file is automatically @generated by Cargo. 2 + # It is not intended for manual editing. 3 + version = 4 4 + 5 + [[package]] 6 + name = "anyhow" 7 + version = "1.0.102" 8 + source = "registry+https://github.com/rust-lang/crates.io-index" 9 + checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" 10 + 11 + [[package]] 12 + name = "atproto-bench" 13 + version = "0.1.0" 14 + dependencies = [ 15 + "ciborium", 16 + "ipld-core", 17 + "iroh-car", 18 + "serde", 19 + "serde_bytes", 20 + "serde_ipld_dagcbor", 21 + "tokio", 22 + ] 23 + 24 + [[package]] 25 + name = "base-x" 26 + version = "0.2.11" 27 + source = "registry+https://github.com/rust-lang/crates.io-index" 28 + checksum = "4cbbc9d0964165b47557570cce6c952866c2678457aca742aafc9fb771d30270" 29 + 30 + [[package]] 31 + name = "base256emoji" 32 + version = "1.0.2" 33 + source = "registry+https://github.com/rust-lang/crates.io-index" 34 + checksum = "b5e9430d9a245a77c92176e649af6e275f20839a48389859d1661e9a128d077c" 35 + dependencies = [ 36 + "const-str", 37 + "match-lookup", 38 + ] 39 + 40 + [[package]] 41 + name = "bytes" 42 + version = "1.11.1" 43 + source = "registry+https://github.com/rust-lang/crates.io-index" 44 + checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" 45 + 46 + [[package]] 47 + name = "cbor4ii" 48 + version = "0.2.14" 49 + source = "registry+https://github.com/rust-lang/crates.io-index" 50 + checksum = "b544cf8c89359205f4f990d0e6f3828db42df85b5dac95d09157a250eb0749c4" 51 + dependencies = [ 52 + "serde", 53 + ] 54 + 55 + [[package]] 56 + name = "cfg-if" 57 + version = "1.0.4" 58 + source = "registry+https://github.com/rust-lang/crates.io-index" 59 + checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" 60 + 61 + [[package]] 62 + name = "ciborium" 63 + version = "0.2.2" 64 + source = "registry+https://github.com/rust-lang/crates.io-index" 65 + checksum = "42e69ffd6f0917f5c029256a24d0161db17cea3997d185db0d35926308770f0e" 66 + dependencies = [ 67 + "ciborium-io", 68 + "ciborium-ll", 69 + "serde", 70 + ] 71 + 72 + [[package]] 73 + name = "ciborium-io" 74 + version = "0.2.2" 75 + source = "registry+https://github.com/rust-lang/crates.io-index" 76 + checksum = "05afea1e0a06c9be33d539b876f1ce3692f4afea2cb41f740e7743225ed1c757" 77 + 78 + [[package]] 79 + name = "ciborium-ll" 80 + version = "0.2.2" 81 + source = "registry+https://github.com/rust-lang/crates.io-index" 82 + checksum = "57663b653d948a338bfb3eeba9bb2fd5fcfaecb9e199e87e1eda4d9e8b240fd9" 83 + dependencies = [ 84 + "ciborium-io", 85 + "half", 86 + ] 87 + 88 + [[package]] 89 + name = "cid" 90 + version = "0.11.1" 91 + source = "registry+https://github.com/rust-lang/crates.io-index" 92 + checksum = "3147d8272e8fa0ccd29ce51194dd98f79ddfb8191ba9e3409884e751798acf3a" 93 + dependencies = [ 94 + "core2", 95 + "multibase", 96 + "multihash", 97 + "serde", 98 + "serde_bytes", 99 + "unsigned-varint 0.8.0", 100 + ] 101 + 102 + [[package]] 103 + name = "const-str" 104 + version = "0.4.3" 105 + source = "registry+https://github.com/rust-lang/crates.io-index" 106 + checksum = "2f421161cb492475f1661ddc9815a745a1c894592070661180fdec3d4872e9c3" 107 + 108 + [[package]] 109 + name = "core2" 110 + version = "0.4.0" 111 + source = "registry+https://github.com/rust-lang/crates.io-index" 112 + checksum = "b49ba7ef1ad6107f8824dbe97de947cbaac53c44e7f9756a1fba0d37c1eec505" 113 + dependencies = [ 114 + "memchr", 115 + ] 116 + 117 + [[package]] 118 + name = "crunchy" 119 + version = "0.2.4" 120 + source = "registry+https://github.com/rust-lang/crates.io-index" 121 + checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" 122 + 123 + [[package]] 124 + name = "data-encoding" 125 + version = "2.10.0" 126 + source = "registry+https://github.com/rust-lang/crates.io-index" 127 + checksum = "d7a1e2f27636f116493b8b860f5546edb47c8d8f8ea73e1d2a20be88e28d1fea" 128 + 129 + [[package]] 130 + name = "data-encoding-macro" 131 + version = "0.1.19" 132 + source = "registry+https://github.com/rust-lang/crates.io-index" 133 + checksum = "8142a83c17aa9461d637e649271eae18bf2edd00e91f2e105df36c3c16355bdb" 134 + dependencies = [ 135 + "data-encoding", 136 + "data-encoding-macro-internal", 137 + ] 138 + 139 + [[package]] 140 + name = "data-encoding-macro-internal" 141 + version = "0.1.17" 142 + source = "registry+https://github.com/rust-lang/crates.io-index" 143 + checksum = "7ab67060fc6b8ef687992d439ca0fa36e7ed17e9a0b16b25b601e8757df720de" 144 + dependencies = [ 145 + "data-encoding", 146 + "syn", 147 + ] 148 + 149 + [[package]] 150 + name = "futures" 151 + version = "0.3.32" 152 + source = "registry+https://github.com/rust-lang/crates.io-index" 153 + checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" 154 + dependencies = [ 155 + "futures-channel", 156 + "futures-core", 157 + "futures-executor", 158 + "futures-io", 159 + "futures-sink", 160 + "futures-task", 161 + "futures-util", 162 + ] 163 + 164 + [[package]] 165 + name = "futures-channel" 166 + version = "0.3.32" 167 + source = "registry+https://github.com/rust-lang/crates.io-index" 168 + checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" 169 + dependencies = [ 170 + "futures-core", 171 + "futures-sink", 172 + ] 173 + 174 + [[package]] 175 + name = "futures-core" 176 + version = "0.3.32" 177 + source = "registry+https://github.com/rust-lang/crates.io-index" 178 + checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" 179 + 180 + [[package]] 181 + name = "futures-executor" 182 + version = "0.3.32" 183 + source = "registry+https://github.com/rust-lang/crates.io-index" 184 + checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" 185 + dependencies = [ 186 + "futures-core", 187 + "futures-task", 188 + "futures-util", 189 + ] 190 + 191 + [[package]] 192 + name = "futures-io" 193 + version = "0.3.32" 194 + source = "registry+https://github.com/rust-lang/crates.io-index" 195 + checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" 196 + 197 + [[package]] 198 + name = "futures-macro" 199 + version = "0.3.32" 200 + source = "registry+https://github.com/rust-lang/crates.io-index" 201 + checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" 202 + dependencies = [ 203 + "proc-macro2", 204 + "quote", 205 + "syn", 206 + ] 207 + 208 + [[package]] 209 + name = "futures-sink" 210 + version = "0.3.32" 211 + source = "registry+https://github.com/rust-lang/crates.io-index" 212 + checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" 213 + 214 + [[package]] 215 + name = "futures-task" 216 + version = "0.3.32" 217 + source = "registry+https://github.com/rust-lang/crates.io-index" 218 + checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" 219 + 220 + [[package]] 221 + name = "futures-util" 222 + version = "0.3.32" 223 + source = "registry+https://github.com/rust-lang/crates.io-index" 224 + checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" 225 + dependencies = [ 226 + "futures-channel", 227 + "futures-core", 228 + "futures-io", 229 + "futures-macro", 230 + "futures-sink", 231 + "futures-task", 232 + "memchr", 233 + "pin-project-lite", 234 + "slab", 235 + ] 236 + 237 + [[package]] 238 + name = "half" 239 + version = "2.7.1" 240 + source = "registry+https://github.com/rust-lang/crates.io-index" 241 + checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" 242 + dependencies = [ 243 + "cfg-if", 244 + "crunchy", 245 + "zerocopy", 246 + ] 247 + 248 + [[package]] 249 + name = "ipld-core" 250 + version = "0.4.3" 251 + source = "registry+https://github.com/rust-lang/crates.io-index" 252 + checksum = "090f624976d72f0b0bb71b86d58dc16c15e069193067cb3a3a09d655246cbbda" 253 + dependencies = [ 254 + "cid", 255 + "serde", 256 + "serde_bytes", 257 + ] 258 + 259 + [[package]] 260 + name = "iroh-car" 261 + version = "0.5.1" 262 + source = "registry+https://github.com/rust-lang/crates.io-index" 263 + checksum = "cb7f8cd4cb9aa083fba8b52e921764252d0b4dcb1cd6d120b809dbfe1106e81a" 264 + dependencies = [ 265 + "anyhow", 266 + "cid", 267 + "futures", 268 + "serde", 269 + "serde_ipld_dagcbor", 270 + "thiserror", 271 + "tokio", 272 + "unsigned-varint 0.7.2", 273 + ] 274 + 275 + [[package]] 276 + name = "match-lookup" 277 + version = "0.1.2" 278 + source = "registry+https://github.com/rust-lang/crates.io-index" 279 + checksum = "757aee279b8bdbb9f9e676796fd459e4207a1f986e87886700abf589f5abf771" 280 + dependencies = [ 281 + "proc-macro2", 282 + "quote", 283 + "syn", 284 + ] 285 + 286 + [[package]] 287 + name = "memchr" 288 + version = "2.8.0" 289 + source = "registry+https://github.com/rust-lang/crates.io-index" 290 + checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" 291 + 292 + [[package]] 293 + name = "multibase" 294 + version = "0.9.2" 295 + source = "registry+https://github.com/rust-lang/crates.io-index" 296 + checksum = "8694bb4835f452b0e3bb06dbebb1d6fc5385b6ca1caf2e55fd165c042390ec77" 297 + dependencies = [ 298 + "base-x", 299 + "base256emoji", 300 + "data-encoding", 301 + "data-encoding-macro", 302 + ] 303 + 304 + [[package]] 305 + name = "multihash" 306 + version = "0.19.3" 307 + source = "registry+https://github.com/rust-lang/crates.io-index" 308 + checksum = "6b430e7953c29dd6a09afc29ff0bb69c6e306329ee6794700aee27b76a1aea8d" 309 + dependencies = [ 310 + "core2", 311 + "serde", 312 + "unsigned-varint 0.8.0", 313 + ] 314 + 315 + [[package]] 316 + name = "pin-project-lite" 317 + version = "0.2.16" 318 + source = "registry+https://github.com/rust-lang/crates.io-index" 319 + checksum = "3b3cff922bd51709b605d9ead9aa71031d81447142d828eb4a6eba76fe619f9b" 320 + 321 + [[package]] 322 + name = "proc-macro2" 323 + version = "1.0.106" 324 + source = "registry+https://github.com/rust-lang/crates.io-index" 325 + checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" 326 + dependencies = [ 327 + "unicode-ident", 328 + ] 329 + 330 + [[package]] 331 + name = "quote" 332 + version = "1.0.44" 333 + source = "registry+https://github.com/rust-lang/crates.io-index" 334 + checksum = "21b2ebcf727b7760c461f091f9f0f539b77b8e87f2fd88131e7f1b433b3cece4" 335 + dependencies = [ 336 + "proc-macro2", 337 + ] 338 + 339 + [[package]] 340 + name = "scopeguard" 341 + version = "1.2.0" 342 + source = "registry+https://github.com/rust-lang/crates.io-index" 343 + checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" 344 + 345 + [[package]] 346 + name = "serde" 347 + version = "1.0.228" 348 + source = "registry+https://github.com/rust-lang/crates.io-index" 349 + checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" 350 + dependencies = [ 351 + "serde_core", 352 + "serde_derive", 353 + ] 354 + 355 + [[package]] 356 + name = "serde_bytes" 357 + version = "0.11.19" 358 + source = "registry+https://github.com/rust-lang/crates.io-index" 359 + checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" 360 + dependencies = [ 361 + "serde", 362 + "serde_core", 363 + ] 364 + 365 + [[package]] 366 + name = "serde_core" 367 + version = "1.0.228" 368 + source = "registry+https://github.com/rust-lang/crates.io-index" 369 + checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" 370 + dependencies = [ 371 + "serde_derive", 372 + ] 373 + 374 + [[package]] 375 + name = "serde_derive" 376 + version = "1.0.228" 377 + source = "registry+https://github.com/rust-lang/crates.io-index" 378 + checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" 379 + dependencies = [ 380 + "proc-macro2", 381 + "quote", 382 + "syn", 383 + ] 384 + 385 + [[package]] 386 + name = "serde_ipld_dagcbor" 387 + version = "0.6.4" 388 + source = "registry+https://github.com/rust-lang/crates.io-index" 389 + checksum = "46182f4f08349a02b45c998ba3215d3f9de826246ba02bb9dddfe9a2a2100778" 390 + dependencies = [ 391 + "cbor4ii", 392 + "ipld-core", 393 + "scopeguard", 394 + "serde", 395 + ] 396 + 397 + [[package]] 398 + name = "slab" 399 + version = "0.4.12" 400 + source = "registry+https://github.com/rust-lang/crates.io-index" 401 + checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" 402 + 403 + [[package]] 404 + name = "syn" 405 + version = "2.0.117" 406 + source = "registry+https://github.com/rust-lang/crates.io-index" 407 + checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" 408 + dependencies = [ 409 + "proc-macro2", 410 + "quote", 411 + "unicode-ident", 412 + ] 413 + 414 + [[package]] 415 + name = "thiserror" 416 + version = "1.0.69" 417 + source = "registry+https://github.com/rust-lang/crates.io-index" 418 + checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52" 419 + dependencies = [ 420 + "thiserror-impl", 421 + ] 422 + 423 + [[package]] 424 + name = "thiserror-impl" 425 + version = "1.0.69" 426 + source = "registry+https://github.com/rust-lang/crates.io-index" 427 + checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" 428 + dependencies = [ 429 + "proc-macro2", 430 + "quote", 431 + "syn", 432 + ] 433 + 434 + [[package]] 435 + name = "tokio" 436 + version = "1.49.0" 437 + source = "registry+https://github.com/rust-lang/crates.io-index" 438 + checksum = "72a2903cd7736441aac9df9d7688bd0ce48edccaadf181c3b90be801e81d3d86" 439 + dependencies = [ 440 + "bytes", 441 + "pin-project-lite", 442 + "tokio-macros", 443 + ] 444 + 445 + [[package]] 446 + name = "tokio-macros" 447 + version = "2.6.0" 448 + source = "registry+https://github.com/rust-lang/crates.io-index" 449 + checksum = "af407857209536a95c8e56f8231ef2c2e2aff839b22e07a1ffcbc617e9db9fa5" 450 + dependencies = [ 451 + "proc-macro2", 452 + "quote", 453 + "syn", 454 + ] 455 + 456 + [[package]] 457 + name = "unicode-ident" 458 + version = "1.0.24" 459 + source = "registry+https://github.com/rust-lang/crates.io-index" 460 + checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" 461 + 462 + [[package]] 463 + name = "unsigned-varint" 464 + version = "0.7.2" 465 + source = "registry+https://github.com/rust-lang/crates.io-index" 466 + checksum = "6889a77d49f1f013504cec6bf97a2c730394adedaeb1deb5ea08949a50541105" 467 + 468 + [[package]] 469 + name = "unsigned-varint" 470 + version = "0.8.0" 471 + source = "registry+https://github.com/rust-lang/crates.io-index" 472 + checksum = "eb066959b24b5196ae73cb057f45598450d2c5f71460e98c49b738086eff9c06" 473 + 474 + [[package]] 475 + name = "zerocopy" 476 + version = "0.8.39" 477 + source = "registry+https://github.com/rust-lang/crates.io-index" 478 + checksum = "db6d35d663eadb6c932438e763b262fe1a70987f9ae936e60158176d710cae4a" 479 + dependencies = [ 480 + "zerocopy-derive", 481 + ] 482 + 483 + [[package]] 484 + name = "zerocopy-derive" 485 + version = "0.8.39" 486 + source = "registry+https://github.com/rust-lang/crates.io-index" 487 + checksum = "4122cd3169e94605190e77839c9a40d40ed048d305bfdc146e7df40ab0f3e517" 488 + dependencies = [ 489 + "proc-macro2", 490 + "quote", 491 + "syn", 492 + ]
+19
rust/Cargo.toml
··· 1 + [package] 2 + name = "atproto-bench" 3 + version = "0.1.0" 4 + edition = "2021" 5 + 6 + [dependencies] 7 + ciborium = "0.2" 8 + serde_ipld_dagcbor = "0.6" 9 + ipld-core = { version = "0.4", features = ["serde"] } 10 + serde = { version = "1", features = ["derive"] } 11 + serde_bytes = "0.11" 12 + tokio = { version = "1", features = ["rt", "rt-multi-thread", "macros"] } 13 + iroh-car = "0.5" 14 + 15 + [profile.release] 16 + opt-level = 3 17 + lto = true 18 + codegen-units = 1 19 + panic = "abort"
+247
rust/src/main.rs
··· 1 + //! atproto firehose benchmarks — rust (jacquard-style) 2 + //! 3 + //! decodes a corpus of real firehose frames using the same decode path as jacquard: 4 + //! - ciborium for the frame header (op + message type) 5 + //! - serde_ipld_dagcbor for the typed Commit body 6 + //! - iroh-car + serde_ipld_dagcbor for CAR block extraction 7 + 8 + use ipld_core::ipld::Ipld; 9 + use serde::Deserialize; 10 + use std::io::Cursor; 11 + use std::time::Instant; 12 + 13 + const WARMUP_PASSES: usize = 2; 14 + const MEASURED_PASSES: usize = 5; 15 + const FIXTURES_DIR: &str = "../fixtures"; 16 + 17 + struct CorpusInfo { 18 + raw: Vec<u8>, 19 + frame_ranges: Vec<(usize, usize)>, 20 + total_bytes: usize, 21 + min_frame: usize, 22 + max_frame: usize, 23 + } 24 + 25 + impl CorpusInfo { 26 + fn frames(&self) -> impl Iterator<Item = &[u8]> { 27 + self.frame_ranges 28 + .iter() 29 + .map(move |&(start, end)| &self.raw[start..end]) 30 + } 31 + } 32 + 33 + #[tokio::main] 34 + async fn main() { 35 + println!("\n=== rust benchmarks ===\n"); 36 + 37 + let corpus = match load_corpus("firehose-frames.bin") { 38 + Ok(c) => c, 39 + Err(e) => { 40 + println!("firehose-frames.bin: SKIP ({e})"); 41 + return; 42 + } 43 + }; 44 + 45 + println!( 46 + "corpus: {} frames, {} bytes total", 47 + corpus.frame_ranges.len(), 48 + corpus.total_bytes 49 + ); 50 + println!( 51 + " frame sizes: {}..{} bytes", 52 + corpus.min_frame, corpus.max_frame 53 + ); 54 + println!( 55 + " passes: {} warmup, {} measured\n", 56 + WARMUP_PASSES, MEASURED_PASSES 57 + ); 58 + 59 + // verify first frame 60 + if let Some(first) = corpus.frames().next() { 61 + match decode_full(first).await { 62 + Ok((commit, block_count)) => { 63 + println!( 64 + "first frame: repo={} ops={} blocks={}", 65 + commit.repo, 66 + commit.ops.len(), 67 + block_count, 68 + ); 69 + } 70 + Err(e) => println!("first frame verify failed: {e}"), 71 + } 72 + } 73 + println!(); 74 + 75 + if let Err(e) = bench_decode(&corpus).await { 76 + println!("decode: SKIP ({e})"); 77 + } 78 + 79 + println!(); 80 + } 81 + 82 + // --- types matching the AT Protocol firehose subscription --- 83 + 84 + #[allow(dead_code)] 85 + #[derive(Deserialize, Debug)] 86 + struct FrameHeader { 87 + op: i64, 88 + #[serde(default)] 89 + t: Option<String>, 90 + } 91 + 92 + #[allow(dead_code)] 93 + #[derive(Deserialize, Debug)] 94 + struct Commit { 95 + seq: i64, 96 + repo: String, 97 + rev: String, 98 + time: String, 99 + #[serde(default)] 100 + since: Option<String>, 101 + #[serde(default, with = "serde_bytes")] 102 + blocks: Vec<u8>, 103 + ops: Vec<RepoOp>, 104 + #[serde(default)] 105 + #[serde(rename = "tooBig")] 106 + too_big: bool, 107 + } 108 + 109 + #[allow(dead_code)] 110 + #[derive(Deserialize, Debug)] 111 + struct RepoOp { 112 + action: String, 113 + path: String, 114 + #[serde(default)] 115 + cid: Option<Ipld>, 116 + } 117 + 118 + // --- header parsing via ciborium (what jacquard does) --- 119 + 120 + /// wrapper that tracks bytes consumed by ciborium::from_reader 121 + struct TrackingReader<R> { 122 + inner: R, 123 + bytes_read: usize, 124 + } 125 + 126 + impl<R: std::io::Read> TrackingReader<R> { 127 + fn new(inner: R) -> Self { 128 + Self { 129 + inner, 130 + bytes_read: 0, 131 + } 132 + } 133 + } 134 + 135 + impl<R: std::io::Read> std::io::Read for TrackingReader<R> { 136 + fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> { 137 + let n = self.inner.read(buf)?; 138 + self.bytes_read += n; 139 + Ok(n) 140 + } 141 + } 142 + 143 + /// full decode: frame header + typed commit + CAR blocks + record decode. 144 + async fn decode_full(data: &[u8]) -> Result<(Commit, usize), Box<dyn std::error::Error>> { 145 + let mut reader = TrackingReader::new(Cursor::new(data)); 146 + let _header: FrameHeader = ciborium::from_reader(&mut reader)?; 147 + let body = &data[reader.bytes_read..]; 148 + let commit: Commit = serde_ipld_dagcbor::from_slice(body)?; 149 + 150 + let mut block_count = 0; 151 + if !commit.blocks.is_empty() { 152 + let mut car_reader = iroh_car::CarReader::new(Cursor::new(&commit.blocks)).await?; 153 + while let Some((_cid, block_data)) = car_reader.next_block().await? { 154 + let _: Ipld = serde_ipld_dagcbor::from_slice(&block_data)?; 155 + block_count += 1; 156 + } 157 + } 158 + 159 + Ok((commit, block_count)) 160 + } 161 + 162 + // --- benchmark --- 163 + 164 + async fn bench_decode(corpus: &CorpusInfo) -> Result<(), Box<dyn std::error::Error>> { 165 + for _ in 0..WARMUP_PASSES { 166 + for frame in corpus.frames() { 167 + decode_full(frame).await?; 168 + } 169 + } 170 + 171 + let mut total_frames: usize = 0; 172 + let start = Instant::now(); 173 + for _ in 0..MEASURED_PASSES { 174 + for frame in corpus.frames() { 175 + decode_full(frame).await?; 176 + total_frames += 1; 177 + } 178 + } 179 + let elapsed = start.elapsed(); 180 + 181 + report_corpus_result("decode", corpus, total_frames, elapsed); 182 + Ok(()) 183 + } 184 + 185 + // --- util --- 186 + 187 + fn report_corpus_result( 188 + name: &str, 189 + corpus: &CorpusInfo, 190 + total_frames: usize, 191 + elapsed: std::time::Duration, 192 + ) { 193 + let elapsed_s = elapsed.as_secs_f64(); 194 + let elapsed_ms = elapsed_s * 1000.0; 195 + let frames_per_sec = total_frames as f64 / elapsed_s; 196 + let total_bytes = corpus.total_bytes as f64 * MEASURED_PASSES as f64; 197 + let throughput_mb = total_bytes / (1024.0 * 1024.0) / elapsed_s; 198 + 199 + println!( 200 + "{:<14} {:>10.0} frames/sec {:>8.1} ms ({:.1} MB/s)", 201 + name, frames_per_sec, elapsed_ms, throughput_mb, 202 + ); 203 + } 204 + 205 + fn load_corpus(name: &str) -> Result<CorpusInfo, Box<dyn std::error::Error>> { 206 + let path = format!("{FIXTURES_DIR}/{name}"); 207 + let raw = std::fs::read(&path).map_err(|e| { 208 + eprintln!("cannot open {path}: {e}"); 209 + eprintln!("run `just capture` first to generate fixtures"); 210 + e 211 + })?; 212 + 213 + if raw.len() < 4 { 214 + return Err("corpus file too small".into()); 215 + } 216 + 217 + let frame_count = u32::from_be_bytes(raw[0..4].try_into().unwrap()) as usize; 218 + let mut frame_ranges = Vec::with_capacity(frame_count); 219 + let mut pos = 4usize; 220 + let mut total_bytes = 0usize; 221 + let mut min_frame = usize::MAX; 222 + let mut max_frame = 0usize; 223 + 224 + for _ in 0..frame_count { 225 + if pos + 4 > raw.len() { 226 + return Err("truncated corpus".into()); 227 + } 228 + let frame_len = u32::from_be_bytes(raw[pos..pos + 4].try_into().unwrap()) as usize; 229 + pos += 4; 230 + if pos + frame_len > raw.len() { 231 + return Err("truncated corpus".into()); 232 + } 233 + frame_ranges.push((pos, pos + frame_len)); 234 + pos += frame_len; 235 + total_bytes += frame_len; 236 + min_frame = min_frame.min(frame_len); 237 + max_frame = max_frame.max(frame_len); 238 + } 239 + 240 + Ok(CorpusInfo { 241 + raw, 242 + frame_ranges, 243 + total_bytes, 244 + min_frame, 245 + max_frame, 246 + }) 247 + }
+53
zig/build.zig
··· 1 + const std = @import("std"); 2 + 3 + pub fn build(b: *std.Build) void { 4 + const target = b.standardTargetOptions(.{}); 5 + const optimize = b.standardOptimizeOption(.{}); 6 + 7 + const zat_dep = b.dependency("zat", .{ 8 + .target = target, 9 + .optimize = optimize, 10 + }); 11 + const zat_mod = zat_dep.module("zat"); 12 + 13 + // get websocket from zat's transitive dependencies (for capture tool) 14 + const ws_mod = zat_dep.builder.dependency("websocket", .{ 15 + .target = target, 16 + .optimize = optimize, 17 + }).module("websocket"); 18 + 19 + // fixture capture tool — needs websocket for raw frame access 20 + const capture = b.addExecutable(.{ 21 + .name = "capture-fixtures", 22 + .root_module = b.createModule(.{ 23 + .root_source_file = b.path("src/capture.zig"), 24 + .target = target, 25 + .optimize = optimize, 26 + .imports = &.{ 27 + .{ .name = "zat", .module = zat_mod }, 28 + .{ .name = "websocket", .module = ws_mod }, 29 + }, 30 + }), 31 + }); 32 + b.installArtifact(capture); 33 + 34 + const run_capture = b.addRunArtifact(capture); 35 + const capture_step = b.step("run-capture", "capture fixtures from live network"); 36 + capture_step.dependOn(&run_capture.step); 37 + 38 + // benchmark 39 + const bench = b.addExecutable(.{ 40 + .name = "bench", 41 + .root_module = b.createModule(.{ 42 + .root_source_file = b.path("src/bench.zig"), 43 + .target = target, 44 + .optimize = optimize, 45 + .imports = &.{.{ .name = "zat", .module = zat_mod }}, 46 + }), 47 + }); 48 + b.installArtifact(bench); 49 + 50 + const run_bench = b.addRunArtifact(bench); 51 + const bench_step = b.step("run-bench", "run benchmarks"); 52 + bench_step.dependOn(&run_bench.step); 53 + }
+16
zig/build.zig.zon
··· 1 + .{ 2 + .name = .atproto_bench, 3 + .version = "0.0.1", 4 + .fingerprint = 0x7cdfbb8d616ff9e0, 5 + .minimum_zig_version = "0.15.0", 6 + .dependencies = .{ 7 + .zat = .{ 8 + .path = "../../zat", 9 + }, 10 + }, 11 + .paths = .{ 12 + "build.zig", 13 + "build.zig.zon", 14 + "src", 15 + }, 16 + }
+160
zig/src/bench.zig
··· 1 + //! atproto firehose benchmarks using zat 2 + //! 3 + //! decodes a corpus of real firehose frames using zat's public API: 4 + //! firehose.decodeFrame — the same call a real consumer makes. 5 + //! produces typed CommitEvent with ops and decoded records from CAR blocks. 6 + 7 + const std = @import("std"); 8 + const zat = @import("zat"); 9 + 10 + const Allocator = std.mem.Allocator; 11 + 12 + const warmup_passes: usize = 2; 13 + const measured_passes: usize = 5; 14 + const fixtures_dir = "../fixtures"; 15 + 16 + const CorpusInfo = struct { 17 + frames: []const []const u8, 18 + total_bytes: usize, 19 + min_frame: usize, 20 + max_frame: usize, 21 + }; 22 + 23 + pub fn main() !void { 24 + const allocator = std.heap.c_allocator; 25 + 26 + const corpus = try loadCorpus(allocator, fixtures_dir ++ "/firehose-frames.bin"); 27 + 28 + std.debug.print("\n=== zat benchmarks ===\n\n", .{}); 29 + std.debug.print("corpus: {d} frames, {d} bytes total\n", .{ corpus.frames.len, corpus.total_bytes }); 30 + std.debug.print(" frame sizes: {d}..{d} bytes\n", .{ corpus.min_frame, corpus.max_frame }); 31 + std.debug.print(" passes: {d} warmup, {d} measured\n\n", .{ warmup_passes, measured_passes }); 32 + 33 + // verify first frame decodes correctly 34 + { 35 + var arena = std.heap.ArenaAllocator.init(allocator); 36 + defer arena.deinit(); 37 + const event = try zat.firehose.decodeFrame(arena.allocator(), corpus.frames[0]); 38 + switch (event) { 39 + .commit => |c| { 40 + std.debug.print("first frame: repo={s} ops={d}\n\n", .{ 41 + c.repo, c.ops.len, 42 + }); 43 + }, 44 + else => { 45 + std.debug.print("first frame: not a commit event\n\n", .{}); 46 + }, 47 + } 48 + } 49 + 50 + benchDecodeFrame(allocator, corpus) catch |err| { 51 + std.debug.print("decode (reuse): SKIP ({s})\n", .{@errorName(err)}); 52 + }; 53 + 54 + benchDecodeFrameAlloc(allocator, corpus) catch |err| { 55 + std.debug.print("decode (alloc): SKIP ({s})\n", .{@errorName(err)}); 56 + }; 57 + 58 + std.debug.print("\n", .{}); 59 + } 60 + 61 + /// arena reuse: production allocation pattern — one arena, reset per frame. 62 + fn benchDecodeFrame(allocator: Allocator, corpus: CorpusInfo) !void { 63 + var arena = std.heap.ArenaAllocator.init(allocator); 64 + defer arena.deinit(); 65 + 66 + for (0..warmup_passes) |_| { 67 + for (corpus.frames) |frame| { 68 + _ = arena.reset(.retain_capacity); 69 + _ = try zat.firehose.decodeFrame(arena.allocator(), frame); 70 + } 71 + } 72 + 73 + var total_frames: usize = 0; 74 + var timer = try std.time.Timer.start(); 75 + for (0..measured_passes) |_| { 76 + for (corpus.frames) |frame| { 77 + _ = arena.reset(.retain_capacity); 78 + _ = try zat.firehose.decodeFrame(arena.allocator(), frame); 79 + total_frames += 1; 80 + } 81 + } 82 + reportCorpusResult("decode (reuse)", corpus, total_frames, timer.read()); 83 + } 84 + 85 + /// arena per-frame: fair cross-language comparison — fresh alloc/free per frame. 86 + fn benchDecodeFrameAlloc(allocator: Allocator, corpus: CorpusInfo) !void { 87 + for (0..warmup_passes) |_| { 88 + for (corpus.frames) |frame| { 89 + var arena = std.heap.ArenaAllocator.init(allocator); 90 + defer arena.deinit(); 91 + _ = try zat.firehose.decodeFrame(arena.allocator(), frame); 92 + } 93 + } 94 + 95 + var total_frames: usize = 0; 96 + var timer = try std.time.Timer.start(); 97 + for (0..measured_passes) |_| { 98 + for (corpus.frames) |frame| { 99 + var arena = std.heap.ArenaAllocator.init(allocator); 100 + defer arena.deinit(); 101 + _ = try zat.firehose.decodeFrame(arena.allocator(), frame); 102 + total_frames += 1; 103 + } 104 + } 105 + reportCorpusResult("decode (alloc)", corpus, total_frames, timer.read()); 106 + } 107 + 108 + fn reportCorpusResult(name: []const u8, corpus: CorpusInfo, total_frames: usize, elapsed_ns: u64) void { 109 + const elapsed_s = @as(f64, @floatFromInt(elapsed_ns)) / 1_000_000_000.0; 110 + const elapsed_ms = elapsed_s * 1000.0; 111 + const frames_per_sec = @as(f64, @floatFromInt(total_frames)) / elapsed_s; 112 + const total_bytes = @as(f64, @floatFromInt(corpus.total_bytes)) * @as(f64, @floatFromInt(measured_passes)); 113 + const throughput_mb = total_bytes / (1024.0 * 1024.0) / elapsed_s; 114 + 115 + std.debug.print("{s: <14} {d:>10.0} frames/sec {d:>8.1} ms ({d:.1} MB/s)\n", .{ 116 + name, 117 + frames_per_sec, 118 + elapsed_ms, 119 + throughput_mb, 120 + }); 121 + } 122 + 123 + fn loadCorpus(allocator: Allocator, path: []const u8) !CorpusInfo { 124 + const file = std.fs.cwd().openFile(path, .{}) catch |err| { 125 + std.debug.print("cannot open {s}: {s}\n", .{ path, @errorName(err) }); 126 + std.debug.print("run `just capture` first to generate fixtures\n", .{}); 127 + return err; 128 + }; 129 + defer file.close(); 130 + const data = try file.readToEndAlloc(allocator, 50 * 1024 * 1024); 131 + 132 + if (data.len < 4) return error.InvalidFormat; 133 + 134 + const frame_count = std.mem.readInt(u32, data[0..4], .big); 135 + var frames: std.ArrayListUnmanaged([]const u8) = .{}; 136 + var pos: usize = 4; 137 + var total_bytes: usize = 0; 138 + var min_frame: usize = std.math.maxInt(usize); 139 + var max_frame: usize = 0; 140 + 141 + for (0..frame_count) |_| { 142 + if (pos + 4 > data.len) return error.InvalidFormat; 143 + const frame_len = std.mem.readInt(u32, data[pos..][0..4], .big); 144 + pos += 4; 145 + if (pos + frame_len > data.len) return error.InvalidFormat; 146 + const frame = data[pos..][0..frame_len]; 147 + try frames.append(allocator, frame); 148 + pos += frame_len; 149 + total_bytes += frame_len; 150 + if (frame_len < min_frame) min_frame = frame_len; 151 + if (frame_len > max_frame) max_frame = frame_len; 152 + } 153 + 154 + return .{ 155 + .frames = try frames.toOwnedSlice(allocator), 156 + .total_bytes = total_bytes, 157 + .min_frame = min_frame, 158 + .max_frame = max_frame, 159 + }; 160 + }
+183
zig/src/capture.zig
··· 1 + //! capture fixtures from live AT Protocol network traffic. 2 + //! 3 + //! connects to jetstream and firehose via raw websocket, captures 4 + //! the first good message from each, and writes to ../fixtures/. 5 + 6 + const std = @import("std"); 7 + const zat = @import("zat"); 8 + const websocket = @import("websocket"); 9 + 10 + const Allocator = std.mem.Allocator; 11 + 12 + const fixtures_dir = "../fixtures"; 13 + const capture_duration_s = 10; 14 + const ns_per_s = std.time.ns_per_s; 15 + 16 + pub fn main() !void { 17 + var gpa: std.heap.GeneralPurposeAllocator(.{}) = .{}; 18 + defer _ = gpa.deinit(); 19 + const allocator = gpa.allocator(); 20 + 21 + std.debug.print("capturing fixtures from live network...\n", .{}); 22 + 23 + try captureJetstream(allocator); 24 + try captureFirehose(allocator); 25 + 26 + std.debug.print("done — all fixtures captured\n", .{}); 27 + } 28 + 29 + fn captureJetstream(allocator: Allocator) !void { 30 + std.debug.print("connecting to jetstream...\n", .{}); 31 + 32 + const host = "jetstream1.us-east.bsky.network"; 33 + var client = try websocket.Client.init(allocator, .{ 34 + .host = host, 35 + .port = 443, 36 + .tls = true, 37 + .max_size = 1024 * 1024, 38 + }); 39 + defer client.deinit(); 40 + 41 + var host_buf: [256]u8 = undefined; 42 + const host_header = try std.fmt.bufPrint(&host_buf, "Host: {s}\r\n", .{host}); 43 + 44 + try client.handshake("/subscribe?wantedCollections=app.bsky.feed.post", .{ 45 + .headers = host_header, 46 + }); 47 + 48 + std.debug.print("jetstream connected, waiting for commit with record...\n", .{}); 49 + 50 + var handler = JetstreamCaptureHandler{ .allocator = allocator }; 51 + client.readLoop(&handler) catch |err| switch (err) { 52 + error.ConnectionClosed => {}, 53 + else => return err, 54 + }; 55 + } 56 + 57 + const JetstreamCaptureHandler = struct { 58 + allocator: Allocator, 59 + 60 + pub fn serverMessage(self: *JetstreamCaptureHandler, data: []const u8) !void { 61 + _ = self; 62 + // jetstream sends JSON text frames — look for a commit with a record 63 + if (std.mem.indexOf(u8, data, "\"commit\"") != null and 64 + std.mem.indexOf(u8, data, "\"record\"") != null and 65 + std.mem.indexOf(u8, data, "\"create\"") != null) 66 + { 67 + try saveFile(fixtures_dir ++ "/jetstream-event.json", data); 68 + std.debug.print("wrote {d} bytes to jetstream-event.json\n", .{data.len}); 69 + 70 + // return error to break out of readLoop 71 + return error.ConnectionClosed; 72 + } 73 + } 74 + 75 + pub fn close(_: *JetstreamCaptureHandler) void {} 76 + }; 77 + 78 + fn captureFirehose(allocator: Allocator) !void { 79 + std.debug.print("connecting to firehose...\n", .{}); 80 + 81 + const host = "bsky.network"; 82 + var client = try websocket.Client.init(allocator, .{ 83 + .host = host, 84 + .port = 443, 85 + .tls = true, 86 + .max_size = 5 * 1024 * 1024, 87 + }); 88 + defer client.deinit(); 89 + 90 + var host_buf: [256]u8 = undefined; 91 + const host_header = try std.fmt.bufPrint(&host_buf, "Host: {s}\r\n", .{host}); 92 + 93 + try client.handshake("/xrpc/com.atproto.sync.subscribeRepos", .{ 94 + .headers = host_header, 95 + }); 96 + 97 + std.debug.print("firehose connected, capturing for ~{d}s...\n", .{capture_duration_s}); 98 + 99 + var handler = FirehoseCaptureHandler{ 100 + .allocator = allocator, 101 + .deadline = @as(i128, std.time.nanoTimestamp()) + @as(i128, capture_duration_s) * ns_per_s, 102 + }; 103 + client.readLoop(&handler) catch |err| switch (err) { 104 + error.ConnectionClosed => {}, 105 + else => return err, 106 + }; 107 + 108 + // write length-prefixed binary format 109 + const frames = handler.frames.items; 110 + if (frames.len == 0) { 111 + std.debug.print("no frames captured!\n", .{}); 112 + return; 113 + } 114 + 115 + var total_bytes: usize = 0; 116 + for (frames) |f| total_bytes += f.len; 117 + 118 + std.debug.print("\ncaptured {d} frames ({d} bytes total)\n", .{ frames.len, total_bytes }); 119 + std.debug.print("writing {s}/firehose-frames.bin...\n", .{fixtures_dir}); 120 + 121 + const file = try std.fs.cwd().createFile(fixtures_dir ++ "/firehose-frames.bin", .{}); 122 + defer file.close(); 123 + 124 + // header: frame count as u32 BE 125 + var count_buf: [4]u8 = undefined; 126 + std.mem.writeInt(u32, &count_buf, @intCast(frames.len), .big); 127 + try file.writeAll(&count_buf); 128 + 129 + // each frame: [u32 BE length][frame bytes] 130 + for (frames) |frame| { 131 + var len_buf: [4]u8 = undefined; 132 + std.mem.writeInt(u32, &len_buf, @intCast(frame.len), .big); 133 + try file.writeAll(&len_buf); 134 + try file.writeAll(frame); 135 + } 136 + 137 + std.debug.print("done — wrote {d} frames to firehose-frames.bin\n", .{frames.len}); 138 + 139 + // free duped frame slices 140 + for (handler.frames.items) |f| allocator.free(f); 141 + handler.frames.deinit(allocator); 142 + } 143 + 144 + const FirehoseCaptureHandler = struct { 145 + allocator: Allocator, 146 + deadline: i128, 147 + frames: std.ArrayListUnmanaged([]const u8) = .{}, 148 + total_bytes: usize = 0, 149 + 150 + pub fn serverMessage(self: *FirehoseCaptureHandler, data: []const u8) !void { 151 + // check deadline 152 + if (std.time.nanoTimestamp() >= self.deadline) { 153 + return error.ConnectionClosed; 154 + } 155 + 156 + // pre-filter: only save frames that decode as commits with ops 157 + var arena = std.heap.ArenaAllocator.init(self.allocator); 158 + defer arena.deinit(); 159 + 160 + const event = zat.firehose.decodeFrame(arena.allocator(), data) catch return; 161 + switch (event) { 162 + .commit => |commit| { 163 + if (commit.ops.len == 0) return; 164 + }, 165 + else => return, 166 + } 167 + 168 + // dupe and buffer the raw frame 169 + const duped = try self.allocator.dupe(u8, data); 170 + try self.frames.append(self.allocator, duped); 171 + self.total_bytes += data.len; 172 + 173 + std.debug.print("\rcaptured {d} frames ({d} bytes)...", .{ self.frames.items.len, self.total_bytes }); 174 + } 175 + 176 + pub fn close(_: *FirehoseCaptureHandler) void {} 177 + }; 178 + 179 + fn saveFile(path: []const u8, data: []const u8) !void { 180 + const file = try std.fs.cwd().createFile(path, .{}); 181 + defer file.close(); 182 + try file.writeAll(data); 183 + }