Ramjet is a relay consumer that supports configurable forward and track collections, as well as record reconciliation.
event-stream
relay
firehose
riblt
atprotocol
release: 0.1.0
+16948
-1
+3
.gitignore
+3
.gitignore
+59
CLAUDE.md
+59
CLAUDE.md
···
1
+
# CLAUDE.md
2
+
3
+
## Project Overview
4
+
5
+
Ramjet is a single-node Rust service that consumes the ATProtocol firehose, persists records for tracked collections in fjall v3 (pure-Rust LSM-tree), and re-emits events via WebSocket fan-out with priority ordering.
6
+
7
+
## Common Commands
8
+
9
+
- **Build**: `cargo build`
10
+
- **Run tests**: `cargo test`
11
+
- **Check code**: `cargo check`
12
+
- **Format code**: `cargo fmt`
13
+
- **Run**: `cargo run -- --db-path /tmp/ramjet-test --listen-addr 127.0.0.1:8080 --tracked-collections "garden.lexicon.**"`
14
+
15
+
## Architecture
16
+
17
+
Three async pipelines connected by channels:
18
+
1. **Firehose Ingester** — WebSocket client connects to relay, decodes CBOR frames (#commit, #identity, #account), sends `IngestEvent` over mpsc channel. Auto-reconnects with 2s delay.
19
+
2. **Batch Writer** — Collects events into configurable batches (size + timeout), writes atomically to fjall via `WriteBatch`. Routes tracked collections to records keyspace + high-priority fan-out, forward-only collections to low-priority fan-out. Filters denied repos. Queues tooBig commits for backfill.
20
+
3. **WebSocket Fan-out** — Serves `com.atproto.sync.subscribeRepos`. Replays from events keyspace on cursor, then streams live via biased `tokio::select!` (high > low > client recv).
21
+
4. **Identity Worker** — Processes identity events from mpsc channel. Resolves DID documents, updates `handle_to_did` and `did_to_doc` keyspaces, removes stale handle mappings.
22
+
5. **Backfill Worker** — Polls `backfill_queue\x00` prefix in meta keyspace every 5s. Fetches CAR from PDS, parses with `atproto_repo::repo::MemoryRepository`, writes records.
23
+
24
+
Eight fjall keyspaces: `records`, `events`, `meta`, `repo_state`, `did_to_doc`, `handle_to_did`, `blobs`, `blob_meta`.
25
+
26
+
## Module Structure
27
+
28
+
- `src/config.rs` — CLI args (clap), `CollectionPattern`/`CollectionMatcher`, `ServiceConfig`
29
+
- `src/errors.rs` — `RamjetError` enum (thiserror)
30
+
- `src/types.rs` — Core domain types: `RecordValue`, `CommitOp`, `CommitOps`, `RepoState`, `OpType`, `AccountStatus`, `EventType`, `FanOutEvent`, `SharedFanOutEvent`
31
+
- `src/storage/mod.rs` — `FjallDb` struct (8 keyspaces), `open()`, `batch()`, `get_cursor()`, `set_cursor()`, `get_repo_state()`, `current_sequence()`, `oldest_event_sequence()`
32
+
- `src/storage/keys.rs` — Key encoding/decoding with `\x00` separators, versioned record keys (`did\x00collection\x00rkey\x00rev`), BE u64 for event keys
33
+
- `src/storage/encoding.rs` — Binary encode/decode for `RecordValue` (CID + data, tombstone = empty), `RepoState`, compact events
34
+
- `src/server/mod.rs` — `AppState`, `build_router()` with 12 routes
35
+
- `src/server/health.rs` — `GET /_health`
36
+
- `src/server/metrics.rs` — Prometheus metrics (`prometheus-client 0.23`)
37
+
- `src/server/xrpc.rs` — XRPC handlers: getRecord, listRecords, describeRepo, resolveIdentity, resolveHandle, resolveDid
38
+
- `src/server/admin.rs` — Admin handlers with JWT auth: getState, setState, resync
39
+
- `src/server/websocket.rs` — WebSocket fan-out with cursor replay and biased priority delivery
40
+
- `src/pipeline/mod.rs` — Module re-exports
41
+
- `src/pipeline/ingester.rs` — Firehose WebSocket client with CBOR decoding (atproto-dasl)
42
+
- `src/pipeline/writer.rs` — Batch writer with collection routing and deny-list filtering
43
+
- `src/pipeline/fanout.rs` — `FanOutChannels` with high/low priority broadcast channels (capacity 8192)
44
+
- `src/pipeline/identity.rs` — Identity resolution worker
45
+
- `src/pipeline/backfill.rs` — Backfill worker with CAR parsing
46
+
47
+
## Error Handling
48
+
49
+
Errors follow: `error-ramjet-{domain}-{number} {message}: {details}`
50
+
51
+
Use `thiserror` for error enums, `anyhow::Result` for function returns.
52
+
53
+
## Key Conventions
54
+
55
+
- Use `tracing` for structured logging. All calls must be fully qualified (`tracing::info!`, etc.).
56
+
- fjall v3 iterators yield `Guard` items — call `guard.into_inner()` to get `Result<(UserKey, UserValue)>`.
57
+
- ATProto crates are path dependencies from `../../atproto-crates-studious-guide/amman-v2/crates/`.
58
+
- Run `cargo fmt` before committing any changes.
59
+
- Edition 2024, rust-version 1.94. `unsafe_code = "forbid"`.
+4707
Cargo.lock
+4707
Cargo.lock
···
1
+
# This file is automatically @generated by Cargo.
2
+
# It is not intended for manual editing.
3
+
version = 4
4
+
5
+
[[package]]
6
+
name = "aho-corasick"
7
+
version = "1.1.4"
8
+
source = "registry+https://github.com/rust-lang/crates.io-index"
9
+
checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301"
10
+
dependencies = [
11
+
"memchr",
12
+
]
13
+
14
+
[[package]]
15
+
name = "allocator-api2"
16
+
version = "0.2.21"
17
+
source = "registry+https://github.com/rust-lang/crates.io-index"
18
+
checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
19
+
20
+
[[package]]
21
+
name = "anstream"
22
+
version = "0.6.21"
23
+
source = "registry+https://github.com/rust-lang/crates.io-index"
24
+
checksum = "43d5b281e737544384e969a5ccad3f1cdd24b48086a0fc1b2a5262a26b8f4f4a"
25
+
dependencies = [
26
+
"anstyle",
27
+
"anstyle-parse",
28
+
"anstyle-query",
29
+
"anstyle-wincon",
30
+
"colorchoice",
31
+
"is_terminal_polyfill",
32
+
"utf8parse",
33
+
]
34
+
35
+
[[package]]
36
+
name = "anstyle"
37
+
version = "1.0.13"
38
+
source = "registry+https://github.com/rust-lang/crates.io-index"
39
+
checksum = "5192cca8006f1fd4f7237516f40fa183bb07f8fbdfedaa0036de5ea9b0b45e78"
40
+
41
+
[[package]]
42
+
name = "anstyle-parse"
43
+
version = "0.2.7"
44
+
source = "registry+https://github.com/rust-lang/crates.io-index"
45
+
checksum = "4e7644824f0aa2c7b9384579234ef10eb7efb6a0deb83f9630a49594dd9c15c2"
46
+
dependencies = [
47
+
"utf8parse",
48
+
]
49
+
50
+
[[package]]
51
+
name = "anstyle-query"
52
+
version = "1.1.5"
53
+
source = "registry+https://github.com/rust-lang/crates.io-index"
54
+
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
55
+
dependencies = [
56
+
"windows-sys 0.61.2",
57
+
]
58
+
59
+
[[package]]
60
+
name = "anstyle-wincon"
61
+
version = "3.0.11"
62
+
source = "registry+https://github.com/rust-lang/crates.io-index"
63
+
checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
64
+
dependencies = [
65
+
"anstyle",
66
+
"once_cell_polyfill",
67
+
"windows-sys 0.61.2",
68
+
]
69
+
70
+
[[package]]
71
+
name = "anyhow"
72
+
version = "1.0.102"
73
+
source = "registry+https://github.com/rust-lang/crates.io-index"
74
+
checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c"
75
+
76
+
[[package]]
77
+
name = "arrayref"
78
+
version = "0.3.9"
79
+
source = "registry+https://github.com/rust-lang/crates.io-index"
80
+
checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb"
81
+
82
+
[[package]]
83
+
name = "arrayvec"
84
+
version = "0.7.6"
85
+
source = "registry+https://github.com/rust-lang/crates.io-index"
86
+
checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
87
+
88
+
[[package]]
89
+
name = "async-trait"
90
+
version = "0.1.89"
91
+
source = "registry+https://github.com/rust-lang/crates.io-index"
92
+
checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb"
93
+
dependencies = [
94
+
"proc-macro2",
95
+
"quote",
96
+
"syn 2.0.117",
97
+
]
98
+
99
+
[[package]]
100
+
name = "atomic"
101
+
version = "0.6.1"
102
+
source = "registry+https://github.com/rust-lang/crates.io-index"
103
+
checksum = "a89cbf775b137e9b968e67227ef7f775587cde3fd31b0d8599dbd0f598a48340"
104
+
dependencies = [
105
+
"bytemuck",
106
+
]
107
+
108
+
[[package]]
109
+
name = "atomic-waker"
110
+
version = "1.1.2"
111
+
source = "registry+https://github.com/rust-lang/crates.io-index"
112
+
checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0"
113
+
114
+
[[package]]
115
+
name = "atproto-dasl"
116
+
version = "0.14.3"
117
+
source = "git+https://tangled.org/ngerakines.me/atproto-crates?branch=main#a402bbaa5f4c44dac2634a51d3c61218f3441292"
118
+
dependencies = [
119
+
"blake3",
120
+
"cid",
121
+
"futures",
122
+
"multihash",
123
+
"serde",
124
+
"serde_bytes",
125
+
"sha2",
126
+
"tempfile",
127
+
"thiserror 2.0.18",
128
+
"tokio",
129
+
"tracing",
130
+
"url",
131
+
]
132
+
133
+
[[package]]
134
+
name = "atproto-identity"
135
+
version = "0.14.3"
136
+
source = "git+https://tangled.org/ngerakines.me/atproto-crates?branch=main#a402bbaa5f4c44dac2634a51d3c61218f3441292"
137
+
dependencies = [
138
+
"anyhow",
139
+
"async-trait",
140
+
"atproto-dasl",
141
+
"base64",
142
+
"chrono",
143
+
"data-encoding",
144
+
"ecdsa",
145
+
"elliptic-curve",
146
+
"hickory-resolver",
147
+
"k256",
148
+
"lru 0.12.5",
149
+
"multibase",
150
+
"p256",
151
+
"p384",
152
+
"rand 0.8.5",
153
+
"reqwest",
154
+
"serde",
155
+
"serde_json",
156
+
"sha2",
157
+
"thiserror 2.0.18",
158
+
"tokio",
159
+
"tracing",
160
+
"url",
161
+
]
162
+
163
+
[[package]]
164
+
name = "atproto-oauth"
165
+
version = "0.14.3"
166
+
source = "git+https://tangled.org/ngerakines.me/atproto-crates?branch=main#a402bbaa5f4c44dac2634a51d3c61218f3441292"
167
+
dependencies = [
168
+
"anyhow",
169
+
"async-trait",
170
+
"atproto-identity",
171
+
"base64",
172
+
"chrono",
173
+
"ecdsa",
174
+
"elliptic-curve",
175
+
"k256",
176
+
"lru 0.12.5",
177
+
"multibase",
178
+
"p256",
179
+
"p384",
180
+
"rand 0.8.5",
181
+
"reqwest",
182
+
"reqwest-chain",
183
+
"reqwest-middleware",
184
+
"serde",
185
+
"serde_json",
186
+
"sha2",
187
+
"thiserror 2.0.18",
188
+
"tokio",
189
+
"tracing",
190
+
"ulid",
191
+
]
192
+
193
+
[[package]]
194
+
name = "atproto-record"
195
+
version = "0.14.3"
196
+
source = "git+https://tangled.org/ngerakines.me/atproto-crates?branch=main#a402bbaa5f4c44dac2634a51d3c61218f3441292"
197
+
dependencies = [
198
+
"anyhow",
199
+
"atproto-dasl",
200
+
"atproto-identity",
201
+
"base64",
202
+
"chrono",
203
+
"cid",
204
+
"multihash",
205
+
"rand 0.8.5",
206
+
"serde",
207
+
"serde_json",
208
+
"sha2",
209
+
"thiserror 2.0.18",
210
+
]
211
+
212
+
[[package]]
213
+
name = "atproto-repo"
214
+
version = "0.14.3"
215
+
source = "git+https://tangled.org/ngerakines.me/atproto-crates?branch=main#a402bbaa5f4c44dac2634a51d3c61218f3441292"
216
+
dependencies = [
217
+
"anyhow",
218
+
"async-trait",
219
+
"atproto-dasl",
220
+
"atproto-identity",
221
+
"cid",
222
+
"futures",
223
+
"multihash",
224
+
"serde",
225
+
"serde_bytes",
226
+
"serde_json",
227
+
"sha2",
228
+
"tempfile",
229
+
"thiserror 2.0.18",
230
+
"tokio",
231
+
"tracing",
232
+
]
233
+
234
+
[[package]]
235
+
name = "atproto-xrpcs"
236
+
version = "0.14.3"
237
+
source = "git+https://tangled.org/ngerakines.me/atproto-crates?branch=main#a402bbaa5f4c44dac2634a51d3c61218f3441292"
238
+
dependencies = [
239
+
"anyhow",
240
+
"async-trait",
241
+
"atproto-identity",
242
+
"atproto-oauth",
243
+
"atproto-record",
244
+
"axum",
245
+
"base64",
246
+
"chrono",
247
+
"elliptic-curve",
248
+
"hickory-resolver",
249
+
"http",
250
+
"rand 0.8.5",
251
+
"reqwest",
252
+
"reqwest-chain",
253
+
"reqwest-middleware",
254
+
"serde",
255
+
"serde_json",
256
+
"thiserror 2.0.18",
257
+
"tokio",
258
+
"tracing",
259
+
]
260
+
261
+
[[package]]
262
+
name = "autocfg"
263
+
version = "1.5.0"
264
+
source = "registry+https://github.com/rust-lang/crates.io-index"
265
+
checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8"
266
+
267
+
[[package]]
268
+
name = "axum"
269
+
version = "0.8.8"
270
+
source = "registry+https://github.com/rust-lang/crates.io-index"
271
+
checksum = "8b52af3cb4058c895d37317bb27508dccc8e5f2d39454016b297bf4a400597b8"
272
+
dependencies = [
273
+
"axum-core",
274
+
"axum-macros",
275
+
"base64",
276
+
"bytes",
277
+
"form_urlencoded",
278
+
"futures-util",
279
+
"http",
280
+
"http-body",
281
+
"http-body-util",
282
+
"hyper",
283
+
"hyper-util",
284
+
"itoa",
285
+
"matchit",
286
+
"memchr",
287
+
"mime",
288
+
"percent-encoding",
289
+
"pin-project-lite",
290
+
"serde_core",
291
+
"serde_json",
292
+
"serde_path_to_error",
293
+
"serde_urlencoded",
294
+
"sha1",
295
+
"sync_wrapper",
296
+
"tokio",
297
+
"tokio-tungstenite",
298
+
"tower",
299
+
"tower-layer",
300
+
"tower-service",
301
+
"tracing",
302
+
]
303
+
304
+
[[package]]
305
+
name = "axum-core"
306
+
version = "0.5.6"
307
+
source = "registry+https://github.com/rust-lang/crates.io-index"
308
+
checksum = "08c78f31d7b1291f7ee735c1c6780ccde7785daae9a9206026862dab7d8792d1"
309
+
dependencies = [
310
+
"bytes",
311
+
"futures-core",
312
+
"http",
313
+
"http-body",
314
+
"http-body-util",
315
+
"mime",
316
+
"pin-project-lite",
317
+
"sync_wrapper",
318
+
"tower-layer",
319
+
"tower-service",
320
+
"tracing",
321
+
]
322
+
323
+
[[package]]
324
+
name = "axum-macros"
325
+
version = "0.5.0"
326
+
source = "registry+https://github.com/rust-lang/crates.io-index"
327
+
checksum = "604fde5e028fea851ce1d8570bbdc034bec850d157f7569d10f347d06808c05c"
328
+
dependencies = [
329
+
"proc-macro2",
330
+
"quote",
331
+
"syn 2.0.117",
332
+
]
333
+
334
+
[[package]]
335
+
name = "base-x"
336
+
version = "0.2.11"
337
+
source = "registry+https://github.com/rust-lang/crates.io-index"
338
+
checksum = "4cbbc9d0964165b47557570cce6c952866c2678457aca742aafc9fb771d30270"
339
+
340
+
[[package]]
341
+
name = "base16ct"
342
+
version = "0.2.0"
343
+
source = "registry+https://github.com/rust-lang/crates.io-index"
344
+
checksum = "4c7f02d4ea65f2c1853089ffd8d2787bdbc63de2f0d29dedbcf8ccdfa0ccd4cf"
345
+
346
+
[[package]]
347
+
name = "base256emoji"
348
+
version = "1.0.2"
349
+
source = "registry+https://github.com/rust-lang/crates.io-index"
350
+
checksum = "b5e9430d9a245a77c92176e649af6e275f20839a48389859d1661e9a128d077c"
351
+
dependencies = [
352
+
"const-str",
353
+
"match-lookup",
354
+
]
355
+
356
+
[[package]]
357
+
name = "base64"
358
+
version = "0.22.1"
359
+
source = "registry+https://github.com/rust-lang/crates.io-index"
360
+
checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
361
+
362
+
[[package]]
363
+
name = "base64ct"
364
+
version = "1.8.3"
365
+
source = "registry+https://github.com/rust-lang/crates.io-index"
366
+
checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06"
367
+
368
+
[[package]]
369
+
name = "bit-set"
370
+
version = "0.5.3"
371
+
source = "registry+https://github.com/rust-lang/crates.io-index"
372
+
checksum = "0700ddab506f33b20a03b13996eccd309a48e5ff77d0d95926aa0210fb4e95f1"
373
+
dependencies = [
374
+
"bit-vec",
375
+
]
376
+
377
+
[[package]]
378
+
name = "bit-vec"
379
+
version = "0.6.3"
380
+
source = "registry+https://github.com/rust-lang/crates.io-index"
381
+
checksum = "349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb"
382
+
383
+
[[package]]
384
+
name = "bitflags"
385
+
version = "1.3.2"
386
+
source = "registry+https://github.com/rust-lang/crates.io-index"
387
+
checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
388
+
389
+
[[package]]
390
+
name = "bitflags"
391
+
version = "2.11.0"
392
+
source = "registry+https://github.com/rust-lang/crates.io-index"
393
+
checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af"
394
+
395
+
[[package]]
396
+
name = "blake3"
397
+
version = "1.8.3"
398
+
source = "registry+https://github.com/rust-lang/crates.io-index"
399
+
checksum = "2468ef7d57b3fb7e16b576e8377cdbde2320c60e1491e961d11da40fc4f02a2d"
400
+
dependencies = [
401
+
"arrayref",
402
+
"arrayvec",
403
+
"cc",
404
+
"cfg-if",
405
+
"constant_time_eq",
406
+
"cpufeatures 0.2.17",
407
+
]
408
+
409
+
[[package]]
410
+
name = "block-buffer"
411
+
version = "0.10.4"
412
+
source = "registry+https://github.com/rust-lang/crates.io-index"
413
+
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
414
+
dependencies = [
415
+
"generic-array",
416
+
]
417
+
418
+
[[package]]
419
+
name = "bumpalo"
420
+
version = "3.20.2"
421
+
source = "registry+https://github.com/rust-lang/crates.io-index"
422
+
checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb"
423
+
424
+
[[package]]
425
+
name = "bytemuck"
426
+
version = "1.25.0"
427
+
source = "registry+https://github.com/rust-lang/crates.io-index"
428
+
checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec"
429
+
430
+
[[package]]
431
+
name = "byteorder-lite"
432
+
version = "0.1.0"
433
+
source = "registry+https://github.com/rust-lang/crates.io-index"
434
+
checksum = "8f1fe948ff07f4bd06c30984e69f5b4899c516a3ef74f34df92a2df2ab535495"
435
+
436
+
[[package]]
437
+
name = "bytes"
438
+
version = "1.11.1"
439
+
source = "registry+https://github.com/rust-lang/crates.io-index"
440
+
checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33"
441
+
442
+
[[package]]
443
+
name = "byteview"
444
+
version = "0.10.1"
445
+
source = "registry+https://github.com/rust-lang/crates.io-index"
446
+
checksum = "1c53ba0f290bfc610084c05582d9c5d421662128fc69f4bf236707af6fd321b9"
447
+
448
+
[[package]]
449
+
name = "castaway"
450
+
version = "0.2.4"
451
+
source = "registry+https://github.com/rust-lang/crates.io-index"
452
+
checksum = "dec551ab6e7578819132c713a93c022a05d60159dc86e7a7050223577484c55a"
453
+
dependencies = [
454
+
"rustversion",
455
+
]
456
+
457
+
[[package]]
458
+
name = "cc"
459
+
version = "1.2.56"
460
+
source = "registry+https://github.com/rust-lang/crates.io-index"
461
+
checksum = "aebf35691d1bfb0ac386a69bac2fde4dd276fb618cf8bf4f5318fe285e821bb2"
462
+
dependencies = [
463
+
"find-msvc-tools",
464
+
"jobserver",
465
+
"libc",
466
+
"shlex",
467
+
]
468
+
469
+
[[package]]
470
+
name = "cfg-if"
471
+
version = "1.0.4"
472
+
source = "registry+https://github.com/rust-lang/crates.io-index"
473
+
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
474
+
475
+
[[package]]
476
+
name = "cfg_aliases"
477
+
version = "0.2.1"
478
+
source = "registry+https://github.com/rust-lang/crates.io-index"
479
+
checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
480
+
481
+
[[package]]
482
+
name = "chacha20"
483
+
version = "0.10.0"
484
+
source = "registry+https://github.com/rust-lang/crates.io-index"
485
+
checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601"
486
+
dependencies = [
487
+
"cfg-if",
488
+
"cpufeatures 0.3.0",
489
+
"rand_core 0.10.0",
490
+
]
491
+
492
+
[[package]]
493
+
name = "chrono"
494
+
version = "0.4.44"
495
+
source = "registry+https://github.com/rust-lang/crates.io-index"
496
+
checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0"
497
+
dependencies = [
498
+
"num-traits",
499
+
"serde",
500
+
]
501
+
502
+
[[package]]
503
+
name = "cid"
504
+
version = "0.11.1"
505
+
source = "registry+https://github.com/rust-lang/crates.io-index"
506
+
checksum = "3147d8272e8fa0ccd29ce51194dd98f79ddfb8191ba9e3409884e751798acf3a"
507
+
dependencies = [
508
+
"core2",
509
+
"multibase",
510
+
"multihash",
511
+
"unsigned-varint",
512
+
]
513
+
514
+
[[package]]
515
+
name = "clap"
516
+
version = "4.5.60"
517
+
source = "registry+https://github.com/rust-lang/crates.io-index"
518
+
checksum = "2797f34da339ce31042b27d23607e051786132987f595b02ba4f6a6dffb7030a"
519
+
dependencies = [
520
+
"clap_builder",
521
+
"clap_derive",
522
+
]
523
+
524
+
[[package]]
525
+
name = "clap_builder"
526
+
version = "4.5.60"
527
+
source = "registry+https://github.com/rust-lang/crates.io-index"
528
+
checksum = "24a241312cea5059b13574bb9b3861cabf758b879c15190b37b6d6fd63ab6876"
529
+
dependencies = [
530
+
"anstream",
531
+
"anstyle",
532
+
"clap_lex",
533
+
"strsim",
534
+
]
535
+
536
+
[[package]]
537
+
name = "clap_derive"
538
+
version = "4.5.55"
539
+
source = "registry+https://github.com/rust-lang/crates.io-index"
540
+
checksum = "a92793da1a46a5f2a02a6f4c46c6496b28c43638adea8306fcb0caa1634f24e5"
541
+
dependencies = [
542
+
"heck",
543
+
"proc-macro2",
544
+
"quote",
545
+
"syn 2.0.117",
546
+
]
547
+
548
+
[[package]]
549
+
name = "clap_lex"
550
+
version = "1.0.0"
551
+
source = "registry+https://github.com/rust-lang/crates.io-index"
552
+
checksum = "3a822ea5bc7590f9d40f1ba12c0dc3c2760f3482c6984db1573ad11031420831"
553
+
554
+
[[package]]
555
+
name = "colorchoice"
556
+
version = "1.0.4"
557
+
source = "registry+https://github.com/rust-lang/crates.io-index"
558
+
checksum = "b05b61dc5112cbb17e4b6cd61790d9845d13888356391624cbe7e41efeac1e75"
559
+
560
+
[[package]]
561
+
name = "compact_str"
562
+
version = "0.9.0"
563
+
source = "registry+https://github.com/rust-lang/crates.io-index"
564
+
checksum = "3fdb1325a1cece981e8a296ab8f0f9b63ae357bd0784a9faaf548cc7b480707a"
565
+
dependencies = [
566
+
"castaway",
567
+
"cfg-if",
568
+
"itoa",
569
+
"rustversion",
570
+
"ryu",
571
+
"static_assertions",
572
+
]
573
+
574
+
[[package]]
575
+
name = "compare"
576
+
version = "0.0.6"
577
+
source = "registry+https://github.com/rust-lang/crates.io-index"
578
+
checksum = "ea0095f6103c2a8b44acd6fd15960c801dafebf02e21940360833e0673f48ba7"
579
+
580
+
[[package]]
581
+
name = "const-oid"
582
+
version = "0.9.6"
583
+
source = "registry+https://github.com/rust-lang/crates.io-index"
584
+
checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8"
585
+
586
+
[[package]]
587
+
name = "const-str"
588
+
version = "0.4.3"
589
+
source = "registry+https://github.com/rust-lang/crates.io-index"
590
+
checksum = "2f421161cb492475f1661ddc9815a745a1c894592070661180fdec3d4872e9c3"
591
+
592
+
[[package]]
593
+
name = "constant_time_eq"
594
+
version = "0.4.2"
595
+
source = "registry+https://github.com/rust-lang/crates.io-index"
596
+
checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b"
597
+
598
+
[[package]]
599
+
name = "convert_case"
600
+
version = "0.10.0"
601
+
source = "registry+https://github.com/rust-lang/crates.io-index"
602
+
checksum = "633458d4ef8c78b72454de2d54fd6ab2e60f9e02be22f3c6104cdc8a4e0fceb9"
603
+
dependencies = [
604
+
"unicode-segmentation",
605
+
]
606
+
607
+
[[package]]
608
+
name = "core-foundation"
609
+
version = "0.9.4"
610
+
source = "registry+https://github.com/rust-lang/crates.io-index"
611
+
checksum = "91e195e091a93c46f7102ec7818a2aa394e1e1771c3ab4825963fa03e45afb8f"
612
+
dependencies = [
613
+
"core-foundation-sys",
614
+
"libc",
615
+
]
616
+
617
+
[[package]]
618
+
name = "core-foundation"
619
+
version = "0.10.1"
620
+
source = "registry+https://github.com/rust-lang/crates.io-index"
621
+
checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6"
622
+
dependencies = [
623
+
"core-foundation-sys",
624
+
"libc",
625
+
]
626
+
627
+
[[package]]
628
+
name = "core-foundation-sys"
629
+
version = "0.8.7"
630
+
source = "registry+https://github.com/rust-lang/crates.io-index"
631
+
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
632
+
633
+
[[package]]
634
+
name = "core2"
635
+
version = "0.4.0"
636
+
source = "registry+https://github.com/rust-lang/crates.io-index"
637
+
checksum = "b49ba7ef1ad6107f8824dbe97de947cbaac53c44e7f9756a1fba0d37c1eec505"
638
+
dependencies = [
639
+
"memchr",
640
+
]
641
+
642
+
[[package]]
643
+
name = "cpufeatures"
644
+
version = "0.2.17"
645
+
source = "registry+https://github.com/rust-lang/crates.io-index"
646
+
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
647
+
dependencies = [
648
+
"libc",
649
+
]
650
+
651
+
[[package]]
652
+
name = "cpufeatures"
653
+
version = "0.3.0"
654
+
source = "registry+https://github.com/rust-lang/crates.io-index"
655
+
checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
656
+
dependencies = [
657
+
"libc",
658
+
]
659
+
660
+
[[package]]
661
+
name = "critical-section"
662
+
version = "1.2.0"
663
+
source = "registry+https://github.com/rust-lang/crates.io-index"
664
+
checksum = "790eea4361631c5e7d22598ecd5723ff611904e3344ce8720784c93e3d83d40b"
665
+
666
+
[[package]]
667
+
name = "crossbeam-channel"
668
+
version = "0.5.15"
669
+
source = "registry+https://github.com/rust-lang/crates.io-index"
670
+
checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2"
671
+
dependencies = [
672
+
"crossbeam-utils",
673
+
]
674
+
675
+
[[package]]
676
+
name = "crossbeam-epoch"
677
+
version = "0.9.18"
678
+
source = "registry+https://github.com/rust-lang/crates.io-index"
679
+
checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e"
680
+
dependencies = [
681
+
"crossbeam-utils",
682
+
]
683
+
684
+
[[package]]
685
+
name = "crossbeam-skiplist"
686
+
version = "0.1.3"
687
+
source = "registry+https://github.com/rust-lang/crates.io-index"
688
+
checksum = "df29de440c58ca2cc6e587ec3d22347551a32435fbde9d2bff64e78a9ffa151b"
689
+
dependencies = [
690
+
"crossbeam-epoch",
691
+
"crossbeam-utils",
692
+
]
693
+
694
+
[[package]]
695
+
name = "crossbeam-utils"
696
+
version = "0.8.21"
697
+
source = "registry+https://github.com/rust-lang/crates.io-index"
698
+
checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28"
699
+
700
+
[[package]]
701
+
name = "crossterm"
702
+
version = "0.29.0"
703
+
source = "registry+https://github.com/rust-lang/crates.io-index"
704
+
checksum = "d8b9f2e4c67f833b660cdb0a3523065869fb35570177239812ed4c905aeff87b"
705
+
dependencies = [
706
+
"bitflags 2.11.0",
707
+
"crossterm_winapi",
708
+
"derive_more",
709
+
"document-features",
710
+
"mio",
711
+
"parking_lot",
712
+
"rustix",
713
+
"signal-hook",
714
+
"signal-hook-mio",
715
+
"winapi",
716
+
]
717
+
718
+
[[package]]
719
+
name = "crossterm_winapi"
720
+
version = "0.9.1"
721
+
source = "registry+https://github.com/rust-lang/crates.io-index"
722
+
checksum = "acdd7c62a3665c7f6830a51635d9ac9b23ed385797f70a83bb8bafe9c572ab2b"
723
+
dependencies = [
724
+
"winapi",
725
+
]
726
+
727
+
[[package]]
728
+
name = "crypto-bigint"
729
+
version = "0.5.5"
730
+
source = "registry+https://github.com/rust-lang/crates.io-index"
731
+
checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76"
732
+
dependencies = [
733
+
"generic-array",
734
+
"rand_core 0.6.4",
735
+
"subtle",
736
+
"zeroize",
737
+
]
738
+
739
+
[[package]]
740
+
name = "crypto-common"
741
+
version = "0.1.6"
742
+
source = "registry+https://github.com/rust-lang/crates.io-index"
743
+
checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3"
744
+
dependencies = [
745
+
"generic-array",
746
+
"typenum",
747
+
]
748
+
749
+
[[package]]
750
+
name = "csscolorparser"
751
+
version = "0.6.2"
752
+
source = "registry+https://github.com/rust-lang/crates.io-index"
753
+
checksum = "eb2a7d3066da2de787b7f032c736763eb7ae5d355f81a68bab2675a96008b0bf"
754
+
dependencies = [
755
+
"lab",
756
+
"phf",
757
+
]
758
+
759
+
[[package]]
760
+
name = "darling"
761
+
version = "0.23.0"
762
+
source = "registry+https://github.com/rust-lang/crates.io-index"
763
+
checksum = "25ae13da2f202d56bd7f91c25fba009e7717a1e4a1cc98a76d844b65ae912e9d"
764
+
dependencies = [
765
+
"darling_core",
766
+
"darling_macro",
767
+
]
768
+
769
+
[[package]]
770
+
name = "darling_core"
771
+
version = "0.23.0"
772
+
source = "registry+https://github.com/rust-lang/crates.io-index"
773
+
checksum = "9865a50f7c335f53564bb694ef660825eb8610e0a53d3e11bf1b0d3df31e03b0"
774
+
dependencies = [
775
+
"ident_case",
776
+
"proc-macro2",
777
+
"quote",
778
+
"strsim",
779
+
"syn 2.0.117",
780
+
]
781
+
782
+
[[package]]
783
+
name = "darling_macro"
784
+
version = "0.23.0"
785
+
source = "registry+https://github.com/rust-lang/crates.io-index"
786
+
checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d"
787
+
dependencies = [
788
+
"darling_core",
789
+
"quote",
790
+
"syn 2.0.117",
791
+
]
792
+
793
+
[[package]]
794
+
name = "dashmap"
795
+
version = "6.1.0"
796
+
source = "registry+https://github.com/rust-lang/crates.io-index"
797
+
checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf"
798
+
dependencies = [
799
+
"cfg-if",
800
+
"crossbeam-utils",
801
+
"hashbrown 0.14.5",
802
+
"lock_api",
803
+
"once_cell",
804
+
"parking_lot_core",
805
+
]
806
+
807
+
[[package]]
808
+
name = "data-encoding"
809
+
version = "2.10.0"
810
+
source = "registry+https://github.com/rust-lang/crates.io-index"
811
+
checksum = "d7a1e2f27636f116493b8b860f5546edb47c8d8f8ea73e1d2a20be88e28d1fea"
812
+
813
+
[[package]]
814
+
name = "data-encoding-macro"
815
+
version = "0.1.19"
816
+
source = "registry+https://github.com/rust-lang/crates.io-index"
817
+
checksum = "8142a83c17aa9461d637e649271eae18bf2edd00e91f2e105df36c3c16355bdb"
818
+
dependencies = [
819
+
"data-encoding",
820
+
"data-encoding-macro-internal",
821
+
]
822
+
823
+
[[package]]
824
+
name = "data-encoding-macro-internal"
825
+
version = "0.1.17"
826
+
source = "registry+https://github.com/rust-lang/crates.io-index"
827
+
checksum = "7ab67060fc6b8ef687992d439ca0fa36e7ed17e9a0b16b25b601e8757df720de"
828
+
dependencies = [
829
+
"data-encoding",
830
+
"syn 2.0.117",
831
+
]
832
+
833
+
[[package]]
834
+
name = "deltae"
835
+
version = "0.3.2"
836
+
source = "registry+https://github.com/rust-lang/crates.io-index"
837
+
checksum = "5729f5117e208430e437df2f4843f5e5952997175992d1414f94c57d61e270b4"
838
+
839
+
[[package]]
840
+
name = "der"
841
+
version = "0.7.10"
842
+
source = "registry+https://github.com/rust-lang/crates.io-index"
843
+
checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb"
844
+
dependencies = [
845
+
"const-oid",
846
+
"pem-rfc7468",
847
+
"zeroize",
848
+
]
849
+
850
+
[[package]]
851
+
name = "deranged"
852
+
version = "0.5.8"
853
+
source = "registry+https://github.com/rust-lang/crates.io-index"
854
+
checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
855
+
dependencies = [
856
+
"powerfmt",
857
+
]
858
+
859
+
[[package]]
860
+
name = "derive_more"
861
+
version = "2.1.1"
862
+
source = "registry+https://github.com/rust-lang/crates.io-index"
863
+
checksum = "d751e9e49156b02b44f9c1815bcb94b984cdcc4396ecc32521c739452808b134"
864
+
dependencies = [
865
+
"derive_more-impl",
866
+
]
867
+
868
+
[[package]]
869
+
name = "derive_more-impl"
870
+
version = "2.1.1"
871
+
source = "registry+https://github.com/rust-lang/crates.io-index"
872
+
checksum = "799a97264921d8623a957f6c3b9011f3b5492f557bbb7a5a19b7fa6d06ba8dcb"
873
+
dependencies = [
874
+
"convert_case",
875
+
"proc-macro2",
876
+
"quote",
877
+
"rustc_version",
878
+
"syn 2.0.117",
879
+
]
880
+
881
+
[[package]]
882
+
name = "digest"
883
+
version = "0.10.7"
884
+
source = "registry+https://github.com/rust-lang/crates.io-index"
885
+
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
886
+
dependencies = [
887
+
"block-buffer",
888
+
"const-oid",
889
+
"crypto-common",
890
+
"subtle",
891
+
]
892
+
893
+
[[package]]
894
+
name = "displaydoc"
895
+
version = "0.2.5"
896
+
source = "registry+https://github.com/rust-lang/crates.io-index"
897
+
checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0"
898
+
dependencies = [
899
+
"proc-macro2",
900
+
"quote",
901
+
"syn 2.0.117",
902
+
]
903
+
904
+
[[package]]
905
+
name = "document-features"
906
+
version = "0.2.12"
907
+
source = "registry+https://github.com/rust-lang/crates.io-index"
908
+
checksum = "d4b8a88685455ed29a21542a33abd9cb6510b6b129abadabdcef0f4c55bc8f61"
909
+
dependencies = [
910
+
"litrs",
911
+
]
912
+
913
+
[[package]]
914
+
name = "dtoa"
915
+
version = "1.0.11"
916
+
source = "registry+https://github.com/rust-lang/crates.io-index"
917
+
checksum = "4c3cf4824e2d5f025c7b531afcb2325364084a16806f6d47fbc1f5fbd9960590"
918
+
919
+
[[package]]
920
+
name = "ecdsa"
921
+
version = "0.16.9"
922
+
source = "registry+https://github.com/rust-lang/crates.io-index"
923
+
checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca"
924
+
dependencies = [
925
+
"der",
926
+
"digest",
927
+
"elliptic-curve",
928
+
"rfc6979",
929
+
"serdect",
930
+
"signature",
931
+
"spki",
932
+
]
933
+
934
+
[[package]]
935
+
name = "either"
936
+
version = "1.15.0"
937
+
source = "registry+https://github.com/rust-lang/crates.io-index"
938
+
checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719"
939
+
940
+
[[package]]
941
+
name = "elliptic-curve"
942
+
version = "0.13.8"
943
+
source = "registry+https://github.com/rust-lang/crates.io-index"
944
+
checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47"
945
+
dependencies = [
946
+
"base16ct",
947
+
"base64ct",
948
+
"crypto-bigint",
949
+
"digest",
950
+
"ff",
951
+
"generic-array",
952
+
"group",
953
+
"hkdf",
954
+
"pem-rfc7468",
955
+
"pkcs8",
956
+
"rand_core 0.6.4",
957
+
"sec1",
958
+
"serde_json",
959
+
"serdect",
960
+
"subtle",
961
+
"zeroize",
962
+
]
963
+
964
+
[[package]]
965
+
name = "encoding_rs"
966
+
version = "0.8.35"
967
+
source = "registry+https://github.com/rust-lang/crates.io-index"
968
+
checksum = "75030f3c4f45dafd7586dd6780965a8c7e8e285a5ecb86713e63a79c5b2766f3"
969
+
dependencies = [
970
+
"cfg-if",
971
+
]
972
+
973
+
[[package]]
974
+
name = "enum-as-inner"
975
+
version = "0.6.1"
976
+
source = "registry+https://github.com/rust-lang/crates.io-index"
977
+
checksum = "a1e6a265c649f3f5979b601d26f1d05ada116434c87741c9493cb56218f76cbc"
978
+
dependencies = [
979
+
"heck",
980
+
"proc-macro2",
981
+
"quote",
982
+
"syn 2.0.117",
983
+
]
984
+
985
+
[[package]]
986
+
name = "enum_dispatch"
987
+
version = "0.3.13"
988
+
source = "registry+https://github.com/rust-lang/crates.io-index"
989
+
checksum = "aa18ce2bc66555b3218614519ac839ddb759a7d6720732f979ef8d13be147ecd"
990
+
dependencies = [
991
+
"once_cell",
992
+
"proc-macro2",
993
+
"quote",
994
+
"syn 2.0.117",
995
+
]
996
+
997
+
[[package]]
998
+
name = "equivalent"
999
+
version = "1.0.2"
1000
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1001
+
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
1002
+
1003
+
[[package]]
1004
+
name = "errno"
1005
+
version = "0.3.14"
1006
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1007
+
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
1008
+
dependencies = [
1009
+
"libc",
1010
+
"windows-sys 0.61.2",
1011
+
]
1012
+
1013
+
[[package]]
1014
+
name = "euclid"
1015
+
version = "0.22.13"
1016
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1017
+
checksum = "df61bf483e837f88d5c2291dcf55c67be7e676b3a51acc48db3a7b163b91ed63"
1018
+
dependencies = [
1019
+
"num-traits",
1020
+
]
1021
+
1022
+
[[package]]
1023
+
name = "fancy-regex"
1024
+
version = "0.11.0"
1025
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1026
+
checksum = "b95f7c0680e4142284cf8b22c14a476e87d61b004a3a0861872b32ef7ead40a2"
1027
+
dependencies = [
1028
+
"bit-set",
1029
+
"regex",
1030
+
]
1031
+
1032
+
[[package]]
1033
+
name = "fastrand"
1034
+
version = "2.3.0"
1035
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1036
+
checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be"
1037
+
1038
+
[[package]]
1039
+
name = "ff"
1040
+
version = "0.13.1"
1041
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1042
+
checksum = "c0b50bfb653653f9ca9095b427bed08ab8d75a137839d9ad64eb11810d5b6393"
1043
+
dependencies = [
1044
+
"rand_core 0.6.4",
1045
+
"subtle",
1046
+
]
1047
+
1048
+
[[package]]
1049
+
name = "filedescriptor"
1050
+
version = "0.8.3"
1051
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1052
+
checksum = "e40758ed24c9b2eeb76c35fb0aebc66c626084edd827e07e1552279814c6682d"
1053
+
dependencies = [
1054
+
"libc",
1055
+
"thiserror 1.0.69",
1056
+
"winapi",
1057
+
]
1058
+
1059
+
[[package]]
1060
+
name = "find-msvc-tools"
1061
+
version = "0.1.9"
1062
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1063
+
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
1064
+
1065
+
[[package]]
1066
+
name = "finl_unicode"
1067
+
version = "1.4.0"
1068
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1069
+
checksum = "9844ddc3a6e533d62bba727eb6c28b5d360921d5175e9ff0f1e621a5c590a4d5"
1070
+
1071
+
[[package]]
1072
+
name = "fixedbitset"
1073
+
version = "0.4.2"
1074
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1075
+
checksum = "0ce7134b9999ecaf8bcd65542e436736ef32ddca1b3e06094cb6ec5755203b80"
1076
+
1077
+
[[package]]
1078
+
name = "fjall"
1079
+
version = "3.1.0"
1080
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1081
+
checksum = "40cb1eb0cef3792900897b32c8282f6417bc978f6af46400a2f14bf0e649ae30"
1082
+
dependencies = [
1083
+
"byteorder-lite",
1084
+
"byteview",
1085
+
"dashmap",
1086
+
"flume",
1087
+
"log",
1088
+
"lsm-tree",
1089
+
"lz4_flex",
1090
+
"tempfile",
1091
+
"xxhash-rust",
1092
+
]
1093
+
1094
+
[[package]]
1095
+
name = "flume"
1096
+
version = "0.12.0"
1097
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1098
+
checksum = "5e139bc46ca777eb5efaf62df0ab8cc5fd400866427e56c68b22e414e53bd3be"
1099
+
dependencies = [
1100
+
"spin",
1101
+
]
1102
+
1103
+
[[package]]
1104
+
name = "fnv"
1105
+
version = "1.0.7"
1106
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1107
+
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
1108
+
1109
+
[[package]]
1110
+
name = "foldhash"
1111
+
version = "0.1.5"
1112
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1113
+
checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
1114
+
1115
+
[[package]]
1116
+
name = "foldhash"
1117
+
version = "0.2.0"
1118
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1119
+
checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
1120
+
1121
+
[[package]]
1122
+
name = "form_urlencoded"
1123
+
version = "1.2.2"
1124
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1125
+
checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf"
1126
+
dependencies = [
1127
+
"percent-encoding",
1128
+
]
1129
+
1130
+
[[package]]
1131
+
name = "futures"
1132
+
version = "0.3.32"
1133
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1134
+
checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d"
1135
+
dependencies = [
1136
+
"futures-channel",
1137
+
"futures-core",
1138
+
"futures-executor",
1139
+
"futures-io",
1140
+
"futures-sink",
1141
+
"futures-task",
1142
+
"futures-util",
1143
+
]
1144
+
1145
+
[[package]]
1146
+
name = "futures-channel"
1147
+
version = "0.3.32"
1148
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1149
+
checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d"
1150
+
dependencies = [
1151
+
"futures-core",
1152
+
"futures-sink",
1153
+
]
1154
+
1155
+
[[package]]
1156
+
name = "futures-core"
1157
+
version = "0.3.32"
1158
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1159
+
checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d"
1160
+
1161
+
[[package]]
1162
+
name = "futures-executor"
1163
+
version = "0.3.32"
1164
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1165
+
checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d"
1166
+
dependencies = [
1167
+
"futures-core",
1168
+
"futures-task",
1169
+
"futures-util",
1170
+
]
1171
+
1172
+
[[package]]
1173
+
name = "futures-io"
1174
+
version = "0.3.32"
1175
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1176
+
checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718"
1177
+
1178
+
[[package]]
1179
+
name = "futures-macro"
1180
+
version = "0.3.32"
1181
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1182
+
checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b"
1183
+
dependencies = [
1184
+
"proc-macro2",
1185
+
"quote",
1186
+
"syn 2.0.117",
1187
+
]
1188
+
1189
+
[[package]]
1190
+
name = "futures-sink"
1191
+
version = "0.3.32"
1192
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1193
+
checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893"
1194
+
1195
+
[[package]]
1196
+
name = "futures-task"
1197
+
version = "0.3.32"
1198
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1199
+
checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393"
1200
+
1201
+
[[package]]
1202
+
name = "futures-timer"
1203
+
version = "3.0.3"
1204
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1205
+
checksum = "f288b0a4f20f9a56b5d1da57e2227c661b7b16168e2f72365f57b63326e29b24"
1206
+
1207
+
[[package]]
1208
+
name = "futures-util"
1209
+
version = "0.3.32"
1210
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1211
+
checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6"
1212
+
dependencies = [
1213
+
"futures-channel",
1214
+
"futures-core",
1215
+
"futures-io",
1216
+
"futures-macro",
1217
+
"futures-sink",
1218
+
"futures-task",
1219
+
"memchr",
1220
+
"pin-project-lite",
1221
+
"slab",
1222
+
]
1223
+
1224
+
[[package]]
1225
+
name = "generic-array"
1226
+
version = "0.14.9"
1227
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1228
+
checksum = "4bb6743198531e02858aeaea5398fcc883e71851fcbcb5a2f773e2fb6cb1edf2"
1229
+
dependencies = [
1230
+
"typenum",
1231
+
"version_check",
1232
+
"zeroize",
1233
+
]
1234
+
1235
+
[[package]]
1236
+
name = "getrandom"
1237
+
version = "0.2.17"
1238
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1239
+
checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0"
1240
+
dependencies = [
1241
+
"cfg-if",
1242
+
"js-sys",
1243
+
"libc",
1244
+
"wasi",
1245
+
"wasm-bindgen",
1246
+
]
1247
+
1248
+
[[package]]
1249
+
name = "getrandom"
1250
+
version = "0.3.4"
1251
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1252
+
checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd"
1253
+
dependencies = [
1254
+
"cfg-if",
1255
+
"js-sys",
1256
+
"libc",
1257
+
"r-efi 5.3.0",
1258
+
"wasip2",
1259
+
"wasm-bindgen",
1260
+
]
1261
+
1262
+
[[package]]
1263
+
name = "getrandom"
1264
+
version = "0.4.2"
1265
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1266
+
checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555"
1267
+
dependencies = [
1268
+
"cfg-if",
1269
+
"libc",
1270
+
"r-efi 6.0.0",
1271
+
"rand_core 0.10.0",
1272
+
"wasip2",
1273
+
"wasip3",
1274
+
]
1275
+
1276
+
[[package]]
1277
+
name = "governor"
1278
+
version = "0.8.1"
1279
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1280
+
checksum = "be93b4ec2e4710b04d9264c0c7350cdd62a8c20e5e4ac732552ebb8f0debe8eb"
1281
+
dependencies = [
1282
+
"cfg-if",
1283
+
"dashmap",
1284
+
"futures-sink",
1285
+
"futures-timer",
1286
+
"futures-util",
1287
+
"getrandom 0.3.4",
1288
+
"no-std-compat",
1289
+
"nonzero_ext",
1290
+
"parking_lot",
1291
+
"portable-atomic",
1292
+
"quanta",
1293
+
"rand 0.9.2",
1294
+
"smallvec",
1295
+
"spinning_top",
1296
+
"web-time",
1297
+
]
1298
+
1299
+
[[package]]
1300
+
name = "group"
1301
+
version = "0.13.0"
1302
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1303
+
checksum = "f0f9ef7462f7c099f518d754361858f86d8a07af53ba9af0fe635bbccb151a63"
1304
+
dependencies = [
1305
+
"ff",
1306
+
"rand_core 0.6.4",
1307
+
"subtle",
1308
+
]
1309
+
1310
+
[[package]]
1311
+
name = "h2"
1312
+
version = "0.4.13"
1313
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1314
+
checksum = "2f44da3a8150a6703ed5d34e164b875fd14c2cdab9af1252a9a1020bde2bdc54"
1315
+
dependencies = [
1316
+
"atomic-waker",
1317
+
"bytes",
1318
+
"fnv",
1319
+
"futures-core",
1320
+
"futures-sink",
1321
+
"http",
1322
+
"indexmap",
1323
+
"slab",
1324
+
"tokio",
1325
+
"tokio-util",
1326
+
"tracing",
1327
+
]
1328
+
1329
+
[[package]]
1330
+
name = "hashbrown"
1331
+
version = "0.14.5"
1332
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1333
+
checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1"
1334
+
1335
+
[[package]]
1336
+
name = "hashbrown"
1337
+
version = "0.15.5"
1338
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1339
+
checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1"
1340
+
dependencies = [
1341
+
"allocator-api2",
1342
+
"equivalent",
1343
+
"foldhash 0.1.5",
1344
+
]
1345
+
1346
+
[[package]]
1347
+
name = "hashbrown"
1348
+
version = "0.16.1"
1349
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1350
+
checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100"
1351
+
dependencies = [
1352
+
"allocator-api2",
1353
+
"equivalent",
1354
+
"foldhash 0.2.0",
1355
+
]
1356
+
1357
+
[[package]]
1358
+
name = "heck"
1359
+
version = "0.5.0"
1360
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1361
+
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
1362
+
1363
+
[[package]]
1364
+
name = "hex"
1365
+
version = "0.4.3"
1366
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1367
+
checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70"
1368
+
1369
+
[[package]]
1370
+
name = "hickory-proto"
1371
+
version = "0.25.2"
1372
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1373
+
checksum = "f8a6fe56c0038198998a6f217ca4e7ef3a5e51f46163bd6dd60b5c71ca6c6502"
1374
+
dependencies = [
1375
+
"async-trait",
1376
+
"cfg-if",
1377
+
"data-encoding",
1378
+
"enum-as-inner",
1379
+
"futures-channel",
1380
+
"futures-io",
1381
+
"futures-util",
1382
+
"idna",
1383
+
"ipnet",
1384
+
"once_cell",
1385
+
"rand 0.9.2",
1386
+
"ring",
1387
+
"thiserror 2.0.18",
1388
+
"tinyvec",
1389
+
"tokio",
1390
+
"tracing",
1391
+
"url",
1392
+
]
1393
+
1394
+
[[package]]
1395
+
name = "hickory-resolver"
1396
+
version = "0.25.2"
1397
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1398
+
checksum = "dc62a9a99b0bfb44d2ab95a7208ac952d31060efc16241c87eaf36406fecf87a"
1399
+
dependencies = [
1400
+
"cfg-if",
1401
+
"futures-util",
1402
+
"hickory-proto",
1403
+
"ipconfig",
1404
+
"moka",
1405
+
"once_cell",
1406
+
"parking_lot",
1407
+
"rand 0.9.2",
1408
+
"resolv-conf",
1409
+
"smallvec",
1410
+
"thiserror 2.0.18",
1411
+
"tokio",
1412
+
"tracing",
1413
+
]
1414
+
1415
+
[[package]]
1416
+
name = "hkdf"
1417
+
version = "0.12.4"
1418
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1419
+
checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7"
1420
+
dependencies = [
1421
+
"hmac",
1422
+
]
1423
+
1424
+
[[package]]
1425
+
name = "hmac"
1426
+
version = "0.12.1"
1427
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1428
+
checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e"
1429
+
dependencies = [
1430
+
"digest",
1431
+
]
1432
+
1433
+
[[package]]
1434
+
name = "http"
1435
+
version = "1.4.0"
1436
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1437
+
checksum = "e3ba2a386d7f85a81f119ad7498ebe444d2e22c2af0b86b069416ace48b3311a"
1438
+
dependencies = [
1439
+
"bytes",
1440
+
"itoa",
1441
+
]
1442
+
1443
+
[[package]]
1444
+
name = "http-body"
1445
+
version = "1.0.1"
1446
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1447
+
checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184"
1448
+
dependencies = [
1449
+
"bytes",
1450
+
"http",
1451
+
]
1452
+
1453
+
[[package]]
1454
+
name = "http-body-util"
1455
+
version = "0.1.3"
1456
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1457
+
checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a"
1458
+
dependencies = [
1459
+
"bytes",
1460
+
"futures-core",
1461
+
"http",
1462
+
"http-body",
1463
+
"pin-project-lite",
1464
+
]
1465
+
1466
+
[[package]]
1467
+
name = "httparse"
1468
+
version = "1.10.1"
1469
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1470
+
checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87"
1471
+
1472
+
[[package]]
1473
+
name = "httpdate"
1474
+
version = "1.0.3"
1475
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1476
+
checksum = "df3b46402a9d5adb4c86a0cf463f42e19994e3ee891101b1841f30a545cb49a9"
1477
+
1478
+
[[package]]
1479
+
name = "hyper"
1480
+
version = "1.8.1"
1481
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1482
+
checksum = "2ab2d4f250c3d7b1c9fcdff1cece94ea4e2dfbec68614f7b87cb205f24ca9d11"
1483
+
dependencies = [
1484
+
"atomic-waker",
1485
+
"bytes",
1486
+
"futures-channel",
1487
+
"futures-core",
1488
+
"h2",
1489
+
"http",
1490
+
"http-body",
1491
+
"httparse",
1492
+
"httpdate",
1493
+
"itoa",
1494
+
"pin-project-lite",
1495
+
"pin-utils",
1496
+
"smallvec",
1497
+
"tokio",
1498
+
"want",
1499
+
]
1500
+
1501
+
[[package]]
1502
+
name = "hyper-rustls"
1503
+
version = "0.27.7"
1504
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1505
+
checksum = "e3c93eb611681b207e1fe55d5a71ecf91572ec8a6705cdb6857f7d8d5242cf58"
1506
+
dependencies = [
1507
+
"http",
1508
+
"hyper",
1509
+
"hyper-util",
1510
+
"rustls",
1511
+
"rustls-pki-types",
1512
+
"tokio",
1513
+
"tokio-rustls",
1514
+
"tower-service",
1515
+
"webpki-roots",
1516
+
]
1517
+
1518
+
[[package]]
1519
+
name = "hyper-util"
1520
+
version = "0.1.20"
1521
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1522
+
checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0"
1523
+
dependencies = [
1524
+
"base64",
1525
+
"bytes",
1526
+
"futures-channel",
1527
+
"futures-util",
1528
+
"http",
1529
+
"http-body",
1530
+
"hyper",
1531
+
"ipnet",
1532
+
"libc",
1533
+
"percent-encoding",
1534
+
"pin-project-lite",
1535
+
"socket2 0.6.3",
1536
+
"system-configuration",
1537
+
"tokio",
1538
+
"tower-service",
1539
+
"tracing",
1540
+
"windows-registry",
1541
+
]
1542
+
1543
+
[[package]]
1544
+
name = "icu_collections"
1545
+
version = "2.1.1"
1546
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1547
+
checksum = "4c6b649701667bbe825c3b7e6388cb521c23d88644678e83c0c4d0a621a34b43"
1548
+
dependencies = [
1549
+
"displaydoc",
1550
+
"potential_utf",
1551
+
"yoke",
1552
+
"zerofrom",
1553
+
"zerovec",
1554
+
]
1555
+
1556
+
[[package]]
1557
+
name = "icu_locale_core"
1558
+
version = "2.1.1"
1559
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1560
+
checksum = "edba7861004dd3714265b4db54a3c390e880ab658fec5f7db895fae2046b5bb6"
1561
+
dependencies = [
1562
+
"displaydoc",
1563
+
"litemap",
1564
+
"tinystr",
1565
+
"writeable",
1566
+
"zerovec",
1567
+
]
1568
+
1569
+
[[package]]
1570
+
name = "icu_normalizer"
1571
+
version = "2.1.1"
1572
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1573
+
checksum = "5f6c8828b67bf8908d82127b2054ea1b4427ff0230ee9141c54251934ab1b599"
1574
+
dependencies = [
1575
+
"icu_collections",
1576
+
"icu_normalizer_data",
1577
+
"icu_properties",
1578
+
"icu_provider",
1579
+
"smallvec",
1580
+
"zerovec",
1581
+
]
1582
+
1583
+
[[package]]
1584
+
name = "icu_normalizer_data"
1585
+
version = "2.1.1"
1586
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1587
+
checksum = "7aedcccd01fc5fe81e6b489c15b247b8b0690feb23304303a9e560f37efc560a"
1588
+
1589
+
[[package]]
1590
+
name = "icu_properties"
1591
+
version = "2.1.2"
1592
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1593
+
checksum = "020bfc02fe870ec3a66d93e677ccca0562506e5872c650f893269e08615d74ec"
1594
+
dependencies = [
1595
+
"icu_collections",
1596
+
"icu_locale_core",
1597
+
"icu_properties_data",
1598
+
"icu_provider",
1599
+
"zerotrie",
1600
+
"zerovec",
1601
+
]
1602
+
1603
+
[[package]]
1604
+
name = "icu_properties_data"
1605
+
version = "2.1.2"
1606
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1607
+
checksum = "616c294cf8d725c6afcd8f55abc17c56464ef6211f9ed59cccffe534129c77af"
1608
+
1609
+
[[package]]
1610
+
name = "icu_provider"
1611
+
version = "2.1.1"
1612
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1613
+
checksum = "85962cf0ce02e1e0a629cc34e7ca3e373ce20dda4c4d7294bbd0bf1fdb59e614"
1614
+
dependencies = [
1615
+
"displaydoc",
1616
+
"icu_locale_core",
1617
+
"writeable",
1618
+
"yoke",
1619
+
"zerofrom",
1620
+
"zerotrie",
1621
+
"zerovec",
1622
+
]
1623
+
1624
+
[[package]]
1625
+
name = "id-arena"
1626
+
version = "2.3.0"
1627
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1628
+
checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954"
1629
+
1630
+
[[package]]
1631
+
name = "ident_case"
1632
+
version = "1.0.1"
1633
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1634
+
checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39"
1635
+
1636
+
[[package]]
1637
+
name = "idna"
1638
+
version = "1.1.0"
1639
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1640
+
checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de"
1641
+
dependencies = [
1642
+
"idna_adapter",
1643
+
"smallvec",
1644
+
"utf8_iter",
1645
+
]
1646
+
1647
+
[[package]]
1648
+
name = "idna_adapter"
1649
+
version = "1.2.1"
1650
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1651
+
checksum = "3acae9609540aa318d1bc588455225fb2085b9ed0c4f6bd0d9d5bcd86f1a0344"
1652
+
dependencies = [
1653
+
"icu_normalizer",
1654
+
"icu_properties",
1655
+
]
1656
+
1657
+
[[package]]
1658
+
name = "indexmap"
1659
+
version = "2.13.0"
1660
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1661
+
checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017"
1662
+
dependencies = [
1663
+
"equivalent",
1664
+
"hashbrown 0.16.1",
1665
+
"serde",
1666
+
"serde_core",
1667
+
]
1668
+
1669
+
[[package]]
1670
+
name = "indoc"
1671
+
version = "2.0.7"
1672
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1673
+
checksum = "79cf5c93f93228cf8efb3ba362535fb11199ac548a09ce117c9b1adc3030d706"
1674
+
dependencies = [
1675
+
"rustversion",
1676
+
]
1677
+
1678
+
[[package]]
1679
+
name = "instability"
1680
+
version = "0.3.11"
1681
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1682
+
checksum = "357b7205c6cd18dd2c86ed312d1e70add149aea98e7ef72b9fdf0270e555c11d"
1683
+
dependencies = [
1684
+
"darling",
1685
+
"indoc",
1686
+
"proc-macro2",
1687
+
"quote",
1688
+
"syn 2.0.117",
1689
+
]
1690
+
1691
+
[[package]]
1692
+
name = "interval-heap"
1693
+
version = "0.0.5"
1694
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1695
+
checksum = "11274e5e8e89b8607cfedc2910b6626e998779b48a019151c7604d0adcb86ac6"
1696
+
dependencies = [
1697
+
"compare",
1698
+
]
1699
+
1700
+
[[package]]
1701
+
name = "ipconfig"
1702
+
version = "0.3.2"
1703
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1704
+
checksum = "b58db92f96b720de98181bbbe63c831e87005ab460c1bf306eb2622b4707997f"
1705
+
dependencies = [
1706
+
"socket2 0.5.10",
1707
+
"widestring",
1708
+
"windows-sys 0.48.0",
1709
+
"winreg",
1710
+
]
1711
+
1712
+
[[package]]
1713
+
name = "ipnet"
1714
+
version = "2.12.0"
1715
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1716
+
checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2"
1717
+
1718
+
[[package]]
1719
+
name = "iri-string"
1720
+
version = "0.7.10"
1721
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1722
+
checksum = "c91338f0783edbd6195decb37bae672fd3b165faffb89bf7b9e6942f8b1a731a"
1723
+
dependencies = [
1724
+
"memchr",
1725
+
"serde",
1726
+
]
1727
+
1728
+
[[package]]
1729
+
name = "is_terminal_polyfill"
1730
+
version = "1.70.2"
1731
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1732
+
checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695"
1733
+
1734
+
[[package]]
1735
+
name = "itertools"
1736
+
version = "0.14.0"
1737
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1738
+
checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285"
1739
+
dependencies = [
1740
+
"either",
1741
+
]
1742
+
1743
+
[[package]]
1744
+
name = "itoa"
1745
+
version = "1.0.17"
1746
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1747
+
checksum = "92ecc6618181def0457392ccd0ee51198e065e016d1d527a7ac1b6dc7c1f09d2"
1748
+
1749
+
[[package]]
1750
+
name = "jobserver"
1751
+
version = "0.1.34"
1752
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1753
+
checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33"
1754
+
dependencies = [
1755
+
"getrandom 0.3.4",
1756
+
"libc",
1757
+
]
1758
+
1759
+
[[package]]
1760
+
name = "js-sys"
1761
+
version = "0.3.91"
1762
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1763
+
checksum = "b49715b7073f385ba4bc528e5747d02e66cb39c6146efb66b781f131f0fb399c"
1764
+
dependencies = [
1765
+
"once_cell",
1766
+
"wasm-bindgen",
1767
+
]
1768
+
1769
+
[[package]]
1770
+
name = "k256"
1771
+
version = "0.13.4"
1772
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1773
+
checksum = "f6e3919bbaa2945715f0bb6d3934a173d1e9a59ac23767fbaaef277265a7411b"
1774
+
dependencies = [
1775
+
"cfg-if",
1776
+
"ecdsa",
1777
+
"elliptic-curve",
1778
+
"once_cell",
1779
+
"sha2",
1780
+
"signature",
1781
+
]
1782
+
1783
+
[[package]]
1784
+
name = "kasuari"
1785
+
version = "0.4.11"
1786
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1787
+
checksum = "8fe90c1150662e858c7d5f945089b7517b0a80d8bf7ba4b1b5ffc984e7230a5b"
1788
+
dependencies = [
1789
+
"hashbrown 0.16.1",
1790
+
"portable-atomic",
1791
+
"thiserror 2.0.18",
1792
+
]
1793
+
1794
+
[[package]]
1795
+
name = "lab"
1796
+
version = "0.11.0"
1797
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1798
+
checksum = "bf36173d4167ed999940f804952e6b08197cae5ad5d572eb4db150ce8ad5d58f"
1799
+
1800
+
[[package]]
1801
+
name = "lazy_static"
1802
+
version = "1.5.0"
1803
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1804
+
checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
1805
+
1806
+
[[package]]
1807
+
name = "leb128fmt"
1808
+
version = "0.1.0"
1809
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1810
+
checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2"
1811
+
1812
+
[[package]]
1813
+
name = "libc"
1814
+
version = "0.2.183"
1815
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1816
+
checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d"
1817
+
1818
+
[[package]]
1819
+
name = "line-clipping"
1820
+
version = "0.3.5"
1821
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1822
+
checksum = "5f4de44e98ddbf09375cbf4d17714d18f39195f4f4894e8524501726fd9a8a4a"
1823
+
dependencies = [
1824
+
"bitflags 2.11.0",
1825
+
]
1826
+
1827
+
[[package]]
1828
+
name = "linux-raw-sys"
1829
+
version = "0.12.1"
1830
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1831
+
checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
1832
+
1833
+
[[package]]
1834
+
name = "litemap"
1835
+
version = "0.8.1"
1836
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1837
+
checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77"
1838
+
1839
+
[[package]]
1840
+
name = "litrs"
1841
+
version = "1.0.0"
1842
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1843
+
checksum = "11d3d7f243d5c5a8b9bb5d6dd2b1602c0cb0b9db1621bafc7ed66e35ff9fe092"
1844
+
1845
+
[[package]]
1846
+
name = "lock_api"
1847
+
version = "0.4.14"
1848
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1849
+
checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965"
1850
+
dependencies = [
1851
+
"scopeguard",
1852
+
]
1853
+
1854
+
[[package]]
1855
+
name = "log"
1856
+
version = "0.4.29"
1857
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1858
+
checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897"
1859
+
1860
+
[[package]]
1861
+
name = "lru"
1862
+
version = "0.12.5"
1863
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1864
+
checksum = "234cf4f4a04dc1f57e24b96cc0cd600cf2af460d4161ac5ecdd0af8e1f3b2a38"
1865
+
dependencies = [
1866
+
"hashbrown 0.15.5",
1867
+
]
1868
+
1869
+
[[package]]
1870
+
name = "lru"
1871
+
version = "0.16.3"
1872
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1873
+
checksum = "a1dc47f592c06f33f8e3aea9591776ec7c9f9e4124778ff8a3c3b87159f7e593"
1874
+
dependencies = [
1875
+
"hashbrown 0.16.1",
1876
+
]
1877
+
1878
+
[[package]]
1879
+
name = "lru-slab"
1880
+
version = "0.1.2"
1881
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1882
+
checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154"
1883
+
1884
+
[[package]]
1885
+
name = "lsm-tree"
1886
+
version = "3.1.0"
1887
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1888
+
checksum = "fc5fa40c207eed45c811085aaa1b0a25fead22e298e286081cd4b98785fe759b"
1889
+
dependencies = [
1890
+
"byteorder-lite",
1891
+
"byteview",
1892
+
"crossbeam-skiplist",
1893
+
"enum_dispatch",
1894
+
"interval-heap",
1895
+
"log",
1896
+
"lz4_flex",
1897
+
"quick_cache",
1898
+
"rustc-hash",
1899
+
"self_cell",
1900
+
"sfa",
1901
+
"tempfile",
1902
+
"varint-rs",
1903
+
"xxhash-rust",
1904
+
]
1905
+
1906
+
[[package]]
1907
+
name = "lz4_flex"
1908
+
version = "0.11.5"
1909
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1910
+
checksum = "08ab2867e3eeeca90e844d1940eab391c9dc5228783db2ed999acbc0a9ed375a"
1911
+
dependencies = [
1912
+
"twox-hash",
1913
+
]
1914
+
1915
+
[[package]]
1916
+
name = "mac_address"
1917
+
version = "1.1.8"
1918
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1919
+
checksum = "c0aeb26bf5e836cc1c341c8106051b573f1766dfa05aa87f0b98be5e51b02303"
1920
+
dependencies = [
1921
+
"nix",
1922
+
"winapi",
1923
+
]
1924
+
1925
+
[[package]]
1926
+
name = "match-lookup"
1927
+
version = "0.1.2"
1928
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1929
+
checksum = "757aee279b8bdbb9f9e676796fd459e4207a1f986e87886700abf589f5abf771"
1930
+
dependencies = [
1931
+
"proc-macro2",
1932
+
"quote",
1933
+
"syn 2.0.117",
1934
+
]
1935
+
1936
+
[[package]]
1937
+
name = "matchers"
1938
+
version = "0.2.0"
1939
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1940
+
checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9"
1941
+
dependencies = [
1942
+
"regex-automata",
1943
+
]
1944
+
1945
+
[[package]]
1946
+
name = "matchit"
1947
+
version = "0.8.4"
1948
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1949
+
checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3"
1950
+
1951
+
[[package]]
1952
+
name = "memchr"
1953
+
version = "2.8.0"
1954
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1955
+
checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79"
1956
+
1957
+
[[package]]
1958
+
name = "memmem"
1959
+
version = "0.1.1"
1960
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1961
+
checksum = "a64a92489e2744ce060c349162be1c5f33c6969234104dbd99ddb5feb08b8c15"
1962
+
1963
+
[[package]]
1964
+
name = "memoffset"
1965
+
version = "0.9.1"
1966
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1967
+
checksum = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a"
1968
+
dependencies = [
1969
+
"autocfg",
1970
+
]
1971
+
1972
+
[[package]]
1973
+
name = "mime"
1974
+
version = "0.3.17"
1975
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1976
+
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
1977
+
1978
+
[[package]]
1979
+
name = "mime_guess"
1980
+
version = "2.0.5"
1981
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1982
+
checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e"
1983
+
dependencies = [
1984
+
"mime",
1985
+
"unicase",
1986
+
]
1987
+
1988
+
[[package]]
1989
+
name = "minimal-lexical"
1990
+
version = "0.2.1"
1991
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1992
+
checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
1993
+
1994
+
[[package]]
1995
+
name = "mio"
1996
+
version = "1.1.1"
1997
+
source = "registry+https://github.com/rust-lang/crates.io-index"
1998
+
checksum = "a69bcab0ad47271a0234d9422b131806bf3968021e5dc9328caf2d4cd58557fc"
1999
+
dependencies = [
2000
+
"libc",
2001
+
"log",
2002
+
"wasi",
2003
+
"windows-sys 0.61.2",
2004
+
]
2005
+
2006
+
[[package]]
2007
+
name = "moka"
2008
+
version = "0.12.14"
2009
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2010
+
checksum = "85f8024e1c8e71c778968af91d43700ce1d11b219d127d79fb2934153b82b42b"
2011
+
dependencies = [
2012
+
"crossbeam-channel",
2013
+
"crossbeam-epoch",
2014
+
"crossbeam-utils",
2015
+
"equivalent",
2016
+
"parking_lot",
2017
+
"portable-atomic",
2018
+
"smallvec",
2019
+
"tagptr",
2020
+
"uuid",
2021
+
]
2022
+
2023
+
[[package]]
2024
+
name = "multibase"
2025
+
version = "0.9.2"
2026
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2027
+
checksum = "8694bb4835f452b0e3bb06dbebb1d6fc5385b6ca1caf2e55fd165c042390ec77"
2028
+
dependencies = [
2029
+
"base-x",
2030
+
"base256emoji",
2031
+
"data-encoding",
2032
+
"data-encoding-macro",
2033
+
]
2034
+
2035
+
[[package]]
2036
+
name = "multihash"
2037
+
version = "0.19.3"
2038
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2039
+
checksum = "6b430e7953c29dd6a09afc29ff0bb69c6e306329ee6794700aee27b76a1aea8d"
2040
+
dependencies = [
2041
+
"core2",
2042
+
"unsigned-varint",
2043
+
]
2044
+
2045
+
[[package]]
2046
+
name = "nix"
2047
+
version = "0.29.0"
2048
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2049
+
checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46"
2050
+
dependencies = [
2051
+
"bitflags 2.11.0",
2052
+
"cfg-if",
2053
+
"cfg_aliases",
2054
+
"libc",
2055
+
"memoffset",
2056
+
]
2057
+
2058
+
[[package]]
2059
+
name = "no-std-compat"
2060
+
version = "0.4.1"
2061
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2062
+
checksum = "b93853da6d84c2e3c7d730d6473e8817692dd89be387eb01b94d7f108ecb5b8c"
2063
+
2064
+
[[package]]
2065
+
name = "nom"
2066
+
version = "7.1.3"
2067
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2068
+
checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
2069
+
dependencies = [
2070
+
"memchr",
2071
+
"minimal-lexical",
2072
+
]
2073
+
2074
+
[[package]]
2075
+
name = "nonzero_ext"
2076
+
version = "0.3.0"
2077
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2078
+
checksum = "38bf9645c8b145698bb0b18a4637dcacbc421ea49bef2317e4fd8065a387cf21"
2079
+
2080
+
[[package]]
2081
+
name = "nu-ansi-term"
2082
+
version = "0.50.3"
2083
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2084
+
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
2085
+
dependencies = [
2086
+
"windows-sys 0.61.2",
2087
+
]
2088
+
2089
+
[[package]]
2090
+
name = "num-conv"
2091
+
version = "0.2.0"
2092
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2093
+
checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050"
2094
+
2095
+
[[package]]
2096
+
name = "num-derive"
2097
+
version = "0.4.2"
2098
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2099
+
checksum = "ed3955f1a9c7c0c15e092f9c887db08b1fc683305fdf6eb6684f22555355e202"
2100
+
dependencies = [
2101
+
"proc-macro2",
2102
+
"quote",
2103
+
"syn 2.0.117",
2104
+
]
2105
+
2106
+
[[package]]
2107
+
name = "num-traits"
2108
+
version = "0.2.19"
2109
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2110
+
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
2111
+
dependencies = [
2112
+
"autocfg",
2113
+
]
2114
+
2115
+
[[package]]
2116
+
name = "num_threads"
2117
+
version = "0.1.7"
2118
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2119
+
checksum = "5c7398b9c8b70908f6371f47ed36737907c87c52af34c268fed0bf0ceb92ead9"
2120
+
dependencies = [
2121
+
"libc",
2122
+
]
2123
+
2124
+
[[package]]
2125
+
name = "once_cell"
2126
+
version = "1.21.3"
2127
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2128
+
checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d"
2129
+
dependencies = [
2130
+
"critical-section",
2131
+
"portable-atomic",
2132
+
]
2133
+
2134
+
[[package]]
2135
+
name = "once_cell_polyfill"
2136
+
version = "1.70.2"
2137
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2138
+
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
2139
+
2140
+
[[package]]
2141
+
name = "openssl-probe"
2142
+
version = "0.2.1"
2143
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2144
+
checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
2145
+
2146
+
[[package]]
2147
+
name = "ordered-float"
2148
+
version = "4.6.0"
2149
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2150
+
checksum = "7bb71e1b3fa6ca1c61f383464aaf2bb0e2f8e772a1f01d486832464de363b951"
2151
+
dependencies = [
2152
+
"num-traits",
2153
+
]
2154
+
2155
+
[[package]]
2156
+
name = "p256"
2157
+
version = "0.13.2"
2158
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2159
+
checksum = "c9863ad85fa8f4460f9c48cb909d38a0d689dba1f6f6988a5e3e0d31071bcd4b"
2160
+
dependencies = [
2161
+
"ecdsa",
2162
+
"elliptic-curve",
2163
+
"primeorder",
2164
+
"serdect",
2165
+
"sha2",
2166
+
]
2167
+
2168
+
[[package]]
2169
+
name = "p384"
2170
+
version = "0.13.1"
2171
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2172
+
checksum = "fe42f1670a52a47d448f14b6a5c61dd78fce51856e68edaa38f7ae3a46b8d6b6"
2173
+
dependencies = [
2174
+
"ecdsa",
2175
+
"elliptic-curve",
2176
+
"primeorder",
2177
+
"serdect",
2178
+
"sha2",
2179
+
]
2180
+
2181
+
[[package]]
2182
+
name = "parking_lot"
2183
+
version = "0.12.5"
2184
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2185
+
checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a"
2186
+
dependencies = [
2187
+
"lock_api",
2188
+
"parking_lot_core",
2189
+
]
2190
+
2191
+
[[package]]
2192
+
name = "parking_lot_core"
2193
+
version = "0.9.12"
2194
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2195
+
checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1"
2196
+
dependencies = [
2197
+
"cfg-if",
2198
+
"libc",
2199
+
"redox_syscall",
2200
+
"smallvec",
2201
+
"windows-link",
2202
+
]
2203
+
2204
+
[[package]]
2205
+
name = "pem-rfc7468"
2206
+
version = "0.7.0"
2207
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2208
+
checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412"
2209
+
dependencies = [
2210
+
"base64ct",
2211
+
]
2212
+
2213
+
[[package]]
2214
+
name = "percent-encoding"
2215
+
version = "2.3.2"
2216
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2217
+
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
2218
+
2219
+
[[package]]
2220
+
name = "pest"
2221
+
version = "2.8.6"
2222
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2223
+
checksum = "e0848c601009d37dfa3430c4666e147e49cdcf1b92ecd3e63657d8a5f19da662"
2224
+
dependencies = [
2225
+
"memchr",
2226
+
"ucd-trie",
2227
+
]
2228
+
2229
+
[[package]]
2230
+
name = "pest_derive"
2231
+
version = "2.8.6"
2232
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2233
+
checksum = "11f486f1ea21e6c10ed15d5a7c77165d0ee443402f0780849d1768e7d9d6fe77"
2234
+
dependencies = [
2235
+
"pest",
2236
+
"pest_generator",
2237
+
]
2238
+
2239
+
[[package]]
2240
+
name = "pest_generator"
2241
+
version = "2.8.6"
2242
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2243
+
checksum = "8040c4647b13b210a963c1ed407c1ff4fdfa01c31d6d2a098218702e6664f94f"
2244
+
dependencies = [
2245
+
"pest",
2246
+
"pest_meta",
2247
+
"proc-macro2",
2248
+
"quote",
2249
+
"syn 2.0.117",
2250
+
]
2251
+
2252
+
[[package]]
2253
+
name = "pest_meta"
2254
+
version = "2.8.6"
2255
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2256
+
checksum = "89815c69d36021a140146f26659a81d6c2afa33d216d736dd4be5381a7362220"
2257
+
dependencies = [
2258
+
"pest",
2259
+
"sha2",
2260
+
]
2261
+
2262
+
[[package]]
2263
+
name = "phf"
2264
+
version = "0.11.3"
2265
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2266
+
checksum = "1fd6780a80ae0c52cc120a26a1a42c1ae51b247a253e4e06113d23d2c2edd078"
2267
+
dependencies = [
2268
+
"phf_macros",
2269
+
"phf_shared",
2270
+
]
2271
+
2272
+
[[package]]
2273
+
name = "phf_codegen"
2274
+
version = "0.11.3"
2275
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2276
+
checksum = "aef8048c789fa5e851558d709946d6d79a8ff88c0440c587967f8e94bfb1216a"
2277
+
dependencies = [
2278
+
"phf_generator",
2279
+
"phf_shared",
2280
+
]
2281
+
2282
+
[[package]]
2283
+
name = "phf_generator"
2284
+
version = "0.11.3"
2285
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2286
+
checksum = "3c80231409c20246a13fddb31776fb942c38553c51e871f8cbd687a4cfb5843d"
2287
+
dependencies = [
2288
+
"phf_shared",
2289
+
"rand 0.8.5",
2290
+
]
2291
+
2292
+
[[package]]
2293
+
name = "phf_macros"
2294
+
version = "0.11.3"
2295
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2296
+
checksum = "f84ac04429c13a7ff43785d75ad27569f2951ce0ffd30a3321230db2fc727216"
2297
+
dependencies = [
2298
+
"phf_generator",
2299
+
"phf_shared",
2300
+
"proc-macro2",
2301
+
"quote",
2302
+
"syn 2.0.117",
2303
+
]
2304
+
2305
+
[[package]]
2306
+
name = "phf_shared"
2307
+
version = "0.11.3"
2308
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2309
+
checksum = "67eabc2ef2a60eb7faa00097bd1ffdb5bd28e62bf39990626a582201b7a754e5"
2310
+
dependencies = [
2311
+
"siphasher",
2312
+
]
2313
+
2314
+
[[package]]
2315
+
name = "pin-project-lite"
2316
+
version = "0.2.17"
2317
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2318
+
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
2319
+
2320
+
[[package]]
2321
+
name = "pin-utils"
2322
+
version = "0.1.0"
2323
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2324
+
checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184"
2325
+
2326
+
[[package]]
2327
+
name = "pkcs8"
2328
+
version = "0.10.2"
2329
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2330
+
checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7"
2331
+
dependencies = [
2332
+
"der",
2333
+
"spki",
2334
+
]
2335
+
2336
+
[[package]]
2337
+
name = "pkg-config"
2338
+
version = "0.3.32"
2339
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2340
+
checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c"
2341
+
2342
+
[[package]]
2343
+
name = "portable-atomic"
2344
+
version = "1.13.1"
2345
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2346
+
checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49"
2347
+
2348
+
[[package]]
2349
+
name = "potential_utf"
2350
+
version = "0.1.4"
2351
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2352
+
checksum = "b73949432f5e2a09657003c25bca5e19a0e9c84f8058ca374f49e0ebe605af77"
2353
+
dependencies = [
2354
+
"zerovec",
2355
+
]
2356
+
2357
+
[[package]]
2358
+
name = "powerfmt"
2359
+
version = "0.2.0"
2360
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2361
+
checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"
2362
+
2363
+
[[package]]
2364
+
name = "ppv-lite86"
2365
+
version = "0.2.21"
2366
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2367
+
checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9"
2368
+
dependencies = [
2369
+
"zerocopy",
2370
+
]
2371
+
2372
+
[[package]]
2373
+
name = "prettyplease"
2374
+
version = "0.2.37"
2375
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2376
+
checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"
2377
+
dependencies = [
2378
+
"proc-macro2",
2379
+
"syn 2.0.117",
2380
+
]
2381
+
2382
+
[[package]]
2383
+
name = "primeorder"
2384
+
version = "0.13.6"
2385
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2386
+
checksum = "353e1ca18966c16d9deb1c69278edbc5f194139612772bd9537af60ac231e1e6"
2387
+
dependencies = [
2388
+
"elliptic-curve",
2389
+
"serdect",
2390
+
]
2391
+
2392
+
[[package]]
2393
+
name = "proc-macro2"
2394
+
version = "1.0.106"
2395
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2396
+
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
2397
+
dependencies = [
2398
+
"unicode-ident",
2399
+
]
2400
+
2401
+
[[package]]
2402
+
name = "prometheus-client"
2403
+
version = "0.23.1"
2404
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2405
+
checksum = "cf41c1a7c32ed72abe5082fb19505b969095c12da9f5732a4bc9878757fd087c"
2406
+
dependencies = [
2407
+
"dtoa",
2408
+
"itoa",
2409
+
"parking_lot",
2410
+
"prometheus-client-derive-encode",
2411
+
]
2412
+
2413
+
[[package]]
2414
+
name = "prometheus-client-derive-encode"
2415
+
version = "0.4.2"
2416
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2417
+
checksum = "440f724eba9f6996b75d63681b0a92b06947f1457076d503a4d2e2c8f56442b8"
2418
+
dependencies = [
2419
+
"proc-macro2",
2420
+
"quote",
2421
+
"syn 2.0.117",
2422
+
]
2423
+
2424
+
[[package]]
2425
+
name = "quanta"
2426
+
version = "0.12.6"
2427
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2428
+
checksum = "f3ab5a9d756f0d97bdc89019bd2e4ea098cf9cde50ee7564dde6b81ccc8f06c7"
2429
+
dependencies = [
2430
+
"crossbeam-utils",
2431
+
"libc",
2432
+
"once_cell",
2433
+
"raw-cpuid",
2434
+
"wasi",
2435
+
"web-sys",
2436
+
"winapi",
2437
+
]
2438
+
2439
+
[[package]]
2440
+
name = "quick_cache"
2441
+
version = "0.6.19"
2442
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2443
+
checksum = "530e84778a55de0f52645a51d4e3b9554978acd6a1e7cd50b6a6784692b3029e"
2444
+
dependencies = [
2445
+
"equivalent",
2446
+
"hashbrown 0.16.1",
2447
+
]
2448
+
2449
+
[[package]]
2450
+
name = "quinn"
2451
+
version = "0.11.9"
2452
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2453
+
checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20"
2454
+
dependencies = [
2455
+
"bytes",
2456
+
"cfg_aliases",
2457
+
"pin-project-lite",
2458
+
"quinn-proto",
2459
+
"quinn-udp",
2460
+
"rustc-hash",
2461
+
"rustls",
2462
+
"socket2 0.6.3",
2463
+
"thiserror 2.0.18",
2464
+
"tokio",
2465
+
"tracing",
2466
+
"web-time",
2467
+
]
2468
+
2469
+
[[package]]
2470
+
name = "quinn-proto"
2471
+
version = "0.11.14"
2472
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2473
+
checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098"
2474
+
dependencies = [
2475
+
"bytes",
2476
+
"getrandom 0.3.4",
2477
+
"lru-slab",
2478
+
"rand 0.9.2",
2479
+
"ring",
2480
+
"rustc-hash",
2481
+
"rustls",
2482
+
"rustls-pki-types",
2483
+
"slab",
2484
+
"thiserror 2.0.18",
2485
+
"tinyvec",
2486
+
"tracing",
2487
+
"web-time",
2488
+
]
2489
+
2490
+
[[package]]
2491
+
name = "quinn-udp"
2492
+
version = "0.5.14"
2493
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2494
+
checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
2495
+
dependencies = [
2496
+
"cfg_aliases",
2497
+
"libc",
2498
+
"once_cell",
2499
+
"socket2 0.6.3",
2500
+
"tracing",
2501
+
"windows-sys 0.60.2",
2502
+
]
2503
+
2504
+
[[package]]
2505
+
name = "quote"
2506
+
version = "1.0.45"
2507
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2508
+
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
2509
+
dependencies = [
2510
+
"proc-macro2",
2511
+
]
2512
+
2513
+
[[package]]
2514
+
name = "r-efi"
2515
+
version = "5.3.0"
2516
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2517
+
checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f"
2518
+
2519
+
[[package]]
2520
+
name = "r-efi"
2521
+
version = "6.0.0"
2522
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2523
+
checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
2524
+
2525
+
[[package]]
2526
+
name = "ramjet"
2527
+
version = "0.1.0"
2528
+
dependencies = [
2529
+
"anyhow",
2530
+
"atproto-dasl",
2531
+
"atproto-identity",
2532
+
"atproto-record",
2533
+
"atproto-repo",
2534
+
"atproto-xrpcs",
2535
+
"axum",
2536
+
"base64",
2537
+
"bytes",
2538
+
"cid",
2539
+
"clap",
2540
+
"crossterm",
2541
+
"fjall",
2542
+
"futures-util",
2543
+
"governor",
2544
+
"http",
2545
+
"multihash",
2546
+
"prometheus-client",
2547
+
"rand 0.10.0",
2548
+
"ratatui",
2549
+
"reqwest",
2550
+
"riblt",
2551
+
"serde",
2552
+
"serde_json",
2553
+
"sha2",
2554
+
"thiserror 2.0.18",
2555
+
"tokio",
2556
+
"tokio-metrics",
2557
+
"tokio-util",
2558
+
"tokio-websockets",
2559
+
"tracing",
2560
+
"tracing-subscriber",
2561
+
"zstd",
2562
+
]
2563
+
2564
+
[[package]]
2565
+
name = "rand"
2566
+
version = "0.8.5"
2567
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2568
+
checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404"
2569
+
dependencies = [
2570
+
"libc",
2571
+
"rand_chacha 0.3.1",
2572
+
"rand_core 0.6.4",
2573
+
]
2574
+
2575
+
[[package]]
2576
+
name = "rand"
2577
+
version = "0.9.2"
2578
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2579
+
checksum = "6db2770f06117d490610c7488547d543617b21bfa07796d7a12f6f1bd53850d1"
2580
+
dependencies = [
2581
+
"rand_chacha 0.9.0",
2582
+
"rand_core 0.9.5",
2583
+
]
2584
+
2585
+
[[package]]
2586
+
name = "rand"
2587
+
version = "0.10.0"
2588
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2589
+
checksum = "bc266eb313df6c5c09c1c7b1fbe2510961e5bcd3add930c1e31f7ed9da0feff8"
2590
+
dependencies = [
2591
+
"chacha20",
2592
+
"getrandom 0.4.2",
2593
+
"rand_core 0.10.0",
2594
+
]
2595
+
2596
+
[[package]]
2597
+
name = "rand_chacha"
2598
+
version = "0.3.1"
2599
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2600
+
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
2601
+
dependencies = [
2602
+
"ppv-lite86",
2603
+
"rand_core 0.6.4",
2604
+
]
2605
+
2606
+
[[package]]
2607
+
name = "rand_chacha"
2608
+
version = "0.9.0"
2609
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2610
+
checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
2611
+
dependencies = [
2612
+
"ppv-lite86",
2613
+
"rand_core 0.9.5",
2614
+
]
2615
+
2616
+
[[package]]
2617
+
name = "rand_core"
2618
+
version = "0.6.4"
2619
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2620
+
checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
2621
+
dependencies = [
2622
+
"getrandom 0.2.17",
2623
+
]
2624
+
2625
+
[[package]]
2626
+
name = "rand_core"
2627
+
version = "0.9.5"
2628
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2629
+
checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
2630
+
dependencies = [
2631
+
"getrandom 0.3.4",
2632
+
]
2633
+
2634
+
[[package]]
2635
+
name = "rand_core"
2636
+
version = "0.10.0"
2637
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2638
+
checksum = "0c8d0fd677905edcbeedbf2edb6494d676f0e98d54d5cf9bda0b061cb8fb8aba"
2639
+
2640
+
[[package]]
2641
+
name = "ratatui"
2642
+
version = "0.30.0"
2643
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2644
+
checksum = "d1ce67fb8ba4446454d1c8dbaeda0557ff5e94d39d5e5ed7f10a65eb4c8266bc"
2645
+
dependencies = [
2646
+
"instability",
2647
+
"ratatui-core",
2648
+
"ratatui-crossterm",
2649
+
"ratatui-macros",
2650
+
"ratatui-termwiz",
2651
+
"ratatui-widgets",
2652
+
]
2653
+
2654
+
[[package]]
2655
+
name = "ratatui-core"
2656
+
version = "0.1.0"
2657
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2658
+
checksum = "5ef8dea09a92caaf73bff7adb70b76162e5937524058a7e5bff37869cbbec293"
2659
+
dependencies = [
2660
+
"bitflags 2.11.0",
2661
+
"compact_str",
2662
+
"hashbrown 0.16.1",
2663
+
"indoc",
2664
+
"itertools",
2665
+
"kasuari",
2666
+
"lru 0.16.3",
2667
+
"strum",
2668
+
"thiserror 2.0.18",
2669
+
"unicode-segmentation",
2670
+
"unicode-truncate",
2671
+
"unicode-width",
2672
+
]
2673
+
2674
+
[[package]]
2675
+
name = "ratatui-crossterm"
2676
+
version = "0.1.0"
2677
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2678
+
checksum = "577c9b9f652b4c121fb25c6a391dd06406d3b092ba68827e6d2f09550edc54b3"
2679
+
dependencies = [
2680
+
"cfg-if",
2681
+
"crossterm",
2682
+
"instability",
2683
+
"ratatui-core",
2684
+
]
2685
+
2686
+
[[package]]
2687
+
name = "ratatui-macros"
2688
+
version = "0.7.0"
2689
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2690
+
checksum = "a7f1342a13e83e4bb9d0b793d0ea762be633f9582048c892ae9041ef39c936f4"
2691
+
dependencies = [
2692
+
"ratatui-core",
2693
+
"ratatui-widgets",
2694
+
]
2695
+
2696
+
[[package]]
2697
+
name = "ratatui-termwiz"
2698
+
version = "0.1.0"
2699
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2700
+
checksum = "0f76fe0bd0ed4295f0321b1676732e2454024c15a35d01904ddb315afd3d545c"
2701
+
dependencies = [
2702
+
"ratatui-core",
2703
+
"termwiz",
2704
+
]
2705
+
2706
+
[[package]]
2707
+
name = "ratatui-widgets"
2708
+
version = "0.3.0"
2709
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2710
+
checksum = "d7dbfa023cd4e604c2553483820c5fe8aa9d71a42eea5aa77c6e7f35756612db"
2711
+
dependencies = [
2712
+
"bitflags 2.11.0",
2713
+
"hashbrown 0.16.1",
2714
+
"indoc",
2715
+
"instability",
2716
+
"itertools",
2717
+
"line-clipping",
2718
+
"ratatui-core",
2719
+
"strum",
2720
+
"time",
2721
+
"unicode-segmentation",
2722
+
"unicode-width",
2723
+
]
2724
+
2725
+
[[package]]
2726
+
name = "raw-cpuid"
2727
+
version = "11.6.0"
2728
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2729
+
checksum = "498cd0dc59d73224351ee52a95fee0f1a617a2eae0e7d9d720cc622c73a54186"
2730
+
dependencies = [
2731
+
"bitflags 2.11.0",
2732
+
]
2733
+
2734
+
[[package]]
2735
+
name = "redox_syscall"
2736
+
version = "0.5.18"
2737
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2738
+
checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d"
2739
+
dependencies = [
2740
+
"bitflags 2.11.0",
2741
+
]
2742
+
2743
+
[[package]]
2744
+
name = "regex"
2745
+
version = "1.12.3"
2746
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2747
+
checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276"
2748
+
dependencies = [
2749
+
"aho-corasick",
2750
+
"memchr",
2751
+
"regex-automata",
2752
+
"regex-syntax",
2753
+
]
2754
+
2755
+
[[package]]
2756
+
name = "regex-automata"
2757
+
version = "0.4.14"
2758
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2759
+
checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f"
2760
+
dependencies = [
2761
+
"aho-corasick",
2762
+
"memchr",
2763
+
"regex-syntax",
2764
+
]
2765
+
2766
+
[[package]]
2767
+
name = "regex-syntax"
2768
+
version = "0.8.10"
2769
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2770
+
checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a"
2771
+
2772
+
[[package]]
2773
+
name = "reqwest"
2774
+
version = "0.12.28"
2775
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2776
+
checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147"
2777
+
dependencies = [
2778
+
"base64",
2779
+
"bytes",
2780
+
"encoding_rs",
2781
+
"futures-channel",
2782
+
"futures-core",
2783
+
"futures-util",
2784
+
"h2",
2785
+
"http",
2786
+
"http-body",
2787
+
"http-body-util",
2788
+
"hyper",
2789
+
"hyper-rustls",
2790
+
"hyper-util",
2791
+
"js-sys",
2792
+
"log",
2793
+
"mime",
2794
+
"mime_guess",
2795
+
"percent-encoding",
2796
+
"pin-project-lite",
2797
+
"quinn",
2798
+
"rustls",
2799
+
"rustls-pki-types",
2800
+
"serde",
2801
+
"serde_json",
2802
+
"serde_urlencoded",
2803
+
"sync_wrapper",
2804
+
"tokio",
2805
+
"tokio-rustls",
2806
+
"tower",
2807
+
"tower-http",
2808
+
"tower-service",
2809
+
"url",
2810
+
"wasm-bindgen",
2811
+
"wasm-bindgen-futures",
2812
+
"web-sys",
2813
+
"webpki-roots",
2814
+
]
2815
+
2816
+
[[package]]
2817
+
name = "reqwest-chain"
2818
+
version = "1.0.0"
2819
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2820
+
checksum = "da5c014fb79a8227db44a0433d748107750d2550b7fca55c59a3d7ee7d2ee2b2"
2821
+
dependencies = [
2822
+
"anyhow",
2823
+
"async-trait",
2824
+
"http",
2825
+
"reqwest-middleware",
2826
+
]
2827
+
2828
+
[[package]]
2829
+
name = "reqwest-middleware"
2830
+
version = "0.4.2"
2831
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2832
+
checksum = "57f17d28a6e6acfe1733fe24bcd30774d13bffa4b8a22535b4c8c98423088d4e"
2833
+
dependencies = [
2834
+
"anyhow",
2835
+
"async-trait",
2836
+
"http",
2837
+
"reqwest",
2838
+
"serde",
2839
+
"thiserror 1.0.69",
2840
+
"tower-service",
2841
+
]
2842
+
2843
+
[[package]]
2844
+
name = "resolv-conf"
2845
+
version = "0.7.6"
2846
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2847
+
checksum = "1e061d1b48cb8d38042de4ae0a7a6401009d6143dc80d2e2d6f31f0bdd6470c7"
2848
+
2849
+
[[package]]
2850
+
name = "rfc6979"
2851
+
version = "0.4.0"
2852
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2853
+
checksum = "f8dd2a808d456c4a54e300a23e9f5a67e122c3024119acbfd73e3bf664491cb2"
2854
+
dependencies = [
2855
+
"hmac",
2856
+
"subtle",
2857
+
]
2858
+
2859
+
[[package]]
2860
+
name = "riblt"
2861
+
version = "0.1.0"
2862
+
source = "git+https://github.com/ngerakines/riblt-rs#1bed49393771f3fc488a49a8b7b4a952273cdd93"
2863
+
2864
+
[[package]]
2865
+
name = "ring"
2866
+
version = "0.17.14"
2867
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2868
+
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
2869
+
dependencies = [
2870
+
"cc",
2871
+
"cfg-if",
2872
+
"getrandom 0.2.17",
2873
+
"libc",
2874
+
"untrusted",
2875
+
"windows-sys 0.52.0",
2876
+
]
2877
+
2878
+
[[package]]
2879
+
name = "rustc-hash"
2880
+
version = "2.1.1"
2881
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2882
+
checksum = "357703d41365b4b27c590e3ed91eabb1b663f07c4c084095e60cbed4362dff0d"
2883
+
2884
+
[[package]]
2885
+
name = "rustc_version"
2886
+
version = "0.4.1"
2887
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2888
+
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
2889
+
dependencies = [
2890
+
"semver",
2891
+
]
2892
+
2893
+
[[package]]
2894
+
name = "rustix"
2895
+
version = "1.1.4"
2896
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2897
+
checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
2898
+
dependencies = [
2899
+
"bitflags 2.11.0",
2900
+
"errno",
2901
+
"libc",
2902
+
"linux-raw-sys",
2903
+
"windows-sys 0.61.2",
2904
+
]
2905
+
2906
+
[[package]]
2907
+
name = "rustls"
2908
+
version = "0.23.37"
2909
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2910
+
checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4"
2911
+
dependencies = [
2912
+
"once_cell",
2913
+
"ring",
2914
+
"rustls-pki-types",
2915
+
"rustls-webpki",
2916
+
"subtle",
2917
+
"zeroize",
2918
+
]
2919
+
2920
+
[[package]]
2921
+
name = "rustls-native-certs"
2922
+
version = "0.8.3"
2923
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2924
+
checksum = "612460d5f7bea540c490b2b6395d8e34a953e52b491accd6c86c8164c5932a63"
2925
+
dependencies = [
2926
+
"openssl-probe",
2927
+
"rustls-pki-types",
2928
+
"schannel",
2929
+
"security-framework",
2930
+
]
2931
+
2932
+
[[package]]
2933
+
name = "rustls-pki-types"
2934
+
version = "1.14.0"
2935
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2936
+
checksum = "be040f8b0a225e40375822a563fa9524378b9d63112f53e19ffff34df5d33fdd"
2937
+
dependencies = [
2938
+
"web-time",
2939
+
"zeroize",
2940
+
]
2941
+
2942
+
[[package]]
2943
+
name = "rustls-webpki"
2944
+
version = "0.103.9"
2945
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2946
+
checksum = "d7df23109aa6c1567d1c575b9952556388da57401e4ace1d15f79eedad0d8f53"
2947
+
dependencies = [
2948
+
"ring",
2949
+
"rustls-pki-types",
2950
+
"untrusted",
2951
+
]
2952
+
2953
+
[[package]]
2954
+
name = "rustversion"
2955
+
version = "1.0.22"
2956
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2957
+
checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
2958
+
2959
+
[[package]]
2960
+
name = "ryu"
2961
+
version = "1.0.23"
2962
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2963
+
checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
2964
+
2965
+
[[package]]
2966
+
name = "schannel"
2967
+
version = "0.1.29"
2968
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2969
+
checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939"
2970
+
dependencies = [
2971
+
"windows-sys 0.61.2",
2972
+
]
2973
+
2974
+
[[package]]
2975
+
name = "scopeguard"
2976
+
version = "1.2.0"
2977
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2978
+
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
2979
+
2980
+
[[package]]
2981
+
name = "sec1"
2982
+
version = "0.7.3"
2983
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2984
+
checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc"
2985
+
dependencies = [
2986
+
"base16ct",
2987
+
"der",
2988
+
"generic-array",
2989
+
"pkcs8",
2990
+
"serdect",
2991
+
"subtle",
2992
+
"zeroize",
2993
+
]
2994
+
2995
+
[[package]]
2996
+
name = "security-framework"
2997
+
version = "3.7.0"
2998
+
source = "registry+https://github.com/rust-lang/crates.io-index"
2999
+
checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d"
3000
+
dependencies = [
3001
+
"bitflags 2.11.0",
3002
+
"core-foundation 0.10.1",
3003
+
"core-foundation-sys",
3004
+
"libc",
3005
+
"security-framework-sys",
3006
+
]
3007
+
3008
+
[[package]]
3009
+
name = "security-framework-sys"
3010
+
version = "2.17.0"
3011
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3012
+
checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3"
3013
+
dependencies = [
3014
+
"core-foundation-sys",
3015
+
"libc",
3016
+
]
3017
+
3018
+
[[package]]
3019
+
name = "self_cell"
3020
+
version = "1.2.2"
3021
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3022
+
checksum = "b12e76d157a900eb52e81bc6e9f3069344290341720e9178cde2407113ac8d89"
3023
+
3024
+
[[package]]
3025
+
name = "semver"
3026
+
version = "1.0.27"
3027
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3028
+
checksum = "d767eb0aabc880b29956c35734170f26ed551a859dbd361d140cdbeca61ab1e2"
3029
+
3030
+
[[package]]
3031
+
name = "serde"
3032
+
version = "1.0.228"
3033
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3034
+
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
3035
+
dependencies = [
3036
+
"serde_core",
3037
+
"serde_derive",
3038
+
]
3039
+
3040
+
[[package]]
3041
+
name = "serde_bytes"
3042
+
version = "0.11.19"
3043
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3044
+
checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8"
3045
+
dependencies = [
3046
+
"serde",
3047
+
"serde_core",
3048
+
]
3049
+
3050
+
[[package]]
3051
+
name = "serde_core"
3052
+
version = "1.0.228"
3053
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3054
+
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
3055
+
dependencies = [
3056
+
"serde_derive",
3057
+
]
3058
+
3059
+
[[package]]
3060
+
name = "serde_derive"
3061
+
version = "1.0.228"
3062
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3063
+
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
3064
+
dependencies = [
3065
+
"proc-macro2",
3066
+
"quote",
3067
+
"syn 2.0.117",
3068
+
]
3069
+
3070
+
[[package]]
3071
+
name = "serde_json"
3072
+
version = "1.0.149"
3073
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3074
+
checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86"
3075
+
dependencies = [
3076
+
"itoa",
3077
+
"memchr",
3078
+
"serde",
3079
+
"serde_core",
3080
+
"zmij",
3081
+
]
3082
+
3083
+
[[package]]
3084
+
name = "serde_path_to_error"
3085
+
version = "0.1.20"
3086
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3087
+
checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457"
3088
+
dependencies = [
3089
+
"itoa",
3090
+
"serde",
3091
+
"serde_core",
3092
+
]
3093
+
3094
+
[[package]]
3095
+
name = "serde_urlencoded"
3096
+
version = "0.7.1"
3097
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3098
+
checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd"
3099
+
dependencies = [
3100
+
"form_urlencoded",
3101
+
"itoa",
3102
+
"ryu",
3103
+
"serde",
3104
+
]
3105
+
3106
+
[[package]]
3107
+
name = "serdect"
3108
+
version = "0.2.0"
3109
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3110
+
checksum = "a84f14a19e9a014bb9f4512488d9829a68e04ecabffb0f9904cd1ace94598177"
3111
+
dependencies = [
3112
+
"base16ct",
3113
+
"serde",
3114
+
]
3115
+
3116
+
[[package]]
3117
+
name = "sfa"
3118
+
version = "1.0.0"
3119
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3120
+
checksum = "a1296838937cab56cd6c4eeeb8718ec777383700c33f060e2869867bd01d1175"
3121
+
dependencies = [
3122
+
"byteorder-lite",
3123
+
"log",
3124
+
"xxhash-rust",
3125
+
]
3126
+
3127
+
[[package]]
3128
+
name = "sha1"
3129
+
version = "0.10.6"
3130
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3131
+
checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba"
3132
+
dependencies = [
3133
+
"cfg-if",
3134
+
"cpufeatures 0.2.17",
3135
+
"digest",
3136
+
]
3137
+
3138
+
[[package]]
3139
+
name = "sha2"
3140
+
version = "0.10.9"
3141
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3142
+
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
3143
+
dependencies = [
3144
+
"cfg-if",
3145
+
"cpufeatures 0.2.17",
3146
+
"digest",
3147
+
]
3148
+
3149
+
[[package]]
3150
+
name = "sharded-slab"
3151
+
version = "0.1.7"
3152
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3153
+
checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6"
3154
+
dependencies = [
3155
+
"lazy_static",
3156
+
]
3157
+
3158
+
[[package]]
3159
+
name = "shlex"
3160
+
version = "1.3.0"
3161
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3162
+
checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
3163
+
3164
+
[[package]]
3165
+
name = "signal-hook"
3166
+
version = "0.3.18"
3167
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3168
+
checksum = "d881a16cf4426aa584979d30bd82cb33429027e42122b169753d6ef1085ed6e2"
3169
+
dependencies = [
3170
+
"libc",
3171
+
"signal-hook-registry",
3172
+
]
3173
+
3174
+
[[package]]
3175
+
name = "signal-hook-mio"
3176
+
version = "0.2.5"
3177
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3178
+
checksum = "b75a19a7a740b25bc7944bdee6172368f988763b744e3d4dfe753f6b4ece40cc"
3179
+
dependencies = [
3180
+
"libc",
3181
+
"mio",
3182
+
"signal-hook",
3183
+
]
3184
+
3185
+
[[package]]
3186
+
name = "signal-hook-registry"
3187
+
version = "1.4.8"
3188
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3189
+
checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b"
3190
+
dependencies = [
3191
+
"errno",
3192
+
"libc",
3193
+
]
3194
+
3195
+
[[package]]
3196
+
name = "signature"
3197
+
version = "2.2.0"
3198
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3199
+
checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de"
3200
+
dependencies = [
3201
+
"digest",
3202
+
"rand_core 0.6.4",
3203
+
]
3204
+
3205
+
[[package]]
3206
+
name = "simdutf8"
3207
+
version = "0.1.5"
3208
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3209
+
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
3210
+
3211
+
[[package]]
3212
+
name = "siphasher"
3213
+
version = "1.0.2"
3214
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3215
+
checksum = "b2aa850e253778c88a04c3d7323b043aeda9d3e30d5971937c1855769763678e"
3216
+
3217
+
[[package]]
3218
+
name = "slab"
3219
+
version = "0.4.12"
3220
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3221
+
checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
3222
+
3223
+
[[package]]
3224
+
name = "smallvec"
3225
+
version = "1.15.1"
3226
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3227
+
checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03"
3228
+
3229
+
[[package]]
3230
+
name = "socket2"
3231
+
version = "0.5.10"
3232
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3233
+
checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678"
3234
+
dependencies = [
3235
+
"libc",
3236
+
"windows-sys 0.52.0",
3237
+
]
3238
+
3239
+
[[package]]
3240
+
name = "socket2"
3241
+
version = "0.6.3"
3242
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3243
+
checksum = "3a766e1110788c36f4fa1c2b71b387a7815aa65f88ce0229841826633d93723e"
3244
+
dependencies = [
3245
+
"libc",
3246
+
"windows-sys 0.61.2",
3247
+
]
3248
+
3249
+
[[package]]
3250
+
name = "spin"
3251
+
version = "0.9.8"
3252
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3253
+
checksum = "6980e8d7511241f8acf4aebddbb1ff938df5eebe98691418c4468d0b72a96a67"
3254
+
dependencies = [
3255
+
"lock_api",
3256
+
]
3257
+
3258
+
[[package]]
3259
+
name = "spinning_top"
3260
+
version = "0.3.0"
3261
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3262
+
checksum = "d96d2d1d716fb500937168cc09353ffdc7a012be8475ac7308e1bdf0e3923300"
3263
+
dependencies = [
3264
+
"lock_api",
3265
+
]
3266
+
3267
+
[[package]]
3268
+
name = "spki"
3269
+
version = "0.7.3"
3270
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3271
+
checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d"
3272
+
dependencies = [
3273
+
"base64ct",
3274
+
"der",
3275
+
]
3276
+
3277
+
[[package]]
3278
+
name = "stable_deref_trait"
3279
+
version = "1.2.1"
3280
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3281
+
checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596"
3282
+
3283
+
[[package]]
3284
+
name = "static_assertions"
3285
+
version = "1.1.0"
3286
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3287
+
checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f"
3288
+
3289
+
[[package]]
3290
+
name = "strsim"
3291
+
version = "0.11.1"
3292
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3293
+
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
3294
+
3295
+
[[package]]
3296
+
name = "strum"
3297
+
version = "0.27.2"
3298
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3299
+
checksum = "af23d6f6c1a224baef9d3f61e287d2761385a5b88fdab4eb4c6f11aeb54c4bcf"
3300
+
dependencies = [
3301
+
"strum_macros",
3302
+
]
3303
+
3304
+
[[package]]
3305
+
name = "strum_macros"
3306
+
version = "0.27.2"
3307
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3308
+
checksum = "7695ce3845ea4b33927c055a39dc438a45b059f7c1b3d91d38d10355fb8cbca7"
3309
+
dependencies = [
3310
+
"heck",
3311
+
"proc-macro2",
3312
+
"quote",
3313
+
"syn 2.0.117",
3314
+
]
3315
+
3316
+
[[package]]
3317
+
name = "subtle"
3318
+
version = "2.6.1"
3319
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3320
+
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
3321
+
3322
+
[[package]]
3323
+
name = "syn"
3324
+
version = "1.0.109"
3325
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3326
+
checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237"
3327
+
dependencies = [
3328
+
"proc-macro2",
3329
+
"quote",
3330
+
"unicode-ident",
3331
+
]
3332
+
3333
+
[[package]]
3334
+
name = "syn"
3335
+
version = "2.0.117"
3336
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3337
+
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
3338
+
dependencies = [
3339
+
"proc-macro2",
3340
+
"quote",
3341
+
"unicode-ident",
3342
+
]
3343
+
3344
+
[[package]]
3345
+
name = "sync_wrapper"
3346
+
version = "1.0.2"
3347
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3348
+
checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
3349
+
dependencies = [
3350
+
"futures-core",
3351
+
]
3352
+
3353
+
[[package]]
3354
+
name = "synstructure"
3355
+
version = "0.13.2"
3356
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3357
+
checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2"
3358
+
dependencies = [
3359
+
"proc-macro2",
3360
+
"quote",
3361
+
"syn 2.0.117",
3362
+
]
3363
+
3364
+
[[package]]
3365
+
name = "system-configuration"
3366
+
version = "0.7.0"
3367
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3368
+
checksum = "a13f3d0daba03132c0aa9767f98351b3488edc2c100cda2d2ec2b04f3d8d3c8b"
3369
+
dependencies = [
3370
+
"bitflags 2.11.0",
3371
+
"core-foundation 0.9.4",
3372
+
"system-configuration-sys",
3373
+
]
3374
+
3375
+
[[package]]
3376
+
name = "system-configuration-sys"
3377
+
version = "0.6.0"
3378
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3379
+
checksum = "8e1d1b10ced5ca923a1fcb8d03e96b8d3268065d724548c0211415ff6ac6bac4"
3380
+
dependencies = [
3381
+
"core-foundation-sys",
3382
+
"libc",
3383
+
]
3384
+
3385
+
[[package]]
3386
+
name = "tagptr"
3387
+
version = "0.2.0"
3388
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3389
+
checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417"
3390
+
3391
+
[[package]]
3392
+
name = "tempfile"
3393
+
version = "3.27.0"
3394
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3395
+
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
3396
+
dependencies = [
3397
+
"fastrand",
3398
+
"getrandom 0.4.2",
3399
+
"once_cell",
3400
+
"rustix",
3401
+
"windows-sys 0.61.2",
3402
+
]
3403
+
3404
+
[[package]]
3405
+
name = "terminfo"
3406
+
version = "0.9.0"
3407
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3408
+
checksum = "d4ea810f0692f9f51b382fff5893887bb4580f5fa246fde546e0b13e7fcee662"
3409
+
dependencies = [
3410
+
"fnv",
3411
+
"nom",
3412
+
"phf",
3413
+
"phf_codegen",
3414
+
]
3415
+
3416
+
[[package]]
3417
+
name = "termios"
3418
+
version = "0.3.3"
3419
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3420
+
checksum = "411c5bf740737c7918b8b1fe232dca4dc9f8e754b8ad5e20966814001ed0ac6b"
3421
+
dependencies = [
3422
+
"libc",
3423
+
]
3424
+
3425
+
[[package]]
3426
+
name = "termwiz"
3427
+
version = "0.23.3"
3428
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3429
+
checksum = "4676b37242ccbd1aabf56edb093a4827dc49086c0ffd764a5705899e0f35f8f7"
3430
+
dependencies = [
3431
+
"anyhow",
3432
+
"base64",
3433
+
"bitflags 2.11.0",
3434
+
"fancy-regex",
3435
+
"filedescriptor",
3436
+
"finl_unicode",
3437
+
"fixedbitset",
3438
+
"hex",
3439
+
"lazy_static",
3440
+
"libc",
3441
+
"log",
3442
+
"memmem",
3443
+
"nix",
3444
+
"num-derive",
3445
+
"num-traits",
3446
+
"ordered-float",
3447
+
"pest",
3448
+
"pest_derive",
3449
+
"phf",
3450
+
"sha2",
3451
+
"signal-hook",
3452
+
"siphasher",
3453
+
"terminfo",
3454
+
"termios",
3455
+
"thiserror 1.0.69",
3456
+
"ucd-trie",
3457
+
"unicode-segmentation",
3458
+
"vtparse",
3459
+
"wezterm-bidi",
3460
+
"wezterm-blob-leases",
3461
+
"wezterm-color-types",
3462
+
"wezterm-dynamic",
3463
+
"wezterm-input-types",
3464
+
"winapi",
3465
+
]
3466
+
3467
+
[[package]]
3468
+
name = "thiserror"
3469
+
version = "1.0.69"
3470
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3471
+
checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52"
3472
+
dependencies = [
3473
+
"thiserror-impl 1.0.69",
3474
+
]
3475
+
3476
+
[[package]]
3477
+
name = "thiserror"
3478
+
version = "2.0.18"
3479
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3480
+
checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4"
3481
+
dependencies = [
3482
+
"thiserror-impl 2.0.18",
3483
+
]
3484
+
3485
+
[[package]]
3486
+
name = "thiserror-impl"
3487
+
version = "1.0.69"
3488
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3489
+
checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1"
3490
+
dependencies = [
3491
+
"proc-macro2",
3492
+
"quote",
3493
+
"syn 2.0.117",
3494
+
]
3495
+
3496
+
[[package]]
3497
+
name = "thiserror-impl"
3498
+
version = "2.0.18"
3499
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3500
+
checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5"
3501
+
dependencies = [
3502
+
"proc-macro2",
3503
+
"quote",
3504
+
"syn 2.0.117",
3505
+
]
3506
+
3507
+
[[package]]
3508
+
name = "thread_local"
3509
+
version = "1.1.9"
3510
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3511
+
checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185"
3512
+
dependencies = [
3513
+
"cfg-if",
3514
+
]
3515
+
3516
+
[[package]]
3517
+
name = "time"
3518
+
version = "0.3.47"
3519
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3520
+
checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c"
3521
+
dependencies = [
3522
+
"deranged",
3523
+
"libc",
3524
+
"num-conv",
3525
+
"num_threads",
3526
+
"powerfmt",
3527
+
"serde_core",
3528
+
"time-core",
3529
+
]
3530
+
3531
+
[[package]]
3532
+
name = "time-core"
3533
+
version = "0.1.8"
3534
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3535
+
checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca"
3536
+
3537
+
[[package]]
3538
+
name = "tinystr"
3539
+
version = "0.8.2"
3540
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3541
+
checksum = "42d3e9c45c09de15d06dd8acf5f4e0e399e85927b7f00711024eb7ae10fa4869"
3542
+
dependencies = [
3543
+
"displaydoc",
3544
+
"zerovec",
3545
+
]
3546
+
3547
+
[[package]]
3548
+
name = "tinyvec"
3549
+
version = "1.10.0"
3550
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3551
+
checksum = "bfa5fdc3bce6191a1dbc8c02d5c8bffcf557bafa17c124c5264a458f1b0613fa"
3552
+
dependencies = [
3553
+
"tinyvec_macros",
3554
+
]
3555
+
3556
+
[[package]]
3557
+
name = "tinyvec_macros"
3558
+
version = "0.1.1"
3559
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3560
+
checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20"
3561
+
3562
+
[[package]]
3563
+
name = "tokio"
3564
+
version = "1.50.0"
3565
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3566
+
checksum = "27ad5e34374e03cfffefc301becb44e9dc3c17584f414349ebe29ed26661822d"
3567
+
dependencies = [
3568
+
"bytes",
3569
+
"libc",
3570
+
"mio",
3571
+
"pin-project-lite",
3572
+
"signal-hook-registry",
3573
+
"socket2 0.6.3",
3574
+
"tokio-macros",
3575
+
"windows-sys 0.61.2",
3576
+
]
3577
+
3578
+
[[package]]
3579
+
name = "tokio-macros"
3580
+
version = "2.6.1"
3581
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3582
+
checksum = "5c55a2eff8b69ce66c84f85e1da1c233edc36ceb85a2058d11b0d6a3c7e7569c"
3583
+
dependencies = [
3584
+
"proc-macro2",
3585
+
"quote",
3586
+
"syn 2.0.117",
3587
+
]
3588
+
3589
+
[[package]]
3590
+
name = "tokio-metrics"
3591
+
version = "0.4.9"
3592
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3593
+
checksum = "0e0410015c6db7b67b9c9ab2a3af4d74a942d637ff248d0d055073750deac6f9"
3594
+
dependencies = [
3595
+
"futures-util",
3596
+
"pin-project-lite",
3597
+
"tokio",
3598
+
"tokio-stream",
3599
+
]
3600
+
3601
+
[[package]]
3602
+
name = "tokio-rustls"
3603
+
version = "0.26.4"
3604
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3605
+
checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61"
3606
+
dependencies = [
3607
+
"rustls",
3608
+
"tokio",
3609
+
]
3610
+
3611
+
[[package]]
3612
+
name = "tokio-stream"
3613
+
version = "0.1.18"
3614
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3615
+
checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70"
3616
+
dependencies = [
3617
+
"futures-core",
3618
+
"pin-project-lite",
3619
+
"tokio",
3620
+
]
3621
+
3622
+
[[package]]
3623
+
name = "tokio-tungstenite"
3624
+
version = "0.28.0"
3625
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3626
+
checksum = "d25a406cddcc431a75d3d9afc6a7c0f7428d4891dd973e4d54c56b46127bf857"
3627
+
dependencies = [
3628
+
"futures-util",
3629
+
"log",
3630
+
"tokio",
3631
+
"tungstenite",
3632
+
]
3633
+
3634
+
[[package]]
3635
+
name = "tokio-util"
3636
+
version = "0.7.18"
3637
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3638
+
checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098"
3639
+
dependencies = [
3640
+
"bytes",
3641
+
"futures-core",
3642
+
"futures-sink",
3643
+
"pin-project-lite",
3644
+
"tokio",
3645
+
]
3646
+
3647
+
[[package]]
3648
+
name = "tokio-websockets"
3649
+
version = "0.13.1"
3650
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3651
+
checksum = "8b6aa6c8b5a31e06fd3760eb5c1b8d9072e30731f0467ee3795617fe768e7449"
3652
+
dependencies = [
3653
+
"base64",
3654
+
"bytes",
3655
+
"fastrand",
3656
+
"futures-core",
3657
+
"futures-sink",
3658
+
"http",
3659
+
"httparse",
3660
+
"ring",
3661
+
"rustls-native-certs",
3662
+
"rustls-pki-types",
3663
+
"simdutf8",
3664
+
"tokio",
3665
+
"tokio-rustls",
3666
+
"tokio-util",
3667
+
]
3668
+
3669
+
[[package]]
3670
+
name = "tower"
3671
+
version = "0.5.3"
3672
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3673
+
checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4"
3674
+
dependencies = [
3675
+
"futures-core",
3676
+
"futures-util",
3677
+
"pin-project-lite",
3678
+
"sync_wrapper",
3679
+
"tokio",
3680
+
"tower-layer",
3681
+
"tower-service",
3682
+
"tracing",
3683
+
]
3684
+
3685
+
[[package]]
3686
+
name = "tower-http"
3687
+
version = "0.6.8"
3688
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3689
+
checksum = "d4e6559d53cc268e5031cd8429d05415bc4cb4aefc4aa5d6cc35fbf5b924a1f8"
3690
+
dependencies = [
3691
+
"bitflags 2.11.0",
3692
+
"bytes",
3693
+
"futures-util",
3694
+
"http",
3695
+
"http-body",
3696
+
"iri-string",
3697
+
"pin-project-lite",
3698
+
"tower",
3699
+
"tower-layer",
3700
+
"tower-service",
3701
+
]
3702
+
3703
+
[[package]]
3704
+
name = "tower-layer"
3705
+
version = "0.3.3"
3706
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3707
+
checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e"
3708
+
3709
+
[[package]]
3710
+
name = "tower-service"
3711
+
version = "0.3.3"
3712
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3713
+
checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3"
3714
+
3715
+
[[package]]
3716
+
name = "tracing"
3717
+
version = "0.1.44"
3718
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3719
+
checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100"
3720
+
dependencies = [
3721
+
"log",
3722
+
"pin-project-lite",
3723
+
"tracing-attributes",
3724
+
"tracing-core",
3725
+
]
3726
+
3727
+
[[package]]
3728
+
name = "tracing-attributes"
3729
+
version = "0.1.31"
3730
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3731
+
checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da"
3732
+
dependencies = [
3733
+
"proc-macro2",
3734
+
"quote",
3735
+
"syn 2.0.117",
3736
+
]
3737
+
3738
+
[[package]]
3739
+
name = "tracing-core"
3740
+
version = "0.1.36"
3741
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3742
+
checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
3743
+
dependencies = [
3744
+
"once_cell",
3745
+
"valuable",
3746
+
]
3747
+
3748
+
[[package]]
3749
+
name = "tracing-log"
3750
+
version = "0.2.0"
3751
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3752
+
checksum = "ee855f1f400bd0e5c02d150ae5de3840039a3f54b025156404e34c23c03f47c3"
3753
+
dependencies = [
3754
+
"log",
3755
+
"once_cell",
3756
+
"tracing-core",
3757
+
]
3758
+
3759
+
[[package]]
3760
+
name = "tracing-subscriber"
3761
+
version = "0.3.22"
3762
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3763
+
checksum = "2f30143827ddab0d256fd843b7a66d164e9f271cfa0dde49142c5ca0ca291f1e"
3764
+
dependencies = [
3765
+
"matchers",
3766
+
"nu-ansi-term",
3767
+
"once_cell",
3768
+
"regex-automata",
3769
+
"sharded-slab",
3770
+
"smallvec",
3771
+
"thread_local",
3772
+
"tracing",
3773
+
"tracing-core",
3774
+
"tracing-log",
3775
+
]
3776
+
3777
+
[[package]]
3778
+
name = "try-lock"
3779
+
version = "0.2.5"
3780
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3781
+
checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
3782
+
3783
+
[[package]]
3784
+
name = "tungstenite"
3785
+
version = "0.28.0"
3786
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3787
+
checksum = "8628dcc84e5a09eb3d8423d6cb682965dea9133204e8fb3efee74c2a0c259442"
3788
+
dependencies = [
3789
+
"bytes",
3790
+
"data-encoding",
3791
+
"http",
3792
+
"httparse",
3793
+
"log",
3794
+
"rand 0.9.2",
3795
+
"sha1",
3796
+
"thiserror 2.0.18",
3797
+
"utf-8",
3798
+
]
3799
+
3800
+
[[package]]
3801
+
name = "twox-hash"
3802
+
version = "2.1.2"
3803
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3804
+
checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c"
3805
+
3806
+
[[package]]
3807
+
name = "typenum"
3808
+
version = "1.19.0"
3809
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3810
+
checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb"
3811
+
3812
+
[[package]]
3813
+
name = "ucd-trie"
3814
+
version = "0.1.7"
3815
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3816
+
checksum = "2896d95c02a80c6d6a5d6e953d479f5ddf2dfdb6a244441010e373ac0fb88971"
3817
+
3818
+
[[package]]
3819
+
name = "ulid"
3820
+
version = "1.2.1"
3821
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3822
+
checksum = "470dbf6591da1b39d43c14523b2b469c86879a53e8b758c8e090a470fe7b1fbe"
3823
+
dependencies = [
3824
+
"rand 0.9.2",
3825
+
"web-time",
3826
+
]
3827
+
3828
+
[[package]]
3829
+
name = "unicase"
3830
+
version = "2.9.0"
3831
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3832
+
checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142"
3833
+
3834
+
[[package]]
3835
+
name = "unicode-ident"
3836
+
version = "1.0.24"
3837
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3838
+
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
3839
+
3840
+
[[package]]
3841
+
name = "unicode-segmentation"
3842
+
version = "1.12.0"
3843
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3844
+
checksum = "f6ccf251212114b54433ec949fd6a7841275f9ada20dddd2f29e9ceea4501493"
3845
+
3846
+
[[package]]
3847
+
name = "unicode-truncate"
3848
+
version = "2.0.1"
3849
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3850
+
checksum = "16b380a1238663e5f8a691f9039c73e1cdae598a30e9855f541d29b08b53e9a5"
3851
+
dependencies = [
3852
+
"itertools",
3853
+
"unicode-segmentation",
3854
+
"unicode-width",
3855
+
]
3856
+
3857
+
[[package]]
3858
+
name = "unicode-width"
3859
+
version = "0.2.2"
3860
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3861
+
checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
3862
+
3863
+
[[package]]
3864
+
name = "unicode-xid"
3865
+
version = "0.2.6"
3866
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3867
+
checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853"
3868
+
3869
+
[[package]]
3870
+
name = "unsigned-varint"
3871
+
version = "0.8.0"
3872
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3873
+
checksum = "eb066959b24b5196ae73cb057f45598450d2c5f71460e98c49b738086eff9c06"
3874
+
3875
+
[[package]]
3876
+
name = "untrusted"
3877
+
version = "0.9.0"
3878
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3879
+
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
3880
+
3881
+
[[package]]
3882
+
name = "url"
3883
+
version = "2.5.8"
3884
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3885
+
checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed"
3886
+
dependencies = [
3887
+
"form_urlencoded",
3888
+
"idna",
3889
+
"percent-encoding",
3890
+
"serde",
3891
+
]
3892
+
3893
+
[[package]]
3894
+
name = "utf-8"
3895
+
version = "0.7.6"
3896
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3897
+
checksum = "09cc8ee72d2a9becf2f2febe0205bbed8fc6615b7cb429ad062dc7b7ddd036a9"
3898
+
3899
+
[[package]]
3900
+
name = "utf8_iter"
3901
+
version = "1.0.4"
3902
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3903
+
checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
3904
+
3905
+
[[package]]
3906
+
name = "utf8parse"
3907
+
version = "0.2.2"
3908
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3909
+
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
3910
+
3911
+
[[package]]
3912
+
name = "uuid"
3913
+
version = "1.22.0"
3914
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3915
+
checksum = "a68d3c8f01c0cfa54a75291d83601161799e4a89a39e0929f4b0354d88757a37"
3916
+
dependencies = [
3917
+
"atomic",
3918
+
"getrandom 0.4.2",
3919
+
"js-sys",
3920
+
"wasm-bindgen",
3921
+
]
3922
+
3923
+
[[package]]
3924
+
name = "valuable"
3925
+
version = "0.1.1"
3926
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3927
+
checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65"
3928
+
3929
+
[[package]]
3930
+
name = "varint-rs"
3931
+
version = "2.2.0"
3932
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3933
+
checksum = "8f54a172d0620933a27a4360d3db3e2ae0dd6cceae9730751a036bbf182c4b23"
3934
+
3935
+
[[package]]
3936
+
name = "version_check"
3937
+
version = "0.9.5"
3938
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3939
+
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
3940
+
3941
+
[[package]]
3942
+
name = "vtparse"
3943
+
version = "0.6.2"
3944
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3945
+
checksum = "6d9b2acfb050df409c972a37d3b8e08cdea3bddb0c09db9d53137e504cfabed0"
3946
+
dependencies = [
3947
+
"utf8parse",
3948
+
]
3949
+
3950
+
[[package]]
3951
+
name = "want"
3952
+
version = "0.3.1"
3953
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3954
+
checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
3955
+
dependencies = [
3956
+
"try-lock",
3957
+
]
3958
+
3959
+
[[package]]
3960
+
name = "wasi"
3961
+
version = "0.11.1+wasi-snapshot-preview1"
3962
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3963
+
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
3964
+
3965
+
[[package]]
3966
+
name = "wasip2"
3967
+
version = "1.0.2+wasi-0.2.9"
3968
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3969
+
checksum = "9517f9239f02c069db75e65f174b3da828fe5f5b945c4dd26bd25d89c03ebcf5"
3970
+
dependencies = [
3971
+
"wit-bindgen",
3972
+
]
3973
+
3974
+
[[package]]
3975
+
name = "wasip3"
3976
+
version = "0.4.0+wasi-0.3.0-rc-2026-01-06"
3977
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3978
+
checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5"
3979
+
dependencies = [
3980
+
"wit-bindgen",
3981
+
]
3982
+
3983
+
[[package]]
3984
+
name = "wasm-bindgen"
3985
+
version = "0.2.114"
3986
+
source = "registry+https://github.com/rust-lang/crates.io-index"
3987
+
checksum = "6532f9a5c1ece3798cb1c2cfdba640b9b3ba884f5db45973a6f442510a87d38e"
3988
+
dependencies = [
3989
+
"cfg-if",
3990
+
"once_cell",
3991
+
"rustversion",
3992
+
"wasm-bindgen-macro",
3993
+
"wasm-bindgen-shared",
3994
+
]
3995
+
3996
+
[[package]]
3997
+
name = "wasm-bindgen-futures"
3998
+
version = "0.4.64"
3999
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4000
+
checksum = "e9c5522b3a28661442748e09d40924dfb9ca614b21c00d3fd135720e48b67db8"
4001
+
dependencies = [
4002
+
"cfg-if",
4003
+
"futures-util",
4004
+
"js-sys",
4005
+
"once_cell",
4006
+
"wasm-bindgen",
4007
+
"web-sys",
4008
+
]
4009
+
4010
+
[[package]]
4011
+
name = "wasm-bindgen-macro"
4012
+
version = "0.2.114"
4013
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4014
+
checksum = "18a2d50fcf105fb33bb15f00e7a77b772945a2ee45dcf454961fd843e74c18e6"
4015
+
dependencies = [
4016
+
"quote",
4017
+
"wasm-bindgen-macro-support",
4018
+
]
4019
+
4020
+
[[package]]
4021
+
name = "wasm-bindgen-macro-support"
4022
+
version = "0.2.114"
4023
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4024
+
checksum = "03ce4caeaac547cdf713d280eda22a730824dd11e6b8c3ca9e42247b25c631e3"
4025
+
dependencies = [
4026
+
"bumpalo",
4027
+
"proc-macro2",
4028
+
"quote",
4029
+
"syn 2.0.117",
4030
+
"wasm-bindgen-shared",
4031
+
]
4032
+
4033
+
[[package]]
4034
+
name = "wasm-bindgen-shared"
4035
+
version = "0.2.114"
4036
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4037
+
checksum = "75a326b8c223ee17883a4251907455a2431acc2791c98c26279376490c378c16"
4038
+
dependencies = [
4039
+
"unicode-ident",
4040
+
]
4041
+
4042
+
[[package]]
4043
+
name = "wasm-encoder"
4044
+
version = "0.244.0"
4045
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4046
+
checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319"
4047
+
dependencies = [
4048
+
"leb128fmt",
4049
+
"wasmparser",
4050
+
]
4051
+
4052
+
[[package]]
4053
+
name = "wasm-metadata"
4054
+
version = "0.244.0"
4055
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4056
+
checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909"
4057
+
dependencies = [
4058
+
"anyhow",
4059
+
"indexmap",
4060
+
"wasm-encoder",
4061
+
"wasmparser",
4062
+
]
4063
+
4064
+
[[package]]
4065
+
name = "wasmparser"
4066
+
version = "0.244.0"
4067
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4068
+
checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe"
4069
+
dependencies = [
4070
+
"bitflags 2.11.0",
4071
+
"hashbrown 0.15.5",
4072
+
"indexmap",
4073
+
"semver",
4074
+
]
4075
+
4076
+
[[package]]
4077
+
name = "web-sys"
4078
+
version = "0.3.91"
4079
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4080
+
checksum = "854ba17bb104abfb26ba36da9729addc7ce7f06f5c0f90f3c391f8461cca21f9"
4081
+
dependencies = [
4082
+
"js-sys",
4083
+
"wasm-bindgen",
4084
+
]
4085
+
4086
+
[[package]]
4087
+
name = "web-time"
4088
+
version = "1.1.0"
4089
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4090
+
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
4091
+
dependencies = [
4092
+
"js-sys",
4093
+
"wasm-bindgen",
4094
+
]
4095
+
4096
+
[[package]]
4097
+
name = "webpki-roots"
4098
+
version = "1.0.6"
4099
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4100
+
checksum = "22cfaf3c063993ff62e73cb4311efde4db1efb31ab78a3e5c457939ad5cc0bed"
4101
+
dependencies = [
4102
+
"rustls-pki-types",
4103
+
]
4104
+
4105
+
[[package]]
4106
+
name = "wezterm-bidi"
4107
+
version = "0.2.3"
4108
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4109
+
checksum = "0c0a6e355560527dd2d1cf7890652f4f09bb3433b6aadade4c9b5ed76de5f3ec"
4110
+
dependencies = [
4111
+
"log",
4112
+
"wezterm-dynamic",
4113
+
]
4114
+
4115
+
[[package]]
4116
+
name = "wezterm-blob-leases"
4117
+
version = "0.1.1"
4118
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4119
+
checksum = "692daff6d93d94e29e4114544ef6d5c942a7ed998b37abdc19b17136ea428eb7"
4120
+
dependencies = [
4121
+
"getrandom 0.3.4",
4122
+
"mac_address",
4123
+
"sha2",
4124
+
"thiserror 1.0.69",
4125
+
"uuid",
4126
+
]
4127
+
4128
+
[[package]]
4129
+
name = "wezterm-color-types"
4130
+
version = "0.3.0"
4131
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4132
+
checksum = "7de81ef35c9010270d63772bebef2f2d6d1f2d20a983d27505ac850b8c4b4296"
4133
+
dependencies = [
4134
+
"csscolorparser",
4135
+
"deltae",
4136
+
"lazy_static",
4137
+
"wezterm-dynamic",
4138
+
]
4139
+
4140
+
[[package]]
4141
+
name = "wezterm-dynamic"
4142
+
version = "0.2.1"
4143
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4144
+
checksum = "5f2ab60e120fd6eaa68d9567f3226e876684639d22a4219b313ff69ec0ccd5ac"
4145
+
dependencies = [
4146
+
"log",
4147
+
"ordered-float",
4148
+
"strsim",
4149
+
"thiserror 1.0.69",
4150
+
"wezterm-dynamic-derive",
4151
+
]
4152
+
4153
+
[[package]]
4154
+
name = "wezterm-dynamic-derive"
4155
+
version = "0.1.1"
4156
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4157
+
checksum = "46c0cf2d539c645b448eaffec9ec494b8b19bd5077d9e58cb1ae7efece8d575b"
4158
+
dependencies = [
4159
+
"proc-macro2",
4160
+
"quote",
4161
+
"syn 1.0.109",
4162
+
]
4163
+
4164
+
[[package]]
4165
+
name = "wezterm-input-types"
4166
+
version = "0.1.0"
4167
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4168
+
checksum = "7012add459f951456ec9d6c7e6fc340b1ce15d6fc9629f8c42853412c029e57e"
4169
+
dependencies = [
4170
+
"bitflags 1.3.2",
4171
+
"euclid",
4172
+
"lazy_static",
4173
+
"serde",
4174
+
"wezterm-dynamic",
4175
+
]
4176
+
4177
+
[[package]]
4178
+
name = "widestring"
4179
+
version = "1.2.1"
4180
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4181
+
checksum = "72069c3113ab32ab29e5584db3c6ec55d416895e60715417b5b883a357c3e471"
4182
+
4183
+
[[package]]
4184
+
name = "winapi"
4185
+
version = "0.3.9"
4186
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4187
+
checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
4188
+
dependencies = [
4189
+
"winapi-i686-pc-windows-gnu",
4190
+
"winapi-x86_64-pc-windows-gnu",
4191
+
]
4192
+
4193
+
[[package]]
4194
+
name = "winapi-i686-pc-windows-gnu"
4195
+
version = "0.4.0"
4196
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4197
+
checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
4198
+
4199
+
[[package]]
4200
+
name = "winapi-x86_64-pc-windows-gnu"
4201
+
version = "0.4.0"
4202
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4203
+
checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
4204
+
4205
+
[[package]]
4206
+
name = "windows-link"
4207
+
version = "0.2.1"
4208
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4209
+
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
4210
+
4211
+
[[package]]
4212
+
name = "windows-registry"
4213
+
version = "0.6.1"
4214
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4215
+
checksum = "02752bf7fbdcce7f2a27a742f798510f3e5ad88dbe84871e5168e2120c3d5720"
4216
+
dependencies = [
4217
+
"windows-link",
4218
+
"windows-result",
4219
+
"windows-strings",
4220
+
]
4221
+
4222
+
[[package]]
4223
+
name = "windows-result"
4224
+
version = "0.4.1"
4225
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4226
+
checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5"
4227
+
dependencies = [
4228
+
"windows-link",
4229
+
]
4230
+
4231
+
[[package]]
4232
+
name = "windows-strings"
4233
+
version = "0.5.1"
4234
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4235
+
checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091"
4236
+
dependencies = [
4237
+
"windows-link",
4238
+
]
4239
+
4240
+
[[package]]
4241
+
name = "windows-sys"
4242
+
version = "0.48.0"
4243
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4244
+
checksum = "677d2418bec65e3338edb076e806bc1ec15693c5d0104683f2efe857f61056a9"
4245
+
dependencies = [
4246
+
"windows-targets 0.48.5",
4247
+
]
4248
+
4249
+
[[package]]
4250
+
name = "windows-sys"
4251
+
version = "0.52.0"
4252
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4253
+
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
4254
+
dependencies = [
4255
+
"windows-targets 0.52.6",
4256
+
]
4257
+
4258
+
[[package]]
4259
+
name = "windows-sys"
4260
+
version = "0.60.2"
4261
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4262
+
checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb"
4263
+
dependencies = [
4264
+
"windows-targets 0.53.5",
4265
+
]
4266
+
4267
+
[[package]]
4268
+
name = "windows-sys"
4269
+
version = "0.61.2"
4270
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4271
+
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
4272
+
dependencies = [
4273
+
"windows-link",
4274
+
]
4275
+
4276
+
[[package]]
4277
+
name = "windows-targets"
4278
+
version = "0.48.5"
4279
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4280
+
checksum = "9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c"
4281
+
dependencies = [
4282
+
"windows_aarch64_gnullvm 0.48.5",
4283
+
"windows_aarch64_msvc 0.48.5",
4284
+
"windows_i686_gnu 0.48.5",
4285
+
"windows_i686_msvc 0.48.5",
4286
+
"windows_x86_64_gnu 0.48.5",
4287
+
"windows_x86_64_gnullvm 0.48.5",
4288
+
"windows_x86_64_msvc 0.48.5",
4289
+
]
4290
+
4291
+
[[package]]
4292
+
name = "windows-targets"
4293
+
version = "0.52.6"
4294
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4295
+
checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973"
4296
+
dependencies = [
4297
+
"windows_aarch64_gnullvm 0.52.6",
4298
+
"windows_aarch64_msvc 0.52.6",
4299
+
"windows_i686_gnu 0.52.6",
4300
+
"windows_i686_gnullvm 0.52.6",
4301
+
"windows_i686_msvc 0.52.6",
4302
+
"windows_x86_64_gnu 0.52.6",
4303
+
"windows_x86_64_gnullvm 0.52.6",
4304
+
"windows_x86_64_msvc 0.52.6",
4305
+
]
4306
+
4307
+
[[package]]
4308
+
name = "windows-targets"
4309
+
version = "0.53.5"
4310
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4311
+
checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3"
4312
+
dependencies = [
4313
+
"windows-link",
4314
+
"windows_aarch64_gnullvm 0.53.1",
4315
+
"windows_aarch64_msvc 0.53.1",
4316
+
"windows_i686_gnu 0.53.1",
4317
+
"windows_i686_gnullvm 0.53.1",
4318
+
"windows_i686_msvc 0.53.1",
4319
+
"windows_x86_64_gnu 0.53.1",
4320
+
"windows_x86_64_gnullvm 0.53.1",
4321
+
"windows_x86_64_msvc 0.53.1",
4322
+
]
4323
+
4324
+
[[package]]
4325
+
name = "windows_aarch64_gnullvm"
4326
+
version = "0.48.5"
4327
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4328
+
checksum = "2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8"
4329
+
4330
+
[[package]]
4331
+
name = "windows_aarch64_gnullvm"
4332
+
version = "0.52.6"
4333
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4334
+
checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3"
4335
+
4336
+
[[package]]
4337
+
name = "windows_aarch64_gnullvm"
4338
+
version = "0.53.1"
4339
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4340
+
checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53"
4341
+
4342
+
[[package]]
4343
+
name = "windows_aarch64_msvc"
4344
+
version = "0.48.5"
4345
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4346
+
checksum = "dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc"
4347
+
4348
+
[[package]]
4349
+
name = "windows_aarch64_msvc"
4350
+
version = "0.52.6"
4351
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4352
+
checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469"
4353
+
4354
+
[[package]]
4355
+
name = "windows_aarch64_msvc"
4356
+
version = "0.53.1"
4357
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4358
+
checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006"
4359
+
4360
+
[[package]]
4361
+
name = "windows_i686_gnu"
4362
+
version = "0.48.5"
4363
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4364
+
checksum = "a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e"
4365
+
4366
+
[[package]]
4367
+
name = "windows_i686_gnu"
4368
+
version = "0.52.6"
4369
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4370
+
checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b"
4371
+
4372
+
[[package]]
4373
+
name = "windows_i686_gnu"
4374
+
version = "0.53.1"
4375
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4376
+
checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3"
4377
+
4378
+
[[package]]
4379
+
name = "windows_i686_gnullvm"
4380
+
version = "0.52.6"
4381
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4382
+
checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66"
4383
+
4384
+
[[package]]
4385
+
name = "windows_i686_gnullvm"
4386
+
version = "0.53.1"
4387
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4388
+
checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c"
4389
+
4390
+
[[package]]
4391
+
name = "windows_i686_msvc"
4392
+
version = "0.48.5"
4393
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4394
+
checksum = "8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406"
4395
+
4396
+
[[package]]
4397
+
name = "windows_i686_msvc"
4398
+
version = "0.52.6"
4399
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4400
+
checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66"
4401
+
4402
+
[[package]]
4403
+
name = "windows_i686_msvc"
4404
+
version = "0.53.1"
4405
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4406
+
checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2"
4407
+
4408
+
[[package]]
4409
+
name = "windows_x86_64_gnu"
4410
+
version = "0.48.5"
4411
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4412
+
checksum = "53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e"
4413
+
4414
+
[[package]]
4415
+
name = "windows_x86_64_gnu"
4416
+
version = "0.52.6"
4417
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4418
+
checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78"
4419
+
4420
+
[[package]]
4421
+
name = "windows_x86_64_gnu"
4422
+
version = "0.53.1"
4423
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4424
+
checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499"
4425
+
4426
+
[[package]]
4427
+
name = "windows_x86_64_gnullvm"
4428
+
version = "0.48.5"
4429
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4430
+
checksum = "0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc"
4431
+
4432
+
[[package]]
4433
+
name = "windows_x86_64_gnullvm"
4434
+
version = "0.52.6"
4435
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4436
+
checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d"
4437
+
4438
+
[[package]]
4439
+
name = "windows_x86_64_gnullvm"
4440
+
version = "0.53.1"
4441
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4442
+
checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1"
4443
+
4444
+
[[package]]
4445
+
name = "windows_x86_64_msvc"
4446
+
version = "0.48.5"
4447
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4448
+
checksum = "ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538"
4449
+
4450
+
[[package]]
4451
+
name = "windows_x86_64_msvc"
4452
+
version = "0.52.6"
4453
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4454
+
checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec"
4455
+
4456
+
[[package]]
4457
+
name = "windows_x86_64_msvc"
4458
+
version = "0.53.1"
4459
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4460
+
checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650"
4461
+
4462
+
[[package]]
4463
+
name = "winreg"
4464
+
version = "0.50.0"
4465
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4466
+
checksum = "524e57b2c537c0f9b1e69f1965311ec12182b4122e45035b1508cd24d2adadb1"
4467
+
dependencies = [
4468
+
"cfg-if",
4469
+
"windows-sys 0.48.0",
4470
+
]
4471
+
4472
+
[[package]]
4473
+
name = "wit-bindgen"
4474
+
version = "0.51.0"
4475
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4476
+
checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5"
4477
+
dependencies = [
4478
+
"wit-bindgen-rust-macro",
4479
+
]
4480
+
4481
+
[[package]]
4482
+
name = "wit-bindgen-core"
4483
+
version = "0.51.0"
4484
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4485
+
checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc"
4486
+
dependencies = [
4487
+
"anyhow",
4488
+
"heck",
4489
+
"wit-parser",
4490
+
]
4491
+
4492
+
[[package]]
4493
+
name = "wit-bindgen-rust"
4494
+
version = "0.51.0"
4495
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4496
+
checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21"
4497
+
dependencies = [
4498
+
"anyhow",
4499
+
"heck",
4500
+
"indexmap",
4501
+
"prettyplease",
4502
+
"syn 2.0.117",
4503
+
"wasm-metadata",
4504
+
"wit-bindgen-core",
4505
+
"wit-component",
4506
+
]
4507
+
4508
+
[[package]]
4509
+
name = "wit-bindgen-rust-macro"
4510
+
version = "0.51.0"
4511
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4512
+
checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a"
4513
+
dependencies = [
4514
+
"anyhow",
4515
+
"prettyplease",
4516
+
"proc-macro2",
4517
+
"quote",
4518
+
"syn 2.0.117",
4519
+
"wit-bindgen-core",
4520
+
"wit-bindgen-rust",
4521
+
]
4522
+
4523
+
[[package]]
4524
+
name = "wit-component"
4525
+
version = "0.244.0"
4526
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4527
+
checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2"
4528
+
dependencies = [
4529
+
"anyhow",
4530
+
"bitflags 2.11.0",
4531
+
"indexmap",
4532
+
"log",
4533
+
"serde",
4534
+
"serde_derive",
4535
+
"serde_json",
4536
+
"wasm-encoder",
4537
+
"wasm-metadata",
4538
+
"wasmparser",
4539
+
"wit-parser",
4540
+
]
4541
+
4542
+
[[package]]
4543
+
name = "wit-parser"
4544
+
version = "0.244.0"
4545
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4546
+
checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736"
4547
+
dependencies = [
4548
+
"anyhow",
4549
+
"id-arena",
4550
+
"indexmap",
4551
+
"log",
4552
+
"semver",
4553
+
"serde",
4554
+
"serde_derive",
4555
+
"serde_json",
4556
+
"unicode-xid",
4557
+
"wasmparser",
4558
+
]
4559
+
4560
+
[[package]]
4561
+
name = "writeable"
4562
+
version = "0.6.2"
4563
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4564
+
checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9"
4565
+
4566
+
[[package]]
4567
+
name = "xxhash-rust"
4568
+
version = "0.8.15"
4569
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4570
+
checksum = "fdd20c5420375476fbd4394763288da7eb0cc0b8c11deed431a91562af7335d3"
4571
+
4572
+
[[package]]
4573
+
name = "yoke"
4574
+
version = "0.8.1"
4575
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4576
+
checksum = "72d6e5c6afb84d73944e5cedb052c4680d5657337201555f9f2a16b7406d4954"
4577
+
dependencies = [
4578
+
"stable_deref_trait",
4579
+
"yoke-derive",
4580
+
"zerofrom",
4581
+
]
4582
+
4583
+
[[package]]
4584
+
name = "yoke-derive"
4585
+
version = "0.8.1"
4586
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4587
+
checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d"
4588
+
dependencies = [
4589
+
"proc-macro2",
4590
+
"quote",
4591
+
"syn 2.0.117",
4592
+
"synstructure",
4593
+
]
4594
+
4595
+
[[package]]
4596
+
name = "zerocopy"
4597
+
version = "0.8.42"
4598
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4599
+
checksum = "f2578b716f8a7a858b7f02d5bd870c14bf4ddbbcf3a4c05414ba6503640505e3"
4600
+
dependencies = [
4601
+
"zerocopy-derive",
4602
+
]
4603
+
4604
+
[[package]]
4605
+
name = "zerocopy-derive"
4606
+
version = "0.8.42"
4607
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4608
+
checksum = "7e6cc098ea4d3bd6246687de65af3f920c430e236bee1e3bf2e441463f08a02f"
4609
+
dependencies = [
4610
+
"proc-macro2",
4611
+
"quote",
4612
+
"syn 2.0.117",
4613
+
]
4614
+
4615
+
[[package]]
4616
+
name = "zerofrom"
4617
+
version = "0.1.6"
4618
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4619
+
checksum = "50cc42e0333e05660c3587f3bf9d0478688e15d870fab3346451ce7f8c9fbea5"
4620
+
dependencies = [
4621
+
"zerofrom-derive",
4622
+
]
4623
+
4624
+
[[package]]
4625
+
name = "zerofrom-derive"
4626
+
version = "0.1.6"
4627
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4628
+
checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502"
4629
+
dependencies = [
4630
+
"proc-macro2",
4631
+
"quote",
4632
+
"syn 2.0.117",
4633
+
"synstructure",
4634
+
]
4635
+
4636
+
[[package]]
4637
+
name = "zeroize"
4638
+
version = "1.8.2"
4639
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4640
+
checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0"
4641
+
4642
+
[[package]]
4643
+
name = "zerotrie"
4644
+
version = "0.2.3"
4645
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4646
+
checksum = "2a59c17a5562d507e4b54960e8569ebee33bee890c70aa3fe7b97e85a9fd7851"
4647
+
dependencies = [
4648
+
"displaydoc",
4649
+
"yoke",
4650
+
"zerofrom",
4651
+
]
4652
+
4653
+
[[package]]
4654
+
name = "zerovec"
4655
+
version = "0.11.5"
4656
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4657
+
checksum = "6c28719294829477f525be0186d13efa9a3c602f7ec202ca9e353d310fb9a002"
4658
+
dependencies = [
4659
+
"yoke",
4660
+
"zerofrom",
4661
+
"zerovec-derive",
4662
+
]
4663
+
4664
+
[[package]]
4665
+
name = "zerovec-derive"
4666
+
version = "0.11.2"
4667
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4668
+
checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3"
4669
+
dependencies = [
4670
+
"proc-macro2",
4671
+
"quote",
4672
+
"syn 2.0.117",
4673
+
]
4674
+
4675
+
[[package]]
4676
+
name = "zmij"
4677
+
version = "1.0.21"
4678
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4679
+
checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa"
4680
+
4681
+
[[package]]
4682
+
name = "zstd"
4683
+
version = "0.13.3"
4684
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4685
+
checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a"
4686
+
dependencies = [
4687
+
"zstd-safe",
4688
+
]
4689
+
4690
+
[[package]]
4691
+
name = "zstd-safe"
4692
+
version = "7.2.4"
4693
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4694
+
checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d"
4695
+
dependencies = [
4696
+
"zstd-sys",
4697
+
]
4698
+
4699
+
[[package]]
4700
+
name = "zstd-sys"
4701
+
version = "2.0.16+zstd.1.5.7"
4702
+
source = "registry+https://github.com/rust-lang/crates.io-index"
4703
+
checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748"
4704
+
dependencies = [
4705
+
"cc",
4706
+
"pkg-config",
4707
+
]
+76
Cargo.toml
+76
Cargo.toml
···
1
+
[package]
2
+
name = "ramjet"
3
+
version = "0.1.0"
4
+
edition = "2024"
5
+
rust-version = "1.94"
6
+
description = "ATProtocol event-stream, record, and blob service built on fjall"
7
+
license = "MIT"
8
+
9
+
[dependencies]
10
+
# ATProto crates
11
+
atproto-dasl = { git = "https://tangled.org/ngerakines.me/atproto-crates", branch = "main" }
12
+
atproto-repo = { git = "https://tangled.org/ngerakines.me/atproto-crates", branch = "main", default-features = false }
13
+
atproto-identity = { git = "https://tangled.org/ngerakines.me/atproto-crates", branch = "main" }
14
+
atproto-record = { git = "https://tangled.org/ngerakines.me/atproto-crates", branch = "main", default-features = false }
15
+
atproto-xrpcs = { git = "https://tangled.org/ngerakines.me/atproto-crates", branch = "main" }
16
+
17
+
# Storage
18
+
fjall = "3"
19
+
20
+
# Compression
21
+
zstd = "0.13"
22
+
23
+
# Web / async
24
+
axum = { version = "0.8", features = ["macros", "ws"] }
25
+
tokio = { version = "1", features = ["macros", "rt", "rt-multi-thread", "signal", "sync", "net"] }
26
+
tokio-websockets = { version = "0.13", features = ["client", "rustls-native-roots", "fastrand", "ring"] }
27
+
tokio-metrics = "0.4"
28
+
tokio-util = { version = "0.7", features = ["io-util"] }
29
+
30
+
# Serialization
31
+
serde = { version = "1.0", features = ["derive"] }
32
+
serde_json = "1.0"
33
+
34
+
# Observability
35
+
tracing = { version = "0.1", features = ["async-await"] }
36
+
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
37
+
prometheus-client = "0.23"
38
+
39
+
# Rate limiting
40
+
governor = "0.8"
41
+
42
+
# Error handling
43
+
thiserror = "2.0"
44
+
anyhow = "1.0"
45
+
46
+
# CLI
47
+
clap = { version = "4.5", features = ["derive", "env"] }
48
+
49
+
# HTTP client (for backfill)
50
+
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "blocking"] }
51
+
52
+
# Async stream utilities
53
+
futures-util = "0.3"
54
+
55
+
# Base64 encoding
56
+
base64 = "0.22"
57
+
58
+
# HTTP types (for tokio-websockets URI)
59
+
http = "1"
60
+
61
+
# TUI
62
+
ratatui = "0.30"
63
+
crossterm = "0.29"
64
+
65
+
# Set reconciliation
66
+
riblt = { git = "https://github.com/ngerakines/riblt-rs" }
67
+
68
+
# Utilities
69
+
cid = "0.11"
70
+
multihash = "0.19"
71
+
sha2 = "0.10"
72
+
bytes = "1"
73
+
rand = "0.10.0"
74
+
75
+
[lints.rust]
76
+
unsafe_code = "forbid"
+196
-1
README.md
+196
-1
README.md
···
1
-
# ramjet-mature-wolfhound
1
+
# Ramjet
2
+
3
+
Single-node Rust service that consumes the ATProtocol firehose, persists records for tracked collections in [fjall](https://github.com/fjall-rs/fjall) (pure-Rust LSM-tree), and re-emits events via WebSocket fan-out with priority ordering.
4
+
5
+
## Motivation
6
+
7
+
ATProtocol applications often need a mix of precisely tracked collections and best-effort network-wide intake. Ramjet solves this with separately configured **tracked** and **forwarded** collection sets:
8
+
9
+
- **Tracked collections** are persisted to storage and emitted on the high-priority fan-out channel — these are the records your application depends on for correctness.
10
+
- **Forwarded collections** are emitted on the low-priority fan-out channel without being persisted — useful for analytics, search indexing, or other consumers that want broad visibility without the storage cost.
11
+
12
+
For example, [Lexicon Garden](https://lexicon.garden) needs accurately tracked lexicon schema and AppView-specific collections, but also wants all records for analytics. [Smoke Signal](https://smokesignal.events) similarly tracks event, RSVP, profile, and location records from different collections explicitly, while still consuming the full network stream.
13
+
14
+
Both collection configurations can be updated at runtime, and the admin API supports on-demand backfill and resync operations for any repository — making it straightforward to add new tracked collections and catch up on historical data.
15
+
16
+
## Quick start
17
+
18
+
```bash
19
+
cargo build --release
20
+
21
+
cargo run --release -- \
22
+
--db-path /var/lib/ramjet/data \
23
+
--listen-addr 0.0.0.0:8080 \
24
+
--relay-host bsky.network \
25
+
--tracked-collections "app.bsky.feed.*" \
26
+
--forward-collections "app.bsky.**"
27
+
```
28
+
29
+
## Configuration
30
+
31
+
All options are available as CLI flags or environment variables.
32
+
33
+
| Flag | Env | Default | Description |
34
+
|---|---|---|---|
35
+
| `--db-path` | `RAMJET_DB_PATH` | `./data/ramjet.db` | fjall database directory |
36
+
| `--listen-addr` | `RAMJET_LISTEN_ADDR` | `0.0.0.0:8080` | HTTP listen address |
37
+
| `--relay-host` | `RAMJET_RELAY_HOST` | `bsky.network` | Firehose relay hostname |
38
+
| `--tracked-collections` | `RAMJET_TRACKED_COLLECTIONS` | `*` | Collections to persist (space-separated patterns) |
39
+
| `--forward-collections` | `RAMJET_FORWARD_COLLECTIONS` | `*` | Collections to forward via WebSocket only |
40
+
| `--event-retention-hours` | `RAMJET_EVENT_RETENTION_HOURS` | `72` | Hours to retain events |
41
+
| `--batch-size` | `RAMJET_BATCH_SIZE` | `500` | Ingester batch size |
42
+
| `--batch-timeout-ms` | `RAMJET_BATCH_TIMEOUT_MS` | `100` | Batch flush timeout (ms) |
43
+
| `--admin-dids` | `RAMJET_ADMIN_DIDS` | *(empty)* | Comma-separated admin DIDs for protected endpoints |
44
+
| `--zstd-dict-path` | `RAMJET_ZSTD_DICT_PATH` | *(none)* | Path to zstd dictionary for event compression |
45
+
| `--backfill` | `RAMJET_BACKFILL` | *(empty)* | Comma-separated DIDs to backfill on startup |
46
+
| `--consumer-group` | `RAMJET_CONSUMER_GROUPS` | *(empty)* | Consumer group definitions (`name:partitions`, comma-separated) |
47
+
48
+
### Collection patterns
49
+
50
+
- `*` — match all collections
51
+
- `com.example.feed` — exact NSID match
52
+
- `com.example.*` — single-segment wildcard (matches `com.example.feed` but not `com.example.feed.like`)
53
+
- `com.example.**` — glob wildcard (matches `com.example.feed`, `com.example.feed.like`, etc.)
54
+
55
+
### Zstd dictionary compression
56
+
57
+
When `--zstd-dict-path` is provided, Ramjet compresses events in the events keyspace using a pre-trained zstd dictionary. The dictionary is identified by hash and stored in the meta keyspace so dictionary changes are detected on restart. Clients can download the active dictionary via the `GET /dictionary` endpoint (supports `If-None-Match` with CID-based ETags).
58
+
59
+
### Consumer groups
60
+
61
+
Consumer groups partition events across multiple WebSocket connections by DID hash, similar to Kafka consumer groups. Define groups at startup:
62
+
63
+
```bash
64
+
cargo run --release -- \
65
+
--consumer-group indexers:3 \
66
+
--consumer-group notifiers:2
67
+
```
68
+
69
+
Clients connect with `?group=indexers&partition=0` to receive only events for DIDs that hash to partition 0. All events for the same DID always route to the same partition. Omitting `group`/`partition` gives broadcast mode (all events).
70
+
71
+
## Architecture
72
+
73
+
Five async pipelines connected by channels:
74
+
75
+
```
76
+
Firehose ──→ Ingester ──mpsc──→ Writer ──broadcast──→ WebSocket Fan-out
77
+
│ │ ↑
78
+
│ └─partitioned──→┤ (consumer groups)
79
+
├─→ fjall storage │
80
+
└─→ events keyspace ───────┘ (cursor replay)
81
+
82
+
Backfill Worker ──→ CAR fetch ──→ fjall storage
83
+
Identity Worker ──→ DID/handle resolution ──→ fjall storage
84
+
```
85
+
86
+
1. **Firehose Ingester** — connects to the relay via WebSocket, decodes CBOR frames (`#commit`, `#identity`, `#account`), and sends events over an mpsc channel. Auto-reconnects with a 2s delay.
87
+
2. **Batch Writer** — collects events into configurable batches (size + timeout), writes to fjall atomically via `WriteBatch`, and broadcasts to fan-out channels with priority routing (tracked = high, forwarded = low). Filters denied repos. Queues `tooBig` commits and desynced repos for backfill.
88
+
3. **WebSocket Fan-out** — serves `dev.ngerakines.ramjet.stream.subscribe` with cursor-based replay from the events keyspace, then switches to live streaming with biased priority delivery (high > low > client recv). Supports both broadcast and partitioned (consumer group) modes.
89
+
4. **Identity Worker** — resolves DID documents and handle mappings via concurrent resolution (bounded by a semaphore), updating the `did_to_doc` and `handle_to_did` keyspaces.
90
+
5. **Backfill Worker** — polls a backfill queue every 5s, fetches full repo CARs from PDS endpoints using `DiskRepository` (spills to disk for large repos), parses records, and writes to storage. Repos exceeding 5 consecutive failures are auto-denied.
91
+
92
+
### Storage
93
+
94
+
Eight fjall keyspaces:
95
+
96
+
| Keyspace | Key format | Value | Purpose |
97
+
|---|---|---|---|
98
+
| `records` | `did\0collection\0rkey\0rev` | CID + DAG-CBOR (empty = tombstone) | Versioned record history |
99
+
| `events` | `u64 BE` | Compact binary event | Event log for cursor replay |
100
+
| `meta` | string key | varies | Cursor, sequence counter, backfill queue, RIBLT cache |
101
+
| `repo_state` | DID bytes | `RepoState` (rev + status + denied + backfilled) | Per-repo state |
102
+
| `did_to_doc` | DID bytes | Timestamped JSON DID document | Identity cache |
103
+
| `handle_to_did` | handle bytes | DID string | Handle-to-DID mapping |
104
+
| `blobs` | blob key | blob data | Blob storage |
105
+
| `blob_meta` | blob key | metadata | Blob metadata |
106
+
107
+
Records are versioned by including the repository revision in the key. Each create/update appends a new entry; deletes append a tombstone (empty value). The latest version of a record is the last entry in a prefix scan over `did\0collection\0rkey\0`. This provides a built-in change history without a separate temporal keyspace.
108
+
109
+
### Event encoding
110
+
111
+
Events are stored in a compact binary format with tag-based versioning:
112
+
- **V1 tags** (0x01-0x03): 1B tag + 8B sequence + variable fields
113
+
- **V2 tags** (0x04-0x06): 1B tag + 8B sequence + 8B timestamp (microseconds) + variable fields
114
+
115
+
Both commit operations, identity events, and account events have dedicated compact encodings. The V2 format adds microsecond timestamps for latency tracking. Events are optionally compressed with zstd dictionary compression before storage.
116
+
117
+
## API
118
+
119
+
### Health and metrics
120
+
121
+
- `GET /_health` — returns `{"status":"ok","version":"0.1.0"}`
122
+
- `GET /metrics` — Prometheus text format (includes HTTP metrics, pipeline counters, tokio task metrics, fan-out queue depths)
123
+
124
+
### XRPC endpoints
125
+
126
+
| Method | Endpoint | Description |
127
+
|---|---|---|
128
+
| GET | `com.atproto.repo.getRecord` | Fetch a single record by repo/collection/rkey |
129
+
| GET | `com.atproto.repo.listRecords` | List records in a collection with cursor pagination |
130
+
| GET | `com.atproto.repo.describeRepo` | Repo metadata, collections, rev, DID document |
131
+
| GET | `com.atproto.identity.resolveIdentity` | Resolve DID or handle to DID document (supports `Cache-Control: max-stale=N`) |
132
+
| GET | `com.atproto.identity.resolveHandle` | Resolve handle to DID |
133
+
| GET | `com.atproto.identity.resolveDid` | Fetch DID document |
134
+
| GET | `dev.ngerakines.ramjet.stream.subscribe` | WebSocket event stream with cursor replay and consumer group support |
135
+
| GET | `dev.ngerakines.ramjet.repos.getReconciliation` | RIBLT sketch for set reconciliation |
136
+
137
+
### Set reconciliation (RIBLT)
138
+
139
+
The `getReconciliation` endpoint returns an [RIBLT](https://github.com/ngerakines/riblt-rs) sketch for a DID's tracked records, allowing consumers to efficiently determine which records they're missing without transferring the full record list.
140
+
141
+
```
142
+
GET /xrpc/dev.ngerakines.ramjet.repos.getReconciliation?did=did:plc:abc123
143
+
```
144
+
145
+
Returns `application/x-riblt` binary with response headers:
146
+
- `ETag` — repo rev (supports `If-None-Match` for 304 Not Modified)
147
+
- `X-Riblt-Records` — number of records encoded
148
+
- `X-Riblt-Rev` — repo revision at time of sketch generation
149
+
150
+
Optional `collection` parameter filters the sketch to a single collection.
151
+
152
+
**Client workflow:**
153
+
1. Build a local sketch from your records using the same symbol encoding (`collection\0rkey\0cid`)
154
+
2. Fetch Ramjet's sketch
155
+
3. Subtract and decode — the symmetric difference tells you which records you're missing (and which you have that Ramjet doesn't)
156
+
4. Fetch missing records via `getRecord` or `listRecords`
157
+
158
+
Sketch sizes are allocated in 25,000-cell increments, bumping to the next tier when the record count exceeds 50% past the current size. Sketches are cached per-DID and automatically invalidated when records change.
159
+
160
+
### Admin endpoints (JWT auth required)
161
+
162
+
| Method | Endpoint | Description |
163
+
|---|---|---|
164
+
| GET | `dev.ngerakines.ramjet.repos.getState` | Get repo state (rev, status, denied) |
165
+
| POST | `dev.ngerakines.ramjet.repos.setState` | Set repo denied flag |
166
+
| POST | `dev.ngerakines.ramjet.repos.resync` | Queue a repo for backfill |
167
+
| POST | `dev.ngerakines.ramjet.repos.rebuildReconciliation` | Force rebuild of RIBLT sketch cache |
168
+
169
+
### Other endpoints
170
+
171
+
| Method | Endpoint | Description |
172
+
|---|---|---|
173
+
| GET | `/dictionary` | Download the active zstd dictionary (supports `If-None-Match` with CID ETags) |
174
+
175
+
## Development
176
+
177
+
```bash
178
+
cargo check # type-check
179
+
cargo test # run tests (50 unit tests)
180
+
cargo fmt # format code
181
+
cargo build # debug build
182
+
```
183
+
184
+
### Binaries
185
+
186
+
- `ramjet` — main service
187
+
- `ramjet-writer` — benchmark binary for ingester-to-writer pipeline analysis
188
+
- `ramjet-data` — data inspection tool
189
+
- `ramjet-dictgen` — zstd dictionary generator
190
+
- `ramjet-forecast` — capacity forecasting
191
+
- `ramjet-consumer` — example WebSocket consumer
192
+
- `rjtop` — TUI dashboard
193
+
194
+
## License
195
+
196
+
MIT
+104
docs/zstd-compression-plan.md
+104
docs/zstd-compression-plan.md
···
1
+
# Zstd Dictionary Compression for Events and Records Keyspaces
2
+
3
+
## Goal
4
+
5
+
Compress data in the events and records keyspaces using zstd with a pre-trained dictionary. Small payloads (100B-few KB) don't compress well with standard zstd but achieve 3-6x compression with a dictionary that captures common patterns (DID prefixes, collection NSIDs, CBOR structure bytes).
6
+
7
+
## Part 1: Dictionary Training Tool
8
+
9
+
Create `src/bin/ramjet_dictgen.rs` that:
10
+
11
+
1. Opens the fjall database read-only
12
+
2. Iterates the events keyspace collecting 10K-100K raw compact event values into `Vec<Vec<u8>>`
13
+
3. Calls `zstd::dict::from_samples(&samples, dict_size)` (COVER algorithm)
14
+
4. Writes dictionary bytes to a file
15
+
5. Prints stats: sample count, dictionary size, average compression ratio with/without dictionary
16
+
17
+
### CLI args (clap)
18
+
19
+
- `--db-path` - path to fjall database (required)
20
+
- `--output` - output dictionary file path (required)
21
+
- `--max-samples` - max events to sample (default 50000)
22
+
- `--dict-size` - dictionary size in bytes (default 65536 = 64KB)
23
+
24
+
### Dependency
25
+
26
+
```toml
27
+
zstd = "0.13"
28
+
```
29
+
30
+
### Notes
31
+
32
+
- Samples should represent the actual distribution of commit/identity/account events
33
+
- 64KB dictionary is a good starting point; test 32KB and 112KB too
34
+
- Retrain when tracked collections change significantly
35
+
36
+
## Part 2: Compression Integration in Ramjet
37
+
38
+
### Config changes (`src/config.rs`)
39
+
40
+
- Add `--zstd-dict-path <PATH>` optional CLI arg to `CliArgs` struct
41
+
- Add `zstd_dict_path: Option<PathBuf>` field to `ServiceConfig`
42
+
- When absent, events are stored uncompressed (current behavior)
43
+
44
+
### Storage changes (`src/storage/mod.rs`)
45
+
46
+
- Add `zstd_dict: Option<Vec<u8>>` field to `FjallDb`
47
+
- Extend `FjallDb::open(path: &Path)` to accept an optional dictionary path, or load it in `main.rs` before passing to a new constructor
48
+
- Add helper methods:
49
+
- `compress_event(&self, data: &[u8]) -> Vec<u8>` - returns compressed if dict loaded, raw otherwise
50
+
- `decompress_event(&self, data: &[u8]) -> anyhow::Result<Vec<u8>>` - auto-detects format
51
+
52
+
### Format detection for backwards compatibility
53
+
54
+
- Zstd frames start with magic bytes `0x28B52FFD`
55
+
- Compact events start with `0x01`-`0x06` (v1: `0x01` commit, `0x02` identity, `0x03` account; v2: `0x04` commit_op, `0x05` identity_v2, `0x06` account_v2)
56
+
- Legacy JSON starts with `0x7B` (`{`)
57
+
- Check first 4 bytes to determine format; decompress only if zstd magic detected
58
+
59
+
### Writer changes (`src/pipeline/writer.rs`)
60
+
61
+
- After `encode_compact_commit_op`/`encode_compact_identity_v2`/`encode_compact_account_v2`, compress via `db.compress_event(&compact_event)`
62
+
- After `RecordValue::encode()`, compress via `db.compress_event(&encoded)` before inserting into records keyspace
63
+
- Create `zstd::bulk::Compressor::with_dictionary(level, &dict)` per batch (cheap to create)
64
+
- Compression level 3 (default) is a good balance
65
+
66
+
### Backfill changes (`src/pipeline/backfill.rs`)
67
+
68
+
- After `RecordValue::encode()`, compress via `db.compress_event(&encoded)` before inserting into records keyspace
69
+
70
+
### Replay changes (`src/server/websocket.rs`)
71
+
72
+
- Before calling `compact_event_to_cbor`, decompress if zstd magic detected
73
+
- Create `zstd::bulk::Decompressor::with_dictionary(&dict)` per connection
74
+
- Use `max_decompressed_size = 4 * 1024 * 1024` (4MB cap)
75
+
76
+
### XRPC changes (`src/server/xrpc.rs`)
77
+
78
+
- `getRecord` and `listRecords` decompress record values via `db.decompress_event()` before `RecordValue::decode()`
79
+
- Tombstones (empty values) pass through decompression unchanged
80
+
81
+
### Meta keyspace
82
+
83
+
- Store `zstd_dict_id` (hash of dictionary bytes) in meta keyspace at startup
84
+
- Log a warning if the stored dict ID doesn't match the loaded dictionary (dictionary was swapped without retraining)
85
+
86
+
### Memory budget
87
+
88
+
- Each Compressor/Decompressor instance: ~128-256KB + dictionary copy
89
+
- Writer needs 1 compressor (reused per batch)
90
+
- Each replaying WebSocket connection needs 1 decompressor
91
+
92
+
## Migration Path
93
+
94
+
1. Deploy with `--zstd-dict-path` absent - no change to existing behavior
95
+
2. Run `ramjet_dictgen` against a live database to train dictionary
96
+
3. Redeploy with `--zstd-dict-path /path/to/dict`
97
+
4. New events and records are compressed; old data (compact binary, legacy JSON, uncompressed records) still readable via format detection
98
+
5. No data migration needed - mixed compressed/uncompressed data coexists in both keyspaces
99
+
100
+
## Expected Results
101
+
102
+
- Identity/account events (~50-80 bytes compact): ~2-3x compression with dictionary
103
+
- Commit events with record bodies (~200B-5KB compact): ~3-6x compression with dictionary
104
+
- CPU overhead: negligible at zstd level 3 for small payloads
+423
src/bin/ramjet_consumer.rs
+423
src/bin/ramjet_consumer.rs
···
1
+
//! Benchmark binary: connects to a ramjet instance's stream.subscribe
2
+
//! WebSocket endpoint and logs ingestion rate with per-collection
3
+
//! breakdown for pipeline analysis and publication baseline.
4
+
5
+
use std::collections::HashMap;
6
+
use std::sync::atomic::{AtomicU64, Ordering};
7
+
use std::sync::{Arc, Mutex};
8
+
use std::time::{Duration, Instant};
9
+
10
+
use clap::Parser;
11
+
use futures_util::StreamExt;
12
+
use tokio::sync::mpsc;
13
+
14
+
#[derive(Parser, Debug)]
15
+
#[command(
16
+
name = "ramjet-consumer",
17
+
about = "Benchmark fan-out consumer with per-collection stats"
18
+
)]
19
+
struct Args {
20
+
/// Ramjet host to connect to.
21
+
#[arg(long, default_value = "localhost:8080")]
22
+
host: String,
23
+
24
+
/// Cursor to resume from (omit to start live).
25
+
#[arg(long)]
26
+
cursor: Option<u64>,
27
+
28
+
/// Stats reporting interval in seconds.
29
+
#[arg(long, default_value = "10")]
30
+
stats_interval: u64,
31
+
32
+
/// Number of top collections to show in stats output.
33
+
#[arg(long, default_value = "10")]
34
+
top_collections: usize,
35
+
36
+
/// Sample rate for printing events to stdout (0.0 = none, 1.0 = all).
37
+
#[arg(long)]
38
+
sample: Option<f64>,
39
+
}
40
+
41
+
/// Minimal event struct for fast CBOR deserialization — skips record data.
42
+
#[derive(serde::Deserialize)]
43
+
struct MinimalEvent {
44
+
#[serde(default)]
45
+
kind: Option<String>,
46
+
#[serde(default)]
47
+
seq: Option<u64>,
48
+
#[serde(default)]
49
+
commit: Option<MinimalCommit>,
50
+
}
51
+
52
+
#[derive(serde::Deserialize)]
53
+
struct MinimalCommit {
54
+
#[serde(default)]
55
+
collection: Option<String>,
56
+
}
57
+
58
+
struct CollectionData {
59
+
interval: HashMap<String, u64>,
60
+
totals: HashMap<String, u64>,
61
+
}
62
+
63
+
struct Stats {
64
+
// Written by receive loop
65
+
events_total: AtomicU64,
66
+
bytes_received: AtomicU64,
67
+
68
+
// Written by processing task
69
+
commits_total: AtomicU64,
70
+
identity_total: AtomicU64,
71
+
account_total: AtomicU64,
72
+
unknown_total: AtomicU64,
73
+
last_seq: AtomicU64,
74
+
seq_gaps: AtomicU64,
75
+
seq_gap_events: AtomicU64,
76
+
process_drops: AtomicU64,
77
+
78
+
/// Single mutex for collection accounting.
79
+
collections: Mutex<CollectionData>,
80
+
}
81
+
82
+
impl Stats {
83
+
fn new() -> Self {
84
+
Self {
85
+
events_total: AtomicU64::new(0),
86
+
commits_total: AtomicU64::new(0),
87
+
identity_total: AtomicU64::new(0),
88
+
account_total: AtomicU64::new(0),
89
+
unknown_total: AtomicU64::new(0),
90
+
bytes_received: AtomicU64::new(0),
91
+
last_seq: AtomicU64::new(0),
92
+
seq_gaps: AtomicU64::new(0),
93
+
seq_gap_events: AtomicU64::new(0),
94
+
process_drops: AtomicU64::new(0),
95
+
collections: Mutex::new(CollectionData {
96
+
interval: HashMap::new(),
97
+
totals: HashMap::new(),
98
+
}),
99
+
}
100
+
}
101
+
102
+
fn take_interval_counts(&self) -> HashMap<String, u64> {
103
+
let mut data = self.collections.lock().unwrap();
104
+
std::mem::take(&mut data.interval)
105
+
}
106
+
107
+
fn total_collections(&self) -> usize {
108
+
self.collections.lock().unwrap().totals.len()
109
+
}
110
+
}
111
+
112
+
#[tokio::main]
113
+
async fn main() -> anyhow::Result<()> {
114
+
let args = Args::parse();
115
+
116
+
tracing_subscriber::fmt()
117
+
.with_env_filter(
118
+
tracing_subscriber::EnvFilter::try_from_default_env()
119
+
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
120
+
)
121
+
.init();
122
+
123
+
tracing::info!(
124
+
host = %args.host,
125
+
cursor = ?args.cursor,
126
+
stats_interval = args.stats_interval,
127
+
top_collections = args.top_collections,
128
+
"starting ramjet-consumer benchmark"
129
+
);
130
+
131
+
let stats = Arc::new(Stats::new());
132
+
133
+
// Processing channel — receive loop sends raw payloads, processing task handles parsing
134
+
let (process_tx, process_rx) = mpsc::channel::<Vec<u8>>(8192);
135
+
136
+
// Spawn processing task
137
+
tokio::spawn({
138
+
let stats = stats.clone();
139
+
let sample_rate = args.sample.unwrap_or(0.0);
140
+
async move {
141
+
process_events(process_rx, stats, sample_rate).await;
142
+
}
143
+
});
144
+
145
+
// Spawn stats reporter
146
+
let _stats_handle = tokio::spawn({
147
+
let stats = stats.clone();
148
+
let interval = Duration::from_secs(args.stats_interval);
149
+
let top_n = args.top_collections;
150
+
async move {
151
+
report_stats_loop(stats, interval, top_n).await;
152
+
}
153
+
});
154
+
155
+
// Connect and consume, using last_seq as cursor on reconnection
156
+
let mut cursor = args.cursor;
157
+
loop {
158
+
if let Err(e) = receive_loop(&args.host, cursor, &stats, &process_tx).await {
159
+
tracing::warn!(error = %e, "consumer disconnected, reconnecting in 2s");
160
+
} else {
161
+
tracing::info!("connection closed, reconnecting in 2s");
162
+
}
163
+
// Use last received seq as cursor for reconnection
164
+
let last = stats.last_seq.load(Ordering::Relaxed);
165
+
if last > 0 {
166
+
cursor = Some(last);
167
+
}
168
+
tokio::time::sleep(Duration::from_secs(2)).await;
169
+
}
170
+
171
+
#[allow(unreachable_code)]
172
+
Ok(())
173
+
}
174
+
175
+
/// Lightweight receive loop — counts bytes/events and forwards raw payloads
176
+
/// to the processing task. No JSON parsing happens here.
177
+
async fn receive_loop(
178
+
host: &str,
179
+
cursor: Option<u64>,
180
+
stats: &Stats,
181
+
process_tx: &mpsc::Sender<Vec<u8>>,
182
+
) -> anyhow::Result<()> {
183
+
let url = if let Some(cursor) = cursor {
184
+
format!("ws://{host}/xrpc/dev.ngerakines.ramjet.stream.subscribe?cursor={cursor}")
185
+
} else {
186
+
format!("ws://{host}/xrpc/dev.ngerakines.ramjet.stream.subscribe")
187
+
};
188
+
189
+
tracing::info!(%url, cursor = ?cursor, "connecting to ramjet");
190
+
191
+
let uri: http::Uri = url.parse()?;
192
+
let (mut ws, _response) = tokio_websockets::ClientBuilder::from_uri(uri)
193
+
.connect()
194
+
.await?;
195
+
196
+
tracing::info!("connected");
197
+
198
+
let mut last_event_at = Instant::now();
199
+
let mut silence_warned = false;
200
+
201
+
loop {
202
+
let deadline = if silence_warned {
203
+
last_event_at + Duration::from_secs(10 * 60)
204
+
} else {
205
+
last_event_at + Duration::from_secs(5 * 60)
206
+
};
207
+
208
+
tokio::select! {
209
+
biased;
210
+
211
+
_ = tokio::time::sleep_until(tokio::time::Instant::from_std(deadline)) => {
212
+
if !silence_warned {
213
+
tracing::warn!(
214
+
silence_secs = last_event_at.elapsed().as_secs(),
215
+
"no events received for 5 minutes, will exit in 5 minutes",
216
+
);
217
+
silence_warned = true;
218
+
} else {
219
+
anyhow::bail!(
220
+
"no events received for 10 minutes, exiting"
221
+
);
222
+
}
223
+
}
224
+
225
+
msg = ws.next() => {
226
+
let Some(result) = msg else {
227
+
return Ok(());
228
+
};
229
+
let message = result?;
230
+
231
+
if !message.is_binary() {
232
+
continue;
233
+
}
234
+
235
+
last_event_at = Instant::now();
236
+
silence_warned = false;
237
+
238
+
let payload = message.into_payload();
239
+
let data: &[u8] = payload.as_ref();
240
+
241
+
stats
242
+
.bytes_received
243
+
.fetch_add(data.len() as u64, Ordering::Relaxed);
244
+
stats.events_total.fetch_add(1, Ordering::Relaxed);
245
+
246
+
if process_tx.try_send(data.to_vec()).is_err() {
247
+
stats.process_drops.fetch_add(1, Ordering::Relaxed);
248
+
}
249
+
}
250
+
}
251
+
}
252
+
}
253
+
254
+
/// Processing task — receives raw payloads from the channel, does minimal
255
+
/// CBOR deserialization (skipping record data), and updates stats.
256
+
async fn process_events(mut rx: mpsc::Receiver<Vec<u8>>, stats: Arc<Stats>, sample_rate: f64) {
257
+
let mut prev_seq: Option<u64> = None;
258
+
259
+
while let Some(data) = rx.recv().await {
260
+
// Minimal deserialization — only extracts kind, seq, commit.collection
261
+
let Ok(event) = atproto_dasl::drisl::from_slice::<MinimalEvent>(data.as_slice()) else {
262
+
stats.unknown_total.fetch_add(1, Ordering::Relaxed);
263
+
continue;
264
+
};
265
+
266
+
// Sequence tracking and gap detection
267
+
if let Some(seq) = event.seq {
268
+
if let Some(prev) = prev_seq {
269
+
let gap = seq.saturating_sub(prev);
270
+
if gap > 1 {
271
+
stats.seq_gaps.fetch_add(1, Ordering::Relaxed);
272
+
stats.seq_gap_events.fetch_add(gap - 1, Ordering::Relaxed);
273
+
}
274
+
}
275
+
prev_seq = Some(seq);
276
+
stats.last_seq.store(seq, Ordering::Relaxed);
277
+
}
278
+
279
+
// Optional sampling — full parse to Ipld for printing
280
+
if sample_rate > 0.0 {
281
+
use rand::RngExt;
282
+
let should_print =
283
+
sample_rate >= 1.0 || rand::rng().random_range(0.0..1.0) < sample_rate;
284
+
if should_print {
285
+
if let Ok(full) =
286
+
atproto_dasl::drisl::from_slice::<atproto_dasl::Ipld>(data.as_slice())
287
+
{
288
+
print_event(&full);
289
+
}
290
+
}
291
+
}
292
+
293
+
match event.kind.as_deref() {
294
+
Some("commit") => {
295
+
stats.commits_total.fetch_add(1, Ordering::Relaxed);
296
+
if let Some(commit) = &event.commit {
297
+
if let Some(collection) = &commit.collection {
298
+
let mut collections = stats.collections.lock().unwrap();
299
+
*collections.interval.entry(collection.clone()).or_default() += 1;
300
+
*collections.totals.entry(collection.clone()).or_default() += 1;
301
+
}
302
+
}
303
+
}
304
+
Some("identity") => {
305
+
stats.identity_total.fetch_add(1, Ordering::Relaxed);
306
+
}
307
+
Some("account") => {
308
+
stats.account_total.fetch_add(1, Ordering::Relaxed);
309
+
}
310
+
_ => {
311
+
stats.unknown_total.fetch_add(1, Ordering::Relaxed);
312
+
}
313
+
}
314
+
}
315
+
}
316
+
317
+
async fn report_stats_loop(stats: Arc<Stats>, interval: Duration, top_n: usize) {
318
+
let mut ticker = tokio::time::interval(interval);
319
+
ticker.tick().await; // skip first immediate tick
320
+
321
+
let mut prev_events: u64 = 0;
322
+
let mut prev_commits: u64 = 0;
323
+
let mut prev_bytes: u64 = 0;
324
+
let mut prev_time = Instant::now();
325
+
326
+
loop {
327
+
ticker.tick().await;
328
+
329
+
let now = Instant::now();
330
+
let elapsed = now.duration_since(prev_time).as_secs_f64();
331
+
prev_time = now;
332
+
333
+
let events = stats.events_total.load(Ordering::Relaxed);
334
+
let commits = stats.commits_total.load(Ordering::Relaxed);
335
+
let identity = stats.identity_total.load(Ordering::Relaxed);
336
+
let account = stats.account_total.load(Ordering::Relaxed);
337
+
let unknown = stats.unknown_total.load(Ordering::Relaxed);
338
+
let bytes = stats.bytes_received.load(Ordering::Relaxed);
339
+
let last_seq = stats.last_seq.load(Ordering::Relaxed);
340
+
let seq_gaps = stats.seq_gaps.load(Ordering::Relaxed);
341
+
let seq_gap_events = stats.seq_gap_events.load(Ordering::Relaxed);
342
+
let process_drops = stats.process_drops.load(Ordering::Relaxed);
343
+
344
+
let d_events = events - prev_events;
345
+
let d_commits = commits - prev_commits;
346
+
let d_bytes = bytes - prev_bytes;
347
+
348
+
prev_events = events;
349
+
prev_commits = commits;
350
+
prev_bytes = bytes;
351
+
352
+
let event_rate = d_events as f64 / elapsed;
353
+
let commit_rate = d_commits as f64 / elapsed;
354
+
let byte_rate = d_bytes as f64 / elapsed;
355
+
356
+
// Get and format top collections for this interval
357
+
let interval_counts = stats.take_interval_counts();
358
+
let top_collections = top_n_collections(&interval_counts, top_n);
359
+
let total_collections = stats.total_collections();
360
+
361
+
tracing::info!(
362
+
events_per_sec = format!("{event_rate:.0}"),
363
+
commits_per_sec = format!("{commit_rate:.0}"),
364
+
throughput = format!("{}/s", format_bytes(byte_rate as u64)),
365
+
interval_events = d_events,
366
+
interval_commits = d_commits,
367
+
total_events = events,
368
+
total_commits = commits,
369
+
total_identity = identity,
370
+
total_account = account,
371
+
total_unknown = unknown,
372
+
total_bytes = format_bytes(bytes),
373
+
last_seq = last_seq,
374
+
unique_collections = total_collections,
375
+
seq_gaps = seq_gaps,
376
+
seq_gap_events = seq_gap_events,
377
+
process_drops = process_drops,
378
+
"stats"
379
+
);
380
+
381
+
if !top_collections.is_empty() {
382
+
for (collection, count) in &top_collections {
383
+
let rate = *count as f64 / elapsed;
384
+
tracing::info!(
385
+
collection = %collection,
386
+
ops = count,
387
+
ops_per_sec = format!("{rate:.1}"),
388
+
" collection"
389
+
);
390
+
}
391
+
}
392
+
}
393
+
}
394
+
395
+
fn print_event(event: &atproto_dasl::Ipld) {
396
+
match serde_json::to_string(event) {
397
+
Ok(json) => println!("{json}"),
398
+
Err(_) => println!("{event:?}"),
399
+
}
400
+
}
401
+
402
+
fn top_n_collections(counts: &HashMap<String, u64>, n: usize) -> Vec<(String, u64)> {
403
+
let mut entries: Vec<(String, u64)> = counts.iter().map(|(k, v)| (k.clone(), *v)).collect();
404
+
entries.sort_by(|a, b| b.1.cmp(&a.1));
405
+
entries.truncate(n);
406
+
entries
407
+
}
408
+
409
+
fn format_bytes(bytes: u64) -> String {
410
+
const KIB: u64 = 1024;
411
+
const MIB: u64 = 1024 * KIB;
412
+
const GIB: u64 = 1024 * MIB;
413
+
414
+
if bytes >= GIB {
415
+
format!("{:.2} GiB", bytes as f64 / GIB as f64)
416
+
} else if bytes >= MIB {
417
+
format!("{:.2} MiB", bytes as f64 / MIB as f64)
418
+
} else if bytes >= KIB {
419
+
format!("{:.2} KiB", bytes as f64 / KIB as f64)
420
+
} else {
421
+
format!("{bytes} B")
422
+
}
423
+
}
+685
src/bin/ramjet_data.rs
+685
src/bin/ramjet_data.rs
···
1
+
//! Interactive TUI for exploring a ramjet fjall database.
2
+
//!
3
+
//! Overview screen shows database metrics and keyspace list.
4
+
//! Selecting a keyspace opens a key browser with live value preview.
5
+
6
+
use std::path::PathBuf;
7
+
8
+
use clap::Parser;
9
+
use crossterm::event::{self, Event, KeyCode, KeyEventKind};
10
+
use crossterm::terminal::{disable_raw_mode, enable_raw_mode};
11
+
use ratatui::DefaultTerminal;
12
+
use ratatui::layout::{Constraint, Direction, Layout};
13
+
use ratatui::prelude::Stylize;
14
+
use ratatui::style::{Color, Modifier, Style};
15
+
use ratatui::text::{Line, Span, Text};
16
+
use ratatui::widgets::{Block, Borders, List, ListItem, ListState, Paragraph, Wrap};
17
+
18
+
use ramjet::storage::FjallDb;
19
+
20
+
#[derive(Parser)]
21
+
#[command(name = "ramjet-data", about = "Interactive database explorer")]
22
+
struct Args {
23
+
/// Path to the fjall database directory.
24
+
#[arg(long, env = "RAMJET_DB_PATH", default_value = "./data/ramjet.db")]
25
+
db_path: PathBuf,
26
+
}
27
+
28
+
/// Names of all 8 keyspaces, matching FjallDb field order.
29
+
const KEYSPACE_NAMES: [&str; 8] = [
30
+
"records",
31
+
"events",
32
+
"meta",
33
+
"repo_state",
34
+
"did_to_doc",
35
+
"handle_to_did",
36
+
"blobs",
37
+
"blob_meta",
38
+
];
39
+
40
+
struct App {
41
+
db: FjallDb,
42
+
db_path: PathBuf,
43
+
screen: Screen,
44
+
// Overview
45
+
overview_state: ListState,
46
+
keyspace_stats: Vec<KeyspaceInfo>,
47
+
disk_size: u64,
48
+
// Explorer
49
+
explorer: ExplorerState,
50
+
}
51
+
52
+
#[derive(PartialEq)]
53
+
enum Screen {
54
+
Overview,
55
+
Explorer,
56
+
}
57
+
58
+
struct KeyspaceInfo {
59
+
name: String,
60
+
count: u64,
61
+
exact: bool,
62
+
}
63
+
64
+
struct ExplorerState {
65
+
keyspace_name: String,
66
+
entries: Vec<(Vec<u8>, Vec<u8>)>,
67
+
list_state: ListState,
68
+
scroll_offset: u16,
69
+
}
70
+
71
+
impl ExplorerState {
72
+
fn new() -> Self {
73
+
Self {
74
+
keyspace_name: String::new(),
75
+
entries: Vec::new(),
76
+
list_state: ListState::default(),
77
+
scroll_offset: 0,
78
+
}
79
+
}
80
+
81
+
fn selected_value(&self) -> Option<&[u8]> {
82
+
self.list_state
83
+
.selected()
84
+
.and_then(|i| self.entries.get(i))
85
+
.map(|(_, v)| v.as_slice())
86
+
}
87
+
88
+
fn selected_key(&self) -> Option<&[u8]> {
89
+
self.list_state
90
+
.selected()
91
+
.and_then(|i| self.entries.get(i))
92
+
.map(|(k, _)| k.as_slice())
93
+
}
94
+
}
95
+
96
+
impl App {
97
+
fn new(db: FjallDb, db_path: PathBuf) -> Self {
98
+
let mut app = Self {
99
+
db,
100
+
db_path,
101
+
screen: Screen::Overview,
102
+
overview_state: ListState::default(),
103
+
keyspace_stats: Vec::new(),
104
+
disk_size: 0,
105
+
explorer: ExplorerState::new(),
106
+
};
107
+
app.overview_state.select(Some(0));
108
+
app.refresh_overview();
109
+
app
110
+
}
111
+
112
+
fn refresh_overview(&mut self) {
113
+
self.keyspace_stats = KEYSPACE_NAMES
114
+
.iter()
115
+
.map(|name| {
116
+
let ks = self.get_keyspace(name);
117
+
let (count, exact) = count_keyspace(ks);
118
+
KeyspaceInfo {
119
+
name: name.to_string(),
120
+
count,
121
+
exact,
122
+
}
123
+
})
124
+
.collect();
125
+
self.disk_size = dir_size(&self.db_path);
126
+
}
127
+
128
+
fn get_keyspace(&self, name: &str) -> &fjall::Keyspace {
129
+
match name {
130
+
"records" => &self.db.records,
131
+
"events" => &self.db.events,
132
+
"meta" => &self.db.meta,
133
+
"repo_state" => &self.db.repo_state,
134
+
"did_to_doc" => &self.db.did_to_doc,
135
+
"handle_to_did" => &self.db.handle_to_did,
136
+
"blobs" => &self.db.blobs,
137
+
"blob_meta" => &self.db.blob_meta,
138
+
_ => &self.db.meta,
139
+
}
140
+
}
141
+
142
+
fn enter_explorer(&mut self) {
143
+
let Some(idx) = self.overview_state.selected() else {
144
+
return;
145
+
};
146
+
let Some(info) = self.keyspace_stats.get(idx) else {
147
+
return;
148
+
};
149
+
150
+
self.explorer.keyspace_name = info.name.clone();
151
+
self.explorer.scroll_offset = 0;
152
+
153
+
let ks = self.get_keyspace(&info.name);
154
+
let mut entries = Vec::new();
155
+
for guard in ks.iter() {
156
+
let Ok((k, v)) = guard.into_inner() else {
157
+
continue;
158
+
};
159
+
let key: &[u8] = &k;
160
+
let val: &[u8] = &v;
161
+
entries.push((key.to_vec(), val.to_vec()));
162
+
if entries.len() >= 50_000 {
163
+
break;
164
+
}
165
+
}
166
+
self.explorer.entries = entries;
167
+
self.explorer.list_state = ListState::default();
168
+
if !self.explorer.entries.is_empty() {
169
+
self.explorer.list_state.select(Some(0));
170
+
}
171
+
172
+
self.screen = Screen::Explorer;
173
+
}
174
+
}
175
+
176
+
fn main() -> anyhow::Result<()> {
177
+
let args = Args::parse();
178
+
let db = FjallDb::open(&args.db_path, None)?;
179
+
let mut app = App::new(db, args.db_path);
180
+
181
+
enable_raw_mode()?;
182
+
let mut terminal = ratatui::init();
183
+
let result = run_app(&mut terminal, &mut app);
184
+
ratatui::restore();
185
+
disable_raw_mode()?;
186
+
187
+
result
188
+
}
189
+
190
+
fn run_app(terminal: &mut DefaultTerminal, app: &mut App) -> anyhow::Result<()> {
191
+
loop {
192
+
terminal.draw(|frame| match app.screen {
193
+
Screen::Overview => draw_overview(frame, app),
194
+
Screen::Explorer => draw_explorer(frame, app),
195
+
})?;
196
+
197
+
if let Event::Key(key) = event::read()? {
198
+
if key.kind != KeyEventKind::Press {
199
+
continue;
200
+
}
201
+
202
+
match app.screen {
203
+
Screen::Overview => match key.code {
204
+
KeyCode::Char('q') => return Ok(()),
205
+
KeyCode::Up => {
206
+
let i = app.overview_state.selected().unwrap_or(0);
207
+
if i > 0 {
208
+
app.overview_state.select(Some(i - 1));
209
+
}
210
+
}
211
+
KeyCode::Down => {
212
+
let i = app.overview_state.selected().unwrap_or(0);
213
+
if i + 1 < KEYSPACE_NAMES.len() {
214
+
app.overview_state.select(Some(i + 1));
215
+
}
216
+
}
217
+
KeyCode::Enter => {
218
+
app.enter_explorer();
219
+
}
220
+
KeyCode::Char('r') => {
221
+
app.refresh_overview();
222
+
}
223
+
_ => {}
224
+
},
225
+
Screen::Explorer => match key.code {
226
+
KeyCode::Esc => {
227
+
app.refresh_overview();
228
+
app.screen = Screen::Overview;
229
+
}
230
+
KeyCode::Up => {
231
+
let i = app.explorer.list_state.selected().unwrap_or(0);
232
+
if i > 0 {
233
+
app.explorer.list_state.select(Some(i - 1));
234
+
app.explorer.scroll_offset = 0;
235
+
}
236
+
}
237
+
KeyCode::Down => {
238
+
let i = app.explorer.list_state.selected().unwrap_or(0);
239
+
if i + 1 < app.explorer.entries.len() {
240
+
app.explorer.list_state.select(Some(i + 1));
241
+
app.explorer.scroll_offset = 0;
242
+
}
243
+
}
244
+
KeyCode::PageUp => {
245
+
let i = app.explorer.list_state.selected().unwrap_or(0);
246
+
let new_i = i.saturating_sub(20);
247
+
app.explorer.list_state.select(Some(new_i));
248
+
app.explorer.scroll_offset = 0;
249
+
}
250
+
KeyCode::PageDown => {
251
+
let i = app.explorer.list_state.selected().unwrap_or(0);
252
+
let max = app.explorer.entries.len().saturating_sub(1);
253
+
let new_i = (i + 20).min(max);
254
+
app.explorer.list_state.select(Some(new_i));
255
+
app.explorer.scroll_offset = 0;
256
+
}
257
+
KeyCode::Home => {
258
+
if !app.explorer.entries.is_empty() {
259
+
app.explorer.list_state.select(Some(0));
260
+
app.explorer.scroll_offset = 0;
261
+
}
262
+
}
263
+
KeyCode::End => {
264
+
if !app.explorer.entries.is_empty() {
265
+
app.explorer
266
+
.list_state
267
+
.select(Some(app.explorer.entries.len() - 1));
268
+
app.explorer.scroll_offset = 0;
269
+
}
270
+
}
271
+
KeyCode::Char('j') => {
272
+
app.explorer.scroll_offset = app.explorer.scroll_offset.saturating_add(1);
273
+
}
274
+
KeyCode::Char('k') => {
275
+
app.explorer.scroll_offset = app.explorer.scroll_offset.saturating_sub(1);
276
+
}
277
+
_ => {}
278
+
},
279
+
}
280
+
}
281
+
}
282
+
}
283
+
284
+
fn draw_overview(frame: &mut ratatui::Frame, app: &mut App) {
285
+
let chunks = Layout::default()
286
+
.direction(Direction::Vertical)
287
+
.constraints([Constraint::Length(7), Constraint::Min(5)])
288
+
.split(frame.area());
289
+
290
+
// Database info panel
291
+
let cursor = app
292
+
.db
293
+
.get_cursor()
294
+
.ok()
295
+
.flatten()
296
+
.map(|c| c.to_string())
297
+
.unwrap_or_else(|| "none".to_string());
298
+
299
+
let event_seq = app.db.current_sequence().to_string();
300
+
301
+
let total_keys: u64 = app.keyspace_stats.iter().map(|ks| ks.count).sum();
302
+
let any_inexact = app.keyspace_stats.iter().any(|ks| !ks.exact);
303
+
let keys_prefix = if any_inexact { ">" } else { "" };
304
+
305
+
let info_text = Text::from(vec![
306
+
Line::from(vec![
307
+
Span::styled("Path: ", Style::default().fg(Color::DarkGray)),
308
+
Span::raw(app.db_path.display().to_string()),
309
+
]),
310
+
Line::from(vec![
311
+
Span::styled("Disk: ", Style::default().fg(Color::DarkGray)),
312
+
Span::raw(format_bytes(app.disk_size)),
313
+
Span::raw(" "),
314
+
Span::styled("Keys: ", Style::default().fg(Color::DarkGray)),
315
+
Span::raw(format!("{keys_prefix}{}", format_count(total_keys))),
316
+
]),
317
+
Line::from(vec![
318
+
Span::styled("Cursor: ", Style::default().fg(Color::DarkGray)),
319
+
Span::raw(cursor),
320
+
Span::raw(" "),
321
+
Span::styled("Event seq: ", Style::default().fg(Color::DarkGray)),
322
+
Span::raw(event_seq),
323
+
]),
324
+
Line::from(""),
325
+
Line::from(Span::styled(
326
+
"[↑↓] navigate [Enter] explore [r] refresh [q] quit",
327
+
Style::default().fg(Color::DarkGray),
328
+
)),
329
+
]);
330
+
331
+
let info_block = Paragraph::new(info_text).block(
332
+
Block::default()
333
+
.borders(Borders::ALL)
334
+
.title(" Database Overview "),
335
+
);
336
+
frame.render_widget(info_block, chunks[0]);
337
+
338
+
// Keyspace list
339
+
let items: Vec<ListItem> = app
340
+
.keyspace_stats
341
+
.iter()
342
+
.map(|ks| {
343
+
let count_str = format_count(ks.count);
344
+
let prefix = if ks.exact { "" } else { ">" };
345
+
ListItem::new(Line::from(vec![
346
+
Span::styled(
347
+
format!("{:<15}", ks.name),
348
+
Style::default().add_modifier(Modifier::BOLD),
349
+
),
350
+
Span::styled(
351
+
format!("{prefix}{count_str:>12} keys"),
352
+
Style::default().fg(Color::Cyan),
353
+
),
354
+
]))
355
+
})
356
+
.collect();
357
+
358
+
let list = List::new(items)
359
+
.block(Block::default().borders(Borders::ALL).title(" Keyspaces "))
360
+
.highlight_style(
361
+
Style::default()
362
+
.bg(Color::DarkGray)
363
+
.add_modifier(Modifier::BOLD),
364
+
)
365
+
.highlight_symbol("▸ ");
366
+
367
+
frame.render_stateful_widget(list, chunks[1], &mut app.overview_state);
368
+
}
369
+
370
+
fn draw_explorer(frame: &mut ratatui::Frame, app: &mut App) {
371
+
let chunks = Layout::default()
372
+
.direction(Direction::Horizontal)
373
+
.constraints([Constraint::Percentage(35), Constraint::Percentage(65)])
374
+
.split(frame.area());
375
+
376
+
// Key list (left panel)
377
+
let items: Vec<ListItem> = app
378
+
.explorer
379
+
.entries
380
+
.iter()
381
+
.map(|(k, v)| {
382
+
let key_display = format_key_display(&app.explorer.keyspace_name, k);
383
+
let size_str = format_bytes_short(v.len() as u64);
384
+
ListItem::new(Line::from(vec![
385
+
Span::raw(key_display),
386
+
Span::styled(
387
+
format!(" ({size_str})"),
388
+
Style::default().fg(Color::DarkGray),
389
+
),
390
+
]))
391
+
})
392
+
.collect();
393
+
394
+
let position = app.explorer.list_state.selected().unwrap_or(0);
395
+
let total = app.explorer.entries.len();
396
+
let key_title = format!(
397
+
" {} [{}/{total}] ",
398
+
app.explorer.keyspace_name,
399
+
position + 1
400
+
);
401
+
402
+
let list = List::new(items)
403
+
.block(
404
+
Block::default()
405
+
.borders(Borders::ALL)
406
+
.title(key_title)
407
+
.title_bottom(
408
+
Line::from(
409
+
" [Esc] back [↑↓ PgUp PgDn Home End] navigate [j/k] scroll value ",
410
+
)
411
+
.fg(Color::DarkGray),
412
+
),
413
+
)
414
+
.highlight_style(
415
+
Style::default()
416
+
.bg(Color::DarkGray)
417
+
.add_modifier(Modifier::BOLD),
418
+
)
419
+
.highlight_symbol("▸ ");
420
+
421
+
frame.render_stateful_widget(list, chunks[0], &mut app.explorer.list_state);
422
+
423
+
// Value preview (right panel)
424
+
let value_text = if let (Some(key), Some(value)) =
425
+
(app.explorer.selected_key(), app.explorer.selected_value())
426
+
{
427
+
format_value_preview(&app.explorer.keyspace_name, key, value)
428
+
} else {
429
+
"No entry selected".to_string()
430
+
};
431
+
432
+
let value_paragraph = Paragraph::new(value_text)
433
+
.block(
434
+
Block::default()
435
+
.borders(Borders::ALL)
436
+
.title(" Value Preview "),
437
+
)
438
+
.wrap(Wrap { trim: false })
439
+
.scroll((app.explorer.scroll_offset, 0));
440
+
441
+
frame.render_widget(value_paragraph, chunks[1]);
442
+
}
443
+
444
+
/// Format a key for display based on which keyspace it belongs to.
445
+
fn format_key_display(keyspace: &str, key: &[u8]) -> String {
446
+
match keyspace {
447
+
"records" => {
448
+
if let Ok((did, coll, rkey, rev)) = ramjet::storage::keys::decode_record_key(key) {
449
+
return format!("{did} / {coll} / {rkey} @ {rev}");
450
+
}
451
+
}
452
+
"events" => {
453
+
if let Ok(seq) = ramjet::storage::keys::decode_event_key(key) {
454
+
return format!("seq:{seq}");
455
+
}
456
+
}
457
+
_ => {}
458
+
}
459
+
460
+
// Fallback: try UTF-8, then hex
461
+
if let Ok(s) = std::str::from_utf8(key) {
462
+
if s.chars().all(|c| !c.is_control()) {
463
+
return s.to_string();
464
+
}
465
+
}
466
+
format_hex(key)
467
+
}
468
+
469
+
/// Format a value for the preview pane with keyspace-aware decoding.
470
+
fn format_value_preview(keyspace: &str, key: &[u8], value: &[u8]) -> String {
471
+
let mut out = String::new();
472
+
473
+
// Key info header
474
+
out.push_str(&format!("Key: {}\n", format_key_display(keyspace, key)));
475
+
out.push_str(&format!(
476
+
"Size: {} ({} bytes)\n",
477
+
format_bytes(value.len() as u64),
478
+
value.len()
479
+
));
480
+
out.push_str("───────────────────────────────────\n");
481
+
482
+
// Keyspace-specific decoding
483
+
match keyspace {
484
+
"records" => {
485
+
if ramjet::types::RecordValue::is_tombstone(value) {
486
+
out.push_str("TOMBSTONE (deleted record)\n");
487
+
return out;
488
+
}
489
+
if let Ok(rv) = ramjet::types::RecordValue::decode(value) {
490
+
out.push_str(&format!("CID: {}\n", rv.cid_string()));
491
+
out.push_str(&format!("Data: {} bytes\n\n", rv.data.len()));
492
+
if !rv.data.is_empty() {
493
+
// Try DAG-CBOR → JSON
494
+
if let Ok(val) = atproto_dasl::from_slice::<serde_json::Value>(&rv.data) {
495
+
if let Ok(pretty) = serde_json::to_string_pretty(&val) {
496
+
out.push_str(&pretty);
497
+
return out;
498
+
}
499
+
}
500
+
// Try plain JSON
501
+
if let Ok(val) = serde_json::from_slice::<serde_json::Value>(&rv.data) {
502
+
if let Ok(pretty) = serde_json::to_string_pretty(&val) {
503
+
out.push_str(&pretty);
504
+
return out;
505
+
}
506
+
}
507
+
out.push_str(&format_hex_block(&rv.data));
508
+
}
509
+
return out;
510
+
}
511
+
}
512
+
"repo_state" => {
513
+
if let Ok(rs) = ramjet::types::RepoState::decode(value) {
514
+
out.push_str(&format!("Rev: {}\n", rs.rev));
515
+
out.push_str(&format!("Status: {}\n", rs.status.as_str()));
516
+
out.push_str(&format!("Denied: {}\n", rs.denied));
517
+
return out;
518
+
}
519
+
}
520
+
"events" => {
521
+
// JSON values
522
+
if let Ok(val) = serde_json::from_slice::<serde_json::Value>(value) {
523
+
if let Ok(pretty) = serde_json::to_string_pretty(&val) {
524
+
out.push_str(&pretty);
525
+
return out;
526
+
}
527
+
}
528
+
}
529
+
"did_to_doc" => {
530
+
// Timestamped DID document: [8B BE timestamp][JSON]
531
+
let (ts, json_bytes) = ramjet::storage::encoding::decode_timestamped_doc(value);
532
+
if ts > 0 {
533
+
out.push_str(&format!("updated_at: {ts} (unix secs)\n\n"));
534
+
}
535
+
if let Ok(val) = serde_json::from_slice::<serde_json::Value>(json_bytes) {
536
+
if let Ok(pretty) = serde_json::to_string_pretty(&val) {
537
+
out.push_str(&pretty);
538
+
return out;
539
+
}
540
+
}
541
+
}
542
+
"handle_to_did" => {
543
+
if let Ok(s) = std::str::from_utf8(value) {
544
+
out.push_str(s);
545
+
return out;
546
+
}
547
+
}
548
+
"meta" => {
549
+
// Try as u64 BE
550
+
if value.len() == 8 {
551
+
let val = u64::from_be_bytes(value.try_into().unwrap());
552
+
out.push_str(&format!("{val} (u64)\n"));
553
+
return out;
554
+
}
555
+
// Try as UTF-8
556
+
if let Ok(s) = std::str::from_utf8(value) {
557
+
if !s.is_empty() && s.chars().all(|c| !c.is_control()) {
558
+
out.push_str(s);
559
+
return out;
560
+
}
561
+
}
562
+
// Try as JSON
563
+
if let Ok(val) = serde_json::from_slice::<serde_json::Value>(value) {
564
+
if let Ok(pretty) = serde_json::to_string_pretty(&val) {
565
+
out.push_str(&pretty);
566
+
return out;
567
+
}
568
+
}
569
+
}
570
+
_ => {}
571
+
}
572
+
573
+
// Fallback: hex dump
574
+
out.push_str(&format_hex_block(value));
575
+
out
576
+
}
577
+
578
+
fn format_hex(bytes: &[u8]) -> String {
579
+
bytes.iter().map(|b| format!("{b:02x}")).collect()
580
+
}
581
+
582
+
fn format_hex_block(bytes: &[u8]) -> String {
583
+
let mut out = String::new();
584
+
for (i, chunk) in bytes.chunks(16).enumerate() {
585
+
let offset = i * 16;
586
+
out.push_str(&format!("{offset:08x} "));
587
+
for (j, b) in chunk.iter().enumerate() {
588
+
out.push_str(&format!("{b:02x} "));
589
+
if j == 7 {
590
+
out.push(' ');
591
+
}
592
+
}
593
+
// Pad if short
594
+
let pad = 16 - chunk.len();
595
+
for _ in 0..pad {
596
+
out.push_str(" ");
597
+
}
598
+
if chunk.len() <= 8 {
599
+
out.push(' ');
600
+
}
601
+
out.push_str(" |");
602
+
for b in chunk {
603
+
if b.is_ascii_graphic() || *b == b' ' {
604
+
out.push(*b as char);
605
+
} else {
606
+
out.push('.');
607
+
}
608
+
}
609
+
out.push_str("|\n");
610
+
}
611
+
out
612
+
}
613
+
614
+
/// Returns (count, exact). If the keyspace has more than 1M keys,
615
+
/// counting stops early and `exact` is false.
616
+
fn count_keyspace(ks: &fjall::Keyspace) -> (u64, bool) {
617
+
let limit: u64 = 1_000_000;
618
+
let mut count: u64 = 0;
619
+
for guard in ks.iter() {
620
+
if guard.into_inner().is_ok() {
621
+
count += 1;
622
+
}
623
+
if count >= limit {
624
+
return (count, false);
625
+
}
626
+
}
627
+
(count, true)
628
+
}
629
+
630
+
fn dir_size(path: &PathBuf) -> u64 {
631
+
let Ok(entries) = std::fs::read_dir(path) else {
632
+
return 0;
633
+
};
634
+
let mut total: u64 = 0;
635
+
for entry in entries.flatten() {
636
+
let Ok(meta) = entry.metadata() else {
637
+
continue;
638
+
};
639
+
if meta.is_dir() {
640
+
total += dir_size(&entry.path().to_path_buf());
641
+
} else {
642
+
total += meta.len();
643
+
}
644
+
}
645
+
total
646
+
}
647
+
648
+
fn format_bytes(bytes: u64) -> String {
649
+
const KIB: u64 = 1024;
650
+
const MIB: u64 = 1024 * KIB;
651
+
const GIB: u64 = 1024 * MIB;
652
+
653
+
if bytes >= GIB {
654
+
format!("{:.2} GiB", bytes as f64 / GIB as f64)
655
+
} else if bytes >= MIB {
656
+
format!("{:.2} MiB", bytes as f64 / MIB as f64)
657
+
} else if bytes >= KIB {
658
+
format!("{:.2} KiB", bytes as f64 / KIB as f64)
659
+
} else {
660
+
format!("{bytes} B")
661
+
}
662
+
}
663
+
664
+
fn format_bytes_short(bytes: u64) -> String {
665
+
const KIB: u64 = 1024;
666
+
const MIB: u64 = 1024 * KIB;
667
+
668
+
if bytes >= MIB {
669
+
format!("{:.1}M", bytes as f64 / MIB as f64)
670
+
} else if bytes >= KIB {
671
+
format!("{:.1}K", bytes as f64 / KIB as f64)
672
+
} else {
673
+
format!("{bytes}B")
674
+
}
675
+
}
676
+
677
+
fn format_count(n: u64) -> String {
678
+
if n >= 1_000_000 {
679
+
format!("{:.2}M", n as f64 / 1_000_000.0)
680
+
} else if n >= 1_000 {
681
+
format!("{:.1}K", n as f64 / 1_000.0)
682
+
} else {
683
+
n.to_string()
684
+
}
685
+
}
+121
src/bin/ramjet_dictgen.rs
+121
src/bin/ramjet_dictgen.rs
···
1
+
//! Zstd dictionary training tool for the events keyspace.
2
+
//!
3
+
//! Samples compact binary events from a fjall database and trains a zstd
4
+
//! dictionary using the COVER algorithm. Prints compression statistics
5
+
//! comparing dictionary-based vs standard compression.
6
+
7
+
use std::path::PathBuf;
8
+
9
+
use clap::Parser;
10
+
11
+
use ramjet::storage::FjallDb;
12
+
13
+
#[derive(Parser)]
14
+
#[command(
15
+
name = "ramjet-dictgen",
16
+
about = "Train a zstd dictionary from the events keyspace"
17
+
)]
18
+
struct Args {
19
+
/// Path to the fjall database directory.
20
+
#[arg(long)]
21
+
db_path: PathBuf,
22
+
23
+
/// Output dictionary file path.
24
+
#[arg(long)]
25
+
output: PathBuf,
26
+
27
+
/// Maximum number of events to sample.
28
+
#[arg(long, default_value = "50000")]
29
+
max_samples: usize,
30
+
31
+
/// Dictionary size in bytes.
32
+
#[arg(long, default_value = "65536")]
33
+
dict_size: usize,
34
+
}
35
+
36
+
fn main() -> anyhow::Result<()> {
37
+
let args = Args::parse();
38
+
39
+
let db = FjallDb::open(&args.db_path, None)?;
40
+
41
+
println!(
42
+
"Sampling up to {} events from {}...",
43
+
args.max_samples,
44
+
args.db_path.display()
45
+
);
46
+
47
+
let mut samples: Vec<Vec<u8>> = Vec::with_capacity(args.max_samples);
48
+
for guard in db.events.iter() {
49
+
let Ok((_key, value)) = guard.into_inner() else {
50
+
continue;
51
+
};
52
+
let bytes: &[u8] = &value;
53
+
samples.push(bytes.to_vec());
54
+
if samples.len() >= args.max_samples {
55
+
break;
56
+
}
57
+
}
58
+
59
+
if samples.is_empty() {
60
+
anyhow::bail!("no events found in the database");
61
+
}
62
+
63
+
println!("Collected {} samples", samples.len());
64
+
65
+
// Compute stats on raw samples
66
+
let total_raw: usize = samples.iter().map(|s| s.len()).sum();
67
+
let avg_raw = total_raw as f64 / samples.len() as f64;
68
+
69
+
println!("Average sample size: {avg_raw:.1} bytes");
70
+
println!("Total raw size: {total_raw} bytes");
71
+
println!();
72
+
73
+
// Train dictionary
74
+
println!(
75
+
"Training dictionary ({} bytes) with COVER algorithm...",
76
+
args.dict_size
77
+
);
78
+
let dict = zstd::dict::from_samples(&samples, args.dict_size)?;
79
+
println!("Dictionary trained: {} bytes", dict.len());
80
+
81
+
// Compress with dictionary and measure
82
+
let mut total_compressed_dict: usize = 0;
83
+
let mut total_compressed_plain: usize = 0;
84
+
85
+
let mut compressor_dict = zstd::bulk::Compressor::with_dictionary(3, &dict)?;
86
+
let mut compressor_plain = zstd::bulk::Compressor::new(3)?;
87
+
88
+
for sample in &samples {
89
+
let compressed = compressor_dict.compress(sample)?;
90
+
total_compressed_dict += compressed.len();
91
+
92
+
let compressed = compressor_plain.compress(sample)?;
93
+
total_compressed_plain += compressed.len();
94
+
}
95
+
96
+
let ratio_dict = total_raw as f64 / total_compressed_dict as f64;
97
+
let ratio_plain = total_raw as f64 / total_compressed_plain as f64;
98
+
99
+
println!();
100
+
println!("Compression Results ({} samples)", samples.len());
101
+
println!("-----------------------------------");
102
+
println!(
103
+
" Without dictionary: {} -> {} ({:.2}x)",
104
+
total_raw, total_compressed_plain, ratio_plain
105
+
);
106
+
println!(
107
+
" With dictionary: {} -> {} ({:.2}x)",
108
+
total_raw, total_compressed_dict, ratio_dict
109
+
);
110
+
println!(
111
+
" Dictionary gain: {:.2}x over plain zstd",
112
+
ratio_dict / ratio_plain
113
+
);
114
+
println!();
115
+
116
+
// Write dictionary
117
+
std::fs::write(&args.output, &dict)?;
118
+
println!("Dictionary written to {}", args.output.display());
119
+
120
+
Ok(())
121
+
}
+630
src/bin/ramjet_forecast.rs
+630
src/bin/ramjet_forecast.rs
···
1
+
//! Storage forecast tool for capacity planning.
2
+
//!
3
+
//! Analyzes a sample database to compute ingestion rates and project
4
+
//! storage requirements across multiple time horizons.
5
+
6
+
use std::collections::HashMap;
7
+
use std::path::PathBuf;
8
+
9
+
use clap::Parser;
10
+
11
+
use ramjet::storage::FjallDb;
12
+
use ramjet::storage::encoding::{self, decode_timestamped_doc};
13
+
use ramjet::storage::keys;
14
+
15
+
#[derive(Parser)]
16
+
#[command(name = "ramjet-forecast", about = "Storage capacity forecast tool")]
17
+
struct Args {
18
+
/// Path to the fjall database directory.
19
+
#[arg(long, env = "RAMJET_DB_PATH", default_value = "./data/ramjet.db")]
20
+
db_path: PathBuf,
21
+
22
+
/// Maximum number of keys to sample per keyspace (0 = unlimited).
23
+
#[arg(long, default_value = "0")]
24
+
sample_limit: u64,
25
+
}
26
+
27
+
fn main() -> anyhow::Result<()> {
28
+
let args = Args::parse();
29
+
30
+
let db = FjallDb::open(&args.db_path, None)?;
31
+
let disk_bytes = dir_size(&args.db_path);
32
+
33
+
println!("Ramjet Storage Forecast");
34
+
println!("=======================");
35
+
println!("Database path: {}", args.db_path.display());
36
+
println!("Disk usage: {}", format_bytes(disk_bytes));
37
+
println!();
38
+
39
+
// -- Events keyspace analysis (primary rate source) --
40
+
let event_stats = analyze_events(&db, args.sample_limit);
41
+
let sample_duration_secs = event_stats.duration_secs;
42
+
43
+
if sample_duration_secs < 60.0 {
44
+
println!(
45
+
"WARNING: Sample duration is only {:.0}s. Run the service longer for accurate projections.",
46
+
sample_duration_secs
47
+
);
48
+
println!();
49
+
}
50
+
51
+
println!("Sample Period");
52
+
println!("-------------");
53
+
println!(
54
+
"Duration: {:.1} hours ({:.0} seconds)",
55
+
sample_duration_secs / 3600.0,
56
+
sample_duration_secs
57
+
);
58
+
println!("Total events: {}", format_count(event_stats.total_events));
59
+
println!(
60
+
"Events/sec: {:.1}",
61
+
event_stats.total_events as f64 / sample_duration_secs
62
+
);
63
+
println!();
64
+
65
+
// -- Event type breakdown --
66
+
println!("Event Breakdown");
67
+
println!("---------------");
68
+
for (event_type, count) in &event_stats.by_type {
69
+
let pct = *count as f64 / event_stats.total_events.max(1) as f64 * 100.0;
70
+
let rate = *count as f64 / sample_duration_secs;
71
+
println!(
72
+
" {:<12} {:>10} ({:>5.1}%) {:.1}/s",
73
+
event_type,
74
+
format_count(*count),
75
+
pct,
76
+
rate
77
+
);
78
+
}
79
+
println!();
80
+
81
+
// -- Keyspace analysis --
82
+
let keyspace_stats = analyze_keyspaces(&db, args.sample_limit);
83
+
84
+
println!("Keyspace Analysis");
85
+
println!("-----------------");
86
+
println!(
87
+
" {:<15} {:>12} {:>12} {:>12} {:>12}",
88
+
"Keyspace", "Keys", "Avg Key", "Avg Value", "Est. Size"
89
+
);
90
+
let mut total_estimated = 0u64;
91
+
for ks in &keyspace_stats {
92
+
let est_size = ks.count as u64 * (ks.avg_key_size + ks.avg_value_size);
93
+
total_estimated += est_size;
94
+
println!(
95
+
" {:<15} {:>12} {:>10} B {:>10} B {:>12}",
96
+
ks.name,
97
+
format_count(ks.count),
98
+
ks.avg_key_size,
99
+
ks.avg_value_size,
100
+
format_bytes(est_size)
101
+
);
102
+
}
103
+
println!(
104
+
" {:<15} {:>12} {:>10} {:>10} {:>12}",
105
+
"",
106
+
"",
107
+
"",
108
+
"",
109
+
format_bytes(total_estimated)
110
+
);
111
+
println!();
112
+
113
+
// -- Collection breakdown in records keyspace --
114
+
let collection_stats = analyze_collections(&db, args.sample_limit);
115
+
if !collection_stats.is_empty() {
116
+
println!("Records by Collection");
117
+
println!("---------------------");
118
+
println!(
119
+
" {:<45} {:>10} {:>10} {:>12}",
120
+
"Collection", "Records", "Avg Size", "Est. Size"
121
+
);
122
+
let mut sorted: Vec<_> = collection_stats.iter().collect();
123
+
sorted.sort_by(|a, b| b.1.total_bytes.cmp(&a.1.total_bytes));
124
+
for (collection, stats) in &sorted {
125
+
let avg = if stats.count > 0 {
126
+
stats.total_bytes / stats.count as u64
127
+
} else {
128
+
0
129
+
};
130
+
println!(
131
+
" {:<45} {:>10} {:>8} B {:>12}",
132
+
truncate(collection, 45),
133
+
format_count(stats.count),
134
+
avg,
135
+
format_bytes(stats.total_bytes)
136
+
);
137
+
}
138
+
println!();
139
+
}
140
+
141
+
// -- Identity stats --
142
+
let identity_stats = analyze_identities(&db, args.sample_limit);
143
+
println!("Identity Cache");
144
+
println!("--------------");
145
+
println!(
146
+
" DID documents: {}",
147
+
format_count(identity_stats.doc_count)
148
+
);
149
+
println!(
150
+
" Handle mappings: {}",
151
+
format_count(identity_stats.handle_count)
152
+
);
153
+
println!(" Avg doc size: {} B", identity_stats.avg_doc_size);
154
+
println!(
155
+
" Unique repos: {}",
156
+
format_count(identity_stats.unique_repos)
157
+
);
158
+
println!();
159
+
160
+
// -- Projections --
161
+
if sample_duration_secs < 1.0 {
162
+
println!("Insufficient data for projections (need at least 1 second of events).");
163
+
return Ok(());
164
+
}
165
+
166
+
// Calculate growth rates from the sample
167
+
let events_per_sec = event_stats.total_events as f64 / sample_duration_secs;
168
+
let events_bytes_per_sec = keyspace_stats
169
+
.iter()
170
+
.find(|ks| ks.name == "events")
171
+
.map(|ks| {
172
+
let total = ks.count as f64 * (ks.avg_key_size + ks.avg_value_size) as f64;
173
+
total / sample_duration_secs
174
+
})
175
+
.unwrap_or(0.0);
176
+
177
+
let records_per_sec = {
178
+
let rec_ks = keyspace_stats.iter().find(|ks| ks.name == "records");
179
+
match rec_ks {
180
+
Some(ks) => ks.count as f64 / sample_duration_secs,
181
+
None => 0.0,
182
+
}
183
+
};
184
+
let records_bytes_per_sec = keyspace_stats
185
+
.iter()
186
+
.find(|ks| ks.name == "records")
187
+
.map(|ks| {
188
+
let total = ks.count as f64 * (ks.avg_key_size + ks.avg_value_size) as f64;
189
+
total / sample_duration_secs
190
+
})
191
+
.unwrap_or(0.0);
192
+
193
+
let identity_bytes_per_sec = {
194
+
let docs = keyspace_stats.iter().find(|ks| ks.name == "did_to_doc");
195
+
let handles = keyspace_stats.iter().find(|ks| ks.name == "handle_to_did");
196
+
let doc_total = docs
197
+
.map(|ks| ks.count as f64 * (ks.avg_key_size + ks.avg_value_size) as f64)
198
+
.unwrap_or(0.0);
199
+
let handle_total = handles
200
+
.map(|ks| ks.count as f64 * (ks.avg_key_size + ks.avg_value_size) as f64)
201
+
.unwrap_or(0.0);
202
+
(doc_total + handle_total) / sample_duration_secs
203
+
};
204
+
205
+
// Total data growth rate
206
+
let total_bytes_per_sec = events_bytes_per_sec + records_bytes_per_sec + identity_bytes_per_sec;
207
+
208
+
// Note: events keyspace has retention, so it eventually plateaus.
209
+
// Records and identity keyspaces grow unbounded (updates overwrite).
210
+
211
+
println!("Growth Rates (from sample)");
212
+
println!("-------------------------");
213
+
println!(
214
+
" Events: {:.1}/s {}/s",
215
+
events_per_sec,
216
+
format_bytes(events_bytes_per_sec as u64)
217
+
);
218
+
println!(
219
+
" Records: {:.1}/s {}/s",
220
+
records_per_sec,
221
+
format_bytes(records_bytes_per_sec as u64)
222
+
);
223
+
println!(
224
+
" Identity: {}/s",
225
+
format_bytes(identity_bytes_per_sec as u64)
226
+
);
227
+
println!(
228
+
" Total: {}/s",
229
+
format_bytes(total_bytes_per_sec as u64)
230
+
);
231
+
println!();
232
+
233
+
// Projection horizons
234
+
let horizons = [
235
+
("1 day", 86400.0),
236
+
("1 week", 86400.0 * 7.0),
237
+
("30 days", 86400.0 * 30.0),
238
+
("90 days", 86400.0 * 90.0),
239
+
("1 year", 86400.0 * 365.0),
240
+
];
241
+
242
+
println!("Storage Projections");
243
+
println!("-------------------");
244
+
println!("Assumes constant ingestion rate from sample period.");
245
+
println!("Note: events keyspace has configurable retention (default 72h).");
246
+
println!(" Identity/repo_state keyspaces grow with unique repos, not linearly.");
247
+
println!();
248
+
println!(
249
+
" {:<10} {:>14} {:>14} {:>14} {:>14}",
250
+
"Horizon", "Events", "Records", "Identity", "Total"
251
+
);
252
+
253
+
for (label, secs) in &horizons {
254
+
let events_size = events_bytes_per_sec * secs;
255
+
let records_size = records_bytes_per_sec * secs;
256
+
let identity_size = identity_bytes_per_sec * secs;
257
+
let total = events_size + records_size + identity_size;
258
+
259
+
println!(
260
+
" {:<10} {:>14} {:>14} {:>14} {:>14}",
261
+
label,
262
+
format_bytes(events_size as u64),
263
+
format_bytes(records_size as u64),
264
+
format_bytes(identity_size as u64),
265
+
format_bytes(total as u64)
266
+
);
267
+
}
268
+
println!();
269
+
270
+
// Event retention cap for events keyspace
271
+
println!("Events Keyspace with Retention Cap");
272
+
println!("----------------------------------");
273
+
let retention_hours = [24, 48, 72, 168, 720];
274
+
println!(" {:<16} {:>14}", "Retention", "Steady-State");
275
+
for hours in retention_hours {
276
+
let steady_state = events_bytes_per_sec * hours as f64 * 3600.0;
277
+
println!(
278
+
" {:<16} {:>14}",
279
+
format!("{hours}h"),
280
+
format_bytes(steady_state as u64)
281
+
);
282
+
}
283
+
println!();
284
+
285
+
// Unique repo growth estimate
286
+
let repos_per_sec = identity_stats.unique_repos as f64 / sample_duration_secs;
287
+
println!("Unique Repo Growth");
288
+
println!("------------------");
289
+
println!(
290
+
" Current repos: {}",
291
+
format_count(identity_stats.unique_repos)
292
+
);
293
+
println!(" Growth rate: {:.1} repos/s", repos_per_sec);
294
+
for (label, secs) in &horizons {
295
+
let projected = identity_stats.unique_repos as f64 + repos_per_sec * secs;
296
+
println!(" {:<10} {}", label, format_count(projected as u64));
297
+
}
298
+
println!();
299
+
300
+
// Per-collection projections for top collections
301
+
if !collection_stats.is_empty() {
302
+
println!("Per-Collection Storage Projections (30 days)");
303
+
println!("--------------------------------------------");
304
+
let thirty_days = 86400.0 * 30.0;
305
+
let mut sorted: Vec<_> = collection_stats.iter().collect();
306
+
sorted.sort_by(|a, b| b.1.total_bytes.cmp(&a.1.total_bytes));
307
+
308
+
println!(
309
+
" {:<45} {:>12} {:>14}",
310
+
"Collection", "Records/day", "30d Storage"
311
+
);
312
+
for (collection, stats) in sorted.iter().take(20) {
313
+
let rate_per_day = stats.count as f64 / sample_duration_secs * 86400.0;
314
+
let bytes_30d = stats.total_bytes as f64 / sample_duration_secs * thirty_days;
315
+
println!(
316
+
" {:<45} {:>12} {:>14}",
317
+
truncate(collection, 45),
318
+
format_count(rate_per_day as u64),
319
+
format_bytes(bytes_30d as u64)
320
+
);
321
+
}
322
+
}
323
+
324
+
Ok(())
325
+
}
326
+
327
+
// -- Analysis types --
328
+
329
+
struct EventStats {
330
+
total_events: u64,
331
+
duration_secs: f64,
332
+
by_type: Vec<(String, u64)>,
333
+
}
334
+
335
+
struct KeyspaceStats {
336
+
name: String,
337
+
count: u64,
338
+
avg_key_size: u64,
339
+
avg_value_size: u64,
340
+
}
341
+
342
+
struct CollectionInfo {
343
+
count: u64,
344
+
total_bytes: u64,
345
+
}
346
+
347
+
struct IdentityStats {
348
+
doc_count: u64,
349
+
handle_count: u64,
350
+
avg_doc_size: u64,
351
+
unique_repos: u64,
352
+
}
353
+
354
+
// -- Analysis functions --
355
+
356
+
fn analyze_events(db: &FjallDb, sample_limit: u64) -> EventStats {
357
+
let mut total: u64 = 0;
358
+
let mut by_type: HashMap<String, u64> = HashMap::new();
359
+
let mut first_seq: Option<u64> = None;
360
+
let mut last_seq: Option<u64> = None;
361
+
362
+
for guard in db.events.iter() {
363
+
let Ok((key, value)) = guard.into_inner() else {
364
+
continue;
365
+
};
366
+
367
+
if let Ok(seq) = keys::decode_event_key(&key) {
368
+
if first_seq.is_none() {
369
+
first_seq = Some(seq);
370
+
}
371
+
last_seq = Some(seq);
372
+
}
373
+
374
+
total += 1;
375
+
376
+
let slice: &[u8] = &value;
377
+
if let Ok(event) = encoding::decode_compact_event(slice) {
378
+
let t = match event {
379
+
encoding::CompactEvent::Commit { .. } => "commit",
380
+
encoding::CompactEvent::CommitOp { .. } => "commit",
381
+
encoding::CompactEvent::Identity { .. } => "identity",
382
+
encoding::CompactEvent::Account { .. } => "account",
383
+
};
384
+
*by_type.entry(t.to_string()).or_default() += 1;
385
+
} else if slice.first() == Some(&b'{') {
386
+
// Legacy JSON events
387
+
if let Ok(event) = serde_json::from_slice::<serde_json::Value>(slice) {
388
+
let t = event
389
+
.get("t")
390
+
.and_then(|v| v.as_str())
391
+
.unwrap_or("unknown")
392
+
.to_string();
393
+
*by_type.entry(t).or_default() += 1;
394
+
}
395
+
}
396
+
397
+
if sample_limit > 0 && total >= sample_limit {
398
+
break;
399
+
}
400
+
}
401
+
402
+
// Estimate duration from sequence numbers.
403
+
// ATProto firehose sequences are microsecond timestamps.
404
+
let duration_secs = match (first_seq, last_seq) {
405
+
(Some(first), Some(last)) if last > first => {
406
+
// Firehose seq numbers are ~microsecond timestamps
407
+
let diff = last - first;
408
+
if diff > 1_000_000_000 {
409
+
// Looks like microsecond timestamps
410
+
diff as f64 / 1_000_000.0
411
+
} else if diff > 1_000_000 {
412
+
// Might be millisecond timestamps
413
+
diff as f64 / 1_000.0
414
+
} else {
415
+
// Simple counter — estimate from event count
416
+
// Fall back to counting events and assuming ~500 events/sec
417
+
total as f64 / 500.0
418
+
}
419
+
}
420
+
_ => 0.0,
421
+
};
422
+
423
+
let mut sorted_types: Vec<_> = by_type.into_iter().collect();
424
+
sorted_types.sort_by(|a, b| b.1.cmp(&a.1));
425
+
426
+
EventStats {
427
+
total_events: total,
428
+
duration_secs: duration_secs.max(1.0),
429
+
by_type: sorted_types,
430
+
}
431
+
}
432
+
433
+
fn analyze_keyspaces(db: &FjallDb, sample_limit: u64) -> Vec<KeyspaceStats> {
434
+
let keyspaces: Vec<(&str, &fjall::Keyspace)> = vec![
435
+
("records", &db.records),
436
+
("events", &db.events),
437
+
("meta", &db.meta),
438
+
("repo_state", &db.repo_state),
439
+
("did_to_doc", &db.did_to_doc),
440
+
("handle_to_did", &db.handle_to_did),
441
+
("blobs", &db.blobs),
442
+
("blob_meta", &db.blob_meta),
443
+
];
444
+
445
+
keyspaces
446
+
.into_iter()
447
+
.map(|(name, ks)| {
448
+
let mut count: u64 = 0;
449
+
let mut total_key_bytes: u64 = 0;
450
+
let mut total_value_bytes: u64 = 0;
451
+
452
+
for guard in ks.iter() {
453
+
let Ok((key, value)) = guard.into_inner() else {
454
+
continue;
455
+
};
456
+
count += 1;
457
+
total_key_bytes += key.len() as u64;
458
+
total_value_bytes += value.len() as u64;
459
+
460
+
if sample_limit > 0 && count >= sample_limit {
461
+
break;
462
+
}
463
+
}
464
+
465
+
KeyspaceStats {
466
+
name: name.to_string(),
467
+
count,
468
+
avg_key_size: if count > 0 {
469
+
total_key_bytes / count
470
+
} else {
471
+
0
472
+
},
473
+
avg_value_size: if count > 0 {
474
+
total_value_bytes / count
475
+
} else {
476
+
0
477
+
},
478
+
}
479
+
})
480
+
.collect()
481
+
}
482
+
483
+
fn analyze_collections(db: &FjallDb, sample_limit: u64) -> HashMap<String, CollectionInfo> {
484
+
let mut collections: HashMap<String, CollectionInfo> = HashMap::new();
485
+
let mut count: u64 = 0;
486
+
487
+
for guard in db.records.iter() {
488
+
let Ok((key, value)) = guard.into_inner() else {
489
+
continue;
490
+
};
491
+
492
+
if let Ok((_, collection, _, _)) = keys::decode_record_key(&key) {
493
+
let entry = collections
494
+
.entry(collection.to_string())
495
+
.or_insert(CollectionInfo {
496
+
count: 0,
497
+
total_bytes: 0,
498
+
});
499
+
entry.count += 1;
500
+
entry.total_bytes += key.len() as u64 + value.len() as u64;
501
+
}
502
+
503
+
count += 1;
504
+
if sample_limit > 0 && count >= sample_limit {
505
+
break;
506
+
}
507
+
}
508
+
509
+
collections
510
+
}
511
+
512
+
fn analyze_identities(db: &FjallDb, sample_limit: u64) -> IdentityStats {
513
+
let mut doc_count: u64 = 0;
514
+
let mut total_doc_bytes: u64 = 0;
515
+
516
+
let mut count: u64 = 0;
517
+
for guard in db.did_to_doc.iter() {
518
+
let Ok((_key, value)) = guard.into_inner() else {
519
+
continue;
520
+
};
521
+
doc_count += 1;
522
+
let slice: &[u8] = &value;
523
+
let (_ts, json_bytes) = decode_timestamped_doc(slice);
524
+
total_doc_bytes += json_bytes.len() as u64;
525
+
526
+
count += 1;
527
+
if sample_limit > 0 && count >= sample_limit {
528
+
break;
529
+
}
530
+
}
531
+
532
+
let mut handle_count: u64 = 0;
533
+
count = 0;
534
+
for guard in db.handle_to_did.iter() {
535
+
let Ok(_) = guard.into_inner() else {
536
+
continue;
537
+
};
538
+
handle_count += 1;
539
+
540
+
count += 1;
541
+
if sample_limit > 0 && count >= sample_limit {
542
+
break;
543
+
}
544
+
}
545
+
546
+
// Count unique repos from repo_state
547
+
let mut unique_repos: u64 = 0;
548
+
count = 0;
549
+
for guard in db.repo_state.iter() {
550
+
let Ok(_) = guard.into_inner() else {
551
+
continue;
552
+
};
553
+
unique_repos += 1;
554
+
555
+
count += 1;
556
+
if sample_limit > 0 && count >= sample_limit {
557
+
break;
558
+
}
559
+
}
560
+
561
+
IdentityStats {
562
+
doc_count,
563
+
handle_count,
564
+
avg_doc_size: if doc_count > 0 {
565
+
total_doc_bytes / doc_count
566
+
} else {
567
+
0
568
+
},
569
+
unique_repos,
570
+
}
571
+
}
572
+
573
+
// -- Formatting helpers --
574
+
575
+
fn format_bytes(bytes: u64) -> String {
576
+
const KB: u64 = 1024;
577
+
const MB: u64 = 1024 * KB;
578
+
const GB: u64 = 1024 * MB;
579
+
const TB: u64 = 1024 * GB;
580
+
581
+
if bytes >= TB {
582
+
format!("{:.2} TB", bytes as f64 / TB as f64)
583
+
} else if bytes >= GB {
584
+
format!("{:.2} GB", bytes as f64 / GB as f64)
585
+
} else if bytes >= MB {
586
+
format!("{:.2} MB", bytes as f64 / MB as f64)
587
+
} else if bytes >= KB {
588
+
format!("{:.2} KB", bytes as f64 / KB as f64)
589
+
} else {
590
+
format!("{bytes} B")
591
+
}
592
+
}
593
+
594
+
fn format_count(n: u64) -> String {
595
+
if n >= 1_000_000_000 {
596
+
format!("{:.2}B", n as f64 / 1_000_000_000.0)
597
+
} else if n >= 1_000_000 {
598
+
format!("{:.2}M", n as f64 / 1_000_000.0)
599
+
} else if n >= 1_000 {
600
+
format!("{:.1}K", n as f64 / 1_000.0)
601
+
} else {
602
+
format!("{n}")
603
+
}
604
+
}
605
+
606
+
fn truncate(s: &str, max: usize) -> String {
607
+
if s.len() <= max {
608
+
s.to_string()
609
+
} else {
610
+
format!("{}...", &s[..max - 3])
611
+
}
612
+
}
613
+
614
+
fn dir_size(path: &PathBuf) -> u64 {
615
+
let Ok(entries) = std::fs::read_dir(path) else {
616
+
return 0;
617
+
};
618
+
let mut total: u64 = 0;
619
+
for entry in entries.flatten() {
620
+
let Ok(meta) = entry.metadata() else {
621
+
continue;
622
+
};
623
+
if meta.is_dir() {
624
+
total += dir_size(&entry.path().to_path_buf());
625
+
} else {
626
+
total += meta.len();
627
+
}
628
+
}
629
+
total
630
+
}
+721
src/bin/ramjet_writer.rs
+721
src/bin/ramjet_writer.rs
···
1
+
//! Benchmark binary: ingests from a relay and writes to fjall,
2
+
//! periodically logging ingestion rate, write rate, and disk usage
3
+
//! for pipeline analysis and capacity planning.
4
+
5
+
use std::collections::HashMap;
6
+
use std::path::PathBuf;
7
+
use std::sync::Arc;
8
+
use std::sync::atomic::{AtomicU64, Ordering};
9
+
use std::time::{Duration, Instant};
10
+
11
+
use clap::Parser;
12
+
use tokio::sync::mpsc;
13
+
use tokio_util::sync::CancellationToken;
14
+
15
+
use ramjet::config::{CollectionMatcher, ServiceConfig};
16
+
use ramjet::pipeline::ingester::{IngestEvent, run_ingester};
17
+
use ramjet::server::dictionary::compute_raw_cid;
18
+
use ramjet::server::metrics::Metrics;
19
+
use ramjet::storage::FjallDb;
20
+
use ramjet::storage::keys;
21
+
use ramjet::types::{AccountStatus, OpType, RecordValue};
22
+
23
+
#[derive(Parser, Debug)]
24
+
#[command(
25
+
name = "ramjet-writer",
26
+
about = "Benchmark ingester→writer pipeline with periodic stats"
27
+
)]
28
+
struct Args {
29
+
/// Path to the fjall database directory.
30
+
#[arg(long, env = "RAMJET_DB_PATH", default_value = "./data/ramjet-bench.db")]
31
+
db_path: PathBuf,
32
+
33
+
/// Upstream relay WebSocket host.
34
+
#[arg(long, env = "RAMJET_RELAY_HOST", default_value = "bsky.network")]
35
+
relay_host: String,
36
+
37
+
/// Collection patterns to persist (space-separated).
38
+
#[arg(long, env = "RAMJET_TRACKED_COLLECTIONS", default_value = "*")]
39
+
tracked_collections: String,
40
+
41
+
/// Maximum events per write batch.
42
+
#[arg(long, default_value = "500")]
43
+
batch_size: usize,
44
+
45
+
/// Maximum wait time (ms) for batch fill before flushing.
46
+
#[arg(long, default_value = "100")]
47
+
batch_timeout_ms: u64,
48
+
49
+
/// Stats reporting interval in seconds.
50
+
#[arg(long, default_value = "10")]
51
+
stats_interval: u64,
52
+
53
+
/// Sample rate for printing events to stdout (0.0 = none, 1.0 = all).
54
+
#[arg(long)]
55
+
sample: Option<f64>,
56
+
57
+
/// Path to a local zstd dictionary file.
58
+
#[arg(long, env = "RAMJET_ZSTD_DICT_PATH")]
59
+
zstd_dict_path: Option<PathBuf>,
60
+
61
+
/// URL of the ramjet server to fetch the zstd dictionary from.
62
+
#[arg(long, env = "RAMJET_URL")]
63
+
ramjet_url: Option<String>,
64
+
65
+
/// Disable zstd dictionary usage entirely.
66
+
#[arg(long, default_value = "false")]
67
+
no_zstd: bool,
68
+
}
69
+
70
+
/// Counters shared between the writer loop and the stats reporter.
71
+
struct Stats {
72
+
events_ingested: AtomicU64,
73
+
events_written: AtomicU64,
74
+
batches_committed: AtomicU64,
75
+
records_written: AtomicU64,
76
+
records_deleted: AtomicU64,
77
+
bytes_written: AtomicU64,
78
+
}
79
+
80
+
impl Stats {
81
+
fn new() -> Self {
82
+
Self {
83
+
events_ingested: AtomicU64::new(0),
84
+
events_written: AtomicU64::new(0),
85
+
batches_committed: AtomicU64::new(0),
86
+
records_written: AtomicU64::new(0),
87
+
records_deleted: AtomicU64::new(0),
88
+
bytes_written: AtomicU64::new(0),
89
+
}
90
+
}
91
+
}
92
+
93
+
#[tokio::main]
94
+
async fn main() -> anyhow::Result<()> {
95
+
let args = Args::parse();
96
+
97
+
tracing_subscriber::fmt()
98
+
.with_env_filter(
99
+
tracing_subscriber::EnvFilter::try_from_default_env()
100
+
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
101
+
)
102
+
.init();
103
+
104
+
tracing::info!(
105
+
db_path = %args.db_path.display(),
106
+
relay_host = %args.relay_host,
107
+
tracked = %args.tracked_collections,
108
+
batch_size = args.batch_size,
109
+
batch_timeout_ms = args.batch_timeout_ms,
110
+
stats_interval = args.stats_interval,
111
+
"starting ramjet-writer benchmark"
112
+
);
113
+
114
+
let dict_path = resolve_dictionary(&args).await?;
115
+
let db = Arc::new(FjallDb::open(&args.db_path, dict_path.as_deref())?);
116
+
tracing::info!("fjall database opened");
117
+
118
+
if let Ok(Some(cursor)) = db.get_cursor() {
119
+
tracing::info!(cursor, "resuming from stored cursor");
120
+
}
121
+
122
+
let tracked = CollectionMatcher::new(&args.tracked_collections);
123
+
let cancel = CancellationToken::new();
124
+
let metrics = Arc::new(Metrics::new());
125
+
let stats = Arc::new(Stats::new());
126
+
127
+
// Build a minimal ServiceConfig for the ingester
128
+
let config = Arc::new(ServiceConfig {
129
+
db_path: args.db_path.clone(),
130
+
relay_host: args.relay_host.clone(),
131
+
listen_addr: "127.0.0.1:0".parse().unwrap(),
132
+
tracked_collections: CollectionMatcher::new(&args.tracked_collections),
133
+
forward_collections: CollectionMatcher::new(""),
134
+
event_retention_hours: 0,
135
+
batch_size: args.batch_size,
136
+
batch_timeout_ms: args.batch_timeout_ms,
137
+
admin_dids: Default::default(),
138
+
zstd_dict_path: None,
139
+
backfill_dids: Vec::new(),
140
+
consumer_groups: Vec::new(),
141
+
});
142
+
143
+
let (tx, rx) = mpsc::channel(4096);
144
+
// Identity channel (unused in benchmark, but required by ingester)
145
+
let (identity_tx, _identity_rx) = mpsc::channel(1024);
146
+
147
+
// Spawn ingester
148
+
let ingester_handle = tokio::spawn({
149
+
let config = config.clone();
150
+
let db = db.clone();
151
+
let metrics = metrics.clone();
152
+
let cancel = cancel.clone();
153
+
async move {
154
+
if let Err(e) = run_ingester(config, db, tx, identity_tx, metrics, cancel).await {
155
+
tracing::error!(error = %e, "ingester failed");
156
+
}
157
+
}
158
+
});
159
+
160
+
// Spawn stats reporter
161
+
let stats_handle = tokio::spawn({
162
+
let stats = stats.clone();
163
+
let db_path = args.db_path.clone();
164
+
let cancel = cancel.clone();
165
+
let interval = Duration::from_secs(args.stats_interval);
166
+
async move {
167
+
report_stats_loop(stats, db_path, interval, cancel).await;
168
+
}
169
+
});
170
+
171
+
// Run writer in the main task
172
+
run_writer_loop(
173
+
db,
174
+
tracked,
175
+
rx,
176
+
stats,
177
+
config.batch_size,
178
+
config.batch_timeout_ms,
179
+
args.sample.unwrap_or(0.0),
180
+
cancel.clone(),
181
+
)
182
+
.await;
183
+
184
+
cancel.cancel();
185
+
let _ = tokio::join!(ingester_handle, stats_handle);
186
+
tracing::info!("ramjet-writer benchmark shut down");
187
+
Ok(())
188
+
}
189
+
190
+
/// Resolve the zstd dictionary to use, fetching from a ramjet server if configured.
191
+
///
192
+
/// Returns the path to the dictionary file, or `None` if no dictionary should be used.
193
+
async fn resolve_dictionary(args: &Args) -> anyhow::Result<Option<PathBuf>> {
194
+
if args.no_zstd {
195
+
tracing::info!("zstd dictionary disabled via --no-zstd");
196
+
return Ok(None);
197
+
}
198
+
199
+
// Compute the CID of the local dictionary if one was provided.
200
+
let local_dict = args
201
+
.zstd_dict_path
202
+
.as_ref()
203
+
.map(|p| std::fs::read(p))
204
+
.transpose()?;
205
+
let local_cid = local_dict.as_ref().map(|d| compute_raw_cid(d));
206
+
207
+
let Some(ramjet_url) = &args.ramjet_url else {
208
+
// No server URL — just use the local dictionary if provided.
209
+
return Ok(args.zstd_dict_path.clone());
210
+
};
211
+
212
+
let url = format!("{}/dictionary", ramjet_url.trim_end_matches('/'));
213
+
let client = reqwest::Client::new();
214
+
let mut request = client.get(&url);
215
+
if let Some(cid) = &local_cid {
216
+
request = request.header("If-None-Match", cid.as_str());
217
+
}
218
+
219
+
let response = match request.send().await {
220
+
Ok(r) => r,
221
+
Err(e) => {
222
+
tracing::warn!(error = %e, "failed to fetch dictionary from server, using local");
223
+
return Ok(args.zstd_dict_path.clone());
224
+
}
225
+
};
226
+
227
+
match response.status() {
228
+
reqwest::StatusCode::NOT_MODIFIED => {
229
+
tracing::info!("server dictionary matches local (CID unchanged)");
230
+
Ok(args.zstd_dict_path.clone())
231
+
}
232
+
reqwest::StatusCode::NOT_FOUND => {
233
+
tracing::info!("server has no zstd dictionary");
234
+
Ok(args.zstd_dict_path.clone())
235
+
}
236
+
reqwest::StatusCode::OK => {
237
+
let server_cid = response
238
+
.headers()
239
+
.get("etag")
240
+
.and_then(|v| v.to_str().ok())
241
+
.map(|s| s.to_string());
242
+
243
+
if let (Some(local), Some(remote)) = (&local_cid, &server_cid) {
244
+
if local != remote {
245
+
tracing::warn!(
246
+
local_cid = %local,
247
+
server_cid = %remote,
248
+
"local zstd dictionary differs from server, using server's"
249
+
);
250
+
}
251
+
}
252
+
253
+
let bytes = response.bytes().await?;
254
+
let dict_path = args.db_path.join("dictionary.zstd");
255
+
std::fs::create_dir_all(&args.db_path)?;
256
+
std::fs::write(&dict_path, &bytes)?;
257
+
tracing::info!(
258
+
dict_size = bytes.len(),
259
+
cid = server_cid.as_deref().unwrap_or("unknown"),
260
+
path = %dict_path.display(),
261
+
"fetched zstd dictionary from server"
262
+
);
263
+
Ok(Some(dict_path))
264
+
}
265
+
status => {
266
+
tracing::warn!(
267
+
%status,
268
+
"unexpected response from dictionary endpoint, using local"
269
+
);
270
+
Ok(args.zstd_dict_path.clone())
271
+
}
272
+
}
273
+
}
274
+
275
+
async fn run_writer_loop(
276
+
db: Arc<FjallDb>,
277
+
tracked: CollectionMatcher,
278
+
mut rx: mpsc::Receiver<IngestEvent>,
279
+
stats: Arc<Stats>,
280
+
batch_size: usize,
281
+
batch_timeout_ms: u64,
282
+
sample_rate: f64,
283
+
cancel: CancellationToken,
284
+
) {
285
+
let batch_timeout = Duration::from_millis(batch_timeout_ms);
286
+
use rand::RngExt;
287
+
let mut rng = rand::rng();
288
+
289
+
loop {
290
+
let mut events = Vec::with_capacity(batch_size);
291
+
292
+
tokio::select! {
293
+
biased;
294
+
_ = cancel.cancelled() => break,
295
+
event = rx.recv() => {
296
+
match event {
297
+
Some(e) => events.push(e),
298
+
None => break,
299
+
}
300
+
}
301
+
}
302
+
303
+
let deadline = tokio::time::Instant::now() + batch_timeout;
304
+
while events.len() < batch_size {
305
+
tokio::select! {
306
+
biased;
307
+
_ = cancel.cancelled() => break,
308
+
_ = tokio::time::sleep_until(deadline) => break,
309
+
event = rx.recv() => {
310
+
match event {
311
+
Some(e) => events.push(e),
312
+
None => break,
313
+
}
314
+
}
315
+
}
316
+
}
317
+
318
+
if events.is_empty() {
319
+
continue;
320
+
}
321
+
322
+
if sample_rate > 0.0 {
323
+
for event in &events {
324
+
if sample_rate >= 1.0 || rng.random_range(0.0..1.0) < sample_rate {
325
+
print_event(event);
326
+
}
327
+
}
328
+
}
329
+
330
+
stats
331
+
.events_ingested
332
+
.fetch_add(events.len() as u64, Ordering::Relaxed);
333
+
334
+
match write_batch(&db, &tracked, &events, &stats) {
335
+
Ok(()) => {
336
+
stats.batches_committed.fetch_add(1, Ordering::Relaxed);
337
+
stats
338
+
.events_written
339
+
.fetch_add(events.len() as u64, Ordering::Relaxed);
340
+
}
341
+
Err(e) => {
342
+
tracing::error!(error = %e, "batch write failed");
343
+
}
344
+
}
345
+
}
346
+
}
347
+
348
+
fn write_batch(
349
+
db: &FjallDb,
350
+
tracked: &CollectionMatcher,
351
+
events: &[IngestEvent],
352
+
stats: &Stats,
353
+
) -> anyhow::Result<()> {
354
+
let mut batch = db.batch();
355
+
let mut max_seq: u64 = 0;
356
+
let mut batch_bytes: u64 = 0;
357
+
358
+
for event in events {
359
+
match event {
360
+
IngestEvent::Commit {
361
+
seq,
362
+
did,
363
+
rev,
364
+
ops,
365
+
blocks,
366
+
..
367
+
} => {
368
+
max_seq = max_seq.max(*seq);
369
+
370
+
let block_map = parse_blocks_car(blocks);
371
+
let mut commit_ops = Vec::new();
372
+
373
+
for op in ops {
374
+
if !tracked.matches(&op.collection) {
375
+
continue;
376
+
}
377
+
378
+
let record_key = keys::encode_record_key(did, &op.collection, &op.rkey, rev);
379
+
380
+
match op.op {
381
+
OpType::Create | OpType::Update => {
382
+
let cid_bytes = op.cid.as_deref().unwrap_or_default().to_vec();
383
+
let data = block_map.get(&cid_bytes).cloned().unwrap_or_default();
384
+
let record_value = RecordValue {
385
+
cid: cid_bytes,
386
+
data,
387
+
};
388
+
let encoded = record_value.encode();
389
+
batch_bytes += (record_key.len() + encoded.len()) as u64;
390
+
batch.insert(&db.records, &record_key, encoded);
391
+
stats.records_written.fetch_add(1, Ordering::Relaxed);
392
+
}
393
+
OpType::Delete => {
394
+
// Tombstone: empty value
395
+
batch.insert(&db.records, &record_key, b"");
396
+
stats.records_deleted.fetch_add(1, Ordering::Relaxed);
397
+
}
398
+
}
399
+
400
+
commit_ops.push(op.clone());
401
+
}
402
+
403
+
let mut repo_state = match db.get_repo_state(did) {
404
+
Ok(Some(rs)) => rs,
405
+
Ok(None) => {
406
+
tracing::debug!(%did, "new repo (no prior state)");
407
+
Default::default()
408
+
}
409
+
Err(e) => {
410
+
tracing::warn!(%did, error = %e, "corrupt repo_state, resetting");
411
+
Default::default()
412
+
}
413
+
};
414
+
repo_state.rev = rev.clone();
415
+
let state_encoded = repo_state.encode();
416
+
batch_bytes += (did.len() + state_encoded.len()) as u64;
417
+
batch.insert(&db.repo_state, did.as_bytes(), state_encoded);
418
+
}
419
+
420
+
IngestEvent::Identity { seq, did, handle } => {
421
+
max_seq = max_seq.max(*seq);
422
+
if let Some(h) = handle {
423
+
let key = h.to_lowercase();
424
+
batch_bytes += (key.len() + did.len()) as u64;
425
+
batch.insert(&db.handle_to_did, key.as_bytes(), did.as_bytes());
426
+
}
427
+
}
428
+
429
+
IngestEvent::Account {
430
+
seq,
431
+
did,
432
+
active,
433
+
status,
434
+
..
435
+
} => {
436
+
max_seq = max_seq.max(*seq);
437
+
let mut repo_state = match db.get_repo_state(did) {
438
+
Ok(Some(rs)) => rs,
439
+
Ok(None) => {
440
+
tracing::debug!(%did, "new account (no prior state)");
441
+
Default::default()
442
+
}
443
+
Err(e) => {
444
+
tracing::warn!(%did, error = %e, "corrupt repo_state, resetting");
445
+
Default::default()
446
+
}
447
+
};
448
+
repo_state.status = if *active {
449
+
AccountStatus::Active
450
+
} else {
451
+
match status.as_deref() {
452
+
Some("deactivated") => AccountStatus::Deactivated,
453
+
Some("suspended") => AccountStatus::Suspended,
454
+
Some("deleted") => AccountStatus::Deleted,
455
+
Some("takendown") => AccountStatus::Takendown,
456
+
_ => AccountStatus::Deactivated,
457
+
}
458
+
};
459
+
let state_encoded = repo_state.encode();
460
+
batch_bytes += (did.len() + state_encoded.len()) as u64;
461
+
batch.insert(&db.repo_state, did.as_bytes(), state_encoded);
462
+
}
463
+
464
+
IngestEvent::Sync { seq, .. } => {
465
+
max_seq = max_seq.max(*seq);
466
+
}
467
+
}
468
+
}
469
+
470
+
if max_seq > 0 {
471
+
batch.insert(&db.meta, b"cursor", &max_seq.to_be_bytes());
472
+
}
473
+
474
+
batch.commit()?;
475
+
stats
476
+
.bytes_written
477
+
.fetch_add(batch_bytes, Ordering::Relaxed);
478
+
Ok(())
479
+
}
480
+
481
+
async fn report_stats_loop(
482
+
stats: Arc<Stats>,
483
+
db_path: PathBuf,
484
+
interval: Duration,
485
+
cancel: CancellationToken,
486
+
) {
487
+
let mut ticker = tokio::time::interval(interval);
488
+
ticker.tick().await; // skip first immediate tick
489
+
490
+
let mut prev_ingested: u64 = 0;
491
+
let mut prev_written: u64 = 0;
492
+
let mut prev_batches: u64 = 0;
493
+
let mut prev_records: u64 = 0;
494
+
let mut prev_deleted: u64 = 0;
495
+
let mut prev_bytes: u64 = 0;
496
+
let mut prev_time = Instant::now();
497
+
498
+
loop {
499
+
tokio::select! {
500
+
biased;
501
+
_ = cancel.cancelled() => break,
502
+
_ = ticker.tick() => {}
503
+
}
504
+
505
+
let now = Instant::now();
506
+
let elapsed = now.duration_since(prev_time).as_secs_f64();
507
+
prev_time = now;
508
+
509
+
let ingested = stats.events_ingested.load(Ordering::Relaxed);
510
+
let written = stats.events_written.load(Ordering::Relaxed);
511
+
let batches = stats.batches_committed.load(Ordering::Relaxed);
512
+
let records = stats.records_written.load(Ordering::Relaxed);
513
+
let deleted = stats.records_deleted.load(Ordering::Relaxed);
514
+
let bytes = stats.bytes_written.load(Ordering::Relaxed);
515
+
516
+
let d_ingested = ingested - prev_ingested;
517
+
let d_written = written - prev_written;
518
+
let d_batches = batches - prev_batches;
519
+
let d_records = records - prev_records;
520
+
let d_deleted = deleted - prev_deleted;
521
+
let d_bytes = bytes - prev_bytes;
522
+
523
+
prev_ingested = ingested;
524
+
prev_written = written;
525
+
prev_batches = batches;
526
+
prev_records = records;
527
+
prev_deleted = deleted;
528
+
prev_bytes = bytes;
529
+
530
+
let ingest_rate = d_ingested as f64 / elapsed;
531
+
let write_rate = d_written as f64 / elapsed;
532
+
let record_rate = d_records as f64 / elapsed;
533
+
let byte_rate = d_bytes as f64 / elapsed;
534
+
535
+
let disk_bytes = dir_size(&db_path);
536
+
537
+
tracing::info!(
538
+
events_per_sec = format!("{ingest_rate:.0}"),
539
+
writes_per_sec = format!("{write_rate:.0}"),
540
+
records_per_sec = format!("{record_rate:.0}"),
541
+
batches = d_batches,
542
+
records_written = d_records,
543
+
records_deleted = d_deleted,
544
+
write_bytes = format_bytes(d_bytes),
545
+
write_throughput = format!("{}/s", format_bytes(byte_rate as u64)),
546
+
total_events = ingested,
547
+
total_records = records,
548
+
total_bytes_written = format_bytes(bytes),
549
+
disk_usage = format_bytes(disk_bytes),
550
+
"stats"
551
+
);
552
+
}
553
+
}
554
+
555
+
fn dir_size(path: &PathBuf) -> u64 {
556
+
walkdir(path)
557
+
}
558
+
559
+
fn walkdir(path: &PathBuf) -> u64 {
560
+
let Ok(entries) = std::fs::read_dir(path) else {
561
+
return 0;
562
+
};
563
+
let mut total: u64 = 0;
564
+
for entry in entries.flatten() {
565
+
let Ok(meta) = entry.metadata() else {
566
+
continue;
567
+
};
568
+
if meta.is_dir() {
569
+
total += walkdir(&entry.path().to_path_buf());
570
+
} else {
571
+
total += meta.len();
572
+
}
573
+
}
574
+
total
575
+
}
576
+
577
+
fn print_event(event: &IngestEvent) {
578
+
match event {
579
+
IngestEvent::Commit {
580
+
seq,
581
+
did,
582
+
rev,
583
+
ops,
584
+
blocks,
585
+
..
586
+
} => {
587
+
let block_map = parse_blocks_car(blocks);
588
+
let mut json_ops = Vec::new();
589
+
for op in ops {
590
+
let cid_str = op
591
+
.cid
592
+
.as_ref()
593
+
.map(|c| {
594
+
cid::Cid::read_bytes(c.as_slice())
595
+
.map(|c| c.to_string())
596
+
.unwrap_or_else(|_| format!("{}B", c.len()))
597
+
})
598
+
.unwrap_or_default();
599
+
let mut op_json = serde_json::json!({
600
+
"op": format!("{:?}", op.op),
601
+
"collection": op.collection,
602
+
"rkey": op.rkey,
603
+
"cid": cid_str,
604
+
});
605
+
if let Some(cid_bytes) = &op.cid {
606
+
if let Some(data) = block_map.get(cid_bytes.as_slice()) {
607
+
if let Ok(ipld) =
608
+
atproto_dasl::drisl::from_slice::<atproto_dasl::Ipld>(data)
609
+
{
610
+
op_json["record"] = ipld_to_json(&ipld);
611
+
} else {
612
+
op_json["record_bytes"] = serde_json::Value::Number(data.len().into());
613
+
}
614
+
}
615
+
}
616
+
json_ops.push(op_json);
617
+
}
618
+
let commit = serde_json::json!({
619
+
"t": "#commit",
620
+
"seq": seq,
621
+
"did": did,
622
+
"rev": rev,
623
+
"blocks_bytes": blocks.len(),
624
+
"ops": json_ops,
625
+
});
626
+
println!(
627
+
"COMMIT {}",
628
+
serde_json::to_string_pretty(&commit).unwrap_or_default()
629
+
);
630
+
}
631
+
IngestEvent::Identity { seq, did, handle } => {
632
+
println!(
633
+
"IDENTITY seq={seq} did={did} handle={}",
634
+
handle.as_deref().unwrap_or("-")
635
+
);
636
+
}
637
+
IngestEvent::Account {
638
+
seq,
639
+
did,
640
+
active,
641
+
status,
642
+
..
643
+
} => {
644
+
println!(
645
+
"ACCOUNT seq={seq} did={did} active={active} status={}",
646
+
status.as_deref().unwrap_or("-")
647
+
);
648
+
}
649
+
IngestEvent::Sync { seq, did, rev } => {
650
+
println!("SYNC seq={seq} did={did} rev={rev}");
651
+
}
652
+
}
653
+
}
654
+
655
+
fn ipld_to_json(ipld: &atproto_dasl::Ipld) -> serde_json::Value {
656
+
use atproto_dasl::Ipld;
657
+
match ipld {
658
+
Ipld::Null => serde_json::Value::Null,
659
+
Ipld::Bool(b) => serde_json::Value::Bool(*b),
660
+
Ipld::Integer(i) => serde_json::json!(*i),
661
+
Ipld::Float(f) => serde_json::json!(*f),
662
+
Ipld::String(s) => serde_json::Value::String(s.clone()),
663
+
Ipld::Bytes(b) => {
664
+
use base64::Engine;
665
+
serde_json::json!({ "$bytes": base64::prelude::BASE64_STANDARD.encode(b) })
666
+
}
667
+
Ipld::List(list) => serde_json::Value::Array(list.iter().map(ipld_to_json).collect()),
668
+
Ipld::Map(map) => {
669
+
let obj: serde_json::Map<String, serde_json::Value> = map
670
+
.iter()
671
+
.map(|(k, v)| (k.clone(), ipld_to_json(v)))
672
+
.collect();
673
+
serde_json::Value::Object(obj)
674
+
}
675
+
Ipld::Link(cid) => {
676
+
serde_json::json!({ "$link": cid.to_string() })
677
+
}
678
+
}
679
+
}
680
+
681
+
fn parse_blocks_car(blocks: &[u8]) -> HashMap<Vec<u8>, Vec<u8>> {
682
+
let mut map = HashMap::new();
683
+
if blocks.is_empty() {
684
+
return map;
685
+
}
686
+
687
+
let mut cursor = std::io::Cursor::new(blocks);
688
+
689
+
if atproto_dasl::CarHeader::read_from(&mut cursor).is_err() {
690
+
return map;
691
+
}
692
+
693
+
loop {
694
+
match atproto_dasl::CarBlock::read_from(&mut cursor) {
695
+
Ok(Some(block)) => {
696
+
let cid_bytes = block.cid.to_bytes();
697
+
map.insert(cid_bytes, block.data);
698
+
}
699
+
Ok(None) => break,
700
+
Err(_) => break,
701
+
}
702
+
}
703
+
704
+
map
705
+
}
706
+
707
+
fn format_bytes(bytes: u64) -> String {
708
+
const KIB: u64 = 1024;
709
+
const MIB: u64 = 1024 * KIB;
710
+
const GIB: u64 = 1024 * MIB;
711
+
712
+
if bytes >= GIB {
713
+
format!("{:.2} GiB", bytes as f64 / GIB as f64)
714
+
} else if bytes >= MIB {
715
+
format!("{:.2} MiB", bytes as f64 / MIB as f64)
716
+
} else if bytes >= KIB {
717
+
format!("{:.2} KiB", bytes as f64 / KIB as f64)
718
+
} else {
719
+
format!("{bytes} B")
720
+
}
721
+
}
+1855
src/bin/rjtop.rs
+1855
src/bin/rjtop.rs
···
1
+
//! rjtop — TUI dashboard for Ramjet runtime metrics.
2
+
//!
3
+
//! Polls `GET /metrics` every 10 seconds and displays key
4
+
//! runtime statistics with sparklines, bar charts, and rates.
5
+
//!
6
+
//! Press [Tab] to cycle between Pipeline and Process panes.
7
+
8
+
use std::collections::{HashMap, VecDeque};
9
+
use std::io;
10
+
use std::time::{Duration, Instant};
11
+
12
+
use crossterm::event::{self, Event, KeyCode, KeyEventKind};
13
+
use crossterm::terminal::{disable_raw_mode, enable_raw_mode};
14
+
use ratatui::Terminal;
15
+
use ratatui::backend::CrosstermBackend;
16
+
use ratatui::layout::{Constraint, Direction, Layout, Rect};
17
+
use ratatui::style::{Color, Modifier, Style};
18
+
use ratatui::text::{Line, Span};
19
+
use ratatui::widgets::{Block, Borders, Gauge, Paragraph, Row, Sparkline, Table, Tabs};
20
+
21
+
const POLL_INTERVAL: Duration = Duration::from_secs(10);
22
+
const HISTORY_LEN: usize = 60; // 60 samples = 10 minutes at 10s intervals
23
+
24
+
/// CLI arguments.
25
+
#[derive(clap::Parser)]
26
+
#[command(name = "rjtop", about = "TUI dashboard for Ramjet metrics")]
27
+
struct Args {
28
+
/// Ramjet metrics endpoint URL.
29
+
#[arg(long, default_value = "http://127.0.0.1:8080/metrics")]
30
+
url: String,
31
+
}
32
+
33
+
fn main() -> anyhow::Result<()> {
34
+
let args = <Args as clap::Parser>::parse();
35
+
36
+
enable_raw_mode()?;
37
+
let mut stdout = io::stdout();
38
+
crossterm::execute!(
39
+
stdout,
40
+
crossterm::terminal::EnterAlternateScreen,
41
+
crossterm::event::EnableMouseCapture
42
+
)?;
43
+
let backend = CrosstermBackend::new(stdout);
44
+
let mut terminal = Terminal::new(backend)?;
45
+
46
+
let result = run_app(&mut terminal, &args.url);
47
+
48
+
disable_raw_mode()?;
49
+
crossterm::execute!(
50
+
terminal.backend_mut(),
51
+
crossterm::terminal::LeaveAlternateScreen,
52
+
crossterm::event::DisableMouseCapture
53
+
)?;
54
+
terminal.show_cursor()?;
55
+
56
+
result
57
+
}
58
+
59
+
// -- Dashboard state with history --
60
+
61
+
const TAB_PIPELINE: usize = 0;
62
+
const TAB_PROCESS: usize = 1;
63
+
const TAB_HTTP: usize = 2;
64
+
const TAB_COUNT: usize = 3;
65
+
66
+
struct DashboardState {
67
+
current: MetricsSnapshot,
68
+
/// Active tab index.
69
+
active_tab: usize,
70
+
/// Rolling history of events/s rates for sparkline.
71
+
events_per_sec: VecDeque<u64>,
72
+
/// Rolling history of batch sizes for sparkline.
73
+
batch_sizes: VecDeque<u64>,
74
+
/// Rolling history of fanout queue depth (high) for sparkline.
75
+
fanout_high_depth: VecDeque<u64>,
76
+
/// Rolling history of fanout queue depth (low) for sparkline.
77
+
fanout_low_depth: VecDeque<u64>,
78
+
/// Rolling history of HTTP req/s for sparkline.
79
+
http_per_sec: VecDeque<u64>,
80
+
/// Rolling history of identity resolve latency (ms) for sparkline.
81
+
identity_latency_ms: VecDeque<u64>,
82
+
/// Rolling history of mean poll duration (us) for sparkline.
83
+
mean_poll_us: VecDeque<u64>,
84
+
/// Rolling history of mean scheduled duration (us) for sparkline.
85
+
mean_scheduled_us: VecDeque<u64>,
86
+
/// Rolling history of slow polls per interval for sparkline.
87
+
slow_polls_per_interval: VecDeque<u64>,
88
+
/// Rolling history of long delays per interval for sparkline.
89
+
long_delays_per_interval: VecDeque<u64>,
90
+
/// Rolling history of HTTP request duration (avg ms) for sparkline.
91
+
http_duration_ms: VecDeque<u64>,
92
+
/// Previous snapshot for computing deltas.
93
+
prev: Option<MetricsSnapshot>,
94
+
/// Timestamp of previous successful poll (for accurate rate computation).
95
+
prev_poll_time: Option<Instant>,
96
+
/// Current computed rates.
97
+
events_rate: f64,
98
+
fanout_rate: f64,
99
+
ws_sent_rate: f64,
100
+
batches_rate: f64,
101
+
http_rate: f64,
102
+
error_msg: Option<String>,
103
+
last_poll: Instant,
104
+
}
105
+
106
+
impl DashboardState {
107
+
fn new() -> Self {
108
+
Self {
109
+
current: MetricsSnapshot::default(),
110
+
active_tab: TAB_PIPELINE,
111
+
events_per_sec: VecDeque::with_capacity(HISTORY_LEN),
112
+
batch_sizes: VecDeque::with_capacity(HISTORY_LEN),
113
+
fanout_high_depth: VecDeque::with_capacity(HISTORY_LEN),
114
+
fanout_low_depth: VecDeque::with_capacity(HISTORY_LEN),
115
+
http_per_sec: VecDeque::with_capacity(HISTORY_LEN),
116
+
identity_latency_ms: VecDeque::with_capacity(HISTORY_LEN),
117
+
mean_poll_us: VecDeque::with_capacity(HISTORY_LEN),
118
+
mean_scheduled_us: VecDeque::with_capacity(HISTORY_LEN),
119
+
slow_polls_per_interval: VecDeque::with_capacity(HISTORY_LEN),
120
+
long_delays_per_interval: VecDeque::with_capacity(HISTORY_LEN),
121
+
http_duration_ms: VecDeque::with_capacity(HISTORY_LEN),
122
+
prev: None,
123
+
prev_poll_time: None,
124
+
events_rate: 0.0,
125
+
fanout_rate: 0.0,
126
+
ws_sent_rate: 0.0,
127
+
batches_rate: 0.0,
128
+
http_rate: 0.0,
129
+
error_msg: None,
130
+
last_poll: Instant::now() - POLL_INTERVAL,
131
+
}
132
+
}
133
+
134
+
fn push_snapshot(&mut self, snap: MetricsSnapshot) {
135
+
let now = Instant::now();
136
+
137
+
// Compute rates from previous snapshot using actual elapsed time
138
+
if let (Some(prev), Some(prev_time)) = (&self.prev, self.prev_poll_time) {
139
+
let interval = prev_time.elapsed().as_secs_f64();
140
+
if interval > 0.0 {
141
+
self.events_rate =
142
+
(snap.firehose_events_total - prev.firehose_events_total) / interval;
143
+
self.fanout_rate =
144
+
(snap.writer_events_fanned_total - prev.writer_events_fanned_total) / interval;
145
+
self.ws_sent_rate =
146
+
(snap.fanout_events_sent_total - prev.fanout_events_sent_total) / interval;
147
+
self.batches_rate =
148
+
(snap.writer_batches_total - prev.writer_batches_total) / interval;
149
+
self.http_rate = (snap.http_requests_total - prev.http_requests_total) / interval;
150
+
}
151
+
152
+
// Avg batch size this interval
153
+
let batch_delta = snap.writer_batch_size_sum - prev.writer_batch_size_sum;
154
+
let count_delta = snap.writer_batch_size_count - prev.writer_batch_size_count;
155
+
let avg_batch = if count_delta > 0.0 {
156
+
batch_delta / count_delta
157
+
} else {
158
+
0.0
159
+
};
160
+
161
+
// Identity latency this interval
162
+
let dur_delta = snap.identity_duration_sum - prev.identity_duration_sum;
163
+
let dur_count = snap.identity_duration_count - prev.identity_duration_count;
164
+
let avg_latency_ms = if dur_count > 0.0 {
165
+
(dur_delta / dur_count) * 1000.0
166
+
} else {
167
+
0.0
168
+
};
169
+
170
+
// HTTP request duration this interval
171
+
let http_dur_delta = snap.http_request_duration_sum - prev.http_request_duration_sum;
172
+
let http_dur_count =
173
+
snap.http_request_duration_count - prev.http_request_duration_count;
174
+
let avg_http_ms = if http_dur_count > 0.0 {
175
+
(http_dur_delta / http_dur_count) * 1000.0
176
+
} else {
177
+
0.0
178
+
};
179
+
180
+
// Tokio metrics deltas
181
+
let slow_poll_delta = snap.tokio_slow_poll_total - prev.tokio_slow_poll_total;
182
+
let long_delay_delta = snap.tokio_long_delay_total - prev.tokio_long_delay_total;
183
+
184
+
push_ring(&mut self.events_per_sec, self.events_rate.max(0.0) as u64);
185
+
push_ring(&mut self.batch_sizes, avg_batch.max(0.0) as u64);
186
+
push_ring(&mut self.http_per_sec, self.http_rate.max(0.0) as u64);
187
+
push_ring(
188
+
&mut self.identity_latency_ms,
189
+
avg_latency_ms.max(0.0) as u64,
190
+
);
191
+
push_ring(&mut self.http_duration_ms, avg_http_ms.max(0.0) as u64);
192
+
push_ring(
193
+
&mut self.slow_polls_per_interval,
194
+
slow_poll_delta.max(0.0) as u64,
195
+
);
196
+
push_ring(
197
+
&mut self.long_delays_per_interval,
198
+
long_delay_delta.max(0.0) as u64,
199
+
);
200
+
}
201
+
202
+
// Gauge snapshots (not rates)
203
+
let high_depth = snap
204
+
.fanout_queue_depth
205
+
.iter()
206
+
.find(|(p, _)| p == "High")
207
+
.map(|(_, v)| *v as u64)
208
+
.unwrap_or(0);
209
+
let low_depth = snap
210
+
.fanout_queue_depth
211
+
.iter()
212
+
.find(|(p, _)| p == "Low")
213
+
.map(|(_, v)| *v as u64)
214
+
.unwrap_or(0);
215
+
push_ring(&mut self.fanout_high_depth, high_depth);
216
+
push_ring(&mut self.fanout_low_depth, low_depth);
217
+
218
+
// Tokio mean gauges (convert seconds to microseconds for sparkline)
219
+
push_ring(
220
+
&mut self.mean_poll_us,
221
+
(snap.tokio_mean_poll_duration_seconds * 1_000_000.0).max(0.0) as u64,
222
+
);
223
+
push_ring(
224
+
&mut self.mean_scheduled_us,
225
+
(snap.tokio_mean_scheduled_duration_seconds * 1_000_000.0).max(0.0) as u64,
226
+
);
227
+
228
+
self.prev = Some(std::mem::take(&mut self.current));
229
+
self.prev_poll_time = Some(now);
230
+
self.current = snap;
231
+
}
232
+
}
233
+
234
+
fn push_ring(ring: &mut VecDeque<u64>, val: u64) {
235
+
if ring.len() >= HISTORY_LEN {
236
+
ring.pop_front();
237
+
}
238
+
ring.push_back(val);
239
+
}
240
+
241
+
fn run_app(terminal: &mut Terminal<CrosstermBackend<io::Stdout>>, url: &str) -> anyhow::Result<()> {
242
+
let client = reqwest::blocking::Client::builder()
243
+
.timeout(Duration::from_secs(5))
244
+
.build()?;
245
+
246
+
let mut state = DashboardState::new();
247
+
248
+
loop {
249
+
if state.last_poll.elapsed() >= POLL_INTERVAL {
250
+
match client.get(url).send().and_then(|r| r.text()) {
251
+
Ok(body) => {
252
+
let snap = parse_metrics(&body);
253
+
state.push_snapshot(snap);
254
+
state.error_msg = None;
255
+
}
256
+
Err(e) => {
257
+
state.error_msg = Some(format!("fetch error: {e}"));
258
+
}
259
+
}
260
+
state.last_poll = Instant::now();
261
+
}
262
+
263
+
terminal.draw(|f| render(f, &state))?;
264
+
265
+
if event::poll(Duration::from_millis(100))? {
266
+
if let Event::Key(key) = event::read()? {
267
+
if key.kind == KeyEventKind::Press {
268
+
match key.code {
269
+
KeyCode::Char('q') | KeyCode::Esc => return Ok(()),
270
+
KeyCode::Tab => {
271
+
state.active_tab = (state.active_tab + 1) % TAB_COUNT;
272
+
}
273
+
KeyCode::BackTab => {
274
+
state.active_tab = (state.active_tab + TAB_COUNT - 1) % TAB_COUNT;
275
+
}
276
+
_ => {}
277
+
}
278
+
}
279
+
}
280
+
}
281
+
}
282
+
}
283
+
284
+
// -- Metrics snapshot --
285
+
286
+
#[derive(Default, Clone)]
287
+
struct MetricsSnapshot {
288
+
firehose_events_total: f64,
289
+
firehose_seq: f64,
290
+
firehose_connection_state: f64,
291
+
firehose_reconnects_total: f64,
292
+
firehose_events_by_type: Vec<(String, f64)>,
293
+
storage_ops: Vec<(String, String, f64)>,
294
+
writer_batches_total: f64,
295
+
writer_batch_size_sum: f64,
296
+
writer_batch_size_count: f64,
297
+
writer_events_fanned_total: f64,
298
+
writer_commits_denied: f64,
299
+
writer_commits_empty_ops: f64,
300
+
writer_commits_filtered: f64,
301
+
writer_ops_by_collection: Vec<(String, String, f64)>,
302
+
identity_resolves: Vec<(String, f64)>,
303
+
identity_duration_sum: f64,
304
+
identity_duration_count: f64,
305
+
fanout_connections_active: f64,
306
+
fanout_connections_total: f64,
307
+
fanout_lagged: Vec<(String, f64)>,
308
+
fanout_queue_depth: Vec<(String, f64)>,
309
+
fanout_connections_evicted: f64,
310
+
fanout_events_sent_total: f64,
311
+
backfill_queue_depth: f64,
312
+
backfill_repos_total: f64,
313
+
backfill_records_total: f64,
314
+
backfill_duration_sum: f64,
315
+
backfill_duration_count: f64,
316
+
backfill_failures_total: f64,
317
+
http_requests_total: f64,
318
+
/// HTTP requests by (method, path, status).
319
+
http_requests_by_route: Vec<(String, String, u16, f64)>,
320
+
/// HTTP request duration histogram sum.
321
+
http_request_duration_sum: f64,
322
+
/// HTTP request duration histogram count.
323
+
http_request_duration_count: f64,
324
+
// Tokio task metrics
325
+
tokio_instrumented_total: f64,
326
+
tokio_dropped_total: f64,
327
+
tokio_first_poll_total: f64,
328
+
tokio_first_poll_delay_seconds_total: f64,
329
+
tokio_idle_duration_seconds_total: f64,
330
+
tokio_scheduled_duration_seconds_total: f64,
331
+
tokio_poll_duration_seconds_total: f64,
332
+
tokio_slow_poll_duration_seconds_total: f64,
333
+
tokio_long_delay_duration_seconds_total: f64,
334
+
tokio_idled_total: f64,
335
+
tokio_scheduled_total: f64,
336
+
tokio_poll_total: f64,
337
+
tokio_slow_poll_total: f64,
338
+
tokio_long_delay_total: f64,
339
+
tokio_mean_poll_duration_seconds: f64,
340
+
tokio_mean_scheduled_duration_seconds: f64,
341
+
}
342
+
343
+
// -- Prometheus text parser --
344
+
345
+
fn parse_metrics(text: &str) -> MetricsSnapshot {
346
+
let mut raw: HashMap<String, f64> = HashMap::new();
347
+
let mut labeled: Vec<(String, HashMap<String, String>, f64)> = Vec::new();
348
+
349
+
for line in text.lines() {
350
+
let line = line.trim();
351
+
if line.is_empty() || line.starts_with('#') {
352
+
continue;
353
+
}
354
+
355
+
if let Some(brace_start) = line.find('{') {
356
+
let name = &line[..brace_start];
357
+
if let Some(brace_end) = line.find('}') {
358
+
let label_str = &line[brace_start + 1..brace_end];
359
+
let value_str = line[brace_end + 1..].trim();
360
+
if let Ok(val) = value_str.parse::<f64>() {
361
+
let labels = parse_labels(label_str);
362
+
labeled.push((name.to_string(), labels, val));
363
+
}
364
+
}
365
+
} else {
366
+
let parts: Vec<&str> = line.split_whitespace().collect();
367
+
if parts.len() >= 2 {
368
+
if let Ok(val) = parts[1].parse::<f64>() {
369
+
raw.insert(parts[0].to_string(), val);
370
+
}
371
+
}
372
+
}
373
+
}
374
+
375
+
let g = |name: &str| -> f64 {
376
+
raw.get(name)
377
+
.or_else(|| raw.get(&format!("{name}_total")))
378
+
.copied()
379
+
.unwrap_or(0.0)
380
+
};
381
+
382
+
let mut snap = MetricsSnapshot::default();
383
+
384
+
snap.firehose_events_total = g("ramjet_firehose_events_total");
385
+
snap.firehose_seq = g("ramjet_firehose_seq");
386
+
snap.firehose_connection_state = g("ramjet_firehose_connection_state");
387
+
snap.firehose_reconnects_total = g("ramjet_firehose_reconnects_total");
388
+
snap.writer_events_fanned_total = g("ramjet_writer_events_fanned");
389
+
snap.writer_commits_denied = g("ramjet_writer_commits_denied");
390
+
snap.writer_commits_empty_ops = g("ramjet_writer_commits_empty_ops");
391
+
snap.writer_commits_filtered = g("ramjet_writer_commits_filtered");
392
+
snap.writer_batches_total = g("ramjet_writer_batches_total");
393
+
snap.writer_batch_size_sum = g("ramjet_writer_batch_size_sum");
394
+
snap.writer_batch_size_count = g("ramjet_writer_batch_size_count");
395
+
snap.fanout_connections_active = g("ramjet_fanout_connections_active");
396
+
snap.fanout_connections_total = g("ramjet_fanout_connections_total");
397
+
snap.fanout_connections_evicted = g("ramjet_fanout_connections_evicted");
398
+
snap.fanout_events_sent_total = g("ramjet_fanout_events_sent");
399
+
snap.backfill_queue_depth = g("ramjet_backfill_queue_depth");
400
+
snap.backfill_repos_total = g("ramjet_backfill_repos_total");
401
+
snap.backfill_records_total = g("ramjet_backfill_records_total");
402
+
snap.backfill_duration_sum = g("ramjet_backfill_duration_seconds_sum");
403
+
snap.backfill_duration_count = g("ramjet_backfill_duration_seconds_count");
404
+
snap.backfill_failures_total = g("ramjet_backfill_failures_total");
405
+
snap.http_requests_total = g("ramjet_http_requests_total");
406
+
snap.http_request_duration_sum = g("ramjet_http_request_duration_seconds_sum");
407
+
snap.http_request_duration_count = g("ramjet_http_request_duration_seconds_count");
408
+
snap.identity_duration_sum = g("ramjet_identity_resolve_duration_seconds_sum");
409
+
snap.identity_duration_count = g("ramjet_identity_resolve_duration_seconds_count");
410
+
411
+
// Tokio task metrics
412
+
snap.tokio_instrumented_total = g("tokio_task_instrumented_total");
413
+
snap.tokio_dropped_total = g("tokio_task_dropped_total");
414
+
snap.tokio_first_poll_total = g("tokio_task_first_poll_total");
415
+
snap.tokio_first_poll_delay_seconds_total = g("tokio_task_first_poll_delay_seconds_total");
416
+
snap.tokio_idle_duration_seconds_total = g("tokio_task_idle_duration_seconds_total");
417
+
snap.tokio_scheduled_duration_seconds_total = g("tokio_task_scheduled_duration_seconds_total");
418
+
snap.tokio_poll_duration_seconds_total = g("tokio_task_poll_duration_seconds_total");
419
+
snap.tokio_slow_poll_duration_seconds_total = g("tokio_task_slow_poll_duration_seconds_total");
420
+
snap.tokio_long_delay_duration_seconds_total =
421
+
g("tokio_task_long_delay_duration_seconds_total");
422
+
snap.tokio_idled_total = g("tokio_task_idled_total");
423
+
snap.tokio_scheduled_total = g("tokio_task_scheduled_total");
424
+
snap.tokio_poll_total = g("tokio_task_poll_total");
425
+
snap.tokio_slow_poll_total = g("tokio_task_slow_poll_total");
426
+
snap.tokio_long_delay_total = g("tokio_task_long_delay_total");
427
+
snap.tokio_mean_poll_duration_seconds = g("tokio_task_mean_poll_duration_seconds");
428
+
snap.tokio_mean_scheduled_duration_seconds = g("tokio_task_mean_scheduled_duration_seconds");
429
+
430
+
for (name, labels, val) in &labeled {
431
+
match name.as_str() {
432
+
n if n.starts_with("ramjet_firehose_events_by_type") => {
433
+
if let Some(et) = labels.get("event_type") {
434
+
snap.firehose_events_by_type.push((et.clone(), *val));
435
+
}
436
+
}
437
+
n if n.starts_with("ramjet_storage_ops") => {
438
+
if let (Some(ks), Some(op)) = (labels.get("keyspace"), labels.get("op")) {
439
+
snap.storage_ops.push((ks.clone(), op.clone(), *val));
440
+
}
441
+
}
442
+
n if n.starts_with("ramjet_writer_ops_by_collection") => {
443
+
if let (Some(coll), Some(op)) = (labels.get("collection"), labels.get("op")) {
444
+
snap.writer_ops_by_collection
445
+
.push((coll.clone(), op.clone(), *val));
446
+
}
447
+
}
448
+
n if n.starts_with("ramjet_identity_resolves") => {
449
+
if let Some(outcome) = labels.get("outcome") {
450
+
snap.identity_resolves.push((outcome.clone(), *val));
451
+
}
452
+
}
453
+
n if n.starts_with("ramjet_fanout_lagged") => {
454
+
if let Some(priority) = labels.get("priority") {
455
+
snap.fanout_lagged.push((priority.clone(), *val));
456
+
}
457
+
}
458
+
n if n.starts_with("ramjet_fanout_queue_depth") => {
459
+
if let Some(priority) = labels.get("priority") {
460
+
snap.fanout_queue_depth.push((priority.clone(), *val));
461
+
}
462
+
}
463
+
n if n.starts_with("ramjet_http_requests_by_route") => {
464
+
if let (Some(method), Some(path), Some(status_str)) = (
465
+
labels.get("method"),
466
+
labels.get("path"),
467
+
labels.get("status"),
468
+
) {
469
+
if let Ok(status) = status_str.parse::<u16>() {
470
+
snap.http_requests_by_route.push((
471
+
method.clone(),
472
+
path.clone(),
473
+
status,
474
+
*val,
475
+
));
476
+
}
477
+
}
478
+
}
479
+
_ => {}
480
+
}
481
+
}
482
+
483
+
snap
484
+
}
485
+
486
+
fn parse_labels(s: &str) -> HashMap<String, String> {
487
+
let mut map = HashMap::new();
488
+
for part in s.split(',') {
489
+
let part = part.trim();
490
+
if let Some((k, v)) = part.split_once('=') {
491
+
let v = v.trim_matches('"');
492
+
map.insert(k.to_string(), v.to_string());
493
+
}
494
+
}
495
+
map
496
+
}
497
+
498
+
// -- Formatting helpers --
499
+
500
+
fn fmt_count(n: f64) -> String {
501
+
if n >= 1_000_000.0 {
502
+
format!("{:.1}M", n / 1_000_000.0)
503
+
} else if n >= 1_000.0 {
504
+
format!("{:.1}K", n / 1_000.0)
505
+
} else {
506
+
format!("{:.0}", n)
507
+
}
508
+
}
509
+
510
+
fn fmt_rate(n: f64) -> String {
511
+
if n >= 1000.0 {
512
+
format!("{:.1}K/s", n / 1000.0)
513
+
} else {
514
+
format!("{:.1}/s", n)
515
+
}
516
+
}
517
+
518
+
fn fmt_duration_us(us: f64) -> String {
519
+
if us >= 1_000_000.0 {
520
+
format!("{:.2}s", us / 1_000_000.0)
521
+
} else if us >= 1_000.0 {
522
+
format!("{:.1}ms", us / 1_000.0)
523
+
} else {
524
+
format!("{:.0}us", us)
525
+
}
526
+
}
527
+
528
+
fn fmt_duration_s(s: f64) -> String {
529
+
fmt_duration_us(s * 1_000_000.0)
530
+
}
531
+
532
+
// -- Rendering --
533
+
534
+
fn render(f: &mut ratatui::Frame, state: &DashboardState) {
535
+
let area = f.area();
536
+
537
+
let outer = Layout::default()
538
+
.direction(Direction::Vertical)
539
+
.constraints([
540
+
Constraint::Length(3), // Header
541
+
Constraint::Length(1), // Tab bar
542
+
Constraint::Min(0), // Content
543
+
])
544
+
.split(area);
545
+
546
+
render_header(f, outer[0], state);
547
+
render_tab_bar(f, outer[1], state);
548
+
549
+
match state.active_tab {
550
+
TAB_PIPELINE => render_pipeline_tab(f, outer[2], state),
551
+
TAB_PROCESS => render_process_tab(f, outer[2], state),
552
+
TAB_HTTP => render_http_tab(f, outer[2], state),
553
+
_ => {}
554
+
}
555
+
}
556
+
557
+
fn render_header(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
558
+
let snap = &state.current;
559
+
560
+
let connected = if snap.firehose_connection_state >= 1.0 {
561
+
Span::styled(
562
+
"CONNECTED",
563
+
Style::default()
564
+
.fg(Color::Green)
565
+
.add_modifier(Modifier::BOLD),
566
+
)
567
+
} else {
568
+
Span::styled(
569
+
"DISCONNECTED",
570
+
Style::default().fg(Color::Red).add_modifier(Modifier::BOLD),
571
+
)
572
+
};
573
+
574
+
let status = if let Some(ref err) = state.error_msg {
575
+
Span::styled(format!(" {err}"), Style::default().fg(Color::Red))
576
+
} else {
577
+
Span::styled(
578
+
format!(" {}s ago", state.last_poll.elapsed().as_secs()),
579
+
Style::default().fg(Color::DarkGray),
580
+
)
581
+
};
582
+
583
+
let info = Span::raw(format!(
584
+
" seq: {:.0} events: {} http: {} reconn: {:.0}",
585
+
snap.firehose_seq,
586
+
fmt_count(snap.firehose_events_total),
587
+
fmt_count(snap.http_requests_total),
588
+
snap.firehose_reconnects_total,
589
+
));
590
+
591
+
let header = Paragraph::new(Line::from(vec![
592
+
Span::styled(
593
+
"rjtop ",
594
+
Style::default()
595
+
.fg(Color::Cyan)
596
+
.add_modifier(Modifier::BOLD),
597
+
),
598
+
connected,
599
+
info,
600
+
status,
601
+
]))
602
+
.block(Block::default().borders(Borders::BOTTOM));
603
+
604
+
f.render_widget(header, area);
605
+
}
606
+
607
+
fn render_tab_bar(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
608
+
let titles = vec![" Pipeline ", " Process ", " HTTP "];
609
+
let tabs = Tabs::new(titles)
610
+
.select(state.active_tab)
611
+
.style(Style::default().fg(Color::DarkGray))
612
+
.highlight_style(
613
+
Style::default()
614
+
.fg(Color::Cyan)
615
+
.add_modifier(Modifier::BOLD),
616
+
)
617
+
.divider(Span::raw("|"));
618
+
f.render_widget(tabs, area);
619
+
}
620
+
621
+
// ============================
622
+
// Tab 1: Pipeline (existing)
623
+
// ============================
624
+
625
+
fn render_pipeline_tab(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
626
+
let snap = &state.current;
627
+
628
+
let cols = Layout::default()
629
+
.direction(Direction::Horizontal)
630
+
.constraints([Constraint::Percentage(50), Constraint::Percentage(50)])
631
+
.split(area);
632
+
633
+
let left = Layout::default()
634
+
.direction(Direction::Vertical)
635
+
.constraints([
636
+
Constraint::Length(7), // Firehose sparkline + stats
637
+
Constraint::Length(8), // Writer sparkline + stats + skip breakdown
638
+
Constraint::Length(9), // Events by type bar chart
639
+
Constraint::Min(4), // Collections table
640
+
])
641
+
.split(cols[0]);
642
+
643
+
let right = Layout::default()
644
+
.direction(Direction::Vertical)
645
+
.constraints([
646
+
Constraint::Length(9), // Storage ops bar chart
647
+
Constraint::Length(9), // Fan-out with sparklines
648
+
Constraint::Length(7), // Identity with sparkline
649
+
Constraint::Min(4), // Backfill
650
+
])
651
+
.split(cols[1]);
652
+
653
+
render_firehose(f, left[0], state);
654
+
render_writer(f, left[1], state);
655
+
render_events_by_type(f, left[2], snap);
656
+
render_collections(f, left[3], snap);
657
+
658
+
render_storage_bar(f, right[0], snap);
659
+
render_fanout(f, right[1], state);
660
+
render_identity(f, right[2], state);
661
+
render_backfill(f, right[3], state);
662
+
}
663
+
664
+
fn render_firehose(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
665
+
let block = Block::default()
666
+
.title(format!(" Firehose {} ", fmt_rate(state.events_rate)))
667
+
.borders(Borders::ALL)
668
+
.border_style(Style::default().fg(Color::Yellow));
669
+
670
+
let inner = block.inner(area);
671
+
f.render_widget(block, area);
672
+
673
+
let chunks = Layout::default()
674
+
.direction(Direction::Vertical)
675
+
.constraints([Constraint::Length(3), Constraint::Min(1)])
676
+
.split(inner);
677
+
678
+
// Sparkline
679
+
let data: Vec<u64> = state.events_per_sec.iter().copied().collect();
680
+
let sparkline = Sparkline::default()
681
+
.data(&data)
682
+
.style(Style::default().fg(Color::Yellow));
683
+
f.render_widget(sparkline, chunks[0]);
684
+
685
+
// Stats
686
+
let snap = &state.current;
687
+
let by_type: Vec<String> = snap
688
+
.firehose_events_by_type
689
+
.iter()
690
+
.map(|(t, v)| format!("{}: {}", t, fmt_count(*v)))
691
+
.collect();
692
+
let stats = Paragraph::new(Line::from(vec![
693
+
Span::styled(" total: ", Style::default().fg(Color::DarkGray)),
694
+
Span::raw(fmt_count(snap.firehose_events_total)),
695
+
Span::raw(" "),
696
+
Span::raw(by_type.join(" ")),
697
+
]));
698
+
f.render_widget(stats, chunks[1]);
699
+
}
700
+
701
+
fn render_writer(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
702
+
let avg_batch = if state.current.writer_batch_size_count > 0.0 {
703
+
state.current.writer_batch_size_sum / state.current.writer_batch_size_count
704
+
} else {
705
+
0.0
706
+
};
707
+
708
+
let block = Block::default()
709
+
.title(format!(
710
+
" Writer {} avg {:.0}/batch fan-out {} ",
711
+
fmt_rate(state.batches_rate),
712
+
avg_batch,
713
+
fmt_rate(state.fanout_rate),
714
+
))
715
+
.borders(Borders::ALL)
716
+
.border_style(Style::default().fg(Color::Blue));
717
+
718
+
let inner = block.inner(area);
719
+
f.render_widget(block, area);
720
+
721
+
let chunks = Layout::default()
722
+
.direction(Direction::Vertical)
723
+
.constraints([
724
+
Constraint::Length(3),
725
+
Constraint::Length(1),
726
+
Constraint::Min(1),
727
+
])
728
+
.split(inner);
729
+
730
+
// Batch size sparkline
731
+
let data: Vec<u64> = state.batch_sizes.iter().copied().collect();
732
+
let sparkline = Sparkline::default()
733
+
.data(&data)
734
+
.style(Style::default().fg(Color::Blue));
735
+
f.render_widget(sparkline, chunks[0]);
736
+
737
+
// Stats line
738
+
let snap = &state.current;
739
+
let stats = Paragraph::new(Line::from(vec![
740
+
Span::styled(" batches: ", Style::default().fg(Color::DarkGray)),
741
+
Span::raw(fmt_count(snap.writer_batches_total)),
742
+
Span::styled(" events: ", Style::default().fg(Color::DarkGray)),
743
+
Span::raw(fmt_count(snap.writer_batch_size_sum)),
744
+
Span::styled(" fanned: ", Style::default().fg(Color::DarkGray)),
745
+
Span::raw(fmt_count(snap.writer_events_fanned_total)),
746
+
Span::styled(" backfill: ", Style::default().fg(Color::DarkGray)),
747
+
Span::raw(format!("{:.0}", snap.backfill_queue_depth)),
748
+
]));
749
+
f.render_widget(stats, chunks[1]);
750
+
751
+
// Skip breakdown line
752
+
let skips = Paragraph::new(Line::from(vec![
753
+
Span::styled(" skipped: ", Style::default().fg(Color::DarkGray)),
754
+
Span::styled("empty ", Style::default().fg(Color::Yellow)),
755
+
Span::raw(fmt_count(snap.writer_commits_empty_ops)),
756
+
Span::styled(" denied ", Style::default().fg(Color::Red)),
757
+
Span::raw(fmt_count(snap.writer_commits_denied)),
758
+
Span::styled(" filtered ", Style::default().fg(Color::Magenta)),
759
+
Span::raw(fmt_count(snap.writer_commits_filtered)),
760
+
]));
761
+
f.render_widget(skips, chunks[2]);
762
+
}
763
+
764
+
fn render_events_by_type(f: &mut ratatui::Frame, area: Rect, snap: &MetricsSnapshot) {
765
+
let total: f64 = snap.firehose_events_by_type.iter().map(|(_, v)| v).sum();
766
+
767
+
let rows: Vec<Row> = snap
768
+
.firehose_events_by_type
769
+
.iter()
770
+
.map(|(name, val)| {
771
+
let color = match name.as_str() {
772
+
"#commit" => Color::Green,
773
+
"#identity" => Color::Cyan,
774
+
"#account" => Color::Magenta,
775
+
_ => Color::White,
776
+
};
777
+
let pct = if total > 0.0 {
778
+
format!("{:.1}%", val / total * 100.0)
779
+
} else {
780
+
"—".to_string()
781
+
};
782
+
Row::new(vec![name.clone(), fmt_count(*val), pct]).style(Style::default().fg(color))
783
+
})
784
+
.collect();
785
+
786
+
let table = Table::new(
787
+
rows,
788
+
[
789
+
Constraint::Min(12),
790
+
Constraint::Length(10),
791
+
Constraint::Length(8),
792
+
],
793
+
)
794
+
.header(
795
+
Row::new(vec!["Type", "Count", "%"]).style(Style::default().add_modifier(Modifier::BOLD)),
796
+
)
797
+
.block(
798
+
Block::default()
799
+
.title(" Events by Type ")
800
+
.borders(Borders::ALL)
801
+
.border_style(Style::default().fg(Color::Green)),
802
+
);
803
+
804
+
f.render_widget(table, area);
805
+
}
806
+
807
+
fn render_collections(f: &mut ratatui::Frame, area: Rect, snap: &MetricsSnapshot) {
808
+
let rows: Vec<Row> = snap
809
+
.writer_ops_by_collection
810
+
.iter()
811
+
.map(|(coll, op, val)| {
812
+
let style = match op.as_str() {
813
+
"Create" => Style::default().fg(Color::Green),
814
+
"Update" => Style::default().fg(Color::Yellow),
815
+
"Delete" => Style::default().fg(Color::Red),
816
+
_ => Style::default(),
817
+
};
818
+
Row::new(vec![coll.clone(), op.clone(), fmt_count(*val)]).style(style)
819
+
})
820
+
.collect();
821
+
822
+
let table = Table::new(
823
+
rows,
824
+
[
825
+
Constraint::Min(28),
826
+
Constraint::Length(8),
827
+
Constraint::Length(10),
828
+
],
829
+
)
830
+
.header(
831
+
Row::new(vec!["Collection", "Op", "Count"])
832
+
.style(Style::default().add_modifier(Modifier::BOLD)),
833
+
)
834
+
.block(
835
+
Block::default()
836
+
.title(" Collections ")
837
+
.borders(Borders::ALL)
838
+
.border_style(Style::default().fg(Color::Magenta)),
839
+
);
840
+
841
+
f.render_widget(table, area);
842
+
}
843
+
844
+
fn render_storage_bar(f: &mut ratatui::Frame, area: Rect, snap: &MetricsSnapshot) {
845
+
// Aggregate by keyspace (sum all ops per keyspace)
846
+
let mut by_keyspace: HashMap<String, f64> = HashMap::new();
847
+
for (ks, _op, val) in &snap.storage_ops {
848
+
*by_keyspace.entry(ks.clone()).or_default() += val;
849
+
}
850
+
let mut sorted: Vec<(String, f64)> = by_keyspace.into_iter().collect();
851
+
sorted.sort_by(|a, b| {
852
+
b.1.partial_cmp(&a.1)
853
+
.unwrap_or(std::cmp::Ordering::Equal)
854
+
.then_with(|| a.0.cmp(&b.0))
855
+
});
856
+
857
+
let total: f64 = sorted.iter().map(|(_, v)| v).sum();
858
+
859
+
let rows: Vec<Row> = sorted
860
+
.iter()
861
+
.map(|(ks, val)| {
862
+
let pct = if total > 0.0 {
863
+
format!("{:.1}%", val / total * 100.0)
864
+
} else {
865
+
"—".to_string()
866
+
};
867
+
Row::new(vec![ks.clone(), fmt_count(*val), pct])
868
+
.style(Style::default().fg(Color::Yellow))
869
+
})
870
+
.collect();
871
+
872
+
let table = Table::new(
873
+
rows,
874
+
[
875
+
Constraint::Min(14),
876
+
Constraint::Length(10),
877
+
Constraint::Length(8),
878
+
],
879
+
)
880
+
.header(
881
+
Row::new(vec!["Keyspace", "Ops", "%"]).style(Style::default().add_modifier(Modifier::BOLD)),
882
+
)
883
+
.block(
884
+
Block::default()
885
+
.title(" Storage Ops ")
886
+
.borders(Borders::ALL)
887
+
.border_style(Style::default().fg(Color::Red)),
888
+
);
889
+
890
+
f.render_widget(table, area);
891
+
}
892
+
893
+
fn render_fanout(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
894
+
let snap = &state.current;
895
+
let block = Block::default()
896
+
.title(format!(
897
+
" Fan-out {} active ws-sent {} ",
898
+
snap.fanout_connections_active as u64,
899
+
fmt_rate(state.ws_sent_rate),
900
+
))
901
+
.borders(Borders::ALL)
902
+
.border_style(Style::default().fg(Color::Green));
903
+
904
+
let inner = block.inner(area);
905
+
f.render_widget(block, area);
906
+
907
+
let rows = Layout::default()
908
+
.direction(Direction::Vertical)
909
+
.constraints([
910
+
Constraint::Length(2), // High priority sparkline
911
+
Constraint::Length(2), // Low priority sparkline
912
+
Constraint::Min(1), // Stats
913
+
])
914
+
.split(inner);
915
+
916
+
// High priority queue sparkline
917
+
let high_data: Vec<u64> = state.fanout_high_depth.iter().copied().collect();
918
+
let high_current = high_data.last().copied().unwrap_or(0);
919
+
let high_label = Paragraph::new(Line::from(vec![
920
+
Span::styled(" High ", Style::default().fg(Color::Yellow)),
921
+
Span::raw(format!("({high_current})")),
922
+
]));
923
+
let high_split = Layout::default()
924
+
.direction(Direction::Horizontal)
925
+
.constraints([Constraint::Length(14), Constraint::Min(1)])
926
+
.split(rows[0]);
927
+
f.render_widget(high_label, high_split[0]);
928
+
let high_spark = Sparkline::default()
929
+
.data(&high_data)
930
+
.style(Style::default().fg(Color::Yellow));
931
+
f.render_widget(high_spark, high_split[1]);
932
+
933
+
// Low priority queue sparkline
934
+
let low_data: Vec<u64> = state.fanout_low_depth.iter().copied().collect();
935
+
let low_current = low_data.last().copied().unwrap_or(0);
936
+
let low_label = Paragraph::new(Line::from(vec![
937
+
Span::styled(" Low ", Style::default().fg(Color::Cyan)),
938
+
Span::raw(format!("({low_current})")),
939
+
]));
940
+
let low_split = Layout::default()
941
+
.direction(Direction::Horizontal)
942
+
.constraints([Constraint::Length(14), Constraint::Min(1)])
943
+
.split(rows[1]);
944
+
f.render_widget(low_label, low_split[0]);
945
+
let low_spark = Sparkline::default()
946
+
.data(&low_data)
947
+
.style(Style::default().fg(Color::Cyan));
948
+
f.render_widget(low_spark, low_split[1]);
949
+
950
+
// Lagged stats
951
+
let lagged_parts: Vec<String> = snap
952
+
.fanout_lagged
953
+
.iter()
954
+
.map(|(p, v)| format!("{p}: {}", fmt_count(*v)))
955
+
.collect();
956
+
let lagged_text = if lagged_parts.is_empty() {
957
+
"none".to_string()
958
+
} else {
959
+
lagged_parts.join(" ")
960
+
};
961
+
let mut stats_spans = vec![
962
+
Span::styled(" conn: ", Style::default().fg(Color::DarkGray)),
963
+
Span::raw(format!("{:.0}", snap.fanout_connections_total)),
964
+
Span::styled(" sent: ", Style::default().fg(Color::DarkGray)),
965
+
Span::raw(fmt_count(snap.fanout_events_sent_total)),
966
+
Span::styled(" lagged: ", Style::default().fg(Color::DarkGray)),
967
+
Span::raw(lagged_text),
968
+
];
969
+
if snap.fanout_connections_evicted > 0.0 {
970
+
stats_spans.push(Span::styled(" evicted: ", Style::default().fg(Color::Red)));
971
+
stats_spans.push(Span::styled(
972
+
format!("{:.0}", snap.fanout_connections_evicted),
973
+
Style::default().fg(Color::Red),
974
+
));
975
+
}
976
+
let stats = Paragraph::new(Line::from(stats_spans));
977
+
f.render_widget(stats, rows[2]);
978
+
}
979
+
980
+
fn render_identity(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
981
+
let snap = &state.current;
982
+
983
+
let total_resolves: f64 = snap.identity_resolves.iter().map(|(_, v)| v).sum();
984
+
let avg_ms = if snap.identity_duration_count > 0.0 {
985
+
(snap.identity_duration_sum / snap.identity_duration_count) * 1000.0
986
+
} else {
987
+
0.0
988
+
};
989
+
990
+
let block = Block::default()
991
+
.title(format!(
992
+
" Identity {} resolves {:.0}ms avg ",
993
+
fmt_count(total_resolves),
994
+
avg_ms,
995
+
))
996
+
.borders(Borders::ALL)
997
+
.border_style(Style::default().fg(Color::Cyan));
998
+
999
+
let inner = block.inner(area);
1000
+
f.render_widget(block, area);
1001
+
1002
+
let chunks = Layout::default()
1003
+
.direction(Direction::Vertical)
1004
+
.constraints([Constraint::Length(2), Constraint::Min(1)])
1005
+
.split(inner);
1006
+
1007
+
// Latency sparkline
1008
+
let lat_data: Vec<u64> = state.identity_latency_ms.iter().copied().collect();
1009
+
let lat_label = Paragraph::new(Line::from(vec![Span::styled(
1010
+
" latency ",
1011
+
Style::default().fg(Color::DarkGray),
1012
+
)]));
1013
+
let lat_split = Layout::default()
1014
+
.direction(Direction::Horizontal)
1015
+
.constraints([Constraint::Length(12), Constraint::Min(1)])
1016
+
.split(chunks[0]);
1017
+
f.render_widget(lat_label, lat_split[0]);
1018
+
let lat_spark = Sparkline::default()
1019
+
.data(&lat_data)
1020
+
.style(Style::default().fg(Color::Cyan));
1021
+
f.render_widget(lat_spark, lat_split[1]);
1022
+
1023
+
// Outcome breakdown with inline gauges
1024
+
let success = snap
1025
+
.identity_resolves
1026
+
.iter()
1027
+
.find(|(o, _)| o == "Success")
1028
+
.map(|(_, v)| *v)
1029
+
.unwrap_or(0.0);
1030
+
let failure = snap
1031
+
.identity_resolves
1032
+
.iter()
1033
+
.find(|(o, _)| o == "Failure")
1034
+
.map(|(_, v)| *v)
1035
+
.unwrap_or(0.0);
1036
+
let ratio = if total_resolves > 0.0 {
1037
+
success / total_resolves
1038
+
} else {
1039
+
1.0
1040
+
};
1041
+
1042
+
let gauge_split = Layout::default()
1043
+
.direction(Direction::Horizontal)
1044
+
.constraints([Constraint::Min(20), Constraint::Length(24)])
1045
+
.split(chunks[1]);
1046
+
1047
+
let gauge = Gauge::default()
1048
+
.gauge_style(Style::default().fg(Color::Green))
1049
+
.ratio(ratio.clamp(0.0, 1.0))
1050
+
.label(format!(
1051
+
"{:.0}% ok ({} / {})",
1052
+
ratio * 100.0,
1053
+
fmt_count(success),
1054
+
fmt_count(total_resolves)
1055
+
));
1056
+
f.render_widget(gauge, gauge_split[0]);
1057
+
1058
+
let fail_text = Paragraph::new(Line::from(vec![
1059
+
Span::styled(" fail: ", Style::default().fg(Color::Red)),
1060
+
Span::raw(fmt_count(failure)),
1061
+
]));
1062
+
f.render_widget(fail_text, gauge_split[1]);
1063
+
}
1064
+
1065
+
fn render_backfill(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1066
+
let snap = &state.current;
1067
+
let avg_duration = if snap.backfill_duration_count > 0.0 {
1068
+
snap.backfill_duration_sum / snap.backfill_duration_count
1069
+
} else {
1070
+
0.0
1071
+
};
1072
+
1073
+
let queue_style = if snap.backfill_queue_depth > 0.0 {
1074
+
Style::default().fg(Color::Yellow)
1075
+
} else {
1076
+
Style::default().fg(Color::DarkGray)
1077
+
};
1078
+
1079
+
let block = Block::default()
1080
+
.title(format!(
1081
+
" Backfill {} queued ",
1082
+
snap.backfill_queue_depth as u64,
1083
+
))
1084
+
.borders(Borders::ALL)
1085
+
.border_style(if snap.backfill_queue_depth > 0.0 {
1086
+
Style::default().fg(Color::Yellow)
1087
+
} else {
1088
+
Style::default().fg(Color::DarkGray)
1089
+
});
1090
+
1091
+
let inner = block.inner(area);
1092
+
f.render_widget(block, area);
1093
+
1094
+
let chunks = Layout::default()
1095
+
.direction(Direction::Vertical)
1096
+
.constraints([
1097
+
Constraint::Length(1),
1098
+
Constraint::Length(1),
1099
+
Constraint::Min(1),
1100
+
])
1101
+
.split(inner);
1102
+
1103
+
let line1 = Paragraph::new(Line::from(vec![
1104
+
Span::styled(" repos: ", Style::default().fg(Color::DarkGray)),
1105
+
Span::raw(fmt_count(snap.backfill_repos_total)),
1106
+
Span::styled(" records: ", Style::default().fg(Color::DarkGray)),
1107
+
Span::raw(fmt_count(snap.backfill_records_total)),
1108
+
Span::styled(" avg: ", Style::default().fg(Color::DarkGray)),
1109
+
Span::raw(fmt_duration_s(avg_duration)),
1110
+
]));
1111
+
f.render_widget(line1, chunks[0]);
1112
+
1113
+
let mut line2_spans = vec![
1114
+
Span::styled(" queue: ", Style::default().fg(Color::DarkGray)),
1115
+
Span::styled(format!("{:.0}", snap.backfill_queue_depth), queue_style),
1116
+
];
1117
+
if snap.backfill_failures_total > 0.0 {
1118
+
line2_spans.push(Span::styled(
1119
+
" failures: ",
1120
+
Style::default().fg(Color::Red),
1121
+
));
1122
+
line2_spans.push(Span::styled(
1123
+
fmt_count(snap.backfill_failures_total),
1124
+
Style::default().fg(Color::Red),
1125
+
));
1126
+
}
1127
+
let line2 = Paragraph::new(Line::from(line2_spans));
1128
+
f.render_widget(line2, chunks[1]);
1129
+
1130
+
let line3 = Paragraph::new(Line::from(vec![
1131
+
Span::styled(" total duration: ", Style::default().fg(Color::DarkGray)),
1132
+
Span::raw(fmt_duration_s(snap.backfill_duration_sum)),
1133
+
]));
1134
+
f.render_widget(line3, chunks[2]);
1135
+
}
1136
+
1137
+
// ============================
1138
+
// Tab 2: Process (tokio metrics)
1139
+
// ============================
1140
+
1141
+
fn render_process_tab(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1142
+
let cols = Layout::default()
1143
+
.direction(Direction::Horizontal)
1144
+
.constraints([Constraint::Percentage(50), Constraint::Percentage(50)])
1145
+
.split(area);
1146
+
1147
+
let left = Layout::default()
1148
+
.direction(Direction::Vertical)
1149
+
.constraints([
1150
+
Constraint::Length(8), // Task lifecycle
1151
+
Constraint::Length(9), // Poll duration sparkline + stats
1152
+
Constraint::Length(9), // Scheduling delay sparkline + stats
1153
+
Constraint::Min(4), // Duration breakdown table
1154
+
])
1155
+
.split(cols[0]);
1156
+
1157
+
let right = Layout::default()
1158
+
.direction(Direction::Vertical)
1159
+
.constraints([
1160
+
Constraint::Length(9), // Slow polls sparkline + stats
1161
+
Constraint::Length(9), // Long delays sparkline + stats
1162
+
Constraint::Min(4), // Event counts table
1163
+
])
1164
+
.split(cols[1]);
1165
+
1166
+
render_task_lifecycle(f, left[0], state);
1167
+
render_poll_duration(f, left[1], state);
1168
+
render_scheduled_duration(f, left[2], state);
1169
+
render_duration_breakdown(f, left[3], state);
1170
+
1171
+
render_slow_polls(f, right[0], state);
1172
+
render_long_delays(f, right[1], state);
1173
+
render_event_counts(f, right[2], state);
1174
+
}
1175
+
1176
+
fn render_task_lifecycle(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1177
+
let snap = &state.current;
1178
+
let active = snap.tokio_instrumented_total - snap.tokio_dropped_total;
1179
+
1180
+
let block = Block::default()
1181
+
.title(format!(" Tasks {:.0} active ", active.max(0.0),))
1182
+
.borders(Borders::ALL)
1183
+
.border_style(Style::default().fg(Color::Cyan));
1184
+
1185
+
let inner = block.inner(area);
1186
+
f.render_widget(block, area);
1187
+
1188
+
let chunks = Layout::default()
1189
+
.direction(Direction::Vertical)
1190
+
.constraints([
1191
+
Constraint::Length(1),
1192
+
Constraint::Length(1),
1193
+
Constraint::Length(1),
1194
+
Constraint::Min(1),
1195
+
])
1196
+
.split(inner);
1197
+
1198
+
let line1 = Paragraph::new(Line::from(vec![
1199
+
Span::styled(" instrumented: ", Style::default().fg(Color::DarkGray)),
1200
+
Span::raw(fmt_count(snap.tokio_instrumented_total)),
1201
+
Span::styled(" dropped: ", Style::default().fg(Color::DarkGray)),
1202
+
Span::raw(fmt_count(snap.tokio_dropped_total)),
1203
+
]));
1204
+
f.render_widget(line1, chunks[0]);
1205
+
1206
+
let line2 = Paragraph::new(Line::from(vec![
1207
+
Span::styled(" first polled: ", Style::default().fg(Color::DarkGray)),
1208
+
Span::raw(fmt_count(snap.tokio_first_poll_total)),
1209
+
Span::styled(" total polls: ", Style::default().fg(Color::DarkGray)),
1210
+
Span::raw(fmt_count(snap.tokio_poll_total)),
1211
+
]));
1212
+
f.render_widget(line2, chunks[1]);
1213
+
1214
+
let line3 = Paragraph::new(Line::from(vec![
1215
+
Span::styled(" scheduled: ", Style::default().fg(Color::DarkGray)),
1216
+
Span::raw(fmt_count(snap.tokio_scheduled_total)),
1217
+
Span::styled(" idled: ", Style::default().fg(Color::DarkGray)),
1218
+
Span::raw(fmt_count(snap.tokio_idled_total)),
1219
+
]));
1220
+
f.render_widget(line3, chunks[2]);
1221
+
1222
+
let pct_slow = if snap.tokio_poll_total > 0.0 {
1223
+
snap.tokio_slow_poll_total / snap.tokio_poll_total * 100.0
1224
+
} else {
1225
+
0.0
1226
+
};
1227
+
let line4 = Paragraph::new(Line::from(vec![
1228
+
Span::styled(" slow polls: ", Style::default().fg(Color::DarkGray)),
1229
+
Span::styled(
1230
+
fmt_count(snap.tokio_slow_poll_total),
1231
+
Style::default().fg(if pct_slow > 1.0 {
1232
+
Color::Red
1233
+
} else if pct_slow > 0.1 {
1234
+
Color::Yellow
1235
+
} else {
1236
+
Color::White
1237
+
}),
1238
+
),
1239
+
Span::styled(
1240
+
format!(" ({pct_slow:.2}%)"),
1241
+
Style::default().fg(Color::DarkGray),
1242
+
),
1243
+
Span::styled(" long delays: ", Style::default().fg(Color::DarkGray)),
1244
+
Span::raw(fmt_count(snap.tokio_long_delay_total)),
1245
+
]));
1246
+
f.render_widget(line4, chunks[3]);
1247
+
}
1248
+
1249
+
fn render_poll_duration(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1250
+
let snap = &state.current;
1251
+
let mean_us = snap.tokio_mean_poll_duration_seconds * 1_000_000.0;
1252
+
1253
+
let block = Block::default()
1254
+
.title(format!(
1255
+
" Poll Duration mean {} ",
1256
+
fmt_duration_us(mean_us),
1257
+
))
1258
+
.borders(Borders::ALL)
1259
+
.border_style(Style::default().fg(Color::Green));
1260
+
1261
+
let inner = block.inner(area);
1262
+
f.render_widget(block, area);
1263
+
1264
+
let chunks = Layout::default()
1265
+
.direction(Direction::Vertical)
1266
+
.constraints([Constraint::Length(3), Constraint::Min(1)])
1267
+
.split(inner);
1268
+
1269
+
// Mean poll duration sparkline (in microseconds)
1270
+
let data: Vec<u64> = state.mean_poll_us.iter().copied().collect();
1271
+
let label = Paragraph::new(Line::from(vec![Span::styled(
1272
+
" mean ",
1273
+
Style::default().fg(Color::DarkGray),
1274
+
)]));
1275
+
let split = Layout::default()
1276
+
.direction(Direction::Horizontal)
1277
+
.constraints([Constraint::Length(8), Constraint::Min(1)])
1278
+
.split(chunks[0]);
1279
+
f.render_widget(label, split[0]);
1280
+
let sparkline = Sparkline::default()
1281
+
.data(&data)
1282
+
.style(Style::default().fg(Color::Green));
1283
+
f.render_widget(sparkline, split[1]);
1284
+
1285
+
let stats = Paragraph::new(Line::from(vec![
1286
+
Span::styled(" total: ", Style::default().fg(Color::DarkGray)),
1287
+
Span::raw(fmt_duration_s(snap.tokio_poll_duration_seconds_total)),
1288
+
Span::styled(" polls: ", Style::default().fg(Color::DarkGray)),
1289
+
Span::raw(fmt_count(snap.tokio_poll_total)),
1290
+
]));
1291
+
f.render_widget(stats, chunks[1]);
1292
+
}
1293
+
1294
+
fn render_scheduled_duration(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1295
+
let snap = &state.current;
1296
+
let mean_us = snap.tokio_mean_scheduled_duration_seconds * 1_000_000.0;
1297
+
1298
+
let block = Block::default()
1299
+
.title(format!(
1300
+
" Scheduling Delay mean {} ",
1301
+
fmt_duration_us(mean_us),
1302
+
))
1303
+
.borders(Borders::ALL)
1304
+
.border_style(Style::default().fg(Color::Yellow));
1305
+
1306
+
let inner = block.inner(area);
1307
+
f.render_widget(block, area);
1308
+
1309
+
let chunks = Layout::default()
1310
+
.direction(Direction::Vertical)
1311
+
.constraints([Constraint::Length(3), Constraint::Min(1)])
1312
+
.split(inner);
1313
+
1314
+
// Mean scheduled duration sparkline (in microseconds)
1315
+
let data: Vec<u64> = state.mean_scheduled_us.iter().copied().collect();
1316
+
let label = Paragraph::new(Line::from(vec![Span::styled(
1317
+
" mean ",
1318
+
Style::default().fg(Color::DarkGray),
1319
+
)]));
1320
+
let split = Layout::default()
1321
+
.direction(Direction::Horizontal)
1322
+
.constraints([Constraint::Length(8), Constraint::Min(1)])
1323
+
.split(chunks[0]);
1324
+
f.render_widget(label, split[0]);
1325
+
let sparkline = Sparkline::default()
1326
+
.data(&data)
1327
+
.style(Style::default().fg(Color::Yellow));
1328
+
f.render_widget(sparkline, split[1]);
1329
+
1330
+
let stats = Paragraph::new(Line::from(vec![
1331
+
Span::styled(" total: ", Style::default().fg(Color::DarkGray)),
1332
+
Span::raw(fmt_duration_s(snap.tokio_scheduled_duration_seconds_total)),
1333
+
Span::styled(" schedules: ", Style::default().fg(Color::DarkGray)),
1334
+
Span::raw(fmt_count(snap.tokio_scheduled_total)),
1335
+
]));
1336
+
f.render_widget(stats, chunks[1]);
1337
+
}
1338
+
1339
+
fn render_duration_breakdown(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1340
+
let snap = &state.current;
1341
+
1342
+
let durations = vec![
1343
+
("Poll", snap.tokio_poll_duration_seconds_total, Color::Green),
1344
+
(
1345
+
"Scheduled",
1346
+
snap.tokio_scheduled_duration_seconds_total,
1347
+
Color::Yellow,
1348
+
),
1349
+
("Idle", snap.tokio_idle_duration_seconds_total, Color::Cyan),
1350
+
(
1351
+
"First poll delay",
1352
+
snap.tokio_first_poll_delay_seconds_total,
1353
+
Color::Magenta,
1354
+
),
1355
+
(
1356
+
"Slow poll",
1357
+
snap.tokio_slow_poll_duration_seconds_total,
1358
+
Color::Red,
1359
+
),
1360
+
(
1361
+
"Long delay",
1362
+
snap.tokio_long_delay_duration_seconds_total,
1363
+
Color::Red,
1364
+
),
1365
+
];
1366
+
1367
+
let total: f64 = snap.tokio_poll_duration_seconds_total
1368
+
+ snap.tokio_scheduled_duration_seconds_total
1369
+
+ snap.tokio_idle_duration_seconds_total;
1370
+
1371
+
let rows: Vec<Row> = durations
1372
+
.iter()
1373
+
.map(|(name, val, color)| {
1374
+
let pct = if total > 0.0 {
1375
+
format!("{:.1}%", val / total * 100.0)
1376
+
} else {
1377
+
"—".to_string()
1378
+
};
1379
+
Row::new(vec![name.to_string(), fmt_duration_s(*val), pct])
1380
+
.style(Style::default().fg(*color))
1381
+
})
1382
+
.collect();
1383
+
1384
+
let table = Table::new(
1385
+
rows,
1386
+
[
1387
+
Constraint::Min(16),
1388
+
Constraint::Length(12),
1389
+
Constraint::Length(8),
1390
+
],
1391
+
)
1392
+
.header(
1393
+
Row::new(vec!["Category", "Total", "%"])
1394
+
.style(Style::default().add_modifier(Modifier::BOLD)),
1395
+
)
1396
+
.block(
1397
+
Block::default()
1398
+
.title(" Duration Breakdown ")
1399
+
.borders(Borders::ALL)
1400
+
.border_style(Style::default().fg(Color::DarkGray)),
1401
+
);
1402
+
1403
+
f.render_widget(table, area);
1404
+
}
1405
+
1406
+
fn render_slow_polls(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1407
+
let snap = &state.current;
1408
+
1409
+
let block = Block::default()
1410
+
.title(format!(
1411
+
" Slow Polls {} total ",
1412
+
fmt_count(snap.tokio_slow_poll_total),
1413
+
))
1414
+
.borders(Borders::ALL)
1415
+
.border_style(Style::default().fg(Color::Red));
1416
+
1417
+
let inner = block.inner(area);
1418
+
f.render_widget(block, area);
1419
+
1420
+
let chunks = Layout::default()
1421
+
.direction(Direction::Vertical)
1422
+
.constraints([Constraint::Length(3), Constraint::Min(1)])
1423
+
.split(inner);
1424
+
1425
+
// Slow polls per interval sparkline
1426
+
let data: Vec<u64> = state.slow_polls_per_interval.iter().copied().collect();
1427
+
let label = Paragraph::new(Line::from(vec![Span::styled(
1428
+
" /interval ",
1429
+
Style::default().fg(Color::DarkGray),
1430
+
)]));
1431
+
let split = Layout::default()
1432
+
.direction(Direction::Horizontal)
1433
+
.constraints([Constraint::Length(12), Constraint::Min(1)])
1434
+
.split(chunks[0]);
1435
+
f.render_widget(label, split[0]);
1436
+
let sparkline = Sparkline::default()
1437
+
.data(&data)
1438
+
.style(Style::default().fg(Color::Red));
1439
+
f.render_widget(sparkline, split[1]);
1440
+
1441
+
let pct = if snap.tokio_poll_total > 0.0 {
1442
+
snap.tokio_slow_poll_total / snap.tokio_poll_total * 100.0
1443
+
} else {
1444
+
0.0
1445
+
};
1446
+
let stats = Paragraph::new(Line::from(vec![
1447
+
Span::styled(" duration: ", Style::default().fg(Color::DarkGray)),
1448
+
Span::raw(fmt_duration_s(snap.tokio_slow_poll_duration_seconds_total)),
1449
+
Span::styled(" ratio: ", Style::default().fg(Color::DarkGray)),
1450
+
Span::styled(
1451
+
format!("{pct:.3}%"),
1452
+
Style::default().fg(if pct > 1.0 {
1453
+
Color::Red
1454
+
} else if pct > 0.1 {
1455
+
Color::Yellow
1456
+
} else {
1457
+
Color::Green
1458
+
}),
1459
+
),
1460
+
]));
1461
+
f.render_widget(stats, chunks[1]);
1462
+
}
1463
+
1464
+
fn render_long_delays(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1465
+
let snap = &state.current;
1466
+
1467
+
let block = Block::default()
1468
+
.title(format!(
1469
+
" Long Delays {} total ",
1470
+
fmt_count(snap.tokio_long_delay_total),
1471
+
))
1472
+
.borders(Borders::ALL)
1473
+
.border_style(Style::default().fg(Color::Magenta));
1474
+
1475
+
let inner = block.inner(area);
1476
+
f.render_widget(block, area);
1477
+
1478
+
let chunks = Layout::default()
1479
+
.direction(Direction::Vertical)
1480
+
.constraints([Constraint::Length(3), Constraint::Min(1)])
1481
+
.split(inner);
1482
+
1483
+
// Long delays per interval sparkline
1484
+
let data: Vec<u64> = state.long_delays_per_interval.iter().copied().collect();
1485
+
let label = Paragraph::new(Line::from(vec![Span::styled(
1486
+
" /interval ",
1487
+
Style::default().fg(Color::DarkGray),
1488
+
)]));
1489
+
let split = Layout::default()
1490
+
.direction(Direction::Horizontal)
1491
+
.constraints([Constraint::Length(12), Constraint::Min(1)])
1492
+
.split(chunks[0]);
1493
+
f.render_widget(label, split[0]);
1494
+
let sparkline = Sparkline::default()
1495
+
.data(&data)
1496
+
.style(Style::default().fg(Color::Magenta));
1497
+
f.render_widget(sparkline, split[1]);
1498
+
1499
+
let pct = if snap.tokio_scheduled_total > 0.0 {
1500
+
snap.tokio_long_delay_total / snap.tokio_scheduled_total * 100.0
1501
+
} else {
1502
+
0.0
1503
+
};
1504
+
let stats = Paragraph::new(Line::from(vec![
1505
+
Span::styled(" duration: ", Style::default().fg(Color::DarkGray)),
1506
+
Span::raw(fmt_duration_s(snap.tokio_long_delay_duration_seconds_total)),
1507
+
Span::styled(" ratio: ", Style::default().fg(Color::DarkGray)),
1508
+
Span::styled(
1509
+
format!("{pct:.3}%"),
1510
+
Style::default().fg(if pct > 1.0 {
1511
+
Color::Red
1512
+
} else if pct > 0.1 {
1513
+
Color::Yellow
1514
+
} else {
1515
+
Color::Green
1516
+
}),
1517
+
),
1518
+
]));
1519
+
f.render_widget(stats, chunks[1]);
1520
+
}
1521
+
1522
+
fn render_event_counts(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1523
+
let snap = &state.current;
1524
+
1525
+
let events = vec![
1526
+
("Polls", snap.tokio_poll_total, Color::Green),
1527
+
("Slow polls", snap.tokio_slow_poll_total, Color::Red),
1528
+
("Scheduled", snap.tokio_scheduled_total, Color::Yellow),
1529
+
("Long delays", snap.tokio_long_delay_total, Color::Magenta),
1530
+
("Idled", snap.tokio_idled_total, Color::Cyan),
1531
+
("First polled", snap.tokio_first_poll_total, Color::White),
1532
+
("Instrumented", snap.tokio_instrumented_total, Color::White),
1533
+
("Dropped", snap.tokio_dropped_total, Color::DarkGray),
1534
+
];
1535
+
1536
+
let rows: Vec<Row> = events
1537
+
.iter()
1538
+
.map(|(name, val, color)| {
1539
+
Row::new(vec![name.to_string(), fmt_count(*val)]).style(Style::default().fg(*color))
1540
+
})
1541
+
.collect();
1542
+
1543
+
let table = Table::new(rows, [Constraint::Min(16), Constraint::Length(12)])
1544
+
.header(
1545
+
Row::new(vec!["Event", "Count"]).style(Style::default().add_modifier(Modifier::BOLD)),
1546
+
)
1547
+
.block(
1548
+
Block::default()
1549
+
.title(" Tokio Events ")
1550
+
.borders(Borders::ALL)
1551
+
.border_style(Style::default().fg(Color::DarkGray)),
1552
+
);
1553
+
1554
+
f.render_widget(table, area);
1555
+
}
1556
+
1557
+
// ============================
1558
+
// Tab 3: HTTP
1559
+
// ============================
1560
+
1561
+
fn render_http_tab(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1562
+
let cols = Layout::default()
1563
+
.direction(Direction::Horizontal)
1564
+
.constraints([Constraint::Percentage(50), Constraint::Percentage(50)])
1565
+
.split(area);
1566
+
1567
+
let left = Layout::default()
1568
+
.direction(Direction::Vertical)
1569
+
.constraints([
1570
+
Constraint::Length(7), // Throughput sparkline + stats
1571
+
Constraint::Length(7), // Latency sparkline + stats
1572
+
Constraint::Min(4), // Routes table
1573
+
])
1574
+
.split(cols[0]);
1575
+
1576
+
let right = Layout::default()
1577
+
.direction(Direction::Vertical)
1578
+
.constraints([
1579
+
Constraint::Length(9), // Status code breakdown
1580
+
Constraint::Min(4), // Top paths
1581
+
])
1582
+
.split(cols[1]);
1583
+
1584
+
render_http_throughput(f, left[0], state);
1585
+
render_http_latency(f, left[1], state);
1586
+
render_http_routes(f, left[2], state);
1587
+
1588
+
render_http_status_codes(f, right[0], state);
1589
+
render_http_top_paths(f, right[1], state);
1590
+
}
1591
+
1592
+
fn render_http_throughput(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1593
+
let block = Block::default()
1594
+
.title(format!(" HTTP Throughput {} ", fmt_rate(state.http_rate)))
1595
+
.borders(Borders::ALL)
1596
+
.border_style(Style::default().fg(Color::Cyan));
1597
+
1598
+
let inner = block.inner(area);
1599
+
f.render_widget(block, area);
1600
+
1601
+
let chunks = Layout::default()
1602
+
.direction(Direction::Vertical)
1603
+
.constraints([Constraint::Length(3), Constraint::Min(1)])
1604
+
.split(inner);
1605
+
1606
+
let data: Vec<u64> = state.http_per_sec.iter().copied().collect();
1607
+
let sparkline = Sparkline::default()
1608
+
.data(&data)
1609
+
.style(Style::default().fg(Color::Cyan));
1610
+
f.render_widget(sparkline, chunks[0]);
1611
+
1612
+
let snap = &state.current;
1613
+
let stats = Paragraph::new(Line::from(vec![
1614
+
Span::styled(" total: ", Style::default().fg(Color::DarkGray)),
1615
+
Span::raw(fmt_count(snap.http_requests_total)),
1616
+
]));
1617
+
f.render_widget(stats, chunks[1]);
1618
+
}
1619
+
1620
+
fn render_http_latency(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1621
+
let snap = &state.current;
1622
+
let avg_ms = if snap.http_request_duration_count > 0.0 {
1623
+
(snap.http_request_duration_sum / snap.http_request_duration_count) * 1000.0
1624
+
} else {
1625
+
0.0
1626
+
};
1627
+
1628
+
let block = Block::default()
1629
+
.title(format!(" HTTP Latency {:.1}ms avg ", avg_ms))
1630
+
.borders(Borders::ALL)
1631
+
.border_style(Style::default().fg(Color::Yellow));
1632
+
1633
+
let inner = block.inner(area);
1634
+
f.render_widget(block, area);
1635
+
1636
+
let chunks = Layout::default()
1637
+
.direction(Direction::Vertical)
1638
+
.constraints([Constraint::Length(3), Constraint::Min(1)])
1639
+
.split(inner);
1640
+
1641
+
let data: Vec<u64> = state.http_duration_ms.iter().copied().collect();
1642
+
let sparkline = Sparkline::default()
1643
+
.data(&data)
1644
+
.style(Style::default().fg(Color::Yellow));
1645
+
f.render_widget(sparkline, chunks[0]);
1646
+
1647
+
let stats = Paragraph::new(Line::from(vec![
1648
+
Span::styled(" total duration: ", Style::default().fg(Color::DarkGray)),
1649
+
Span::raw(fmt_duration_s(snap.http_request_duration_sum)),
1650
+
Span::styled(" samples: ", Style::default().fg(Color::DarkGray)),
1651
+
Span::raw(fmt_count(snap.http_request_duration_count)),
1652
+
]));
1653
+
f.render_widget(stats, chunks[1]);
1654
+
}
1655
+
1656
+
fn render_http_routes(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1657
+
let snap = &state.current;
1658
+
1659
+
// Aggregate by (method, path) with status breakdown
1660
+
let mut route_map: HashMap<(String, String), Vec<(u16, f64)>> = HashMap::new();
1661
+
for (method, path, status, count) in &snap.http_requests_by_route {
1662
+
route_map
1663
+
.entry((method.clone(), path.clone()))
1664
+
.or_default()
1665
+
.push((*status, *count));
1666
+
}
1667
+
1668
+
let mut sorted_routes: Vec<((String, String), Vec<(u16, f64)>)> =
1669
+
route_map.into_iter().collect();
1670
+
sorted_routes.sort_by(|a, b| {
1671
+
let total_a: f64 = a.1.iter().map(|(_, c)| c).sum();
1672
+
let total_b: f64 = b.1.iter().map(|(_, c)| c).sum();
1673
+
total_b
1674
+
.partial_cmp(&total_a)
1675
+
.unwrap_or(std::cmp::Ordering::Equal)
1676
+
});
1677
+
1678
+
let rows: Vec<Row> = sorted_routes
1679
+
.iter()
1680
+
.map(|((method, path), statuses)| {
1681
+
let total: f64 = statuses.iter().map(|(_, c)| c).sum();
1682
+
let status_parts: Vec<String> = {
1683
+
let mut s = statuses.clone();
1684
+
s.sort_by_key(|(code, _)| *code);
1685
+
s.iter()
1686
+
.map(|(code, count)| format!("{}:{}", code, fmt_count(*count)))
1687
+
.collect()
1688
+
};
1689
+
let color = match method.as_str() {
1690
+
"GET" => Color::Green,
1691
+
"POST" => Color::Yellow,
1692
+
"PUT" => Color::Blue,
1693
+
"DELETE" => Color::Red,
1694
+
_ => Color::White,
1695
+
};
1696
+
Row::new(vec![
1697
+
method.clone(),
1698
+
path.clone(),
1699
+
fmt_count(total),
1700
+
status_parts.join(" "),
1701
+
])
1702
+
.style(Style::default().fg(color))
1703
+
})
1704
+
.collect();
1705
+
1706
+
let table = Table::new(
1707
+
rows,
1708
+
[
1709
+
Constraint::Length(6),
1710
+
Constraint::Min(30),
1711
+
Constraint::Length(10),
1712
+
Constraint::Length(20),
1713
+
],
1714
+
)
1715
+
.header(
1716
+
Row::new(vec!["Method", "Path", "Count", "Status"])
1717
+
.style(Style::default().add_modifier(Modifier::BOLD)),
1718
+
)
1719
+
.block(
1720
+
Block::default()
1721
+
.title(" Routes ")
1722
+
.borders(Borders::ALL)
1723
+
.border_style(Style::default().fg(Color::Green)),
1724
+
);
1725
+
1726
+
f.render_widget(table, area);
1727
+
}
1728
+
1729
+
fn render_http_status_codes(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1730
+
let snap = &state.current;
1731
+
1732
+
// Aggregate by status code class
1733
+
let mut by_class: HashMap<String, f64> = HashMap::new();
1734
+
for (_, _, status, count) in &snap.http_requests_by_route {
1735
+
let class = match status / 100 {
1736
+
2 => "2xx",
1737
+
3 => "3xx",
1738
+
4 => "4xx",
1739
+
5 => "5xx",
1740
+
_ => "other",
1741
+
};
1742
+
*by_class.entry(class.to_string()).or_default() += count;
1743
+
}
1744
+
1745
+
let total: f64 = by_class.values().sum();
1746
+
1747
+
let classes = ["2xx", "3xx", "4xx", "5xx"];
1748
+
let rows: Vec<Row> = classes
1749
+
.iter()
1750
+
.map(|class| {
1751
+
let count = by_class.get(*class).copied().unwrap_or(0.0);
1752
+
let pct = if total > 0.0 {
1753
+
format!("{:.1}%", count / total * 100.0)
1754
+
} else {
1755
+
"—".to_string()
1756
+
};
1757
+
let color = match *class {
1758
+
"2xx" => Color::Green,
1759
+
"3xx" => Color::Cyan,
1760
+
"4xx" => Color::Yellow,
1761
+
"5xx" => Color::Red,
1762
+
_ => Color::White,
1763
+
};
1764
+
Row::new(vec![class.to_string(), fmt_count(count), pct])
1765
+
.style(Style::default().fg(color))
1766
+
})
1767
+
.collect();
1768
+
1769
+
let table = Table::new(
1770
+
rows,
1771
+
[
1772
+
Constraint::Length(6),
1773
+
Constraint::Length(10),
1774
+
Constraint::Length(8),
1775
+
],
1776
+
)
1777
+
.header(
1778
+
Row::new(vec!["Status", "Count", "%"]).style(Style::default().add_modifier(Modifier::BOLD)),
1779
+
)
1780
+
.block(
1781
+
Block::default()
1782
+
.title(" Status Codes ")
1783
+
.borders(Borders::ALL)
1784
+
.border_style(Style::default().fg(Color::Magenta)),
1785
+
);
1786
+
1787
+
f.render_widget(table, area);
1788
+
}
1789
+
1790
+
fn render_http_top_paths(f: &mut ratatui::Frame, area: Rect, state: &DashboardState) {
1791
+
let snap = &state.current;
1792
+
1793
+
// Aggregate by path, sorted by count descending
1794
+
let mut by_path: HashMap<String, f64> = HashMap::new();
1795
+
let mut errors_by_path: HashMap<String, f64> = HashMap::new();
1796
+
for (_, path, status, count) in &snap.http_requests_by_route {
1797
+
*by_path.entry(path.clone()).or_default() += count;
1798
+
if *status >= 400 {
1799
+
*errors_by_path.entry(path.clone()).or_default() += count;
1800
+
}
1801
+
}
1802
+
1803
+
let total: f64 = by_path.values().sum();
1804
+
1805
+
let mut sorted: Vec<(String, f64)> = by_path.into_iter().collect();
1806
+
sorted.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
1807
+
1808
+
let rows: Vec<Row> = sorted
1809
+
.iter()
1810
+
.map(|(path, count)| {
1811
+
let errors = errors_by_path.get(path).copied().unwrap_or(0.0);
1812
+
let pct = if total > 0.0 {
1813
+
format!("{:.1}%", count / total * 100.0)
1814
+
} else {
1815
+
"—".to_string()
1816
+
};
1817
+
let error_str = if errors > 0.0 {
1818
+
fmt_count(errors)
1819
+
} else {
1820
+
"—".to_string()
1821
+
};
1822
+
let color = if *count > 0.0 && errors / count > 0.5 {
1823
+
Color::Red
1824
+
} else if errors > 0.0 {
1825
+
Color::Yellow
1826
+
} else {
1827
+
Color::Green
1828
+
};
1829
+
Row::new(vec![path.clone(), fmt_count(*count), pct, error_str])
1830
+
.style(Style::default().fg(color))
1831
+
})
1832
+
.collect();
1833
+
1834
+
let table = Table::new(
1835
+
rows,
1836
+
[
1837
+
Constraint::Min(30),
1838
+
Constraint::Length(10),
1839
+
Constraint::Length(8),
1840
+
Constraint::Length(8),
1841
+
],
1842
+
)
1843
+
.header(
1844
+
Row::new(vec!["Path", "Count", "%", "Errors"])
1845
+
.style(Style::default().add_modifier(Modifier::BOLD)),
1846
+
)
1847
+
.block(
1848
+
Block::default()
1849
+
.title(" Top Paths ")
1850
+
.borders(Borders::ALL)
1851
+
.border_style(Style::default().fg(Color::Red)),
1852
+
);
1853
+
1854
+
f.render_widget(table, area);
1855
+
}
+346
src/config.rs
+346
src/config.rs
···
1
+
//! Configuration types for the Ramjet service.
2
+
//!
3
+
//! Includes CLI argument parsing via clap, collection pattern matching
4
+
//! for routing firehose events, and service-level configuration.
5
+
6
+
use std::collections::HashSet;
7
+
use std::net::SocketAddr;
8
+
use std::path::PathBuf;
9
+
10
+
use clap::Parser;
11
+
12
+
/// Collection pattern for matching ATProtocol NSIDs.
13
+
///
14
+
/// Supports exact match, single-segment wildcard (`.*`), and
15
+
/// glob wildcard (`.**`) patterns.
16
+
#[derive(Debug, Clone)]
17
+
pub enum CollectionPattern {
18
+
/// Empty pattern — matches nothing.
19
+
None,
20
+
/// Wildcard `*` — matches every collection.
21
+
MatchAll,
22
+
/// Exact NSID match (e.g., `com.example`).
23
+
Exact(String),
24
+
/// Single-segment wildcard (e.g., `com.example.*` matches `com.example.foo`).
25
+
SingleWild(String),
26
+
/// Glob wildcard (e.g., `com.example.**` matches `com.example.foo.bar`).
27
+
GlobWild(String),
28
+
}
29
+
30
+
impl CollectionPattern {
31
+
/// Parse a pattern string into a `CollectionPattern`.
32
+
pub fn parse(pattern: &str) -> Self {
33
+
let trimmed = pattern.trim();
34
+
if trimmed.is_empty() {
35
+
CollectionPattern::None
36
+
} else if trimmed == "*" {
37
+
CollectionPattern::MatchAll
38
+
} else if let Some(prefix) = trimmed.strip_suffix(".**") {
39
+
CollectionPattern::GlobWild(format!("{}.", prefix))
40
+
} else if let Some(prefix) = trimmed.strip_suffix(".*") {
41
+
CollectionPattern::SingleWild(format!("{}.", prefix))
42
+
} else {
43
+
CollectionPattern::Exact(trimmed.to_string())
44
+
}
45
+
}
46
+
47
+
/// Test whether a collection NSID matches this pattern.
48
+
pub fn matches(&self, collection: &str) -> bool {
49
+
match self {
50
+
CollectionPattern::None => false,
51
+
CollectionPattern::MatchAll => true,
52
+
CollectionPattern::Exact(expected) => collection == expected,
53
+
CollectionPattern::SingleWild(prefix) => {
54
+
collection.starts_with(prefix.as_str())
55
+
&& !collection[prefix.len()..].contains('.')
56
+
&& collection.len() > prefix.len()
57
+
}
58
+
CollectionPattern::GlobWild(prefix) => {
59
+
collection.starts_with(prefix.as_str()) && collection.len() > prefix.len()
60
+
}
61
+
}
62
+
}
63
+
}
64
+
65
+
/// A set of collection patterns evaluated with OR semantics.
66
+
#[derive(Debug, Clone)]
67
+
pub struct CollectionMatcher {
68
+
patterns: Vec<CollectionPattern>,
69
+
}
70
+
71
+
impl CollectionMatcher {
72
+
/// Create a matcher from a space-separated pattern string.
73
+
///
74
+
/// - `"*"` matches everything
75
+
/// - `""` matches nothing
76
+
/// - `"app.bsky.** community.lexicon.**"` matches multiple prefixes
77
+
pub fn new(pattern_str: &str) -> Self {
78
+
let trimmed = pattern_str.trim();
79
+
if trimmed.is_empty() {
80
+
return Self {
81
+
patterns: vec![CollectionPattern::None],
82
+
};
83
+
}
84
+
Self {
85
+
patterns: trimmed
86
+
.split_whitespace()
87
+
.map(CollectionPattern::parse)
88
+
.collect(),
89
+
}
90
+
}
91
+
92
+
/// Test whether a collection NSID matches any pattern in this matcher.
93
+
pub fn matches(&self, collection: &str) -> bool {
94
+
self.patterns.iter().any(|p| p.matches(collection))
95
+
}
96
+
}
97
+
98
+
/// Ramjet service configuration, parsed from CLI arguments and environment variables.
99
+
#[derive(Parser, Debug)]
100
+
#[command(
101
+
name = "ramjet",
102
+
version,
103
+
about = "ATProtocol event-stream, record, and blob service"
104
+
)]
105
+
pub struct CliArgs {
106
+
/// Path to the fjall database directory.
107
+
#[arg(long, env = "RAMJET_DB_PATH", default_value = "./data/ramjet.db")]
108
+
pub db_path: PathBuf,
109
+
110
+
/// Upstream relay WebSocket host.
111
+
#[arg(long, env = "RAMJET_RELAY_HOST", default_value = "bsky.network")]
112
+
pub relay_host: String,
113
+
114
+
/// HTTP bind address.
115
+
#[arg(long, env = "RAMJET_LISTEN_ADDR", default_value = "0.0.0.0:8080")]
116
+
pub listen_addr: SocketAddr,
117
+
118
+
/// Space-separated collection patterns to persist (e.g., `"garden.lexicon.** com.atproto.lexicon.**"`).
119
+
#[arg(long, env = "RAMJET_TRACKED_COLLECTIONS", default_value = "*")]
120
+
pub tracked_collections: String,
121
+
122
+
/// Space-separated collection patterns to forward as low-priority events.
123
+
/// `"*"` forwards everything not already tracked (default).
124
+
/// `""` forwards nothing.
125
+
#[arg(long, env = "RAMJET_FORWARD_COLLECTIONS", default_value = "*")]
126
+
pub forward_collections: String,
127
+
128
+
/// Event retention window in hours.
129
+
#[arg(long, env = "RAMJET_EVENT_RETENTION_HOURS", default_value = "72")]
130
+
pub event_retention_hours: u64,
131
+
132
+
/// Maximum events per WriteBatch.
133
+
#[arg(long, env = "RAMJET_BATCH_SIZE", default_value = "500")]
134
+
pub batch_size: usize,
135
+
136
+
/// Maximum wait time (ms) for batch fill before flushing.
137
+
#[arg(long, env = "RAMJET_BATCH_TIMEOUT_MS", default_value = "100")]
138
+
pub batch_timeout_ms: u64,
139
+
140
+
/// Comma-separated list of admin DIDs.
141
+
#[arg(long, env = "ADMIN_DIDS", default_value = "")]
142
+
pub admin_dids: String,
143
+
144
+
/// Path to a zstd dictionary file for event compression.
145
+
/// When absent, events are stored uncompressed.
146
+
#[arg(long, env = "RAMJET_ZSTD_DICT_PATH")]
147
+
pub zstd_dict_path: Option<PathBuf>,
148
+
149
+
/// Comma-separated list of DIDs to backfill on startup (skips already-backfilled repos).
150
+
#[arg(long, env = "RAMJET_BACKFILL", default_value = "")]
151
+
pub backfill: String,
152
+
153
+
/// Consumer group definitions (repeatable). Format: `name:partition_count`.
154
+
/// Example: `--consumer-group indexers:3 --consumer-group notifiers:2`
155
+
#[arg(long, env = "RAMJET_CONSUMER_GROUPS", value_delimiter = ',')]
156
+
pub consumer_group: Vec<String>,
157
+
}
158
+
159
+
/// A pre-defined consumer group with a fixed number of partitions.
160
+
#[derive(Debug, Clone)]
161
+
pub struct ConsumerGroup {
162
+
/// Group name (e.g., "indexers").
163
+
pub name: String,
164
+
/// Number of partitions. Must be >= 1.
165
+
pub partition_count: u16,
166
+
}
167
+
168
+
impl ConsumerGroup {
169
+
/// Parse a `name:count` string into a `ConsumerGroup`.
170
+
pub fn parse(s: &str) -> Option<Self> {
171
+
let (name, count_str) = s.split_once(':')?;
172
+
let name = name.trim();
173
+
let count: u16 = count_str.trim().parse().ok()?;
174
+
if name.is_empty() || count == 0 {
175
+
return None;
176
+
}
177
+
Some(Self {
178
+
name: name.to_string(),
179
+
partition_count: count,
180
+
})
181
+
}
182
+
}
183
+
184
+
/// Resolved service configuration with parsed matchers and sets.
185
+
pub struct ServiceConfig {
186
+
/// Path to the fjall database directory.
187
+
pub db_path: PathBuf,
188
+
/// Upstream relay WebSocket host.
189
+
pub relay_host: String,
190
+
/// HTTP bind address.
191
+
pub listen_addr: SocketAddr,
192
+
/// Matcher for collections to persist.
193
+
pub tracked_collections: CollectionMatcher,
194
+
/// Matcher for collections to forward as low-priority.
195
+
pub forward_collections: CollectionMatcher,
196
+
/// Event retention window in hours.
197
+
pub event_retention_hours: u64,
198
+
/// Maximum events per WriteBatch.
199
+
pub batch_size: usize,
200
+
/// Maximum wait time (ms) for batch fill.
201
+
pub batch_timeout_ms: u64,
202
+
/// Set of admin DIDs.
203
+
pub admin_dids: HashSet<String>,
204
+
/// Optional path to a zstd dictionary file for event compression.
205
+
pub zstd_dict_path: Option<PathBuf>,
206
+
/// DIDs to backfill on startup.
207
+
pub backfill_dids: Vec<String>,
208
+
/// Pre-defined consumer groups.
209
+
pub consumer_groups: Vec<ConsumerGroup>,
210
+
}
211
+
212
+
impl From<CliArgs> for ServiceConfig {
213
+
fn from(args: CliArgs) -> Self {
214
+
let admin_dids: HashSet<String> = args
215
+
.admin_dids
216
+
.split(',')
217
+
.map(|s| s.trim().to_string())
218
+
.filter(|s| !s.is_empty())
219
+
.collect();
220
+
221
+
let backfill_dids: Vec<String> = args
222
+
.backfill
223
+
.split(',')
224
+
.map(|s| s.trim().to_string())
225
+
.filter(|s| !s.is_empty())
226
+
.collect();
227
+
228
+
let consumer_groups: Vec<ConsumerGroup> = args
229
+
.consumer_group
230
+
.iter()
231
+
.filter_map(|s| {
232
+
let parsed = ConsumerGroup::parse(s);
233
+
if parsed.is_none() {
234
+
tracing::warn!(value = %s, "ignoring invalid --consumer-group value, expected name:count");
235
+
}
236
+
parsed
237
+
})
238
+
.collect();
239
+
240
+
Self {
241
+
db_path: args.db_path,
242
+
relay_host: args.relay_host,
243
+
listen_addr: args.listen_addr,
244
+
tracked_collections: CollectionMatcher::new(&args.tracked_collections),
245
+
forward_collections: CollectionMatcher::new(&args.forward_collections),
246
+
event_retention_hours: args.event_retention_hours,
247
+
batch_size: args.batch_size,
248
+
batch_timeout_ms: args.batch_timeout_ms,
249
+
admin_dids,
250
+
zstd_dict_path: args.zstd_dict_path,
251
+
backfill_dids,
252
+
consumer_groups,
253
+
}
254
+
}
255
+
}
256
+
257
+
#[cfg(test)]
258
+
mod tests {
259
+
use super::*;
260
+
261
+
#[test]
262
+
fn test_exact_match() {
263
+
let p = CollectionPattern::parse("com.example");
264
+
assert!(p.matches("com.example"));
265
+
assert!(!p.matches("com.example.foo"));
266
+
assert!(!p.matches("com"));
267
+
}
268
+
269
+
#[test]
270
+
fn test_single_wildcard() {
271
+
let p = CollectionPattern::parse("com.example.*");
272
+
assert!(p.matches("com.example.foo"));
273
+
assert!(p.matches("com.example.bar"));
274
+
assert!(!p.matches("com.example"));
275
+
assert!(!p.matches("com.example.foo.baz"));
276
+
assert!(!p.matches("com"));
277
+
}
278
+
279
+
#[test]
280
+
fn test_glob_wildcard() {
281
+
let p = CollectionPattern::parse("com.example.**");
282
+
assert!(p.matches("com.example.foo"));
283
+
assert!(p.matches("com.example.bar"));
284
+
assert!(p.matches("com.example.foo.bar"));
285
+
assert!(p.matches("com.example.bar.baz.qux"));
286
+
assert!(!p.matches("com.example"));
287
+
assert!(!p.matches("com"));
288
+
}
289
+
290
+
#[test]
291
+
fn test_match_all() {
292
+
let p = CollectionPattern::parse("*");
293
+
assert!(p.matches("anything"));
294
+
assert!(p.matches("com.example.foo"));
295
+
}
296
+
297
+
#[test]
298
+
fn test_none() {
299
+
let p = CollectionPattern::parse("");
300
+
assert!(!p.matches("anything"));
301
+
}
302
+
303
+
#[test]
304
+
fn test_matcher_multiple_patterns() {
305
+
let m = CollectionMatcher::new("garden.lexicon.** com.atproto.lexicon.**");
306
+
assert!(m.matches("garden.lexicon.foo"));
307
+
assert!(m.matches("garden.lexicon.foo.bar"));
308
+
assert!(m.matches("com.atproto.lexicon.def"));
309
+
assert!(!m.matches("app.bsky.feed.post"));
310
+
assert!(!m.matches("garden.lexicon"));
311
+
}
312
+
313
+
#[test]
314
+
fn test_matcher_star() {
315
+
let m = CollectionMatcher::new("*");
316
+
assert!(m.matches("anything"));
317
+
}
318
+
319
+
#[test]
320
+
fn test_matcher_empty() {
321
+
let m = CollectionMatcher::new("");
322
+
assert!(!m.matches("anything"));
323
+
}
324
+
325
+
#[test]
326
+
fn test_admin_dids_parsing() {
327
+
let args = CliArgs {
328
+
db_path: PathBuf::from("/tmp"),
329
+
relay_host: "bsky.network".to_string(),
330
+
listen_addr: "0.0.0.0:8080".parse().unwrap(),
331
+
tracked_collections: "*".to_string(),
332
+
forward_collections: "*".to_string(),
333
+
event_retention_hours: 72,
334
+
batch_size: 500,
335
+
batch_timeout_ms: 100,
336
+
admin_dids: "did:plc:abc, did:plc:def".to_string(),
337
+
zstd_dict_path: None,
338
+
backfill: String::new(),
339
+
consumer_group: Vec::new(),
340
+
};
341
+
let config = ServiceConfig::from(args);
342
+
assert!(config.admin_dids.contains("did:plc:abc"));
343
+
assert!(config.admin_dids.contains("did:plc:def"));
344
+
assert_eq!(config.admin_dids.len(), 2);
345
+
}
346
+
}
+59
src/errors.rs
+59
src/errors.rs
···
1
+
//! Error types for the Ramjet service.
2
+
//!
3
+
//! All errors follow the convention: `error-ramjet-{domain}-{number} {message}: {details}`
4
+
5
+
/// Top-level error type for Ramjet operations.
6
+
#[derive(Debug, thiserror::Error)]
7
+
pub enum RamjetError {
8
+
/// Fjall storage engine error.
9
+
#[error("error-ramjet-storage-1 fjall operation failed: {0}")]
10
+
Fjall(#[from] fjall::Error),
11
+
12
+
/// Key encoding or decoding error.
13
+
#[error("error-ramjet-storage-2 key encoding failed: {reason}")]
14
+
KeyEncoding {
15
+
/// Description of the encoding failure.
16
+
reason: String,
17
+
},
18
+
19
+
/// Value encoding or decoding error.
20
+
#[error("error-ramjet-storage-3 value decoding failed: {reason}")]
21
+
ValueDecoding {
22
+
/// Description of the decoding failure.
23
+
reason: String,
24
+
},
25
+
26
+
/// Invalid configuration.
27
+
#[error("error-ramjet-config-1 invalid configuration: {reason}")]
28
+
Config {
29
+
/// Description of the configuration error.
30
+
reason: String,
31
+
},
32
+
33
+
/// HTTP server error.
34
+
#[error("error-ramjet-server-1 server error: {reason}")]
35
+
Server {
36
+
/// Description of the server error.
37
+
reason: String,
38
+
},
39
+
40
+
/// Pipeline processing error.
41
+
#[error("error-ramjet-pipeline-1 pipeline error: {reason}")]
42
+
Pipeline {
43
+
/// Description of the pipeline error.
44
+
reason: String,
45
+
},
46
+
47
+
/// I/O error.
48
+
#[error("error-ramjet-io-1 I/O error: {0}")]
49
+
Io(#[from] std::io::Error),
50
+
51
+
/// Failed to load zstd dictionary file.
52
+
#[error("error-ramjet-storage-4 failed to load zstd dictionary at {path}: {source}")]
53
+
DictLoad {
54
+
/// Path to the dictionary file.
55
+
path: std::path::PathBuf,
56
+
/// Underlying I/O error.
57
+
source: std::io::Error,
58
+
},
59
+
}
+13
src/lib.rs
+13
src/lib.rs
···
1
+
//! Ramjet: Rust-native ATProtocol event-stream, record, and blob service.
2
+
//!
3
+
//! Built on fjall v3 (pure-Rust LSM-tree) for storage, axum for HTTP/WS,
4
+
//! and the `atproto-crates` ecosystem for protocol types and identity.
5
+
6
+
#![forbid(unsafe_code)]
7
+
8
+
pub mod config;
9
+
pub mod errors;
10
+
pub mod pipeline;
11
+
pub mod server;
12
+
pub mod storage;
13
+
pub mod types;
+237
src/main.rs
+237
src/main.rs
···
1
+
//! Ramjet service entry point.
2
+
//!
3
+
//! Initializes configuration, storage, metrics, and the HTTP server,
4
+
//! then spawns the async pipeline tasks.
5
+
6
+
use std::sync::Arc;
7
+
8
+
use atproto_identity::resolve::{
9
+
HickoryDnsResolver, InnerIdentityResolver, SharedIdentityResolver,
10
+
};
11
+
use atproto_identity::traits::IdentityResolver;
12
+
use clap::Parser;
13
+
use tokio::net::TcpListener;
14
+
use tokio::sync::{Semaphore, mpsc};
15
+
use tokio_util::sync::CancellationToken;
16
+
17
+
use ramjet::config::{CliArgs, ServiceConfig};
18
+
use ramjet::pipeline::backfill::run_backfill_worker;
19
+
use ramjet::pipeline::fanout::FanOutChannels;
20
+
use ramjet::pipeline::identity::run_identity_worker;
21
+
use ramjet::pipeline::ingester::run_ingester;
22
+
use ramjet::pipeline::writer::run_writer;
23
+
use ramjet::server::metrics::Metrics;
24
+
use ramjet::server::{AppState, build_router};
25
+
use ramjet::storage::FjallDb;
26
+
27
+
#[tokio::main]
28
+
async fn main() -> anyhow::Result<()> {
29
+
// Parse CLI arguments and build config
30
+
let args = CliArgs::parse();
31
+
let config = ServiceConfig::from(args);
32
+
33
+
// Initialize tracing
34
+
tracing_subscriber::fmt()
35
+
.with_env_filter(
36
+
tracing_subscriber::EnvFilter::try_from_default_env()
37
+
.unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("info")),
38
+
)
39
+
.init();
40
+
41
+
tracing::info!(
42
+
db_path = %config.db_path.display(),
43
+
listen_addr = %config.listen_addr,
44
+
relay_host = %config.relay_host,
45
+
"starting ramjet"
46
+
);
47
+
48
+
// Open storage
49
+
let db = Arc::new(FjallDb::open(
50
+
&config.db_path,
51
+
config.zstd_dict_path.as_deref(),
52
+
)?);
53
+
tracing::info!("fjall database opened");
54
+
55
+
if let Ok(Some(cursor)) = db.get_cursor() {
56
+
tracing::info!(cursor, "resuming from stored cursor");
57
+
} else {
58
+
tracing::info!("no stored cursor, will start from live head");
59
+
}
60
+
61
+
// Queue any DIDs specified via --backfill that haven't been backfilled yet.
62
+
for did in &config.backfill_dids {
63
+
let already_backfilled = db
64
+
.get_repo_state(did)
65
+
.ok()
66
+
.flatten()
67
+
.is_some_and(|rs| rs.backfilled);
68
+
if already_backfilled {
69
+
tracing::info!(%did, "skipping --backfill DID, already backfilled");
70
+
} else {
71
+
let queue_key = format!("backfill_queue\x00{did}");
72
+
db.meta.insert(queue_key.as_bytes(), b"")?;
73
+
tracing::info!(%did, "queued --backfill DID for backfill");
74
+
}
75
+
}
76
+
77
+
// Create shared state
78
+
let config = Arc::new(config);
79
+
let fanout = Arc::new(FanOutChannels::with_consumer_groups(
80
+
8192,
81
+
&config.consumer_groups,
82
+
));
83
+
for group in &config.consumer_groups {
84
+
tracing::info!(
85
+
group = %group.name,
86
+
partitions = group.partition_count,
87
+
"registered consumer group"
88
+
);
89
+
}
90
+
let metrics = Arc::new(Metrics::new());
91
+
let cancel = CancellationToken::new();
92
+
93
+
// Build identity resolver
94
+
let dns_resolver = HickoryDnsResolver::create_resolver(&[]);
95
+
let http_client = reqwest::Client::new();
96
+
let identity_resolver: Arc<dyn IdentityResolver> =
97
+
Arc::new(SharedIdentityResolver(Arc::new(InnerIdentityResolver {
98
+
dns_resolver: Arc::new(dns_resolver),
99
+
http_client,
100
+
plc_hostname: "plc.directory".to_string(),
101
+
})));
102
+
103
+
let state = AppState {
104
+
db: db.clone(),
105
+
config: config.clone(),
106
+
metrics: metrics.clone(),
107
+
fanout: fanout.clone(),
108
+
resolver: identity_resolver.clone(),
109
+
};
110
+
111
+
// Ingester → Writer channel
112
+
let (ingest_tx, ingest_rx) = mpsc::channel(81920);
113
+
114
+
// Identity event channel
115
+
let (identity_tx, identity_rx) = mpsc::channel(40960);
116
+
117
+
// Task monitor for tokio-metrics instrumentation
118
+
let task_monitor = metrics.tokio_metrics.monitor().clone();
119
+
120
+
// Spawn tokio-metrics collector (updates prometheus gauges every 10s)
121
+
let tokio_metrics_handle = tokio::spawn({
122
+
let tokio_metrics = metrics.tokio_metrics.clone();
123
+
let cancel = cancel.clone();
124
+
async move {
125
+
tokio_metrics
126
+
.run_collector(std::time::Duration::from_secs(10), cancel)
127
+
.await;
128
+
}
129
+
});
130
+
131
+
// Spawn pipeline tasks (all instrumented with tokio-metrics)
132
+
let ingester_handle = tokio::spawn(task_monitor.instrument({
133
+
let config = config.clone();
134
+
let db = db.clone();
135
+
let identity_tx = identity_tx.clone();
136
+
let metrics = metrics.clone();
137
+
let cancel = cancel.clone();
138
+
async move {
139
+
if let Err(e) = run_ingester(config, db, ingest_tx, identity_tx, metrics, cancel).await
140
+
{
141
+
tracing::error!(error = %e, "ingester failed");
142
+
}
143
+
}
144
+
}));
145
+
146
+
let writer_handle = tokio::spawn(task_monitor.instrument({
147
+
let config = config.clone();
148
+
let db = db.clone();
149
+
let fanout = fanout.clone();
150
+
let metrics = metrics.clone();
151
+
let cancel = cancel.clone();
152
+
async move {
153
+
if let Err(e) = run_writer(config, db, ingest_rx, fanout, metrics, cancel).await {
154
+
tracing::error!(error = %e, "writer failed");
155
+
}
156
+
}
157
+
}));
158
+
159
+
let identity_handle = tokio::spawn(task_monitor.instrument({
160
+
let db = db.clone();
161
+
let cancel = cancel.clone();
162
+
let semaphore = Arc::new(Semaphore::new(10));
163
+
let resolver = identity_resolver.clone();
164
+
let metrics = metrics.clone();
165
+
async move {
166
+
if let Err(e) =
167
+
run_identity_worker(db, resolver, semaphore, metrics, identity_rx, cancel).await
168
+
{
169
+
tracing::error!(error = %e, "identity worker failed");
170
+
}
171
+
}
172
+
}));
173
+
174
+
let backfill_handle = tokio::spawn(task_monitor.instrument({
175
+
let db = db.clone();
176
+
let config = config.clone();
177
+
let fanout = fanout.clone();
178
+
let resolver = identity_resolver.clone();
179
+
let metrics = metrics.clone();
180
+
let cancel = cancel.clone();
181
+
async move {
182
+
if let Err(e) = run_backfill_worker(db, config, fanout, resolver, metrics, cancel).await
183
+
{
184
+
tracing::error!(error = %e, "backfill worker failed");
185
+
}
186
+
}
187
+
}));
188
+
189
+
// Build router and bind
190
+
let router = build_router(state);
191
+
let listener = TcpListener::bind(config.listen_addr).await?;
192
+
tracing::info!(addr = %config.listen_addr, "HTTP server listening");
193
+
194
+
// Serve with graceful shutdown
195
+
axum::serve(listener, router)
196
+
.with_graceful_shutdown(shutdown_signal(cancel.clone()))
197
+
.await?;
198
+
199
+
// Cancel all pipeline tasks
200
+
cancel.cancel();
201
+
let _ = tokio::join!(
202
+
ingester_handle,
203
+
writer_handle,
204
+
identity_handle,
205
+
backfill_handle,
206
+
tokio_metrics_handle,
207
+
);
208
+
209
+
tracing::info!("ramjet shut down");
210
+
Ok(())
211
+
}
212
+
213
+
async fn shutdown_signal(cancel: CancellationToken) {
214
+
let ctrl_c = async {
215
+
tokio::signal::ctrl_c()
216
+
.await
217
+
.expect("failed to install Ctrl+C handler");
218
+
};
219
+
220
+
#[cfg(unix)]
221
+
let terminate = async {
222
+
tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
223
+
.expect("failed to install SIGTERM handler")
224
+
.recv()
225
+
.await;
226
+
};
227
+
228
+
#[cfg(not(unix))]
229
+
let terminate = std::future::pending::<()>();
230
+
231
+
tokio::select! {
232
+
() = ctrl_c => tracing::info!("received Ctrl+C"),
233
+
() = terminate => tracing::info!("received SIGTERM"),
234
+
}
235
+
236
+
cancel.cancel();
237
+
}
+411
src/pipeline/backfill.rs
+411
src/pipeline/backfill.rs
···
1
+
//! Backfill worker pipeline.
2
+
//!
3
+
//! Fetches full repository data from PDS endpoints for repos that need
4
+
//! resynchronization. Downloads are buffered via [`SpillableBuffer`] (spills
5
+
//! to disk when the response exceeds 10 MB) and parsed with
6
+
//! [`DiskRepository`] so that large repos never blow up memory.
7
+
8
+
use std::collections::HashMap;
9
+
use std::sync::Arc;
10
+
use std::time::{Instant, SystemTime, UNIX_EPOCH};
11
+
12
+
use atproto_dasl::Ipld;
13
+
use atproto_identity::traits::IdentityResolver;
14
+
use atproto_repo::SpillableBuffer;
15
+
use bytes::Bytes;
16
+
use tokio_util::sync::CancellationToken;
17
+
18
+
use crate::config::ServiceConfig;
19
+
use crate::pipeline::fanout::FanOutChannels;
20
+
use crate::server::metrics::{KeyspaceOpLabels, Metrics, StorageOp};
21
+
use crate::server::reconciliation;
22
+
use crate::storage::FjallDb;
23
+
use crate::storage::encoding::{
24
+
self, CompactOp, decode_timestamped_doc, encode_timestamped_doc, ipld_map, to_dag_cbor_bytes,
25
+
};
26
+
use crate::storage::keys;
27
+
use crate::types::{EventType, FanOutEvent, OpType, RecordValue};
28
+
29
+
const BACKFILL_QUEUE_PREFIX: &[u8] = b"backfill_queue\x00";
30
+
const POLL_INTERVAL_SECS: u64 = 5;
31
+
const MAX_BACKFILL_FAILURES: u32 = 5;
32
+
33
+
/// Run the backfill worker task.
34
+
pub async fn run_backfill_worker(
35
+
db: Arc<FjallDb>,
36
+
config: Arc<ServiceConfig>,
37
+
fanout: Arc<FanOutChannels>,
38
+
resolver: Arc<dyn IdentityResolver>,
39
+
metrics: Arc<Metrics>,
40
+
cancel: CancellationToken,
41
+
) -> anyhow::Result<()> {
42
+
tracing::info!("backfill worker started");
43
+
44
+
let client = reqwest::Client::builder()
45
+
.timeout(std::time::Duration::from_secs(60))
46
+
.build()?;
47
+
48
+
let mut failure_counts: HashMap<String, u32> = HashMap::new();
49
+
50
+
loop {
51
+
tokio::select! {
52
+
biased;
53
+
_ = cancel.cancelled() => break,
54
+
_ = tokio::time::sleep(std::time::Duration::from_secs(POLL_INTERVAL_SECS)) => {}
55
+
}
56
+
57
+
let mut queue_entries = Vec::new();
58
+
for guard in db.meta.prefix(BACKFILL_QUEUE_PREFIX) {
59
+
let Ok((key, _value)) = guard.into_inner() else {
60
+
continue;
61
+
};
62
+
let key_bytes: &[u8] = &key;
63
+
if let Some(did_bytes) = key_bytes.strip_prefix(BACKFILL_QUEUE_PREFIX) {
64
+
if let Ok(did) = std::str::from_utf8(did_bytes) {
65
+
queue_entries.push(did.to_string());
66
+
}
67
+
}
68
+
}
69
+
70
+
metrics.backfill_queue_depth.set(queue_entries.len() as u64);
71
+
72
+
for did in &queue_entries {
73
+
if cancel.is_cancelled() {
74
+
break;
75
+
}
76
+
77
+
let start = Instant::now();
78
+
match process_backfill(&db, &config, &fanout, &*resolver, &metrics, &client, did).await
79
+
{
80
+
Ok(()) => {
81
+
let queue_key = format!("backfill_queue\x00{did}");
82
+
let _ = db.meta.remove(queue_key.as_bytes());
83
+
failure_counts.remove(did.as_str());
84
+
85
+
// Mark repo as backfilled so it won't be re-queued.
86
+
if let Ok(Some(mut repo_state)) = db.get_repo_state(did) {
87
+
repo_state.backfilled = true;
88
+
let _ = db.repo_state.insert(did.as_bytes(), repo_state.encode());
89
+
metrics
90
+
.storage_ops_total
91
+
.get_or_create(&KeyspaceOpLabels {
92
+
keyspace: "repo_state".to_string(),
93
+
op: StorageOp::Write,
94
+
})
95
+
.inc();
96
+
}
97
+
98
+
metrics.backfill_repos_total.inc();
99
+
metrics
100
+
.backfill_duration_seconds
101
+
.observe(start.elapsed().as_secs_f64());
102
+
tracing::info!(%did, "backfill completed");
103
+
}
104
+
Err(e) => {
105
+
metrics.backfill_failures_total.inc();
106
+
let count = failure_counts.entry(did.clone()).or_insert(0);
107
+
*count += 1;
108
+
109
+
if *count >= MAX_BACKFILL_FAILURES {
110
+
tracing::warn!(
111
+
%did,
112
+
failures = *count,
113
+
error = %e,
114
+
"backfill exceeded max failures, denying repo"
115
+
);
116
+
117
+
let queue_key = format!("backfill_queue\x00{did}");
118
+
let _ = db.meta.remove(queue_key.as_bytes());
119
+
failure_counts.remove(did.as_str());
120
+
121
+
let mut repo_state =
122
+
db.get_repo_state(did).ok().flatten().unwrap_or_default();
123
+
repo_state.denied = true;
124
+
let _ = db.repo_state.insert(did.as_bytes(), repo_state.encode());
125
+
} else {
126
+
tracing::warn!(
127
+
%did,
128
+
failures = *count,
129
+
error = %e,
130
+
"backfill failed, will retry"
131
+
);
132
+
}
133
+
}
134
+
}
135
+
136
+
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
137
+
}
138
+
}
139
+
140
+
tracing::info!("backfill worker shutting down");
141
+
Ok(())
142
+
}
143
+
144
+
/// Memory threshold for the spillable download buffer (10 MB).
145
+
const SPILL_THRESHOLD: usize = 10 * 1024 * 1024;
146
+
147
+
async fn process_backfill(
148
+
db: &FjallDb,
149
+
config: &ServiceConfig,
150
+
fanout: &FanOutChannels,
151
+
resolver: &dyn IdentityResolver,
152
+
metrics: &Metrics,
153
+
client: &reqwest::Client,
154
+
did: &str,
155
+
) -> anyhow::Result<()> {
156
+
let pds_url = find_pds_endpoint(db, resolver, did).await?;
157
+
158
+
tracing::info!(%did, %pds_url, "starting backfill");
159
+
160
+
let url = format!("{pds_url}/xrpc/com.atproto.sync.getRepo?did={did}");
161
+
let response = client.get(&url).send().await?;
162
+
163
+
if !response.status().is_success() {
164
+
anyhow::bail!(
165
+
"getRepo returned {}: {}",
166
+
response.status(),
167
+
response.text().await.unwrap_or_default()
168
+
);
169
+
}
170
+
171
+
// Use content-length hint to decide memory vs disk up front.
172
+
let content_length = response.content_length();
173
+
let mut buffer = SpillableBuffer::with_size_hint(SPILL_THRESHOLD, content_length).await?;
174
+
175
+
let mut response = response;
176
+
while let Some(chunk) = response.chunk().await? {
177
+
buffer.write_chunk(&chunk).await?;
178
+
}
179
+
180
+
tracing::info!(
181
+
%did,
182
+
bytes = buffer.bytes_written(),
183
+
on_disk = buffer.is_on_disk(),
184
+
"downloaded repo CAR"
185
+
);
186
+
187
+
let reader = buffer.into_reader().await?;
188
+
189
+
// Parse the CAR with disk-backed storage so large MSTs don't blow up memory.
190
+
let backfill_config = atproto_repo::RepoConfig::default()
191
+
.with_limits(atproto_dasl::LimitsConfig::high_throughput());
192
+
let repo = atproto_repo::DiskRepository::from_car(reader, backfill_config).await?;
193
+
194
+
let commit_rev = repo.rev().to_string();
195
+
let collections = repo.list_collections().await?;
196
+
197
+
let mut batch = db.batch();
198
+
let mut record_count: u64 = 0;
199
+
200
+
let time_us = now_micros();
201
+
202
+
/// A backfilled record pending fan-out after batch commit.
203
+
struct PendingEvent {
204
+
seq: u64,
205
+
collection: String,
206
+
payload: Bytes,
207
+
}
208
+
209
+
let mut pending_events: Vec<PendingEvent> = Vec::new();
210
+
211
+
for collection in &collections {
212
+
if !config.tracked_collections.matches(collection) {
213
+
continue;
214
+
}
215
+
216
+
let entries = repo.list_collection(collection).await?;
217
+
for (path, cid) in &entries {
218
+
let record_bytes = repo.get_record_bytes(path).await?;
219
+
let Some(data) = record_bytes else { continue };
220
+
221
+
let record_key = keys::encode_record_key(did, collection, &path.rkey, &commit_rev);
222
+
let cid_bytes: Vec<u8> = cid.to_bytes();
223
+
224
+
let record_value = RecordValue {
225
+
cid: cid_bytes.clone(),
226
+
data: data.clone(),
227
+
};
228
+
229
+
let compressed_record = db.compress_event(&record_value.encode());
230
+
batch.insert(&db.records, &record_key, compressed_record);
231
+
record_count += 1;
232
+
233
+
// Allocate event sequence and persist compact event for replay.
234
+
let event_seq = db.next_event_seq();
235
+
let compact_op = CompactOp {
236
+
action: OpType::Create,
237
+
collection: collection.clone(),
238
+
rkey: path.rkey.clone(),
239
+
cid: Some(cid_bytes.clone()),
240
+
data: data.clone(),
241
+
};
242
+
let compact_event = encoding::encode_compact_commit_op(
243
+
event_seq,
244
+
time_us,
245
+
did,
246
+
&commit_rev,
247
+
&compact_op,
248
+
);
249
+
let compressed = db.compress_event(&compact_event);
250
+
let event_key = keys::encode_event_key(event_seq);
251
+
batch.insert(&db.events, &event_key, &compressed);
252
+
metrics
253
+
.storage_ops_total
254
+
.get_or_create(&KeyspaceOpLabels {
255
+
keyspace: "events".to_string(),
256
+
op: StorageOp::Write,
257
+
})
258
+
.inc();
259
+
260
+
// Build fan-out payload.
261
+
let payload = create_backfill_commit_payload(
262
+
event_seq,
263
+
time_us,
264
+
did,
265
+
&commit_rev,
266
+
collection,
267
+
&path.rkey,
268
+
&cid_bytes,
269
+
&data,
270
+
);
271
+
pending_events.push(PendingEvent {
272
+
seq: event_seq,
273
+
collection: collection.clone(),
274
+
payload,
275
+
});
276
+
}
277
+
}
278
+
279
+
let mut repo_state = db.get_repo_state(did)?.unwrap_or_default();
280
+
repo_state.rev = commit_rev;
281
+
batch.insert(&db.repo_state, did.as_bytes(), repo_state.encode());
282
+
283
+
// Persist the current event sequence counter.
284
+
batch.insert(&db.meta, b"event_seq", &db.current_sequence().to_be_bytes());
285
+
286
+
batch.commit()?;
287
+
288
+
// Invalidate RIBLT sketch cache after records change.
289
+
reconciliation::invalidate_sketch_cache(db, did);
290
+
291
+
// Fan-out events only after persistence succeeds.
292
+
for event in pending_events {
293
+
let fan_event = Arc::new(FanOutEvent {
294
+
seq: event.seq,
295
+
did: did.into(),
296
+
event_type: EventType::Commit {
297
+
collection: event.collection,
298
+
},
299
+
payload: event.payload,
300
+
});
301
+
let _ = fanout.high_priority_tx.send(fan_event.clone());
302
+
fanout.send_partitioned(&fan_event);
303
+
}
304
+
305
+
metrics
306
+
.storage_ops_total
307
+
.get_or_create(&KeyspaceOpLabels {
308
+
keyspace: "records".to_string(),
309
+
op: StorageOp::Write,
310
+
})
311
+
.inc_by(record_count);
312
+
metrics
313
+
.storage_ops_total
314
+
.get_or_create(&KeyspaceOpLabels {
315
+
keyspace: "repo_state".to_string(),
316
+
op: StorageOp::Write,
317
+
})
318
+
.inc();
319
+
metrics.backfill_records_total.inc_by(record_count);
320
+
321
+
tracing::info!(%did, record_count, "backfill wrote records");
322
+
Ok(())
323
+
}
324
+
325
+
/// Current time in microseconds since Unix epoch.
326
+
fn now_micros() -> u64 {
327
+
std::time::SystemTime::now()
328
+
.duration_since(std::time::UNIX_EPOCH)
329
+
.unwrap_or_default()
330
+
.as_micros() as u64
331
+
}
332
+
333
+
/// Create the DAG-CBOR payload for a backfilled record (create operation).
334
+
fn create_backfill_commit_payload(
335
+
seq: u64,
336
+
_time_us: u64,
337
+
did: &str,
338
+
rev: &str,
339
+
collection: &str,
340
+
rkey: &str,
341
+
cid_bytes: &[u8],
342
+
data: &[u8],
343
+
) -> Bytes {
344
+
let cid_str = cid::Cid::read_bytes(cid_bytes)
345
+
.map(|c| c.to_string())
346
+
.unwrap_or_default();
347
+
348
+
let mut commit_fields: Vec<(&str, Ipld)> = vec![
349
+
("rev", Ipld::String(rev.to_string())),
350
+
("operation", Ipld::String("create".to_string())),
351
+
("collection", Ipld::String(collection.to_string())),
352
+
("rkey", Ipld::String(rkey.to_string())),
353
+
("cid", Ipld::String(cid_str)),
354
+
];
355
+
356
+
if let Ok(ipld) = atproto_dasl::drisl::from_slice::<Ipld>(data) {
357
+
commit_fields.push(("record", ipld));
358
+
}
359
+
360
+
let event = ipld_map(vec![
361
+
("seq", Ipld::Integer(seq.into())),
362
+
("did", Ipld::String(did.to_string())),
363
+
("kind", Ipld::String("commit".to_string())),
364
+
("commit", ipld_map(commit_fields)),
365
+
]);
366
+
367
+
to_dag_cbor_bytes(&event).into()
368
+
}
369
+
370
+
async fn find_pds_endpoint(
371
+
db: &FjallDb,
372
+
resolver: &dyn IdentityResolver,
373
+
did: &str,
374
+
) -> anyhow::Result<String> {
375
+
// Try cached DID document first.
376
+
if let Some(doc_bytes) = db.did_to_doc.get(did.as_bytes())? {
377
+
let slice: &[u8] = &doc_bytes;
378
+
let (_ts, json_bytes) = decode_timestamped_doc(slice);
379
+
if let Ok(pds) = extract_pds_from_json(json_bytes) {
380
+
return Ok(pds);
381
+
}
382
+
}
383
+
384
+
// No cached document (or no PDS in it) — resolve live and cache.
385
+
tracing::info!(%did, "resolving DID document for backfill");
386
+
let document = resolver.resolve(did).await?;
387
+
let doc_json = serde_json::to_vec(&document)?;
388
+
let now = SystemTime::now()
389
+
.duration_since(UNIX_EPOCH)
390
+
.unwrap_or_default()
391
+
.as_secs();
392
+
let value = encode_timestamped_doc(now, &doc_json);
393
+
let _ = db.did_to_doc.insert(did.as_bytes(), &value);
394
+
395
+
extract_pds_from_json(&doc_json)
396
+
}
397
+
398
+
fn extract_pds_from_json(json_bytes: &[u8]) -> anyhow::Result<String> {
399
+
let doc: serde_json::Value = serde_json::from_slice(json_bytes)?;
400
+
if let Some(services) = doc.get("service").and_then(|v| v.as_array()) {
401
+
for svc in services {
402
+
let svc_type = svc.get("type").and_then(|v| v.as_str()).unwrap_or("");
403
+
if svc_type == "AtprotoPersonalDataServer" {
404
+
if let Some(endpoint) = svc.get("serviceEndpoint").and_then(|v| v.as_str()) {
405
+
return Ok(endpoint.trim_end_matches('/').to_string());
406
+
}
407
+
}
408
+
}
409
+
}
410
+
anyhow::bail!("no PDS endpoint found in DID document")
411
+
}
+115
src/pipeline/fanout.rs
+115
src/pipeline/fanout.rs
···
1
+
//! Fan-out channels for WebSocket event delivery.
2
+
//!
3
+
//! Two broadcast channels (high-priority and low-priority) connect
4
+
//! the batch writer to per-consumer WebSocket tasks. Optional consumer
5
+
//! groups add per-partition channels for sharded delivery.
6
+
7
+
use std::collections::HashMap;
8
+
use std::hash::{DefaultHasher, Hash, Hasher};
9
+
10
+
use crate::config::ConsumerGroup;
11
+
use crate::types::SharedFanOutEvent;
12
+
use tokio::sync::broadcast;
13
+
14
+
/// Default capacity for broadcast channels.
15
+
const DEFAULT_CAPACITY: usize = 8192;
16
+
17
+
/// Broadcast channels for event fan-out to WebSocket consumers.
18
+
pub struct FanOutChannels {
19
+
/// High-priority channel for tracked collections, identity, account, and sync events.
20
+
pub high_priority_tx: broadcast::Sender<SharedFanOutEvent>,
21
+
/// Low-priority channel for forwarded (non-tracked) collection events.
22
+
pub low_priority_tx: broadcast::Sender<SharedFanOutEvent>,
23
+
/// Per-group per-partition channels. Key is group name.
24
+
consumer_groups: HashMap<String, GroupChannels>,
25
+
}
26
+
27
+
/// Per-partition broadcast channels for a consumer group.
28
+
struct GroupChannels {
29
+
/// One sender per partition, indexed by partition number.
30
+
partitions: Vec<broadcast::Sender<SharedFanOutEvent>>,
31
+
}
32
+
33
+
impl FanOutChannels {
34
+
/// Create new fan-out channels with the given capacity.
35
+
pub fn new(capacity: usize) -> Self {
36
+
let (high_priority_tx, _) = broadcast::channel(capacity);
37
+
let (low_priority_tx, _) = broadcast::channel(capacity);
38
+
Self {
39
+
high_priority_tx,
40
+
low_priority_tx,
41
+
consumer_groups: HashMap::new(),
42
+
}
43
+
}
44
+
45
+
/// Create fan-out channels with consumer group support.
46
+
pub fn with_consumer_groups(capacity: usize, groups: &[ConsumerGroup]) -> Self {
47
+
let mut channels = Self::new(capacity);
48
+
for group in groups {
49
+
let partitions = (0..group.partition_count)
50
+
.map(|_| broadcast::channel(capacity).0)
51
+
.collect();
52
+
channels
53
+
.consumer_groups
54
+
.insert(group.name.clone(), GroupChannels { partitions });
55
+
}
56
+
channels
57
+
}
58
+
59
+
/// Subscribe to the high-priority channel.
60
+
pub fn subscribe_high(&self) -> broadcast::Receiver<SharedFanOutEvent> {
61
+
self.high_priority_tx.subscribe()
62
+
}
63
+
64
+
/// Subscribe to the low-priority channel.
65
+
pub fn subscribe_low(&self) -> broadcast::Receiver<SharedFanOutEvent> {
66
+
self.low_priority_tx.subscribe()
67
+
}
68
+
69
+
/// Subscribe to a specific partition of a consumer group.
70
+
/// Returns `None` if the group or partition doesn't exist.
71
+
pub fn subscribe_partition(
72
+
&self,
73
+
group: &str,
74
+
partition: u16,
75
+
) -> Option<broadcast::Receiver<SharedFanOutEvent>> {
76
+
let gc = self.consumer_groups.get(group)?;
77
+
gc.partitions
78
+
.get(partition as usize)
79
+
.map(|tx| tx.subscribe())
80
+
}
81
+
82
+
/// Get the partition count for a consumer group.
83
+
/// Returns `None` if the group doesn't exist.
84
+
pub fn partition_count(&self, group: &str) -> Option<u16> {
85
+
self.consumer_groups
86
+
.get(group)
87
+
.map(|gc| gc.partitions.len() as u16)
88
+
}
89
+
90
+
/// Route an event to the correct partition channel for each consumer group.
91
+
/// The DID is hashed to determine the target partition.
92
+
pub fn send_partitioned(&self, event: &SharedFanOutEvent) {
93
+
if self.consumer_groups.is_empty() {
94
+
return;
95
+
}
96
+
let hash = did_hash(&event.did);
97
+
for gc in self.consumer_groups.values() {
98
+
let partition = (hash % gc.partitions.len() as u64) as usize;
99
+
let _ = gc.partitions[partition].send(event.clone());
100
+
}
101
+
}
102
+
}
103
+
104
+
impl Default for FanOutChannels {
105
+
fn default() -> Self {
106
+
Self::new(DEFAULT_CAPACITY)
107
+
}
108
+
}
109
+
110
+
/// Compute a deterministic hash of a DID string for partition assignment.
111
+
pub fn did_hash(did: &str) -> u64 {
112
+
let mut hasher = DefaultHasher::new();
113
+
did.hash(&mut hasher);
114
+
hasher.finish()
115
+
}
+187
src/pipeline/identity.rs
+187
src/pipeline/identity.rs
···
1
+
//! Identity resolution pipeline.
2
+
//!
3
+
//! Processes `#identity` events by resolving full DID documents via an
4
+
//! `IdentityResolver` and updating the `did_to_doc` and `handle_to_did`
5
+
//! keyspaces.
6
+
7
+
use std::sync::Arc;
8
+
use std::time::{Instant, SystemTime, UNIX_EPOCH};
9
+
10
+
use atproto_identity::traits::IdentityResolver;
11
+
use tokio::sync::Semaphore;
12
+
use tokio::sync::mpsc;
13
+
use tokio_util::sync::CancellationToken;
14
+
15
+
use crate::server::metrics::{
16
+
IdentityOutcome, IdentityOutcomeLabels, KeyspaceOpLabels, Metrics, StorageOp,
17
+
};
18
+
use crate::storage::FjallDb;
19
+
use crate::storage::encoding::{decode_timestamped_doc, encode_timestamped_doc};
20
+
21
+
/// An identity event to process.
22
+
#[derive(Debug)]
23
+
pub struct IdentityEvent {
24
+
pub did: String,
25
+
pub handle: Option<String>,
26
+
}
27
+
28
+
/// Run the identity resolution worker.
29
+
pub async fn run_identity_worker(
30
+
db: Arc<FjallDb>,
31
+
resolver: Arc<dyn IdentityResolver>,
32
+
semaphore: Arc<Semaphore>,
33
+
metrics: Arc<Metrics>,
34
+
mut rx: mpsc::Receiver<IdentityEvent>,
35
+
cancel: CancellationToken,
36
+
) -> anyhow::Result<()> {
37
+
tracing::info!("identity worker started");
38
+
39
+
loop {
40
+
tokio::select! {
41
+
biased;
42
+
_ = cancel.cancelled() => break,
43
+
event = rx.recv() => {
44
+
let Some(event) = event else { break };
45
+
let permit = semaphore.clone().acquire_owned().await?;
46
+
let db = db.clone();
47
+
let resolver = resolver.clone();
48
+
let metrics = metrics.clone();
49
+
tokio::spawn(async move {
50
+
process_identity_event(&db, &*resolver, &metrics, &event).await;
51
+
drop(permit);
52
+
});
53
+
}
54
+
}
55
+
}
56
+
57
+
tracing::info!("identity worker shutting down");
58
+
Ok(())
59
+
}
60
+
61
+
async fn process_identity_event(
62
+
db: &FjallDb,
63
+
resolver: &dyn IdentityResolver,
64
+
metrics: &Metrics,
65
+
event: &IdentityEvent,
66
+
) {
67
+
let start = Instant::now();
68
+
69
+
// Resolve the full DID document
70
+
let document = match resolver.resolve(&event.did).await {
71
+
Ok(doc) => doc,
72
+
Err(e) => {
73
+
let elapsed = start.elapsed().as_secs_f64();
74
+
metrics.identity_resolve_duration_seconds.observe(elapsed);
75
+
metrics
76
+
.identity_resolves_total
77
+
.get_or_create(&IdentityOutcomeLabels {
78
+
outcome: IdentityOutcome::Failure,
79
+
})
80
+
.inc();
81
+
82
+
tracing::warn!(did = %event.did, error = %e, "failed to resolve DID document");
83
+
// If we have a handle from the firehose event, store a minimal mapping
84
+
// even if full resolution fails
85
+
if let Some(ref handle) = event.handle {
86
+
if let Err(e) = update_handle_mapping(db, &event.did, handle) {
87
+
tracing::warn!(did = %event.did, error = %e, "failed to update handle mapping");
88
+
}
89
+
}
90
+
return;
91
+
}
92
+
};
93
+
94
+
let elapsed = start.elapsed().as_secs_f64();
95
+
metrics.identity_resolve_duration_seconds.observe(elapsed);
96
+
metrics
97
+
.identity_resolves_total
98
+
.get_or_create(&IdentityOutcomeLabels {
99
+
outcome: IdentityOutcome::Success,
100
+
})
101
+
.inc();
102
+
103
+
// Serialize and store the full DID document with timestamp
104
+
let doc_json = match serde_json::to_vec(&document) {
105
+
Ok(j) => j,
106
+
Err(e) => {
107
+
tracing::warn!(did = %event.did, error = %e, "failed to serialize DID document");
108
+
return;
109
+
}
110
+
};
111
+
let now = SystemTime::now()
112
+
.duration_since(UNIX_EPOCH)
113
+
.unwrap_or_default()
114
+
.as_secs();
115
+
let value = encode_timestamped_doc(now, &doc_json);
116
+
if let Err(e) = db.did_to_doc.insert(event.did.as_bytes(), &value) {
117
+
tracing::warn!(did = %event.did, error = %e, "failed to store DID document");
118
+
return;
119
+
}
120
+
metrics
121
+
.storage_ops_total
122
+
.get_or_create(&KeyspaceOpLabels {
123
+
keyspace: "did_to_doc".to_string(),
124
+
op: StorageOp::Write,
125
+
})
126
+
.inc();
127
+
128
+
// Extract handle from the resolved document's alsoKnownAs
129
+
let new_handle = document
130
+
.also_known_as
131
+
.iter()
132
+
.find_map(|alias| alias.strip_prefix("at://"));
133
+
134
+
// Remove old handle mappings that differ from the new one
135
+
if let Some(new_handle) = new_handle {
136
+
if let Err(e) = remove_stale_handle_mappings(db, &event.did, new_handle) {
137
+
tracing::warn!(did = %event.did, error = %e, "failed to remove stale handle mappings");
138
+
}
139
+
let _ = db
140
+
.handle_to_did
141
+
.insert(new_handle.to_lowercase().as_bytes(), event.did.as_bytes());
142
+
metrics
143
+
.storage_ops_total
144
+
.get_or_create(&KeyspaceOpLabels {
145
+
keyspace: "handle_to_did".to_string(),
146
+
op: StorageOp::Write,
147
+
})
148
+
.inc();
149
+
}
150
+
151
+
tracing::debug!(
152
+
did = %event.did,
153
+
handle = ?new_handle,
154
+
elapsed_ms = elapsed * 1000.0,
155
+
"stored DID document"
156
+
);
157
+
}
158
+
159
+
/// Update handle→DID mapping without a full DID document resolution.
160
+
fn update_handle_mapping(db: &FjallDb, did: &str, handle: &str) -> anyhow::Result<()> {
161
+
remove_stale_handle_mappings(db, did, handle)?;
162
+
db.handle_to_did
163
+
.insert(handle.to_lowercase().as_bytes(), did.as_bytes())?;
164
+
Ok(())
165
+
}
166
+
167
+
/// Remove old handle→DID mappings for this DID that differ from the new handle.
168
+
fn remove_stale_handle_mappings(db: &FjallDb, did: &str, new_handle: &str) -> anyhow::Result<()> {
169
+
if let Ok(Some(doc_bytes)) = db.did_to_doc.get(did.as_bytes()) {
170
+
let slice: &[u8] = &doc_bytes;
171
+
let (_ts, json_bytes) = decode_timestamped_doc(slice);
172
+
if let Ok(doc) = serde_json::from_slice::<serde_json::Value>(json_bytes) {
173
+
if let Some(aliases) = doc.get("alsoKnownAs").and_then(|v| v.as_array()) {
174
+
for alias in aliases {
175
+
if let Some(old_handle) = alias.as_str().and_then(|s| s.strip_prefix("at://")) {
176
+
let old_lower = old_handle.to_lowercase();
177
+
let new_lower = new_handle.to_lowercase();
178
+
if old_lower != new_lower {
179
+
let _ = db.handle_to_did.remove(old_lower.as_bytes());
180
+
}
181
+
}
182
+
}
183
+
}
184
+
}
185
+
}
186
+
Ok(())
187
+
}
+570
src/pipeline/ingester.rs
+570
src/pipeline/ingester.rs
···
1
+
//! Firehose ingester pipeline.
2
+
//!
3
+
//! Maintains a WebSocket connection to the upstream relay, decodes
4
+
//! CBOR frames, and dispatches events to the batch writer channel.
5
+
6
+
use std::collections::VecDeque;
7
+
use std::sync::Arc;
8
+
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
9
+
10
+
use atproto_dasl::Ipld;
11
+
use futures_util::{SinkExt, StreamExt};
12
+
use tokio::sync::mpsc;
13
+
use tokio_util::sync::CancellationToken;
14
+
15
+
use crate::config::ServiceConfig;
16
+
use crate::pipeline::identity::IdentityEvent;
17
+
use crate::server::metrics::Metrics;
18
+
use crate::storage::FjallDb;
19
+
use crate::storage::encoding::decode_timestamped_doc;
20
+
use crate::types::{CommitOp, OpType};
21
+
22
+
/// How old a DID document can be before we re-resolve it.
23
+
const IDENTITY_REFRESH_SECS: u64 = 24 * 60 * 60;
24
+
25
+
/// How long without receiving an event before we log a warning.
26
+
const SILENCE_WARN_SECS: u64 = 3 * 60;
27
+
28
+
/// How long after the silence warning before we force-disconnect.
29
+
const SILENCE_DISCONNECT_SECS: u64 = 7 * 60;
30
+
31
+
/// Local buffer capacity between WebSocket reads and writer channel sends.
32
+
/// Decouples relay reading from writer backpressure so the relay connection
33
+
/// stays alive during transient writer stalls.
34
+
const LOCAL_BUFFER_CAP: usize = 16384;
35
+
36
+
/// How often to send a client-initiated WebSocket ping to the relay.
37
+
const PING_INTERVAL_SECS: u64 = 30;
38
+
39
+
/// Decoded event from the firehose, sent to the batch writer.
40
+
#[derive(Debug)]
41
+
pub enum IngestEvent {
42
+
/// A commit with record operations.
43
+
Commit {
44
+
seq: u64,
45
+
did: String,
46
+
rev: String,
47
+
since: Option<String>,
48
+
ops: Vec<CommitOp>,
49
+
blocks: Vec<u8>,
50
+
too_big: bool,
51
+
},
52
+
/// An identity change event.
53
+
Identity {
54
+
seq: u64,
55
+
did: String,
56
+
handle: Option<String>,
57
+
},
58
+
/// An account status change event.
59
+
Account {
60
+
seq: u64,
61
+
did: String,
62
+
active: bool,
63
+
status: Option<String>,
64
+
},
65
+
/// A sync (state-assertion) event indicating the current rev of a repo.
66
+
Sync { seq: u64, did: String, rev: String },
67
+
}
68
+
69
+
impl IngestEvent {
70
+
pub fn seq(&self) -> u64 {
71
+
match self {
72
+
Self::Commit { seq, .. }
73
+
| Self::Identity { seq, .. }
74
+
| Self::Account { seq, .. }
75
+
| Self::Sync { seq, .. } => *seq,
76
+
}
77
+
}
78
+
}
79
+
80
+
/// Run the firehose ingester task.
81
+
pub async fn run_ingester(
82
+
config: Arc<ServiceConfig>,
83
+
db: Arc<FjallDb>,
84
+
tx: mpsc::Sender<IngestEvent>,
85
+
identity_tx: mpsc::Sender<IdentityEvent>,
86
+
metrics: Arc<Metrics>,
87
+
cancel: CancellationToken,
88
+
) -> anyhow::Result<()> {
89
+
tracing::info!(relay = %config.relay_host, "firehose ingester started");
90
+
91
+
loop {
92
+
if cancel.is_cancelled() {
93
+
break;
94
+
}
95
+
96
+
let cursor = db.get_cursor().unwrap_or(None);
97
+
let url = if let Some(c) = cursor {
98
+
format!(
99
+
"wss://{}/xrpc/com.atproto.sync.subscribeRepos?cursor={}",
100
+
config.relay_host, c
101
+
)
102
+
} else {
103
+
format!(
104
+
"wss://{}/xrpc/com.atproto.sync.subscribeRepos",
105
+
config.relay_host
106
+
)
107
+
};
108
+
109
+
tracing::info!(%url, "connecting to relay");
110
+
111
+
let uri: http::Uri = match url.parse() {
112
+
Ok(u) => u,
113
+
Err(e) => {
114
+
tracing::error!(error = %e, "invalid relay URI");
115
+
tokio::time::sleep(std::time::Duration::from_secs(5)).await;
116
+
continue;
117
+
}
118
+
};
119
+
120
+
let connect_result = tokio_websockets::ClientBuilder::from_uri(uri)
121
+
.connect()
122
+
.await;
123
+
124
+
let (ws, _response) = match connect_result {
125
+
Ok(r) => r,
126
+
Err(e) => {
127
+
tracing::warn!(error = %e, "failed to connect to relay, retrying in 5s");
128
+
metrics.firehose_reconnects_total.inc();
129
+
metrics.firehose_connection_state.set(0);
130
+
tokio::time::sleep(std::time::Duration::from_secs(5)).await;
131
+
continue;
132
+
}
133
+
};
134
+
135
+
metrics.firehose_connection_state.set(1);
136
+
metrics.firehose_reconnects_total.inc();
137
+
tracing::info!("connected to relay");
138
+
139
+
let (mut ws_sink, mut ws_stream) = ws.split();
140
+
141
+
let connected_at = Instant::now();
142
+
let mut last_event_at = Instant::now();
143
+
let mut events_this_connection: u64 = 0;
144
+
let mut silence_warned = false;
145
+
let mut buf: VecDeque<IngestEvent> = VecDeque::with_capacity(LOCAL_BUFFER_CAP);
146
+
let mut ping_interval = tokio::time::interval(Duration::from_secs(PING_INTERVAL_SECS));
147
+
ping_interval.reset(); // don't fire immediately
148
+
149
+
'connection: loop {
150
+
// Drain local buffer to writer channel (non-blocking)
151
+
while let Some(event) = buf.pop_front() {
152
+
let process_start = Instant::now();
153
+
match tx.try_send(event) {
154
+
Ok(()) => {
155
+
metrics
156
+
.ingester_process_duration_seconds
157
+
.observe(process_start.elapsed().as_secs_f64());
158
+
}
159
+
Err(mpsc::error::TrySendError::Full(event)) => {
160
+
buf.push_front(event);
161
+
break;
162
+
}
163
+
Err(mpsc::error::TrySendError::Closed(_)) => {
164
+
tracing::error!("writer channel closed");
165
+
cancel.cancel();
166
+
return Ok(());
167
+
}
168
+
}
169
+
}
170
+
metrics.ingester_buffer_depth.set(buf.len() as u64);
171
+
172
+
if buf.len() >= LOCAL_BUFFER_CAP {
173
+
// Buffer full, channel full — must block on send to make progress.
174
+
// The relay connection stays alive via TCP buffers while we wait.
175
+
// Also keep sending pings so the relay doesn't consider us dead.
176
+
tokio::select! {
177
+
biased;
178
+
_ = cancel.cancelled() => {
179
+
tracing::info!("ingester cancellation requested");
180
+
break 'connection;
181
+
}
182
+
_ = ping_interval.tick() => {
183
+
if ws_sink.send(tokio_websockets::Message::ping("")).await.is_err() {
184
+
tracing::warn!("failed to send ping, disconnecting");
185
+
break 'connection;
186
+
}
187
+
}
188
+
permit = tx.reserve() => {
189
+
match permit {
190
+
Ok(permit) => {
191
+
let process_start = Instant::now();
192
+
permit.send(buf.pop_front().unwrap());
193
+
metrics
194
+
.ingester_process_duration_seconds
195
+
.observe(process_start.elapsed().as_secs_f64());
196
+
}
197
+
Err(_) => {
198
+
tracing::error!("writer channel closed");
199
+
cancel.cancel();
200
+
return Ok(());
201
+
}
202
+
}
203
+
}
204
+
}
205
+
continue;
206
+
}
207
+
208
+
// Buffer has room — read from relay
209
+
let deadline = if silence_warned {
210
+
last_event_at + Duration::from_secs(SILENCE_WARN_SECS + SILENCE_DISCONNECT_SECS)
211
+
} else {
212
+
last_event_at + Duration::from_secs(SILENCE_WARN_SECS)
213
+
};
214
+
215
+
tokio::select! {
216
+
biased;
217
+
218
+
_ = cancel.cancelled() => {
219
+
tracing::info!("ingester cancellation requested");
220
+
break 'connection;
221
+
}
222
+
223
+
_ = tokio::time::sleep_until(tokio::time::Instant::from_std(deadline)) => {
224
+
if !silence_warned {
225
+
tracing::warn!(
226
+
silence_secs = last_event_at.elapsed().as_secs(),
227
+
events_this_connection,
228
+
buf_len = buf.len(),
229
+
"no events received from relay for 3 minutes, will force-disconnect in 7 minutes",
230
+
);
231
+
silence_warned = true;
232
+
} else {
233
+
tracing::warn!(
234
+
silence_secs = last_event_at.elapsed().as_secs(),
235
+
events_this_connection,
236
+
buf_len = buf.len(),
237
+
"no events received from relay for 10 minutes, force-disconnecting",
238
+
);
239
+
break 'connection;
240
+
}
241
+
}
242
+
243
+
_ = ping_interval.tick() => {
244
+
if ws_sink.send(tokio_websockets::Message::ping("")).await.is_err() {
245
+
tracing::warn!("failed to send ping, disconnecting");
246
+
break 'connection;
247
+
}
248
+
}
249
+
250
+
msg = ws_stream.next() => {
251
+
let Some(result) = msg else {
252
+
tracing::warn!(
253
+
events_this_connection,
254
+
uptime_secs = connected_at.elapsed().as_secs(),
255
+
"relay connection closed (stream ended)"
256
+
);
257
+
break 'connection;
258
+
};
259
+
260
+
let message = match result {
261
+
Ok(m) => m,
262
+
Err(e) => {
263
+
tracing::warn!(
264
+
error = %e,
265
+
events_this_connection,
266
+
uptime_secs = connected_at.elapsed().as_secs(),
267
+
"websocket error, disconnecting"
268
+
);
269
+
break 'connection;
270
+
}
271
+
};
272
+
273
+
if message.is_pong() {
274
+
continue;
275
+
}
276
+
277
+
if !message.is_binary() {
278
+
continue;
279
+
}
280
+
281
+
let payload = message.into_payload();
282
+
let data: &[u8] = payload.as_ref();
283
+
284
+
match decode_frame(data) {
285
+
Ok(Some(event)) => {
286
+
if event.seq() == 0 {
287
+
tracing::warn!("received event with seq 0, skipping");
288
+
last_event_at = Instant::now();
289
+
continue;
290
+
}
291
+
last_event_at = Instant::now();
292
+
silence_warned = false;
293
+
events_this_connection += 1;
294
+
metrics.firehose_events_total.inc();
295
+
let seq = event.seq();
296
+
metrics.firehose_seq.set(seq);
297
+
298
+
let event_type = match &event {
299
+
IngestEvent::Commit { .. } => "#commit",
300
+
IngestEvent::Identity { .. } => "#identity",
301
+
IngestEvent::Account { .. } => "#account",
302
+
IngestEvent::Sync { .. } => "#sync",
303
+
};
304
+
metrics
305
+
.firehose_events_by_type
306
+
.get_or_create(&crate::server::metrics::EventTypeLabels {
307
+
event_type: event_type.to_string(),
308
+
})
309
+
.inc();
310
+
311
+
maybe_refresh_identity(&event, &config, &db, &identity_tx).await;
312
+
313
+
buf.push_back(event);
314
+
metrics.ingester_buffer_depth.set(buf.len() as u64);
315
+
}
316
+
Ok(None) => {
317
+
last_event_at = Instant::now();
318
+
silence_warned = false;
319
+
}
320
+
Err(e) => {
321
+
tracing::debug!(error = %e, "failed to decode firehose frame");
322
+
}
323
+
}
324
+
}
325
+
}
326
+
}
327
+
328
+
metrics.firehose_connection_state.set(0);
329
+
330
+
if cancel.is_cancelled() {
331
+
break;
332
+
}
333
+
334
+
tracing::info!("reconnecting to relay in 2s");
335
+
tokio::time::sleep(std::time::Duration::from_secs(2)).await;
336
+
}
337
+
338
+
tracing::info!("firehose ingester shutting down");
339
+
Ok(())
340
+
}
341
+
342
+
/// If the event is a commit with tracked collections and the DID's identity
343
+
/// hasn't been resolved in the last 24 hours, queue a resolution.
344
+
async fn maybe_refresh_identity(
345
+
event: &IngestEvent,
346
+
config: &ServiceConfig,
347
+
db: &FjallDb,
348
+
identity_tx: &mpsc::Sender<IdentityEvent>,
349
+
) {
350
+
let IngestEvent::Commit { did, ops, .. } = event else {
351
+
return;
352
+
};
353
+
354
+
// Check if any op is for a tracked collection
355
+
let has_tracked = ops
356
+
.iter()
357
+
.any(|op| config.tracked_collections.matches(&op.collection));
358
+
if !has_tracked {
359
+
return;
360
+
}
361
+
362
+
// Check if the DID document is missing or stale
363
+
let needs_refresh = match db.did_to_doc.get(did.as_bytes()) {
364
+
Ok(Some(bytes)) => {
365
+
let slice: &[u8] = &bytes;
366
+
let (ts, _) = decode_timestamped_doc(slice);
367
+
let now = SystemTime::now()
368
+
.duration_since(UNIX_EPOCH)
369
+
.unwrap_or_default()
370
+
.as_secs();
371
+
now.saturating_sub(ts) > IDENTITY_REFRESH_SECS
372
+
}
373
+
_ => true,
374
+
};
375
+
376
+
if needs_refresh {
377
+
let _ = identity_tx.try_send(IdentityEvent {
378
+
did: did.clone(),
379
+
handle: None,
380
+
});
381
+
}
382
+
}
383
+
384
+
/// Decode a firehose frame (two consecutive CBOR items: header + payload).
385
+
fn decode_frame(data: &[u8]) -> anyhow::Result<Option<IngestEvent>> {
386
+
// The firehose sends two CBOR items concatenated in one binary frame.
387
+
// Use atproto-dasl's non-strict deserializer to decode them sequentially.
388
+
let mut reader = std::io::Cursor::new(data);
389
+
let mut de = atproto_dasl::drisl::de::Deserializer::non_strict(&mut reader);
390
+
let header: Ipld = serde::Deserialize::deserialize(&mut de)?;
391
+
let body: Ipld = serde::Deserialize::deserialize(&mut de)?;
392
+
393
+
let Ipld::Map(ref header_map) = header else {
394
+
anyhow::bail!("header is not a map");
395
+
};
396
+
397
+
let op = header_map
398
+
.get("op")
399
+
.and_then(|v| ipld_integer(v))
400
+
.unwrap_or(0);
401
+
if op != 1 {
402
+
return Ok(None); // op=-1 is error, skip
403
+
}
404
+
405
+
let event_type = header_map
406
+
.get("t")
407
+
.and_then(|v| ipld_string(v))
408
+
.unwrap_or_default();
409
+
410
+
let Ipld::Map(ref body_map) = body else {
411
+
anyhow::bail!("body is not a map");
412
+
};
413
+
414
+
match event_type.as_str() {
415
+
"#commit" => decode_commit(body_map),
416
+
"#identity" => decode_identity(body_map),
417
+
"#account" => decode_account(body_map),
418
+
"#sync" => decode_sync(body_map),
419
+
_ => Ok(None),
420
+
}
421
+
}
422
+
423
+
fn decode_commit(
424
+
body: &std::collections::BTreeMap<String, Ipld>,
425
+
) -> anyhow::Result<Option<IngestEvent>> {
426
+
let seq = get_integer(body, "seq")
427
+
.and_then(|v| u64::try_from(v).ok())
428
+
.unwrap_or(0);
429
+
let did = get_string(body, "repo").unwrap_or_default();
430
+
let rev = get_string(body, "rev").unwrap_or_default();
431
+
let since = get_string_opt(body, "since");
432
+
let too_big = get_bool(body, "tooBig").unwrap_or(false);
433
+
434
+
let blocks = get_bytes(body, "blocks").unwrap_or_default();
435
+
436
+
let mut ops = Vec::new();
437
+
if let Some(Ipld::List(op_array)) = body.get("ops") {
438
+
for op_val in op_array {
439
+
if let Ipld::Map(op_map) = op_val {
440
+
let action = get_string(op_map, "action").unwrap_or_default();
441
+
let path = get_string(op_map, "path").unwrap_or_default();
442
+
let cid_bytes = get_link_bytes(op_map, "cid").or_else(|| get_bytes(op_map, "cid"));
443
+
444
+
let op_type = match action.as_str() {
445
+
"create" => OpType::Create,
446
+
"update" => OpType::Update,
447
+
"delete" => OpType::Delete,
448
+
_ => continue,
449
+
};
450
+
451
+
let (collection, rkey) = match path.split_once('/') {
452
+
Some((c, r)) => (c.to_string(), r.to_string()),
453
+
None => continue,
454
+
};
455
+
456
+
ops.push(CommitOp {
457
+
op: op_type,
458
+
collection,
459
+
rkey,
460
+
cid: cid_bytes,
461
+
});
462
+
}
463
+
}
464
+
}
465
+
466
+
Ok(Some(IngestEvent::Commit {
467
+
seq,
468
+
did,
469
+
rev,
470
+
since,
471
+
ops,
472
+
blocks,
473
+
too_big,
474
+
}))
475
+
}
476
+
477
+
fn decode_identity(
478
+
body: &std::collections::BTreeMap<String, Ipld>,
479
+
) -> anyhow::Result<Option<IngestEvent>> {
480
+
let seq = get_integer(body, "seq")
481
+
.and_then(|v| u64::try_from(v).ok())
482
+
.unwrap_or(0);
483
+
let did = get_string(body, "did").unwrap_or_default();
484
+
let handle = get_string_opt(body, "handle");
485
+
486
+
Ok(Some(IngestEvent::Identity { seq, did, handle }))
487
+
}
488
+
489
+
fn decode_account(
490
+
body: &std::collections::BTreeMap<String, Ipld>,
491
+
) -> anyhow::Result<Option<IngestEvent>> {
492
+
let seq = get_integer(body, "seq")
493
+
.and_then(|v| u64::try_from(v).ok())
494
+
.unwrap_or(0);
495
+
let did = get_string(body, "did").unwrap_or_default();
496
+
let active = get_bool(body, "active").unwrap_or(true);
497
+
let status = get_string_opt(body, "status");
498
+
499
+
Ok(Some(IngestEvent::Account {
500
+
seq,
501
+
did,
502
+
active,
503
+
status,
504
+
}))
505
+
}
506
+
507
+
fn decode_sync(
508
+
body: &std::collections::BTreeMap<String, Ipld>,
509
+
) -> anyhow::Result<Option<IngestEvent>> {
510
+
let seq = get_integer(body, "seq")
511
+
.and_then(|v| u64::try_from(v).ok())
512
+
.unwrap_or(0);
513
+
let did = get_string(body, "repo").unwrap_or_default();
514
+
let rev = get_string(body, "rev").unwrap_or_default();
515
+
516
+
Ok(Some(IngestEvent::Sync { seq, did, rev }))
517
+
}
518
+
519
+
// Ipld map helpers
520
+
521
+
fn ipld_string(v: &Ipld) -> Option<String> {
522
+
match v {
523
+
Ipld::String(s) => Some(s.clone()),
524
+
_ => None,
525
+
}
526
+
}
527
+
528
+
fn ipld_integer(v: &Ipld) -> Option<i128> {
529
+
match v {
530
+
Ipld::Integer(i) => Some(*i),
531
+
_ => None,
532
+
}
533
+
}
534
+
535
+
fn get_string(map: &std::collections::BTreeMap<String, Ipld>, key: &str) -> Option<String> {
536
+
map.get(key).and_then(ipld_string)
537
+
}
538
+
539
+
fn get_string_opt(map: &std::collections::BTreeMap<String, Ipld>, key: &str) -> Option<String> {
540
+
match map.get(key) {
541
+
Some(Ipld::String(s)) => Some(s.clone()),
542
+
Some(Ipld::Null) | None => None,
543
+
_ => None,
544
+
}
545
+
}
546
+
547
+
fn get_integer(map: &std::collections::BTreeMap<String, Ipld>, key: &str) -> Option<i128> {
548
+
map.get(key).and_then(ipld_integer)
549
+
}
550
+
551
+
fn get_bool(map: &std::collections::BTreeMap<String, Ipld>, key: &str) -> Option<bool> {
552
+
match map.get(key) {
553
+
Some(Ipld::Bool(b)) => Some(*b),
554
+
_ => None,
555
+
}
556
+
}
557
+
558
+
fn get_bytes(map: &std::collections::BTreeMap<String, Ipld>, key: &str) -> Option<Vec<u8>> {
559
+
match map.get(key) {
560
+
Some(Ipld::Bytes(b)) => Some(b.clone()),
561
+
_ => None,
562
+
}
563
+
}
564
+
565
+
fn get_link_bytes(map: &std::collections::BTreeMap<String, Ipld>, key: &str) -> Option<Vec<u8>> {
566
+
match map.get(key) {
567
+
Some(Ipld::Link(cid)) => Some(cid.to_bytes()),
568
+
_ => None,
569
+
}
570
+
}
+14
src/pipeline/mod.rs
+14
src/pipeline/mod.rs
···
1
+
//! Async pipeline components for the Ramjet service.
2
+
//!
3
+
//! Three main pipelines connect via channels:
4
+
//! 1. **Ingester** — firehose WebSocket → mpsc channel
5
+
//! 2. **Writer** — mpsc channel → fjall WriteBatch → broadcast channels
6
+
//! 3. **Fan-out** — broadcast channels → per-client WebSocket
7
+
//!
8
+
//! Plus supporting workers for identity resolution and backfill.
9
+
10
+
pub mod backfill;
11
+
pub mod fanout;
12
+
pub mod identity;
13
+
pub mod ingester;
14
+
pub mod writer;
+649
src/pipeline/writer.rs
+649
src/pipeline/writer.rs
···
1
+
//! Batch writer pipeline.
2
+
//!
3
+
//! Drains the ingester channel in batches, constructs fjall WriteBatch
4
+
//! operations with atomic writes across keyspaces, and publishes events
5
+
//! to the fan-out broadcast channels.
6
+
7
+
use std::collections::HashMap;
8
+
use std::sync::Arc;
9
+
10
+
use tokio::sync::mpsc;
11
+
use tokio_util::sync::CancellationToken;
12
+
13
+
use crate::config::ServiceConfig;
14
+
use crate::pipeline::fanout::FanOutChannels;
15
+
use crate::pipeline::ingester::IngestEvent;
16
+
use crate::server::metrics::{
17
+
CollectionOpLabels, EventOp, FanoutPriority, FanoutPriorityLabels, KeyspaceOpLabels, Metrics,
18
+
StorageOp,
19
+
};
20
+
use crate::storage::FjallDb;
21
+
use atproto_dasl::Ipld;
22
+
use bytes::Bytes;
23
+
24
+
use crate::server::reconciliation;
25
+
use crate::storage::encoding::{self, CompactOp, ipld_map, to_dag_cbor_bytes};
26
+
use crate::storage::keys;
27
+
use crate::types::{
28
+
AccountStatus, CommitOp, EventType, FanOutEvent, OpType, RecordValue, RepoState,
29
+
SharedFanOutEvent,
30
+
};
31
+
32
+
/// Current time in microseconds since Unix epoch.
33
+
fn now_micros() -> u64 {
34
+
std::time::SystemTime::now()
35
+
.duration_since(std::time::UNIX_EPOCH)
36
+
.unwrap_or_default()
37
+
.as_micros() as u64
38
+
}
39
+
40
+
/// Run the batch writer task.
41
+
pub async fn run_writer(
42
+
config: Arc<ServiceConfig>,
43
+
db: Arc<FjallDb>,
44
+
mut rx: mpsc::Receiver<IngestEvent>,
45
+
fanout: Arc<FanOutChannels>,
46
+
metrics: Arc<Metrics>,
47
+
cancel: CancellationToken,
48
+
) -> anyhow::Result<()> {
49
+
tracing::info!(
50
+
batch_size = config.batch_size,
51
+
batch_timeout_ms = config.batch_timeout_ms,
52
+
"batch writer started"
53
+
);
54
+
55
+
let batch_timeout = std::time::Duration::from_millis(config.batch_timeout_ms);
56
+
57
+
loop {
58
+
let mut events = Vec::with_capacity(config.batch_size);
59
+
60
+
tokio::select! {
61
+
biased;
62
+
_ = cancel.cancelled() => break,
63
+
event = rx.recv() => {
64
+
match event {
65
+
Some(e) => events.push(e),
66
+
None => break,
67
+
}
68
+
}
69
+
}
70
+
71
+
let deadline = tokio::time::Instant::now() + batch_timeout;
72
+
while events.len() < config.batch_size {
73
+
tokio::select! {
74
+
biased;
75
+
_ = cancel.cancelled() => break,
76
+
_ = tokio::time::sleep_until(deadline) => break,
77
+
event = rx.recv() => {
78
+
match event {
79
+
Some(e) => events.push(e),
80
+
None => break,
81
+
}
82
+
}
83
+
}
84
+
}
85
+
86
+
if events.is_empty() {
87
+
continue;
88
+
}
89
+
90
+
let batch_len = events.len();
91
+
let db2 = db.clone();
92
+
let config2 = config.clone();
93
+
let metrics2 = metrics.clone();
94
+
let result =
95
+
tokio::task::spawn_blocking(move || process_batch(&db2, &config2, &metrics2, &events))
96
+
.await;
97
+
98
+
let pending = match result {
99
+
Ok(Ok(pending)) => pending,
100
+
Ok(Err(e)) => {
101
+
tracing::error!(error = %e, "batch write failed");
102
+
continue;
103
+
}
104
+
Err(e) => {
105
+
tracing::error!(error = %e, "spawn_blocking panicked");
106
+
continue;
107
+
}
108
+
};
109
+
110
+
// Fan-out: broadcast sends are non-blocking and fast, keep in async context
111
+
for (fan_event, is_high) in pending {
112
+
if is_high {
113
+
let _ = fanout.high_priority_tx.send(fan_event.clone());
114
+
} else {
115
+
let _ = fanout.low_priority_tx.send(fan_event.clone());
116
+
}
117
+
fanout.send_partitioned(&fan_event);
118
+
metrics.writer_events_fanned_total.inc();
119
+
}
120
+
121
+
metrics
122
+
.fanout_queue_depth
123
+
.get_or_create(&FanoutPriorityLabels {
124
+
priority: FanoutPriority::High,
125
+
})
126
+
.set(fanout.high_priority_tx.len() as u64);
127
+
metrics
128
+
.fanout_queue_depth
129
+
.get_or_create(&FanoutPriorityLabels {
130
+
priority: FanoutPriority::Low,
131
+
})
132
+
.set(fanout.low_priority_tx.len() as u64);
133
+
134
+
metrics.writer_batch_size.observe(batch_len as f64);
135
+
metrics.writer_batches_total.inc();
136
+
}
137
+
138
+
tracing::info!("batch writer shutting down");
139
+
Ok(())
140
+
}
141
+
142
+
fn process_batch(
143
+
db: &FjallDb,
144
+
config: &ServiceConfig,
145
+
metrics: &Metrics,
146
+
events: &[IngestEvent],
147
+
) -> anyhow::Result<Vec<(SharedFanOutEvent, bool)>> {
148
+
let mut batch = db.batch();
149
+
let mut max_seq: u64 = 0;
150
+
let mut pending_fanout: Vec<(SharedFanOutEvent, bool)> = Vec::new();
151
+
let mut repo_states: HashMap<String, RepoState> = HashMap::new();
152
+
let mut riblt_invalidated: std::collections::HashSet<String> = std::collections::HashSet::new();
153
+
154
+
// Helper: get repo state from in-batch cache, falling back to db.
155
+
let get_repo_state =
156
+
|did: &str, cache: &HashMap<String, RepoState>, db: &FjallDb| -> RepoState {
157
+
if let Some(state) = cache.get(did) {
158
+
return state.clone();
159
+
}
160
+
match db.get_repo_state(did) {
161
+
Ok(Some(rs)) => rs,
162
+
Ok(None) | Err(_) => Default::default(),
163
+
}
164
+
};
165
+
166
+
for event in events {
167
+
match event {
168
+
IngestEvent::Commit {
169
+
seq,
170
+
did,
171
+
rev,
172
+
since,
173
+
ops,
174
+
blocks,
175
+
too_big,
176
+
} => {
177
+
max_seq = max_seq.max(*seq);
178
+
179
+
let state = get_repo_state(did, &repo_states, db);
180
+
if state.denied {
181
+
metrics.writer_commits_denied.inc();
182
+
continue;
183
+
}
184
+
185
+
if ops.is_empty() {
186
+
metrics.writer_commits_empty_ops.inc();
187
+
}
188
+
189
+
// Parse blocks CAR once to extract record data by CID
190
+
let block_map = parse_blocks_car(blocks);
191
+
192
+
// Collect filtered ops with routing info
193
+
struct FilteredOp {
194
+
op: CommitOp,
195
+
is_tracked: bool,
196
+
data: Vec<u8>,
197
+
}
198
+
199
+
let mut filtered_ops: Vec<FilteredOp> = Vec::new();
200
+
let mut has_tracked_op = false;
201
+
202
+
for op in ops {
203
+
let is_tracked = config.tracked_collections.matches(&op.collection);
204
+
let is_forwarded = config.forward_collections.matches(&op.collection);
205
+
206
+
if !is_tracked && !is_forwarded {
207
+
continue;
208
+
}
209
+
210
+
if is_tracked {
211
+
has_tracked_op = true;
212
+
riblt_invalidated.insert(did.clone());
213
+
214
+
let record_key =
215
+
keys::encode_record_key(did, &op.collection, &op.rkey, rev);
216
+
217
+
match op.op {
218
+
OpType::Create | OpType::Update => {
219
+
let cid_bytes = op.cid.as_deref().unwrap_or_default().to_vec();
220
+
let data = block_map.get(&cid_bytes).cloned().unwrap_or_default();
221
+
let record_value = RecordValue {
222
+
cid: cid_bytes,
223
+
data,
224
+
};
225
+
let compressed_record = db.compress_event(&record_value.encode());
226
+
batch.insert(&db.records, &record_key, compressed_record);
227
+
metrics
228
+
.storage_ops_total
229
+
.get_or_create(&KeyspaceOpLabels {
230
+
keyspace: "records".to_string(),
231
+
op: StorageOp::Write,
232
+
})
233
+
.inc();
234
+
}
235
+
OpType::Delete => {
236
+
batch.insert(&db.records, &record_key, b"");
237
+
metrics
238
+
.storage_ops_total
239
+
.get_or_create(&KeyspaceOpLabels {
240
+
keyspace: "records".to_string(),
241
+
op: StorageOp::Delete,
242
+
})
243
+
.inc();
244
+
}
245
+
}
246
+
247
+
let event_op = match op.op {
248
+
OpType::Create => EventOp::Create,
249
+
OpType::Update => EventOp::Update,
250
+
OpType::Delete => EventOp::Delete,
251
+
};
252
+
metrics
253
+
.writer_ops_by_collection
254
+
.get_or_create(&CollectionOpLabels {
255
+
collection: op.collection.clone(),
256
+
op: event_op,
257
+
})
258
+
.inc();
259
+
}
260
+
261
+
let data = op
262
+
.cid
263
+
.as_ref()
264
+
.and_then(|cid_bytes| block_map.get(cid_bytes.as_slice()))
265
+
.cloned()
266
+
.unwrap_or_default();
267
+
268
+
filtered_ops.push(FilteredOp {
269
+
op: op.clone(),
270
+
is_tracked,
271
+
data,
272
+
});
273
+
}
274
+
275
+
// Update repo_state using in-batch cache.
276
+
let mut repo_state = get_repo_state(did, &repo_states, db);
277
+
278
+
// Detect desync: if `since` doesn't match our stored rev, we
279
+
// missed intermediate commits and need to re-backfill.
280
+
let is_desynced = if let Some(s) = since {
281
+
!repo_state.rev.is_empty() && s != &repo_state.rev
282
+
} else {
283
+
false
284
+
};
285
+
286
+
let prev_rev = repo_state.rev.clone();
287
+
repo_state.rev = rev.clone();
288
+
let needs_backfill = has_tracked_op && !repo_state.backfilled;
289
+
let needs_resync = has_tracked_op && repo_state.backfilled && is_desynced;
290
+
if needs_resync {
291
+
repo_state.backfilled = false;
292
+
tracing::info!(
293
+
%did,
294
+
since = since.as_deref().unwrap_or(""),
295
+
stored_rev = %prev_rev,
296
+
"commit desync detected, queueing re-backfill"
297
+
);
298
+
}
299
+
batch.insert(&db.repo_state, did.as_bytes(), repo_state.encode());
300
+
repo_states.insert(did.clone(), repo_state);
301
+
metrics
302
+
.storage_ops_total
303
+
.get_or_create(&KeyspaceOpLabels {
304
+
keyspace: "repo_state".to_string(),
305
+
op: StorageOp::Write,
306
+
})
307
+
.inc();
308
+
309
+
if *too_big || needs_backfill || needs_resync {
310
+
let queue_key = format!("backfill_queue\x00{did}");
311
+
batch.insert(&db.meta, queue_key.as_bytes(), b"");
312
+
}
313
+
314
+
if filtered_ops.is_empty() && !ops.is_empty() {
315
+
metrics.writer_commits_filtered.inc();
316
+
}
317
+
318
+
// Emit one event per filtered op (granular fan-out)
319
+
let time_us = now_micros();
320
+
for fop in &filtered_ops {
321
+
let event_seq = db.next_event_seq();
322
+
323
+
let compact_op = CompactOp {
324
+
action: fop.op.op,
325
+
collection: fop.op.collection.clone(),
326
+
rkey: fop.op.rkey.clone(),
327
+
cid: fop.op.cid.clone(),
328
+
data: fop.data.clone(),
329
+
};
330
+
331
+
let compact_event = encoding::encode_compact_commit_op(
332
+
event_seq,
333
+
time_us,
334
+
did,
335
+
rev,
336
+
&compact_op,
337
+
);
338
+
let compressed = db.compress_event(&compact_event);
339
+
let event_key = keys::encode_event_key(event_seq);
340
+
batch.insert(&db.events, &event_key, &compressed);
341
+
metrics
342
+
.storage_ops_total
343
+
.get_or_create(&KeyspaceOpLabels {
344
+
keyspace: "events".to_string(),
345
+
op: StorageOp::Write,
346
+
})
347
+
.inc();
348
+
349
+
let payload =
350
+
create_commit_op_payload(event_seq, time_us, did, rev, &fop.op, &block_map);
351
+
let fan_event = Arc::new(FanOutEvent {
352
+
seq: event_seq,
353
+
did: did.as_str().into(),
354
+
event_type: EventType::Commit {
355
+
collection: fop.op.collection.clone(),
356
+
},
357
+
payload,
358
+
});
359
+
360
+
pending_fanout.push((fan_event, fop.is_tracked));
361
+
}
362
+
}
363
+
364
+
IngestEvent::Identity { seq, did, handle } => {
365
+
max_seq = max_seq.max(*seq);
366
+
367
+
if let Some(h) = handle {
368
+
batch.insert(
369
+
&db.handle_to_did,
370
+
h.to_lowercase().as_bytes(),
371
+
did.as_bytes(),
372
+
);
373
+
metrics
374
+
.storage_ops_total
375
+
.get_or_create(&KeyspaceOpLabels {
376
+
keyspace: "handle_to_did".to_string(),
377
+
op: StorageOp::Write,
378
+
})
379
+
.inc();
380
+
}
381
+
382
+
let event_seq = db.next_event_seq();
383
+
let time_us = now_micros();
384
+
385
+
let compact_event = encoding::encode_compact_identity_v2(
386
+
event_seq,
387
+
time_us,
388
+
did,
389
+
handle.as_deref(),
390
+
);
391
+
let compressed = db.compress_event(&compact_event);
392
+
let event_key = keys::encode_event_key(event_seq);
393
+
batch.insert(&db.events, &event_key, &compressed);
394
+
metrics
395
+
.storage_ops_total
396
+
.get_or_create(&KeyspaceOpLabels {
397
+
keyspace: "events".to_string(),
398
+
op: StorageOp::Write,
399
+
})
400
+
.inc();
401
+
402
+
let payload = create_identity_payload(event_seq, time_us, did, handle.as_deref());
403
+
let fan_event = Arc::new(FanOutEvent {
404
+
seq: event_seq,
405
+
did: did.as_str().into(),
406
+
event_type: EventType::Identity,
407
+
payload,
408
+
});
409
+
410
+
pending_fanout.push((fan_event, true));
411
+
}
412
+
413
+
IngestEvent::Account {
414
+
seq,
415
+
did,
416
+
active,
417
+
status,
418
+
..
419
+
} => {
420
+
max_seq = max_seq.max(*seq);
421
+
422
+
let mut repo_state = get_repo_state(did, &repo_states, db);
423
+
repo_state.status = if *active {
424
+
AccountStatus::Active
425
+
} else {
426
+
match status.as_deref() {
427
+
Some("deactivated") => AccountStatus::Deactivated,
428
+
Some("suspended") => AccountStatus::Suspended,
429
+
Some("deleted") => AccountStatus::Deleted,
430
+
Some("takendown") => AccountStatus::Takendown,
431
+
_ => AccountStatus::Deactivated,
432
+
}
433
+
};
434
+
batch.insert(&db.repo_state, did.as_bytes(), repo_state.encode());
435
+
repo_states.insert(did.clone(), repo_state);
436
+
metrics
437
+
.storage_ops_total
438
+
.get_or_create(&KeyspaceOpLabels {
439
+
keyspace: "repo_state".to_string(),
440
+
op: StorageOp::Write,
441
+
})
442
+
.inc();
443
+
444
+
let event_seq = db.next_event_seq();
445
+
let time_us = now_micros();
446
+
447
+
let compact_event = encoding::encode_compact_account_v2(
448
+
event_seq,
449
+
time_us,
450
+
did,
451
+
*active,
452
+
status.as_deref(),
453
+
);
454
+
let compressed = db.compress_event(&compact_event);
455
+
let event_key = keys::encode_event_key(event_seq);
456
+
batch.insert(&db.events, &event_key, &compressed);
457
+
metrics
458
+
.storage_ops_total
459
+
.get_or_create(&KeyspaceOpLabels {
460
+
keyspace: "events".to_string(),
461
+
op: StorageOp::Write,
462
+
})
463
+
.inc();
464
+
465
+
let payload =
466
+
create_account_payload(event_seq, time_us, did, *active, status.as_deref());
467
+
let fan_event = Arc::new(FanOutEvent {
468
+
seq: event_seq,
469
+
did: did.as_str().into(),
470
+
event_type: EventType::Account,
471
+
payload,
472
+
});
473
+
474
+
pending_fanout.push((fan_event, true));
475
+
}
476
+
477
+
IngestEvent::Sync { seq, did, rev } => {
478
+
max_seq = max_seq.max(*seq);
479
+
480
+
let repo_state = get_repo_state(did, &repo_states, db);
481
+
482
+
// Detect desync: if our stored rev doesn't match the sync
483
+
// event's rev, we're out of date and need to re-backfill.
484
+
if !repo_state.rev.is_empty() && repo_state.rev != *rev {
485
+
let stored_rev = repo_state.rev.clone();
486
+
let queue_key = format!("backfill_queue\x00{did}");
487
+
batch.insert(&db.meta, queue_key.as_bytes(), b"");
488
+
489
+
// Mark as not backfilled so the backfill worker will
490
+
// re-process it.
491
+
let mut updated_state = repo_state;
492
+
updated_state.backfilled = false;
493
+
batch.insert(&db.repo_state, did.as_bytes(), updated_state.encode());
494
+
repo_states.insert(did.clone(), updated_state);
495
+
metrics
496
+
.storage_ops_total
497
+
.get_or_create(&KeyspaceOpLabels {
498
+
keyspace: "repo_state".to_string(),
499
+
op: StorageOp::Write,
500
+
})
501
+
.inc();
502
+
503
+
tracing::info!(
504
+
%did,
505
+
%stored_rev,
506
+
sync_rev = %rev,
507
+
"sync desync detected, queueing re-backfill"
508
+
);
509
+
}
510
+
}
511
+
}
512
+
}
513
+
514
+
if max_seq > 0 {
515
+
batch.insert(&db.meta, b"cursor", &max_seq.to_be_bytes());
516
+
}
517
+
batch.insert(&db.meta, b"event_seq", &db.current_sequence().to_be_bytes());
518
+
519
+
// COMMIT — only return fan-out events after persistence succeeds
520
+
batch.commit()?;
521
+
522
+
// Invalidate RIBLT sketch caches for DIDs with tracked record changes.
523
+
for did in &riblt_invalidated {
524
+
reconciliation::invalidate_sketch_cache(db, did);
525
+
}
526
+
527
+
Ok(pending_fanout)
528
+
}
529
+
530
+
/// Create the DAG-CBOR payload for a single commit operation.
531
+
fn create_commit_op_payload(
532
+
seq: u64,
533
+
_time_us: u64,
534
+
did: &str,
535
+
rev: &str,
536
+
op: &CommitOp,
537
+
block_map: &HashMap<Vec<u8>, Vec<u8>>,
538
+
) -> Bytes {
539
+
let operation = match op.op {
540
+
OpType::Create => "create",
541
+
OpType::Update => "update",
542
+
OpType::Delete => "delete",
543
+
};
544
+
545
+
let mut commit_fields: Vec<(&str, Ipld)> = vec![
546
+
("rev", Ipld::String(rev.to_string())),
547
+
("operation", Ipld::String(operation.to_string())),
548
+
("collection", Ipld::String(op.collection.clone())),
549
+
("rkey", Ipld::String(op.rkey.clone())),
550
+
];
551
+
552
+
// Deletes omit record and cid
553
+
if op.op != OpType::Delete {
554
+
if let Some(cid_bytes) = &op.cid {
555
+
let cid_str = cid::Cid::read_bytes(cid_bytes.as_slice())
556
+
.map(|c| c.to_string())
557
+
.unwrap_or_default();
558
+
commit_fields.push(("cid", Ipld::String(cid_str)));
559
+
560
+
if let Some(data) = block_map.get(cid_bytes.as_slice()) {
561
+
if let Ok(ipld) = atproto_dasl::drisl::from_slice::<Ipld>(data) {
562
+
commit_fields.push(("record", ipld));
563
+
}
564
+
}
565
+
}
566
+
}
567
+
568
+
let event = ipld_map(vec![
569
+
("seq", Ipld::Integer(seq.into())),
570
+
("did", Ipld::String(did.to_string())),
571
+
("kind", Ipld::String("commit".to_string())),
572
+
("commit", ipld_map(commit_fields)),
573
+
]);
574
+
575
+
to_dag_cbor_bytes(&event).into()
576
+
}
577
+
578
+
/// Create the DAG-CBOR payload for an identity event.
579
+
fn create_identity_payload(seq: u64, _time_us: u64, did: &str, handle: Option<&str>) -> Bytes {
580
+
let mut identity_fields: Vec<(&str, Ipld)> = Vec::new();
581
+
if let Some(h) = handle {
582
+
identity_fields.push(("handle", Ipld::String(h.to_string())));
583
+
}
584
+
585
+
let event = ipld_map(vec![
586
+
("seq", Ipld::Integer(seq.into())),
587
+
("did", Ipld::String(did.to_string())),
588
+
("kind", Ipld::String("identity".to_string())),
589
+
("identity", ipld_map(identity_fields)),
590
+
]);
591
+
592
+
to_dag_cbor_bytes(&event).into()
593
+
}
594
+
595
+
/// Create the DAG-CBOR payload for an account event.
596
+
fn create_account_payload(
597
+
seq: u64,
598
+
_time_us: u64,
599
+
did: &str,
600
+
active: bool,
601
+
status: Option<&str>,
602
+
) -> Bytes {
603
+
let mut account_fields: Vec<(&str, Ipld)> = vec![("active", Ipld::Bool(active))];
604
+
if let Some(s) = status {
605
+
account_fields.push(("status", Ipld::String(s.to_string())));
606
+
}
607
+
608
+
let event = ipld_map(vec![
609
+
("seq", Ipld::Integer(seq.into())),
610
+
("did", Ipld::String(did.to_string())),
611
+
("kind", Ipld::String("account".to_string())),
612
+
("account", ipld_map(account_fields)),
613
+
]);
614
+
615
+
to_dag_cbor_bytes(&event).into()
616
+
}
617
+
618
+
/// Parse a blocks CAR slice into a map of CID bytes → record data.
619
+
///
620
+
/// The firehose sends record data inside a CAR (Content-Addressable aRchive)
621
+
/// in the `blocks` field of `#commit` events. Each block's CID corresponds
622
+
/// to an op's CID, and its data is the DAG-CBOR encoded record.
623
+
fn parse_blocks_car(blocks: &[u8]) -> HashMap<Vec<u8>, Vec<u8>> {
624
+
let mut map = HashMap::new();
625
+
if blocks.is_empty() {
626
+
return map;
627
+
}
628
+
629
+
let mut cursor = std::io::Cursor::new(blocks);
630
+
631
+
// Skip the CAR header
632
+
if atproto_dasl::CarHeader::read_from(&mut cursor).is_err() {
633
+
return map;
634
+
}
635
+
636
+
// Read all blocks
637
+
loop {
638
+
match atproto_dasl::CarBlock::read_from(&mut cursor) {
639
+
Ok(Some(block)) => {
640
+
let cid_bytes = block.cid.to_bytes();
641
+
map.insert(cid_bytes, block.data);
642
+
}
643
+
Ok(None) => break,
644
+
Err(_) => break,
645
+
}
646
+
}
647
+
648
+
map
649
+
}
+238
src/server/admin.rs
+238
src/server/admin.rs
···
1
+
//! Admin XRPC endpoint handlers with authorization.
2
+
3
+
use axum::Json;
4
+
use axum::extract::{Query, State};
5
+
use axum::http::StatusCode;
6
+
use serde::Deserialize;
7
+
use serde_json::{Value, json};
8
+
9
+
use crate::server::AppState;
10
+
use crate::server::reconciliation;
11
+
use crate::types::{AccountStatus, RepoState};
12
+
13
+
#[derive(Deserialize)]
14
+
pub struct GetStateParams {
15
+
pub did: Option<String>,
16
+
}
17
+
18
+
#[derive(Deserialize)]
19
+
pub struct SetStateBody {
20
+
pub did: String,
21
+
pub denied: bool,
22
+
}
23
+
24
+
#[derive(Deserialize)]
25
+
pub struct ResyncBody {
26
+
pub did: String,
27
+
}
28
+
29
+
#[derive(Deserialize)]
30
+
pub struct RebuildReconciliationBody {
31
+
pub did: String,
32
+
}
33
+
34
+
fn error_response(status: StatusCode, error: &str, message: &str) -> (StatusCode, Json<Value>) {
35
+
(status, Json(json!({ "error": error, "message": message })))
36
+
}
37
+
38
+
/// Extract admin DID from Authorization header.
39
+
fn check_admin_auth(
40
+
headers: &axum::http::HeaderMap,
41
+
state: &AppState,
42
+
) -> Result<String, (StatusCode, Json<Value>)> {
43
+
let auth_header = headers
44
+
.get(axum::http::header::AUTHORIZATION)
45
+
.and_then(|v| v.to_str().ok())
46
+
.and_then(|s| s.strip_prefix("Bearer "));
47
+
48
+
let Some(token) = auth_header else {
49
+
return Err(error_response(
50
+
StatusCode::UNAUTHORIZED,
51
+
"AuthRequired",
52
+
"authorization required",
53
+
));
54
+
};
55
+
56
+
let parts: Vec<&str> = token.split('.').collect();
57
+
if parts.len() != 3 {
58
+
return Err(error_response(
59
+
StatusCode::UNAUTHORIZED,
60
+
"InvalidToken",
61
+
"invalid JWT format",
62
+
));
63
+
}
64
+
65
+
use base64::Engine;
66
+
let payload_bytes = base64::engine::general_purpose::URL_SAFE_NO_PAD
67
+
.decode(parts[1])
68
+
.map_err(|_| {
69
+
error_response(
70
+
StatusCode::UNAUTHORIZED,
71
+
"InvalidToken",
72
+
"invalid JWT payload encoding",
73
+
)
74
+
})?;
75
+
76
+
let claims: Value = serde_json::from_slice(&payload_bytes).map_err(|_| {
77
+
error_response(
78
+
StatusCode::UNAUTHORIZED,
79
+
"InvalidToken",
80
+
"invalid JWT claims",
81
+
)
82
+
})?;
83
+
84
+
let issuer = claims
85
+
.get("iss")
86
+
.or_else(|| claims.get("sub"))
87
+
.and_then(|v| v.as_str())
88
+
.ok_or_else(|| {
89
+
error_response(StatusCode::UNAUTHORIZED, "InvalidToken", "no issuer in JWT")
90
+
})?;
91
+
92
+
if !state.config.admin_dids.contains(issuer) {
93
+
return Err(error_response(
94
+
StatusCode::FORBIDDEN,
95
+
"Forbidden",
96
+
"not an admin",
97
+
));
98
+
}
99
+
100
+
Ok(issuer.to_string())
101
+
}
102
+
103
+
/// Handle `GET /xrpc/dev.ngerakines.ramjet.repos.getState`.
104
+
pub async fn handle_get_state(
105
+
State(state): State<AppState>,
106
+
Query(params): Query<GetStateParams>,
107
+
) -> (StatusCode, Json<Value>) {
108
+
let Some(did) = params.did.as_deref().filter(|s| !s.is_empty()) else {
109
+
return error_response(
110
+
StatusCode::BAD_REQUEST,
111
+
"InvalidRequest",
112
+
"missing required parameter: did",
113
+
);
114
+
};
115
+
116
+
match state.db.get_repo_state(did) {
117
+
Ok(Some(rs)) => (
118
+
StatusCode::OK,
119
+
Json(json!({
120
+
"did": did,
121
+
"rev": rs.rev,
122
+
"status": rs.status.as_str(),
123
+
"denied": rs.denied,
124
+
})),
125
+
),
126
+
Ok(None) => error_response(StatusCode::NOT_FOUND, "RepoNotFound", "repo not found"),
127
+
Err(e) => error_response(
128
+
StatusCode::INTERNAL_SERVER_ERROR,
129
+
"InternalError",
130
+
&format!("storage error: {e}"),
131
+
),
132
+
}
133
+
}
134
+
135
+
/// Handle `POST /xrpc/dev.ngerakines.ramjet.repos.setState`.
136
+
pub async fn handle_set_state(
137
+
State(state): State<AppState>,
138
+
headers: axum::http::HeaderMap,
139
+
Json(body): Json<SetStateBody>,
140
+
) -> (StatusCode, Json<Value>) {
141
+
if let Err(resp) = check_admin_auth(&headers, &state) {
142
+
return resp;
143
+
}
144
+
145
+
let mut repo_state = match state.db.get_repo_state(&body.did) {
146
+
Ok(Some(rs)) => rs,
147
+
Ok(None) => RepoState {
148
+
rev: "0000000000000".to_string(),
149
+
status: AccountStatus::Active,
150
+
denied: false,
151
+
backfilled: false,
152
+
},
153
+
Err(e) => {
154
+
return error_response(
155
+
StatusCode::INTERNAL_SERVER_ERROR,
156
+
"InternalError",
157
+
&format!("storage error: {e}"),
158
+
);
159
+
}
160
+
};
161
+
162
+
repo_state.denied = body.denied;
163
+
164
+
if let Err(e) = state
165
+
.db
166
+
.repo_state
167
+
.insert(body.did.as_bytes(), repo_state.encode())
168
+
{
169
+
return error_response(
170
+
StatusCode::INTERNAL_SERVER_ERROR,
171
+
"InternalError",
172
+
&format!("storage error: {e}"),
173
+
);
174
+
}
175
+
176
+
(
177
+
StatusCode::OK,
178
+
Json(json!({
179
+
"did": body.did,
180
+
"rev": repo_state.rev,
181
+
"status": repo_state.status.as_str(),
182
+
"denied": repo_state.denied,
183
+
})),
184
+
)
185
+
}
186
+
187
+
/// Handle `POST /xrpc/dev.ngerakines.ramjet.repos.resync`.
188
+
pub async fn handle_resync(
189
+
State(state): State<AppState>,
190
+
headers: axum::http::HeaderMap,
191
+
Json(body): Json<ResyncBody>,
192
+
) -> (StatusCode, Json<Value>) {
193
+
if let Err(resp) = check_admin_auth(&headers, &state) {
194
+
return resp;
195
+
}
196
+
197
+
let queue_key = format!("backfill_queue\x00{}", body.did);
198
+
if let Err(e) = state.db.meta.insert(queue_key.as_bytes(), b"") {
199
+
return error_response(
200
+
StatusCode::INTERNAL_SERVER_ERROR,
201
+
"InternalError",
202
+
&format!("storage error: {e}"),
203
+
);
204
+
}
205
+
206
+
(
207
+
StatusCode::OK,
208
+
Json(json!({
209
+
"did": body.did,
210
+
"queued": true,
211
+
})),
212
+
)
213
+
}
214
+
215
+
/// Handle `POST /xrpc/dev.ngerakines.ramjet.repos.rebuildReconciliation`.
216
+
///
217
+
/// Forces a rebuild of the RIBLT reconciliation sketch for a DID by
218
+
/// invalidating the cache. The next `getReconciliation` call will
219
+
/// build a fresh sketch from the records keyspace.
220
+
pub async fn handle_rebuild_reconciliation(
221
+
State(state): State<AppState>,
222
+
headers: axum::http::HeaderMap,
223
+
Json(body): Json<RebuildReconciliationBody>,
224
+
) -> (StatusCode, Json<Value>) {
225
+
if let Err(resp) = check_admin_auth(&headers, &state) {
226
+
return resp;
227
+
}
228
+
229
+
reconciliation::invalidate_sketch_cache(&state.db, &body.did);
230
+
231
+
(
232
+
StatusCode::OK,
233
+
Json(json!({
234
+
"did": body.did,
235
+
"invalidated": true,
236
+
})),
237
+
)
238
+
}
+45
src/server/dictionary.rs
+45
src/server/dictionary.rs
···
1
+
//! Zstd dictionary download endpoint.
2
+
3
+
use axum::extract::State;
4
+
use axum::http::StatusCode;
5
+
use axum::http::header::{ETAG, IF_NONE_MATCH};
6
+
use axum::response::{IntoResponse, Response};
7
+
8
+
use super::AppState;
9
+
10
+
/// Compute a CIDv1 (raw codec, SHA-256) string for the given bytes.
11
+
pub fn compute_raw_cid(data: &[u8]) -> String {
12
+
use sha2::Digest;
13
+
let digest = sha2::Sha256::digest(data);
14
+
let mh = multihash::Multihash::<64>::wrap(0x12, &digest).expect("SHA-256 digest is 32 bytes");
15
+
let cid = cid::Cid::new_v1(0x55, mh);
16
+
cid.to_string()
17
+
}
18
+
19
+
/// Handle `GET /dictionary`.
20
+
pub async fn handle_dictionary(
21
+
State(state): State<AppState>,
22
+
headers: axum::http::HeaderMap,
23
+
) -> Response {
24
+
let Some(dict_bytes) = state.db.zstd_dict() else {
25
+
return StatusCode::NOT_FOUND.into_response();
26
+
};
27
+
28
+
let etag = compute_raw_cid(dict_bytes);
29
+
30
+
if let Some(if_none_match) = headers.get(IF_NONE_MATCH).and_then(|v| v.to_str().ok()) {
31
+
if if_none_match == etag {
32
+
return (StatusCode::NOT_MODIFIED, [(ETAG, etag)]).into_response();
33
+
}
34
+
}
35
+
36
+
(
37
+
[(ETAG, etag)],
38
+
[(
39
+
axum::http::header::CONTENT_TYPE,
40
+
"application/octet-stream".to_string(),
41
+
)],
42
+
dict_bytes.to_vec(),
43
+
)
44
+
.into_response()
45
+
}
+21
src/server/health.rs
+21
src/server/health.rs
···
1
+
//! Health check endpoint.
2
+
3
+
use axum::Json;
4
+
use serde::Serialize;
5
+
6
+
/// Health check response.
7
+
#[derive(Serialize)]
8
+
pub struct HealthResponse {
9
+
/// Service status.
10
+
pub status: &'static str,
11
+
/// Service version.
12
+
pub version: &'static str,
13
+
}
14
+
15
+
/// Handle `GET /_health`.
16
+
pub async fn handle_health() -> Json<HealthResponse> {
17
+
Json(HealthResponse {
18
+
status: "ok",
19
+
version: env!("CARGO_PKG_VERSION"),
20
+
})
21
+
}
+451
src/server/metrics.rs
+451
src/server/metrics.rs
···
1
+
//! Prometheus metrics registry and HTTP handler.
2
+
3
+
use axum::http::header::CONTENT_TYPE;
4
+
use axum::response::IntoResponse;
5
+
use prometheus_client::encoding::text::encode;
6
+
use prometheus_client::encoding::{EncodeLabelSet, EncodeLabelValue};
7
+
use prometheus_client::metrics::counter::Counter;
8
+
use prometheus_client::metrics::family::Family;
9
+
use prometheus_client::metrics::gauge::Gauge;
10
+
use prometheus_client::metrics::histogram::{Histogram, exponential_buckets};
11
+
use prometheus_client::registry::Registry;
12
+
use std::sync::Arc;
13
+
use std::sync::atomic::AtomicU64;
14
+
15
+
use super::tokio_metrics::TokioMetrics;
16
+
17
+
// -- Label types --
18
+
19
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
20
+
pub struct KeyspaceOpLabels {
21
+
pub keyspace: String,
22
+
pub op: StorageOp,
23
+
}
24
+
25
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelValue)]
26
+
pub enum StorageOp {
27
+
Write,
28
+
Read,
29
+
Delete,
30
+
}
31
+
32
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
33
+
pub struct CollectionOpLabels {
34
+
pub collection: String,
35
+
pub op: EventOp,
36
+
}
37
+
38
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelValue)]
39
+
pub enum EventOp {
40
+
Create,
41
+
Update,
42
+
Delete,
43
+
}
44
+
45
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
46
+
pub struct EventTypeLabels {
47
+
pub event_type: String,
48
+
}
49
+
50
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
51
+
pub struct IdentityOutcomeLabels {
52
+
pub outcome: IdentityOutcome,
53
+
}
54
+
55
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelValue)]
56
+
pub enum IdentityOutcome {
57
+
Success,
58
+
Failure,
59
+
Skipped,
60
+
}
61
+
62
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
63
+
pub struct FanoutPriorityLabels {
64
+
pub priority: FanoutPriority,
65
+
}
66
+
67
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelValue)]
68
+
pub enum FanoutPriority {
69
+
High,
70
+
Low,
71
+
}
72
+
73
+
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
74
+
pub struct HttpRequestLabels {
75
+
pub method: String,
76
+
pub path: String,
77
+
pub status: u16,
78
+
}
79
+
80
+
/// Prometheus metrics for the Ramjet service.
81
+
pub struct Metrics {
82
+
/// The prometheus-client registry.
83
+
pub registry: Registry,
84
+
85
+
// -- Firehose --
86
+
/// Total events received from the upstream relay.
87
+
pub firehose_events_total: Counter<u64, AtomicU64>,
88
+
/// Current upstream sequence number.
89
+
pub firehose_seq: Gauge<u64, AtomicU64>,
90
+
/// Whether the firehose is connected (1) or not (0).
91
+
pub firehose_connection_state: Gauge<u64, AtomicU64>,
92
+
/// Total reconnections to the upstream relay.
93
+
pub firehose_reconnects_total: Counter<u64, AtomicU64>,
94
+
/// Firehose events by type (#commit, #identity, #account).
95
+
pub firehose_events_by_type: Family<EventTypeLabels, Counter<u64, AtomicU64>>,
96
+
97
+
/// Current depth of the ingester local buffer (VecDeque).
98
+
pub ingester_buffer_depth: Gauge<u64, AtomicU64>,
99
+
/// Time spent processing each item popped from the ingester work queue.
100
+
pub ingester_process_duration_seconds: Histogram,
101
+
102
+
// -- Storage --
103
+
/// Storage operations by keyspace and operation type.
104
+
pub storage_ops_total: Family<KeyspaceOpLabels, Counter<u64, AtomicU64>>,
105
+
106
+
// -- Writer --
107
+
/// Total WriteBatch commits.
108
+
pub writer_batches_total: Counter<u64, AtomicU64>,
109
+
/// Events processed per batch.
110
+
pub writer_batch_size: Histogram,
111
+
/// Event-stream operations by collection and op type.
112
+
pub writer_ops_by_collection: Family<CollectionOpLabels, Counter<u64, AtomicU64>>,
113
+
/// Total events sent to fan-out channels.
114
+
pub writer_events_fanned_total: Counter<u64, AtomicU64>,
115
+
/// Commits skipped because repo is denied.
116
+
pub writer_commits_denied: Counter<u64, AtomicU64>,
117
+
/// Commits with zero ops in the ops array.
118
+
pub writer_commits_empty_ops: Counter<u64, AtomicU64>,
119
+
/// Commits where ops existed but none matched collection filters.
120
+
pub writer_commits_filtered: Counter<u64, AtomicU64>,
121
+
122
+
// -- Identity --
123
+
/// Identity resolution attempts total.
124
+
pub identity_resolves_total: Family<IdentityOutcomeLabels, Counter<u64, AtomicU64>>,
125
+
/// Identity resolution latency in seconds.
126
+
pub identity_resolve_duration_seconds: Histogram,
127
+
128
+
// -- Fan-out --
129
+
/// Current number of connected WebSocket consumers.
130
+
pub fanout_connections_active: Gauge<u64, AtomicU64>,
131
+
/// Total WebSocket connections accepted.
132
+
pub fanout_connections_total: Counter<u64, AtomicU64>,
133
+
/// Total events dropped due to consumer lag.
134
+
pub fanout_lagged_total: Family<FanoutPriorityLabels, Counter<u64, AtomicU64>>,
135
+
/// Fan-out broadcast queue depth (approximate).
136
+
pub fanout_queue_depth: Family<FanoutPriorityLabels, Gauge<u64, AtomicU64>>,
137
+
/// Connections evicted due to sustained consumer lag.
138
+
pub fanout_connections_evicted: Counter<u64, AtomicU64>,
139
+
/// Total events actually sent over WebSocket to consumers.
140
+
pub fanout_events_sent_total: Counter<u64, AtomicU64>,
141
+
142
+
// -- Backfill --
143
+
/// Number of repos pending backfill.
144
+
pub backfill_queue_depth: Gauge<u64, AtomicU64>,
145
+
/// Total repos successfully backfilled.
146
+
pub backfill_repos_total: Counter<u64, AtomicU64>,
147
+
/// Total records written during backfill.
148
+
pub backfill_records_total: Counter<u64, AtomicU64>,
149
+
/// Backfill duration per repo in seconds.
150
+
pub backfill_duration_seconds: Histogram,
151
+
/// Total backfill failures.
152
+
pub backfill_failures_total: Counter<u64, AtomicU64>,
153
+
154
+
// -- HTTP --
155
+
/// Total HTTP requests.
156
+
pub http_requests_total: Counter<u64, AtomicU64>,
157
+
/// HTTP requests by method, path, and status code.
158
+
pub http_requests_by_route: Family<HttpRequestLabels, Counter<u64, AtomicU64>>,
159
+
/// HTTP request duration in seconds by method and path.
160
+
pub http_request_duration_seconds: Histogram,
161
+
162
+
// -- Tokio task metrics --
163
+
/// Tokio task monitor metrics (poll durations, scheduling delays, etc.).
164
+
pub tokio_metrics: Arc<TokioMetrics>,
165
+
}
166
+
167
+
impl Metrics {
168
+
/// Create a new metrics instance with all counters registered.
169
+
pub fn new() -> Self {
170
+
let mut registry = Registry::default();
171
+
172
+
// -- Firehose --
173
+
let firehose_events_total = Counter::default();
174
+
registry.register(
175
+
"ramjet_firehose_events_total",
176
+
"Total events received from the upstream relay",
177
+
firehose_events_total.clone(),
178
+
);
179
+
180
+
let firehose_seq = Gauge::default();
181
+
registry.register(
182
+
"ramjet_firehose_seq",
183
+
"Most recently processed upstream sequence number",
184
+
firehose_seq.clone(),
185
+
);
186
+
187
+
let firehose_connection_state = Gauge::default();
188
+
registry.register(
189
+
"ramjet_firehose_connection_state",
190
+
"1 if connected to relay, 0 if disconnected",
191
+
firehose_connection_state.clone(),
192
+
);
193
+
194
+
let firehose_reconnects_total = Counter::default();
195
+
registry.register(
196
+
"ramjet_firehose_reconnects_total",
197
+
"Total reconnections to the upstream relay",
198
+
firehose_reconnects_total.clone(),
199
+
);
200
+
201
+
let firehose_events_by_type = Family::default();
202
+
registry.register(
203
+
"ramjet_firehose_events_by_type_total",
204
+
"Firehose events by type",
205
+
firehose_events_by_type.clone(),
206
+
);
207
+
208
+
let ingester_buffer_depth = Gauge::default();
209
+
registry.register(
210
+
"ramjet_ingester_buffer_depth",
211
+
"Current depth of the ingester local buffer",
212
+
ingester_buffer_depth.clone(),
213
+
);
214
+
215
+
let ingester_process_duration_seconds =
216
+
Histogram::new(exponential_buckets(0.000_001, 4.0, 12));
217
+
registry.register(
218
+
"ramjet_ingester_process_duration_seconds",
219
+
"Time spent processing each item popped from the ingester work queue",
220
+
ingester_process_duration_seconds.clone(),
221
+
);
222
+
223
+
// -- Storage --
224
+
let storage_ops_total = Family::default();
225
+
registry.register(
226
+
"ramjet_storage_ops_total",
227
+
"Storage operations by keyspace and operation type",
228
+
storage_ops_total.clone(),
229
+
);
230
+
231
+
// -- Writer --
232
+
let writer_batches_total = Counter::default();
233
+
registry.register(
234
+
"ramjet_writer_batches_total",
235
+
"Total WriteBatch commits",
236
+
writer_batches_total.clone(),
237
+
);
238
+
239
+
let writer_batch_size = Histogram::new(exponential_buckets(1.0, 2.0, 12));
240
+
registry.register(
241
+
"ramjet_writer_batch_size",
242
+
"Number of events per writer batch",
243
+
writer_batch_size.clone(),
244
+
);
245
+
246
+
let writer_ops_by_collection = Family::default();
247
+
registry.register(
248
+
"ramjet_writer_ops_by_collection_total",
249
+
"Record operations by collection and operation type",
250
+
writer_ops_by_collection.clone(),
251
+
);
252
+
253
+
let writer_events_fanned_total = Counter::default();
254
+
registry.register(
255
+
"ramjet_writer_events_fanned",
256
+
"Total events sent to fan-out channels",
257
+
writer_events_fanned_total.clone(),
258
+
);
259
+
260
+
let writer_commits_denied = Counter::default();
261
+
registry.register(
262
+
"ramjet_writer_commits_denied",
263
+
"Commits skipped because repo is denied",
264
+
writer_commits_denied.clone(),
265
+
);
266
+
267
+
let writer_commits_empty_ops = Counter::default();
268
+
registry.register(
269
+
"ramjet_writer_commits_empty_ops",
270
+
"Commits with zero ops in the ops array",
271
+
writer_commits_empty_ops.clone(),
272
+
);
273
+
274
+
let writer_commits_filtered = Counter::default();
275
+
registry.register(
276
+
"ramjet_writer_commits_filtered",
277
+
"Commits where ops existed but none matched collection filters",
278
+
writer_commits_filtered.clone(),
279
+
);
280
+
281
+
// -- Identity --
282
+
let identity_resolves_total = Family::default();
283
+
registry.register(
284
+
"ramjet_identity_resolves_total",
285
+
"Identity resolution attempts by outcome",
286
+
identity_resolves_total.clone(),
287
+
);
288
+
289
+
let identity_resolve_duration_seconds = Histogram::new(exponential_buckets(0.01, 2.0, 12));
290
+
registry.register(
291
+
"ramjet_identity_resolve_duration_seconds",
292
+
"Identity resolution latency in seconds",
293
+
identity_resolve_duration_seconds.clone(),
294
+
);
295
+
296
+
// -- Fan-out --
297
+
let fanout_connections_active = Gauge::default();
298
+
registry.register(
299
+
"ramjet_fanout_connections_active",
300
+
"Currently connected WebSocket consumers",
301
+
fanout_connections_active.clone(),
302
+
);
303
+
304
+
let fanout_connections_total = Counter::default();
305
+
registry.register(
306
+
"ramjet_fanout_connections_total",
307
+
"Total WebSocket connections accepted",
308
+
fanout_connections_total.clone(),
309
+
);
310
+
311
+
let fanout_lagged_total = Family::default();
312
+
registry.register(
313
+
"ramjet_fanout_lagged_total",
314
+
"Events dropped due to consumer lag",
315
+
fanout_lagged_total.clone(),
316
+
);
317
+
318
+
let fanout_queue_depth = Family::default();
319
+
registry.register(
320
+
"ramjet_fanout_queue_depth",
321
+
"Approximate broadcast queue depth",
322
+
fanout_queue_depth.clone(),
323
+
);
324
+
325
+
let fanout_connections_evicted = Counter::default();
326
+
registry.register(
327
+
"ramjet_fanout_connections_evicted",
328
+
"Connections evicted due to sustained consumer lag",
329
+
fanout_connections_evicted.clone(),
330
+
);
331
+
332
+
let fanout_events_sent_total = Counter::default();
333
+
registry.register(
334
+
"ramjet_fanout_events_sent",
335
+
"Total events actually sent over WebSocket to consumers",
336
+
fanout_events_sent_total.clone(),
337
+
);
338
+
339
+
// -- Backfill --
340
+
let backfill_queue_depth = Gauge::default();
341
+
registry.register(
342
+
"ramjet_backfill_queue_depth",
343
+
"Repositories pending backfill",
344
+
backfill_queue_depth.clone(),
345
+
);
346
+
347
+
let backfill_repos_total = Counter::default();
348
+
registry.register(
349
+
"ramjet_backfill_repos_total",
350
+
"Total repos successfully backfilled",
351
+
backfill_repos_total.clone(),
352
+
);
353
+
354
+
let backfill_records_total = Counter::default();
355
+
registry.register(
356
+
"ramjet_backfill_records_total",
357
+
"Total records written during backfill",
358
+
backfill_records_total.clone(),
359
+
);
360
+
361
+
let backfill_duration_seconds = Histogram::new(exponential_buckets(0.1, 2.0, 12));
362
+
registry.register(
363
+
"ramjet_backfill_duration_seconds",
364
+
"Backfill duration per repo in seconds",
365
+
backfill_duration_seconds.clone(),
366
+
);
367
+
368
+
let backfill_failures_total = Counter::default();
369
+
registry.register(
370
+
"ramjet_backfill_failures_total",
371
+
"Total backfill failures",
372
+
backfill_failures_total.clone(),
373
+
);
374
+
375
+
// -- HTTP --
376
+
let http_requests_total = Counter::default();
377
+
registry.register(
378
+
"ramjet_http_requests_total",
379
+
"Total HTTP requests",
380
+
http_requests_total.clone(),
381
+
);
382
+
383
+
let http_requests_by_route = Family::default();
384
+
registry.register(
385
+
"ramjet_http_requests_by_route_total",
386
+
"HTTP requests by method, path, and status code",
387
+
http_requests_by_route.clone(),
388
+
);
389
+
390
+
let http_request_duration_seconds = Histogram::new(exponential_buckets(0.000_1, 4.0, 12));
391
+
registry.register(
392
+
"ramjet_http_request_duration_seconds",
393
+
"HTTP request duration in seconds",
394
+
http_request_duration_seconds.clone(),
395
+
);
396
+
397
+
// -- Tokio task metrics --
398
+
let tokio_metrics = Arc::new(TokioMetrics::register(&mut registry));
399
+
400
+
Self {
401
+
registry,
402
+
firehose_events_total,
403
+
firehose_seq,
404
+
firehose_connection_state,
405
+
firehose_reconnects_total,
406
+
firehose_events_by_type,
407
+
ingester_buffer_depth,
408
+
ingester_process_duration_seconds,
409
+
storage_ops_total,
410
+
writer_batches_total,
411
+
writer_batch_size,
412
+
writer_ops_by_collection,
413
+
writer_events_fanned_total,
414
+
writer_commits_denied,
415
+
writer_commits_empty_ops,
416
+
writer_commits_filtered,
417
+
identity_resolves_total,
418
+
identity_resolve_duration_seconds,
419
+
fanout_connections_active,
420
+
fanout_connections_total,
421
+
fanout_lagged_total,
422
+
fanout_queue_depth,
423
+
fanout_connections_evicted,
424
+
fanout_events_sent_total,
425
+
backfill_queue_depth,
426
+
backfill_repos_total,
427
+
backfill_records_total,
428
+
backfill_duration_seconds,
429
+
backfill_failures_total,
430
+
http_requests_total,
431
+
http_requests_by_route,
432
+
http_request_duration_seconds,
433
+
tokio_metrics,
434
+
}
435
+
}
436
+
}
437
+
438
+
impl Default for Metrics {
439
+
fn default() -> Self {
440
+
Self::new()
441
+
}
442
+
}
443
+
444
+
/// Handle `GET /metrics` — returns Prometheus text exposition format.
445
+
pub async fn handle_metrics(
446
+
axum::extract::State(metrics): axum::extract::State<std::sync::Arc<Metrics>>,
447
+
) -> impl IntoResponse {
448
+
let mut buf = String::new();
449
+
encode(&mut buf, &metrics.registry).expect("metrics encoding failed");
450
+
([(CONTENT_TYPE, "text/plain; version=0.0.4")], buf)
451
+
}
+138
src/server/mod.rs
+138
src/server/mod.rs
···
1
+
//! HTTP server with XRPC endpoints, health check, and metrics.
2
+
3
+
pub mod admin;
4
+
pub mod dictionary;
5
+
pub mod health;
6
+
pub mod metrics;
7
+
pub mod reconciliation;
8
+
pub mod tokio_metrics;
9
+
pub mod websocket;
10
+
pub mod xrpc;
11
+
12
+
use std::sync::Arc;
13
+
14
+
use atproto_identity::traits::IdentityResolver;
15
+
use axum::Router;
16
+
use axum::middleware;
17
+
use axum::routing::{get, post};
18
+
19
+
use crate::config::ServiceConfig;
20
+
use crate::pipeline::fanout::FanOutChannels;
21
+
use crate::storage::FjallDb;
22
+
23
+
use self::metrics::Metrics;
24
+
25
+
/// Shared application state passed to all handlers.
26
+
#[derive(Clone)]
27
+
pub struct AppState {
28
+
/// Fjall database handle.
29
+
pub db: Arc<FjallDb>,
30
+
/// Service configuration.
31
+
pub config: Arc<ServiceConfig>,
32
+
/// Prometheus metrics.
33
+
pub metrics: Arc<Metrics>,
34
+
/// Fan-out broadcast channels.
35
+
pub fanout: Arc<FanOutChannels>,
36
+
/// Identity resolver for on-demand DID resolution.
37
+
pub resolver: Arc<dyn IdentityResolver>,
38
+
}
39
+
40
+
/// Build the axum router with all endpoints.
41
+
pub fn build_router(state: AppState) -> Router {
42
+
Router::new()
43
+
// Health and metrics
44
+
.route("/_health", get(health::handle_health))
45
+
.route(
46
+
"/metrics",
47
+
get(metrics::handle_metrics).with_state(state.metrics.clone()),
48
+
)
49
+
// Standard XRPC endpoints
50
+
.route(
51
+
"/xrpc/com.atproto.repo.getRecord",
52
+
get(xrpc::handle_get_record),
53
+
)
54
+
.route(
55
+
"/xrpc/com.atproto.repo.listRecords",
56
+
get(xrpc::handle_list_records),
57
+
)
58
+
.route(
59
+
"/xrpc/com.atproto.repo.describeRepo",
60
+
get(xrpc::handle_describe_repo),
61
+
)
62
+
.route(
63
+
"/xrpc/com.atproto.identity.resolveIdentity",
64
+
get(xrpc::handle_resolve_identity),
65
+
)
66
+
.route(
67
+
"/xrpc/com.atproto.identity.resolveHandle",
68
+
get(xrpc::handle_resolve_handle),
69
+
)
70
+
.route(
71
+
"/xrpc/com.atproto.identity.resolveDid",
72
+
get(xrpc::handle_resolve_did),
73
+
)
74
+
// WebSocket subscription
75
+
.route(
76
+
"/xrpc/dev.ngerakines.ramjet.stream.subscribe",
77
+
get(websocket::handle_subscribe),
78
+
)
79
+
// Admin endpoints
80
+
.route(
81
+
"/xrpc/dev.ngerakines.ramjet.repos.getState",
82
+
get(admin::handle_get_state),
83
+
)
84
+
.route(
85
+
"/xrpc/dev.ngerakines.ramjet.repos.setState",
86
+
post(admin::handle_set_state),
87
+
)
88
+
.route(
89
+
"/xrpc/dev.ngerakines.ramjet.repos.resync",
90
+
post(admin::handle_resync),
91
+
)
92
+
.route(
93
+
"/xrpc/dev.ngerakines.ramjet.repos.getReconciliation",
94
+
get(reconciliation::handle_get_reconciliation),
95
+
)
96
+
.route(
97
+
"/xrpc/dev.ngerakines.ramjet.repos.rebuildReconciliation",
98
+
post(admin::handle_rebuild_reconciliation),
99
+
)
100
+
// Dictionary download
101
+
.route("/dictionary", get(dictionary::handle_dictionary))
102
+
.layer(middleware::from_fn_with_state(
103
+
state.clone(),
104
+
count_requests_middleware,
105
+
))
106
+
.with_state(state)
107
+
}
108
+
109
+
async fn count_requests_middleware(
110
+
axum::extract::State(state): axum::extract::State<AppState>,
111
+
req: axum::extract::Request,
112
+
next: middleware::Next,
113
+
) -> axum::response::Response {
114
+
state.metrics.http_requests_total.inc();
115
+
116
+
let method = req.method().to_string();
117
+
let path = req.uri().path().to_string();
118
+
let start = std::time::Instant::now();
119
+
120
+
let response = next.run(req).await;
121
+
122
+
let status = response.status().as_u16();
123
+
state
124
+
.metrics
125
+
.http_requests_by_route
126
+
.get_or_create(&metrics::HttpRequestLabels {
127
+
method,
128
+
path,
129
+
status,
130
+
})
131
+
.inc();
132
+
state
133
+
.metrics
134
+
.http_request_duration_seconds
135
+
.observe(start.elapsed().as_secs_f64());
136
+
137
+
response
138
+
}
+371
src/server/reconciliation.rs
+371
src/server/reconciliation.rs
···
1
+
//! RIBLT set reconciliation endpoint.
2
+
//!
3
+
//! Provides an XRPC endpoint that returns an RIBLT sketch for a DID's
4
+
//! tracked records, allowing consumers to efficiently determine which
5
+
//! records they're missing via set reconciliation.
6
+
7
+
use axum::extract::{Query, State};
8
+
use axum::http::{HeaderMap, StatusCode, header};
9
+
use axum::response::{IntoResponse, Response};
10
+
use serde::Deserialize;
11
+
use serde_json::json;
12
+
13
+
use riblt::Encoder;
14
+
use riblt::byte_symbol::ByteSymbol;
15
+
use riblt::file_format::RibltFile;
16
+
17
+
use crate::server::AppState;
18
+
use crate::storage::FjallDb;
19
+
use crate::storage::keys;
20
+
use crate::types::RecordValue;
21
+
22
+
/// Sketch sizes are always multiples of this value.
23
+
const SKETCH_INCREMENT: usize = 25_000;
24
+
25
+
#[derive(Deserialize)]
26
+
pub struct GetReconciliationParams {
27
+
pub did: Option<String>,
28
+
pub collection: Option<String>,
29
+
}
30
+
31
+
/// Compute the sketch size for a given record count.
32
+
///
33
+
/// Sizes are multiples of 25,000. The sketch stays at the current tier
34
+
/// until the record count crosses 50% into the next increment, then bumps.
35
+
fn sketch_size_for_records(record_count: usize) -> usize {
36
+
let mut size = SKETCH_INCREMENT;
37
+
while record_count >= size + SKETCH_INCREMENT / 2 {
38
+
size += SKETCH_INCREMENT;
39
+
}
40
+
size
41
+
}
42
+
43
+
/// Build a ByteSymbol for a record identified by (collection, rkey, cid).
44
+
fn make_record_symbol(collection: &str, rkey: &str, cid: &[u8]) -> ByteSymbol {
45
+
let mut bytes = Vec::with_capacity(collection.len() + 1 + rkey.len() + 1 + cid.len());
46
+
bytes.extend_from_slice(collection.as_bytes());
47
+
bytes.push(0x00);
48
+
bytes.extend_from_slice(rkey.as_bytes());
49
+
bytes.push(0x00);
50
+
bytes.extend_from_slice(cid);
51
+
ByteSymbol(bytes)
52
+
}
53
+
54
+
/// Handle `GET /xrpc/dev.ngerakines.ramjet.repos.getReconciliation`.
55
+
///
56
+
/// Returns an RIBLT sketch as binary (`application/x-riblt`) for the
57
+
/// given DID's tracked records. Clients build their own sketch from local
58
+
/// records, subtract, and decode to find the symmetric difference.
59
+
///
60
+
/// Supports `If-None-Match` with a repo rev — returns 304 if unchanged.
61
+
pub async fn handle_get_reconciliation(
62
+
State(state): State<AppState>,
63
+
headers: HeaderMap,
64
+
Query(params): Query<GetReconciliationParams>,
65
+
) -> Response {
66
+
let Some(did) = params.did.as_deref().filter(|s| !s.is_empty()) else {
67
+
return error_json(
68
+
StatusCode::BAD_REQUEST,
69
+
"InvalidRequest",
70
+
"missing required parameter: did",
71
+
);
72
+
};
73
+
74
+
let repo_state = match state.db.get_repo_state(did) {
75
+
Ok(Some(rs)) => rs,
76
+
Ok(None) => {
77
+
return error_json(StatusCode::NOT_FOUND, "RepoNotFound", "repo not found");
78
+
}
79
+
Err(e) => {
80
+
return error_json(
81
+
StatusCode::INTERNAL_SERVER_ERROR,
82
+
"InternalError",
83
+
&format!("storage error: {e}"),
84
+
);
85
+
}
86
+
};
87
+
88
+
// If-None-Match: client already has a sketch at this rev
89
+
if let Some(etag) = headers
90
+
.get(header::IF_NONE_MATCH)
91
+
.and_then(|v| v.to_str().ok())
92
+
{
93
+
let etag = etag.trim_matches('"');
94
+
if etag == repo_state.rev {
95
+
return StatusCode::NOT_MODIFIED.into_response();
96
+
}
97
+
}
98
+
99
+
// Check for cached sketch (full-repo only, not collection-filtered)
100
+
let use_cache = params.collection.is_none();
101
+
if use_cache {
102
+
let cache_key = riblt_cache_key(did);
103
+
if let Ok(Some(cached)) = state.db.meta.get(cache_key.as_bytes()) {
104
+
let slice: &[u8] = &cached;
105
+
if let Some((cached_rev, record_count, riblt_bytes)) = decode_cache_value(slice) {
106
+
if cached_rev == repo_state.rev {
107
+
return riblt_response(riblt_bytes, &repo_state.rev, record_count);
108
+
}
109
+
}
110
+
}
111
+
}
112
+
113
+
// Build sketch from records scan
114
+
let db = state.db.clone();
115
+
let did_owned = did.to_string();
116
+
let collection_filter = params.collection.clone();
117
+
118
+
let result = tokio::task::spawn_blocking(move || {
119
+
build_sketch(&db, &did_owned, collection_filter.as_deref())
120
+
})
121
+
.await;
122
+
123
+
match result {
124
+
Ok(Ok((riblt_bytes, record_count, rev))) => {
125
+
// Cache full-repo sketches
126
+
if use_cache {
127
+
let cache_key = riblt_cache_key(did);
128
+
let cache_value = encode_cache_value(&rev, record_count, &riblt_bytes);
129
+
let _ = state.db.meta.insert(cache_key.as_bytes(), &cache_value);
130
+
}
131
+
riblt_response(&riblt_bytes, &rev, record_count)
132
+
}
133
+
Ok(Err(e)) => error_json(
134
+
StatusCode::INTERNAL_SERVER_ERROR,
135
+
"InternalError",
136
+
&format!("failed to build sketch: {e}"),
137
+
),
138
+
Err(e) => error_json(
139
+
StatusCode::INTERNAL_SERVER_ERROR,
140
+
"InternalError",
141
+
&format!("task failed: {e}"),
142
+
),
143
+
}
144
+
}
145
+
146
+
/// Build an RIBLT sketch from the records keyspace for a DID.
147
+
///
148
+
/// Scans all records, groups by (collection, rkey), takes the latest
149
+
/// non-tombstone revision of each, and produces coded symbols.
150
+
fn build_sketch(
151
+
db: &FjallDb,
152
+
did: &str,
153
+
collection_filter: Option<&str>,
154
+
) -> anyhow::Result<(Vec<u8>, u64, String)> {
155
+
let prefix = if let Some(coll) = collection_filter {
156
+
keys::encode_collection_prefix(did, coll)
157
+
} else {
158
+
keys::encode_did_prefix(did)
159
+
};
160
+
161
+
let mut symbols: Vec<ByteSymbol> = Vec::new();
162
+
163
+
// Track the current (collection, rkey) group and its latest CID.
164
+
let mut current_collection: Option<String> = None;
165
+
let mut current_rkey: Option<String> = None;
166
+
let mut current_cid: Option<Vec<u8>> = None;
167
+
let mut is_tombstone = false;
168
+
169
+
for guard in db.records.prefix(&prefix) {
170
+
let Ok((key, value)) = guard.into_inner() else {
171
+
continue;
172
+
};
173
+
let Ok((_, collection, rkey, _rev)) = keys::decode_record_key(&key) else {
174
+
continue;
175
+
};
176
+
177
+
let same_record = current_collection.as_deref() == Some(collection)
178
+
&& current_rkey.as_deref() == Some(rkey);
179
+
180
+
if same_record {
181
+
// Same record, newer revision — overwrite
182
+
let decompressed = db
183
+
.decompress_event(&value)
184
+
.unwrap_or_else(|_| value.to_vec());
185
+
if RecordValue::is_tombstone(&decompressed) {
186
+
is_tombstone = true;
187
+
current_cid = None;
188
+
} else if let Ok(rv) = RecordValue::decode(&decompressed) {
189
+
is_tombstone = false;
190
+
current_cid = Some(rv.cid);
191
+
}
192
+
continue;
193
+
}
194
+
195
+
// New record — emit previous if live
196
+
if let (Some(coll), Some(rk)) = (¤t_collection, ¤t_rkey) {
197
+
if !is_tombstone {
198
+
if let Some(cid) = ¤t_cid {
199
+
symbols.push(make_record_symbol(coll, rk, cid));
200
+
}
201
+
}
202
+
}
203
+
204
+
// Start new group
205
+
let decompressed = db
206
+
.decompress_event(&value)
207
+
.unwrap_or_else(|_| value.to_vec());
208
+
if RecordValue::is_tombstone(&decompressed) {
209
+
is_tombstone = true;
210
+
current_cid = None;
211
+
} else if let Ok(rv) = RecordValue::decode(&decompressed) {
212
+
is_tombstone = false;
213
+
current_cid = Some(rv.cid);
214
+
} else {
215
+
is_tombstone = true;
216
+
current_cid = None;
217
+
}
218
+
current_collection = Some(collection.to_string());
219
+
current_rkey = Some(rkey.to_string());
220
+
}
221
+
222
+
// Emit final record
223
+
if let (Some(coll), Some(rk)) = (¤t_collection, ¤t_rkey) {
224
+
if !is_tombstone {
225
+
if let Some(cid) = ¤t_cid {
226
+
symbols.push(make_record_symbol(coll, rk, cid));
227
+
}
228
+
}
229
+
}
230
+
231
+
let record_count = symbols.len() as u64;
232
+
let sketch_size = sketch_size_for_records(symbols.len());
233
+
234
+
let mut encoder = Encoder::<ByteSymbol>::new();
235
+
for sym in &symbols {
236
+
encoder.add_symbol(sym.clone());
237
+
}
238
+
239
+
let coded_symbols = (0..sketch_size)
240
+
.map(|_| encoder.produce_next_coded_symbol())
241
+
.collect();
242
+
243
+
let file = RibltFile {
244
+
num_records: record_count,
245
+
coded_symbols,
246
+
};
247
+
248
+
let mut buf = Vec::new();
249
+
file.write_to(&mut buf)?;
250
+
251
+
let rev = db.get_repo_state(did)?.map(|rs| rs.rev).unwrap_or_default();
252
+
253
+
Ok((buf, record_count, rev))
254
+
}
255
+
256
+
/// Invalidate the cached RIBLT sketch for a DID.
257
+
///
258
+
/// Called when records change so the next request rebuilds the sketch.
259
+
pub fn invalidate_sketch_cache(db: &FjallDb, did: &str) {
260
+
let cache_key = riblt_cache_key(did);
261
+
let _ = db.meta.remove(cache_key.as_bytes());
262
+
}
263
+
264
+
// -- Cache encoding --
265
+
//
266
+
// Format: [2B rev_len LE][rev bytes][8B record_count LE][RibltFile bytes]
267
+
268
+
fn riblt_cache_key(did: &str) -> String {
269
+
format!("riblt\x00{did}")
270
+
}
271
+
272
+
fn encode_cache_value(rev: &str, record_count: u64, riblt_bytes: &[u8]) -> Vec<u8> {
273
+
let rev_bytes = rev.as_bytes();
274
+
let mut buf = Vec::with_capacity(2 + rev_bytes.len() + 8 + riblt_bytes.len());
275
+
buf.extend_from_slice(&(rev_bytes.len() as u16).to_le_bytes());
276
+
buf.extend_from_slice(rev_bytes);
277
+
buf.extend_from_slice(&record_count.to_le_bytes());
278
+
buf.extend_from_slice(riblt_bytes);
279
+
buf
280
+
}
281
+
282
+
fn decode_cache_value(bytes: &[u8]) -> Option<(&str, u64, &[u8])> {
283
+
if bytes.len() < 2 {
284
+
return None;
285
+
}
286
+
let rev_len = u16::from_le_bytes([bytes[0], bytes[1]]) as usize;
287
+
if bytes.len() < 2 + rev_len + 8 {
288
+
return None;
289
+
}
290
+
let rev = std::str::from_utf8(&bytes[2..2 + rev_len]).ok()?;
291
+
let record_count = u64::from_le_bytes(bytes[2 + rev_len..2 + rev_len + 8].try_into().ok()?);
292
+
let riblt_bytes = &bytes[2 + rev_len + 8..];
293
+
Some((rev, record_count, riblt_bytes))
294
+
}
295
+
296
+
fn error_json(status: StatusCode, error: &str, message: &str) -> Response {
297
+
(
298
+
status,
299
+
axum::Json(json!({ "error": error, "message": message })),
300
+
)
301
+
.into_response()
302
+
}
303
+
304
+
fn riblt_response(data: &[u8], rev: &str, record_count: u64) -> Response {
305
+
Response::builder()
306
+
.status(StatusCode::OK)
307
+
.header(header::CONTENT_TYPE, "application/x-riblt")
308
+
.header(header::ETAG, format!("\"{rev}\""))
309
+
.header("X-Riblt-Records", record_count.to_string())
310
+
.header("X-Riblt-Rev", rev)
311
+
.body(axum::body::Body::from(data.to_vec()))
312
+
.unwrap_or_else(|_| StatusCode::INTERNAL_SERVER_ERROR.into_response())
313
+
}
314
+
315
+
#[cfg(test)]
316
+
mod tests {
317
+
use super::*;
318
+
319
+
#[test]
320
+
fn test_sketch_size_zero_records() {
321
+
assert_eq!(sketch_size_for_records(0), 25_000);
322
+
}
323
+
324
+
#[test]
325
+
fn test_sketch_size_within_first_tier() {
326
+
assert_eq!(sketch_size_for_records(1), 25_000);
327
+
assert_eq!(sketch_size_for_records(25_000), 25_000);
328
+
}
329
+
330
+
#[test]
331
+
fn test_sketch_size_bump_at_threshold() {
332
+
// 50% of next increment = 12,500
333
+
// Bump at 25,000 + 12,500 = 37,500
334
+
assert_eq!(sketch_size_for_records(37_499), 25_000);
335
+
assert_eq!(sketch_size_for_records(37_500), 50_000);
336
+
}
337
+
338
+
#[test]
339
+
fn test_sketch_size_second_tier() {
340
+
assert_eq!(sketch_size_for_records(50_000), 50_000);
341
+
assert_eq!(sketch_size_for_records(62_499), 50_000);
342
+
assert_eq!(sketch_size_for_records(62_500), 75_000);
343
+
}
344
+
345
+
#[test]
346
+
fn test_sketch_size_third_tier() {
347
+
assert_eq!(sketch_size_for_records(75_000), 75_000);
348
+
assert_eq!(sketch_size_for_records(87_499), 75_000);
349
+
assert_eq!(sketch_size_for_records(87_500), 100_000);
350
+
}
351
+
352
+
#[test]
353
+
fn test_make_record_symbol() {
354
+
let sym = make_record_symbol("app.bsky.feed.post", "3jzfcij", &[0x01, 0x71, 0x12]);
355
+
assert_eq!(sym.0, b"app.bsky.feed.post\x003jzfcij\x00\x01\x71\x12");
356
+
}
357
+
358
+
#[test]
359
+
fn test_cache_roundtrip() {
360
+
let rev = "3jzfcijpj2z2b";
361
+
let record_count = 42;
362
+
let riblt_bytes = b"RIBLT1\x01test-data";
363
+
364
+
let encoded = encode_cache_value(rev, record_count, riblt_bytes);
365
+
let (decoded_rev, decoded_count, decoded_data) = decode_cache_value(&encoded).unwrap();
366
+
367
+
assert_eq!(decoded_rev, rev);
368
+
assert_eq!(decoded_count, record_count);
369
+
assert_eq!(decoded_data, riblt_bytes);
370
+
}
371
+
}
+269
src/server/tokio_metrics.rs
+269
src/server/tokio_metrics.rs
···
1
+
//! Tokio task metrics collection using stable TaskMonitor API.
2
+
//!
3
+
//! This module bridges `tokio-metrics` TaskMonitor to `prometheus-client` Registry,
4
+
//! providing metrics about instrumented async task performance without requiring
5
+
//! unstable Tokio features.
6
+
7
+
use prometheus_client::metrics::gauge::Gauge;
8
+
use prometheus_client::registry::Registry;
9
+
use std::sync::Arc;
10
+
use std::sync::atomic::AtomicU64;
11
+
use std::time::Duration;
12
+
use tokio_metrics::TaskMonitor;
13
+
use tokio_util::sync::CancellationToken;
14
+
15
+
/// Tokio task metrics collector using stable TaskMonitor API.
16
+
///
17
+
/// Provides a `TaskMonitor` for instrumenting async tasks and exposes
18
+
/// aggregate metrics to Prometheus.
19
+
///
20
+
/// All cumulative metrics are exposed as gauges that contain the running total.
21
+
/// Use Prometheus `rate()` or `increase()` functions to compute rates.
22
+
pub struct TokioMetrics {
23
+
/// The task monitor for instrumenting async tasks
24
+
monitor: TaskMonitor,
25
+
26
+
// Task lifecycle gauges (cumulative totals)
27
+
instrumented_total: Gauge,
28
+
dropped_total: Gauge,
29
+
first_poll_total: Gauge,
30
+
31
+
// Duration gauges (cumulative totals in seconds)
32
+
first_poll_delay_seconds_total: Gauge<f64, AtomicU64>,
33
+
idle_duration_seconds_total: Gauge<f64, AtomicU64>,
34
+
scheduled_duration_seconds_total: Gauge<f64, AtomicU64>,
35
+
poll_duration_seconds_total: Gauge<f64, AtomicU64>,
36
+
slow_poll_duration_seconds_total: Gauge<f64, AtomicU64>,
37
+
long_delay_duration_seconds_total: Gauge<f64, AtomicU64>,
38
+
39
+
// Event gauges (cumulative totals)
40
+
idled_total: Gauge,
41
+
scheduled_total: Gauge,
42
+
poll_total: Gauge,
43
+
slow_poll_total: Gauge,
44
+
long_delay_total: Gauge,
45
+
46
+
// Instantaneous gauges (computed averages)
47
+
mean_poll_duration_seconds: Gauge<f64, AtomicU64>,
48
+
mean_scheduled_duration_seconds: Gauge<f64, AtomicU64>,
49
+
}
50
+
51
+
impl TokioMetrics {
52
+
/// Create and register tokio task metrics with the registry.
53
+
///
54
+
/// Uses default thresholds: 50μs for slow polls and long delays.
55
+
pub fn register(registry: &mut Registry) -> Self {
56
+
let monitor = TaskMonitor::new();
57
+
58
+
// Create a sub-registry with "tokio_task" prefix
59
+
let sub_registry = registry.sub_registry_with_prefix("tokio_task");
60
+
61
+
// Task lifecycle gauges
62
+
let instrumented_total = Gauge::default();
63
+
sub_registry.register(
64
+
"instrumented_total",
65
+
"Total number of tasks instrumented",
66
+
instrumented_total.clone(),
67
+
);
68
+
69
+
let dropped_total = Gauge::default();
70
+
sub_registry.register(
71
+
"dropped_total",
72
+
"Total number of tasks dropped",
73
+
dropped_total.clone(),
74
+
);
75
+
76
+
let first_poll_total = Gauge::default();
77
+
sub_registry.register(
78
+
"first_poll_total",
79
+
"Total number of tasks polled for the first time",
80
+
first_poll_total.clone(),
81
+
);
82
+
83
+
// Duration gauges
84
+
let first_poll_delay_seconds_total = Gauge::<f64, AtomicU64>::default();
85
+
sub_registry.register(
86
+
"first_poll_delay_seconds_total",
87
+
"Total time from task creation to first poll in seconds",
88
+
first_poll_delay_seconds_total.clone(),
89
+
);
90
+
91
+
let idle_duration_seconds_total = Gauge::<f64, AtomicU64>::default();
92
+
sub_registry.register(
93
+
"idle_duration_seconds_total",
94
+
"Total time tasks spent idle waiting for events in seconds",
95
+
idle_duration_seconds_total.clone(),
96
+
);
97
+
98
+
let scheduled_duration_seconds_total = Gauge::<f64, AtomicU64>::default();
99
+
sub_registry.register(
100
+
"scheduled_duration_seconds_total",
101
+
"Total time tasks spent waiting to be executed in seconds",
102
+
scheduled_duration_seconds_total.clone(),
103
+
);
104
+
105
+
let poll_duration_seconds_total = Gauge::<f64, AtomicU64>::default();
106
+
sub_registry.register(
107
+
"poll_duration_seconds_total",
108
+
"Total time spent polling tasks in seconds",
109
+
poll_duration_seconds_total.clone(),
110
+
);
111
+
112
+
let slow_poll_duration_seconds_total = Gauge::<f64, AtomicU64>::default();
113
+
sub_registry.register(
114
+
"slow_poll_duration_seconds_total",
115
+
"Total time spent in slow polls in seconds",
116
+
slow_poll_duration_seconds_total.clone(),
117
+
);
118
+
119
+
let long_delay_duration_seconds_total = Gauge::<f64, AtomicU64>::default();
120
+
sub_registry.register(
121
+
"long_delay_duration_seconds_total",
122
+
"Total time spent in long scheduling delays in seconds",
123
+
long_delay_duration_seconds_total.clone(),
124
+
);
125
+
126
+
// Event gauges
127
+
let idled_total = Gauge::default();
128
+
sub_registry.register(
129
+
"idled_total",
130
+
"Total number of times tasks idled waiting for events",
131
+
idled_total.clone(),
132
+
);
133
+
134
+
let scheduled_total = Gauge::default();
135
+
sub_registry.register(
136
+
"scheduled_total",
137
+
"Total number of times tasks were scheduled for execution",
138
+
scheduled_total.clone(),
139
+
);
140
+
141
+
let poll_total = Gauge::default();
142
+
sub_registry.register(
143
+
"poll_total",
144
+
"Total number of task poll operations",
145
+
poll_total.clone(),
146
+
);
147
+
148
+
let slow_poll_total = Gauge::default();
149
+
sub_registry.register(
150
+
"slow_poll_total",
151
+
"Total number of polls exceeding slow poll threshold",
152
+
slow_poll_total.clone(),
153
+
);
154
+
155
+
let long_delay_total = Gauge::default();
156
+
sub_registry.register(
157
+
"long_delay_total",
158
+
"Total number of long scheduling delays",
159
+
long_delay_total.clone(),
160
+
);
161
+
162
+
// Instantaneous gauges for computed averages
163
+
let mean_poll_duration_seconds = Gauge::<f64, AtomicU64>::default();
164
+
sub_registry.register(
165
+
"mean_poll_duration_seconds",
166
+
"Mean duration of task polls in seconds",
167
+
mean_poll_duration_seconds.clone(),
168
+
);
169
+
170
+
let mean_scheduled_duration_seconds = Gauge::<f64, AtomicU64>::default();
171
+
sub_registry.register(
172
+
"mean_scheduled_duration_seconds",
173
+
"Mean time tasks waited to be executed in seconds",
174
+
mean_scheduled_duration_seconds.clone(),
175
+
);
176
+
177
+
Self {
178
+
monitor,
179
+
instrumented_total,
180
+
dropped_total,
181
+
first_poll_total,
182
+
first_poll_delay_seconds_total,
183
+
idle_duration_seconds_total,
184
+
scheduled_duration_seconds_total,
185
+
poll_duration_seconds_total,
186
+
slow_poll_duration_seconds_total,
187
+
long_delay_duration_seconds_total,
188
+
idled_total,
189
+
scheduled_total,
190
+
poll_total,
191
+
slow_poll_total,
192
+
long_delay_total,
193
+
mean_poll_duration_seconds,
194
+
mean_scheduled_duration_seconds,
195
+
}
196
+
}
197
+
198
+
/// Get the TaskMonitor for instrumenting async tasks.
199
+
///
200
+
/// Use `monitor.instrument(future)` to track a task's metrics.
201
+
pub fn monitor(&self) -> &TaskMonitor {
202
+
&self.monitor
203
+
}
204
+
205
+
/// Update Prometheus metrics from cumulative TaskMetrics.
206
+
fn update(&self, metrics: &tokio_metrics::TaskMetrics) {
207
+
// Update lifecycle gauges
208
+
self.instrumented_total
209
+
.set(metrics.instrumented_count as i64);
210
+
self.dropped_total.set(metrics.dropped_count as i64);
211
+
self.first_poll_total.set(metrics.first_poll_count as i64);
212
+
213
+
// Update duration gauges (convert to seconds)
214
+
self.first_poll_delay_seconds_total
215
+
.set(metrics.total_first_poll_delay.as_secs_f64());
216
+
self.idle_duration_seconds_total
217
+
.set(metrics.total_idle_duration.as_secs_f64());
218
+
self.scheduled_duration_seconds_total
219
+
.set(metrics.total_scheduled_duration.as_secs_f64());
220
+
self.poll_duration_seconds_total
221
+
.set(metrics.total_poll_duration.as_secs_f64());
222
+
self.slow_poll_duration_seconds_total
223
+
.set(metrics.total_slow_poll_duration.as_secs_f64());
224
+
self.long_delay_duration_seconds_total
225
+
.set(metrics.total_long_delay_duration.as_secs_f64());
226
+
227
+
// Update event gauges
228
+
self.idled_total.set(metrics.total_idled_count as i64);
229
+
self.scheduled_total
230
+
.set(metrics.total_scheduled_count as i64);
231
+
self.poll_total.set(metrics.total_poll_count as i64);
232
+
self.slow_poll_total
233
+
.set(metrics.total_slow_poll_count as i64);
234
+
self.long_delay_total
235
+
.set(metrics.total_long_delay_count as i64);
236
+
237
+
// Update computed averages
238
+
self.mean_poll_duration_seconds
239
+
.set(metrics.mean_poll_duration().as_secs_f64());
240
+
self.mean_scheduled_duration_seconds
241
+
.set(metrics.mean_scheduled_duration().as_secs_f64());
242
+
}
243
+
244
+
/// Run the metrics collector, periodically updating Prometheus metrics.
245
+
///
246
+
/// This should be spawned as a background task.
247
+
pub async fn run_collector(self: Arc<Self>, interval: Duration, cancel: CancellationToken) {
248
+
let mut ticker = tokio::time::interval(interval);
249
+
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
250
+
251
+
loop {
252
+
tokio::select! {
253
+
_ = cancel.cancelled() => {
254
+
tracing::info!("tokio metrics collector shutting down");
255
+
break;
256
+
}
257
+
_ = ticker.tick() => {
258
+
let metrics = self.monitor.cumulative();
259
+
self.update(&metrics);
260
+
tracing::trace!(
261
+
instrumented = metrics.instrumented_count,
262
+
polls = metrics.total_poll_count,
263
+
"updated tokio task metrics"
264
+
);
265
+
}
266
+
}
267
+
}
268
+
}
269
+
}
+659
src/server/websocket.rs
+659
src/server/websocket.rs
···
1
+
//! WebSocket fan-out handler for dev.ngerakines.ramjet.stream.subscribe.
2
+
//!
3
+
//! Supports cursor-based replay from the events keyspace, then switches
4
+
//! to live streaming. A per-connection drain task reads from the shared
5
+
//! broadcast channels into a bounded mpsc buffer, decoupling broadcast
6
+
//! drain speed from WebSocket send speed.
7
+
//!
8
+
//! Consumers can optionally join a pre-defined consumer group by passing
9
+
//! `group` and `partition` query parameters. In that mode, only events
10
+
//! whose DID hashes to the requested partition are delivered.
11
+
12
+
use std::sync::atomic::Ordering;
13
+
14
+
use axum::extract::ws::{Message, WebSocket, WebSocketUpgrade};
15
+
use axum::extract::{Query, State};
16
+
use axum::response::IntoResponse;
17
+
use serde::Deserialize;
18
+
use tokio::sync::{broadcast, mpsc};
19
+
20
+
use atproto_dasl::Ipld;
21
+
use bytes::Bytes;
22
+
23
+
use crate::pipeline::fanout::did_hash;
24
+
use crate::server::AppState;
25
+
use crate::server::metrics::{FanoutPriority, FanoutPriorityLabels};
26
+
use crate::storage::encoding::{
27
+
CompactEvent, CompactOp, decode_compact_event, extract_did_from_compact, ipld_map,
28
+
to_dag_cbor_bytes,
29
+
};
30
+
use crate::storage::keys;
31
+
use crate::types::{OpType, SharedFanOutEvent};
32
+
33
+
/// Per-connection buffer capacity. Large enough to absorb bursts from
34
+
/// the writer without dropping events when the WebSocket send is slow.
35
+
const CONNECTION_BUFFER_SIZE: usize = 16384;
36
+
37
+
/// Number of consecutive dropped events before evicting the connection.
38
+
/// At ~900 events/s this is roughly 18s of sustained full-buffer lag.
39
+
const EVICTION_THRESHOLD: u64 = 16384;
40
+
41
+
/// Maximum number of events to replay from cursor before cutting off.
42
+
/// Prevents an old cursor from blocking a tokio worker for minutes.
43
+
const MAX_REPLAY_EVENTS: usize = 100_000;
44
+
45
+
/// Yield to the tokio scheduler every this many events during replay.
46
+
const REPLAY_YIELD_INTERVAL: usize = 256;
47
+
48
+
#[derive(Deserialize)]
49
+
pub struct SubscribeParams {
50
+
pub cursor: Option<u64>,
51
+
/// Consumer group name. When set with `partition`, the client receives
52
+
/// only events whose DID maps to the requested partition.
53
+
pub group: Option<String>,
54
+
/// Partition index within the consumer group (0-based).
55
+
pub partition: Option<u16>,
56
+
}
57
+
58
+
/// Resolved subscription mode after validating query parameters.
59
+
enum SubscriptionMode {
60
+
/// Broadcast mode: receive all events via high/low priority channels.
61
+
Broadcast,
62
+
/// Consumer group mode: receive only events for one partition.
63
+
Partitioned {
64
+
group: String,
65
+
partition: u16,
66
+
partition_count: u16,
67
+
},
68
+
}
69
+
70
+
/// Handle `GET /xrpc/dev.ngerakines.ramjet.stream.subscribe`.
71
+
pub async fn handle_subscribe(
72
+
State(state): State<AppState>,
73
+
Query(params): Query<SubscribeParams>,
74
+
ws: WebSocketUpgrade,
75
+
) -> impl IntoResponse {
76
+
ws.on_upgrade(move |socket| handle_connection(socket, state, params))
77
+
}
78
+
79
+
async fn handle_connection(mut socket: WebSocket, state: AppState, params: SubscribeParams) {
80
+
state
81
+
.metrics
82
+
.fanout_connections_active
83
+
.inner()
84
+
.fetch_add(1, Ordering::Relaxed);
85
+
state.metrics.fanout_connections_total.inc();
86
+
87
+
let result = run_consumer(&mut socket, &state, params).await;
88
+
if let Err(e) = result {
89
+
tracing::debug!(error = %e, "websocket consumer ended");
90
+
}
91
+
92
+
state
93
+
.metrics
94
+
.fanout_connections_active
95
+
.inner()
96
+
.fetch_sub(1, Ordering::Relaxed);
97
+
}
98
+
99
+
async fn run_consumer(
100
+
socket: &mut WebSocket,
101
+
state: &AppState,
102
+
params: SubscribeParams,
103
+
) -> anyhow::Result<()> {
104
+
// Determine subscription mode from query parameters.
105
+
let mode = match (params.group, params.partition) {
106
+
(Some(group), Some(partition)) => {
107
+
let Some(partition_count) = state.fanout.partition_count(&group) else {
108
+
let err_msg = to_dag_cbor_bytes(&ipld_map(vec![
109
+
("error", Ipld::String("InvalidGroup".to_string())),
110
+
(
111
+
"message",
112
+
Ipld::String(format!("consumer group '{}' does not exist", group)),
113
+
),
114
+
]));
115
+
socket.send(Message::Binary(err_msg.into())).await?;
116
+
return Ok(());
117
+
};
118
+
if partition >= partition_count {
119
+
let err_msg = to_dag_cbor_bytes(&ipld_map(vec![
120
+
("error", Ipld::String("InvalidPartition".to_string())),
121
+
(
122
+
"message",
123
+
Ipld::String(format!(
124
+
"partition {} out of range for group '{}' (0..{})",
125
+
partition, group, partition_count
126
+
)),
127
+
),
128
+
]));
129
+
socket.send(Message::Binary(err_msg.into())).await?;
130
+
return Ok(());
131
+
}
132
+
SubscriptionMode::Partitioned {
133
+
group,
134
+
partition,
135
+
partition_count,
136
+
}
137
+
}
138
+
(Some(group), None) => {
139
+
let err_msg = to_dag_cbor_bytes(&ipld_map(vec![
140
+
("error", Ipld::String("MissingPartition".to_string())),
141
+
(
142
+
"message",
143
+
Ipld::String(format!(
144
+
"group '{}' specified without partition parameter",
145
+
group
146
+
)),
147
+
),
148
+
]));
149
+
socket.send(Message::Binary(err_msg.into())).await?;
150
+
return Ok(());
151
+
}
152
+
(None, Some(_)) => {
153
+
let err_msg = to_dag_cbor_bytes(&ipld_map(vec![
154
+
("error", Ipld::String("MissingGroup".to_string())),
155
+
(
156
+
"message",
157
+
Ipld::String("partition specified without group parameter".to_string()),
158
+
),
159
+
]));
160
+
socket.send(Message::Binary(err_msg.into())).await?;
161
+
return Ok(());
162
+
}
163
+
(None, None) => SubscriptionMode::Broadcast,
164
+
};
165
+
166
+
// Partition filter for replay (None means accept all).
167
+
let partition_filter: Option<(u16, u16)> = match &mode {
168
+
SubscriptionMode::Partitioned {
169
+
partition,
170
+
partition_count,
171
+
..
172
+
} => Some((*partition, *partition_count)),
173
+
SubscriptionMode::Broadcast => None,
174
+
};
175
+
176
+
// Subscribe BEFORE replay so we don't miss events broadcast during the
177
+
// gap between the end of the range scan and when we start consuming live.
178
+
let live_rx: LiveReceiver = match &mode {
179
+
SubscriptionMode::Broadcast => LiveReceiver::Broadcast {
180
+
high_rx: state.fanout.subscribe_high(),
181
+
low_rx: state.fanout.subscribe_low(),
182
+
},
183
+
SubscriptionMode::Partitioned {
184
+
group, partition, ..
185
+
} => LiveReceiver::Partitioned {
186
+
rx: state
187
+
.fanout
188
+
.subscribe_partition(group, *partition)
189
+
.expect("partition validated above"),
190
+
},
191
+
};
192
+
193
+
// Replay from cursor if provided, tracking the last replayed sequence
194
+
// so the drain task can skip duplicates from the overlap window.
195
+
let mut last_replayed_seq: u64 = 0;
196
+
197
+
if let Some(cursor_seq) = params.cursor {
198
+
let oldest = state.db.oldest_event_sequence()?;
199
+
if cursor_seq < oldest && oldest > 0 {
200
+
let err_msg = to_dag_cbor_bytes(&ipld_map(vec![
201
+
("error", Ipld::String("FutureCursor".to_string())),
202
+
(
203
+
"message",
204
+
Ipld::String(format!(
205
+
"cursor {} is before oldest available event {}",
206
+
cursor_seq, oldest
207
+
)),
208
+
),
209
+
]));
210
+
socket.send(Message::Binary(err_msg.into())).await?;
211
+
return Ok(());
212
+
}
213
+
214
+
last_replayed_seq = cursor_seq;
215
+
let start_key = keys::encode_event_key(cursor_seq + 1);
216
+
let mut replay_count: usize = 0;
217
+
218
+
for guard in state.db.events.range(start_key..) {
219
+
let Ok((key, value)) = guard.into_inner() else {
220
+
continue;
221
+
};
222
+
223
+
// Track the sequence of the last event we replayed
224
+
if let Ok(seq) = keys::decode_event_key(&key) {
225
+
last_replayed_seq = seq;
226
+
}
227
+
228
+
// Decompress if zstd-compressed, then decode compact binary event
229
+
// and convert to CBOR for the client.
230
+
// V1 grouped commits expand to multiple messages (one per op).
231
+
let raw: &[u8] = &value;
232
+
let decompressed = match state.db.decompress_event(raw) {
233
+
Ok(d) => d,
234
+
Err(e) => {
235
+
tracing::warn!(error = %e, "failed to decompress event during replay");
236
+
continue;
237
+
}
238
+
};
239
+
240
+
// For partitioned consumers, filter by DID hash before decoding.
241
+
if let Some((partition, partition_count)) = partition_filter {
242
+
if let Some(did) = extract_did_from_compact(&decompressed) {
243
+
let h = did_hash(did);
244
+
if (h % partition_count as u64) as u16 != partition {
245
+
continue;
246
+
}
247
+
}
248
+
}
249
+
250
+
let cbor_messages = compact_event_to_cbor(&decompressed);
251
+
for cbor_bytes in cbor_messages {
252
+
if socket.send(Message::Binary(cbor_bytes)).await.is_err() {
253
+
return Ok(());
254
+
}
255
+
256
+
replay_count += 1;
257
+
258
+
if replay_count % REPLAY_YIELD_INTERVAL == 0 {
259
+
tokio::task::yield_now().await;
260
+
}
261
+
262
+
if replay_count >= MAX_REPLAY_EVENTS {
263
+
let err_msg = to_dag_cbor_bytes(&ipld_map(vec![
264
+
("error", Ipld::String("ReplayLimitExceeded".to_string())),
265
+
(
266
+
"message",
267
+
Ipld::String(format!(
268
+
"cursor replay exceeded maximum of {} events; reconnect with a newer cursor",
269
+
MAX_REPLAY_EVENTS
270
+
)),
271
+
),
272
+
]));
273
+
socket.send(Message::Binary(err_msg.into())).await?;
274
+
return Ok(());
275
+
}
276
+
}
277
+
}
278
+
}
279
+
280
+
// Per-connection buffer: drain task writes here, send loop reads from here.
281
+
let (buf_tx, mut buf_rx) = mpsc::channel::<SharedFanOutEvent>(CONNECTION_BUFFER_SIZE);
282
+
283
+
// Spawn drain task — reads from broadcast channels into the per-connection
284
+
// mpsc buffer. This never blocks on IO, so it keeps up with the broadcast
285
+
// even when the WebSocket send is slow. Returns true if evicted due to
286
+
// sustained lag. Events at or below last_replayed_seq are skipped to
287
+
// deduplicate the overlap window between replay and live subscription.
288
+
let drain_handle = tokio::spawn({
289
+
let metrics = state.metrics.clone();
290
+
async move { drain_live(live_rx, &buf_tx, &metrics, last_replayed_seq).await }
291
+
});
292
+
293
+
// Send loop — reads from per-connection buffer and writes to WebSocket.
294
+
// Batch-drains up to SEND_BATCH events per iteration to amortize
295
+
// the cost of tokio::select! and reduce per-event syscall overhead.
296
+
const SEND_BATCH: usize = 64;
297
+
loop {
298
+
tokio::select! {
299
+
biased;
300
+
301
+
event = buf_rx.recv() => {
302
+
let Some(event) = event else {
303
+
// Drain task exited — either broadcast closed or connection evicted
304
+
break;
305
+
};
306
+
if socket.send(Message::Binary(event.payload.clone())).await.is_err() {
307
+
break;
308
+
}
309
+
state.metrics.fanout_events_sent_total.inc();
310
+
311
+
// Drain additional buffered events without re-entering select
312
+
let mut broke = false;
313
+
for _ in 1..SEND_BATCH {
314
+
match buf_rx.try_recv() {
315
+
Ok(event) => {
316
+
if socket.send(Message::Binary(event.payload.clone())).await.is_err() {
317
+
broke = true;
318
+
break;
319
+
}
320
+
state.metrics.fanout_events_sent_total.inc();
321
+
}
322
+
Err(_) => break,
323
+
}
324
+
}
325
+
if broke {
326
+
break;
327
+
}
328
+
}
329
+
330
+
msg = socket.recv() => {
331
+
match msg {
332
+
Some(Ok(_)) => {}
333
+
_ => break,
334
+
}
335
+
}
336
+
}
337
+
}
338
+
339
+
// Check if drain task evicted this connection
340
+
let evicted = match drain_handle.await {
341
+
Ok(was_evicted) => was_evicted,
342
+
Err(_) => false, // task was cancelled
343
+
};
344
+
345
+
if evicted {
346
+
state.metrics.fanout_connections_evicted.inc();
347
+
tracing::warn!("evicted WebSocket consumer due to sustained lag");
348
+
}
349
+
350
+
Ok(())
351
+
}
352
+
353
+
/// Encapsulates the live event source for a WebSocket consumer.
354
+
enum LiveReceiver {
355
+
/// Broadcast mode: receives from both high and low priority channels.
356
+
Broadcast {
357
+
high_rx: broadcast::Receiver<SharedFanOutEvent>,
358
+
low_rx: broadcast::Receiver<SharedFanOutEvent>,
359
+
},
360
+
/// Consumer group mode: receives from a single partition channel.
361
+
Partitioned {
362
+
rx: broadcast::Receiver<SharedFanOutEvent>,
363
+
},
364
+
}
365
+
366
+
/// Drain live events into the per-connection buffer.
367
+
/// Returns `true` if the connection was evicted due to sustained lag.
368
+
async fn drain_live(
369
+
receiver: LiveReceiver,
370
+
buf_tx: &mpsc::Sender<SharedFanOutEvent>,
371
+
metrics: &crate::server::metrics::Metrics,
372
+
last_replayed_seq: u64,
373
+
) -> bool {
374
+
match receiver {
375
+
LiveReceiver::Broadcast {
376
+
mut high_rx,
377
+
mut low_rx,
378
+
} => {
379
+
drain_broadcast(
380
+
&mut high_rx,
381
+
&mut low_rx,
382
+
buf_tx,
383
+
metrics,
384
+
last_replayed_seq,
385
+
)
386
+
.await
387
+
}
388
+
LiveReceiver::Partitioned { mut rx } => {
389
+
drain_partition(&mut rx, buf_tx, metrics, last_replayed_seq).await
390
+
}
391
+
}
392
+
}
393
+
394
+
/// Drains both broadcast channels into the per-connection mpsc buffer.
395
+
/// Uses biased select to prioritize high-priority events. When the
396
+
/// connection buffer is full, drops events and counts them as lagged.
397
+
/// Events with `seq <= last_replayed_seq` are silently skipped to
398
+
/// deduplicate the overlap window between cursor replay and live
399
+
/// subscription.
400
+
/// Returns `true` if the connection was evicted due to sustained lag.
401
+
async fn drain_broadcast(
402
+
high_rx: &mut broadcast::Receiver<SharedFanOutEvent>,
403
+
low_rx: &mut broadcast::Receiver<SharedFanOutEvent>,
404
+
buf_tx: &mpsc::Sender<SharedFanOutEvent>,
405
+
metrics: &crate::server::metrics::Metrics,
406
+
last_replayed_seq: u64,
407
+
) -> bool {
408
+
let mut consecutive_drops: u64 = 0;
409
+
410
+
loop {
411
+
tokio::select! {
412
+
biased;
413
+
414
+
msg = high_rx.recv() => {
415
+
match msg {
416
+
Ok(event) => {
417
+
// Skip events already sent during cursor replay
418
+
if event.seq <= last_replayed_seq {
419
+
continue;
420
+
}
421
+
if buf_tx.try_send(event).is_err() {
422
+
consecutive_drops += 1;
423
+
metrics
424
+
.fanout_lagged_total
425
+
.get_or_create(&FanoutPriorityLabels {
426
+
priority: FanoutPriority::High,
427
+
})
428
+
.inc();
429
+
if consecutive_drops >= EVICTION_THRESHOLD {
430
+
return true;
431
+
}
432
+
} else {
433
+
consecutive_drops = 0;
434
+
}
435
+
}
436
+
Err(broadcast::error::RecvError::Lagged(n)) => {
437
+
consecutive_drops += n;
438
+
tracing::warn!(n, "high-priority broadcast lagged");
439
+
metrics
440
+
.fanout_lagged_total
441
+
.get_or_create(&FanoutPriorityLabels {
442
+
priority: FanoutPriority::High,
443
+
})
444
+
.inc_by(n);
445
+
if consecutive_drops >= EVICTION_THRESHOLD {
446
+
return true;
447
+
}
448
+
}
449
+
Err(broadcast::error::RecvError::Closed) => return false,
450
+
}
451
+
}
452
+
453
+
msg = low_rx.recv() => {
454
+
match msg {
455
+
Ok(event) => {
456
+
// Skip events already sent during cursor replay
457
+
if event.seq <= last_replayed_seq {
458
+
continue;
459
+
}
460
+
if buf_tx.try_send(event).is_err() {
461
+
consecutive_drops += 1;
462
+
metrics
463
+
.fanout_lagged_total
464
+
.get_or_create(&FanoutPriorityLabels {
465
+
priority: FanoutPriority::Low,
466
+
})
467
+
.inc();
468
+
if consecutive_drops >= EVICTION_THRESHOLD {
469
+
return true;
470
+
}
471
+
} else {
472
+
consecutive_drops = 0;
473
+
}
474
+
}
475
+
Err(broadcast::error::RecvError::Lagged(n)) => {
476
+
consecutive_drops += n;
477
+
tracing::warn!(n, "low-priority broadcast lagged");
478
+
metrics
479
+
.fanout_lagged_total
480
+
.get_or_create(&FanoutPriorityLabels {
481
+
priority: FanoutPriority::Low,
482
+
})
483
+
.inc_by(n);
484
+
if consecutive_drops >= EVICTION_THRESHOLD {
485
+
return true;
486
+
}
487
+
}
488
+
Err(broadcast::error::RecvError::Closed) => return false,
489
+
}
490
+
}
491
+
}
492
+
}
493
+
}
494
+
495
+
/// Drains a single partition channel into the per-connection buffer.
496
+
/// Returns `true` if the connection was evicted due to sustained lag.
497
+
async fn drain_partition(
498
+
rx: &mut broadcast::Receiver<SharedFanOutEvent>,
499
+
buf_tx: &mpsc::Sender<SharedFanOutEvent>,
500
+
metrics: &crate::server::metrics::Metrics,
501
+
last_replayed_seq: u64,
502
+
) -> bool {
503
+
let mut consecutive_drops: u64 = 0;
504
+
505
+
loop {
506
+
match rx.recv().await {
507
+
Ok(event) => {
508
+
if event.seq <= last_replayed_seq {
509
+
continue;
510
+
}
511
+
if buf_tx.try_send(event).is_err() {
512
+
consecutive_drops += 1;
513
+
metrics
514
+
.fanout_lagged_total
515
+
.get_or_create(&FanoutPriorityLabels {
516
+
priority: FanoutPriority::High,
517
+
})
518
+
.inc();
519
+
if consecutive_drops >= EVICTION_THRESHOLD {
520
+
return true;
521
+
}
522
+
} else {
523
+
consecutive_drops = 0;
524
+
}
525
+
}
526
+
Err(broadcast::error::RecvError::Lagged(n)) => {
527
+
consecutive_drops += n;
528
+
tracing::warn!(n, "partition broadcast lagged");
529
+
metrics
530
+
.fanout_lagged_total
531
+
.get_or_create(&FanoutPriorityLabels {
532
+
priority: FanoutPriority::High,
533
+
})
534
+
.inc_by(n);
535
+
if consecutive_drops >= EVICTION_THRESHOLD {
536
+
return true;
537
+
}
538
+
}
539
+
Err(broadcast::error::RecvError::Closed) => return false,
540
+
}
541
+
}
542
+
}
543
+
544
+
/// Convert a compact binary event to one or more CBOR messages for WebSocket delivery.
545
+
///
546
+
/// V1 grouped commits expand to one message per op. V2 per-op events produce
547
+
/// a single message. Legacy raw JSON events are passed through as-is.
548
+
fn compact_event_to_cbor(bytes: &[u8]) -> Vec<Bytes> {
549
+
// Legacy detection: JSON events start with '{' (0x7B).
550
+
// Pass through as-is for backward compat.
551
+
if bytes.first() == Some(&b'{') {
552
+
return vec![Bytes::copy_from_slice(bytes)];
553
+
}
554
+
555
+
let event = match decode_compact_event(bytes) {
556
+
Ok(e) => e,
557
+
Err(_) => return vec![Bytes::copy_from_slice(bytes)],
558
+
};
559
+
560
+
match event {
561
+
// V1 grouped commit — expand to one message per op
562
+
CompactEvent::Commit { seq, did, rev, ops } => {
563
+
if ops.is_empty() {
564
+
return vec![];
565
+
}
566
+
ops.iter()
567
+
.enumerate()
568
+
.map(|(i, op)| {
569
+
// All ops from same commit share the base seq; offset by index
570
+
commit_op_to_cbor(seq + i as u64, 0, &did, &rev, op).into()
571
+
})
572
+
.collect()
573
+
}
574
+
// V2 per-op commit — single message
575
+
CompactEvent::CommitOp {
576
+
seq,
577
+
time_us,
578
+
did,
579
+
rev,
580
+
op,
581
+
} => {
582
+
vec![commit_op_to_cbor(seq, time_us, &did, &rev, &op).into()]
583
+
}
584
+
CompactEvent::Identity {
585
+
seq, did, handle, ..
586
+
} => {
587
+
let mut identity_fields: Vec<(&str, Ipld)> = Vec::new();
588
+
if let Some(h) = handle {
589
+
identity_fields.push(("handle", Ipld::String(h)));
590
+
}
591
+
let event = ipld_map(vec![
592
+
("seq", Ipld::Integer(seq.into())),
593
+
("did", Ipld::String(did)),
594
+
("kind", Ipld::String("identity".to_string())),
595
+
("identity", ipld_map(identity_fields)),
596
+
]);
597
+
vec![to_dag_cbor_bytes(&event).into()]
598
+
}
599
+
CompactEvent::Account {
600
+
seq,
601
+
did,
602
+
active,
603
+
status,
604
+
..
605
+
} => {
606
+
let mut account_fields: Vec<(&str, Ipld)> = vec![("active", Ipld::Bool(active))];
607
+
if let Some(s) = status {
608
+
account_fields.push(("status", Ipld::String(s)));
609
+
}
610
+
let event = ipld_map(vec![
611
+
("seq", Ipld::Integer(seq.into())),
612
+
("did", Ipld::String(did)),
613
+
("kind", Ipld::String("account".to_string())),
614
+
("account", ipld_map(account_fields)),
615
+
]);
616
+
vec![to_dag_cbor_bytes(&event).into()]
617
+
}
618
+
}
619
+
}
620
+
621
+
/// Build the DAG-CBOR bytes for a single commit operation.
622
+
fn commit_op_to_cbor(seq: u64, _time_us: u64, did: &str, rev: &str, op: &CompactOp) -> Vec<u8> {
623
+
let operation = match op.action {
624
+
OpType::Create => "create",
625
+
OpType::Update => "update",
626
+
OpType::Delete => "delete",
627
+
};
628
+
629
+
let mut commit_fields: Vec<(&str, Ipld)> = vec![
630
+
("rev", Ipld::String(rev.to_string())),
631
+
("operation", Ipld::String(operation.to_string())),
632
+
("collection", Ipld::String(op.collection.clone())),
633
+
("rkey", Ipld::String(op.rkey.clone())),
634
+
];
635
+
636
+
if op.action != OpType::Delete {
637
+
if let Some(cid_bytes) = &op.cid {
638
+
let cid_str = cid::Cid::read_bytes(cid_bytes.as_slice())
639
+
.map(|c| c.to_string())
640
+
.unwrap_or_default();
641
+
commit_fields.push(("cid", Ipld::String(cid_str)));
642
+
}
643
+
644
+
if !op.data.is_empty() {
645
+
if let Ok(ipld) = atproto_dasl::drisl::from_slice::<Ipld>(&op.data) {
646
+
commit_fields.push(("record", ipld));
647
+
}
648
+
}
649
+
}
650
+
651
+
let event = ipld_map(vec![
652
+
("seq", Ipld::Integer(seq.into())),
653
+
("did", Ipld::String(did.to_string())),
654
+
("kind", Ipld::String("commit".to_string())),
655
+
("commit", ipld_map(commit_fields)),
656
+
]);
657
+
658
+
to_dag_cbor_bytes(&event)
659
+
}
+618
src/server/xrpc.rs
+618
src/server/xrpc.rs
···
1
+
//! XRPC endpoint handlers for ATProtocol repo and identity queries.
2
+
3
+
use std::time::{Instant, SystemTime, UNIX_EPOCH};
4
+
5
+
use axum::Json;
6
+
use axum::extract::{Query, State};
7
+
use axum::http::{HeaderMap, StatusCode};
8
+
use serde::Deserialize;
9
+
use serde_json::{Value, json};
10
+
11
+
use crate::server::AppState;
12
+
use crate::server::metrics::{IdentityOutcome, IdentityOutcomeLabels, KeyspaceOpLabels, StorageOp};
13
+
use crate::storage::encoding::{decode_timestamped_doc, encode_timestamped_doc};
14
+
use crate::storage::keys;
15
+
use crate::types::RecordValue;
16
+
17
+
#[derive(Deserialize)]
18
+
pub struct GetRecordParams {
19
+
pub repo: Option<String>,
20
+
pub collection: Option<String>,
21
+
pub rkey: Option<String>,
22
+
}
23
+
24
+
#[derive(Deserialize)]
25
+
pub struct ListRecordsParams {
26
+
pub repo: Option<String>,
27
+
pub collection: Option<String>,
28
+
pub limit: Option<usize>,
29
+
pub cursor: Option<String>,
30
+
}
31
+
32
+
#[derive(Deserialize)]
33
+
pub struct DescribeRepoParams {
34
+
pub repo: Option<String>,
35
+
}
36
+
37
+
#[derive(Deserialize)]
38
+
pub struct ResolveIdentityParams {
39
+
pub identifier: Option<String>,
40
+
}
41
+
42
+
#[derive(Deserialize)]
43
+
pub struct ResolveHandleParams {
44
+
pub handle: Option<String>,
45
+
}
46
+
47
+
#[derive(Deserialize)]
48
+
pub struct ResolveDidParams {
49
+
pub did: Option<String>,
50
+
}
51
+
52
+
fn error_response(status: StatusCode, error: &str, message: &str) -> (StatusCode, Json<Value>) {
53
+
(status, Json(json!({ "error": error, "message": message })))
54
+
}
55
+
56
+
fn missing_param(name: &str) -> (StatusCode, Json<Value>) {
57
+
error_response(
58
+
StatusCode::BAD_REQUEST,
59
+
"InvalidRequest",
60
+
&format!("missing required parameter: {name}"),
61
+
)
62
+
}
63
+
64
+
/// Try to decode data as DAG-CBOR, then plain JSON, then base64.
65
+
fn decode_record_value(data: &[u8]) -> Value {
66
+
if let Ok(ipld) = atproto_dasl::drisl::from_slice::<atproto_dasl::Ipld>(data) {
67
+
return ipld_to_json(&ipld);
68
+
}
69
+
if let Ok(val) = serde_json::from_slice::<Value>(data) {
70
+
return val;
71
+
}
72
+
use base64::Engine;
73
+
json!({ "$bytes": base64::engine::general_purpose::STANDARD.encode(data) })
74
+
}
75
+
76
+
/// Convert an `Ipld` value to `serde_json::Value`, encoding CID links as
77
+
/// `{"$link": "<cid-string>"}` and bytes as `{"$bytes": "<base64>"}`.
78
+
fn ipld_to_json(ipld: &atproto_dasl::Ipld) -> Value {
79
+
use atproto_dasl::Ipld;
80
+
match ipld {
81
+
Ipld::Null => Value::Null,
82
+
Ipld::Bool(b) => Value::Bool(*b),
83
+
Ipld::Integer(i) => json!(*i),
84
+
Ipld::Float(f) => json!(*f),
85
+
Ipld::String(s) => Value::String(s.clone()),
86
+
Ipld::Bytes(b) => {
87
+
use base64::Engine;
88
+
json!({ "$bytes": base64::prelude::BASE64_STANDARD.encode(b) })
89
+
}
90
+
Ipld::List(list) => Value::Array(list.iter().map(ipld_to_json).collect()),
91
+
Ipld::Map(map) => {
92
+
let obj: serde_json::Map<String, Value> = map
93
+
.iter()
94
+
.map(|(k, v)| (k.clone(), ipld_to_json(v)))
95
+
.collect();
96
+
Value::Object(obj)
97
+
}
98
+
Ipld::Link(cid) => {
99
+
json!({ "$link": cid.to_string() })
100
+
}
101
+
}
102
+
}
103
+
104
+
/// Handle `GET /xrpc/com.atproto.repo.getRecord`.
105
+
///
106
+
/// Finds the latest version of a record by prefix-scanning all revisions
107
+
/// and taking the last entry. A tombstone (empty value) means deleted.
108
+
pub async fn handle_get_record(
109
+
State(state): State<AppState>,
110
+
Query(params): Query<GetRecordParams>,
111
+
) -> (StatusCode, Json<Value>) {
112
+
let Some(repo) = params.repo.as_deref().filter(|s| !s.is_empty()) else {
113
+
return missing_param("repo");
114
+
};
115
+
let Some(collection) = params.collection.as_deref().filter(|s| !s.is_empty()) else {
116
+
return missing_param("collection");
117
+
};
118
+
let Some(rkey) = params.rkey.as_deref().filter(|s| !s.is_empty()) else {
119
+
return missing_param("rkey");
120
+
};
121
+
122
+
let prefix = keys::encode_record_prefix(repo, collection, rkey);
123
+
124
+
// Scan all versions, keep the last (latest rev sorts last lexicographically)
125
+
let mut latest: Option<(Vec<u8>, Vec<u8>)> = None;
126
+
for guard in state.db.records.prefix(&prefix) {
127
+
let Ok((key, value)) = guard.into_inner() else {
128
+
continue;
129
+
};
130
+
latest = Some((key.to_vec(), value.to_vec()));
131
+
}
132
+
133
+
match latest {
134
+
Some((_key, value)) if RecordValue::is_tombstone(&value) => {
135
+
error_response(StatusCode::NOT_FOUND, "RecordNotFound", "record not found")
136
+
}
137
+
Some((key, value)) => {
138
+
let decompressed = match state.db.decompress_event(&value) {
139
+
Ok(d) => d,
140
+
Err(e) => {
141
+
return error_response(
142
+
StatusCode::INTERNAL_SERVER_ERROR,
143
+
"InternalError",
144
+
&format!("failed to decompress record: {e}"),
145
+
);
146
+
}
147
+
};
148
+
let rev = keys::decode_record_key(&key)
149
+
.map(|(_, _, _, r)| r.to_string())
150
+
.unwrap_or_default();
151
+
match RecordValue::decode(&decompressed) {
152
+
Ok(record) => (
153
+
StatusCode::OK,
154
+
Json(json!({
155
+
"uri": format!("at://{repo}/{collection}/{rkey}"),
156
+
"cid": record.cid_string(),
157
+
"rev": rev,
158
+
"value": decode_record_value(&record.data),
159
+
})),
160
+
),
161
+
Err(e) => error_response(
162
+
StatusCode::INTERNAL_SERVER_ERROR,
163
+
"InternalError",
164
+
&format!("failed to decode record: {e}"),
165
+
),
166
+
}
167
+
}
168
+
None => error_response(StatusCode::NOT_FOUND, "RecordNotFound", "record not found"),
169
+
}
170
+
}
171
+
172
+
/// Handle `GET /xrpc/com.atproto.repo.listRecords`.
173
+
///
174
+
/// With versioned keys (`did\x00collection\x00rkey\x00rev`), multiple entries
175
+
/// exist per record. We group by rkey, take the latest version of each, and
176
+
/// skip tombstoned records.
177
+
pub async fn handle_list_records(
178
+
State(state): State<AppState>,
179
+
Query(params): Query<ListRecordsParams>,
180
+
) -> (StatusCode, Json<Value>) {
181
+
let Some(repo) = params.repo.as_deref().filter(|s| !s.is_empty()) else {
182
+
return missing_param("repo");
183
+
};
184
+
let Some(collection) = params.collection.as_deref().filter(|s| !s.is_empty()) else {
185
+
return missing_param("collection");
186
+
};
187
+
188
+
let limit = params.limit.unwrap_or(50).min(100);
189
+
let prefix = keys::encode_collection_prefix(repo, collection);
190
+
191
+
let mut records = Vec::new();
192
+
let mut last_rkey = None;
193
+
194
+
// Track the current rkey group and its latest value
195
+
let mut current_rkey: Option<String> = None;
196
+
let mut current_value: Option<Vec<u8>> = None;
197
+
198
+
for guard in state.db.records.prefix(&prefix) {
199
+
let Ok((key, value)) = guard.into_inner() else {
200
+
continue;
201
+
};
202
+
let Ok((_, _, rkey, _rev)) = keys::decode_record_key(&key) else {
203
+
continue;
204
+
};
205
+
206
+
let decompressed = match state.db.decompress_event(&value) {
207
+
Ok(d) => d,
208
+
Err(_) => continue,
209
+
};
210
+
211
+
if current_rkey.as_deref() == Some(rkey) {
212
+
// Same rkey — this is a newer revision, overwrite
213
+
current_value = Some(decompressed);
214
+
continue;
215
+
}
216
+
217
+
// New rkey — emit the previous group if it was a live record
218
+
if let (Some(prev_rkey), Some(prev_value)) = (¤t_rkey, ¤t_value) {
219
+
if !RecordValue::is_tombstone(prev_value) {
220
+
let past_cursor = params
221
+
.cursor
222
+
.as_ref()
223
+
.is_none_or(|c| prev_rkey.as_str() > c.as_str());
224
+
if past_cursor {
225
+
if let Ok(record) = RecordValue::decode(prev_value) {
226
+
records.push(json!({
227
+
"uri": format!("at://{repo}/{collection}/{prev_rkey}"),
228
+
"cid": record.cid_string(),
229
+
"value": decode_record_value(&record.data),
230
+
}));
231
+
last_rkey = Some(prev_rkey.clone());
232
+
}
233
+
}
234
+
}
235
+
if records.len() >= limit {
236
+
break;
237
+
}
238
+
}
239
+
240
+
current_rkey = Some(rkey.to_string());
241
+
current_value = Some(decompressed);
242
+
}
243
+
244
+
// Emit final group
245
+
if records.len() < limit {
246
+
if let (Some(prev_rkey), Some(prev_value)) = (¤t_rkey, ¤t_value) {
247
+
if !RecordValue::is_tombstone(prev_value) {
248
+
let past_cursor = params
249
+
.cursor
250
+
.as_ref()
251
+
.is_none_or(|c| prev_rkey.as_str() > c.as_str());
252
+
if past_cursor {
253
+
if let Ok(record) = RecordValue::decode(prev_value) {
254
+
records.push(json!({
255
+
"uri": format!("at://{repo}/{collection}/{prev_rkey}"),
256
+
"cid": record.cid_string(),
257
+
"value": decode_record_value(&record.data),
258
+
}));
259
+
last_rkey = Some(prev_rkey.clone());
260
+
}
261
+
}
262
+
}
263
+
}
264
+
}
265
+
266
+
let mut response = json!({ "records": records });
267
+
if records.len() >= limit {
268
+
if let Some(cursor) = last_rkey {
269
+
response["cursor"] = json!(cursor);
270
+
}
271
+
}
272
+
273
+
(StatusCode::OK, Json(response))
274
+
}
275
+
276
+
/// Handle `GET /xrpc/com.atproto.repo.describeRepo`.
277
+
pub async fn handle_describe_repo(
278
+
State(state): State<AppState>,
279
+
Query(params): Query<DescribeRepoParams>,
280
+
) -> (StatusCode, Json<Value>) {
281
+
let Some(repo) = params.repo.as_deref().filter(|s| !s.is_empty()) else {
282
+
return missing_param("repo");
283
+
};
284
+
285
+
let repo_state = match state.db.get_repo_state(repo) {
286
+
Ok(Some(rs)) => rs,
287
+
Ok(None) => return error_response(StatusCode::NOT_FOUND, "RepoNotFound", "repo not found"),
288
+
Err(e) => {
289
+
return error_response(
290
+
StatusCode::INTERNAL_SERVER_ERROR,
291
+
"InternalError",
292
+
&format!("storage error: {e}"),
293
+
);
294
+
}
295
+
};
296
+
297
+
let handle = find_handle_for_did(&state, repo);
298
+
299
+
let did_doc = state
300
+
.db
301
+
.did_to_doc
302
+
.get(repo.as_bytes())
303
+
.ok()
304
+
.flatten()
305
+
.and_then(|bytes| {
306
+
let slice: &[u8] = &bytes;
307
+
let (_ts, json_bytes) = decode_timestamped_doc(slice);
308
+
serde_json::from_slice::<Value>(json_bytes).ok()
309
+
});
310
+
311
+
// Collect unique collections by seeking past each collection's records.
312
+
// Keys are sorted as DID\x00collection\x00rkey, so after finding one record
313
+
// in a collection we skip ahead to DID\x00collection\x00\xFF which is past
314
+
// any valid rkey (rkeys are printable ASCII, all < 0xFF).
315
+
let did_prefix = keys::encode_did_prefix(repo);
316
+
let mut collections = Vec::new();
317
+
let mut seek_key = did_prefix.clone();
318
+
319
+
const MAX_COLLECTIONS: usize = 1000;
320
+
321
+
while collections.len() < MAX_COLLECTIONS {
322
+
let mut found = false;
323
+
for guard in state.db.records.range(seek_key.clone()..) {
324
+
let Ok((key, _value)) = guard.into_inner() else {
325
+
continue;
326
+
};
327
+
let key_bytes: &[u8] = &key;
328
+
// Check we're still within this DID's prefix
329
+
if !key_bytes.starts_with(&did_prefix) {
330
+
break;
331
+
}
332
+
if let Ok((_, coll, _, _)) = keys::decode_record_key(key_bytes) {
333
+
collections.push(coll.to_string());
334
+
// Construct a seek key past all records in this collection
335
+
seek_key = keys::encode_collection_prefix(repo, coll);
336
+
seek_key.push(0xFF);
337
+
found = true;
338
+
break;
339
+
}
340
+
break;
341
+
}
342
+
if !found {
343
+
break;
344
+
}
345
+
}
346
+
347
+
let mut response = json!({
348
+
"did": repo,
349
+
"handle": handle.unwrap_or_default(),
350
+
"collections": collections,
351
+
"rev": repo_state.rev,
352
+
"status": repo_state.status.as_str(),
353
+
});
354
+
if let Some(doc) = did_doc {
355
+
response["didDoc"] = doc;
356
+
}
357
+
358
+
(StatusCode::OK, Json(response))
359
+
}
360
+
361
+
/// Handle `GET /xrpc/com.atproto.identity.resolveIdentity`.
362
+
pub async fn handle_resolve_identity(
363
+
State(state): State<AppState>,
364
+
headers: HeaderMap,
365
+
Query(params): Query<ResolveIdentityParams>,
366
+
) -> (StatusCode, Json<Value>) {
367
+
let Some(identifier) = params.identifier.as_deref().filter(|s| !s.is_empty()) else {
368
+
return missing_param("identifier");
369
+
};
370
+
371
+
let max_stale = parse_max_stale(&headers);
372
+
373
+
// If the identifier is a handle, try resolving it
374
+
let did = if identifier.starts_with("did:") {
375
+
identifier.to_string()
376
+
} else {
377
+
match lookup_handle(&state, identifier) {
378
+
Some(d) => d,
379
+
None => {
380
+
// Handle not in cache — try a live resolve
381
+
match resolve_and_cache(&state, identifier).await {
382
+
Ok(doc) => return (StatusCode::OK, Json(doc)),
383
+
Err(_) => {
384
+
return error_response(
385
+
StatusCode::NOT_FOUND,
386
+
"IdentityNotFound",
387
+
"identity not found",
388
+
);
389
+
}
390
+
}
391
+
}
392
+
}
393
+
};
394
+
395
+
lookup_or_resolve_did_doc(&state, &did, max_stale).await
396
+
}
397
+
398
+
/// Handle `GET /xrpc/com.atproto.identity.resolveHandle`.
399
+
pub async fn handle_resolve_handle(
400
+
State(state): State<AppState>,
401
+
headers: HeaderMap,
402
+
Query(params): Query<ResolveHandleParams>,
403
+
) -> (StatusCode, Json<Value>) {
404
+
let Some(handle) = params.handle.as_deref().filter(|s| !s.is_empty()) else {
405
+
return missing_param("handle");
406
+
};
407
+
408
+
// Check cache first
409
+
if let Some(did) = lookup_handle(&state, handle) {
410
+
return (StatusCode::OK, Json(json!({ "did": did })));
411
+
}
412
+
413
+
// Not cached — resolve live
414
+
let _ = parse_max_stale(&headers); // acknowledged but handle lookups are boolean
415
+
match resolve_and_cache(&state, handle).await {
416
+
Ok(doc) => {
417
+
if let Some(did) = doc.get("id").and_then(|v| v.as_str()) {
418
+
(StatusCode::OK, Json(json!({ "did": did })))
419
+
} else {
420
+
error_response(StatusCode::NOT_FOUND, "HandleNotFound", "handle not found")
421
+
}
422
+
}
423
+
Err(_) => error_response(StatusCode::NOT_FOUND, "HandleNotFound", "handle not found"),
424
+
}
425
+
}
426
+
427
+
/// Handle `GET /xrpc/com.atproto.identity.resolveDid`.
428
+
pub async fn handle_resolve_did(
429
+
State(state): State<AppState>,
430
+
headers: HeaderMap,
431
+
Query(params): Query<ResolveDidParams>,
432
+
) -> (StatusCode, Json<Value>) {
433
+
let Some(did) = params.did.as_deref().filter(|s| !s.is_empty()) else {
434
+
return missing_param("did");
435
+
};
436
+
437
+
let max_stale = parse_max_stale(&headers);
438
+
lookup_or_resolve_did_doc(&state, did, max_stale).await
439
+
}
440
+
441
+
/// Parse `Cache-Control: max-stale=N` from request headers.
442
+
/// Returns `Some(seconds)` if present, `None` otherwise.
443
+
fn parse_max_stale(headers: &HeaderMap) -> Option<u64> {
444
+
let value = headers.get("cache-control")?.to_str().ok()?;
445
+
for directive in value.split(',') {
446
+
let directive = directive.trim();
447
+
if let Some(val) = directive.strip_prefix("max-stale=") {
448
+
return val.trim().parse().ok();
449
+
}
450
+
}
451
+
None
452
+
}
453
+
454
+
fn lookup_handle(state: &AppState, handle: &str) -> Option<String> {
455
+
state
456
+
.metrics
457
+
.storage_ops_total
458
+
.get_or_create(&KeyspaceOpLabels {
459
+
keyspace: "handle_to_did".to_string(),
460
+
op: StorageOp::Read,
461
+
})
462
+
.inc();
463
+
state
464
+
.db
465
+
.handle_to_did
466
+
.get(handle.to_lowercase().as_bytes())
467
+
.ok()
468
+
.flatten()
469
+
.and_then(|bytes| {
470
+
let slice: &[u8] = &bytes;
471
+
std::str::from_utf8(slice).ok().map(|s| s.to_string())
472
+
})
473
+
}
474
+
475
+
/// Look up a DID document from cache, respecting max-stale.
476
+
/// If missing or too stale, resolve live and cache the result.
477
+
async fn lookup_or_resolve_did_doc(
478
+
state: &AppState,
479
+
did: &str,
480
+
max_stale: Option<u64>,
481
+
) -> (StatusCode, Json<Value>) {
482
+
let now = SystemTime::now()
483
+
.duration_since(UNIX_EPOCH)
484
+
.unwrap_or_default()
485
+
.as_secs();
486
+
487
+
// Check cache
488
+
state
489
+
.metrics
490
+
.storage_ops_total
491
+
.get_or_create(&KeyspaceOpLabels {
492
+
keyspace: "did_to_doc".to_string(),
493
+
op: StorageOp::Read,
494
+
})
495
+
.inc();
496
+
if let Ok(Some(bytes)) = state.db.did_to_doc.get(did.as_bytes()) {
497
+
let slice: &[u8] = &bytes;
498
+
let (ts, json_bytes) = decode_timestamped_doc(slice);
499
+
let age = now.saturating_sub(ts);
500
+
501
+
// If client provided max-stale and the document is within that window, return it
502
+
if let Some(max_stale_secs) = max_stale {
503
+
if age <= max_stale_secs {
504
+
if let Ok(doc) = serde_json::from_slice::<Value>(json_bytes) {
505
+
return (StatusCode::OK, Json(doc));
506
+
}
507
+
}
508
+
}
509
+
510
+
// If the document is fresh (less than 24h old), return it regardless
511
+
if age < 24 * 60 * 60 {
512
+
if let Ok(doc) = serde_json::from_slice::<Value>(json_bytes) {
513
+
return (StatusCode::OK, Json(doc));
514
+
}
515
+
}
516
+
}
517
+
518
+
// Cache miss or stale — resolve live
519
+
match resolve_and_cache(state, did).await {
520
+
Ok(doc) => (StatusCode::OK, Json(doc)),
521
+
Err(e) => error_response(
522
+
StatusCode::NOT_FOUND,
523
+
"DidNotFound",
524
+
&format!("failed to resolve DID: {e}"),
525
+
),
526
+
}
527
+
}
528
+
529
+
/// Resolve a subject (DID or handle) via the identity resolver,
530
+
/// cache the result, and return the document as JSON.
531
+
async fn resolve_and_cache(state: &AppState, subject: &str) -> anyhow::Result<Value> {
532
+
let start = Instant::now();
533
+
534
+
let document = match state.resolver.resolve(subject).await {
535
+
Ok(doc) => {
536
+
let elapsed = start.elapsed().as_secs_f64();
537
+
state
538
+
.metrics
539
+
.identity_resolve_duration_seconds
540
+
.observe(elapsed);
541
+
state
542
+
.metrics
543
+
.identity_resolves_total
544
+
.get_or_create(&IdentityOutcomeLabels {
545
+
outcome: IdentityOutcome::Success,
546
+
})
547
+
.inc();
548
+
doc
549
+
}
550
+
Err(e) => {
551
+
let elapsed = start.elapsed().as_secs_f64();
552
+
state
553
+
.metrics
554
+
.identity_resolve_duration_seconds
555
+
.observe(elapsed);
556
+
state
557
+
.metrics
558
+
.identity_resolves_total
559
+
.get_or_create(&IdentityOutcomeLabels {
560
+
outcome: IdentityOutcome::Failure,
561
+
})
562
+
.inc();
563
+
return Err(e);
564
+
}
565
+
};
566
+
567
+
// Cache the document with timestamp
568
+
let doc_json = serde_json::to_vec(&document)?;
569
+
let now = SystemTime::now()
570
+
.duration_since(UNIX_EPOCH)
571
+
.unwrap_or_default()
572
+
.as_secs();
573
+
let value = encode_timestamped_doc(now, &doc_json);
574
+
state.db.did_to_doc.insert(document.id.as_bytes(), &value)?;
575
+
state
576
+
.metrics
577
+
.storage_ops_total
578
+
.get_or_create(&KeyspaceOpLabels {
579
+
keyspace: "did_to_doc".to_string(),
580
+
op: StorageOp::Write,
581
+
})
582
+
.inc();
583
+
584
+
// Update handle→DID mapping
585
+
for alias in &document.also_known_as {
586
+
if let Some(handle) = alias.strip_prefix("at://") {
587
+
state
588
+
.db
589
+
.handle_to_did
590
+
.insert(handle.to_lowercase().as_bytes(), document.id.as_bytes())?;
591
+
state
592
+
.metrics
593
+
.storage_ops_total
594
+
.get_or_create(&KeyspaceOpLabels {
595
+
keyspace: "handle_to_did".to_string(),
596
+
op: StorageOp::Write,
597
+
})
598
+
.inc();
599
+
}
600
+
}
601
+
602
+
let doc_value = serde_json::to_value(&document)?;
603
+
Ok(doc_value)
604
+
}
605
+
606
+
fn find_handle_for_did(state: &AppState, did: &str) -> Option<String> {
607
+
let bytes = state.db.did_to_doc.get(did.as_bytes()).ok()??;
608
+
let slice: &[u8] = &bytes;
609
+
let (_ts, json_bytes) = decode_timestamped_doc(slice);
610
+
let doc: Value = serde_json::from_slice(json_bytes).ok()?;
611
+
let aliases = doc.get("alsoKnownAs")?.as_array()?;
612
+
for alias in aliases {
613
+
if let Some(handle) = alias.as_str()?.strip_prefix("at://") {
614
+
return Some(handle.to_string());
615
+
}
616
+
}
617
+
None
618
+
}
+1308
src/storage/encoding.rs
+1308
src/storage/encoding.rs
···
1
+
//! Binary encoding and decoding for fjall keyspace values.
2
+
//!
3
+
//! Uses compact binary formats with length-prefixed variable-length fields
4
+
//! to avoid the overhead of general-purpose serializers for internal data.
5
+
6
+
use crate::errors::RamjetError;
7
+
use crate::types::{AccountStatus, CommitOp, CommitOps, OpType, RecordValue, RepoState};
8
+
9
+
/// CIDv1 dag-cbor SHA-256 is always 36 bytes:
10
+
/// 1B version (0x01) + 1B codec (0x71) + 1B hash fn (0x12) + 1B digest len (0x20) + 32B digest.
11
+
const CID_LENGTH: usize = 36;
12
+
13
+
// -- RecordValue: [36B cid][remaining = data] --
14
+
//
15
+
// The revision is stored in the key (`did\x00collection\x00rkey\x00rev`),
16
+
// not in the value. An empty value (0 bytes) represents a tombstone.
17
+
18
+
impl RecordValue {
19
+
/// Encode to binary for storage.
20
+
pub fn encode(&self) -> Vec<u8> {
21
+
let total = CID_LENGTH + self.data.len();
22
+
let mut buf = Vec::with_capacity(total);
23
+
buf.extend_from_slice(&self.cid);
24
+
buf.extend_from_slice(&self.data);
25
+
buf
26
+
}
27
+
28
+
/// Decode from binary storage.
29
+
pub fn decode(bytes: &[u8]) -> Result<Self, RamjetError> {
30
+
if bytes.len() < CID_LENGTH {
31
+
return Err(RamjetError::ValueDecoding {
32
+
reason: format!(
33
+
"RecordValue too short: need at least {} bytes, got {}",
34
+
CID_LENGTH,
35
+
bytes.len()
36
+
),
37
+
});
38
+
}
39
+
40
+
let cid = bytes[..CID_LENGTH].to_vec();
41
+
let data = bytes[CID_LENGTH..].to_vec();
42
+
43
+
Ok(Self { cid, data })
44
+
}
45
+
46
+
/// Returns true if this value is a tombstone (deleted record).
47
+
pub fn is_tombstone(bytes: &[u8]) -> bool {
48
+
bytes.is_empty()
49
+
}
50
+
51
+
/// Get the CID as a multibase-encoded string (for XRPC responses).
52
+
pub fn cid_string(&self) -> String {
53
+
match cid::Cid::read_bytes(self.cid.as_slice()) {
54
+
Ok(c) => c.to_string(),
55
+
Err(_) => hex_encode(&self.cid),
56
+
}
57
+
}
58
+
}
59
+
60
+
// -- RepoState: [13B rev][1B status][1B denied] --
61
+
62
+
impl RepoState {
63
+
/// Encode to binary for storage.
64
+
///
65
+
/// Format: `[rev bytes (variable length)][1B status][1B denied][1B backfilled]`
66
+
pub fn encode(&self) -> Vec<u8> {
67
+
let rev_bytes = self.rev.as_bytes();
68
+
let mut buf = Vec::with_capacity(rev_bytes.len() + 3);
69
+
buf.extend_from_slice(rev_bytes);
70
+
buf.push(self.status.as_u8());
71
+
buf.push(u8::from(self.denied));
72
+
buf.push(u8::from(self.backfilled));
73
+
buf
74
+
}
75
+
76
+
/// Decode from binary storage.
77
+
///
78
+
/// V2 format: `[rev bytes (variable length)][1B status][1B denied][1B backfilled]`
79
+
/// V1 format: `[rev bytes (variable length)][1B status][1B denied]`
80
+
///
81
+
/// V1 entries (2 trailer bytes) are decoded with `backfilled = false`.
82
+
pub fn decode(bytes: &[u8]) -> Result<Self, RamjetError> {
83
+
if bytes.len() < 2 {
84
+
return Err(RamjetError::ValueDecoding {
85
+
reason: format!(
86
+
"RepoState too short: need at least 2 bytes, got {}",
87
+
bytes.len()
88
+
),
89
+
});
90
+
}
91
+
92
+
// Detect v1 (2 trailer bytes) vs v2 (3 trailer bytes).
93
+
// Rev is always a 13-character TID or empty. We use the total length
94
+
// to determine the format: if (len - 3) is 0 or 13 it's v2, otherwise v1.
95
+
let trailer_len = if bytes.len() >= 3 {
96
+
let candidate_rev_len = bytes.len() - 3;
97
+
if candidate_rev_len == 0 || candidate_rev_len == 13 {
98
+
3
99
+
} else {
100
+
2
101
+
}
102
+
} else {
103
+
2
104
+
};
105
+
106
+
let rev_end = bytes.len() - trailer_len;
107
+
let rev =
108
+
std::str::from_utf8(&bytes[..rev_end]).map_err(|_| RamjetError::ValueDecoding {
109
+
reason: "invalid UTF-8 in rev".to_string(),
110
+
})?;
111
+
112
+
let status =
113
+
AccountStatus::from_u8(bytes[rev_end]).ok_or_else(|| RamjetError::ValueDecoding {
114
+
reason: format!("invalid account status byte: {}", bytes[rev_end]),
115
+
})?;
116
+
117
+
let denied = bytes[rev_end + 1] != 0;
118
+
let backfilled = if trailer_len == 3 {
119
+
bytes[rev_end + 2] != 0
120
+
} else {
121
+
false
122
+
};
123
+
124
+
Ok(Self {
125
+
rev: rev.to_string(),
126
+
status,
127
+
denied,
128
+
backfilled,
129
+
})
130
+
}
131
+
132
+
/// Get the rev as a string (for XRPC responses).
133
+
pub fn rev_string(&self) -> String {
134
+
self.rev.clone()
135
+
}
136
+
137
+
/// Get the status as a string (for XRPC responses).
138
+
pub fn status_string(&self) -> Option<String> {
139
+
Some(self.status.as_str().to_string())
140
+
}
141
+
}
142
+
143
+
// -- CommitOps: [2B count][per op: 1B type, 2B coll_len, coll, 2B rkey_len, rkey, 4B cid_len, cid] --
144
+
145
+
impl CommitOps {
146
+
/// Encode to binary for storage.
147
+
pub fn encode(&self) -> Vec<u8> {
148
+
let count = self.ops.len() as u16;
149
+
let mut buf = Vec::new();
150
+
buf.extend_from_slice(&count.to_be_bytes());
151
+
152
+
for op in &self.ops {
153
+
buf.push(op.op.as_u8());
154
+
155
+
let coll_len = op.collection.len() as u16;
156
+
buf.extend_from_slice(&coll_len.to_be_bytes());
157
+
buf.extend_from_slice(op.collection.as_bytes());
158
+
159
+
let rkey_len = op.rkey.len() as u16;
160
+
buf.extend_from_slice(&rkey_len.to_be_bytes());
161
+
buf.extend_from_slice(op.rkey.as_bytes());
162
+
163
+
match &op.cid {
164
+
Some(cid) => {
165
+
let cid_len = cid.len() as u32;
166
+
buf.extend_from_slice(&cid_len.to_be_bytes());
167
+
buf.extend_from_slice(cid);
168
+
}
169
+
None => {
170
+
buf.extend_from_slice(&0u32.to_be_bytes());
171
+
}
172
+
}
173
+
}
174
+
175
+
buf
176
+
}
177
+
178
+
/// Decode from binary storage.
179
+
pub fn decode(bytes: &[u8]) -> Result<Self, RamjetError> {
180
+
if bytes.len() < 2 {
181
+
return Err(RamjetError::ValueDecoding {
182
+
reason: "CommitOps too short".to_string(),
183
+
});
184
+
}
185
+
186
+
let count = u16::from_be_bytes([bytes[0], bytes[1]]) as usize;
187
+
let mut pos = 2;
188
+
let mut ops = Vec::with_capacity(count);
189
+
190
+
for _ in 0..count {
191
+
if pos >= bytes.len() {
192
+
return Err(RamjetError::ValueDecoding {
193
+
reason: "CommitOps truncated".to_string(),
194
+
});
195
+
}
196
+
197
+
let op_type =
198
+
OpType::from_u8(bytes[pos]).ok_or_else(|| RamjetError::ValueDecoding {
199
+
reason: format!("invalid op type byte: {}", bytes[pos]),
200
+
})?;
201
+
pos += 1;
202
+
203
+
let coll_len = read_u16(bytes, &mut pos)?;
204
+
let collection = read_string(bytes, &mut pos, coll_len)?;
205
+
206
+
let rkey_len = read_u16(bytes, &mut pos)?;
207
+
let rkey = read_string(bytes, &mut pos, rkey_len)?;
208
+
209
+
let cid_len = read_u32(bytes, &mut pos)?;
210
+
let cid = if cid_len > 0 {
211
+
Some(read_bytes(bytes, &mut pos, cid_len)?)
212
+
} else {
213
+
None
214
+
};
215
+
216
+
ops.push(CommitOp {
217
+
op: op_type,
218
+
collection,
219
+
rkey,
220
+
cid,
221
+
});
222
+
}
223
+
224
+
Ok(Self { ops })
225
+
}
226
+
}
227
+
228
+
fn read_u16(bytes: &[u8], pos: &mut usize) -> Result<usize, RamjetError> {
229
+
if *pos + 2 > bytes.len() {
230
+
return Err(RamjetError::ValueDecoding {
231
+
reason: "unexpected end of data reading u16".to_string(),
232
+
});
233
+
}
234
+
let val = u16::from_be_bytes([bytes[*pos], bytes[*pos + 1]]) as usize;
235
+
*pos += 2;
236
+
Ok(val)
237
+
}
238
+
239
+
fn read_u32(bytes: &[u8], pos: &mut usize) -> Result<usize, RamjetError> {
240
+
if *pos + 4 > bytes.len() {
241
+
return Err(RamjetError::ValueDecoding {
242
+
reason: "unexpected end of data reading u32".to_string(),
243
+
});
244
+
}
245
+
let val = u32::from_be_bytes(bytes[*pos..*pos + 4].try_into().unwrap()) as usize;
246
+
*pos += 4;
247
+
Ok(val)
248
+
}
249
+
250
+
fn read_string(bytes: &[u8], pos: &mut usize, len: usize) -> Result<String, RamjetError> {
251
+
if *pos + len > bytes.len() {
252
+
return Err(RamjetError::ValueDecoding {
253
+
reason: "unexpected end of data reading string".to_string(),
254
+
});
255
+
}
256
+
let s = std::str::from_utf8(&bytes[*pos..*pos + len])
257
+
.map_err(|_| RamjetError::ValueDecoding {
258
+
reason: "invalid UTF-8 in string field".to_string(),
259
+
})?
260
+
.to_string();
261
+
*pos += len;
262
+
Ok(s)
263
+
}
264
+
265
+
fn read_bytes(bytes: &[u8], pos: &mut usize, len: usize) -> Result<Vec<u8>, RamjetError> {
266
+
if *pos + len > bytes.len() {
267
+
return Err(RamjetError::ValueDecoding {
268
+
reason: "unexpected end of data reading bytes".to_string(),
269
+
});
270
+
}
271
+
let b = bytes[*pos..*pos + len].to_vec();
272
+
*pos += len;
273
+
Ok(b)
274
+
}
275
+
276
+
fn hex_encode(bytes: &[u8]) -> String {
277
+
bytes.iter().map(|b| format!("{b:02x}")).collect()
278
+
}
279
+
280
+
// -- Compact event encoding for the events keyspace --
281
+
//
282
+
// Stores events in a compact binary format instead of JSON.
283
+
// Record bodies are kept as raw DAG-CBOR bytes (not expanded to JSON).
284
+
// Conversion to JSON happens at read time (cursor replay).
285
+
286
+
/// Tag bytes for event types in the compact binary format.
287
+
/// V1 tags (legacy grouped format).
288
+
const EVENT_TAG_COMMIT: u8 = 0x01;
289
+
const EVENT_TAG_IDENTITY: u8 = 0x02;
290
+
const EVENT_TAG_ACCOUNT: u8 = 0x03;
291
+
/// V2 tags (per-op commits, all events carry `time_us`).
292
+
const EVENT_TAG_COMMIT_OP: u8 = 0x04;
293
+
const EVENT_TAG_IDENTITY_V2: u8 = 0x05;
294
+
const EVENT_TAG_ACCOUNT_V2: u8 = 0x06;
295
+
296
+
/// Extract the DID from a compact binary event without fully decoding it.
297
+
///
298
+
/// This is used for fast partition filtering during cursor replay.
299
+
/// Returns `None` if the bytes are too short or the tag is unrecognized.
300
+
pub fn extract_did_from_compact(bytes: &[u8]) -> Option<&str> {
301
+
if bytes.len() < 12 {
302
+
return None;
303
+
}
304
+
let tag = bytes[0];
305
+
// DID offset depends on whether the format includes time_us (8 bytes).
306
+
// V1 tags (0x01..=0x03): [1B tag][8B seq][2B did_len][did...]
307
+
// V2 tags (0x04..=0x06): [1B tag][8B seq][8B time_us][2B did_len][did...]
308
+
let did_len_offset = match tag {
309
+
EVENT_TAG_COMMIT | EVENT_TAG_IDENTITY | EVENT_TAG_ACCOUNT => 9,
310
+
EVENT_TAG_COMMIT_OP | EVENT_TAG_IDENTITY_V2 | EVENT_TAG_ACCOUNT_V2 => 17,
311
+
_ => return None,
312
+
};
313
+
if bytes.len() < did_len_offset + 2 {
314
+
return None;
315
+
}
316
+
let did_len = u16::from_be_bytes([bytes[did_len_offset], bytes[did_len_offset + 1]]) as usize;
317
+
let did_start = did_len_offset + 2;
318
+
if bytes.len() < did_start + did_len {
319
+
return None;
320
+
}
321
+
std::str::from_utf8(&bytes[did_start..did_start + did_len]).ok()
322
+
}
323
+
324
+
/// A decoded compact event from the events keyspace.
325
+
#[derive(Debug)]
326
+
pub enum CompactEvent {
327
+
/// V1 grouped commit (legacy). Multiple ops per event.
328
+
Commit {
329
+
seq: u64,
330
+
did: String,
331
+
rev: String,
332
+
ops: Vec<CompactOp>,
333
+
},
334
+
/// V2 per-operation commit with microsecond timestamp.
335
+
CommitOp {
336
+
seq: u64,
337
+
time_us: u64,
338
+
did: String,
339
+
rev: String,
340
+
op: CompactOp,
341
+
},
342
+
Identity {
343
+
seq: u64,
344
+
time_us: u64,
345
+
did: String,
346
+
handle: Option<String>,
347
+
},
348
+
Account {
349
+
seq: u64,
350
+
time_us: u64,
351
+
did: String,
352
+
active: bool,
353
+
status: Option<String>,
354
+
},
355
+
}
356
+
357
+
/// A single op within a compact commit event.
358
+
#[derive(Debug)]
359
+
pub struct CompactOp {
360
+
pub action: OpType,
361
+
pub collection: String,
362
+
pub rkey: String,
363
+
pub cid: Option<Vec<u8>>,
364
+
/// Raw DAG-CBOR record body (empty for deletes).
365
+
pub data: Vec<u8>,
366
+
}
367
+
368
+
/// Encode a commit event in compact binary format.
369
+
///
370
+
/// Format: `[1B tag][8B seq][2B did_len][did][13B rev][2B op_count]`
371
+
/// Per op: `[1B action][2B coll_len][coll][2B rkey_len][rkey][1B cid_len][cid][4B data_len][data]`
372
+
pub fn encode_compact_commit(seq: u64, did: &str, rev: &str, ops: &[CompactOp]) -> Vec<u8> {
373
+
let mut buf = Vec::with_capacity(128);
374
+
buf.push(EVENT_TAG_COMMIT);
375
+
buf.extend_from_slice(&seq.to_be_bytes());
376
+
let did_len = did.len() as u16;
377
+
buf.extend_from_slice(&did_len.to_be_bytes());
378
+
buf.extend_from_slice(did.as_bytes());
379
+
// Rev is padded/truncated to exactly 13 bytes
380
+
let rev_bytes = rev.as_bytes();
381
+
if rev_bytes.len() >= 13 {
382
+
buf.extend_from_slice(&rev_bytes[..13]);
383
+
} else {
384
+
buf.extend_from_slice(rev_bytes);
385
+
buf.extend(std::iter::repeat_n(0u8, 13 - rev_bytes.len()));
386
+
}
387
+
let op_count = ops.len() as u16;
388
+
buf.extend_from_slice(&op_count.to_be_bytes());
389
+
for op in ops {
390
+
buf.push(op.action.as_u8());
391
+
let coll_len = op.collection.len() as u16;
392
+
buf.extend_from_slice(&coll_len.to_be_bytes());
393
+
buf.extend_from_slice(op.collection.as_bytes());
394
+
let rkey_len = op.rkey.len() as u16;
395
+
buf.extend_from_slice(&rkey_len.to_be_bytes());
396
+
buf.extend_from_slice(op.rkey.as_bytes());
397
+
match &op.cid {
398
+
Some(cid) => {
399
+
buf.push(cid.len() as u8);
400
+
buf.extend_from_slice(cid);
401
+
}
402
+
None => {
403
+
buf.push(0);
404
+
}
405
+
}
406
+
let data_len = op.data.len() as u32;
407
+
buf.extend_from_slice(&data_len.to_be_bytes());
408
+
buf.extend_from_slice(&op.data);
409
+
}
410
+
buf
411
+
}
412
+
413
+
/// Encode an identity event in compact binary format.
414
+
///
415
+
/// Format: `[1B tag][8B seq][2B did_len][did][1B has_handle][2B handle_len][handle]`
416
+
pub fn encode_compact_identity(seq: u64, did: &str, handle: Option<&str>) -> Vec<u8> {
417
+
let mut buf = Vec::with_capacity(64);
418
+
buf.push(EVENT_TAG_IDENTITY);
419
+
buf.extend_from_slice(&seq.to_be_bytes());
420
+
let did_len = did.len() as u16;
421
+
buf.extend_from_slice(&did_len.to_be_bytes());
422
+
buf.extend_from_slice(did.as_bytes());
423
+
match handle {
424
+
Some(h) => {
425
+
buf.push(1);
426
+
let h_len = h.len() as u16;
427
+
buf.extend_from_slice(&h_len.to_be_bytes());
428
+
buf.extend_from_slice(h.as_bytes());
429
+
}
430
+
None => {
431
+
buf.push(0);
432
+
}
433
+
}
434
+
buf
435
+
}
436
+
437
+
/// Encode an account event in compact binary format.
438
+
///
439
+
/// Format: `[1B tag][8B seq][2B did_len][did][1B active][1B has_status][2B status_len][status]`
440
+
pub fn encode_compact_account(seq: u64, did: &str, active: bool, status: Option<&str>) -> Vec<u8> {
441
+
let mut buf = Vec::with_capacity(64);
442
+
buf.push(EVENT_TAG_ACCOUNT);
443
+
buf.extend_from_slice(&seq.to_be_bytes());
444
+
let did_len = did.len() as u16;
445
+
buf.extend_from_slice(&did_len.to_be_bytes());
446
+
buf.extend_from_slice(did.as_bytes());
447
+
buf.push(u8::from(active));
448
+
match status {
449
+
Some(s) => {
450
+
buf.push(1);
451
+
let s_len = s.len() as u16;
452
+
buf.extend_from_slice(&s_len.to_be_bytes());
453
+
buf.extend_from_slice(s.as_bytes());
454
+
}
455
+
None => {
456
+
buf.push(0);
457
+
}
458
+
}
459
+
buf
460
+
}
461
+
462
+
/// Encode a single commit op in compact binary v2 format.
463
+
///
464
+
/// Format: `[1B tag][8B seq][8B time_us][2B did_len][did][13B rev][1B action][2B coll_len][coll][2B rkey_len][rkey][1B cid_len][cid][4B data_len][data]`
465
+
pub fn encode_compact_commit_op(
466
+
seq: u64,
467
+
time_us: u64,
468
+
did: &str,
469
+
rev: &str,
470
+
op: &CompactOp,
471
+
) -> Vec<u8> {
472
+
let mut buf = Vec::with_capacity(128);
473
+
buf.push(EVENT_TAG_COMMIT_OP);
474
+
buf.extend_from_slice(&seq.to_be_bytes());
475
+
buf.extend_from_slice(&time_us.to_be_bytes());
476
+
let did_len = did.len() as u16;
477
+
buf.extend_from_slice(&did_len.to_be_bytes());
478
+
buf.extend_from_slice(did.as_bytes());
479
+
let rev_bytes = rev.as_bytes();
480
+
if rev_bytes.len() >= 13 {
481
+
buf.extend_from_slice(&rev_bytes[..13]);
482
+
} else {
483
+
buf.extend_from_slice(rev_bytes);
484
+
buf.extend(std::iter::repeat_n(0u8, 13 - rev_bytes.len()));
485
+
}
486
+
buf.push(op.action.as_u8());
487
+
let coll_len = op.collection.len() as u16;
488
+
buf.extend_from_slice(&coll_len.to_be_bytes());
489
+
buf.extend_from_slice(op.collection.as_bytes());
490
+
let rkey_len = op.rkey.len() as u16;
491
+
buf.extend_from_slice(&rkey_len.to_be_bytes());
492
+
buf.extend_from_slice(op.rkey.as_bytes());
493
+
match &op.cid {
494
+
Some(cid) => {
495
+
buf.push(cid.len() as u8);
496
+
buf.extend_from_slice(cid);
497
+
}
498
+
None => {
499
+
buf.push(0);
500
+
}
501
+
}
502
+
let data_len = op.data.len() as u32;
503
+
buf.extend_from_slice(&data_len.to_be_bytes());
504
+
buf.extend_from_slice(&op.data);
505
+
buf
506
+
}
507
+
508
+
/// Encode an identity event in compact binary v2 format.
509
+
///
510
+
/// Format: `[1B tag][8B seq][8B time_us][2B did_len][did][1B has_handle][2B handle_len][handle]`
511
+
pub fn encode_compact_identity_v2(
512
+
seq: u64,
513
+
time_us: u64,
514
+
did: &str,
515
+
handle: Option<&str>,
516
+
) -> Vec<u8> {
517
+
let mut buf = Vec::with_capacity(64);
518
+
buf.push(EVENT_TAG_IDENTITY_V2);
519
+
buf.extend_from_slice(&seq.to_be_bytes());
520
+
buf.extend_from_slice(&time_us.to_be_bytes());
521
+
let did_len = did.len() as u16;
522
+
buf.extend_from_slice(&did_len.to_be_bytes());
523
+
buf.extend_from_slice(did.as_bytes());
524
+
match handle {
525
+
Some(h) => {
526
+
buf.push(1);
527
+
let h_len = h.len() as u16;
528
+
buf.extend_from_slice(&h_len.to_be_bytes());
529
+
buf.extend_from_slice(h.as_bytes());
530
+
}
531
+
None => {
532
+
buf.push(0);
533
+
}
534
+
}
535
+
buf
536
+
}
537
+
538
+
/// Encode an account event in compact binary v2 format.
539
+
///
540
+
/// Format: `[1B tag][8B seq][8B time_us][2B did_len][did][1B active][1B has_status][2B status_len][status]`
541
+
pub fn encode_compact_account_v2(
542
+
seq: u64,
543
+
time_us: u64,
544
+
did: &str,
545
+
active: bool,
546
+
status: Option<&str>,
547
+
) -> Vec<u8> {
548
+
let mut buf = Vec::with_capacity(64);
549
+
buf.push(EVENT_TAG_ACCOUNT_V2);
550
+
buf.extend_from_slice(&seq.to_be_bytes());
551
+
buf.extend_from_slice(&time_us.to_be_bytes());
552
+
let did_len = did.len() as u16;
553
+
buf.extend_from_slice(&did_len.to_be_bytes());
554
+
buf.extend_from_slice(did.as_bytes());
555
+
buf.push(u8::from(active));
556
+
match status {
557
+
Some(s) => {
558
+
buf.push(1);
559
+
let s_len = s.len() as u16;
560
+
buf.extend_from_slice(&s_len.to_be_bytes());
561
+
buf.extend_from_slice(s.as_bytes());
562
+
}
563
+
None => {
564
+
buf.push(0);
565
+
}
566
+
}
567
+
buf
568
+
}
569
+
570
+
/// Decode a compact event from the events keyspace.
571
+
pub fn decode_compact_event(bytes: &[u8]) -> Result<CompactEvent, RamjetError> {
572
+
if bytes.is_empty() {
573
+
return Err(RamjetError::ValueDecoding {
574
+
reason: "empty compact event".to_string(),
575
+
});
576
+
}
577
+
578
+
let tag = bytes[0];
579
+
let mut pos = 1;
580
+
581
+
match tag {
582
+
EVENT_TAG_COMMIT => {
583
+
let seq = read_u64(bytes, &mut pos)?;
584
+
let did_len = read_u16(bytes, &mut pos)?;
585
+
let did = read_string(bytes, &mut pos, did_len)?;
586
+
let rev_raw = read_bytes(bytes, &mut pos, 13)?;
587
+
// Trim trailing null padding from rev
588
+
let rev_end = rev_raw
589
+
.iter()
590
+
.position(|&b| b == 0)
591
+
.unwrap_or(rev_raw.len());
592
+
let rev = std::str::from_utf8(&rev_raw[..rev_end])
593
+
.map_err(|_| RamjetError::ValueDecoding {
594
+
reason: "invalid UTF-8 in rev".to_string(),
595
+
})?
596
+
.to_string();
597
+
let op_count = read_u16(bytes, &mut pos)?;
598
+
let mut ops = Vec::with_capacity(op_count);
599
+
for _ in 0..op_count {
600
+
if pos >= bytes.len() {
601
+
return Err(RamjetError::ValueDecoding {
602
+
reason: "compact commit event truncated".to_string(),
603
+
});
604
+
}
605
+
let action =
606
+
OpType::from_u8(bytes[pos]).ok_or_else(|| RamjetError::ValueDecoding {
607
+
reason: format!("invalid op type byte: {}", bytes[pos]),
608
+
})?;
609
+
pos += 1;
610
+
let coll_len = read_u16(bytes, &mut pos)?;
611
+
let collection = read_string(bytes, &mut pos, coll_len)?;
612
+
let rkey_len = read_u16(bytes, &mut pos)?;
613
+
let rkey = read_string(bytes, &mut pos, rkey_len)?;
614
+
if pos >= bytes.len() {
615
+
return Err(RamjetError::ValueDecoding {
616
+
reason: "compact commit event truncated at cid_len".to_string(),
617
+
});
618
+
}
619
+
let cid_len = bytes[pos] as usize;
620
+
pos += 1;
621
+
let cid = if cid_len > 0 {
622
+
Some(read_bytes(bytes, &mut pos, cid_len)?)
623
+
} else {
624
+
None
625
+
};
626
+
let data_len = read_u32(bytes, &mut pos)?;
627
+
let data = if data_len > 0 {
628
+
read_bytes(bytes, &mut pos, data_len)?
629
+
} else {
630
+
Vec::new()
631
+
};
632
+
ops.push(CompactOp {
633
+
action,
634
+
collection,
635
+
rkey,
636
+
cid,
637
+
data,
638
+
});
639
+
}
640
+
Ok(CompactEvent::Commit { seq, did, rev, ops })
641
+
}
642
+
EVENT_TAG_IDENTITY => {
643
+
let seq = read_u64(bytes, &mut pos)?;
644
+
let did_len = read_u16(bytes, &mut pos)?;
645
+
let did = read_string(bytes, &mut pos, did_len)?;
646
+
if pos >= bytes.len() {
647
+
return Err(RamjetError::ValueDecoding {
648
+
reason: "compact identity event truncated".to_string(),
649
+
});
650
+
}
651
+
let has_handle = bytes[pos] != 0;
652
+
pos += 1;
653
+
let handle = if has_handle {
654
+
let h_len = read_u16(bytes, &mut pos)?;
655
+
Some(read_string(bytes, &mut pos, h_len)?)
656
+
} else {
657
+
None
658
+
};
659
+
Ok(CompactEvent::Identity {
660
+
seq,
661
+
time_us: 0,
662
+
did,
663
+
handle,
664
+
})
665
+
}
666
+
EVENT_TAG_ACCOUNT => {
667
+
let seq = read_u64(bytes, &mut pos)?;
668
+
let did_len = read_u16(bytes, &mut pos)?;
669
+
let did = read_string(bytes, &mut pos, did_len)?;
670
+
if pos + 1 >= bytes.len() {
671
+
return Err(RamjetError::ValueDecoding {
672
+
reason: "compact account event truncated".to_string(),
673
+
});
674
+
}
675
+
let active = bytes[pos] != 0;
676
+
pos += 1;
677
+
let has_status = bytes[pos] != 0;
678
+
pos += 1;
679
+
let status = if has_status {
680
+
let s_len = read_u16(bytes, &mut pos)?;
681
+
Some(read_string(bytes, &mut pos, s_len)?)
682
+
} else {
683
+
None
684
+
};
685
+
Ok(CompactEvent::Account {
686
+
seq,
687
+
time_us: 0,
688
+
did,
689
+
active,
690
+
status,
691
+
})
692
+
}
693
+
EVENT_TAG_COMMIT_OP => {
694
+
let seq = read_u64(bytes, &mut pos)?;
695
+
let time_us = read_u64(bytes, &mut pos)?;
696
+
let did_len = read_u16(bytes, &mut pos)?;
697
+
let did = read_string(bytes, &mut pos, did_len)?;
698
+
let rev_raw = read_bytes(bytes, &mut pos, 13)?;
699
+
let rev_end = rev_raw
700
+
.iter()
701
+
.position(|&b| b == 0)
702
+
.unwrap_or(rev_raw.len());
703
+
let rev = std::str::from_utf8(&rev_raw[..rev_end])
704
+
.map_err(|_| RamjetError::ValueDecoding {
705
+
reason: "invalid UTF-8 in rev".to_string(),
706
+
})?
707
+
.to_string();
708
+
if pos >= bytes.len() {
709
+
return Err(RamjetError::ValueDecoding {
710
+
reason: "compact commit op truncated".to_string(),
711
+
});
712
+
}
713
+
let action = OpType::from_u8(bytes[pos]).ok_or_else(|| RamjetError::ValueDecoding {
714
+
reason: format!("invalid op type byte: {}", bytes[pos]),
715
+
})?;
716
+
pos += 1;
717
+
let coll_len = read_u16(bytes, &mut pos)?;
718
+
let collection = read_string(bytes, &mut pos, coll_len)?;
719
+
let rkey_len = read_u16(bytes, &mut pos)?;
720
+
let rkey = read_string(bytes, &mut pos, rkey_len)?;
721
+
if pos >= bytes.len() {
722
+
return Err(RamjetError::ValueDecoding {
723
+
reason: "compact commit op truncated at cid_len".to_string(),
724
+
});
725
+
}
726
+
let cid_len = bytes[pos] as usize;
727
+
pos += 1;
728
+
let cid = if cid_len > 0 {
729
+
Some(read_bytes(bytes, &mut pos, cid_len)?)
730
+
} else {
731
+
None
732
+
};
733
+
let data_len = read_u32(bytes, &mut pos)?;
734
+
let data = if data_len > 0 {
735
+
read_bytes(bytes, &mut pos, data_len)?
736
+
} else {
737
+
Vec::new()
738
+
};
739
+
Ok(CompactEvent::CommitOp {
740
+
seq,
741
+
time_us,
742
+
did,
743
+
rev,
744
+
op: CompactOp {
745
+
action,
746
+
collection,
747
+
rkey,
748
+
cid,
749
+
data,
750
+
},
751
+
})
752
+
}
753
+
EVENT_TAG_IDENTITY_V2 => {
754
+
let seq = read_u64(bytes, &mut pos)?;
755
+
let time_us = read_u64(bytes, &mut pos)?;
756
+
let did_len = read_u16(bytes, &mut pos)?;
757
+
let did = read_string(bytes, &mut pos, did_len)?;
758
+
if pos >= bytes.len() {
759
+
return Err(RamjetError::ValueDecoding {
760
+
reason: "compact identity v2 event truncated".to_string(),
761
+
});
762
+
}
763
+
let has_handle = bytes[pos] != 0;
764
+
pos += 1;
765
+
let handle = if has_handle {
766
+
let h_len = read_u16(bytes, &mut pos)?;
767
+
Some(read_string(bytes, &mut pos, h_len)?)
768
+
} else {
769
+
None
770
+
};
771
+
Ok(CompactEvent::Identity {
772
+
seq,
773
+
time_us,
774
+
did,
775
+
handle,
776
+
})
777
+
}
778
+
EVENT_TAG_ACCOUNT_V2 => {
779
+
let seq = read_u64(bytes, &mut pos)?;
780
+
let time_us = read_u64(bytes, &mut pos)?;
781
+
let did_len = read_u16(bytes, &mut pos)?;
782
+
let did = read_string(bytes, &mut pos, did_len)?;
783
+
if pos + 1 >= bytes.len() {
784
+
return Err(RamjetError::ValueDecoding {
785
+
reason: "compact account v2 event truncated".to_string(),
786
+
});
787
+
}
788
+
let active = bytes[pos] != 0;
789
+
pos += 1;
790
+
let has_status = bytes[pos] != 0;
791
+
pos += 1;
792
+
let status = if has_status {
793
+
let s_len = read_u16(bytes, &mut pos)?;
794
+
Some(read_string(bytes, &mut pos, s_len)?)
795
+
} else {
796
+
None
797
+
};
798
+
Ok(CompactEvent::Account {
799
+
seq,
800
+
time_us,
801
+
did,
802
+
active,
803
+
status,
804
+
})
805
+
}
806
+
_ => Err(RamjetError::ValueDecoding {
807
+
reason: format!("unknown compact event tag: {tag:#04x}"),
808
+
}),
809
+
}
810
+
}
811
+
812
+
fn read_u64(bytes: &[u8], pos: &mut usize) -> Result<u64, RamjetError> {
813
+
if *pos + 8 > bytes.len() {
814
+
return Err(RamjetError::ValueDecoding {
815
+
reason: "unexpected end of data reading u64".to_string(),
816
+
});
817
+
}
818
+
let val = u64::from_be_bytes(bytes[*pos..*pos + 8].try_into().unwrap());
819
+
*pos += 8;
820
+
Ok(val)
821
+
}
822
+
823
+
// -- Timestamped DID document: [8B BE unix secs][JSON bytes] --
824
+
825
+
const TIMESTAMP_LEN: usize = 8;
826
+
827
+
/// Encode a DID document with a timestamp prefix.
828
+
pub fn encode_timestamped_doc(timestamp_secs: u64, doc_json: &[u8]) -> Vec<u8> {
829
+
let mut buf = Vec::with_capacity(TIMESTAMP_LEN + doc_json.len());
830
+
buf.extend_from_slice(×tamp_secs.to_be_bytes());
831
+
buf.extend_from_slice(doc_json);
832
+
buf
833
+
}
834
+
835
+
/// Decode a DID document value, returning `(timestamp_secs, json_bytes)`.
836
+
/// Falls back to treating the entire value as JSON (timestamp = 0) for
837
+
/// values written before the timestamp prefix was added.
838
+
pub fn decode_timestamped_doc(bytes: &[u8]) -> (u64, &[u8]) {
839
+
if bytes.len() >= TIMESTAMP_LEN && bytes[0] != b'{' {
840
+
let ts = u64::from_be_bytes(bytes[..TIMESTAMP_LEN].try_into().unwrap());
841
+
(ts, &bytes[TIMESTAMP_LEN..])
842
+
} else {
843
+
// Legacy format: raw JSON without timestamp
844
+
(0, bytes)
845
+
}
846
+
}
847
+
848
+
// -- CBOR wire format helpers for WebSocket event delivery --
849
+
850
+
/// Build an `Ipld::Map` from string-key / Ipld pairs.
851
+
pub fn ipld_map(entries: Vec<(&str, atproto_dasl::Ipld)>) -> atproto_dasl::Ipld {
852
+
let map: std::collections::BTreeMap<String, atproto_dasl::Ipld> = entries
853
+
.into_iter()
854
+
.map(|(k, v)| (k.to_string(), v))
855
+
.collect();
856
+
atproto_dasl::Ipld::Map(map)
857
+
}
858
+
859
+
/// Serialize an `Ipld` value to DAG-CBOR bytes.
860
+
pub fn to_dag_cbor_bytes(value: &atproto_dasl::Ipld) -> Vec<u8> {
861
+
atproto_dasl::drisl::to_vec(value).unwrap_or_default()
862
+
}
863
+
864
+
#[cfg(test)]
865
+
mod tests {
866
+
use super::*;
867
+
868
+
/// Build a valid 36-byte CIDv1 dag-cbor SHA-256 for testing.
869
+
fn test_cid() -> Vec<u8> {
870
+
let mut cid = vec![0x01, 0x71, 0x12, 0x20]; // version, codec, hash fn, digest len
871
+
cid.extend_from_slice(&[0xAA; 32]); // 32-byte digest
872
+
cid
873
+
}
874
+
875
+
#[test]
876
+
fn test_record_value_roundtrip() {
877
+
let original = RecordValue {
878
+
cid: test_cid(),
879
+
data: vec![0xA1, 0x64, 0x74, 0x65, 0x73, 0x74],
880
+
};
881
+
882
+
let encoded = original.encode();
883
+
let decoded = RecordValue::decode(&encoded).unwrap();
884
+
885
+
assert_eq!(decoded.cid, original.cid);
886
+
assert_eq!(decoded.data, original.data);
887
+
}
888
+
889
+
#[test]
890
+
fn test_record_value_empty_data() {
891
+
let original = RecordValue {
892
+
cid: test_cid(),
893
+
data: vec![],
894
+
};
895
+
896
+
let encoded = original.encode();
897
+
let decoded = RecordValue::decode(&encoded).unwrap();
898
+
899
+
assert_eq!(decoded.cid, original.cid);
900
+
assert!(decoded.data.is_empty());
901
+
}
902
+
903
+
#[test]
904
+
fn test_record_value_tombstone() {
905
+
assert!(RecordValue::is_tombstone(b""));
906
+
assert!(!RecordValue::is_tombstone(&test_cid()));
907
+
}
908
+
909
+
#[test]
910
+
fn test_record_value_cid_string() {
911
+
let record = RecordValue {
912
+
cid: test_cid(),
913
+
data: vec![],
914
+
};
915
+
let cid_str = record.cid_string();
916
+
// CIDv1 multibase strings start with 'b' (base32lower)
917
+
assert!(
918
+
cid_str.starts_with('b'),
919
+
"CID string should be multibase base32lower: {cid_str}"
920
+
);
921
+
}
922
+
923
+
#[test]
924
+
fn test_repo_state_empty_rev() {
925
+
let original = RepoState {
926
+
rev: String::new(),
927
+
status: AccountStatus::Active,
928
+
denied: false,
929
+
backfilled: false,
930
+
};
931
+
932
+
let encoded = original.encode();
933
+
assert_eq!(encoded.len(), 3);
934
+
let decoded = RepoState::decode(&encoded).unwrap();
935
+
936
+
assert_eq!(decoded.rev, "");
937
+
assert_eq!(decoded.status, AccountStatus::Active);
938
+
assert!(!decoded.denied);
939
+
assert!(!decoded.backfilled);
940
+
}
941
+
942
+
#[test]
943
+
fn test_repo_state_roundtrip() {
944
+
let original = RepoState {
945
+
rev: "3jzfcijpj2z2a".to_string(),
946
+
status: AccountStatus::Suspended,
947
+
denied: true,
948
+
backfilled: true,
949
+
};
950
+
951
+
let encoded = original.encode();
952
+
let decoded = RepoState::decode(&encoded).unwrap();
953
+
954
+
assert_eq!(decoded.rev, original.rev);
955
+
assert_eq!(decoded.status, original.status);
956
+
assert_eq!(decoded.denied, original.denied);
957
+
assert_eq!(decoded.backfilled, original.backfilled);
958
+
}
959
+
960
+
#[test]
961
+
fn test_repo_state_default() {
962
+
let state = RepoState {
963
+
rev: "2222222222222".to_string(),
964
+
status: AccountStatus::Active,
965
+
denied: false,
966
+
backfilled: false,
967
+
};
968
+
969
+
let encoded = state.encode();
970
+
let decoded = RepoState::decode(&encoded).unwrap();
971
+
972
+
assert_eq!(decoded.status, AccountStatus::Active);
973
+
assert!(!decoded.denied);
974
+
assert!(!decoded.backfilled);
975
+
}
976
+
977
+
#[test]
978
+
fn test_repo_state_v1_compat() {
979
+
// V1 format: 13-byte rev + 2 trailer bytes (no backfilled field)
980
+
let mut v1_bytes = b"3jzfcijpj2z2a".to_vec();
981
+
v1_bytes.push(0); // status = Active
982
+
v1_bytes.push(1); // denied = true
983
+
let decoded = RepoState::decode(&v1_bytes).unwrap();
984
+
985
+
assert_eq!(decoded.rev, "3jzfcijpj2z2a");
986
+
assert_eq!(decoded.status, AccountStatus::Active);
987
+
assert!(decoded.denied);
988
+
assert!(!decoded.backfilled); // defaults to false for v1
989
+
}
990
+
991
+
#[test]
992
+
fn test_commit_ops_roundtrip() {
993
+
let original = CommitOps {
994
+
ops: vec![
995
+
CommitOp {
996
+
op: OpType::Create,
997
+
collection: "app.bsky.feed.post".to_string(),
998
+
rkey: "abc123".to_string(),
999
+
cid: Some(vec![0x01, 0x71]),
1000
+
},
1001
+
CommitOp {
1002
+
op: OpType::Delete,
1003
+
collection: "app.bsky.feed.like".to_string(),
1004
+
rkey: "def456".to_string(),
1005
+
cid: None,
1006
+
},
1007
+
CommitOp {
1008
+
op: OpType::Update,
1009
+
collection: "app.bsky.actor.profile".to_string(),
1010
+
rkey: "self".to_string(),
1011
+
cid: Some(vec![0x01, 0x71, 0x12]),
1012
+
},
1013
+
],
1014
+
};
1015
+
1016
+
let encoded = original.encode();
1017
+
let decoded = CommitOps::decode(&encoded).unwrap();
1018
+
1019
+
assert_eq!(decoded.ops.len(), 3);
1020
+
assert_eq!(decoded.ops[0].op, OpType::Create);
1021
+
assert_eq!(decoded.ops[0].collection, "app.bsky.feed.post");
1022
+
assert_eq!(decoded.ops[0].rkey, "abc123");
1023
+
assert_eq!(decoded.ops[0].cid, Some(vec![0x01, 0x71]));
1024
+
1025
+
assert_eq!(decoded.ops[1].op, OpType::Delete);
1026
+
assert!(decoded.ops[1].cid.is_none());
1027
+
1028
+
assert_eq!(decoded.ops[2].op, OpType::Update);
1029
+
assert_eq!(decoded.ops[2].rkey, "self");
1030
+
}
1031
+
1032
+
#[test]
1033
+
fn test_commit_ops_empty() {
1034
+
let original = CommitOps { ops: vec![] };
1035
+
let encoded = original.encode();
1036
+
let decoded = CommitOps::decode(&encoded).unwrap();
1037
+
assert!(decoded.ops.is_empty());
1038
+
}
1039
+
1040
+
#[test]
1041
+
fn test_account_status_all_variants() {
1042
+
for status in [
1043
+
AccountStatus::Active,
1044
+
AccountStatus::Deactivated,
1045
+
AccountStatus::Suspended,
1046
+
AccountStatus::Deleted,
1047
+
AccountStatus::Takendown,
1048
+
] {
1049
+
let byte = status.as_u8();
1050
+
let decoded = AccountStatus::from_u8(byte).unwrap();
1051
+
assert_eq!(decoded, status);
1052
+
}
1053
+
}
1054
+
1055
+
#[test]
1056
+
fn test_compact_commit_roundtrip() {
1057
+
let ops = vec![
1058
+
CompactOp {
1059
+
action: OpType::Create,
1060
+
collection: "app.bsky.feed.post".to_string(),
1061
+
rkey: "abc123".to_string(),
1062
+
cid: Some(test_cid()),
1063
+
data: vec![0xA1, 0x64, 0x74, 0x65, 0x73, 0x74],
1064
+
},
1065
+
CompactOp {
1066
+
action: OpType::Delete,
1067
+
collection: "app.bsky.feed.like".to_string(),
1068
+
rkey: "def456".to_string(),
1069
+
cid: None,
1070
+
data: vec![],
1071
+
},
1072
+
];
1073
+
1074
+
let encoded = encode_compact_commit(42, "did:plc:abc123", "3jzfcijpj2z2a", &ops);
1075
+
let decoded = decode_compact_event(&encoded).unwrap();
1076
+
1077
+
let CompactEvent::Commit {
1078
+
seq,
1079
+
did,
1080
+
rev,
1081
+
ops: decoded_ops,
1082
+
} = decoded
1083
+
else {
1084
+
panic!("expected Commit event");
1085
+
};
1086
+
1087
+
assert_eq!(seq, 42);
1088
+
assert_eq!(did, "did:plc:abc123");
1089
+
assert_eq!(rev, "3jzfcijpj2z2a");
1090
+
assert_eq!(decoded_ops.len(), 2);
1091
+
assert_eq!(decoded_ops[0].collection, "app.bsky.feed.post");
1092
+
assert_eq!(decoded_ops[0].rkey, "abc123");
1093
+
assert_eq!(decoded_ops[0].cid, Some(test_cid()));
1094
+
assert_eq!(
1095
+
decoded_ops[0].data,
1096
+
vec![0xA1, 0x64, 0x74, 0x65, 0x73, 0x74]
1097
+
);
1098
+
assert_eq!(decoded_ops[1].collection, "app.bsky.feed.like");
1099
+
assert!(decoded_ops[1].cid.is_none());
1100
+
assert!(decoded_ops[1].data.is_empty());
1101
+
}
1102
+
1103
+
#[test]
1104
+
fn test_compact_identity_roundtrip() {
1105
+
let encoded = encode_compact_identity(100, "did:plc:test123", Some("alice.bsky.social"));
1106
+
let decoded = decode_compact_event(&encoded).unwrap();
1107
+
1108
+
let CompactEvent::Identity {
1109
+
seq,
1110
+
time_us,
1111
+
did,
1112
+
handle,
1113
+
} = decoded
1114
+
else {
1115
+
panic!("expected Identity event");
1116
+
};
1117
+
assert_eq!(seq, 100);
1118
+
assert_eq!(time_us, 0); // v1 format has no timestamp
1119
+
assert_eq!(did, "did:plc:test123");
1120
+
assert_eq!(handle.as_deref(), Some("alice.bsky.social"));
1121
+
}
1122
+
1123
+
#[test]
1124
+
fn test_compact_identity_no_handle() {
1125
+
let encoded = encode_compact_identity(101, "did:plc:test456", None);
1126
+
let decoded = decode_compact_event(&encoded).unwrap();
1127
+
1128
+
let CompactEvent::Identity {
1129
+
seq,
1130
+
time_us,
1131
+
did,
1132
+
handle,
1133
+
} = decoded
1134
+
else {
1135
+
panic!("expected Identity event");
1136
+
};
1137
+
assert_eq!(seq, 101);
1138
+
assert_eq!(time_us, 0);
1139
+
assert_eq!(did, "did:plc:test456");
1140
+
assert!(handle.is_none());
1141
+
}
1142
+
1143
+
#[test]
1144
+
fn test_compact_account_roundtrip() {
1145
+
let encoded = encode_compact_account(200, "did:plc:acct1", false, Some("suspended"));
1146
+
let decoded = decode_compact_event(&encoded).unwrap();
1147
+
1148
+
let CompactEvent::Account {
1149
+
seq,
1150
+
time_us,
1151
+
did,
1152
+
active,
1153
+
status,
1154
+
} = decoded
1155
+
else {
1156
+
panic!("expected Account event");
1157
+
};
1158
+
assert_eq!(seq, 200);
1159
+
assert_eq!(time_us, 0);
1160
+
assert_eq!(did, "did:plc:acct1");
1161
+
assert!(!active);
1162
+
assert_eq!(status.as_deref(), Some("suspended"));
1163
+
}
1164
+
1165
+
#[test]
1166
+
fn test_compact_account_no_status() {
1167
+
let encoded = encode_compact_account(201, "did:plc:acct2", true, None);
1168
+
let decoded = decode_compact_event(&encoded).unwrap();
1169
+
1170
+
let CompactEvent::Account {
1171
+
seq,
1172
+
time_us,
1173
+
did,
1174
+
active,
1175
+
status,
1176
+
} = decoded
1177
+
else {
1178
+
panic!("expected Account event");
1179
+
};
1180
+
assert_eq!(seq, 201);
1181
+
assert_eq!(time_us, 0);
1182
+
assert_eq!(did, "did:plc:acct2");
1183
+
assert!(active);
1184
+
assert!(status.is_none());
1185
+
}
1186
+
1187
+
#[test]
1188
+
fn test_compact_commit_empty_ops() {
1189
+
let encoded = encode_compact_commit(1, "did:plc:x", "3jzfcijpj2z2a", &[]);
1190
+
let decoded = decode_compact_event(&encoded).unwrap();
1191
+
1192
+
let CompactEvent::Commit { ops, .. } = decoded else {
1193
+
panic!("expected Commit event");
1194
+
};
1195
+
assert!(ops.is_empty());
1196
+
}
1197
+
1198
+
#[test]
1199
+
fn test_compact_commit_op_v2_roundtrip() {
1200
+
let op = CompactOp {
1201
+
action: OpType::Create,
1202
+
collection: "app.bsky.feed.post".to_string(),
1203
+
rkey: "abc123".to_string(),
1204
+
cid: Some(test_cid()),
1205
+
data: vec![0xA1, 0x64, 0x74, 0x65, 0x73, 0x74],
1206
+
};
1207
+
1208
+
let encoded =
1209
+
encode_compact_commit_op(42, 1725911162329308, "did:plc:abc123", "3jzfcijpj2z2a", &op);
1210
+
let decoded = decode_compact_event(&encoded).unwrap();
1211
+
1212
+
let CompactEvent::CommitOp {
1213
+
seq,
1214
+
time_us,
1215
+
did,
1216
+
rev,
1217
+
op: decoded_op,
1218
+
} = decoded
1219
+
else {
1220
+
panic!("expected CommitOp event");
1221
+
};
1222
+
1223
+
assert_eq!(seq, 42);
1224
+
assert_eq!(time_us, 1725911162329308);
1225
+
assert_eq!(did, "did:plc:abc123");
1226
+
assert_eq!(rev, "3jzfcijpj2z2a");
1227
+
assert_eq!(decoded_op.action, OpType::Create);
1228
+
assert_eq!(decoded_op.collection, "app.bsky.feed.post");
1229
+
assert_eq!(decoded_op.rkey, "abc123");
1230
+
assert_eq!(decoded_op.cid, Some(test_cid()));
1231
+
assert_eq!(decoded_op.data, vec![0xA1, 0x64, 0x74, 0x65, 0x73, 0x74]);
1232
+
}
1233
+
1234
+
#[test]
1235
+
fn test_compact_commit_op_v2_delete() {
1236
+
let op = CompactOp {
1237
+
action: OpType::Delete,
1238
+
collection: "app.bsky.feed.like".to_string(),
1239
+
rkey: "def456".to_string(),
1240
+
cid: None,
1241
+
data: vec![],
1242
+
};
1243
+
1244
+
let encoded =
1245
+
encode_compact_commit_op(43, 1725911162400000, "did:plc:xyz", "3jzfcijpj2z2a", &op);
1246
+
let decoded = decode_compact_event(&encoded).unwrap();
1247
+
1248
+
let CompactEvent::CommitOp { op: decoded_op, .. } = decoded else {
1249
+
panic!("expected CommitOp event");
1250
+
};
1251
+
assert_eq!(decoded_op.action, OpType::Delete);
1252
+
assert!(decoded_op.cid.is_none());
1253
+
assert!(decoded_op.data.is_empty());
1254
+
}
1255
+
1256
+
#[test]
1257
+
fn test_compact_identity_v2_roundtrip() {
1258
+
let encoded = encode_compact_identity_v2(
1259
+
100,
1260
+
1725911162329308,
1261
+
"did:plc:test123",
1262
+
Some("alice.bsky.social"),
1263
+
);
1264
+
let decoded = decode_compact_event(&encoded).unwrap();
1265
+
1266
+
let CompactEvent::Identity {
1267
+
seq,
1268
+
time_us,
1269
+
did,
1270
+
handle,
1271
+
} = decoded
1272
+
else {
1273
+
panic!("expected Identity event");
1274
+
};
1275
+
assert_eq!(seq, 100);
1276
+
assert_eq!(time_us, 1725911162329308);
1277
+
assert_eq!(did, "did:plc:test123");
1278
+
assert_eq!(handle.as_deref(), Some("alice.bsky.social"));
1279
+
}
1280
+
1281
+
#[test]
1282
+
fn test_compact_account_v2_roundtrip() {
1283
+
let encoded = encode_compact_account_v2(
1284
+
200,
1285
+
1725911162329308,
1286
+
"did:plc:acct1",
1287
+
false,
1288
+
Some("suspended"),
1289
+
);
1290
+
let decoded = decode_compact_event(&encoded).unwrap();
1291
+
1292
+
let CompactEvent::Account {
1293
+
seq,
1294
+
time_us,
1295
+
did,
1296
+
active,
1297
+
status,
1298
+
} = decoded
1299
+
else {
1300
+
panic!("expected Account event");
1301
+
};
1302
+
assert_eq!(seq, 200);
1303
+
assert_eq!(time_us, 1725911162329308);
1304
+
assert_eq!(did, "did:plc:acct1");
1305
+
assert!(!active);
1306
+
assert_eq!(status.as_deref(), Some("suspended"));
1307
+
}
1308
+
}
+251
src/storage/keys.rs
+251
src/storage/keys.rs
···
1
+
//! Key encoding and decoding for fjall keyspaces.
2
+
//!
3
+
//! All primary keys use `\x00` null byte separators, which is safe because
4
+
//! DIDs, NSIDs, record keys, and TID revisions are restricted to printable ASCII.
5
+
6
+
use crate::errors::RamjetError;
7
+
8
+
/// Encode a record primary key: `DID \x00 collection \x00 rkey \x00 rev`.
9
+
///
10
+
/// Including the revision in the key creates a versioned history of each record.
11
+
/// The latest version is the last entry when scanning the prefix
12
+
/// `DID \x00 collection \x00 rkey \x00`.
13
+
pub fn encode_record_key(did: &str, collection: &str, rkey: &str, rev: &str) -> Vec<u8> {
14
+
let mut key = Vec::with_capacity(did.len() + collection.len() + rkey.len() + rev.len() + 3);
15
+
key.extend_from_slice(did.as_bytes());
16
+
key.push(0x00);
17
+
key.extend_from_slice(collection.as_bytes());
18
+
key.push(0x00);
19
+
key.extend_from_slice(rkey.as_bytes());
20
+
key.push(0x00);
21
+
key.extend_from_slice(rev.as_bytes());
22
+
key
23
+
}
24
+
25
+
/// Decode a record primary key into `(did, collection, rkey, rev)`.
26
+
pub fn decode_record_key(key: &[u8]) -> Result<(&str, &str, &str, &str), RamjetError> {
27
+
let first_sep =
28
+
key.iter()
29
+
.position(|&b| b == 0x00)
30
+
.ok_or_else(|| RamjetError::KeyEncoding {
31
+
reason: "missing first separator in record key".to_string(),
32
+
})?;
33
+
34
+
let second_sep = key[first_sep + 1..]
35
+
.iter()
36
+
.position(|&b| b == 0x00)
37
+
.map(|pos| first_sep + 1 + pos)
38
+
.ok_or_else(|| RamjetError::KeyEncoding {
39
+
reason: "missing second separator in record key".to_string(),
40
+
})?;
41
+
42
+
let third_sep = key[second_sep + 1..]
43
+
.iter()
44
+
.position(|&b| b == 0x00)
45
+
.map(|pos| second_sep + 1 + pos)
46
+
.ok_or_else(|| RamjetError::KeyEncoding {
47
+
reason: "missing third separator in record key".to_string(),
48
+
})?;
49
+
50
+
let did = std::str::from_utf8(&key[..first_sep]).map_err(|_| RamjetError::KeyEncoding {
51
+
reason: "invalid UTF-8 in DID".to_string(),
52
+
})?;
53
+
let collection = std::str::from_utf8(&key[first_sep + 1..second_sep]).map_err(|_| {
54
+
RamjetError::KeyEncoding {
55
+
reason: "invalid UTF-8 in collection".to_string(),
56
+
}
57
+
})?;
58
+
let rkey = std::str::from_utf8(&key[second_sep + 1..third_sep]).map_err(|_| {
59
+
RamjetError::KeyEncoding {
60
+
reason: "invalid UTF-8 in rkey".to_string(),
61
+
}
62
+
})?;
63
+
let rev = std::str::from_utf8(&key[third_sep + 1..]).map_err(|_| RamjetError::KeyEncoding {
64
+
reason: "invalid UTF-8 in rev".to_string(),
65
+
})?;
66
+
67
+
Ok((did, collection, rkey, rev))
68
+
}
69
+
70
+
/// Encode a record prefix for looking up all versions of a specific record:
71
+
/// `DID \x00 collection \x00 rkey \x00`.
72
+
pub fn encode_record_prefix(did: &str, collection: &str, rkey: &str) -> Vec<u8> {
73
+
let mut key = Vec::with_capacity(did.len() + collection.len() + rkey.len() + 3);
74
+
key.extend_from_slice(did.as_bytes());
75
+
key.push(0x00);
76
+
key.extend_from_slice(collection.as_bytes());
77
+
key.push(0x00);
78
+
key.extend_from_slice(rkey.as_bytes());
79
+
key.push(0x00);
80
+
key
81
+
}
82
+
83
+
/// Encode a collection prefix for prefix scans: `DID \x00 collection \x00`.
84
+
pub fn encode_collection_prefix(did: &str, collection: &str) -> Vec<u8> {
85
+
let mut key = Vec::with_capacity(did.len() + collection.len() + 2);
86
+
key.extend_from_slice(did.as_bytes());
87
+
key.push(0x00);
88
+
key.extend_from_slice(collection.as_bytes());
89
+
key.push(0x00);
90
+
key
91
+
}
92
+
93
+
/// Encode a DID prefix for prefix scans: `DID \x00`.
94
+
pub fn encode_did_prefix(did: &str) -> Vec<u8> {
95
+
let mut key = Vec::with_capacity(did.len() + 1);
96
+
key.extend_from_slice(did.as_bytes());
97
+
key.push(0x00);
98
+
key
99
+
}
100
+
101
+
/// Extract the collection NSID from a record key.
102
+
pub fn extract_collection(key: &[u8]) -> Result<&str, RamjetError> {
103
+
let (_, collection, _, _) = decode_record_key(key)?;
104
+
Ok(collection)
105
+
}
106
+
107
+
/// Extract the rkey from a record key.
108
+
pub fn extract_rkey(key: &[u8]) -> Result<&str, RamjetError> {
109
+
let (_, _, rkey, _) = decode_record_key(key)?;
110
+
Ok(rkey)
111
+
}
112
+
113
+
/// Encode a global event log key: big-endian u64 sequence number.
114
+
pub fn encode_event_key(seq: u64) -> [u8; 8] {
115
+
seq.to_be_bytes()
116
+
}
117
+
118
+
/// Decode a global event log key from big-endian u64 bytes.
119
+
pub fn decode_event_key(key: &[u8]) -> Result<u64, RamjetError> {
120
+
let bytes: [u8; 8] = key.try_into().map_err(|_| RamjetError::KeyEncoding {
121
+
reason: format!("event key must be 8 bytes, got {}", key.len()),
122
+
})?;
123
+
Ok(u64::from_be_bytes(bytes))
124
+
}
125
+
126
+
#[cfg(test)]
127
+
mod tests {
128
+
use super::*;
129
+
130
+
#[test]
131
+
fn test_record_key_roundtrip() {
132
+
let did = "did:plc:abc123";
133
+
let collection = "app.bsky.feed.post";
134
+
let rkey = "3jzfcijpj2z2a";
135
+
let rev = "3jzfcijpj2z2b";
136
+
137
+
let key = encode_record_key(did, collection, rkey, rev);
138
+
let (d, c, r, v) = decode_record_key(&key).unwrap();
139
+
assert_eq!(d, did);
140
+
assert_eq!(c, collection);
141
+
assert_eq!(r, rkey);
142
+
assert_eq!(v, rev);
143
+
}
144
+
145
+
#[test]
146
+
fn test_record_key_special_rkeys() {
147
+
for rkey in &["self", "app", "some-slug", "a-uuid-value"] {
148
+
let key = encode_record_key("did:plc:test", "com.example", rkey, "3jzfcijpj2z2a");
149
+
let (_, _, r, _) = decode_record_key(&key).unwrap();
150
+
assert_eq!(r, *rkey);
151
+
}
152
+
}
153
+
154
+
#[test]
155
+
fn test_collection_prefix() {
156
+
let prefix = encode_collection_prefix("did:plc:abc", "app.bsky.feed.post");
157
+
let key = encode_record_key("did:plc:abc", "app.bsky.feed.post", "rkey1", "rev1");
158
+
assert!(key.starts_with(&prefix));
159
+
}
160
+
161
+
#[test]
162
+
fn test_did_prefix() {
163
+
let prefix = encode_did_prefix("did:plc:abc");
164
+
let key = encode_record_key("did:plc:abc", "app.bsky.feed.post", "rkey1", "rev1");
165
+
assert!(key.starts_with(&prefix));
166
+
}
167
+
168
+
#[test]
169
+
fn test_record_prefix() {
170
+
let prefix = encode_record_prefix("did:plc:abc", "app.bsky.feed.post", "rkey1");
171
+
let key1 = encode_record_key("did:plc:abc", "app.bsky.feed.post", "rkey1", "rev1");
172
+
let key2 = encode_record_key("did:plc:abc", "app.bsky.feed.post", "rkey1", "rev2");
173
+
let key3 = encode_record_key("did:plc:abc", "app.bsky.feed.post", "rkey2", "rev1");
174
+
assert!(key1.starts_with(&prefix));
175
+
assert!(key2.starts_with(&prefix));
176
+
assert!(!key3.starts_with(&prefix));
177
+
}
178
+
179
+
#[test]
180
+
fn test_record_key_version_ordering() {
181
+
// TID revisions sort lexicographically, so later revisions come after earlier ones
182
+
let key1 = encode_record_key(
183
+
"did:plc:abc",
184
+
"app.bsky.feed.post",
185
+
"rkey1",
186
+
"2222222222222",
187
+
);
188
+
let key2 = encode_record_key(
189
+
"did:plc:abc",
190
+
"app.bsky.feed.post",
191
+
"rkey1",
192
+
"3333333333333",
193
+
);
194
+
assert!(key1 < key2);
195
+
}
196
+
197
+
#[test]
198
+
fn test_event_key_roundtrip() {
199
+
let seq = 42u64;
200
+
let key = encode_event_key(seq);
201
+
let decoded = decode_event_key(&key).unwrap();
202
+
assert_eq!(decoded, seq);
203
+
}
204
+
205
+
#[test]
206
+
fn test_event_key_ordering() {
207
+
let k1 = encode_event_key(1);
208
+
let k2 = encode_event_key(2);
209
+
let k3 = encode_event_key(1000);
210
+
assert!(k1 < k2);
211
+
assert!(k2 < k3);
212
+
}
213
+
214
+
#[test]
215
+
fn test_extract_collection() {
216
+
let key = encode_record_key("did:plc:abc", "app.bsky.feed.post", "rkey1", "rev1");
217
+
assert_eq!(extract_collection(&key).unwrap(), "app.bsky.feed.post");
218
+
}
219
+
220
+
#[test]
221
+
fn test_extract_rkey() {
222
+
let key = encode_record_key("did:plc:abc", "app.bsky.feed.post", "rkey1", "rev1");
223
+
assert_eq!(extract_rkey(&key).unwrap(), "rkey1");
224
+
}
225
+
226
+
#[test]
227
+
fn test_decode_record_key_missing_separator() {
228
+
let bad_key = b"no-separators";
229
+
assert!(decode_record_key(bad_key).is_err());
230
+
}
231
+
232
+
#[test]
233
+
fn test_decode_record_key_one_separator() {
234
+
let mut key = Vec::new();
235
+
key.extend_from_slice(b"did:plc:abc");
236
+
key.push(0x00);
237
+
key.extend_from_slice(b"collection-only");
238
+
assert!(decode_record_key(&key).is_err());
239
+
}
240
+
241
+
#[test]
242
+
fn test_decode_record_key_two_separators() {
243
+
let mut key = Vec::new();
244
+
key.extend_from_slice(b"did:plc:abc");
245
+
key.push(0x00);
246
+
key.extend_from_slice(b"collection");
247
+
key.push(0x00);
248
+
key.extend_from_slice(b"rkey");
249
+
assert!(decode_record_key(&key).is_err());
250
+
}
251
+
}
+226
src/storage/mod.rs
+226
src/storage/mod.rs
···
1
+
//! Storage layer wrapping fjall v3 with 8 keyspace partitions.
2
+
//!
3
+
//! The `FjallDb` struct provides access to all keyspaces and helper
4
+
//! methods for common operations like cursor management. Optionally
5
+
//! supports zstd dictionary compression for the events keyspace.
6
+
7
+
pub mod encoding;
8
+
pub mod keys;
9
+
10
+
use std::path::Path;
11
+
use std::sync::atomic::{AtomicU64, Ordering};
12
+
13
+
use crate::errors::RamjetError;
14
+
use crate::types::RepoState;
15
+
16
+
/// Zstd frame magic number (first 4 bytes of any zstd-compressed frame).
17
+
const ZSTD_MAGIC: [u8; 4] = [0x28, 0xB5, 0x2F, 0xFD];
18
+
19
+
/// Maximum decompressed event size (4 MB safety cap).
20
+
const MAX_DECOMPRESSED_SIZE: usize = 4 * 1024 * 1024;
21
+
22
+
/// Central storage abstraction wrapping a fjall database with 8 keyspaces.
23
+
pub struct FjallDb {
24
+
/// The fjall database handle.
25
+
pub db: fjall::Database,
26
+
/// Versioned record store: `DID\x00collection\x00rkey\x00rev → CID + CBOR`.
27
+
/// Empty value = tombstone (deleted record).
28
+
pub records: fjall::Keyspace,
29
+
/// Global replay buffer: `BE u64 seq → compact event`.
30
+
pub events: fjall::Keyspace,
31
+
/// Service metadata: cursor, config, backfill queue.
32
+
pub meta: fjall::Keyspace,
33
+
/// Per-repo sync tracking: `DID → latest rev + status + denied`.
34
+
pub repo_state: fjall::Keyspace,
35
+
/// DID → `[8B BE timestamp][JSON DID document]`.
36
+
pub did_to_doc: fjall::Keyspace,
37
+
/// Handle (lowercase) → DID.
38
+
pub handle_to_did: fjall::Keyspace,
39
+
/// Small blob storage: `CID → blob data`.
40
+
pub blobs: fjall::Keyspace,
41
+
/// Blob metadata: `CID → ref count + size + path`.
42
+
pub blob_meta: fjall::Keyspace,
43
+
/// Optional zstd dictionary bytes for event compression.
44
+
zstd_dict: Option<Vec<u8>>,
45
+
/// Atomic event sequence counter shared between writer and backfill.
46
+
event_seq: AtomicU64,
47
+
}
48
+
49
+
impl FjallDb {
50
+
/// Open or create the database at the given path, optionally loading a
51
+
/// zstd dictionary for event compression.
52
+
pub fn open(path: &Path, zstd_dict_path: Option<&Path>) -> Result<Self, RamjetError> {
53
+
let db = fjall::Database::builder(path).open()?;
54
+
55
+
let ks = |name: &str| -> Result<fjall::Keyspace, RamjetError> {
56
+
Ok(db.keyspace(name, || fjall::KeyspaceCreateOptions::default())?)
57
+
};
58
+
59
+
let meta = ks("meta")?;
60
+
61
+
let zstd_dict = if let Some(dict_path) = zstd_dict_path {
62
+
let bytes = std::fs::read(dict_path).map_err(|e| RamjetError::DictLoad {
63
+
path: dict_path.to_path_buf(),
64
+
source: e,
65
+
})?;
66
+
67
+
// Store a hash of the dictionary in meta so we can detect swaps.
68
+
let dict_hash = simple_hash(&bytes).to_be_bytes();
69
+
let hash_key = b"zstd_dict_id";
70
+
if let Some(stored) = meta.get(hash_key)? {
71
+
let stored_slice: &[u8] = &stored;
72
+
if stored_slice != dict_hash {
73
+
tracing::warn!("zstd dictionary changed since last run");
74
+
}
75
+
}
76
+
meta.insert(hash_key, &dict_hash)?;
77
+
db.persist(fjall::PersistMode::SyncAll)?;
78
+
79
+
tracing::info!(
80
+
dict_size = bytes.len(),
81
+
dict_path = %dict_path.display(),
82
+
"loaded zstd dictionary for event compression"
83
+
);
84
+
Some(bytes)
85
+
} else {
86
+
None
87
+
};
88
+
89
+
// Initialize the atomic event sequence counter from persisted state.
90
+
let initial_seq = match meta.get(b"event_seq")? {
91
+
Some(bytes) => {
92
+
let slice: &[u8] = &bytes;
93
+
let arr: [u8; 8] = slice.try_into().map_err(|_| RamjetError::ValueDecoding {
94
+
reason: "event_seq must be 8 bytes".to_string(),
95
+
})?;
96
+
u64::from_be_bytes(arr)
97
+
}
98
+
None => 0,
99
+
};
100
+
101
+
Ok(Self {
102
+
records: ks("records")?,
103
+
events: ks("events")?,
104
+
repo_state: ks("repo_state")?,
105
+
did_to_doc: ks("did_to_doc")?,
106
+
handle_to_did: ks("handle_to_did")?,
107
+
blobs: ks("blobs")?,
108
+
blob_meta: ks("blob_meta")?,
109
+
meta,
110
+
db,
111
+
zstd_dict,
112
+
event_seq: AtomicU64::new(initial_seq),
113
+
})
114
+
}
115
+
116
+
/// Compress an event using the loaded zstd dictionary.
117
+
/// Returns the raw bytes unchanged if no dictionary is loaded.
118
+
pub fn compress_event(&self, data: &[u8]) -> Vec<u8> {
119
+
let Some(dict) = &self.zstd_dict else {
120
+
return data.to_vec();
121
+
};
122
+
let compressor = zstd::bulk::Compressor::with_dictionary(3, dict);
123
+
match compressor {
124
+
Ok(mut c) => c.compress(data).unwrap_or_else(|_| data.to_vec()),
125
+
Err(_) => data.to_vec(),
126
+
}
127
+
}
128
+
129
+
/// Decompress an event, auto-detecting the format.
130
+
///
131
+
/// - If the first 4 bytes match the zstd magic number, decompress using
132
+
/// the loaded dictionary.
133
+
/// - Otherwise, return the raw bytes as-is (uncompressed compact binary,
134
+
/// legacy JSON, etc.).
135
+
pub fn decompress_event(&self, data: &[u8]) -> Result<Vec<u8>, RamjetError> {
136
+
if data.len() >= 4 && data[..4] == ZSTD_MAGIC {
137
+
let Some(dict) = &self.zstd_dict else {
138
+
return Err(RamjetError::ValueDecoding {
139
+
reason: "zstd-compressed event but no dictionary loaded".to_string(),
140
+
});
141
+
};
142
+
let mut decompressor =
143
+
zstd::bulk::Decompressor::with_dictionary(dict).map_err(|e| {
144
+
RamjetError::ValueDecoding {
145
+
reason: format!("failed to create zstd decompressor: {e}"),
146
+
}
147
+
})?;
148
+
decompressor
149
+
.decompress(data, MAX_DECOMPRESSED_SIZE)
150
+
.map_err(|e| RamjetError::ValueDecoding {
151
+
reason: format!("zstd decompression failed: {e}"),
152
+
})
153
+
} else {
154
+
Ok(data.to_vec())
155
+
}
156
+
}
157
+
158
+
/// Create a new write batch for atomic multi-keyspace writes.
159
+
pub fn batch(&self) -> fjall::OwnedWriteBatch {
160
+
self.db.batch()
161
+
}
162
+
163
+
/// Get the current firehose cursor (last processed sequence number).
164
+
pub fn get_cursor(&self) -> Result<Option<u64>, RamjetError> {
165
+
match self.meta.get(b"cursor")? {
166
+
Some(bytes) => {
167
+
let slice: &[u8] = &bytes;
168
+
let arr: [u8; 8] = slice.try_into().map_err(|_| RamjetError::ValueDecoding {
169
+
reason: "cursor must be 8 bytes".to_string(),
170
+
})?;
171
+
Ok(Some(u64::from_be_bytes(arr)))
172
+
}
173
+
None => Ok(None),
174
+
}
175
+
}
176
+
177
+
/// Get the repo state for a DID.
178
+
pub fn get_repo_state(&self, did: &str) -> Result<Option<RepoState>, RamjetError> {
179
+
match self.repo_state.get(did.as_bytes())? {
180
+
Some(bytes) => {
181
+
let slice: &[u8] = &bytes;
182
+
Ok(Some(RepoState::decode(slice)?))
183
+
}
184
+
None => Ok(None),
185
+
}
186
+
}
187
+
188
+
/// Get the current event sequence number from the atomic counter.
189
+
pub fn current_sequence(&self) -> u64 {
190
+
self.event_seq.load(Ordering::Relaxed)
191
+
}
192
+
193
+
/// Allocate the next event sequence number (thread-safe).
194
+
pub fn next_event_seq(&self) -> u64 {
195
+
self.event_seq.fetch_add(1, Ordering::Relaxed) + 1
196
+
}
197
+
198
+
/// Get the zstd dictionary bytes, if loaded.
199
+
pub fn zstd_dict(&self) -> Option<&[u8]> {
200
+
self.zstd_dict.as_deref()
201
+
}
202
+
203
+
/// Get the oldest retained event sequence number.
204
+
pub fn oldest_event_sequence(&self) -> Result<u64, RamjetError> {
205
+
match self.meta.get(b"oldest_event_seq")? {
206
+
Some(bytes) => {
207
+
let slice: &[u8] = &bytes;
208
+
let arr: [u8; 8] = slice.try_into().map_err(|_| RamjetError::ValueDecoding {
209
+
reason: "oldest_event_seq must be 8 bytes".to_string(),
210
+
})?;
211
+
Ok(u64::from_be_bytes(arr))
212
+
}
213
+
None => Ok(0),
214
+
}
215
+
}
216
+
}
217
+
218
+
/// FNV-1a 64-bit hash — fast, non-cryptographic, good enough for dict identity.
219
+
fn simple_hash(data: &[u8]) -> u64 {
220
+
let mut hash: u64 = 0xcbf29ce484222325;
221
+
for &b in data {
222
+
hash ^= b as u64;
223
+
hash = hash.wrapping_mul(0x100000001b3);
224
+
}
225
+
hash
226
+
}
+172
src/types.rs
+172
src/types.rs
···
1
+
//! Core domain types for the Ramjet service.
2
+
//!
3
+
//! These types represent the data model stored in fjall keyspaces
4
+
//! and exchanged between pipeline components.
5
+
6
+
use std::sync::Arc;
7
+
8
+
use bytes::Bytes;
9
+
10
+
/// Operation type for record changes.
11
+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
12
+
#[repr(u8)]
13
+
pub enum OpType {
14
+
/// A new record was created.
15
+
Create = 0,
16
+
/// An existing record was updated.
17
+
Update = 1,
18
+
/// A record was deleted.
19
+
Delete = 2,
20
+
}
21
+
22
+
impl OpType {
23
+
/// Encode as a single byte.
24
+
pub fn as_u8(self) -> u8 {
25
+
self as u8
26
+
}
27
+
28
+
/// Decode from a single byte.
29
+
pub fn from_u8(value: u8) -> Option<Self> {
30
+
match value {
31
+
0 => Some(Self::Create),
32
+
1 => Some(Self::Update),
33
+
2 => Some(Self::Delete),
34
+
_ => None,
35
+
}
36
+
}
37
+
}
38
+
39
+
/// Account hosting status from `#account` events.
40
+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
41
+
#[repr(u8)]
42
+
pub enum AccountStatus {
43
+
/// Account is active.
44
+
#[default]
45
+
Active = 0,
46
+
/// Account has been deactivated by the user.
47
+
Deactivated = 1,
48
+
/// Account has been suspended by the host.
49
+
Suspended = 2,
50
+
/// Account has been deleted.
51
+
Deleted = 3,
52
+
/// Account has been taken down.
53
+
Takendown = 4,
54
+
}
55
+
56
+
impl AccountStatus {
57
+
/// Encode as a single byte.
58
+
pub fn as_u8(self) -> u8 {
59
+
self as u8
60
+
}
61
+
62
+
/// Decode from a single byte.
63
+
pub fn from_u8(value: u8) -> Option<Self> {
64
+
match value {
65
+
0 => Some(Self::Active),
66
+
1 => Some(Self::Deactivated),
67
+
2 => Some(Self::Suspended),
68
+
3 => Some(Self::Deleted),
69
+
4 => Some(Self::Takendown),
70
+
_ => None,
71
+
}
72
+
}
73
+
74
+
/// Convert to the string representation used in XRPC responses.
75
+
pub fn as_str(&self) -> &'static str {
76
+
match self {
77
+
Self::Active => "active",
78
+
Self::Deactivated => "deactivated",
79
+
Self::Suspended => "suspended",
80
+
Self::Deleted => "deleted",
81
+
Self::Takendown => "takendown",
82
+
}
83
+
}
84
+
}
85
+
86
+
/// Per-repository state stored in the `repo_state` keyspace.
87
+
#[derive(Debug, Clone)]
88
+
pub struct RepoState {
89
+
/// Latest known revision TID (13-character string).
90
+
pub rev: String,
91
+
/// Account hosting status.
92
+
pub status: AccountStatus,
93
+
/// Whether this repository is on the operator deny-list.
94
+
pub denied: bool,
95
+
/// Whether this repository has been backfilled.
96
+
pub backfilled: bool,
97
+
}
98
+
99
+
impl Default for RepoState {
100
+
fn default() -> Self {
101
+
Self {
102
+
rev: String::new(),
103
+
status: AccountStatus::Active,
104
+
denied: false,
105
+
backfilled: false,
106
+
}
107
+
}
108
+
}
109
+
110
+
/// Value stored in the `records` keyspace.
111
+
///
112
+
/// The key includes the revision, so rev is not stored in the value.
113
+
/// An empty value (0 bytes) represents a tombstone (deleted record).
114
+
#[derive(Debug, Clone)]
115
+
pub struct RecordValue {
116
+
/// CID bytes of the record.
117
+
pub cid: Vec<u8>,
118
+
/// Raw DAG-CBOR record data.
119
+
pub data: Vec<u8>,
120
+
}
121
+
122
+
/// A single operation within a commit.
123
+
#[derive(Debug, Clone)]
124
+
pub struct CommitOp {
125
+
/// The type of operation.
126
+
pub op: OpType,
127
+
/// Collection NSID.
128
+
pub collection: String,
129
+
/// Record key.
130
+
pub rkey: String,
131
+
/// CID bytes (None for deletes).
132
+
pub cid: Option<Vec<u8>>,
133
+
}
134
+
135
+
/// All operations from a single commit.
136
+
#[derive(Debug, Clone)]
137
+
pub struct CommitOps {
138
+
/// Operations in this commit.
139
+
pub ops: Vec<CommitOp>,
140
+
}
141
+
142
+
/// Event type classification for fan-out routing.
143
+
#[derive(Debug, Clone)]
144
+
pub enum EventType {
145
+
/// A commit event with record operations.
146
+
Commit {
147
+
/// The primary collection affected by this commit.
148
+
collection: String,
149
+
},
150
+
/// An identity change event.
151
+
Identity,
152
+
/// An account status change event.
153
+
Account,
154
+
/// A sync/state-assertion event.
155
+
Sync,
156
+
}
157
+
158
+
/// An event to be delivered to WebSocket consumers.
159
+
#[derive(Debug, Clone)]
160
+
pub struct FanOutEvent {
161
+
/// Monotonic sequence number.
162
+
pub seq: u64,
163
+
/// The DID that this event pertains to (used for consumer group partitioning).
164
+
pub did: Arc<str>,
165
+
/// Event classification for routing.
166
+
pub event_type: EventType,
167
+
/// Pre-serialized CBOR frame bytes (header + payload).
168
+
pub payload: Bytes,
169
+
}
170
+
171
+
/// Type alias for shared fan-out events.
172
+
pub type SharedFanOutEvent = Arc<FanOutEvent>;