+4

.gitignore

reviewed

··· 17 17 *.db 18 18 *.db-shm 19 19 *.db-wal 20 20 + 21 21 + # local scratch files (content folded into docs/ops-changelog.md) 22 22 + context.md 23 23 + TODO.md

-73

TODO.md

reviewed

··· 1 1 - # zlay: next steps 2 2 - 3 3 - ## context 4 4 - 5 5 - we just shipped inductive proof chain plumbing to zlay (commit `1ecf365` in 6 6 - `@zzstoatzz.io/zlay`). this session was run from the zat repo directory — going 7 7 - forward, work directly from this relay repo instead. 8 8 - 9 9 - ### what landed 10 10 - 11 11 - - **phase 1**: fixed `extractOps`/`checkCommitStructure` in `validator.zig` to 12 12 - read the firehose `path` field instead of nonexistent `collection`/`rkey`. 13 13 - `verify_commit_diff` was dead code before this fix. 14 14 - - **phase 2**: added `since`/`prevData` chain continuity checks and future-rev 15 15 - rejection in `frame_worker.zig` and `subscriber.zig`. log-only + `chain_breaks` 16 16 - metric (no commits dropped). panel added to grafana dashboard. 17 17 - - **phase 3**: conditional upsert (`WHERE rev < EXCLUDED.rev`) on 18 18 - `updateAccountState` in `event_log.zig` to prevent concurrent workers from 19 19 - rolling back rev on same DID. 20 20 - 21 21 - ### where to find more context 22 22 - 23 23 - - plan transcript: `~/.claude/projects/-Users-nate-tangled-sh--zzstoatzz-io-zat/5335a98d-69e2-44d6-9596-5832272df710.jsonl` 24 24 - - memory file: `~/.claude/projects/-Users-nate-tangled-sh--zzstoatzz-io-zat/memory/MEMORY.md` 25 25 - - zlay CLAUDE.md: `@zzstoatzz.io/zlay/CLAUDE.md` (deploy instructions, known issues) 26 26 - - dashboard: `zlay/deploy/zlay-dashboard.json` (in this repo) 27 27 - - grafana: `https://zlay-metrics.waow.tech` 28 28 - 29 29 - ### phase 4 (not yet implemented) 30 30 - 31 31 - desync + resync on chain breaks. collect `chain_breaks` data from phase 2 first 32 32 - to understand frequency and patterns before building the resync machinery. 33 33 - design sketch is in the plan transcript. 34 34 - 35 35 - ## investigate: memory fragmentation 36 36 - 37 37 - **this is the main open item.** pre-existing, unrelated to the chain plumbing work. 38 38 - 39 39 - ### what we observed (3h window, grafana) 40 40 - 41 41 - - malloc arena (claimed from OS) climbing steadily toward ~3.5 GiB 42 42 - - malloc in-use much lower (~500 MB) — the gap is fragmentation 43 43 - - process RSS stairstepping upward to ~4 GiB, never returning pages to OS 44 44 - - memory limit is 8 GiB (`zlay/deploy/zlay-values.yaml:57`) 45 45 - 46 46 - ### what's already been tried 47 47 - 48 48 - `zlay/deploy/zlay-values.yaml` already sets: 49 49 - - `MALLOC_ARENA_MAX=2` (limit glibc arena count) 50 50 - - `MALLOC_TRIM_THRESHOLD_=131072` (return freed pages when free > 128 KB) 51 51 - 52 52 - fragmentation is happening despite these tunings. 53 53 - 54 54 - ### likely cause 55 55 - 56 56 - arena-per-frame pattern: every websocket frame creates an `ArenaAllocator`, 57 57 - decodes CBOR into it, processes, then frees. at ~700 frames/sec with ~2,748 58 58 - concurrent PDS connections, this creates massive allocation churn. glibc ptmalloc 59 59 - is known to be bad at returning pages to the OS under this pattern. 60 60 - 61 61 - ### things to investigate 62 62 - 63 63 - - periodic `malloc_trim(0)` call (e.g. every 10s on a timer thread) to force 64 64 - page return. cheapest fix if it works. 65 65 - - `MALLOC_MMAP_THRESHOLD_` tuning — lower it so more allocations go through 66 66 - mmap (which gets returned to OS on free) instead of sbrk 67 67 - - switch to jemalloc or mimalloc via `LD_PRELOAD` — both handle arena churn 68 68 - better than ptmalloc. mimalloc in particular is good at returning pages. 69 69 - - zig's `std.heap.c_allocator` vs `std.heap.page_allocator` — check what the 70 70 - ArenaAllocator is backed by and whether switching the backing allocator helps 71 71 - - profile allocation sizes: if most arenas are small and similar-sized, a 72 72 - fixed-size pool allocator that reuses arenas across frames would eliminate 73 73 - the churn entirely

-127

context.md

reviewed

··· 1 1 - # context: what's going on here 2 2 - 3 3 - this file is a companion to [TODO.md](TODO.md). the TODO focuses on zlay's internal implementation work — validation plumbing, memory fragmentation, backfill mechanics. this file covers the operational picture from the outside: what the relays are doing in production, how they compare, and what a new operator should know. 4 4 - 5 5 - ## the two relays 6 6 - 7 7 - we run two independent ATProto relays on separate Hetzner nodes: 8 8 - 9 9 - - **relay.waow.tech** — the Go implementation ([indigo](https://github.com/bluesky-social/indigo)), Ashburn VA. this is the "known quantity." it's been running longer, it's the reference codebase, and it's well understood. collectiondir runs as a sidecar with a pebble DB. 10 10 - - **zlay.waow.tech** — the Zig implementation ([zlay](https://tangled.org/zzstoatzz.io/zlay)), Hillsboro OR. this is the experimental one. has an inline RocksDB collection index and uses about half the memory of indigo for the same workload. 11 11 - 12 12 - both relays work the same way architecturally: they connect directly to each PDS on the network via individual `subscribeRepos` WebSocket connections (one per host, ~2,800 concurrent). neither is a "fan-out relay" that re-serves an upstream firehose. `RELAY_UPSTREAM` in zlay (and the equivalent in indigo) is used once at bootstrap to call `listHosts` for PDS discovery — no event data flows through it after that. 13 13 - 14 14 - both serve the full ATProto firehose and `listReposByCollection`. they're deployed on independent k3s clusters and don't depend on each other. 15 15 - 16 16 - ## what we've been measuring 17 17 - 18 18 - we run coverage comparisons periodically (see `docs/coverage-comparison-*.md`) using: 19 19 - 20 20 - - **[pulsar](https://tangled.org/mackuba.eu/pulsar)** — subscribes to multiple relay firehoses at once, counts events and unique DIDs over a 2-minute window. tells us whether the relays are seeing the same traffic. 21 21 - - **coldir-compare** (a bash script at `/tmp/coldir-compare.sh`) — queries `listReposByCollection` across relay, zlay, and bsky.network for every indie NSID from [lexicon garden](https://lexicon.garden). tells us whether the collection directories are complete. 22 22 - - **[tap](https://github.com/bluesky-social/indigo/tree/main/cmd/tap)** — an indigo tool that discovers repos via `listReposByCollection`, backfills from PDS, and subscribes to the firehose. tells us whether the data is correct, not just present. 23 23 - 24 24 - the reference relay for comparison is always **bsky.network** (Bluesky's own relay). we also include third-party relays — see "other public relays" below. 25 25 - 26 26 - ## what the numbers say (as of 2026-03-05) 27 27 - 28 28 - ### firehose: parity 29 29 - 30 30 - relay and zlay are within ~2% of each other on both event count and unique DIDs. third-party relays (firehose.network, atproto.africa) show comparable numbers when included in tests. all relays on the network are seeing the same traffic. 31 31 - 32 32 - ### collection directory: zlay leads on most indie collections 33 33 - 34 34 - zlay's backfill imported DIDs from bsky.network, so it starts with at least what bsky has. on many collections it has 30-60% more DIDs than the indigo relay, and matches or slightly trails bsky.network. 35 35 - 36 36 - the indigo relay's collectiondir trails because: 37 37 - 1. its crawl-based backfill is slower and more fragile (bsky PDS shards rate-limit aggressively, crawl state is in-memory only, pod restart = progress lost) 38 38 - 2. it still carries ghost DIDs (accounts whose PDS is gone) and deactivated accounts that inflate some counts while leaving real gaps in others 39 39 - 40 40 - zlay has its own gaps — 6 long-tail collections where it returns 0 results. these are collections not in lexicon garden's llms.txt and not present on bsky.network, so the backfill never found them. the live firehose will pick them up if anyone creates new records in those collections. 41 41 - 42 42 - ### the one collection where indigo leads 43 43 - 44 44 - `xyz.statusphere.status` — indigo has ~848 DIDs, zlay has 815. statusphere accounts were probably indexed during an early crawl that zlay's backfill didn't cover. not a systemic issue. 45 45 - 46 46 - ## things a new operator should know 47 47 - 48 48 - ### zlay restarts 49 49 - 50 50 - zlay's RSS grows over time due to glibc memory fragmentation (see TODO.md for details). the memory limit is 5 GiB. when it gets OOM-killed, k8s restarts it automatically. this is the main operational concern right now. the relay recovers cleanly — postgres has the cursor, RocksDB has the index — but there's a brief outage window. 51 51 - 52 52 - the TODO has the investigation plan. short version: the arena-per-frame allocation pattern doesn't play well with ptmalloc. jemalloc/mimalloc via `LD_PRELOAD` or periodic `malloc_trim(0)` are the most promising fixes. 53 53 - 54 54 - ### indigo collectiondir is fragile 55 55 - 56 56 - the indigo collectiondir's backfill has several sharp edges: 57 57 - - bsky PDS shards share an IP-based rate limit. >2 concurrent crawls = HTTP 429, which kills the crawl with no retry. 58 58 - - crawl state is in-memory. pod restart = all progress lost. 59 59 - - port-forwards to the collectiondir die after ~80 minutes. crawls continue server-side but you can't monitor or submit new batches. 60 60 - 61 61 - we built a [micro-PDS trick](docs/hacks.md) to work around the rate limit for targeted backfills — stand up a fake PDS serving just the DIDs you need, point `requestCrawl` at it. works great for small gaps, doesn't scale to full-network backfill. 62 62 - 63 63 - submit bsky shards one at a time (`--batch-size 1`). indie PDS hosts can be batched freely. 64 64 - 65 65 - ### zlay's collection index backfill is better 66 66 - 67 67 - zlay imports from bsky.network via `listReposByCollection` — no PDS crawling, no rate limit issues. progress is tracked in postgres (crash-resumable). it's currently at ~13.6M+ DIDs across ~1,000 collections. triggered via admin API, not a crawl job. 68 68 - 69 69 - ### admin auth differs 70 70 - 71 71 - - **indigo relay**: HTTP basic auth (`admin:$RELAY_ADMIN_PASSWORD`) 72 72 - - **indigo collectiondir**: bearer token (`Authorization: Bearer $COLLECTIONDIR_ADMIN_TOKEN`) 73 73 - - **zlay**: bearer token on admin endpoints (port 3001) 74 74 - 75 75 - ### split ports on zlay 76 76 - 77 77 - zlay serves the WebSocket firehose on port 3000 and HTTP (health, metrics, admin, XRPC) on port 3001. indigo serves everything on 2470 (metrics on 2471). this matters for health checks, port-forwards, and ServiceMonitor configuration. 78 78 - 79 79 - ### `listReposByCollection` limits 80 80 - 81 81 - - indigo collectiondir: max limit 2000 82 82 - - zlay: max limit 1000 83 83 - - bsky.network: max limit 1000 84 84 - 85 85 - if you're writing tooling that paginates, use limit ≤ 1000 to work against all three. 86 86 - 87 87 - ### DNS is manual 88 88 - 89 89 - both relay domains are managed in Cloudflare. there's no terraform for DNS — just A records pointing at the server IPs (`just indigo server-ip` / `just zlay server-ip`). 90 90 - 91 91 - ### monitoring 92 92 - 93 93 - each cluster has its own kube-prometheus-stack: 94 94 - - `relay-metrics.waow.tech` — indigo grafana 95 95 - - `zlay-metrics.waow.tech` — zlay grafana 96 96 - 97 97 - dashboards are provisioned via configmaps. the layouts are aligned (events/sec, hosts, memory, threads/goroutines) so you can compare side-by-side. 98 98 - 99 99 - ## other public relays 100 100 - 101 101 - we're not the only independent relay operators. worth knowing about: 102 102 - 103 103 - ### firehose.network (sri) 104 104 - 105 105 - sri ([firehose.network](https://firehose.network), [status](https://status.vayumandala.com)) runs 3 full indigo relays globally with 72-hour replay windows: 106 106 - 107 107 - | region | hostname | 108 108 - |---|---| 109 109 - | North America | `northamerica.firehose.network` | 110 110 - | Europe | `europe.firehose.network` | 111 111 - | Asia | `asia.firehose.network` | 112 112 - 113 113 - plus 6 public Jetstream instances at `*.firehose.stream` (NYC, SFO, London, Frankfurt, Chennai, Canada). mature ops — automated PDS discovery, monitoring, public status page. uptime is consistently 99.9%+. these are firehose-only relays (no `listReposByCollection`). 114 114 - 115 115 - sri also builds [lexicon.store](https://lexicon.store) and [goals.garden](https://goals.garden). 116 116 - 117 117 - ### atproto.africa (BlackSky) 118 118 - 119 119 - firehose-only relay, no collection directory. comparable firehose coverage in our tests. 120 120 - 121 121 - ## what's next 122 122 - 123 123 - the TODO covers zlay-specific implementation work. from the operational side: 124 124 - 125 125 - 1. **fix zlay's memory growth** — this is the most impactful item. the relay works correctly; it just can't stay up indefinitely. 126 126 - 2. **close zlay's collection gaps** — the 6 zero-result collections need their NSIDs added to the backfill source list, or the collections need to appear on bsky.network so the importer picks them up. 127 127 - 3. **keep running coverage comparisons** — the `docs/coverage-comparison-*.md` files track progress over time. run pulsar + coldir-compare periodically to catch regressions.

+72 -2

docs/ops-changelog.md

reviewed

··· 1 1 # ops changelog 2 2 3 3 reverse-chronological log of operational changes, debugging sessions, and 4 4 - deployment decisions. complements `context.md` (the big picture) and zlay's 5 5 - `EXPERIMENTS.md` (memory leak experiments specifically). 4 4 + deployment decisions for both relays (indigo @ relay.waow.tech, zlay @ zlay.waow.tech). 5 5 + 6 6 + --- 7 7 + 8 8 + ## open items 9 9 + 10 10 + ### OutdatedCursor info frame 11 11 + 12 12 + `replayTo` (broadcaster.zig) needs to detect when the requested cursor 13 13 + is older than the replay buffer and send an OutdatedCursor error frame 14 14 + before closing. currently silently starts from oldest available. 15 15 + 16 16 + ### broadcast after flush (architectural) 17 17 + 18 18 + indigo's `flushLog` writes to disk first, then calls `broadcast()` for 19 19 + each event in the buffer. zlay broadcasts immediately after buffering to 20 20 + memory, flush happens async in `flushLoop`. crash between broadcast and 21 21 + flush = consumers received events that replay can't produce. 22 22 + 23 23 + fix: move broadcast into `flushLocked()`, after the disk write succeeds. 24 24 + files: event_log.zig (flushLocked, persist), frame_worker.zig:227, 25 25 + subscriber.zig:633 26 26 + 27 27 + ### getRepo redirect 28 28 + 29 29 + router.zig:68 has no getRepo. should redirect to the PDS hosting the 30 30 + repo (like indigo service.go:153). needs DID → host lookup + new handler. 31 31 + 32 32 + --- 33 33 + 34 34 + ## 2026-03-09 35 35 + 36 36 + ### fix: collection index missing `update` ops (0adf187, deployed) 37 37 + 38 38 + **problem**: `trackCommitOps` in collection_index.zig only indexed `create` ops. 39 39 + `putRecord` to an existing rkey emits an `update` op, so DIDs that only updated 40 40 + records (never created new ones while zlay was watching) were never added to the 41 41 + collection index. indigo indexes both `create` and `update`. 42 42 + 43 43 + **fix**: changed the condition in `trackCommitOps` to match on both actions. 44 44 + `addCollection` is idempotent (RocksDB put is a no-op for existing keys), so 45 45 + no cost to calling on updates. 46 46 + 47 47 + **verification**: evan's `relay-compare` tool showed 3 missing DIDs across 48 48 + `io.atcr.*` collections. after fix + backfill reset, 9/10 collections fully 49 49 + in sync. the remaining gap (`sailor.profile`) is bsky.network missing 3 DIDs 50 50 + that zlay has — zlay has MORE coverage. 51 51 + 52 52 + ### bump zat to v0.2.16 (0adf187) 53 53 + 54 54 + brings in the websocket.zig TCP split-write fix (395d0f4). HTTP response 55 55 + bodies that arrive in separate TCP segments from headers are now fully read. 56 56 + this was the root cause of sporadic HTTP failures in the DID resolver path. 57 57 + 58 58 + ### deploy includes previously pending changes 59 59 + 60 60 + the 0adf187 build also ships: 61 61 + - error frame wire format fix (9a22a23) — drop `t` field from `op=-1` frames 62 62 + - host authority enforcement (ac3f10a) — events from non-authoritative hosts dropped 63 63 + - api conformance batch (5baf376) — was already deployed separately 64 64 + 65 65 + ### reconnect cronjobs updated 66 66 + 67 67 + both indigo and zlay reconnect cronjobs now include phase 2: sync hosts from 68 68 + bsky.network's `listHosts` endpoint in addition to the mary-ext PDS list. 69 69 + 70 70 + ### added relay-compare tool 71 71 + 72 72 + evan jarrett's `relay-compare` Go script added to `scripts/relay-compare/`. 73 73 + compares `io.atcr.*` collection coverage across relays using 74 74 + `listReposByCollection`, with optional `--verify` flag to check records 75 75 + against the PDS. 6 76 7 77 --- 8 78

+59 -23

indigo/deploy/reconnect-cronjob.yaml

reviewed

··· 30 30 - python3 31 31 - -c 32 32 - | 33 33 - import json, urllib.request, time, os, base64, sys 33 33 + import json, urllib.request, urllib.error, time, os, base64, sys 34 34 35 35 PDS_LIST_URL = "https://raw.githubusercontent.com/mary-ext/atproto-scraping/refs/heads/trunk/state.json" 36 36 RELAY_URL = "http://relay.relay.svc.cluster.local:2470" 37 37 PASSWORD = os.environ["RELAY_ADMIN_PASSWORD"] 38 38 AUTH = base64.b64encode(f"admin:{PASSWORD}".encode()).decode() 39 39 + HEADERS = {"Content-Type": "application/json", "Authorization": f"Basic {AUTH}"} 39 40 40 40 - print(f"fetching PDS list from {PDS_LIST_URL}...") 41 41 - with urllib.request.urlopen(PDS_LIST_URL, timeout=30) as resp: 42 42 - data = json.loads(resp.read()) 43 43 - hosts = [url.rstrip("/") for url in data.get("pdses", {}).keys() if url.startswith("https://")] 44 44 - print(f"found {len(hosts)} PDS hosts") 45 45 - 46 46 - ok = errors = 0 47 47 - start = time.time() 48 48 - 49 49 - for i, host in enumerate(hosts): 50 50 - payload = json.dumps({"hostname": host}).encode() 41 41 + def request_crawl(hostname): 42 42 + payload = json.dumps({"hostname": hostname}).encode() 51 43 req = urllib.request.Request( 52 44 f"{RELAY_URL}/admin/pds/requestCrawl", 53 53 - data=payload, 54 54 - headers={"Content-Type": "application/json", "Authorization": f"Basic {AUTH}"}, 55 55 - method="POST", 45 45 + data=payload, headers=HEADERS, method="POST", 56 46 ) 57 47 try: 58 48 with urllib.request.urlopen(req, timeout=10) as resp: 49 49 + return True 50 50 + except (urllib.error.HTTPError, ConnectionError, OSError, urllib.error.URLError): 51 51 + return False 52 52 + 53 53 + def submit_hosts(hosts, label): 54 54 + ok = errors = 0 55 55 + start = time.time() 56 56 + for i, host in enumerate(hosts): 57 57 + if request_crawl(host): 59 58 ok += 1 60 60 - except urllib.error.HTTPError: 61 61 - errors += 1 62 62 - except (ConnectionError, OSError, urllib.error.URLError): 63 63 - errors += 1 59 59 + else: 60 60 + errors += 1 61 61 + if (i + 1) % 500 == 0: 62 62 + print(f" {i + 1}/{len(hosts)} ({ok} ok, {errors} errors, {time.time() - start:.0f}s)") 63 63 + time.sleep(0.05) 64 64 + print(f"{label}: {ok} ok, {errors} errors, {time.time() - start:.0f}s") 64 65 65 65 - if (i + 1) % 500 == 0: 66 66 - print(f" {i + 1}/{len(hosts)} ({ok} ok, {errors} errors, {time.time() - start:.0f}s)") 66 66 + # phase 1: mary-ext scraping list (reconnect existing hosts) 67 67 + print(f"phase 1: fetching PDS list from {PDS_LIST_URL}...") 68 68 + with urllib.request.urlopen(PDS_LIST_URL, timeout=30) as resp: 69 69 + data = json.loads(resp.read()) 70 70 + hosts = [url.rstrip("/") for url in data.get("pdses", {}).keys() if url.startswith("https://")] 71 71 + print(f" {len(hosts)} hosts") 72 72 + submit_hosts(hosts, "phase 1 (mary-ext)") 67 73 68 68 - time.sleep(0.05) 74 74 + # phase 2: discover new hosts from bsky.network 75 75 + print("\nphase 2: pulling hosts from bsky.network...") 76 76 + our_hosts = set() 77 77 + try: 78 78 + resp = urllib.request.urlopen(f"{RELAY_URL}/admin/pds/list", timeout=30) 79 79 + for rec in json.loads(resp.read()): 80 80 + our_hosts.add(rec["Host"]) 81 81 + except Exception as e: 82 82 + print(f" warning: could not fetch our host list: {e}") 83 83 + 84 84 + bsky_hosts = {} 85 85 + cursor = "" 86 86 + while True: 87 87 + url = f"https://bsky.network/xrpc/com.atproto.sync.listHosts?limit=1000" 88 88 + if cursor: 89 89 + url += f"&cursor={cursor}" 90 90 + try: 91 91 + with urllib.request.urlopen(url, timeout=30) as resp: 92 92 + page = json.loads(resp.read()) 93 93 + except Exception: 94 94 + break 95 95 + for h in page.get("hosts", []): 96 96 + status = h.get("status", "unknown") 97 97 + if status in ("active", "idle"): 98 98 + bsky_hosts[h["hostname"]] = status 99 99 + cursor = page.get("cursor", "") 100 100 + if not cursor: 101 101 + break 69 102 70 70 - print(f"done: {ok} ok, {errors} errors, {time.time() - start:.0f}s") 103 103 + missing = [h for h in bsky_hosts if h not in our_hosts] 104 104 + print(f" bsky.network: {len(bsky_hosts)} active/idle hosts, {len(missing)} new") 105 105 + if missing: 106 106 + submit_hosts(missing, "phase 2 (bsky.network)")

+11 -1

indigo/deploy/relay-dashboard.json

reviewed

··· 191 191 "spanNulls": false 192 192 } 193 193 }, 194 194 - "overrides": [] 194 194 + "overrides": [ 195 195 + { 196 196 + "matcher": { "id": "byName", "options": "limit" }, 197 197 + "properties": [ 198 198 + { "id": "custom.lineStyle", "value": { "fill": "dash", "dash": [10, 10] } }, 199 199 + { "id": "custom.fillOpacity", "value": 0 }, 200 200 + { "id": "custom.lineWidth", "value": 3 }, 201 201 + { "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } } 202 202 + ] 203 203 + } 204 204 + ] 195 205 }, 196 206 "targets": [ 197 207 {

+6 -4

indigo/deploy/relay-values.yaml

reviewed

··· 6 6 containers: 7 7 main: 8 8 image: 9 9 - repository: ghcr.io/bluesky-social/indigo 10 10 - tag: relay-bf41e2ee75ab75997bf8cdd92b063c0a96db4aaf 9 9 + repository: atcr.io/zzstoatzz.io/relay 10 10 + tag: outdated-cursor-fix 11 11 env: 12 12 # DATABASE_URL injected from secret via envFrom 13 13 RELAY_PERSIST_DIR: /data 14 14 RELAY_REPLAY_WINDOW: "2h" 15 15 RELAY_IDENT_CACHE_SIZE: "500000" 16 16 LOG_LEVEL: "info" 17 17 - GOMEMLIMIT: "3GiB" 17 17 + GOMEMLIMIT: "7GiB" 18 18 GOMAXPROCS: "4" 19 19 envFrom: 20 20 - secretRef: ··· 37 37 memory: 1Gi 38 38 cpu: 500m 39 39 limits: 40 40 - memory: 4Gi 40 40 + memory: 8Gi 41 41 cpu: "4" 42 42 43 43 defaultPodOptions: 44 44 + imagePullSecrets: 45 45 + - name: atcr-creds 44 46 # hostNetwork recommended for full-network relays (high packet volume). 45 47 # traefik ingress still works: it routes to the pod via the service, 46 48 # which resolves to the host IP when hostNetwork is true.

+12

indigo/justfile

reviewed

··· 244 244 -t atcr.io/zzstoatzz.io/collectiondir:latest "$TMPDIR" 245 245 ATCR_AUTO_AUTH=1 docker push atcr.io/zzstoatzz.io/collectiondir:latest 246 246 247 247 + # build and push relay image from our fork (outdated cursor fix) 248 248 + relay-publish: 249 249 + #!/usr/bin/env bash 250 250 + set -euo pipefail 251 251 + TMPDIR=$(mktemp -d) 252 252 + trap "rm -rf $TMPDIR" EXIT 253 253 + git clone --depth 1 -b fix/outdated-cursor-error-frame https://github.com/zzstoatzz/indigo "$TMPDIR" 254 254 + docker build --platform linux/amd64 \ 255 255 + -f "$TMPDIR/cmd/relay/Dockerfile" \ 256 256 + -t atcr.io/zzstoatzz.io/relay:outdated-cursor-fix "$TMPDIR" 257 257 + ATCR_AUTO_AUTH=1 docker push atcr.io/zzstoatzz.io/relay:outdated-cursor-fix 258 258 + 247 259 # --- scripts --- 248 260 249 261 # reconnect relay to all known PDS hosts (run periodically, e.g. every 4 hours)

+66

scripts/relay-compare/go.mod

reviewed

··· 1 1 + module relay-compare 2 2 + 3 3 + go 1.25.1 4 4 + 5 5 + require github.com/bluesky-social/indigo v0.0.0-20260309144133-7e1240ec4113 6 6 + 7 7 + require ( 8 8 + github.com/beorn7/perks v1.0.1 // indirect 9 9 + github.com/cespare/xxhash/v2 v2.2.0 // indirect 10 10 + github.com/earthboundkid/versioninfo/v2 v2.24.1 // indirect 11 11 + github.com/felixge/httpsnoop v1.0.4 // indirect 12 12 + github.com/go-logr/logr v1.4.1 // indirect 13 13 + github.com/go-logr/stdr v1.2.2 // indirect 14 14 + github.com/gogo/protobuf v1.3.2 // indirect 15 15 + github.com/google/uuid v1.4.0 // indirect 16 16 + github.com/hashicorp/go-cleanhttp v0.5.2 // indirect 17 17 + github.com/hashicorp/go-retryablehttp v0.7.5 // indirect 18 18 + github.com/hashicorp/golang-lru v1.0.2 // indirect 19 19 + github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect 20 20 + github.com/ipfs/bbloom v0.0.4 // indirect 21 21 + github.com/ipfs/go-block-format v0.2.0 // indirect 22 22 + github.com/ipfs/go-cid v0.4.1 // indirect 23 23 + github.com/ipfs/go-datastore v0.6.0 // indirect 24 24 + github.com/ipfs/go-ipfs-blockstore v1.3.1 // indirect 25 25 + github.com/ipfs/go-ipfs-ds-help v1.1.1 // indirect 26 26 + github.com/ipfs/go-ipfs-util v0.0.3 // indirect 27 27 + github.com/ipfs/go-ipld-cbor v0.1.0 // indirect 28 28 + github.com/ipfs/go-ipld-format v0.6.0 // indirect 29 29 + github.com/ipfs/go-log v1.0.5 // indirect 30 30 + github.com/ipfs/go-log/v2 v2.5.1 // indirect 31 31 + github.com/ipfs/go-metrics-interface v0.0.1 // indirect 32 32 + github.com/jbenet/goprocess v0.1.4 // indirect 33 33 + github.com/klauspost/cpuid/v2 v2.2.7 // indirect 34 34 + github.com/mattn/go-isatty v0.0.20 // indirect 35 35 + github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 // indirect 36 36 + github.com/minio/sha256-simd v1.0.1 // indirect 37 37 + github.com/mr-tron/base58 v1.2.0 // indirect 38 38 + github.com/multiformats/go-base32 v0.1.0 // indirect 39 39 + github.com/multiformats/go-base36 v0.2.0 // indirect 40 40 + github.com/multiformats/go-multibase v0.2.0 // indirect 41 41 + github.com/multiformats/go-multihash v0.2.3 // indirect 42 42 + github.com/multiformats/go-varint v0.0.7 // indirect 43 43 + github.com/opentracing/opentracing-go v1.2.0 // indirect 44 44 + github.com/polydawn/refmt v0.89.1-0.20221221234430-40501e09de1f // indirect 45 45 + github.com/prometheus/client_golang v1.17.0 // indirect 46 46 + github.com/prometheus/client_model v0.5.0 // indirect 47 47 + github.com/prometheus/common v0.45.0 // indirect 48 48 + github.com/prometheus/procfs v0.12.0 // indirect 49 49 + github.com/spaolacci/murmur3 v1.1.0 // indirect 50 50 + github.com/whyrusleeping/cbor-gen v0.2.1-0.20241030202151-b7a6831be65e // indirect 51 51 + gitlab.com/yawning/secp256k1-voi v0.0.0-20230925100816-f2616030848b // indirect 52 52 + gitlab.com/yawning/tuplehash v0.0.0-20230713102510-df83abbf9a02 // indirect 53 53 + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.46.1 // indirect 54 54 + go.opentelemetry.io/otel v1.21.0 // indirect 55 55 + go.opentelemetry.io/otel/metric v1.21.0 // indirect 56 56 + go.opentelemetry.io/otel/trace v1.21.0 // indirect 57 57 + go.uber.org/atomic v1.11.0 // indirect 58 58 + go.uber.org/multierr v1.11.0 // indirect 59 59 + go.uber.org/zap v1.26.0 // indirect 60 60 + golang.org/x/crypto v0.21.0 // indirect 61 61 + golang.org/x/sys v0.22.0 // indirect 62 62 + golang.org/x/time v0.3.0 // indirect 63 63 + golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect 64 64 + google.golang.org/protobuf v1.33.0 // indirect 65 65 + lukechampine.com/blake3 v1.2.1 // indirect 66 66 + )

+241

scripts/relay-compare/go.sum

reviewed

··· 1 1 + github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= 2 2 + github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= 3 3 + github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= 4 4 + github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= 5 5 + github.com/bluesky-social/indigo v0.0.0-20260309144133-7e1240ec4113 h1:2cX25bto05aCy5M5E8aB2pP6FiqdV1tUx0c8UE9uS+Y= 6 6 + github.com/bluesky-social/indigo v0.0.0-20260309144133-7e1240ec4113/go.mod h1:VG/LeqLGNI3Ew7lsYixajnZGFfWPv144qbUddh+Oyag= 7 7 + github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= 8 8 + github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= 9 9 + github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU= 10 10 + github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 11 11 + github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 12 12 + github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 13 13 + github.com/earthboundkid/versioninfo/v2 v2.24.1 h1:SJTMHaoUx3GzjjnUO1QzP3ZXK6Ee/nbWyCm58eY3oUg= 14 14 + github.com/earthboundkid/versioninfo/v2 v2.24.1/go.mod h1:VcWEooDEuyUJnMfbdTh0uFN4cfEIg+kHMuWB2CDCLjw= 15 15 + github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= 16 16 + github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= 17 17 + github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= 18 18 + github.com/go-logr/logr v1.4.1 h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ= 19 19 + github.com/go-logr/logr v1.4.1/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= 20 20 + github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= 21 21 + github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= 22 22 + github.com/go-yaml/yaml v2.1.0+incompatible/go.mod h1:w2MrLa16VYP0jy6N7M5kHaCkaLENm+P+Tv+MfurjSw0= 23 23 + github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= 24 24 + github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= 25 25 + github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= 26 26 + github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= 27 27 + github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI= 28 28 + github.com/google/uuid v1.4.0 h1:MtMxsa51/r9yyhkyLsVeVt0B+BGQZzpQiTQ4eHZ8bc4= 29 29 + github.com/google/uuid v1.4.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 30 30 + github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8= 31 31 + github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= 32 32 + github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= 33 33 + github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= 34 34 + github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI= 35 35 + github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ= 36 36 + github.com/hashicorp/go-retryablehttp v0.7.5 h1:bJj+Pj19UZMIweq/iie+1u5YCdGrnxCT9yvm0e+Nd5M= 37 37 + github.com/hashicorp/go-retryablehttp v0.7.5/go.mod h1:Jy/gPYAdjqffZ/yFGCFV2doI5wjtH1ewM9u8iYVjtX8= 38 38 + github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iPY6p1c= 39 39 + github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4= 40 40 + github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= 41 41 + github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM= 42 42 + github.com/ipfs/bbloom v0.0.4 h1:Gi+8EGJ2y5qiD5FbsbpX/TMNcJw8gSqr7eyjHa4Fhvs= 43 43 + github.com/ipfs/bbloom v0.0.4/go.mod h1:cS9YprKXpoZ9lT0n/Mw/a6/aFV6DTjTLYHeA+gyqMG0= 44 44 + github.com/ipfs/go-block-format v0.2.0 h1:ZqrkxBA2ICbDRbK8KJs/u0O3dlp6gmAuuXUJNiW1Ycs= 45 45 + github.com/ipfs/go-block-format v0.2.0/go.mod h1:+jpL11nFx5A/SPpsoBn6Bzkra/zaArfSmsknbPMYgzM= 46 46 + github.com/ipfs/go-cid v0.4.1 h1:A/T3qGvxi4kpKWWcPC/PgbvDA2bjVLO7n4UeVwnbs/s= 47 47 + github.com/ipfs/go-cid v0.4.1/go.mod h1:uQHwDeX4c6CtyrFwdqyhpNcxVewur1M7l7fNU7LKwZk= 48 48 + github.com/ipfs/go-datastore v0.6.0 h1:JKyz+Gvz1QEZw0LsX1IBn+JFCJQH4SJVFtM4uWU0Myk= 49 49 + github.com/ipfs/go-datastore v0.6.0/go.mod h1:rt5M3nNbSO/8q1t4LNkLyUwRs8HupMeN/8O4Vn9YAT8= 50 50 + github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ6cOk= 51 51 + github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= 52 52 + github.com/ipfs/go-ipfs-blockstore v1.3.1 h1:cEI9ci7V0sRNivqaOr0elDsamxXFxJMMMy7PTTDQNsQ= 53 53 + github.com/ipfs/go-ipfs-blockstore v1.3.1/go.mod h1:KgtZyc9fq+P2xJUiCAzbRdhhqJHvsw8u2Dlqy2MyRTE= 54 54 + github.com/ipfs/go-ipfs-ds-help v1.1.1 h1:B5UJOH52IbcfS56+Ul+sv8jnIV10lbjLF5eOO0C66Nw= 55 55 + github.com/ipfs/go-ipfs-ds-help v1.1.1/go.mod h1:75vrVCkSdSFidJscs8n4W+77AtTpCIAdDGAwjitJMIo= 56 56 + github.com/ipfs/go-ipfs-util v0.0.3 h1:2RFdGez6bu2ZlZdI+rWfIdbQb1KudQp3VGwPtdNCmE0= 57 57 + github.com/ipfs/go-ipfs-util v0.0.3/go.mod h1:LHzG1a0Ig4G+iZ26UUOMjHd+lfM84LZCrn17xAKWBvs= 58 58 + github.com/ipfs/go-ipld-cbor v0.1.0 h1:dx0nS0kILVivGhfWuB6dUpMa/LAwElHPw1yOGYopoYs= 59 59 + github.com/ipfs/go-ipld-cbor v0.1.0/go.mod h1:U2aYlmVrJr2wsUBU67K4KgepApSZddGRDWBYR0H4sCk= 60 60 + github.com/ipfs/go-ipld-format v0.6.0 h1:VEJlA2kQ3LqFSIm5Vu6eIlSxD/Ze90xtc4Meten1F5U= 61 61 + github.com/ipfs/go-ipld-format v0.6.0/go.mod h1:g4QVMTn3marU3qXchwjpKPKgJv+zF+OlaKMyhJ4LHPg= 62 62 + github.com/ipfs/go-log v1.0.5 h1:2dOuUCB1Z7uoczMWgAyDck5JLb72zHzrMnGnCNNbvY8= 63 63 + github.com/ipfs/go-log v1.0.5/go.mod h1:j0b8ZoR+7+R99LD9jZ6+AJsrzkPbSXbZfGakb5JPtIo= 64 64 + github.com/ipfs/go-log/v2 v2.1.3/go.mod h1:/8d0SH3Su5Ooc31QlL1WysJhvyOTDCjcCZ9Axpmri6g= 65 65 + github.com/ipfs/go-log/v2 v2.5.1 h1:1XdUzF7048prq4aBjDQQ4SL5RxftpRGdXhNRwKSAlcY= 66 66 + github.com/ipfs/go-log/v2 v2.5.1/go.mod h1:prSpmC1Gpllc9UYWxDiZDreBYw7zp4Iqp1kOLU9U5UI= 67 67 + github.com/ipfs/go-metrics-interface v0.0.1 h1:j+cpbjYvu4R8zbleSs36gvB7jR+wsL2fGD6n0jO4kdg= 68 68 + github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j/b/tL7HTWtJ4VPgWY= 69 69 + github.com/jbenet/go-cienv v0.1.0/go.mod h1:TqNnHUmJgXau0nCzC7kXWeotg3J9W34CUv5Djy1+FlA= 70 70 + github.com/jbenet/goprocess v0.1.4 h1:DRGOFReOMqqDNXwW70QkacFW0YN9QnwLV0Vqk+3oU0o= 71 71 + github.com/jbenet/goprocess v0.1.4/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4= 72 72 + github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo= 73 73 + github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= 74 74 + github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= 75 75 + github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= 76 76 + github.com/klauspost/cpuid/v2 v2.2.7 h1:ZWSB3igEs+d0qvnxR/ZBzXVmxkgt8DdzP6m9pfuVLDM= 77 77 + github.com/klauspost/cpuid/v2 v2.2.7/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws= 78 78 + github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= 79 79 + github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= 80 80 + github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= 81 81 + github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 82 82 + github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 83 83 + github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= 84 84 + github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= 85 85 + github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= 86 86 + github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= 87 87 + github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= 88 88 + github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 h1:jWpvCLoY8Z/e3VKvlsiIGKtc+UG6U5vzxaoagmhXfyg= 89 89 + github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0/go.mod h1:QUyp042oQthUoa9bqDv0ER0wrtXnBruoNd7aNjkbP+k= 90 90 + github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM= 91 91 + github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8= 92 92 + github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o= 93 93 + github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc= 94 94 + github.com/multiformats/go-base32 v0.1.0 h1:pVx9xoSPqEIQG8o+UbAe7DNi51oej1NtK+aGkbLYxPE= 95 95 + github.com/multiformats/go-base32 v0.1.0/go.mod h1:Kj3tFY6zNr+ABYMqeUNeGvkIC/UYgtWibDcT0rExnbI= 96 96 + github.com/multiformats/go-base36 v0.2.0 h1:lFsAbNOGeKtuKozrtBsAkSVhv1p9D0/qedU9rQyccr0= 97 97 + github.com/multiformats/go-base36 v0.2.0/go.mod h1:qvnKE++v+2MWCfePClUEjE78Z7P2a1UV0xHgWc0hkp4= 98 98 + github.com/multiformats/go-multibase v0.2.0 h1:isdYCVLvksgWlMW9OZRYJEa9pZETFivncJHmHnnd87g= 99 99 + github.com/multiformats/go-multibase v0.2.0/go.mod h1:bFBZX4lKCA/2lyOFSAoKH5SS6oPyjtnzK/XTFDPkNuk= 100 100 + github.com/multiformats/go-multihash v0.2.3 h1:7Lyc8XfX/IY2jWb/gI7JP+o7JEq9hOa7BFvVU9RSh+U= 101 101 + github.com/multiformats/go-multihash v0.2.3/go.mod h1:dXgKXCXjBzdscBLk9JkjINiEsCKRVch90MdaGiKsvSM= 102 102 + github.com/multiformats/go-varint v0.0.7 h1:sWSGR+f/eu5ABZA2ZpYKBILXTTs9JWpdEM/nEGOHFS8= 103 103 + github.com/multiformats/go-varint v0.0.7/go.mod h1:r8PUYw/fD/SjBCiKOoDlGF6QawOELpZAu9eioSos/OU= 104 104 + github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs= 105 105 + github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc= 106 106 + github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 107 107 + github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 108 108 + github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 109 109 + github.com/polydawn/refmt v0.89.1-0.20221221234430-40501e09de1f h1:VXTQfuJj9vKR4TCkEuWIckKvdHFeJH/huIFJ9/cXOB0= 110 110 + github.com/polydawn/refmt v0.89.1-0.20221221234430-40501e09de1f/go.mod h1:/zvteZs/GwLtCgZ4BL6CBsk9IKIlexP43ObX9AxTqTw= 111 111 + github.com/prometheus/client_golang v1.17.0 h1:rl2sfwZMtSthVU752MqfjQozy7blglC+1SOtjMAMh+Q= 112 112 + github.com/prometheus/client_golang v1.17.0/go.mod h1:VeL+gMmOAxkS2IqfCq0ZmHSL+LjWfWDUmp1mBz9JgUY= 113 113 + github.com/prometheus/client_model v0.5.0 h1:VQw1hfvPvk3Uv6Qf29VrPF32JB6rtbgI6cYPYQjL0Qw= 114 114 + github.com/prometheus/client_model v0.5.0/go.mod h1:dTiFglRmd66nLR9Pv9f0mZi7B7fk5Pm3gvsjB5tr+kI= 115 115 + github.com/prometheus/common v0.45.0 h1:2BGz0eBc2hdMDLnO/8n0jeB3oPrt2D08CekT0lneoxM= 116 116 + github.com/prometheus/common v0.45.0/go.mod h1:YJmSTw9BoKxJplESWWxlbyttQR4uaEcGyv9MZjVOJsY= 117 117 + github.com/prometheus/procfs v0.12.0 h1:jluTpSng7V9hY0O2R9DzzJHYb2xULk9VTR1V1R/k6Bo= 118 118 + github.com/prometheus/procfs v0.12.0/go.mod h1:pcuDEFsWDnvcgNzo4EEweacyhjeA9Zk3cnaOZAZEfOo= 119 119 + github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= 120 120 + github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= 121 121 + github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= 122 122 + github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= 123 123 + github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= 124 124 + github.com/smartystreets/assertions v1.2.0 h1:42S6lae5dvLc7BrLu/0ugRtcFVjoJNMC/N3yZFZkDFs= 125 125 + github.com/smartystreets/assertions v1.2.0/go.mod h1:tcbTF8ujkAEcZ8TElKY+i30BzYlVhC/LOxJk7iOWnoo= 126 126 + github.com/smartystreets/goconvey v1.7.2 h1:9RBaZCeXEQ3UselpuwUQHltGVXvdwm6cv1hgR6gDIPg= 127 127 + github.com/smartystreets/goconvey v1.7.2/go.mod h1:Vw0tHAZW6lzCRk3xgdin6fKYcG+G3Pg9vgXWeJpQFMM= 128 128 + github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= 129 129 + github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= 130 130 + github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 131 131 + github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 132 132 + github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 133 133 + github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= 134 134 + github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 135 135 + github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA= 136 136 + github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= 137 137 + github.com/urfave/cli v1.22.10/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0= 138 138 + github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0 h1:GDDkbFiaK8jsSDJfjId/PEGEShv6ugrt4kYsC5UIDaQ= 139 139 + github.com/warpfork/go-wish v0.0.0-20220906213052-39a1cc7a02d0/go.mod h1:x6AKhvSSexNrVSrViXSHUEbICjmGXhtgABaHIySUSGw= 140 140 + github.com/whyrusleeping/cbor-gen v0.2.1-0.20241030202151-b7a6831be65e h1:28X54ciEwwUxyHn9yrZfl5ojgF4CBNLWX7LR0rvBkf4= 141 141 + github.com/whyrusleeping/cbor-gen v0.2.1-0.20241030202151-b7a6831be65e/go.mod h1:pM99HXyEbSQHcosHc0iW7YFmwnscr+t9Te4ibko05so= 142 142 + github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 143 143 + github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 144 144 + github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= 145 145 + gitlab.com/yawning/secp256k1-voi v0.0.0-20230925100816-f2616030848b h1:CzigHMRySiX3drau9C6Q5CAbNIApmLdat5jPMqChvDA= 146 146 + gitlab.com/yawning/secp256k1-voi v0.0.0-20230925100816-f2616030848b/go.mod h1:/y/V339mxv2sZmYYR64O07VuCpdNZqCTwO8ZcouTMI8= 147 147 + gitlab.com/yawning/tuplehash v0.0.0-20230713102510-df83abbf9a02 h1:qwDnMxjkyLmAFgcfgTnfJrmYKWhHnci3GjDqcZp1M3Q= 148 148 + gitlab.com/yawning/tuplehash v0.0.0-20230713102510-df83abbf9a02/go.mod h1:JTnUj0mpYiAsuZLmKjTx/ex3AtMowcCgnE7YNyCEP0I= 149 149 + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.46.1 h1:aFJWCqJMNjENlcleuuOkGAPH82y0yULBScfXcIEdS24= 150 150 + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.46.1/go.mod h1:sEGXWArGqc3tVa+ekntsN65DmVbVeW+7lTKTjZF3/Fo= 151 151 + go.opentelemetry.io/otel v1.21.0 h1:hzLeKBZEL7Okw2mGzZ0cc4k/A7Fta0uoPgaJCr8fsFc= 152 152 + go.opentelemetry.io/otel v1.21.0/go.mod h1:QZzNPQPm1zLX4gZK4cMi+71eaorMSGT3A4znnUvNNEo= 153 153 + go.opentelemetry.io/otel/metric v1.21.0 h1:tlYWfeo+Bocx5kLEloTjbcDwBuELRrIFxwdQ36PlJu4= 154 154 + go.opentelemetry.io/otel/metric v1.21.0/go.mod h1:o1p3CA8nNHW8j5yuQLdc1eeqEaPfzug24uvsyIEJRWM= 155 155 + go.opentelemetry.io/otel/trace v1.21.0 h1:WD9i5gzvoUPuXIXH24ZNBudiarZDKuekPqi/E8fpfLc= 156 156 + go.opentelemetry.io/otel/trace v1.21.0/go.mod h1:LGbsEB0f9LGjN+OZaQQ26sohbOmiMR+BaslueVtS/qQ= 157 157 + go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ= 158 158 + go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= 159 159 + go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= 160 160 + go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= 161 161 + go.uber.org/goleak v1.1.11-0.20210813005559-691160354723/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ= 162 162 + go.uber.org/goleak v1.2.0 h1:xqgm/S+aQvhWFTtR0XK3Jvg7z8kGV8P4X14IzwN3Eqk= 163 163 + go.uber.org/goleak v1.2.0/go.mod h1:XJYK+MuIchqpmGmUSAzotztawfKvYLUIgg7guXrwVUo= 164 164 + go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= 165 165 + go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= 166 166 + go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= 167 167 + go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y= 168 168 + go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= 169 169 + go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ= 170 170 + go.uber.org/zap v1.19.1/go.mod h1:j3DNczoxDZroyBnOT1L/Q79cfUMGZxlv/9dzN7SM1rI= 171 171 + go.uber.org/zap v1.26.0 h1:sI7k6L95XOKS281NhVKOFCUNIvv9e0w4BF8N3u+tCRo= 172 172 + go.uber.org/zap v1.26.0/go.mod h1:dtElttAiwGvoJ/vj4IwHBS/gXsEu/pZ50mUIRWuG0so= 173 173 + golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 174 174 + golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 175 175 + golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 176 176 + golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 177 177 + golang.org/x/crypto v0.21.0 h1:X31++rzVUdKhX5sWmSOFZxx8UW/ldWx55cbf08iNAMA= 178 178 + golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs= 179 179 + golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 180 180 + golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= 181 181 + golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 182 182 + golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 183 183 + golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 184 184 + golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 185 185 + golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 186 186 + golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 187 187 + golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 188 188 + golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 189 189 + golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= 190 190 + golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 191 191 + golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 192 192 + golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 193 193 + golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 194 194 + golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 195 195 + golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 196 196 + golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 197 197 + golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 198 198 + golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 199 199 + golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 200 200 + golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 201 201 + golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 202 202 + golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 203 203 + golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI= 204 204 + golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= 205 205 + golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= 206 206 + golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 207 207 + golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 208 208 + golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= 209 209 + golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 210 210 + golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 211 211 + golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 212 212 + golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 213 213 + golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= 214 214 + golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 215 215 + golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 216 216 + golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 217 217 + golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= 218 218 + golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= 219 219 + golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= 220 220 + golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 221 221 + golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 222 222 + golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 223 223 + golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 224 224 + golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 h1:+cNy6SZtPcJQH3LJVLOSmiC7MMxXNOb3PU/VUEz+EhU= 225 225 + golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90= 226 226 + google.golang.org/protobuf v1.33.0 h1:uNO2rsAINq/JlFpSdYEKIZ0uKD/R9cpdv0T+yoGwGmI= 227 227 + google.golang.org/protobuf v1.33.0/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos= 228 228 + gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 229 229 + gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 230 230 + gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= 231 231 + gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= 232 232 + gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= 233 233 + gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 234 234 + gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 235 235 + gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 236 236 + gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 237 237 + gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 238 238 + gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 239 239 + honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg= 240 240 + lukechampine.com/blake3 v1.2.1 h1:YuqqRuaqsGV71BV/nm9xlI0MKUv4QC54jQnBChWbGnI= 241 241 + lukechampine.com/blake3 v1.2.1/go.mod h1:0OFRp7fBtAylGVCO40o87sbupkyIGgbpv1+M1k1LM6k=

+616

scripts/relay-compare/main.go

reviewed

··· 1 1 + // relay-compare compares ATProto relays by querying listReposByCollection 2 2 + // for all io.atcr.* record types and showing what's missing from each relay. 3 3 + // 4 4 + // Usage: 5 5 + // 6 6 + // go run ./cmd/relay-compare https://relay1.us-east.bsky.network https://relay1.us-west.bsky.network 7 7 + package main 8 8 + 9 9 + import ( 10 10 + "context" 11 11 + "encoding/json" 12 12 + "flag" 13 13 + "fmt" 14 14 + "net/http" 15 15 + "net/url" 16 16 + "os" 17 17 + "sort" 18 18 + "strings" 19 19 + "sync" 20 20 + "time" 21 21 + 22 22 + "github.com/bluesky-social/indigo/atproto/identity" 23 23 + "github.com/bluesky-social/indigo/atproto/syntax" 24 24 + "github.com/bluesky-social/indigo/xrpc" 25 25 + ) 26 26 + 27 27 + // ANSI color codes (disabled via --no-color or NO_COLOR env) 28 28 + var ( 29 29 + cRed = "\033[31m" 30 30 + cGreen = "\033[32m" 31 31 + cYellow = "\033[33m" 32 32 + cCyan = "\033[36m" 33 33 + cBold = "\033[1m" 34 34 + cDim = "\033[2m" 35 35 + cReset = "\033[0m" 36 36 + ) 37 37 + 38 38 + func disableColors() { 39 39 + cRed, cGreen, cYellow, cCyan, cBold, cDim, cReset = "", "", "", "", "", "", "" 40 40 + } 41 41 + 42 42 + // All io.atcr.* collections to compare 43 43 + var allCollections = []string{ 44 44 + "io.atcr.manifest", 45 45 + "io.atcr.tag", 46 46 + "io.atcr.sailor.profile", 47 47 + "io.atcr.sailor.star", 48 48 + "io.atcr.repo.page", 49 49 + "io.atcr.hold.captain", 50 50 + "io.atcr.hold.crew", 51 51 + "io.atcr.hold.layer", 52 52 + "io.atcr.hold.stats", 53 53 + "io.atcr.hold.scan", 54 54 + } 55 55 + 56 56 + type summaryRow struct { 57 57 + collection string 58 58 + counts []int 59 59 + status string // "sync", "diff", "error" 60 60 + diffCount int 61 61 + realGaps int // verified: record exists on PDS but relay is missing it 62 62 + ghosts int // verified: record doesn't exist on PDS, relay has stale entry 63 63 + deactivated int // verified: account deactivated/deleted on PDS 64 64 + } 65 65 + 66 66 + // verifyResult holds the PDS verification result for a (DID, collection) pair. 67 67 + type verifyResult struct { 68 68 + exists bool 69 69 + deactivated bool // account deactivated/deleted on PDS 70 70 + err error 71 71 + } 72 72 + 73 73 + // key identifies a (collection, relay-or-DID) pair for result lookups. 74 74 + type key struct{ col, relay string } 75 75 + 76 76 + // diffEntry represents a DID missing from a specific relay for a collection. 77 77 + type diffEntry struct { 78 78 + did string 79 79 + collection string 80 80 + relayIdx int 81 81 + } 82 82 + 83 83 + // XRPC response types for listReposByCollection 84 84 + type listReposByCollectionResult struct { 85 85 + Repos []repoRef `json:"repos"` 86 86 + Cursor string `json:"cursor,omitempty"` 87 87 + } 88 88 + 89 89 + type repoRef struct { 90 90 + DID string `json:"did"` 91 91 + } 92 92 + 93 93 + // XRPC response types for listRecords 94 94 + type listRecordsResult struct { 95 95 + Records []json.RawMessage `json:"records"` 96 96 + Cursor string `json:"cursor,omitempty"` 97 97 + } 98 98 + 99 99 + // Shared identity directory for DID resolution 100 100 + var dir identity.Directory 101 101 + 102 102 + func main() { 103 103 + noColor := flag.Bool("no-color", false, "disable colored output") 104 104 + verify := flag.Bool("verify", false, "verify diffs against PDS to distinguish real gaps from ghost entries") 105 105 + hideGhosts := flag.Bool("hide-ghosts", false, "with --verify, hide ghost and deactivated entries from output") 106 106 + collection := flag.String("collection", "", "compare only this collection") 107 107 + timeout := flag.Duration("timeout", 2*time.Minute, "timeout for all relay queries") 108 108 + flag.Usage = func() { 109 109 + fmt.Fprintf(os.Stderr, "Compare ATProto relays by querying listReposByCollection for io.atcr.* records.\n\n") 110 110 + fmt.Fprintf(os.Stderr, "Usage:\n relay-compare [flags] <relay-url> <relay-url> [relay-url...]\n\n") 111 111 + fmt.Fprintf(os.Stderr, "Example:\n") 112 112 + fmt.Fprintf(os.Stderr, " go run ./cmd/relay-compare https://relay1.us-east.bsky.network https://relay1.us-west.bsky.network\n\n") 113 113 + fmt.Fprintf(os.Stderr, "Flags:\n") 114 114 + flag.PrintDefaults() 115 115 + } 116 116 + flag.Parse() 117 117 + 118 118 + if *noColor || os.Getenv("NO_COLOR") != "" { 119 119 + disableColors() 120 120 + } 121 121 + 122 122 + relays := flag.Args() 123 123 + if len(relays) < 2 { 124 124 + flag.Usage() 125 125 + os.Exit(1) 126 126 + } 127 127 + 128 128 + for i, r := range relays { 129 129 + relays[i] = strings.TrimRight(r, "/") 130 130 + } 131 131 + 132 132 + cols := allCollections 133 133 + if *collection != "" { 134 134 + cols = []string{*collection} 135 135 + } 136 136 + 137 137 + ctx, cancel := context.WithTimeout(context.Background(), *timeout) 138 138 + defer cancel() 139 139 + 140 140 + dir = identity.DefaultDirectory() 141 141 + 142 142 + // Short display names for each relay 143 143 + names := make([]string, len(relays)) 144 144 + maxNameLen := 0 145 145 + for i, r := range relays { 146 146 + names[i] = shortName(r) 147 147 + if len(names[i]) > maxNameLen { 148 148 + maxNameLen = len(names[i]) 149 149 + } 150 150 + } 151 151 + 152 152 + fmt.Printf("%sFetching %d collections from %d relays...%s\n", cDim, len(cols), len(relays), cReset) 153 153 + 154 154 + // Fetch all data in parallel: every (collection, relay) pair concurrently 155 155 + type fetchResult struct { 156 156 + dids map[string]struct{} 157 157 + err error 158 158 + } 159 159 + allResults := make(map[key]fetchResult) 160 160 + var mu sync.Mutex 161 161 + var wg sync.WaitGroup 162 162 + 163 163 + for _, col := range cols { 164 164 + for _, relay := range relays { 165 165 + wg.Add(1) 166 166 + go func(col, relay string) { 167 167 + defer wg.Done() 168 168 + dids, err := fetchAllDIDs(ctx, relay, col) 169 169 + mu.Lock() 170 170 + allResults[key{col, relay}] = fetchResult{dids, err} 171 171 + mu.Unlock() 172 172 + }(col, relay) 173 173 + } 174 174 + } 175 175 + wg.Wait() 176 176 + 177 177 + // Collect all diffs across collections (for optional verification) 178 178 + var allDiffs []diffEntry 179 179 + 180 180 + // First pass: compute diffs per collection 181 181 + type colDiffs struct { 182 182 + hasError bool 183 183 + counts []int 184 184 + // per-relay missing DIDs (sorted) 185 185 + missing [][]string 186 186 + } 187 187 + colResults := make(map[string]*colDiffs) 188 188 + 189 189 + for _, col := range cols { 190 190 + cd := &colDiffs{counts: make([]int, len(relays)), missing: make([][]string, len(relays))} 191 191 + colResults[col] = cd 192 192 + 193 193 + for ri, relay := range relays { 194 194 + r := allResults[key{col, relay}] 195 195 + if r.err != nil { 196 196 + cd.hasError = true 197 197 + } else { 198 198 + cd.counts[ri] = len(r.dids) 199 199 + } 200 200 + } 201 201 + 202 202 + if cd.hasError { 203 203 + continue 204 204 + } 205 205 + 206 206 + // Build union of all DIDs across relays 207 207 + union := make(map[string]struct{}) 208 208 + for _, relay := range relays { 209 209 + for did := range allResults[key{col, relay}].dids { 210 210 + union[did] = struct{}{} 211 211 + } 212 212 + } 213 213 + 214 214 + for ri, relay := range relays { 215 215 + var missing []string 216 216 + for did := range union { 217 217 + if _, ok := allResults[key{col, relay}].dids[did]; !ok { 218 218 + missing = append(missing, did) 219 219 + } 220 220 + } 221 221 + sort.Strings(missing) 222 222 + cd.missing[ri] = missing 223 223 + for _, did := range missing { 224 224 + allDiffs = append(allDiffs, diffEntry{did: did, collection: col, relayIdx: ri}) 225 225 + } 226 226 + } 227 227 + } 228 228 + 229 229 + // Optionally verify diffs against PDS 230 230 + verified := make(map[key]verifyResult) 231 231 + if *verify && len(allDiffs) > 0 { 232 232 + verified = verifyDiffs(ctx, allDiffs) 233 233 + } 234 234 + 235 235 + // Display per-collection diffs and collect summary 236 236 + var summary []summaryRow 237 237 + totalMissing := 0 238 238 + totalRealGaps := 0 239 239 + totalGhosts := 0 240 240 + totalDeactivated := 0 241 241 + 242 242 + for _, col := range cols { 243 243 + fmt.Printf("\n%s%s━━━ %s ━━━%s\n", cBold, cCyan, col, cReset) 244 244 + 245 245 + cd := colResults[col] 246 246 + row := summaryRow{collection: col, counts: cd.counts} 247 247 + 248 248 + if cd.hasError { 249 249 + for ri, relay := range relays { 250 250 + r := allResults[key{col, relay}] 251 251 + if r.err != nil { 252 252 + fmt.Printf(" %-*s %s%serror%s: %v\n", maxNameLen, names[ri], cBold, cRed, cReset, r.err) 253 253 + } else { 254 254 + fmt.Printf(" %-*s %s%d%s DIDs\n", maxNameLen, names[ri], cBold, len(r.dids), cReset) 255 255 + } 256 256 + } 257 257 + row.status = "error" 258 258 + summary = append(summary, row) 259 259 + continue 260 260 + } 261 261 + 262 262 + // Show counts per relay 263 263 + for ri := range relays { 264 264 + fmt.Printf(" %-*s %s%d%s DIDs\n", maxNameLen, names[ri], cBold, cd.counts[ri], cReset) 265 265 + } 266 266 + 267 267 + // Show missing DIDs per relay 268 268 + inSync := true 269 269 + for ri := range relays { 270 270 + missing := cd.missing[ri] 271 271 + if len(missing) == 0 { 272 272 + continue 273 273 + } 274 274 + 275 275 + inSync = false 276 276 + totalMissing += len(missing) 277 277 + row.diffCount += len(missing) 278 278 + 279 279 + fmt.Printf("\n %sMissing from %s (%d):%s\n", cRed, names[ri], len(missing), cReset) 280 280 + for _, did := range missing { 281 281 + suffix := "" 282 282 + skip := false 283 283 + if *verify { 284 284 + vr, ok := verified[key{col, did}] 285 285 + if !ok { 286 286 + suffix = fmt.Sprintf(" %s(verify: unknown)%s", cDim, cReset) 287 287 + } else if vr.err != nil { 288 288 + suffix = fmt.Sprintf(" %s(verify: %s)%s", cDim, vr.err, cReset) 289 289 + } else if vr.deactivated { 290 290 + suffix = fmt.Sprintf(" %s← deactivated%s", cDim, cReset) 291 291 + row.deactivated++ 292 292 + totalDeactivated++ 293 293 + skip = *hideGhosts 294 294 + } else if vr.exists { 295 295 + suffix = fmt.Sprintf(" %s← real gap%s", cRed, cReset) 296 296 + row.realGaps++ 297 297 + totalRealGaps++ 298 298 + } else { 299 299 + suffix = fmt.Sprintf(" %s← ghost (not on PDS)%s", cDim, cReset) 300 300 + row.ghosts++ 301 301 + totalGhosts++ 302 302 + skip = *hideGhosts 303 303 + } 304 304 + } 305 305 + if !skip { 306 306 + fmt.Printf(" %s- %s%s%s\n", cRed, did, cReset, suffix) 307 307 + } 308 308 + } 309 309 + } 310 310 + 311 311 + // When verifying, ghost/deactivated-only diffs are considered in sync 312 312 + if !inSync && *verify && row.realGaps == 0 { 313 313 + inSync = true 314 314 + } 315 315 + 316 316 + if inSync { 317 317 + notes := "" 318 318 + if !*hideGhosts { 319 319 + notes = formatSyncNotes(row.ghosts, row.deactivated) 320 320 + } 321 321 + if notes != "" { 322 322 + fmt.Printf(" %s✓ in sync%s %s(%s)%s\n", cGreen, cReset, cDim, notes, cReset) 323 323 + } else { 324 324 + fmt.Printf(" %s✓ in sync%s\n", cGreen, cReset) 325 325 + } 326 326 + row.status = "sync" 327 327 + } else { 328 328 + row.status = "diff" 329 329 + } 330 330 + summary = append(summary, row) 331 331 + } 332 332 + 333 333 + // Summary table 334 334 + printSummary(summary, names, maxNameLen, totalMissing, *verify, *hideGhosts, totalRealGaps, totalGhosts, totalDeactivated) 335 335 + } 336 336 + 337 337 + func printSummary(rows []summaryRow, names []string, maxNameLen, totalMissing int, showVerify, hideGhosts bool, totalRealGaps, totalGhosts, totalDeactivated int) { 338 338 + fmt.Printf("\n%s%s━━━ Summary ━━━%s\n\n", cBold, cCyan, cReset) 339 339 + 340 340 + // Build short labels (A, B, C, ...) for compact columns 341 341 + labels := make([]string, len(names)) 342 342 + for i, name := range names { 343 343 + labels[i] = string(rune('A' + i)) 344 344 + fmt.Printf(" %s%s%s: %s\n", cBold, labels[i], cReset, name) 345 345 + } 346 346 + fmt.Println() 347 347 + 348 348 + colW := len("Collection") 349 349 + for _, row := range rows { 350 350 + if len(row.collection) > colW { 351 351 + colW = len(row.collection) 352 352 + } 353 353 + } 354 354 + relayW := 6 355 355 + 356 356 + // Header 357 357 + fmt.Printf(" %-*s", colW, "Collection") 358 358 + for _, label := range labels { 359 359 + fmt.Printf(" %*s", relayW, label) 360 360 + } 361 361 + fmt.Printf(" Status\n") 362 362 + 363 363 + // Separator 364 364 + fmt.Printf(" %s", strings.Repeat("─", colW)) 365 365 + for range labels { 366 366 + fmt.Printf(" %s", strings.Repeat("─", relayW)) 367 367 + } 368 368 + fmt.Printf(" %s\n", strings.Repeat("─", 14)) 369 369 + 370 370 + // Data rows 371 371 + for _, row := range rows { 372 372 + fmt.Printf(" %-*s", colW, row.collection) 373 373 + for _, c := range row.counts { 374 374 + switch row.status { 375 375 + case "error": 376 376 + fmt.Printf(" %*s", relayW, fmt.Sprintf("%s—%s", cDim, cReset)) 377 377 + default: 378 378 + fmt.Printf(" %*d", relayW, c) 379 379 + } 380 380 + } 381 381 + switch row.status { 382 382 + case "sync": 383 383 + notes := "" 384 384 + if !hideGhosts { 385 385 + notes = formatSyncNotes(row.ghosts, row.deactivated) 386 386 + } 387 387 + if notes != "" { 388 388 + fmt.Printf(" %s✓ in sync%s %s(%s)%s", cGreen, cReset, cDim, notes, cReset) 389 389 + } else { 390 390 + fmt.Printf(" %s✓ in sync%s", cGreen, cReset) 391 391 + } 392 392 + case "diff": 393 393 + if showVerify { 394 394 + if hideGhosts { 395 395 + fmt.Printf(" %s≠ %d missing%s", cYellow, row.realGaps, cReset) 396 396 + } else { 397 397 + notes := formatSyncNotes(row.ghosts, row.deactivated) 398 398 + if notes != "" { 399 399 + notes = ", " + notes 400 400 + } 401 401 + fmt.Printf(" %s≠ %d missing%s %s(%d real%s)%s", 402 402 + cYellow, row.realGaps, cReset, cDim, row.realGaps, notes, cReset) 403 403 + } 404 404 + } else { 405 405 + fmt.Printf(" %s≠ %d missing%s", cYellow, row.diffCount, cReset) 406 406 + } 407 407 + case "error": 408 408 + fmt.Printf(" %s✗ error%s", cRed, cReset) 409 409 + } 410 410 + fmt.Println() 411 411 + } 412 412 + 413 413 + // Footer 414 414 + fmt.Println() 415 415 + if totalMissing > 0 { 416 416 + if showVerify && totalRealGaps == 0 { 417 417 + if hideGhosts { 418 418 + fmt.Printf("%s✓ All relays in sync%s\n", cGreen, cReset) 419 419 + } else { 420 420 + notes := formatSyncNotes(totalGhosts, totalDeactivated) 421 421 + fmt.Printf("%s✓ All relays in sync%s %s(%s)%s\n", cGreen, cReset, cDim, notes, cReset) 422 422 + } 423 423 + } else { 424 424 + if showVerify { 425 425 + fmt.Printf("%s%d real gaps across relays%s", cYellow, totalRealGaps, cReset) 426 426 + if !hideGhosts { 427 427 + notes := formatSyncNotes(totalGhosts, totalDeactivated) 428 428 + if notes != "" { 429 429 + fmt.Printf(" %s(%s)%s", cDim, notes, cReset) 430 430 + } 431 431 + } 432 432 + fmt.Println() 433 433 + } else { 434 434 + fmt.Printf("%s%d total missing DID-collection pairs across relays%s\n", cYellow, totalMissing, cReset) 435 435 + } 436 436 + } 437 437 + } else { 438 438 + fmt.Printf("%s✓ All relays fully in sync%s\n", cGreen, cReset) 439 439 + } 440 440 + } 441 441 + 442 442 + // formatSyncNotes builds a parenthetical like "2 ghost, 1 deactivated" for sync status. 443 443 + // Returns empty string if both counts are zero. 444 444 + func formatSyncNotes(ghosts, deactivated int) string { 445 445 + var parts []string 446 446 + if ghosts > 0 { 447 447 + parts = append(parts, fmt.Sprintf("%d ghost", ghosts)) 448 448 + } 449 449 + if deactivated > 0 { 450 450 + parts = append(parts, fmt.Sprintf("%d deactivated", deactivated)) 451 451 + } 452 452 + return strings.Join(parts, ", ") 453 453 + } 454 454 + 455 455 + // verifyDiffs resolves each diff DID to its PDS and checks if records actually exist. 456 456 + func verifyDiffs(ctx context.Context, diffs []diffEntry) map[key]verifyResult { 457 457 + // Collect unique (DID, collection) pairs to verify 458 458 + type didCol struct{ did, col string } 459 459 + unique := make(map[didCol]struct{}) 460 460 + for _, d := range diffs { 461 461 + unique[didCol{d.did, d.collection}] = struct{}{} 462 462 + } 463 463 + 464 464 + // Resolve unique DIDs to PDS endpoints (deduplicate across collections) 465 465 + uniqueDIDs := make(map[string]struct{}) 466 466 + for dc := range unique { 467 467 + uniqueDIDs[dc.did] = struct{}{} 468 468 + } 469 469 + 470 470 + fmt.Printf("\n%sVerifying %d DID-collection pairs (%d unique DIDs)...%s\n", cDim, len(unique), len(uniqueDIDs), cReset) 471 471 + 472 472 + pdsEndpoints := make(map[string]string) // DID → PDS URL 473 473 + pdsErrors := make(map[string]error) // DID → resolution error 474 474 + var mu sync.Mutex 475 475 + var wg sync.WaitGroup 476 476 + sem := make(chan struct{}, 10) // concurrency limit 477 477 + 478 478 + for did := range uniqueDIDs { 479 479 + wg.Add(1) 480 480 + go func(did string) { 481 481 + defer wg.Done() 482 482 + sem <- struct{}{} 483 483 + defer func() { <-sem }() 484 484 + 485 485 + pds, err := resolveDIDToPDS(ctx, did) 486 486 + mu.Lock() 487 487 + if err != nil { 488 488 + pdsErrors[did] = err 489 489 + } else { 490 490 + pdsEndpoints[did] = pds 491 491 + } 492 492 + mu.Unlock() 493 493 + }(did) 494 494 + } 495 495 + wg.Wait() 496 496 + 497 497 + // Check each (DID, collection) pair against the resolved PDS 498 498 + results := make(map[key]verifyResult) 499 499 + 500 500 + for dc := range unique { 501 501 + wg.Add(1) 502 502 + go func(dc didCol) { 503 503 + defer wg.Done() 504 504 + sem <- struct{}{} 505 505 + defer func() { <-sem }() 506 506 + 507 507 + k := key{dc.col, dc.did} 508 508 + 509 509 + // Check if DID resolution failed — could mean account is deactivated/tombstoned 510 510 + if err, ok := pdsErrors[dc.did]; ok { 511 511 + errStr := err.Error() 512 512 + if strings.Contains(errStr, "no PDS endpoint") || 513 513 + strings.Contains(errStr, "not found") { 514 514 + mu.Lock() 515 515 + results[k] = verifyResult{deactivated: true} 516 516 + mu.Unlock() 517 517 + } else { 518 518 + mu.Lock() 519 519 + results[k] = verifyResult{err: fmt.Errorf("DID resolution failed: %w", err)} 520 520 + mu.Unlock() 521 521 + } 522 522 + return 523 523 + } 524 524 + 525 525 + pds := pdsEndpoints[dc.did] 526 526 + client := &xrpc.Client{Host: pds, Client: http.DefaultClient} 527 527 + var listResult listRecordsResult 528 528 + err := client.LexDo(ctx, "GET", "", "com.atproto.repo.listRecords", map[string]any{ 529 529 + "repo": dc.did, 530 530 + "collection": dc.col, 531 531 + "limit": 1, 532 532 + }, nil, &listResult) 533 533 + mu.Lock() 534 534 + if err != nil { 535 535 + errStr := err.Error() 536 536 + if strings.Contains(errStr, "Could not find repo") || 537 537 + strings.Contains(errStr, "RepoDeactivated") || 538 538 + strings.Contains(errStr, "RepoTakendown") || 539 539 + strings.Contains(errStr, "RepoSuspended") { 540 540 + results[k] = verifyResult{deactivated: true} 541 541 + } else { 542 542 + results[k] = verifyResult{err: err} 543 543 + } 544 544 + } else { 545 545 + results[k] = verifyResult{exists: len(listResult.Records) > 0} 546 546 + } 547 547 + mu.Unlock() 548 548 + }(dc) 549 549 + } 550 550 + wg.Wait() 551 551 + 552 552 + return results 553 553 + } 554 554 + 555 555 + // resolveDIDToPDS resolves a DID to its PDS endpoint using the shared identity directory. 556 556 + func resolveDIDToPDS(ctx context.Context, did string) (string, error) { 557 557 + didParsed, err := syntax.ParseDID(did) 558 558 + if err != nil { 559 559 + return "", fmt.Errorf("invalid DID: %w", err) 560 560 + } 561 561 + 562 562 + ident, err := dir.LookupDID(ctx, didParsed) 563 563 + if err != nil { 564 564 + return "", fmt.Errorf("failed to resolve DID: %w", err) 565 565 + } 566 566 + 567 567 + pdsEndpoint := ident.PDSEndpoint() 568 568 + if pdsEndpoint == "" { 569 569 + return "", fmt.Errorf("no PDS endpoint found for DID") 570 570 + } 571 571 + 572 572 + return pdsEndpoint, nil 573 573 + } 574 574 + 575 575 + // fetchAllDIDs paginates through listReposByCollection to collect all DIDs. 576 576 + func fetchAllDIDs(ctx context.Context, relay, collection string) (map[string]struct{}, error) { 577 577 + client := &xrpc.Client{Host: relay, Client: http.DefaultClient} 578 578 + dids := make(map[string]struct{}) 579 579 + var cursor string 580 580 + 581 581 + for { 582 582 + params := map[string]any{ 583 583 + "collection": collection, 584 584 + "limit": 1000, 585 585 + } 586 586 + if cursor != "" { 587 587 + params["cursor"] = cursor 588 588 + } 589 589 + 590 590 + var result listReposByCollectionResult 591 591 + err := client.LexDo(ctx, "GET", "", "com.atproto.sync.listReposByCollection", params, nil, &result) 592 592 + if err != nil { 593 593 + return dids, fmt.Errorf("listReposByCollection failed: %w", err) 594 594 + } 595 595 + 596 596 + for _, repo := range result.Repos { 597 597 + dids[repo.DID] = struct{}{} 598 598 + } 599 599 + 600 600 + if result.Cursor == "" { 601 601 + break 602 602 + } 603 603 + cursor = result.Cursor 604 604 + } 605 605 + 606 606 + return dids, nil 607 607 + } 608 608 + 609 609 + // shortName extracts the hostname from a relay URL for display. 610 610 + func shortName(relayURL string) string { 611 611 + u, err := url.Parse(relayURL) 612 612 + if err != nil { 613 613 + return relayURL 614 614 + } 615 615 + return u.Hostname() 616 616 + }

scripts/relay-compare/relay-compare

reviewed

This is a binary file and will not be displayed.

+91 -29

zlay/deploy/zlay-reconnect-cronjob.yaml

reviewed

··· 25 25 - python3 26 26 - -c 27 27 - | 28 28 - import json, urllib.request, time, sys 28 28 + import json, urllib.request, urllib.error, socket, time, sys 29 29 30 30 PDS_LIST_URL = "https://raw.githubusercontent.com/mary-ext/atproto-scraping/refs/heads/trunk/state.json" 31 31 - ZLAY_URL = "http://zlay.zlay.svc.cluster.local:3000" 31 31 + ZLAY_HOST = "zlay.zlay.svc.cluster.local" 32 32 + ZLAY_PORT = 3000 32 33 33 33 - print(f"fetching PDS list from {PDS_LIST_URL}...") 34 34 - with urllib.request.urlopen(PDS_LIST_URL, timeout=30) as resp: 35 35 - data = json.loads(resp.read()) 36 36 - hosts = [url.rstrip("/") for url in data.get("pdses", {}).keys() if url.startswith("https://")] 37 37 - print(f"found {len(hosts)} PDS hosts") 38 38 - 39 39 - ok = errors = 0 40 40 - start = time.time() 34 34 + def request_crawl(hostname): 35 35 + """POST requestCrawl using raw socket (single write). 41 36 42 42 - for i, host in enumerate(hosts): 43 43 - # strip scheme — zlay's requestCrawl expects bare hostname 44 44 - hostname = host.replace("https://", "").replace("http://", "") 45 45 - payload = json.dumps({"hostname": hostname}).encode() 46 46 - req = urllib.request.Request( 47 47 - f"{ZLAY_URL}/xrpc/com.atproto.sync.requestCrawl", 48 48 - data=payload, 49 49 - headers={"Content-Type": "application/json"}, 50 50 - method="POST", 51 51 - ) 37 37 + zlay's HTTP parser reads headers+body in one buffer read. 38 38 + urllib sends them as separate writes, losing the body. 39 39 + """ 40 40 + body = json.dumps({"hostname": hostname}, separators=(",", ":")).encode() 41 41 + request = ( 42 42 + f"POST /xrpc/com.atproto.sync.requestCrawl HTTP/1.1\r\n" 43 43 + f"Host: {ZLAY_HOST}\r\n" 44 44 + f"Content-Type: application/json\r\n" 45 45 + f"Content-Length: {len(body)}\r\n" 46 46 + f"Connection: close\r\n" 47 47 + f"\r\n" 48 48 + ).encode() + body 52 49 try: 53 53 - with urllib.request.urlopen(req, timeout=10) as resp: 50 50 + sock = socket.create_connection((ZLAY_HOST, ZLAY_PORT), timeout=10) 51 51 + sock.sendall(request) 52 52 + resp = b"" 53 53 + while True: 54 54 + chunk = sock.recv(4096) 55 55 + if not chunk: 56 56 + break 57 57 + resp += chunk 58 58 + sock.close() 59 59 + status = int(resp.split(b"\r\n", 1)[0].split(b" ", 2)[1]) 60 60 + return status == 200 61 61 + except (ConnectionError, OSError, socket.timeout): 62 62 + return False 63 63 + 64 64 + def submit_hosts(hosts, label): 65 65 + ok = errors = 0 66 66 + start = time.time() 67 67 + for i, host in enumerate(hosts): 68 68 + if request_crawl(host): 54 69 ok += 1 55 55 - except urllib.error.HTTPError: 56 56 - errors += 1 57 57 - except (ConnectionError, OSError, urllib.error.URLError): 58 58 - errors += 1 70 70 + else: 71 71 + errors += 1 72 72 + if (i + 1) % 500 == 0: 73 73 + print(f" {i + 1}/{len(hosts)} ({ok} ok, {errors} errors, {time.time() - start:.0f}s)") 74 74 + time.sleep(0.05) 75 75 + print(f"{label}: {ok} ok, {errors} errors, {time.time() - start:.0f}s") 59 76 60 60 - if (i + 1) % 500 == 0: 61 61 - print(f" {i + 1}/{len(hosts)} ({ok} ok, {errors} errors, {time.time() - start:.0f}s)") 77 77 + # phase 1: mary-ext scraping list (reconnect existing hosts) 78 78 + print(f"phase 1: fetching PDS list from {PDS_LIST_URL}...") 79 79 + with urllib.request.urlopen(PDS_LIST_URL, timeout=30) as resp: 80 80 + data = json.loads(resp.read()) 81 81 + pds_urls = [url.rstrip("/") for url in data.get("pdses", {}).keys() if url.startswith("https://")] 82 82 + hosts = [url.replace("https://", "").replace("http://", "") for url in pds_urls] 83 83 + print(f" {len(hosts)} hosts") 84 84 + submit_hosts(hosts, "phase 1 (mary-ext)") 62 85 63 63 - time.sleep(0.05) 86 86 + # phase 2: discover new hosts from bsky.network 87 87 + print("\nphase 2: pulling hosts from bsky.network...") 88 88 + our_hosts = set() 89 89 + cursor = "" 90 90 + while True: 91 91 + url = f"https://zlay.waow.tech/xrpc/com.atproto.sync.listHosts?limit=1000" 92 92 + if cursor: 93 93 + url += f"&cursor={cursor}" 94 94 + try: 95 95 + with urllib.request.urlopen(url, timeout=30) as resp: 96 96 + page = json.loads(resp.read()) 97 97 + except Exception: 98 98 + break 99 99 + for h in page.get("hosts", []): 100 100 + our_hosts.add(h["hostname"]) 101 101 + cursor = page.get("cursor", "") 102 102 + if not cursor: 103 103 + break 64 104 65 65 - print(f"done: {ok} ok, {errors} errors, {time.time() - start:.0f}s") 105 105 + bsky_hosts = {} 106 106 + cursor = "" 107 107 + while True: 108 108 + url = f"https://bsky.network/xrpc/com.atproto.sync.listHosts?limit=1000" 109 109 + if cursor: 110 110 + url += f"&cursor={cursor}" 111 111 + try: 112 112 + with urllib.request.urlopen(url, timeout=30) as resp: 113 113 + page = json.loads(resp.read()) 114 114 + except Exception: 115 115 + break 116 116 + for h in page.get("hosts", []): 117 117 + status = h.get("status", "unknown") 118 118 + if status in ("active", "idle"): 119 119 + bsky_hosts[h["hostname"]] = status 120 120 + cursor = page.get("cursor", "") 121 121 + if not cursor: 122 122 + break 123 123 + 124 124 + missing = [h for h in bsky_hosts if h not in our_hosts] 125 125 + print(f" bsky.network: {len(bsky_hosts)} active/idle hosts, {len(missing)} new") 126 126 + if missing: 127 127 + submit_hosts(missing, "phase 2 (bsky.network)")