atproto user agency toolkit for individuals and groups
8
fork

Configure Feed

Select the types of activity you want to include in your feed.

Add local lexicon index and update README

Aggregate lexicon NSIDs from replicated record paths into a
searchable index. Three public endpoints (search, list, stats),
incremental updates on sync/firehose, full rebuild on startup,
and a Lit UI component with prefix search.

README updated with lexicon index, PLC log archiving, rotation
key management, and kadDHT client mode.

+397 -4
+29 -2
README.md
··· 119 119 120 120 Challenge-response protocol: `StorageChallenge` → `StorageChallengeResponse` → `StorageChallengeResult`. Deterministic generation from epoch + DIDs + nonce. Transport-agnostic with libp2p primary and HTTP fallback. 121 121 122 + ### Lexicon index 123 + 124 + Every record path stored during sync includes a lexicon NSID (the collection portion of `collection/rkey`). P2PDS aggregates these into a local lexicon index — a catalog of every lexicon encountered across all replicated repos. 125 + 126 + - **Automatic population** — updated incrementally after every full sync and firehose event, rebuilt from scratch on startup 127 + - **Public API** — three unauthenticated endpoints for querying: 128 + - `GET /xrpc/org.p2pds.lexicon.search?q=app.bsky&limit=50` — prefix search 129 + - `GET /xrpc/org.p2pds.lexicon.list?limit=100` — all NSIDs by record count 130 + - `GET /xrpc/org.p2pds.lexicon.stats` — aggregate stats (unique NSIDs, total records) 131 + - **UI** — searchable table in the web interface showing NSID, record count, repo count, first/last seen dates 132 + 133 + This is the foundation for distributed lexicon discovery — nodes can query each other's indexes to find who stores what. 134 + 122 135 ### Storage 123 136 124 137 All persistent state in a single SQLite database (`pds.db`): ··· 126 139 - **IPFS blocks/datastore** — SQLite-backed, no filesystem churn 127 140 - **Replication state** — sync progress, peer info, block/blob tracking, firehose cursor 128 141 - **Challenge history** — proof-of-storage results and peer reliability scores 142 + - **Lexicon index** — aggregated NSID usage across all replicated repos 143 + - **PLC mirror** — archived PLC operation logs for tracked DIDs 129 144 - **Node identity** — DID + handle, established on first OAuth login 130 145 146 + ### PLC log archiving 147 + 148 + P2PDS mirrors PLC operation logs for all tracked `did:plc` DIDs. The PLC directory is the root of trust for DID resolution — if it goes down or loses data, DID documents become unresolvable. By archiving PLC logs locally, each node maintains an independent backup of the identity layer. 149 + 150 + - **Automatic** — logs are fetched on first sync and refreshed every 6 hours 151 + - **Cross-node sharing** — public endpoint `GET /xrpc/org.p2pds.plc.getLog?did=...` lets nodes fetch PLC logs from each other, not just from the central PLC directory 152 + - **Per-DID status** — the UI shows PLC archive status (op count, last fetch, tombstone state) for each tracked DID 153 + - **Validation** — operation chains are validated on fetch 154 + 131 155 ### Policy engine 132 156 133 157 Every replication relationship is backed by a policy object with lifecycle state, consent tracking, and merge rules. Policies are created automatically from offer negotiation or manually via config/UI. ··· 166 190 167 191 **Handle survival:** If you used a custom domain handle (verified via DNS), it survives migration — the DNS record still points to your DID. If you used a `.bsky.social` handle, it's gone — that subdomain is controlled by Bluesky. 168 192 193 + **Rotation key management in p2pds:** P2PDS includes a UI for adding rotation keys to your PLC document. The flow: request a PLC operation token (triggers an email from your PDS), enter the token, and submit your public key. This uses the standard `com.atproto.identity.signPlcOperation` API. You can also view your current rotation keys from the web interface. 194 + 169 195 **The honest reality for most Bluesky users today:** Bluesky has not yet shipped user-facing rotation key management. Most users don't independently hold a rotation key. If Bluesky's PDS infrastructure disappeared tomorrow, those users would have their data (thanks to p2pds) but could not recover their identity. This is an upstream gap in the atproto ecosystem, not something p2pds can solve — but it's important to understand. The data preservation is still valuable: your posts, social graph, and media survive, even if reattaching them to the same DID requires key management tooling that doesn't exist yet. 170 196 171 197 ## Stack ··· 173 199 - **Runtime**: Node.js, TypeScript (ES2022, strict) 174 200 - **HTTP**: Hono 175 201 - **Database**: better-sqlite3 176 - - **IPFS**: Helia with minimal libp2p (TCP + noise + yamux + autoNAT) 202 + - **IPFS**: Helia with minimal libp2p (TCP + noise + yamux + autoNAT + kadDHT client mode) 177 203 - **UI**: Lit web components, esbuild-bundled 178 204 - **Identity**: AT Protocol DIDs via PLC directory 179 205 - **Auth**: OAuth (primary) or legacy JWT (fallback) ··· 210 236 ipfs.ts IpfsService (Helia wrapper, SQLite-backed) 211 237 build-ui.ts esbuild bundler for Lit UI 212 238 ui/ Lit web components (app shell, cards, state) 213 - replication/ Sync, verification, challenges, offers 239 + replication/ Sync, verification, challenges, offers, lexicon index 214 240 policy/ Policy engine types, engine, presets 241 + identity/ PLC mirror, rotation key management 215 242 oauth/ OAuth client, routes, PdsClient 216 243 xrpc/ XRPC endpoint handlers 217 244 middleware/ Auth, rate limiting, body limits
+20
src/index.ts
··· 187 187 }), 188 188 ); 189 189 190 + // Lexicon index endpoints (unauthenticated, rate-limited) 191 + const lexiconRL = rateLimitMiddleware(rateLimiter, { 192 + pool: "lexicon", 193 + rule: { maxRequests: 300, windowMs: w }, 194 + }); 195 + app.use("/xrpc/org.p2pds.lexicon.search", lexiconRL); 196 + app.use("/xrpc/org.p2pds.lexicon.list", lexiconRL); 197 + app.use("/xrpc/org.p2pds.lexicon.stats", lexiconRL); 198 + 190 199 // App endpoints 191 200 const appRL = rateLimitMiddleware(rateLimiter, { 192 201 pool: "app", ··· 701 710 // Public PLC log endpoint (unauthenticated — PLC logs are public data) 702 711 app.get("/xrpc/org.p2pds.plc.getLog", (c) => 703 712 app_routes.getPlcLogPublic(c, db), 713 + ); 714 + 715 + // Lexicon index endpoints (unauthenticated) 716 + app.get("/xrpc/org.p2pds.lexicon.search", (c) => 717 + app_routes.searchLexicons(c, replicationManager), 718 + ); 719 + app.get("/xrpc/org.p2pds.lexicon.list", (c) => 720 + app_routes.listLexicons(c, replicationManager), 721 + ); 722 + app.get("/xrpc/org.p2pds.lexicon.stats", (c) => 723 + app_routes.getLexiconStats(c, replicationManager), 704 724 ); 705 725 706 726 app.post("/xrpc/org.p2pds.replication.revokeOffer", requireAuth, async (c) => {
+41
src/replication/replication-manager.ts
··· 59 59 missingBlocks?: number; 60 60 } 61 61 62 + /** Extract unique NSIDs from record paths (collection/rkey format). */ 63 + function extractNsids(paths: string[]): string[] { 64 + const nsids = new Set<string>(); 65 + for (const p of paths) { 66 + const slash = p.indexOf("/"); 67 + if (slash > 0) nsids.add(p.slice(0, slash)); 68 + } 69 + return [...nsids]; 70 + } 71 + 62 72 /** How old cached peer info can be before re-fetching (1 hour). */ 63 73 const PEER_INFO_TTL_MS = 60 * 60 * 1000; 64 74 ··· 1215 1225 ); 1216 1226 this.syncStorage.clearRecordPaths(did); 1217 1227 this.syncStorage.trackRecordPaths(did, recordPaths); 1228 + // Update lexicon index with NSIDs from these paths 1229 + const nsids = extractNsids(recordPaths); 1230 + if (nsids.length > 0) { 1231 + this.syncStorage.updateLexiconIndex(nsids); 1232 + } 1218 1233 } catch { 1219 1234 // Non-fatal: path extraction is best-effort 1220 1235 } ··· 1513 1528 console.error("[replication] Initial PLC mirror fetch error:", err); 1514 1529 }); 1515 1530 1531 + // Build lexicon index from existing record paths 1532 + try { 1533 + this.syncStorage.rebuildLexiconIndex(); 1534 + } catch (err) { 1535 + console.error("[replication] Lexicon index rebuild error:", err); 1536 + } 1537 + 1516 1538 // Run verification once on startup, then on a timer 1517 1539 this.runVerification().catch((err) => { 1518 1540 console.error("Initial verification error:", err); ··· 1792 1814 .map((op) => op.path); 1793 1815 if (createdPaths.length > 0) { 1794 1816 this.syncStorage.trackRecordPaths(did, createdPaths); 1817 + // Update lexicon index with NSIDs from created paths 1818 + const nsids = extractNsids(createdPaths); 1819 + if (nsids.length > 0) { 1820 + this.syncStorage.updateLexiconIndex(nsids); 1821 + } 1795 1822 } 1796 1823 if (deletedPaths.length > 0) { 1797 1824 this.syncStorage.removeRecordPaths(did, deletedPaths); ··· 2114 2141 */ 2115 2142 getSyncStates(): SyncState[] { 2116 2143 return this.syncStorage.getAllStates(); 2144 + } 2145 + 2146 + /** 2147 + * Query the lexicon index with optional prefix filter. 2148 + */ 2149 + getLexiconIndex(prefix?: string, limit?: number) { 2150 + return this.syncStorage.getLexiconIndex(prefix, limit); 2151 + } 2152 + 2153 + /** 2154 + * Get aggregate lexicon stats. 2155 + */ 2156 + getLexiconStats() { 2157 + return this.syncStorage.getLexiconStats(); 2117 2158 } 2118 2159 2119 2160 /**
+133
src/replication/sync-storage.ts
··· 112 112 ); 113 113 `); 114 114 115 + // Lexicon index table: aggregates NSID usage across all replicated repos. 116 + this.db.exec(` 117 + CREATE TABLE IF NOT EXISTS lexicon_index ( 118 + nsid TEXT PRIMARY KEY, 119 + first_seen_at TEXT NOT NULL, 120 + last_seen_at TEXT NOT NULL, 121 + record_count INTEGER NOT NULL DEFAULT 0, 122 + repo_count INTEGER NOT NULL DEFAULT 0 123 + ); 124 + `); 125 + 115 126 // Offered DIDs table: tracks DIDs we've offered to replicate 116 127 // but don't yet have mutual consent for. 117 128 this.db.exec(` ··· 1152 1163 return row?.root_cid ?? null; 1153 1164 } 1154 1165 1166 + // ============================================ 1167 + // Lexicon index 1168 + // ============================================ 1169 + 1170 + /** 1171 + * Upsert NSIDs into the lexicon index. 1172 + * Updates last_seen_at and recomputes record_count/repo_count from source data. 1173 + */ 1174 + updateLexiconIndex(nsids: string[]): void { 1175 + if (nsids.length === 0) return; 1176 + const now = new Date().toISOString(); 1177 + const upsert = this.db.prepare( 1178 + `INSERT INTO lexicon_index (nsid, first_seen_at, last_seen_at, record_count, repo_count) 1179 + VALUES (?, ?, ?, 0, 0) 1180 + ON CONFLICT(nsid) DO UPDATE SET last_seen_at = excluded.last_seen_at`, 1181 + ); 1182 + const countRecords = this.db.prepare( 1183 + `SELECT COUNT(*) as cnt FROM replication_record_paths 1184 + WHERE record_path LIKE ? || '/%'`, 1185 + ); 1186 + const countRepos = this.db.prepare( 1187 + `SELECT COUNT(DISTINCT did) as cnt FROM replication_record_paths 1188 + WHERE record_path LIKE ? || '/%'`, 1189 + ); 1190 + const updateCounts = this.db.prepare( 1191 + `UPDATE lexicon_index SET record_count = ?, repo_count = ? WHERE nsid = ?`, 1192 + ); 1193 + 1194 + const batch = this.db.transaction((items: string[]) => { 1195 + for (const nsid of items) { 1196 + upsert.run(nsid, now, now); 1197 + const rc = countRecords.get(nsid) as { cnt: number }; 1198 + const rp = countRepos.get(nsid) as { cnt: number }; 1199 + updateCounts.run(rc.cnt, rp.cnt, nsid); 1200 + } 1201 + }); 1202 + batch(nsids); 1203 + } 1204 + 1205 + /** 1206 + * Full rebuild of the lexicon index from replication_record_paths. 1207 + */ 1208 + rebuildLexiconIndex(): void { 1209 + const now = new Date().toISOString(); 1210 + this.db.exec("DELETE FROM lexicon_index"); 1211 + 1212 + const rows = this.db.prepare( 1213 + `SELECT 1214 + SUBSTR(record_path, 1, INSTR(record_path, '/') - 1) as nsid, 1215 + COUNT(*) as record_count, 1216 + COUNT(DISTINCT did) as repo_count, 1217 + MIN(?) as first_seen_at, 1218 + MAX(?) as last_seen_at 1219 + FROM replication_record_paths 1220 + WHERE INSTR(record_path, '/') > 0 1221 + GROUP BY SUBSTR(record_path, 1, INSTR(record_path, '/') - 1)`, 1222 + ).all(now, now) as Array<{ 1223 + nsid: string; 1224 + record_count: number; 1225 + repo_count: number; 1226 + first_seen_at: string; 1227 + last_seen_at: string; 1228 + }>; 1229 + 1230 + if (rows.length === 0) return; 1231 + 1232 + const insert = this.db.prepare( 1233 + `INSERT INTO lexicon_index (nsid, first_seen_at, last_seen_at, record_count, repo_count) 1234 + VALUES (?, ?, ?, ?, ?)`, 1235 + ); 1236 + const batch = this.db.transaction(() => { 1237 + for (const r of rows) { 1238 + insert.run(r.nsid, r.first_seen_at, r.last_seen_at, r.record_count, r.repo_count); 1239 + } 1240 + }); 1241 + batch(); 1242 + } 1243 + 1244 + /** 1245 + * Query the lexicon index with optional NSID prefix filter. 1246 + */ 1247 + getLexiconIndex(prefix?: string, limit: number = 100): Array<{ 1248 + nsid: string; 1249 + firstSeenAt: string; 1250 + lastSeenAt: string; 1251 + recordCount: number; 1252 + repoCount: number; 1253 + }> { 1254 + const query = prefix 1255 + ? `SELECT * FROM lexicon_index WHERE nsid LIKE ? || '%' ORDER BY record_count DESC LIMIT ?` 1256 + : `SELECT * FROM lexicon_index ORDER BY record_count DESC LIMIT ?`; 1257 + const params = prefix ? [prefix, limit] : [limit]; 1258 + const rows = this.db.prepare(query).all(...params) as Array<{ 1259 + nsid: string; 1260 + first_seen_at: string; 1261 + last_seen_at: string; 1262 + record_count: number; 1263 + repo_count: number; 1264 + }>; 1265 + return rows.map((r) => ({ 1266 + nsid: r.nsid, 1267 + firstSeenAt: r.first_seen_at, 1268 + lastSeenAt: r.last_seen_at, 1269 + recordCount: r.record_count, 1270 + repoCount: r.repo_count, 1271 + })); 1272 + } 1273 + 1274 + /** 1275 + * Get aggregate lexicon stats. 1276 + */ 1277 + getLexiconStats(): { uniqueNsids: number; totalRecords: number } { 1278 + const row = this.db.prepare( 1279 + `SELECT COUNT(*) as unique_nsids, COALESCE(SUM(record_count), 0) as total_records FROM lexicon_index`, 1280 + ).get() as { unique_nsids: number; total_records: number }; 1281 + return { 1282 + uniqueNsids: row.unique_nsids, 1283 + totalRecords: row.total_records, 1284 + }; 1285 + } 1286 + 1155 1287 /** 1156 1288 * Delete all data from all replication tables in a single transaction. 1157 1289 * Used during full disconnect to wipe the node clean. ··· 1169 1301 this.db.prepare("DELETE FROM sync_history").run(); 1170 1302 this.db.prepare("DELETE FROM firehose_cursor").run(); 1171 1303 this.db.prepare("DELETE FROM plc_mirror").run(); 1304 + this.db.prepare("DELETE FROM lexicon_index").run(); 1172 1305 }); 1173 1306 purge(); 1174 1307 }
+1
src/ui/app.ts
··· 21 21 import "./components/verification-card"; 22 22 import "./components/confirm-dialog"; 23 23 import "./components/recovery-key-dialog"; 24 + import "./components/lexicon-index";
+2
src/ui/components/app-shell.ts
··· 178 178 179 179 <p2p-verification .data=${this.overview}></p2p-verification> 180 180 181 + <p2p-lexicon-index></p2p-lexicon-index> 182 + 181 183 <p2p-confirm-dialog></p2p-confirm-dialog> 182 184 `; 183 185 }
+121
src/ui/components/lexicon-index.ts
··· 1 + import { LitElement, html } from "lit"; 2 + import { customElement, state } from "lit/decorators.js"; 3 + 4 + interface LexiconEntry { 5 + nsid: string; 6 + firstSeenAt: string; 7 + lastSeenAt: string; 8 + recordCount: number; 9 + repoCount: number; 10 + } 11 + 12 + @customElement("p2p-lexicon-index") 13 + export class LexiconIndex extends LitElement { 14 + createRenderRoot() { return this; } 15 + 16 + @state() lexicons: LexiconEntry[] = []; 17 + @state() query = ""; 18 + @state() uniqueNsids = 0; 19 + @state() loading = false; 20 + 21 + private debounceTimer: ReturnType<typeof setTimeout> | null = null; 22 + 23 + connectedCallback() { 24 + super.connectedCallback(); 25 + this.fetchData(); 26 + } 27 + 28 + private async fetchData() { 29 + this.loading = true; 30 + try { 31 + const endpoint = this.query 32 + ? `/xrpc/org.p2pds.lexicon.search?q=${encodeURIComponent(this.query)}&limit=50` 33 + : `/xrpc/org.p2pds.lexicon.list?limit=100`; 34 + const res = await fetch(endpoint); 35 + if (res.ok) { 36 + const data = await res.json(); 37 + this.lexicons = data.lexicons ?? []; 38 + } 39 + 40 + const statsRes = await fetch("/xrpc/org.p2pds.lexicon.stats"); 41 + if (statsRes.ok) { 42 + const stats = await statsRes.json(); 43 + this.uniqueNsids = stats.uniqueNsids ?? 0; 44 + } 45 + } catch { 46 + // Fetch failed 47 + } 48 + this.loading = false; 49 + } 50 + 51 + private handleInput(e: Event) { 52 + const input = e.target as HTMLInputElement; 53 + this.query = input.value; 54 + if (this.debounceTimer) clearTimeout(this.debounceTimer); 55 + this.debounceTimer = setTimeout(() => this.fetchData(), 300); 56 + } 57 + 58 + private fmtDate(iso: string): string { 59 + if (!iso) return "-"; 60 + const d = new Date(iso); 61 + return d.toLocaleDateString(undefined, { month: "short", day: "numeric" }); 62 + } 63 + 64 + render() { 65 + return html` 66 + <div class="card"> 67 + <details> 68 + <summary> 69 + Lexicons 70 + <span class="count-badge">${this.uniqueNsids} unique</span> 71 + </summary> 72 + <div style="margin-bottom: 0.4rem;"> 73 + <input 74 + type="text" 75 + placeholder="Filter by prefix (e.g. app.bsky)" 76 + .value=${this.query} 77 + @input=${this.handleInput} 78 + style="width: 100%; padding: 0.3rem 0.5rem; font-family: inherit; font-size: 0.8rem; border: 1px solid var(--input-border); border-radius: 3px; background: var(--input-bg); color: var(--fg); outline: none; box-sizing: border-box;" 79 + /> 80 + </div> 81 + ${this.loading 82 + ? html`<p class="loading">Loading...</p>` 83 + : this.lexicons.length === 0 84 + ? html`<p style="font-size: 0.8rem; color: var(--muted);">No lexicons found${this.query ? ` matching "${this.query}"` : ""}.</p>` 85 + : html` 86 + <div class="scroll-container"> 87 + <table> 88 + <thead> 89 + <tr> 90 + <th>NSID</th> 91 + <th>Records</th> 92 + <th>Repos</th> 93 + <th>First Seen</th> 94 + <th>Last Seen</th> 95 + </tr> 96 + </thead> 97 + <tbody> 98 + ${this.lexicons.map((lex) => html` 99 + <tr> 100 + <td>${lex.nsid}</td> 101 + <td>${lex.recordCount.toLocaleString()}</td> 102 + <td>${lex.repoCount}</td> 103 + <td>${this.fmtDate(lex.firstSeenAt)}</td> 104 + <td>${this.fmtDate(lex.lastSeenAt)}</td> 105 + </tr> 106 + `)} 107 + </tbody> 108 + </table> 109 + </div> 110 + `} 111 + </details> 112 + </div> 113 + `; 114 + } 115 + } 116 + 117 + declare global { 118 + interface HTMLElementTagNameMap { 119 + "p2p-lexicon-index": LexiconIndex; 120 + } 121 + }
+1 -1
src/ui/components/recovery-key-dialog.ts
··· 1 1 import { LitElement, html, nothing } from "lit"; 2 2 import { customElement, state } from "lit/decorators.js"; 3 3 import { generateMnemonic, mnemonicToSeedSync } from "@scure/bip39"; 4 - import { wordlist } from "@scure/bip39/wordlists/english"; 4 + import { wordlist } from "@scure/bip39/wordlists/english.js"; 5 5 import { HDKey } from "@scure/bip32"; 6 6 import { secp256k1 } from "@noble/curves/secp256k1"; 7 7 import { base58btc } from "multiformats/bases/base58";
+2 -1
src/ui/styles/theme.ts
··· 222 222 .account-selected { 223 223 display: flex; align-items: center; gap: 0.4rem; padding: 0.35rem 0.5rem; 224 224 background: var(--selected-bg); border: 1px solid var(--selected-border); border-radius: 4px; 225 - height: 100%; box-sizing: border-box; position: relative; 225 + box-sizing: border-box; position: relative; 226 226 } 227 + .add-did-row .account-selected { height: 100%; } 227 228 .account-selected-avatar { width: 22px; height: 22px; border-radius: 50%; flex-shrink: 0; } 228 229 .account-selected-placeholder { 229 230 width: 22px; height: 22px; border-radius: 50%; background: var(--acct-placeholder-bg);
+47
src/xrpc/app.ts
··· 1062 1062 } 1063 1063 1064 1064 // ============================================ 1065 + // Lexicon index 1066 + // ============================================ 1067 + 1068 + export function searchLexicons( 1069 + c: Context<AppEnv>, 1070 + replicationManager: ReplicationManager | undefined, 1071 + ): Response { 1072 + if (!replicationManager) { 1073 + return c.json({ error: "ReplicationNotEnabled", message: "Replication is not enabled" }, 400); 1074 + } 1075 + 1076 + const q = c.req.query("q") || ""; 1077 + const limitStr = c.req.query("limit"); 1078 + const limit = limitStr ? Math.min(Math.max(parseInt(limitStr, 10) || 50, 1), 200) : 50; 1079 + 1080 + const results = replicationManager.getLexiconIndex(q || undefined, limit); 1081 + return c.json({ lexicons: results, query: q }); 1082 + } 1083 + 1084 + export function listLexicons( 1085 + c: Context<AppEnv>, 1086 + replicationManager: ReplicationManager | undefined, 1087 + ): Response { 1088 + if (!replicationManager) { 1089 + return c.json({ error: "ReplicationNotEnabled", message: "Replication is not enabled" }, 400); 1090 + } 1091 + 1092 + const limitStr = c.req.query("limit"); 1093 + const limit = limitStr ? Math.min(Math.max(parseInt(limitStr, 10) || 100, 1), 500) : 100; 1094 + 1095 + const results = replicationManager.getLexiconIndex(undefined, limit); 1096 + return c.json({ lexicons: results }); 1097 + } 1098 + 1099 + export function getLexiconStats( 1100 + c: Context<AppEnv>, 1101 + replicationManager: ReplicationManager | undefined, 1102 + ): Response { 1103 + if (!replicationManager) { 1104 + return c.json({ error: "ReplicationNotEnabled", message: "Replication is not enabled" }, 400); 1105 + } 1106 + 1107 + const stats = replicationManager.getLexiconStats(); 1108 + return c.json(stats); 1109 + } 1110 + 1111 + // ============================================ 1065 1112 // PLC rotation key management 1066 1113 // ============================================ 1067 1114