================================================================================ *roadmap.txt* PDSharp: F#/Giraffe PDS Implementation Roadmap ================================================================================ PHASE 1: IMPLEMENTATION (Build) ================================================================================ Milestone A: Giraffe XRPC Shell -------------------------------------------------------------------------------- - [x] Implement /xrpc/com.atproto.server.describeServer (GET) - [x] Implement NSID-based routing with structured error responses DoD: describeServer responds with stable JSON -------------------------------------------------------------------------------- Milestone B: Identity + Crypto Primitives -------------------------------------------------------------------------------- - [x] DID document fetch/parse for signing key and PDS endpoint - [x] SHA-256 hashing, ECDSA sign/verify (p256 + k256), low-S enforcement DoD: Sign and verify atproto commit hash with low-S -------------------------------------------------------------------------------- Milestone C: DAG-CBOR + CID -------------------------------------------------------------------------------- - [x] Canonical DAG-CBOR encode/decode with IPLD link tagging - [x] CID creation/parsing (multicodec dag-cbor, sha2-256) DoD: Record JSON → stable DAG-CBOR bytes → deterministic CID -------------------------------------------------------------------------------- Milestone D: MST Implementation -------------------------------------------------------------------------------- - [x] Merkle Search Tree per repository spec - [x] Key depth = leading zero bits in SHA-256(key) counted in 2-bit chunks - [x] Node encoding: (l, e[p,k,v,t]) with key prefix compression DoD: Insert/update/delete yields reproducible root CID -------------------------------------------------------------------------------- Milestone E: Commit + BlockStore + putRecord -------------------------------------------------------------------------------- - [x] BlockStore: cid → bytes, indexed by DID/rev/head - [x] Commit signing: UnsignedCommit → DAG-CBOR → sha256 → ECDSA sign - [x] Implement com.atproto.repo.putRecord/createRecord DoD: Write and read records by path/AT-URI -------------------------------------------------------------------------------- Milestone F: CAR Export + Sync Endpoints -------------------------------------------------------------------------------- - [x] CARv1 writer (roots = commit CID, blocks stream) - [x] Implement: sync.getRepo, sync.getBlocks, sync.getBlob DoD: External services can fetch repo snapshot + blocks -------------------------------------------------------------------------------- Milestone G: subscribeRepos Firehose -------------------------------------------------------------------------------- - [x] Monotonic sequence number + commit event generation - [x] WebSocket streaming for subscribeRepos DoD: Relay/client receives commit events after writes -------------------------------------------------------------------------------- Milestone H: Account + Sessions -------------------------------------------------------------------------------- - [x] Implement: server.createAccount, server.createSession, refreshSession - [x] Password/app-password hashing + JWT issuance DoD: Authenticate and write records with accessJwt -------------------------------------------------------------------------------- Milestone I: Lexicon Validation + Conformance -------------------------------------------------------------------------------- - [x] Lexicon validation for writes (app.bsky.* records) - [x] Conformance testing: diff CIDs/CARs/signatures vs reference PDS DoD: Same inputs → same outputs for repo/sync surfaces -------------------------------------------------------------------------------- Milestone J: Persistence + Backups -------------------------------------------------------------------------------- Deliverables: - BackupOps module in Core (scheduler unit / cron / scripts, plus Litestream config) Backups (SQLite) [ ] Set PDS_SQLITE_DISABLE_WAL_AUTO_CHECKPOINT=true (Litestream-friendly) [ ] Run a scheduled backup/replication job that: - finds recently updated DBs - backs up /pds/actors/* and PDS-wide DBs - runs on SIGTERM during deploys (avoid missing last writes) Backups (Blobs) [ ] Configurable Options (app settings): (A) Disk blobs: include /pds/blocks in backups (B) S3-compatible blobstore: rely on object-store durability Guardrails [ ] Uptime check: https:///xrpc/_health [ ] Alert if “latest backup” is older than N minutes. [ ] Alert on disk pressure for /pds. DoD: - You can restore onto a fresh host and pass the P3 verification checklist. - Backups run automatically and are observable (“last successful backup”). - Backup set is explicitly documented (DBs + blobs decision). ================================================================================ PHASE 2: DEPLOYMENT (Self-Host) ================================================================================ Milestone J: Topology + Domain Planning -------------------------------------------------------------------------------- - Choose PDS hostname (pds.example.com) vs handle domain (example.com) - Obtain domain, DNS access, VPS with static IP, reverse proxy DoD: Clear plan for PDS location, handle, and DID resolution -------------------------------------------------------------------------------- Milestone K: DNS + TLS + Reverse Proxy -------------------------------------------------------------------------------- - DNS A/AAAA records for PDS hostname - TLS certs (ACME) via Caddy DoD: https:// responds with valid cert -------------------------------------------------------------------------------- Milestone L: Deploy PDSharp -------------------------------------------------------------------------------- - Deploy built PDS with persistence (SQLite/Postgres + blob storage) - Verify /xrpc/com.atproto.server.describeServer DoD: describeServer returns capabilities payload -------------------------------------------------------------------------------- Milestone M: Account Creation -------------------------------------------------------------------------------- - Create account using admin tooling - Verify authentication: createSession DoD: Obtain session and perform authenticated write -------------------------------------------------------------------------------- Milestone N: Smoke Test Repo + Blobs -------------------------------------------------------------------------------- - Write record via putRecord - Upload blob, verify retrieval via sync.getBlob DoD: Posts appear in clients, media loads reliably -------------------------------------------------------------------------------- Milestone O: Account Migration -------------------------------------------------------------------------------- - Export/import from bsky.social - Update DID service endpoint - Verify handle/DID resolution DoD: Handle unchanged, DID points to your PDS -------------------------------------------------------------------------------- Milestone P: Reliability -------------------------------------------------------------------------------- - Backups: repo storage + database + blobs - Restore drill on fresh instance - Monitoring: uptime checks for describeServer + getBlob DoD: Restore from backup passes smoke tests -------------------------------------------------------------------------------- Milestone Q: Updates + Security -------------------------------------------------------------------------------- - Update cadence with rollback plan - Rate limits and access controls at proxy - Log retention and disk growth alerts DoD: Update smoothly, maintain stable federation ================================================================================ QUICK CHECKLIST ================================================================================ [x] describeServer endpoint working [x] Crypto primitives (sha256, ECDSA p256/k256, low-S) [x] DAG-CBOR + CID generation correct [x] MST producing deterministic root CIDs [x] putRecord + blockstore operational [x] CAR export + sync endpoints [x] subscribeRepos firehose [x] Authentication (createAccount, createSession) [ ] Lexicon validation [ ] Domain + TLS configured [ ] PDS deployed and reachable [ ] Account created, session works [ ] Writes + blobs verified [ ] Backups + monitoring in place ================================================================================ REFERENCES ================================================================================ https://atproto.com/guides/self-hosting https://github.com/bluesky-social/pds https://atproto.com/specs/repository https://atproto.com/specs/sync https://atproto.com/specs/blob https://docs.bsky.app/docs/api/com-atproto-server-describe-server https://docs.bsky.app/docs/api/com-atproto-server-create-session https://docs.bsky.app/docs/api/com-atproto-repo-put-record https://docs.bsky.app/docs/api/com-atproto-sync-get-blob