P2PDS#

Peer-to-peer replication infrastructure for AT Protocol. Backs up and serves atproto account data over IPFS, acting on behalf of authenticated users.

• Syncs and stores repos and blobs for configured accounts
• Provides data on P2P networks (IPFS/libp2p) for other nodes to replicate
• Fetches and stores data from P2P networks for serviced accounts
• Mutual replication agreements between peers via on-protocol consent records
• Push-based offer discovery: nodes notify each other of replication offers

P2PDS is infrastructure — like a torrent client for atproto data. It does not have its own identity. Users authenticate with their own atproto accounts, and records (org.p2pds.peer, org.p2pds.replication.offer) are published to the user's own repo via their PDS.

Stack#

• Runtime: Node.js, TypeScript (ES2022, strict)
• Base: Generalized from Cirrus
• HTTP: Hono
• Database: better-sqlite3 (sync API) — all state in a single pds.db file
• IPFS: Helia with minimal libp2p (TCP + noise + yamux), SQLite-backed blockstore
• Identity: AT Protocol DIDs via PLC directory
• Auth: OAuth (primary) or legacy JWT (fallback)
• Desktop: Tauri v2 (optional, apps/desktop/)
• Content addressing: DASL CIDs (CIDv1, SHA-256, dag-cbor/raw, base32lower)

Architecture#

User's atproto account (any PDS: Bluesky, Cirrus, self-hosted)
        │
        ▼
   ┌─────────┐
   │  p2pds   │  ← replication infrastructure (local, cloud, or co-located)
   │          │
   │  SQLite  │  blocks, blobs, sync state, peer routing, challenge history
   │  Helia   │  IPFS storage, direct peer connections, bitswap
   │  Hono    │  XRPC endpoints, admin dashboard, RASL
   └─────────┘
        │
        ▼
   Other p2pds nodes (mutual replication via offer records)

Storage#

All persistent state lives in a single SQLite database (pds.db):

• IPFS blocks (ipfs_blocks) — replaces filesystem blockstore, avoids thousands of tiny files
• IPFS datastore (ipfs_datastore) — replaces filesystem datastore for libp2p peer/routing data
• Replication state — sync progress, peer info, block/blob tracking, firehose cursor
• Challenge history — proof-of-storage results and peer reliability scores
• Incoming offers — offers from other nodes awaiting accept/reject
• Node identity — DID + handle, established on first OAuth login

Identity model#

P2PDS starts without an identity. On first OAuth login, the user's DID becomes the node identity and is persisted in SQLite. Subsequent restarts load the identity from the database. This "lazy identity" model means:

• No DID or signing key required in config
• Identity established interactively via the dashboard
• RepoManager is optional throughout (firehose, replication, startup all handle its absence)

Libp2p configuration#

Helia runs with a minimal libp2p stack: TCP transport, Noise encryption, Yamux multiplexing, and Identify only. No DHT, gossipsub, relay, autoNAT, UPnP, or WebRTC — those services peg CPU connecting to random peers. P2PDS dials known peers directly using multiaddrs from org.p2pds.peer records.

Replication flow#

1. User adds a DID via the dashboard → publishes an org.p2pds.replication.offer record
2. Node resolves the target's org.p2pds.peer record to find their p2pds endpoint
3. Node POSTs a notification to the target's notifyOffer endpoint
4. Target verifies the offer exists in the offerer's repo (anti-spoofing)
5. Target's dashboard shows the incoming offer with Accept/Reject buttons
6. Accepting creates a reciprocal offer + push notification back
7. Both nodes detect mutual agreement → promote to active replication
8. Sync loop: fetch repo, store blocks/blobs, verify, announce

Design choices#

• DHT only for discovery/routing — no IPNI or centralized indexers
• Slow data is fine as a tradeoff for resilience and decentralization
• Transport-agnostic verification — RASL works over any HTTP transport
• DASL-compliant content addressing — all CIDs are CIDv1 + SHA-256 with either dag-cbor (0x71) or raw (0x55) codec, encoded as base32lower (b prefix). Enforced by @atcute/cid.
• No node identity — p2pds acts on behalf of users, not as its own entity
• SQLite everywhere — single-file database, no filesystem blockstore churn
• Consent-gated replication — "Add" publishes an offer, not an immediate sync. Replication only begins when both sides agree.

Deployment flexibility#

P2PDS works with any combination of:

• PDS: Bluesky, Cirrus, self-hosted, any atproto-compatible PDS
• Location: local desktop (via Tauri app), cloud (Railway, etc.), co-located server

Lexicons#

P2PDS defines two record types for the on-protocol interop surface:

Schemas are in lexicons/ and validated by src/lexicons.ts.

Offer negotiation: Peers publish offer records declaring willingness to replicate specific DIDs. When two peers have mutual offers (A offers to replicate B, B offers to replicate A), a replication agreement is automatically formed. Parameters are merged: max(minCopies), min(intervalSec), max(priority). Revoking an offer = deleting the record.

Push notifications: When a node publishes an offer, it resolves the target's org.p2pds.peer record to find their p2pds HTTP endpoint and POSTs a notification. The receiving node verifies the offer exists in the sender's repo before storing it (prevents spoofing).

Verification#

Content-addressed retrieval is unforgeable: if a peer returns the correct bytes for a CID, they have the data. The verification stack exploits this property at multiple layers:

Challenge-response protocol: Three message types (StorageChallenge → StorageChallengeResponse → StorageChallengeResult). Deterministic generation from epoch + DIDs + nonce. Transport-agnostic with libp2p primary and HTTP fallback (FailoverChallengeTransport). Challenge history and peer reliability tracked in SQLite.

Replication#

Sync loop (per DID, periodic with policy-driven intervals):

1. Resolve DID → PDS endpoint (via PLC directory)
2. Try libp2p peer-first sync if peer info is known
3. Fall back to HTTP PDS fetch (com.atproto.sync.getRepo, incremental via since)
4. Parse CAR, store blocks in IPFS, fetch and store blobs
5. Track block/blob CIDs, populate record paths via MST walk
6. Verify local block availability
7. If source PDS fails, fall back to peer endpoints

Real-time sync: Firehose subscription (com.atproto.sync.subscribeRepos) with cursor persistence, DID filtering, and incremental block application. Gossipsub commit notifications for low-latency cross-node coordination.

GC and tombstones: Deferred GC via needs_gc flag on delete/update ops. Full block/blob reconciliation during sync via MST walk. Cross-DID block sharing safety. Tombstone detection via firehose #account events with 24hr grace period.

Policy Engine#

Declarative, deterministic, transport-agnostic policy system operating on atproto accounts:

• Mutual aid: N-of-M redundancy between cooperating peers
• SaaS: SLA compliance with minimum copy counts and sync intervals
• Group governance: Multi-party replication agreements

Policies drive sync intervals, priority ordering, and shouldReplicate filtering in the replication manager. P2P policies are auto-generated from mutual offer records with p2p: prefixed IDs.

App#

• Dashboard: Server-rendered HTML at / with auto-refresh, account search, incoming offer notifications
• API: Authenticated XRPC endpoints for overview, per-DID status, network status, policies, sync history
• DID management: Add/remove/offer DIDs at runtime via dashboard or API
• Incoming offers: Accept/reject replication offers from other nodes via dashboard
• Rate limiting: Per-IP limits across all endpoint groups (meta, sync, session, read, write, challenge, app, notifyOffer)

Desktop App#

Optional Tauri v2 wrapper at apps/desktop/. Spawns p2pds as a sidecar process and loads the dashboard in a webview.

cd apps/desktop
npm run build:sidecar   # compile p2pds to standalone binary via pkg
cargo tauri dev          # run in development
cargo tauri build        # build distributable

Development#

npm install
npm test
npm run dev

Two-node manual testing#

Scripts for running two p2pds nodes locally for manual testing:

npm run start:both       # Build and start both nodes on random ports
npm run start:node1      # Start node 1 only
npm run start:node2      # Start node 2 only (uses data-node2/.env)
npm run stop             # Stop both nodes
npm run clean            # Wipe data for both nodes (keeps .env files)
npm run logs             # Show recent logs for both nodes
npm run test:add-did     # Offer a DID on a running node

Node 2 requires a data-node2/.env file with a separate OAUTH_ENABLED=true and DATA_DIR=./data-node2 config. Both nodes pick random ports and write them to /tmp/p2pds-node{1,2}.port.

Project structure#

src/
  index.ts              Hono app with all routes
  server.ts             HTTP server entry point
  start.ts              Server startup orchestrator
  config.ts             Config interface + loadConfig()
  ipfs.ts               IpfsService (Helia wrapper, SQLite-backed)
  sqlite-blockstore.ts  SQLite blockstore for Helia (replaces FsBlockstore)
  sqlite-datastore.ts   SQLite datastore for libp2p (replaces FsDatastore)
  repo-manager.ts       Local repo management
  storage.ts            SQLite block storage
  blobs.ts              Blob storage
  middleware/auth.ts     Auth middleware (OAuth + legacy JWT)
  oauth/                OAuth client, routes, session/state stores, PdsClient
  replication/           Sync, verification, challenges, offers, gossipsub
  policy/                Policy engine types, engine, presets
  xrpc/                  XRPC endpoint handlers
scripts/                 Two-node testing scripts
lexicons/                Lexicon JSON schemas
apps/desktop/            Tauri desktop app

Configuration#

Environment variables (or .env file):

Legacy auth (when OAUTH_ENABLED=false):

Status#

1. Single-user PDS — done
2. Record replication with IPFS storage — done
3. Real-time firehose sync — done
4. Layered verification (L0-L3) — done
5. Challenge-response proof-of-storage — done
6. Policy engine — done
7. P2P offer negotiation — done
8. Consent-gated replication — done
9. Incoming offer discovery via push notification — done
10. Admin dashboard + DID management — done
11. Rate limiting — done
12. Architecture refactor (user-DID model, lazy identity) — done
13. SQLite-backed IPFS storage — done
14. Lexicon definitions — done
15. Desktop app skeleton — done