atproto user agency toolkit for individuals and groups
7
fork

Configure Feed

Select the types of activity you want to include in your feed.

Add policy lifecycle, consent records, OAuth default-on, and terminology cleanup

- Policy lifecycle: StoredPolicy with state machine (proposed/active/
suspended/terminated/purged), consent status, timestamps, persistence
via PolicyStorage. New configArchive() and archive() presets.
- Consent records: CONSENT_NSID + ConsentRecord type, consent.json lexicon
- OAuth default-on: OAUTH_ENABLED defaults true (opt-out via =false)
- Session safety: purge leftover data on new OAuth login
- Scripts: --clean flag, fixed ports (6700/6701), check-api/health/etc
- Terminology: "dashboard" -> "app", "admin" -> "user" throughout

+1667 -84
+9 -9
README.md
··· 33 33 │ │ 34 34 │ SQLite │ blocks, blobs, sync state, peer routing, challenge history 35 35 │ Helia │ IPFS storage, direct peer connections, bitswap 36 - │ Hono │ XRPC endpoints, admin dashboard, RASL 36 + │ Hono │ XRPC endpoints, app, RASL 37 37 └─────────┘ 38 38 39 39 ··· 56 56 P2PDS starts without an identity. On first OAuth login, the user's DID becomes the node identity and is persisted in SQLite. Subsequent restarts load the identity from the database. This "lazy identity" model means: 57 57 58 58 - No DID or signing key required in config 59 - - Identity established interactively via the dashboard 59 + - Identity established interactively via the app 60 60 - `RepoManager` is optional throughout (firehose, replication, startup all handle its absence) 61 61 62 62 ### Libp2p configuration ··· 65 65 66 66 ### Replication flow 67 67 68 - 1. User adds a DID via the dashboard → publishes an `org.p2pds.replication.offer` record 68 + 1. User adds a DID via the app → publishes an `org.p2pds.replication.offer` record 69 69 2. Node resolves the target's `org.p2pds.peer` record to find their p2pds endpoint 70 70 3. Node POSTs a notification to the target's `notifyOffer` endpoint 71 71 4. Target verifies the offer exists in the offerer's repo (anti-spoofing) 72 - 5. Target's dashboard shows the incoming offer with Accept/Reject buttons 72 + 5. Target's app shows the incoming offer with Accept/Reject buttons 73 73 6. Accepting creates a reciprocal offer + push notification back 74 74 7. Both nodes detect mutual agreement → promote to active replication 75 75 8. Sync loop: fetch repo, store blocks/blobs, verify, announce ··· 146 146 147 147 ## App 148 148 149 - - **Dashboard**: Server-rendered HTML at `/` with auto-refresh, account search, incoming offer notifications 149 + - **App**: Server-rendered HTML at `/` with auto-refresh, account search, incoming offer notifications 150 150 - **API**: Authenticated XRPC endpoints for overview, per-DID status, network status, policies, sync history 151 - - **DID management**: Add/remove/offer DIDs at runtime via dashboard or API 152 - - **Incoming offers**: Accept/reject replication offers from other nodes via dashboard 151 + - **DID management**: Add/remove/offer DIDs at runtime via app or API 152 + - **Incoming offers**: Accept/reject replication offers from other nodes via app 153 153 - **Rate limiting**: Per-IP limits across all endpoint groups (meta, sync, session, read, write, challenge, app, notifyOffer) 154 154 155 155 ## Desktop App 156 156 157 - Optional Tauri v2 wrapper at `apps/desktop/`. Spawns p2pds as a sidecar process and loads the dashboard in a webview. 157 + Optional Tauri v2 wrapper at `apps/desktop/`. Spawns p2pds as a sidecar process and loads the app in a webview. 158 158 159 159 ``` 160 160 cd apps/desktop ··· 252 252 7. P2P offer negotiation — done 253 253 8. Consent-gated replication — done 254 254 9. Incoming offer discovery via push notification — done 255 - 10. Admin dashboard + DID management — done 255 + 10. App UI + DID management — done 256 256 11. Rate limiting — done 257 257 12. Architecture refactor (user-DID model, lazy identity) — done 258 258 13. SQLite-backed IPFS storage — done
+3 -3
apps/desktop/src/main.ts
··· 24 24 return `http://127.0.0.1:${port}/xrpc/_health`; 25 25 } 26 26 27 - function dashboardUrl(port: number): string { 27 + function appUrl(port: number): string { 28 28 return `http://127.0.0.1:${port}/`; 29 29 } 30 30 ··· 128 128 setStatus("Waiting for server..."); 129 129 await waitForServer(port); 130 130 131 - setStatus("Redirecting to dashboard..."); 132 - window.location.href = dashboardUrl(port); 131 + setStatus("Redirecting to app..."); 132 + window.location.href = appUrl(port); 133 133 } catch (err) { 134 134 const message = err instanceof Error ? err.message : String(err); 135 135 showError(message);
+734
docs/policy-engine-design.md
··· 1 + # Policy Engine Design 2 + 3 + ## 1. Current State Analysis 4 + 5 + ### What Exists 6 + 7 + The policy system is spread across four locations, each with its own notion of "this DID should be replicated": 8 + 9 + **Source 1: Config REPLICATE_DIDS** (`src/config.ts`) 10 + - Comma-separated DID list from environment variable 11 + - Cannot be modified at runtime 12 + - Always replicated, regardless of policy engine state 13 + - No parameters (uses global defaults) 14 + 15 + **Source 2: admin_tracked_dids** (`src/replication/sync-storage.ts`) 16 + - SQLite table populated by `addDid()` and the offer-acceptance flow 17 + - Mutable at runtime via XRPC endpoints 18 + - Also used as the promotion target when mutual offers are detected: `offerDid()` publishes an offer, and when mutual agreement is discovered, `addDid()` inserts the DID here 19 + - No parameters attached to the row itself 20 + 21 + **Source 3: PolicyEngine** (`src/policy/engine.ts`) 22 + - In-memory `Policy[]` with JSON serialization 23 + - Loaded from `POLICY_FILE` or created empty 24 + - `p2p:`-prefixed policies are auto-generated by OfferManager when mutual offers are detected 25 + - Can be manually populated via presets (reciprocal, saas, groupGovernance) 26 + - Pure evaluation: `evaluate(did)` merges all matching policies into an `EffectivePolicy` 27 + 28 + **Source 4: offered_dids / incoming_offers** (`src/replication/sync-storage.ts`) 29 + - `offered_dids`: DIDs we have published an offer for but that are not yet actively replicated (awaiting mutual consent) 30 + - `incoming_offers`: offers from remote peers, awaiting user acceptance 31 + - These are staging tables that feed into Source 2 upon mutual agreement 32 + 33 + ### What's Working 34 + 35 + The consent-gated flow is functional end-to-end: 36 + 1. User clicks "Add" on a DID -> publishes `org.p2pds.replication.offer` record to their PDS 37 + 2. Remote peer receives push notification -> incoming_offers row created 38 + 3. Remote user clicks "Accept" -> publishes reciprocal offer, triggers offer discovery 39 + 4. Mutual agreement detected -> DID promoted to admin_tracked_dids, P2P policy created, sync begins 40 + 5. Revocation: either side deletes offer -> peer notified -> data purged 41 + 42 + The PolicyEngine correctly merges multiple overlapping policies (max minCopies, min interval, etc.) and the ReplicationManager respects per-DID sync intervals. 43 + 44 + ### What's Messy 45 + 46 + **1. Three-source merging creates implicit priority rules.** 47 + `getReplicateDids()` unions config DIDs, admin DIDs, and policy explicit DIDs. Config DIDs and admin DIDs always replicate regardless of policy evaluation. This means the policy engine is advisory for some DIDs but authoritative for others, and the logic for which is which is buried in `getReplicateDids()`. 48 + 49 + **2. admin_tracked_dids conflates manual tracking with agreement-driven tracking.** 50 + When a user manually adds a DID via the UI, it goes into admin_tracked_dids. When a mutual offer promotes a DID, it also goes into admin_tracked_dids. There's no way to distinguish "I manually archived this DID" from "this DID was auto-promoted from a P2P agreement." This matters for revocation: revoking a P2P agreement should remove the DID, but removing a manual archive should not touch any offer records. 51 + 52 + **3. Policy is stateless; agreement state lives elsewhere.** 53 + The P2P policy (`p2p:did:plc:xxx`) is a plain `Policy` object with no lifecycle metadata: no creation timestamp, no agreement state, no link back to the offers that generated it. OfferManager maintains this linkage implicitly by re-scanning offers during `discoverAndSync()`, but if the process restarts, P2P policies must be rediscovered from scratch. 54 + 55 + **4. No violation/degradation model.** 56 + The current system is binary: either a DID is replicated or it isn't. There's no notion of "this agreement is in trouble" (peer stopped syncing, challenges failing, peer unreachable). The challenge-response system produces reliability scores, but nothing acts on them. 57 + 58 + **5. UI shows raw state, not semantic status.** 59 + The app shows sync states (pending/syncing/synced/error/tombstoned) and DID sources (config/user/policy), but not agreement-level status. A user can't see "your reciprocal replication agreement with @alice.bsky.social is healthy" -- they see a DID, a sync timestamp, and a source label. 60 + 61 + --- 62 + 63 + ## 2. Proposed Architecture 64 + 65 + ### Core Principle: Policy as the Single Source of Truth 66 + 67 + Every replicated DID must be justified by a policy. The policy engine becomes the authoritative answer to: 68 + - "Should this DID be replicated?" (yes/no) 69 + - "How should it be replicated?" (parameters) 70 + - "Why is it being replicated?" (policy type + provenance) 71 + - "What is the state of that replication relationship?" (lifecycle) 72 + 73 + Config REPLICATE_DIDS and admin_tracked_dids stop being independent sources. Instead, they become policy *generators*: inputs that produce policy objects which flow through the same evaluation pipeline as everything else. 74 + 75 + ### Architecture Layers 76 + 77 + ``` 78 + +---------------------------------------------------------+ 79 + | UI / XRPC Layer | 80 + | Shows policies, agreements, lifecycle states, actions | 81 + +---------------------------------------------------------+ 82 + | 83 + +---------------------------------------------------------+ 84 + | Policy Engine | 85 + | - Stores all policies (SQLite-backed) | 86 + | - Evaluates effective policy per DID | 87 + | - Manages lifecycle state transitions | 88 + | - Emits events on state changes | 89 + +---------------------------------------------------------+ 90 + | | | 91 + +-----------------+ +------------------+ +--------------+ 92 + | Policy Sources | | Lifecycle Driver | | Health Monitor| 93 + | - Config DIDs | | - Offer flow | | - Sync status | 94 + | - Manual pins | | - Acceptance | | - Challenges | 95 + | - P2P offers | | - Revocation | | - Reachability| 96 + | - File presets | | - Expiration | | - Grace periods| 97 + +-----------------+ +------------------+ +--------------+ 98 + ``` 99 + 100 + ### Policy Storage 101 + 102 + Policies move from in-memory-only to SQLite-backed with the following schema: 103 + 104 + ```sql 105 + CREATE TABLE policies ( 106 + id TEXT PRIMARY KEY, 107 + type TEXT NOT NULL, -- 'reciprocal', 'archive', 'config', 'saas', 'group' 108 + state TEXT NOT NULL DEFAULT 'active', -- lifecycle state 109 + consent TEXT NOT NULL DEFAULT 'unconsented', -- 'reciprocal', 'consented', 'unconsented', 'revoked' 110 + name TEXT NOT NULL, 111 + target_json TEXT NOT NULL, -- JSON: PolicyTarget 112 + replication_json TEXT NOT NULL, -- JSON: ReplicationGoals 113 + sync_json TEXT NOT NULL, -- JSON: SyncConfig 114 + retention_json TEXT NOT NULL, -- JSON: RetentionConfig 115 + priority INTEGER NOT NULL DEFAULT 50, 116 + enabled INTEGER NOT NULL DEFAULT 1, 117 + 118 + -- Lifecycle metadata 119 + created_at TEXT NOT NULL, 120 + activated_at TEXT, 121 + suspended_at TEXT, 122 + terminated_at TEXT, 123 + expires_at TEXT, -- optional TTL 124 + 125 + -- Provenance (who/what created this policy) 126 + created_by TEXT, -- DID of the user who created it, or 'system' 127 + source TEXT, -- 'config', 'file', 'offer', 'manual', 'api' 128 + 129 + -- Agreement linkage (for P2P policies) 130 + counterparty_did TEXT, -- the other party in a bilateral agreement 131 + local_offer_uri TEXT, -- AT URI of our offer record 132 + remote_offer_uri TEXT -- AT URI of their offer record 133 + ); 134 + ``` 135 + 136 + This replaces the current in-memory `Policy[]` and the separate `admin_tracked_dids`, `offered_dids` tables. (The `incoming_offers` table remains -- it represents *pending* proposals that have not yet become policies.) 137 + 138 + --- 139 + 140 + ## 3. Consent Model 141 + 142 + Consent and reciprocity are orthogonal dimensions. Every replication relationship can be characterized along both axes: 143 + 144 + | | **Consensual** | **Non-consensual** | 145 + |--|---------------|-------------------| 146 + | **Reciprocal** | Both parties explicitly agree to archive each other. The current offer/accept flow. | Both parties independently archive each other without coordination. An emergent state, not a policy type — detected when two independent archive policies point at each other. | 147 + | **Non-reciprocal** | Subject grants permission to be archived (open consent or directed consent), but the archiver does not share their data back. | Archiver replicates public data without the subject's explicit permission. The data is public anyway (it's on their PDS), but the subject hasn't opted in. | 148 + 149 + ### Consent mechanisms 150 + 151 + **Explicit consent (directed):** The current offer/accept flow. Account A offers to replicate Account B. Account B accepts, granting consent and optionally reciprocating. Consent is recorded as `org.p2pds.replication.offer` records in both repos. 152 + 153 + **Open consent:** An account publishes a standing record (`org.p2pds.replication.consent`) declaring "anyone may archive my data." Any p2pds node that discovers this record can create an archive policy without needing a per-peer handshake. Open consent can be revoked at any time by deleting the record. 154 + 155 + **No consent (unilateral):** An archiver replicates public atproto data without any consent signal from the subject. This is always technically possible (the data is public), but the policy engine should distinguish it so the UI can surface the consent status clearly. 156 + 157 + ### How consent flows through the system 158 + 159 + 1. **Archiver initiates:** User clicks "Archive" on a DID → policy engine checks for consent signals. 160 + 2. **Consent check:** Is there an open consent record? Is there an existing directed offer from the subject? If yes → archive with consent. If no → archive without consent (unilateral). 161 + 3. **Reciprocity check:** Does the subject also archive the archiver? If yes → the relationship is reciprocal. The policy engine detects this and can surface it in the UI ("reciprocal") even if neither side explicitly requested reciprocity. 162 + 4. **Consent revocation:** Subject deletes their consent/offer record → the archiver's policy transitions. For reciprocal policies, this terminates the agreement. For unilateral archives, the data remains (it's public) but the consent status updates to "no consent." 163 + 164 + ### Consent status as policy metadata 165 + 166 + Rather than consent being a policy *type*, it's a property of every policy: 167 + 168 + ```typescript 169 + type ConsentStatus = 170 + | "reciprocal" // Both parties consented and archive each other 171 + | "consented" // Subject explicitly consented (open or directed) 172 + | "unconsented" // No consent signal from subject 173 + | "revoked"; // Subject previously consented, then revoked 174 + ``` 175 + 176 + This means the policy types become simpler — they describe *what* is happening, while consent describes the *relationship*: 177 + 178 + ## 4. Policy Type Taxonomy 179 + 180 + ### 4.1 Archive (`archive`) 181 + 182 + The fundamental operation: replicate a DID's data on this node. Every replication relationship starts as an archive. 183 + 184 + **Consent variants:** 185 + - **Consented archive:** Subject has an open consent record or accepted a directed offer. The archiver can prove permission. 186 + - **Unconsented archive:** No consent signal. The data is public, so this is always possible, but the UI makes the distinction visible. 187 + 188 + **Parameters:** 189 + - `minCopies`: 1 (just this node) 190 + - `intervalSec`: 300 (default) 191 + - `priority`: 40 192 + - `consent`: ConsentStatus 193 + 194 + **Provenance:** Created by user action ("Archive" button), config, or API. 195 + 196 + **Invariant:** Active as long as the archiver wants it. If consent was present and gets revoked, the archive can continue (data is public) but consent status changes to `revoked`. 197 + 198 + ### 4.2 Reciprocal (`reciprocal`) 199 + 200 + A reciprocal policy is an archive with a bilateral consent agreement: both parties archive each other and both have consented. This is the current offer/accept flow. 201 + 202 + **How it emerges:** 203 + 1. Account A offers to archive Account B (publishes offer, creates a `proposed` reciprocal policy) 204 + 2. Account B accepts (publishes reciprocal offer, creating their own reciprocal policy) 205 + 3. Both policies transition to `active` when mutual offers are detected 206 + 207 + **Can also emerge organically:** If Account A creates an archive policy for B, and Account B independently creates an archive policy for A, the policy engine detects the mutual relationship and can prompt: "You're both archiving each other. Convert to a reciprocal agreement?" 208 + 209 + **Parameters:** 210 + - `minCopies`: 2 (at least two copies across the agreement) 211 + - `intervalSec`: 600 (default) 212 + - `priority`: 50 (higher than unilateral archive) 213 + - `counterpartyDid`: the other party's DID 214 + - `consent`: always `"reciprocal"` 215 + 216 + **Provenance:** Created by the offer/accept consent flow. Both sides hold a reciprocal policy pointing at each other. 217 + 218 + **Invariant:** Valid only when both parties have active offer records. If either side revokes consent, the policy terminates. The remaining party can choose to continue as a unilateral archive (downgrade to `archive` with `consent: "revoked"`). 219 + 220 + ### 4.3 Config (`config`) 221 + 222 + DIDs specified via the `REPLICATE_DIDS` environment variable. Infrastructure-level archives that persist across restarts and can't be modified from the UI. 223 + 224 + **Parameters:** 225 + - `minCopies`: 1 226 + - `intervalSec`: 300 227 + - `priority`: 30 (lowest — infrastructure baseline) 228 + - `consent`: typically `"unconsented"` (infra doesn't negotiate) 229 + 230 + **Provenance:** Generated at startup from config. Policy IDs are `config:{did}`. 231 + 232 + **Invariant:** Exists as long as the DID is in the config. Recreated on every startup. 233 + 234 + ### 4.4 Open Consent Record (`org.p2pds.replication.consent`) 235 + 236 + Not a policy type, but a mechanism that feeds into policy creation. An account publishes this record to declare open consent: 237 + 238 + ```json 239 + { 240 + "$type": "org.p2pds.replication.consent", 241 + "scope": "any", 242 + "createdAt": "2026-02-16T..." 243 + } 244 + ``` 245 + 246 + When a p2pds node discovers this record (during archival or via firehose), it can upgrade any unconsented archive of that DID to `consent: "consented"`. It also enables the UI to show "this account welcomes archiving" when a user is deciding whether to archive someone. 247 + 248 + Future extensions: `scope` could be `"any"`, `"followers"`, or an explicit DID list for directed open consent. 249 + 250 + ### 4.5 Future: SaaS (`saas`) 251 + 252 + An operator guarantees replication for a set of accounts under an SLA. Not implemented now, but the type system accommodates it. 253 + 254 + **Parameters:** 255 + - `minCopies`: 3+ (SLA-driven) 256 + - `intervalSec`: 60 (aggressive) 257 + - `priority`: 80+ (high) 258 + - SLA metadata: uptime target, response time, penalty terms 259 + 260 + ### 4.6 Future: Group / Mutual Aid (`group`) 261 + 262 + A community collectively decides which accounts to archive. This is where true mutual aid lives — community-level coordination rather than bilateral agreements. Requires quorum/voting mechanisms. 263 + 264 + **Parameters:** 265 + - `minCopies`: varies by group decision 266 + - `memberDids`: the governance group 267 + - `approvalThreshold`: N-of-M required 268 + - `priority`: 60 269 + 270 + --- 271 + 272 + ## 4. Lifecycle State Machine 273 + 274 + Every policy goes through a lifecycle. The states and transitions depend on the policy type, but the core state machine is shared: 275 + 276 + ``` 277 + +-----------+ 278 + create --> | proposed |----> [reject] ----> terminated 279 + +-----------+ 280 + | 281 + [activate] 282 + | 283 + v 284 + +-----------+ 285 + | active |<---- [resume] 286 + +-----------+ 287 + | | 288 + [suspend]| |[terminate] 289 + v | 290 + +-----------+| 291 + | suspended || 292 + +-----------+| 293 + | | 294 + [resume] | | 295 + (back up) | 296 + v 297 + +-----------+ 298 + | terminated| 299 + +-----------+ 300 + | 301 + [purge data] 302 + | 303 + +-----------+ 304 + | purged | 305 + +-----------+ 306 + ``` 307 + 308 + ### State Definitions 309 + 310 + | State | Meaning | Data Kept? | Sync Active? | 311 + |-------|---------|------------|--------------| 312 + | `proposed` | Policy created but not yet activated (e.g., offer sent, awaiting acceptance) | No data yet | No | 313 + | `active` | Policy is live; DID is being replicated according to parameters | Yes | Yes | 314 + | `suspended` | Temporarily paused (health check failure, peer unreachable, user pause) | Yes (retained) | No | 315 + | `terminated` | Policy ended (revocation, offer deleted, manual removal) | Retained briefly | No | 316 + | `purged` | Data has been deleted; policy record kept for audit trail | No | No | 317 + 318 + ### State Transitions by Policy Type 319 + 320 + **Reciprocal:** 321 + ``` 322 + User clicks "Replicate" on a DID 323 + -> proposed (offer published, awaiting consent from counterparty) 324 + 325 + Counterparty consents (accepts offer, publishes reciprocal offer) 326 + -> active, consent: reciprocal (both sides syncing) 327 + 328 + Challenge failures exceed threshold 329 + -> suspended (grace period, retry) 330 + 331 + Grace period expires with no recovery 332 + -> terminated (notify counterparty) 333 + 334 + Either side revokes consent 335 + -> terminated (notify counterparty, begin data retention countdown) 336 + 337 + Data retention period expires 338 + -> purged 339 + ``` 340 + 341 + **Archive:** 342 + ``` 343 + User clicks "Archive" on a DID 344 + -> active (immediate; consent status set based on subject's consent signals) 345 + -> consent: consented (if open consent record exists) 346 + -> consent: unconsented (if no consent signal) 347 + 348 + Subject publishes open consent record 349 + -> consent status upgrades to "consented" 350 + 351 + Subject revokes open consent 352 + -> consent status changes to "revoked" (archive continues, data is public) 353 + 354 + User pauses 355 + -> suspended 356 + 357 + User resumes 358 + -> active 359 + 360 + User removes 361 + -> terminated -> purged (immediate, no retention needed) 362 + ``` 363 + 364 + **Config:** 365 + ``` 366 + Startup, DID in REPLICATE_DIDS 367 + -> active (immediate) 368 + 369 + DID removed from config + restart 370 + -> (policy simply not recreated) 371 + ``` 372 + 373 + ### Grace Periods and Escalation 374 + 375 + When a reciprocal policy enters `suspended` state, the system follows this escalation path: 376 + 377 + 1. **Soft suspension** (0-1 hour): Sync paused, challenge retries continue. No user notification. Peer may just be temporarily offline. 378 + 2. **Warning** (1-6 hours): UI shows warning badge. Push notification to the user. Continue monitoring. 379 + 3. **Hard suspension** (6-24 hours): UI shows alert. Data retained but no sync attempts. Peer gets a "your agreement is at risk" signal. 380 + 4. **Termination** (24+ hours): Agreement terminated. Begin data retention countdown (configurable, default 7 days). 381 + 382 + This is configuration within the policy itself, not hardcoded. The reciprocal preset provides sensible defaults, but users can adjust thresholds. 383 + 384 + ```typescript 385 + interface ViolationPolicy { 386 + /** Seconds of peer unreachability before soft suspension. */ 387 + softSuspensionAfterSec: number; // default: 3600 (1 hour) 388 + /** Seconds before warning escalation. */ 389 + warningAfterSec: number; // default: 21600 (6 hours) 390 + /** Seconds before hard suspension. */ 391 + hardSuspensionAfterSec: number; // default: 86400 (24 hours) 392 + /** Seconds before automatic termination. */ 393 + terminationAfterSec: number; // default: 86400 (24 hours) 394 + /** Seconds to retain data after termination before purge. */ 395 + dataRetentionAfterTerminationSec: number; // default: 604800 (7 days) 396 + /** Challenge failure count before suspension. */ 397 + challengeFailureThreshold: number; // default: 3 398 + } 399 + ``` 400 + 401 + --- 402 + 403 + ## 5. UI Language Mapping 404 + 405 + The policy type and lifecycle state determine what the user sees. This section maps each state to display language for the primary UI contexts. 406 + 407 + ### 5.1 Action Buttons 408 + 409 + | Context | Current Language | Proposed Language | Notes | 410 + |---------|-----------------|-------------------|-------| 411 + | Adding a DID — request reciprocal | "Add" | "Replicate" | Initiates a reciprocal proposal (publishes offer, awaits consent) | 412 + | Adding a DID — unilateral archive | (not distinct) | "Archive" | Creates an archive policy; no consent required | 413 + | Responding to an incoming offer | "Accept" | "Accept & Replicate" | Makes it clear: you consent and replicate them back | 414 + | Declining an incoming offer | "Reject" | "Decline" | Less hostile than "reject" | 415 + | Stopping a reciprocal agreement | "Remove" | "Revoke" | Revokes your consent; agreement ends for both sides | 416 + | Stopping a unilateral archive | "Remove" | "Remove" | Simple removal, no consent implications | 417 + | Temporarily pausing | (not available) | "Pause" | Suspends without terminating | 418 + 419 + ### 5.2 Status Labels 420 + 421 + | Policy State | Consent | User-Facing Label | Icon/Badge | Description Text | 422 + |-------------|---------|-------------------|------------|------------------| 423 + | proposed | — | "Waiting for consent" | Clock | "You've offered to replicate this account. Waiting for them to accept." | 424 + | active | reciprocal | "Reciprocal" | Green shield | "You and {handle} are replicating each other's data." | 425 + | active | consented | "Archiving (consented)" | Green dot | "Archiving with {handle}'s consent." | 426 + | active | unconsented | "Archiving" | Blue dot | "Archiving public data." | 427 + | active | revoked | "Archiving (consent revoked)" | Yellow dot | "{handle} revoked consent. Data is still public." | 428 + | suspended (soft) | any | "Paused" | Yellow dot | "Sync paused. Retrying automatically." | 429 + | suspended (warning) | reciprocal | "Connection issues" | Orange warning | "{handle}'s node hasn't responded in {duration}." | 430 + | suspended (hard) | reciprocal | "Agreement at risk" | Red warning | "No response for {duration}. Agreement will end in {time remaining}." | 431 + | terminated | any | "Ended" | Gray X | "This replication relationship has ended." | 432 + | purged | any | (not shown) | — | Removed from active UI; available in history | 433 + 434 + ### 5.3 DID Source Labels 435 + 436 + Replace the current "config"/"user"/"policy" source labels with policy-aware labels: 437 + 438 + | Policy Type | Consent | Source Label | 439 + |-------------|---------|-------------| 440 + | reciprocal | reciprocal | "Reciprocal with {handle}" | 441 + | archive | consented | "Archived (consented)" | 442 + | archive | unconsented | "Archived" | 443 + | archive | revoked | "Archived (consent revoked)" | 444 + | config | any | "Infrastructure" | 445 + | saas | any | "SLA: {policy name}" | 446 + | group | any | "Group: {group name}" | 447 + 448 + ### 5.4 Notification Language 449 + 450 + | Event | Notification Text | 451 + |-------|-------------------| 452 + | Incoming offer | "{handle} wants to replicate your data. Accept to replicate each other." | 453 + | Offer accepted | "{handle} accepted. You're now replicating each other's data." | 454 + | Consent revoked by subject | "{handle} revoked archiving consent. Your archive continues (public data)." | 455 + | Reciprocal agreement revoked by peer | "{handle} revoked your replication agreement. Your data was removed from their node." | 456 + | Soft suspension | (no notification — auto-retry is silent) | 457 + | Warning | "{handle}'s node hasn't synced in {duration}." | 458 + | Termination by timeout | "Agreement with {handle} ended due to inactivity." | 459 + | Accidental reciprocity detected | "You and {handle} are both archiving each other. Formalize as a reciprocal agreement?" | 460 + 461 + ### 5.5 App Sections 462 + 463 + Restructure the app around policies and consent status: 464 + 465 + ``` 466 + Reciprocal Agreements 467 + @alice.bsky.social — Replicating (last sync: 2m ago) [Revoke] 468 + @bob.example.com — Waiting for consent [Cancel] 469 + 470 + Archives 471 + @carol.bsky.social — Archiving (consented) [Remove] 472 + did:plc:xyz123 — Archiving [Remove] 473 + 474 + Infrastructure 475 + did:plc:abc789 — Replicating (from config) 476 + 477 + Incoming Requests 478 + @dave.bsky.social wants to replicate your data. [Accept & Replicate] [Decline] 479 + ``` 480 + 481 + --- 482 + 483 + ## 6. Migration Path 484 + 485 + ### Phase 0: No-op foundation (prepare the ground) 486 + 487 + **Goal:** Add the `policies` table and policy persistence without changing any behavior. 488 + 489 + 1. Add SQLite `policies` table schema to SyncStorage (or a new PolicyStorage class). 490 + 2. Extend the `Policy` type with lifecycle fields (`state`, `createdAt`, `type`, `source`, `counterpartyDid`). 491 + 3. PolicyEngine gains `persist()` and `loadFromDb()` methods alongside the existing in-memory operations. 492 + 4. On startup, load policies from DB. Existing `POLICY_FILE` loading still works and seeds the DB. 493 + 494 + **Migration:** On first startup with the new schema, the `policies` table is empty. No behavior change. Existing in-memory policies continue to work. 495 + 496 + ### Phase 1: Config and archive policies 497 + 498 + **Goal:** Config DIDs and admin_tracked_dids generate policy objects. 499 + 500 + 1. At startup, for each DID in `REPLICATE_DIDS`, create a `config:{did}` policy if it doesn't exist. 501 + 2. When `addDid()` is called, instead of inserting into `admin_tracked_dids`, create a `archive` policy. 502 + 3. `getReplicateDids()` changes to: query all policies where `state = 'active'` and `enabled = 1`, collect target DIDs. No more three-source merging. 503 + 4. `getDidSource()` changes to: look up the policy type for the DID. 504 + 5. `removeDid()` changes to: transition the policy to `terminated`/`purged`. 505 + 506 + **Migration:** On first startup after this change, scan `admin_tracked_dids` and create `archive` policies for each entry. After successful migration, the `admin_tracked_dids` table becomes unused (keep it around briefly for rollback safety). 507 + 508 + ### Phase 2: Reciprocal lifecycle 509 + 510 + **Goal:** The offer flow creates and manages reciprocal policies with lifecycle states. 511 + 512 + 1. `offerDid()` creates a `reciprocal` policy in `proposed` state (replaces `offered_dids` insertion). 513 + 2. `acceptOffer()` transitions the policy to `active` (replaces the addDid() promotion). 514 + 3. `revokeReplication()` transitions to `terminated`. 515 + 4. `runOfferDiscovery()` no longer promotes offered_dids to admin_tracked_dids. Instead, it transitions `proposed` policies to `active` when mutual agreement is detected. 516 + 5. Broken-agreement detection transitions `active` policies to `terminated`. 517 + 518 + **Migration:** On first startup, scan `offered_dids` and create `proposed` reciprocal policies. Scan `admin_tracked_dids` entries that have corresponding P2P policies and convert them to `active` reciprocal policies. 519 + 520 + ### Phase 3: Health monitoring and suspension 521 + 522 + **Goal:** Policies can be suspended based on health signals. 523 + 524 + 1. Integrate challenge-response results: if a peer fails N consecutive challenges, suspend the policy. 525 + 2. Integrate sync failure tracking: if sync fails repeatedly for a DID, suspend. 526 + 3. Implement grace period escalation (soft -> warning -> hard -> termination). 527 + 4. Implement data retention countdown after termination. 528 + 5. Add health status to the policy evaluation output so the UI can display it. 529 + 530 + ### Phase 4: UI restructuring 531 + 532 + **Goal:** The dashboard is organized by policy, not by raw DID list. 533 + 534 + 1. API returns policies with lifecycle state, counterparty info, health status. 535 + 2. Dashboard groups DIDs by policy type. 536 + 3. Action buttons reflect policy semantics ("End agreement" vs "Remove archive"). 537 + 4. Notifications use policy-aware language. 538 + 539 + --- 540 + 541 + ## 7. Implementation Plan 542 + 543 + ### Phase 0: Foundation (estimated: 1-2 sessions) 544 + 545 + Files to modify: 546 + - `src/policy/types.ts` -- extend Policy with lifecycle fields 547 + - `src/policy/engine.ts` -- add persistence, load from DB 548 + - `src/replication/sync-storage.ts` -- add policies table schema (or new file `src/policy/storage.ts`) 549 + - `src/start.ts` -- load policies from DB on startup, seed from file 550 + 551 + Tests: 552 + - PolicyEngine persistence round-trip 553 + - Schema migration on fresh DB 554 + 555 + ### Phase 1: Config and Archive Policies (estimated: 2-3 sessions) 556 + 557 + Files to modify: 558 + - `src/start.ts` -- generate config policies at startup 559 + - `src/replication/replication-manager.ts` -- `addDid()` creates archive policy, `getReplicateDids()` reads from policy engine, `removeDid()` transitions policy 560 + - `src/xrpc/app.ts` -- update API responses to use policy-based source info 561 + - `src/policy/presets.ts` -- add `archive()` and `configArchive()` preset factories 562 + 563 + Migration script: 564 + - `src/policy/migrate.ts` -- scan admin_tracked_dids, create archive policies 565 + 566 + Tests: 567 + - addDid creates archive policy 568 + - getReplicateDids returns policy-driven list 569 + - Config DIDs generate config policies 570 + - Removal transitions policy state 571 + 572 + ### Phase 2: Reciprocal Lifecycle (estimated: 3-4 sessions) 573 + 574 + Files to modify: 575 + - `src/replication/replication-manager.ts` -- `offerDid()`, `acceptOffer()`, `revokeReplication()` operate on policies 576 + - `src/replication/offer-manager.ts` -- `syncPolicies()` transitions policy states instead of add/remove 577 + - `src/policy/engine.ts` -- lifecycle transition methods (`propose()`, `activate()`, `suspend()`, `terminate()`) 578 + - `src/policy/types.ts` -- add ViolationPolicy, lifecycle state enum 579 + 580 + Migration: 581 + - Convert offered_dids to proposed policies 582 + - Convert P2P-origin admin_tracked_dids to active reciprocal policies 583 + 584 + Tests: 585 + - Full offer -> accept -> active -> revoke -> terminated lifecycle 586 + - Broken agreement detection transitions to terminated 587 + - Policy state persists across restarts 588 + 589 + ### Phase 3: Health and Suspension (estimated: 2-3 sessions) 590 + 591 + Files to modify: 592 + - `src/policy/engine.ts` -- health evaluation, suspension logic 593 + - `src/replication/challenge-response/challenge-scheduler.ts` -- report results to policy engine 594 + - `src/replication/replication-manager.ts` -- sync failure reporting to policy engine 595 + - `src/policy/types.ts` -- ViolationPolicy defaults in presets 596 + 597 + Tests: 598 + - Challenge failures trigger suspension 599 + - Grace period escalation 600 + - Termination after timeout 601 + - Recovery from suspension when peer comes back 602 + 603 + ### Phase 4: UI Restructuring (estimated: 2-3 sessions) 604 + 605 + Files to modify: 606 + - `src/xrpc/app.ts` -- new API shape (policies with lifecycle state) 607 + - `src/index.ts` -- dashboard HTML rendering 608 + - UI template (inline HTML in index.ts or separate template) 609 + 610 + Tests: 611 + - API returns correct policy groupings 612 + - Status labels match lifecycle state 613 + - Action buttons match policy type 614 + 615 + --- 616 + 617 + ## 8. Detailed Type Definitions (Proposed) 618 + 619 + ```typescript 620 + /** Policy lifecycle states. */ 621 + type PolicyState = 622 + | "proposed" // Created, awaiting activation (e.g., offer sent) 623 + | "active" // Live, DID is being replicated 624 + | "suspended" // Temporarily paused (health issue, user pause) 625 + | "terminated" // Ended, data retained briefly 626 + | "purged"; // Data deleted, record kept for history 627 + 628 + /** What kind of policy this is. */ 629 + type PolicyType = 630 + | "reciprocal" // Bilateral agreement 631 + | "archive" // Unilateral personal archive 632 + | "config" // Infrastructure, from env var 633 + | "saas" // SLA compliance (future) 634 + | "group"; // Governance group (future) 635 + 636 + /** How the policy was created. */ 637 + type PolicySource = 638 + | "config" // REPLICATE_DIDS env var 639 + | "file" // POLICY_FILE JSON 640 + | "offer" // P2P offer flow 641 + | "manual" // User action in the UI 642 + | "api"; // Programmatic API call 643 + 644 + /** Consent status of a replication relationship. */ 645 + type ConsentStatus = 646 + | "reciprocal" // Both parties consented and archive each other 647 + | "consented" // Subject explicitly consented (open or directed) 648 + | "unconsented" // No consent signal from subject 649 + | "revoked"; // Subject previously consented, then revoked 650 + 651 + /** Extended policy with lifecycle and consent. */ 652 + interface PolicyV2 extends Policy { 653 + type: PolicyType; 654 + state: PolicyState; 655 + source: PolicySource; 656 + consent: ConsentStatus; 657 + 658 + // Timestamps 659 + createdAt: string; 660 + activatedAt: string | null; 661 + suspendedAt: string | null; 662 + terminatedAt: string | null; 663 + expiresAt: string | null; 664 + 665 + // Provenance 666 + createdBy: string; // DID or 'system' 667 + 668 + // Agreement linkage (reciprocal only) 669 + counterpartyDid: string | null; 670 + localOfferUri: string | null; 671 + remoteOfferUri: string | null; 672 + 673 + // Violation handling 674 + violation: ViolationPolicy; 675 + 676 + // Health snapshot (computed, not stored) 677 + health?: PolicyHealth; 678 + } 679 + 680 + /** Health status computed from sync/challenge data. */ 681 + interface PolicyHealth { 682 + /** Overall health: green/yellow/orange/red */ 683 + level: "healthy" | "degraded" | "warning" | "critical"; 684 + /** Last successful sync timestamp. */ 685 + lastSyncAt: string | null; 686 + /** Consecutive sync failures. */ 687 + consecutiveSyncFailures: number; 688 + /** Latest challenge result (if applicable). */ 689 + latestChallengeResult: "pass" | "fail" | "pending" | null; 690 + /** Peer reliability score (0-1). */ 691 + peerReliability: number | null; 692 + /** Seconds until next escalation (if degraded). */ 693 + escalationInSec: number | null; 694 + } 695 + ``` 696 + 697 + --- 698 + 699 + ## 9. Open Questions 700 + 701 + **Q1: What happens when an archive subject revokes consent?** 702 + If a subject had open consent and revokes it, should existing archives be terminated or just marked `consent: "revoked"`? The data is public regardless — revoking consent doesn't make the data private. But it does signal "I don't want you archiving me." 703 + 704 + Recommendation: Default behavior is to mark `consent: "revoked"` but keep the archive. The UI clearly shows the consent status change. The archiver can choose to respect the revocation and delete, or continue. For reciprocal policies, revocation terminates the agreement (since reciprocity requires bilateral consent). 705 + 706 + **Q2: How should the UI present unconsented archives?** 707 + Unconsented archiving is always technically possible (public data). But we should be thoughtful about the UX. Should the UI show a warning? Should it be a separate flow from consented archiving? 708 + 709 + Recommendation: The "Archive" action always works. If the subject has open consent, the UI shows a green consent badge. If not, the UI shows a neutral "no consent signal" indicator — not a warning, but visible. The distinction matters for trust and transparency, not for blocking the action. 710 + 711 + **Q3: How does open consent interact with reciprocal detection?** 712 + If Account A archives Account B (who has open consent), and Account B independently archives Account A, this is technically reciprocal but wasn't negotiated. Should the policy engine auto-detect this and offer to upgrade? 713 + 714 + Recommendation: Yes. The policy engine periodically checks for "accidental reciprocity" — two independent archive policies pointing at each other. It surfaces a prompt: "You and {handle} are both archiving each other. Would you like to formalize this as a reciprocal agreement?" This is opt-in, not automatic. 715 + 716 + **Q4: How long should terminated reciprocal data be retained?** 717 + The current implementation purges immediately on revocation. A grace period (e.g., 7 days) would allow the user to re-establish the agreement without re-syncing everything. But it also means storing data the peer has explicitly asked you to delete. 718 + 719 + Recommendation: Default 7-day retention for reciprocal termination (configurable to 0 for immediate purge). The data is content-addressed and already public (it's on their PDS), so retaining it briefly is not a privacy concern. The peer record itself says "terminated" — we're not claiming to still serve it. 720 + 721 + **Q5: Should the policy engine emit events?** 722 + State transitions (proposed → active, active → suspended, etc.) could emit events that other components subscribe to (e.g., UI notifications, sync scheduling changes, gossipsub announcements). 723 + 724 + Recommendation: Yes, but as a simple callback/listener pattern, not a full event bus. The PolicyEngine gains an `onStateChange(callback)` method. ReplicationManager and the UI layer register listeners. 725 + 726 + **Q6: What happens to policies when the user logs out?** 727 + Currently, logout disconnects OAuth and prevents offer management. Should policies survive logout? Reciprocal policies depend on offer records on the PDS, which require an active session to manage. 728 + 729 + Recommendation: Policies survive logout. Active policies continue to replicate (sync doesn't require authentication). But the user can't create, accept, or revoke offers without logging back in. The UI shows "Log in to manage agreements." 730 + 731 + **Q7: Policy versioning and schema evolution.** 732 + The current PolicySet has `version: 1`. As PolicyV2 adds fields, how do we handle upgrades? 733 + 734 + Recommendation: Version the `policies` table schema, not the policy objects. Use SQLite `ALTER TABLE ADD COLUMN` with defaults for additive changes. For breaking changes, write a migration function. The JSON fields (target_json, etc.) are forward-compatible by nature — unknown keys are ignored.
+27
lexicons/org/p2pds/replication/consent.json
··· 1 + { 2 + "lexicon": 1, 3 + "id": "org.p2pds.replication.consent", 4 + "defs": { 5 + "main": { 6 + "type": "record", 7 + "description": "Declares open consent for replication of this account's data. Published in the user's own repo. When present, any p2pds node may replicate this account without requiring a mutual offer.", 8 + "key": "literal:self", 9 + "record": { 10 + "type": "object", 11 + "required": ["scope", "createdAt"], 12 + "properties": { 13 + "scope": { 14 + "type": "string", 15 + "knownValues": ["any"], 16 + "description": "Scope of consent. Currently only 'any' is supported, meaning any node may replicate." 17 + }, 18 + "createdAt": { 19 + "type": "string", 20 + "format": "datetime", 21 + "description": "Timestamp when consent was granted." 22 + } 23 + } 24 + } 25 + } 26 + } 27 + }
+6 -1
package.json
··· 18 18 "clean": "bash scripts/clean.sh", 19 19 "logs": "bash scripts/logs.sh", 20 20 "test:add-did": "bash scripts/test-add-did.sh", 21 - "serve": "tsc && node dist/server.js > /tmp/p2pds-node1.log 2>&1 & echo $! > /tmp/p2pds-node1.pid && sleep 2 && cat /tmp/p2pds-node1.log" 21 + "serve": "tsc && node dist/server.js > /tmp/p2pds-node1.log 2>&1 & echo $! > /tmp/p2pds-node1.pid && sleep 2 && cat /tmp/p2pds-node1.log", 22 + "check-token": "bash scripts/check-token.sh", 23 + "health": "bash -c 'PORT=$(cat /tmp/p2pds-node1.port 2>/dev/null || echo 3000) && curl -s http://localhost:$PORT/xrpc/_health | python3 -m json.tool'", 24 + "check-api": "bash scripts/check-api.sh", 25 + "open": "bash scripts/open.sh", 26 + "restart": "bash scripts/restart.sh" 22 27 }, 23 28 "dependencies": { 24 29 "@atcute/atproto": "^3.1.10",
+49
scripts/check-api.sh
··· 1 + #!/bin/bash 2 + # Check API responses from running node 1. 3 + # Usage: npm run check-api 4 + set -e 5 + 6 + PORTFILE="/tmp/p2pds-node1.port" 7 + if [ ! -f "$PORTFILE" ]; then 8 + echo "Node 1 not running. Start with: npm run start:node1" 9 + exit 1 10 + fi 11 + PORT=$(cat "$PORTFILE") 12 + BASE="http://localhost:$PORT" 13 + 14 + # Get auth token from the page 15 + HTML=$(curl -s "$BASE/") 16 + TOKEN=$(echo "$HTML" | grep -o 'const TOKEN = "[^"]*"' | sed 's/const TOKEN = "//;s/"//') 17 + 18 + echo "=== Server ===" 19 + echo " URL: $BASE" 20 + echo " Token: ${TOKEN:0:16}..." 21 + 22 + echo "" 23 + echo "=== OAuth Status ===" 24 + curl -s "$BASE/oauth/status" | python3 -m json.tool 25 + 26 + echo "" 27 + echo "=== Overview (key fields) ===" 28 + curl -s -H "Authorization: Bearer $TOKEN" "$BASE/xrpc/org.p2pds.app.getOverview" | python3 -c " 29 + import json, sys 30 + d = json.load(sys.stdin) 31 + if 'error' in d: 32 + print(' ERROR:', d.get('error'), '-', d.get('message')) 33 + else: 34 + print(' did:', d.get('did')) 35 + print(' handle:', d.get('handle')) 36 + print(' replication.enabled:', d.get('replication', {}).get('enabled')) 37 + tracked = d.get('replication', {}).get('trackedDids', []) 38 + print(' trackedDids:', tracked) 39 + sources = d.get('replication', {}).get('didSources', {}) 40 + print(' didSources:', sources) 41 + print(' policies:', d.get('policy', {}).get('policyCount')) 42 + " 43 + 44 + echo "" 45 + echo "=== Page HTML markers ===" 46 + echo " Has TOKEN: $(echo "$HTML" | grep -c 'const TOKEN')" 47 + echo " Has 'Connect an account': $(echo "$HTML" | grep -c 'Connect an account')" 48 + echo " Has refreshAccount: $(echo "$HTML" | grep -c 'refreshAccount')" 49 + echo " Cache-Control: $(curl -sI "$BASE/" | grep -i cache-control || echo 'none')"
+23
scripts/check-js.sh
··· 1 + #!/bin/bash 2 + # Extract the <script> block from the served page and check for JS syntax errors. 3 + # Usage: npm run check-js 4 + PORTFILE="/tmp/p2pds-node1.port" 5 + if [ ! -f "$PORTFILE" ]; then 6 + echo "Node 1 not running. Start with: npm run start:node1" 7 + exit 1 8 + fi 9 + PORT=$(cat "$PORTFILE") 10 + BASE="http://localhost:$PORT" 11 + 12 + # Extract JS between <script> and </script> 13 + curl -s "$BASE/" | sed -n '/<script>/,/<\/script>/p' | sed '1d;$d' > /tmp/p2pds-page-script.js 14 + 15 + echo "Script size: $(wc -c < /tmp/p2pds-page-script.js) bytes, $(wc -l < /tmp/p2pds-page-script.js) lines" 16 + 17 + # Check syntax with Node 18 + node --check /tmp/p2pds-page-script.js 2>&1 19 + if [ $? -eq 0 ]; then 20 + echo "JS syntax: OK" 21 + else 22 + echo "JS syntax: ERRORS FOUND" 23 + fi
+29
scripts/check-token.sh
··· 1 + #!/bin/bash 2 + # Check if the auth token persists across restarts. 3 + # Usage: npm run check-token 4 + # Reads the token from the running node 1, restarts it, and compares. 5 + set -e 6 + 7 + PORTFILE="/tmp/p2pds-node1.port" 8 + if [ ! -f "$PORTFILE" ]; then 9 + echo "Node 1 not running. Start with: npm run start:node1" 10 + exit 1 11 + fi 12 + PORT=$(cat "$PORTFILE") 13 + 14 + TOKEN_BEFORE=$(curl -s "http://localhost:$PORT/" | grep -o 'const TOKEN = "[^"]*"' | sed 's/const TOKEN = "//;s/"//') 15 + echo "Token before restart: ${TOKEN_BEFORE:0:16}..." 16 + 17 + # Restart 18 + npm run start:node1 >/dev/null 2>&1 19 + PORT=$(cat "$PORTFILE") 20 + 21 + TOKEN_AFTER=$(curl -s "http://localhost:$PORT/" | grep -o 'const TOKEN = "[^"]*"' | sed 's/const TOKEN = "//;s/"//') 22 + echo "Token after restart: ${TOKEN_AFTER:0:16}..." 23 + 24 + if [ "$TOKEN_BEFORE" = "$TOKEN_AFTER" ]; then 25 + echo "PASS: Token persisted across restart" 26 + else 27 + echo "FAIL: Token changed on restart" 28 + exit 1 29 + fi
+2 -2
scripts/demo-replication.ts
··· 1 1 /** 2 2 * Demo script: starts two nodes, creates records on Node A, 3 3 * replicates to Node B, then leaves Node B running so you 4 - * can inspect the admin dashboard with real metrics. 4 + * can inspect the app with real metrics. 5 5 * 6 6 * Usage: npx tsx scripts/demo-replication.ts 7 7 */ ··· 199 199 } 200 200 } 201 201 202 - console.log(pc.bold(pc.green(`\n✓ Dashboard ready at: http://127.0.0.1:${PORT_B}/`))); 202 + console.log(pc.bold(pc.green(`\n✓ App ready at: http://127.0.0.1:${PORT_B}/`))); 203 203 console.log(pc.dim(" Auth token: demo-token")); 204 204 console.log(pc.dim(" Press Ctrl+C to stop\n")); 205 205
+13
scripts/open.sh
··· 1 + #!/bin/bash 2 + # Open the running node in the default browser. 3 + # Usage: npm run open [-- 1|2] 4 + NODE=${1:-1} 5 + PORTFILE="/tmp/p2pds-node${NODE}.port" 6 + if [ ! -f "$PORTFILE" ]; then 7 + echo "Node $NODE not running. Start with: npm run start:node${NODE}" 8 + exit 1 9 + fi 10 + PORT=$(cat "$PORTFILE") 11 + URL="http://localhost:$PORT" 12 + echo "$URL" 13 + open "$URL"
+6
scripts/restart.sh
··· 1 + #!/bin/bash 2 + # Restart node 1 (stop + start). Passes args through (e.g. --clean). 3 + # Usage: npm run restart [-- --clean] 4 + set -e 5 + npm run stop 6 + npm run start:node1 -- "$@"
+9
scripts/start-both.sh
··· 1 1 #!/bin/bash 2 2 # Start both p2pds nodes for two-node testing 3 + # Usage: scripts/start-both.sh [--clean] 3 4 set -e 4 5 5 6 # Stop previous instances if pid files exist 6 7 bash scripts/stop-both.sh 2>/dev/null || true 8 + 9 + # Optionally wipe all data 10 + if [ "$1" = "--clean" ]; then 11 + echo "Cleaning node data..." 12 + rm -rf data/pds.db data/pds.db-shm data/pds.db-wal data/blobs data/ipfs 13 + rm -rf data-node2/pds.db data-node2/pds.db-shm data-node2/pds.db-wal data-node2/blobs data-node2/ipfs 14 + echo "Data wiped (kept .env files)" 15 + fi 7 16 8 17 # Build first 9 18 npm run build
+11 -3
scripts/start-node.sh
··· 1 1 #!/bin/bash 2 - # Start a single p2pds node: scripts/start-node.sh [1|2] 2 + # Start a single p2pds node: scripts/start-node.sh [1|2] [--clean] 3 3 set -e 4 4 NODE=${1:-1} 5 5 PIDFILE="/tmp/p2pds-node${NODE}.pid" ··· 13 13 sleep 1 14 14 fi 15 15 16 + # Optionally wipe data 17 + if [ "$2" = "--clean" ] || [ "$1" = "--clean" ]; then 18 + DATADIR="data" 19 + [ "$NODE" = "2" ] && DATADIR="data-node2" 20 + echo "Cleaning $DATADIR..." 21 + rm -rf "$DATADIR/pds.db" "$DATADIR/pds.db-shm" "$DATADIR/pds.db-wal" "$DATADIR/blobs" "$DATADIR/ipfs" 22 + fi 23 + 16 24 npm run build 17 25 18 - # Pick a random high port 19 - PORT=$(python3 -c 'import socket; s=socket.socket(); s.bind(("",0)); print(s.getsockname()[1]); s.close()') 26 + # Fixed high port per node (node1=6700, node2=6701) 27 + PORT=$((6699 + NODE)) 20 28 export PORT 21 29 22 30 if [ "$NODE" = "2" ]; then
+6 -8
src/bidirectional-replication.test.ts
··· 481 481 expect(agreementsA[0]!.effectiveParams.intervalSec).toBe(300); // min(300, 600) 482 482 expect(agreementsA[0]!.effectiveParams.priority).toBe(75); // max(50, 75) 483 483 484 - // Verify policies were created in the policy engine 485 - const policiesA = peA.getPolicies(); 486 - expect(policiesA.length).toBe(1); 487 - expect(policiesA[0]!.id).toBe(`p2p:${DID_NODE_B}`); 488 - expect(policiesA[0]!.replication.minCopies).toBe(3); 484 + // Verify P2P policies were created in the policy engine 485 + const p2pPolicyA = peA.getPolicies().find((p) => p.id === `p2p:${DID_NODE_B}`); 486 + expect(p2pPolicyA).toBeDefined(); 487 + expect(p2pPolicyA!.replication.minCopies).toBe(3); 489 488 490 - const policiesB = peB.getPolicies(); 491 - expect(policiesB.length).toBe(1); 492 - expect(policiesB[0]!.id).toBe(`p2p:${DID_NODE_A}`); 489 + const p2pPolicyB = peB.getPolicies().find((p) => p.id === `p2p:${DID_NODE_A}`); 490 + expect(p2pPolicyB).toBeDefined(); 493 491 }, 30_000); 494 492 }); 495 493
+3 -3
src/config.ts
··· 35 35 RATE_LIMIT_CHALLENGE_PER_MIN: number; 36 36 RATE_LIMIT_MAX_CONNECTIONS: number; 37 37 RATE_LIMIT_FIREHOSE_PER_IP: number; 38 - /** Whether OAuth login is enabled for remote PDS publishing (default false). */ 38 + /** Whether OAuth login is enabled for remote PDS publishing (default true). */ 39 39 OAUTH_ENABLED: boolean; 40 40 /** Public URL of this p2pds instance, used for push notifications between nodes. */ 41 41 PUBLIC_URL: string; ··· 94 94 loadDotEnv(dotenvPath); 95 95 96 96 // Validate required variables (legacy auth fields only required without OAuth) 97 - const oauthEnabled = process.env.OAUTH_ENABLED === "true"; 97 + const oauthEnabled = process.env.OAUTH_ENABLED !== "false"; 98 98 if (!oauthEnabled) { 99 99 const missing: string[] = []; 100 100 for (const key of LEGACY_REQUIRED_KEYS) { ··· 137 137 RATE_LIMIT_CHALLENGE_PER_MIN: parseInt(process.env.RATE_LIMIT_CHALLENGE_PER_MIN ?? "20", 10), 138 138 RATE_LIMIT_MAX_CONNECTIONS: parseInt(process.env.RATE_LIMIT_MAX_CONNECTIONS ?? "100", 10), 139 139 RATE_LIMIT_FIREHOSE_PER_IP: parseInt(process.env.RATE_LIMIT_FIREHOSE_PER_IP ?? "3", 10), 140 - OAUTH_ENABLED: process.env.OAUTH_ENABLED === "true", 140 + OAUTH_ENABLED: process.env.OAUTH_ENABLED !== "false", 141 141 PUBLIC_URL: process.env.PUBLIC_URL || `http://localhost:${parseInt(process.env.PORT ?? "3000", 10)}`, 142 142 }; 143 143 }
+2 -2
src/didless-startup.test.ts
··· 2 2 * Tests for DID-less server startup and lazy identity establishment. 3 3 * 4 4 * Verifies that a p2pds node can start with no DID/SIGNING_KEY, 5 - * serve its dashboard and health check, replicate data, and 5 + * serve its app and health check, replicate data, and 6 6 * establish identity via the node_identity table. 7 7 */ 8 8 ··· 81 81 expect(body.status).toBe("ok"); 82 82 }); 83 83 84 - it("dashboard loads without DID", async () => { 84 + it("app loads without DID", async () => { 85 85 tmpDir = mkdtempSync(join(tmpdir(), "didless-startup-")); 86 86 const config = didlessConfig(tmpDir); 87 87 handle = await startServer(config);
+17
src/oauth/routes.ts
··· 83 83 ), 403); 84 84 } 85 85 86 + // Safety net: purge any leftover data from a previous session 87 + // (e.g. if a disconnect failed partway through) 88 + if (replicationManager) { 89 + try { replicationManager.getSyncStorage().purgeAllData(); } 90 + catch (err) { console.warn("[oauth] Failed to purge leftover sync data:", err instanceof Error ? err.message : String(err)); } 91 + try { replicationManager.getChallengeStorage().purgeAll(); } 92 + catch (err) { console.warn("[oauth] Failed to purge leftover challenge data:", err instanceof Error ? err.message : String(err)); } 93 + try { replicationManager.getPolicyEngine()?.clear(); } 94 + catch (err) { console.warn("[oauth] Failed to purge leftover policies:", err instanceof Error ? err.message : String(err)); } 95 + } 96 + if (ipfsService) { 97 + try { ipfsService.clearBlockstore(); } 98 + catch (err) { console.warn("[oauth] Failed to purge leftover IPFS blocks:", err instanceof Error ? err.message : String(err)); } 99 + } 100 + try { rmSync(join(config.DATA_DIR, "blobs"), { recursive: true, force: true }); } 101 + catch (err) { console.warn("[oauth] Failed to purge leftover blobs:", err instanceof Error ? err.message : String(err)); } 102 + 86 103 // First login: establish node identity 87 104 const isFirstLogin = !config.DID; 88 105 if (isFirstLogin) {
+138 -2
src/policy/engine.test.ts
··· 1 - import { describe, it, expect } from "vitest"; 1 + import { describe, it, expect, beforeEach, afterEach } from "vitest"; 2 + import { mkdtempSync, rmSync } from "node:fs"; 3 + import { tmpdir } from "node:os"; 4 + import { join } from "node:path"; 5 + import Database from "better-sqlite3"; 2 6 3 7 import { PolicyEngine, matchesTarget } from "./engine.js"; 8 + import { PolicyStorage } from "./storage.js"; 4 9 import type { 5 10 Policy, 6 11 PolicySet, ··· 11 16 DEFAULT_REPLICATION, 12 17 DEFAULT_SYNC, 13 18 DEFAULT_RETENTION, 19 + toStoredPolicy, 14 20 } from "./types.js"; 15 - import { mutualAid, saas, groupGovernance } from "./presets.js"; 21 + import { mutualAid, saas, groupGovernance, configArchive, archive } from "./presets.js"; 16 22 17 23 // ============================================ 18 24 // Helpers ··· 722 728 expect(r1.shouldReplicate).toBe(r2.shouldReplicate); 723 729 }); 724 730 }); 731 + 732 + // ============================================ 733 + // PolicyEngine: persistence 734 + // ============================================ 735 + 736 + describe("PolicyEngine: persistence", () => { 737 + let tmpDir: string; 738 + let db: InstanceType<typeof Database>; 739 + let storage: PolicyStorage; 740 + 741 + beforeEach(() => { 742 + tmpDir = mkdtempSync(join(tmpdir(), "policy-engine-persist-test-")); 743 + db = new Database(join(tmpDir, "test.db")); 744 + storage = new PolicyStorage(db); 745 + storage.initSchema(); 746 + }); 747 + 748 + afterEach(() => { 749 + db.close(); 750 + rmSync(tmpDir, { recursive: true, force: true }); 751 + }); 752 + 753 + it("addPolicy auto-persists StoredPolicy to DB", () => { 754 + const engine = new PolicyEngine(undefined, storage); 755 + const policy = configArchive("did:plc:alice"); 756 + engine.addPolicy(policy); 757 + 758 + expect(storage.hasPolicy("config:did:plc:alice")).toBe(true); 759 + const loaded = storage.getPolicy("config:did:plc:alice")!; 760 + expect(loaded.type).toBe("config"); 761 + expect(loaded.state).toBe("active"); 762 + }); 763 + 764 + it("addPolicy does NOT persist plain Policy to DB", () => { 765 + const engine = new PolicyEngine(undefined, storage); 766 + const plain = makePolicy({ id: "plain" }); 767 + engine.addPolicy(plain); 768 + 769 + expect(storage.hasPolicy("plain")).toBe(false); 770 + expect(engine.getPolicies()).toHaveLength(1); 771 + }); 772 + 773 + it("removePolicy auto-deletes from DB", () => { 774 + const engine = new PolicyEngine(undefined, storage); 775 + engine.addPolicy(configArchive("did:plc:alice")); 776 + expect(storage.hasPolicy("config:did:plc:alice")).toBe(true); 777 + 778 + engine.removePolicy("config:did:plc:alice"); 779 + expect(storage.hasPolicy("config:did:plc:alice")).toBe(false); 780 + }); 781 + 782 + it("loadFromDb restores persisted policies", () => { 783 + // Write directly to storage 784 + storage.upsertPolicy(configArchive("did:plc:alice")); 785 + storage.upsertPolicy(archive("did:plc:bob")); 786 + 787 + const engine = new PolicyEngine(undefined, storage); 788 + expect(engine.getPolicies()).toHaveLength(0); 789 + 790 + engine.loadFromDb(); 791 + expect(engine.getPolicies()).toHaveLength(2); 792 + expect(engine.getActiveDids().sort()).toEqual(["did:plc:alice", "did:plc:bob"]); 793 + }); 794 + 795 + it("loadFromDb deduplicates with existing in-memory policies", () => { 796 + const policy = configArchive("did:plc:alice"); 797 + storage.upsertPolicy(policy); 798 + 799 + const engine = new PolicyEngine(undefined, storage); 800 + engine.addPolicy(policy); // already in memory 801 + engine.loadFromDb(); // should not duplicate 802 + 803 + expect(engine.getPolicies()).toHaveLength(1); 804 + }); 805 + 806 + it("persistAll saves all StoredPolicy objects", () => { 807 + const engine = new PolicyEngine(undefined, storage); 808 + // Add without storage (simulate file-loaded policies) 809 + const policy = configArchive("did:plc:alice"); 810 + engine.addPolicy(policy); 811 + 812 + // Verify it was auto-persisted 813 + expect(storage.count()).toBe(1); 814 + 815 + // Add a second one and persistAll 816 + const p2 = archive("did:plc:bob"); 817 + engine.addPolicy(p2); 818 + engine.persistAll(); 819 + expect(storage.count()).toBe(2); 820 + }); 821 + 822 + it("transitionPolicy updates state in memory and DB", () => { 823 + const engine = new PolicyEngine(undefined, storage); 824 + engine.addPolicy(archive("did:plc:alice")); 825 + 826 + const result = engine.transitionPolicy("archive:did:plc:alice", "suspended"); 827 + expect(result).toBe(true); 828 + 829 + // In-memory 830 + const stored = engine.getStoredPolicy("archive:did:plc:alice")!; 831 + expect(stored.state).toBe("suspended"); 832 + expect(stored.suspendedAt).not.toBeNull(); 833 + 834 + // In DB 835 + const dbPolicy = storage.getPolicy("archive:did:plc:alice")!; 836 + expect(dbPolicy.state).toBe("suspended"); 837 + }); 838 + 839 + it("getActiveDids excludes non-active StoredPolicies", () => { 840 + const engine = new PolicyEngine(undefined, storage); 841 + engine.addPolicy(configArchive("did:plc:alice")); 842 + engine.addPolicy(archive("did:plc:bob")); 843 + 844 + engine.transitionPolicy("archive:did:plc:bob", "terminated"); 845 + 846 + const dids = engine.getActiveDids(); 847 + expect(dids).toContain("did:plc:alice"); 848 + expect(dids).not.toContain("did:plc:bob"); 849 + }); 850 + 851 + it("getActiveDids includes plain (non-stored) enabled policies", () => { 852 + const engine = new PolicyEngine(undefined, storage); 853 + engine.addPolicy(makePolicy({ 854 + id: "plain", 855 + target: { type: "list", dids: ["did:plc:plain"] }, 856 + })); 857 + 858 + expect(engine.getActiveDids()).toContain("did:plc:plain"); 859 + }); 860 + });
+121
src/policy/migrate.test.ts
··· 1 + import { describe, it, expect, beforeEach, afterEach } from "vitest"; 2 + import { mkdtempSync, rmSync } from "node:fs"; 3 + import { tmpdir } from "node:os"; 4 + import { join } from "node:path"; 5 + import Database from "better-sqlite3"; 6 + 7 + import { PolicyStorage } from "./storage.js"; 8 + import { PolicyEngine } from "./engine.js"; 9 + import { migrateToNewPolicies } from "./migrate.js"; 10 + import type { StoredPolicy } from "./types.js"; 11 + 12 + describe("migrateToNewPolicies", () => { 13 + let tmpDir: string; 14 + let db: InstanceType<typeof Database>; 15 + let storage: PolicyStorage; 16 + let engine: PolicyEngine; 17 + 18 + beforeEach(() => { 19 + tmpDir = mkdtempSync(join(tmpdir(), "policy-migrate-test-")); 20 + db = new Database(join(tmpDir, "test.db")); 21 + storage = new PolicyStorage(db); 22 + storage.initSchema(); 23 + engine = new PolicyEngine(undefined, storage); 24 + 25 + // Create the admin_tracked_dids table (normally done by SyncStorage) 26 + db.exec(` 27 + CREATE TABLE IF NOT EXISTS admin_tracked_dids ( 28 + did TEXT PRIMARY KEY, 29 + added_at TEXT NOT NULL DEFAULT (datetime('now')) 30 + ) 31 + `); 32 + }); 33 + 34 + afterEach(() => { 35 + db.close(); 36 + rmSync(tmpDir, { recursive: true, force: true }); 37 + }); 38 + 39 + it("migrates config DIDs to config: policies", () => { 40 + const count = migrateToNewPolicies(db, storage, engine, [ 41 + "did:plc:alice", 42 + "did:plc:bob", 43 + ]); 44 + 45 + expect(count).toBe(2); 46 + expect(storage.hasPolicy("config:did:plc:alice")).toBe(true); 47 + expect(storage.hasPolicy("config:did:plc:bob")).toBe(true); 48 + 49 + const policy = storage.getPolicy("config:did:plc:alice")!; 50 + expect(policy.type).toBe("config"); 51 + expect(policy.source).toBe("config"); 52 + expect(policy.state).toBe("active"); 53 + expect(policy.priority).toBe(30); 54 + }); 55 + 56 + it("migrates admin_tracked_dids to archive: policies", () => { 57 + db.prepare("INSERT INTO admin_tracked_dids (did) VALUES (?)").run("did:plc:carol"); 58 + db.prepare("INSERT INTO admin_tracked_dids (did) VALUES (?)").run("did:plc:dave"); 59 + 60 + const count = migrateToNewPolicies(db, storage, engine, []); 61 + 62 + expect(count).toBe(2); 63 + expect(storage.hasPolicy("archive:did:plc:carol")).toBe(true); 64 + expect(storage.hasPolicy("archive:did:plc:dave")).toBe(true); 65 + 66 + const policy = storage.getPolicy("archive:did:plc:carol")!; 67 + expect(policy.type).toBe("archive"); 68 + expect(policy.source).toBe("manual"); 69 + expect(policy.createdBy).toBe("migration"); 70 + expect(policy.priority).toBe(40); 71 + }); 72 + 73 + it("is idempotent: running twice creates no duplicates", () => { 74 + db.prepare("INSERT INTO admin_tracked_dids (did) VALUES (?)").run("did:plc:carol"); 75 + 76 + const count1 = migrateToNewPolicies(db, storage, engine, ["did:plc:alice"]); 77 + expect(count1).toBe(2); 78 + 79 + // Run again — should skip existing 80 + const count2 = migrateToNewPolicies(db, storage, engine, ["did:plc:alice"]); 81 + expect(count2).toBe(0); 82 + 83 + expect(storage.count()).toBe(2); 84 + }); 85 + 86 + it("skips admin DID if config policy already exists for that DID", () => { 87 + // Config DID and admin DID for the same account 88 + db.prepare("INSERT INTO admin_tracked_dids (did) VALUES (?)").run("did:plc:alice"); 89 + 90 + const count = migrateToNewPolicies(db, storage, engine, ["did:plc:alice"]); 91 + 92 + // Should create config: but NOT archive: for the same DID 93 + expect(count).toBe(1); 94 + expect(storage.hasPolicy("config:did:plc:alice")).toBe(true); 95 + expect(storage.hasPolicy("archive:did:plc:alice")).toBe(false); 96 + }); 97 + 98 + it("handles missing admin_tracked_dids table gracefully", () => { 99 + db.exec("DROP TABLE IF EXISTS admin_tracked_dids"); 100 + const count = migrateToNewPolicies(db, storage, engine, ["did:plc:alice"]); 101 + expect(count).toBe(1); 102 + }); 103 + 104 + it("does NOT drop the old admin_tracked_dids table", () => { 105 + db.prepare("INSERT INTO admin_tracked_dids (did) VALUES (?)").run("did:plc:carol"); 106 + 107 + migrateToNewPolicies(db, storage, engine, []); 108 + 109 + // Old table should still exist and have data 110 + const rows = db.prepare("SELECT * FROM admin_tracked_dids").all(); 111 + expect(rows).toHaveLength(1); 112 + }); 113 + 114 + it("policies appear in engine's getActiveDids()", () => { 115 + db.prepare("INSERT INTO admin_tracked_dids (did) VALUES (?)").run("did:plc:carol"); 116 + migrateToNewPolicies(db, storage, engine, ["did:plc:alice"]); 117 + 118 + const activeDids = engine.getActiveDids().sort(); 119 + expect(activeDids).toEqual(["did:plc:alice", "did:plc:carol"]); 120 + }); 121 + });
+63
src/policy/migrate.ts
··· 1 + /** 2 + * One-time migration: convert legacy DID sources to PolicyEngine policies. 3 + * 4 + * - Config DIDs (REPLICATE_DIDS) → `config:{did}` policies 5 + * - Admin-tracked DIDs (admin_tracked_dids table) → `archive:{did}` policies 6 + * 7 + * Idempotent: skips policies that already exist. Does NOT drop old tables. 8 + */ 9 + 10 + import type Database from "better-sqlite3"; 11 + import type { PolicyStorage } from "./storage.js"; 12 + import type { PolicyEngine } from "./engine.js"; 13 + import { configArchive, archive } from "./presets.js"; 14 + 15 + /** 16 + * Migrate legacy DID sources into policy engine policies. 17 + * Returns the number of policies created. 18 + */ 19 + export function migrateToNewPolicies( 20 + db: Database.Database, 21 + policyStorage: PolicyStorage, 22 + policyEngine: PolicyEngine, 23 + configDids: string[], 24 + ): number { 25 + let created = 0; 26 + 27 + // 1. Migrate config DIDs 28 + for (const did of configDids) { 29 + const policyId = `config:${did}`; 30 + if (policyStorage.hasPolicy(policyId)) continue; 31 + if (policyEngine.getPolicies().some((p) => p.id === policyId)) continue; 32 + 33 + const policy = configArchive(did); 34 + policyEngine.addPolicy(policy); 35 + created++; 36 + } 37 + 38 + // 2. Migrate admin-tracked DIDs (if table exists) 39 + try { 40 + const rows = db 41 + .prepare("SELECT did FROM admin_tracked_dids ORDER BY added_at") 42 + .all() as Array<{ did: string }>; 43 + 44 + for (const row of rows) { 45 + const policyId = `archive:${row.did}`; 46 + if (policyStorage.hasPolicy(policyId)) continue; 47 + if (policyEngine.getPolicies().some((p) => p.id === policyId)) continue; 48 + 49 + // Skip if there's already a config policy for this DID 50 + const configPolicyId = `config:${row.did}`; 51 + if (policyStorage.hasPolicy(configPolicyId)) continue; 52 + if (policyEngine.getPolicies().some((p) => p.id === configPolicyId)) continue; 53 + 54 + const policy = archive(row.did, { createdBy: "migration" }); 55 + policyEngine.addPolicy(policy); 56 + created++; 57 + } 58 + } catch { 59 + // admin_tracked_dids table may not exist yet — that's fine 60 + } 61 + 62 + return created; 63 + }
+59 -1
src/policy/presets.ts
··· 6 6 * write policies from scratch. 7 7 */ 8 8 9 - import type { Policy } from "./types.js"; 9 + import type { Policy, StoredPolicy } from "./types.js"; 10 10 import { 11 11 DEFAULT_REPLICATION, 12 12 DEFAULT_SYNC, 13 13 DEFAULT_RETENTION, 14 14 DEFAULT_PRIORITY, 15 + toStoredPolicy, 15 16 } from "./types.js"; 16 17 17 18 // ============================================ ··· 161 162 enabled: true, 162 163 }; 163 164 } 165 + 166 + // ============================================ 167 + // configArchive: policy for a config-origin DID 168 + // ============================================ 169 + 170 + /** 171 + * Create a StoredPolicy for a DID originating from REPLICATE_DIDS config. 172 + * ID: `config:{did}`, type: config, priority 30. 173 + */ 174 + export function configArchive(did: string): StoredPolicy { 175 + const base: Policy = { 176 + id: `config:${did}`, 177 + name: `Config: ${did}`, 178 + target: { type: "list", dids: [did] }, 179 + replication: { ...DEFAULT_REPLICATION }, 180 + sync: { ...DEFAULT_SYNC }, 181 + retention: { ...DEFAULT_RETENTION }, 182 + priority: 30, 183 + enabled: true, 184 + }; 185 + return toStoredPolicy(base, "config", { source: "config", consent: "unconsented" }); 186 + } 187 + 188 + // ============================================ 189 + // archive: policy for a user-added DID 190 + // ============================================ 191 + 192 + export interface ArchiveOptions { 193 + /** Priority. Default: 40. */ 194 + priority?: number; 195 + /** Sync interval in seconds. Default: 300. */ 196 + intervalSec?: number; 197 + /** Who created the policy. Default: "user". */ 198 + createdBy?: string; 199 + } 200 + 201 + /** 202 + * Create a StoredPolicy for a user-added (archive) DID. 203 + * ID: `archive:{did}`, type: archive, priority 40. 204 + */ 205 + export function archive(did: string, opts?: ArchiveOptions): StoredPolicy { 206 + const base: Policy = { 207 + id: `archive:${did}`, 208 + name: `Archive: ${did}`, 209 + target: { type: "list", dids: [did] }, 210 + replication: { ...DEFAULT_REPLICATION }, 211 + sync: { intervalSec: opts?.intervalSec ?? DEFAULT_SYNC.intervalSec }, 212 + retention: { ...DEFAULT_RETENTION }, 213 + priority: opts?.priority ?? 40, 214 + enabled: true, 215 + }; 216 + return toStoredPolicy(base, "archive", { 217 + source: "manual", 218 + consent: "unconsented", 219 + createdBy: opts?.createdBy ?? "user", 220 + }); 221 + }
+205
src/policy/storage.test.ts
··· 1 + import { describe, it, expect, beforeEach, afterEach } from "vitest"; 2 + import { mkdtempSync, rmSync } from "node:fs"; 3 + import { tmpdir } from "node:os"; 4 + import { join } from "node:path"; 5 + import Database from "better-sqlite3"; 6 + 7 + import { PolicyStorage } from "./storage.js"; 8 + import type { StoredPolicy } from "./types.js"; 9 + import { toStoredPolicy } from "./types.js"; 10 + import { 11 + DEFAULT_REPLICATION, 12 + DEFAULT_SYNC, 13 + DEFAULT_RETENTION, 14 + } from "./types.js"; 15 + import type { Policy } from "./types.js"; 16 + 17 + function makePolicy(overrides: Partial<Policy> & { id: string }): Policy { 18 + return { 19 + name: overrides.id, 20 + target: { type: "list", dids: ["did:plc:test"] }, 21 + replication: { ...DEFAULT_REPLICATION }, 22 + sync: { ...DEFAULT_SYNC }, 23 + retention: { ...DEFAULT_RETENTION }, 24 + priority: 50, 25 + enabled: true, 26 + ...overrides, 27 + }; 28 + } 29 + 30 + function makeStoredPolicy(id: string, did: string = "did:plc:test"): StoredPolicy { 31 + return toStoredPolicy( 32 + makePolicy({ id, target: { type: "list", dids: [did] } }), 33 + "archive", 34 + { source: "manual", createdBy: "test" }, 35 + ); 36 + } 37 + 38 + describe("PolicyStorage", () => { 39 + let tmpDir: string; 40 + let db: InstanceType<typeof Database>; 41 + let storage: PolicyStorage; 42 + 43 + beforeEach(() => { 44 + tmpDir = mkdtempSync(join(tmpdir(), "policy-storage-test-")); 45 + db = new Database(join(tmpDir, "test.db")); 46 + storage = new PolicyStorage(db); 47 + storage.initSchema(); 48 + }); 49 + 50 + afterEach(() => { 51 + db.close(); 52 + rmSync(tmpDir, { recursive: true, force: true }); 53 + }); 54 + 55 + describe("round-trip", () => { 56 + it("upsert and load a policy", () => { 57 + const policy = makeStoredPolicy("test-1", "did:plc:alice"); 58 + storage.upsertPolicy(policy); 59 + 60 + const loaded = storage.getPolicy("test-1"); 61 + expect(loaded).not.toBeNull(); 62 + expect(loaded!.id).toBe("test-1"); 63 + expect(loaded!.type).toBe("archive"); 64 + expect(loaded!.state).toBe("active"); 65 + expect(loaded!.source).toBe("manual"); 66 + expect(loaded!.consent).toBe("unconsented"); 67 + expect(loaded!.createdBy).toBe("test"); 68 + expect(loaded!.target).toEqual({ type: "list", dids: ["did:plc:alice"] }); 69 + expect(loaded!.replication).toEqual(DEFAULT_REPLICATION); 70 + expect(loaded!.sync).toEqual(DEFAULT_SYNC); 71 + expect(loaded!.retention).toEqual(DEFAULT_RETENTION); 72 + expect(loaded!.priority).toBe(50); 73 + expect(loaded!.enabled).toBe(true); 74 + }); 75 + 76 + it("upsert updates an existing policy", () => { 77 + const policy = makeStoredPolicy("test-1"); 78 + storage.upsertPolicy(policy); 79 + 80 + const updated = { ...policy, name: "Updated Name", priority: 99 }; 81 + storage.upsertPolicy(updated); 82 + 83 + const loaded = storage.getPolicy("test-1"); 84 + expect(loaded!.name).toBe("Updated Name"); 85 + expect(loaded!.priority).toBe(99); 86 + }); 87 + 88 + it("preserves JSON fields correctly", () => { 89 + const policy = toStoredPolicy( 90 + makePolicy({ 91 + id: "json-test", 92 + replication: { minCopies: 3, preferredPeers: ["did:plc:peer1"] }, 93 + retention: { maxAgeSec: 86400, keepHistory: true }, 94 + }), 95 + "saas", 96 + ); 97 + storage.upsertPolicy(policy); 98 + 99 + const loaded = storage.getPolicy("json-test")!; 100 + expect(loaded.replication.minCopies).toBe(3); 101 + expect(loaded.replication.preferredPeers).toEqual(["did:plc:peer1"]); 102 + expect(loaded.retention.maxAgeSec).toBe(86400); 103 + expect(loaded.retention.keepHistory).toBe(true); 104 + }); 105 + }); 106 + 107 + describe("loadPolicies", () => { 108 + it("loads all policies when no state filter", () => { 109 + storage.upsertPolicy(makeStoredPolicy("p1")); 110 + storage.upsertPolicy(makeStoredPolicy("p2")); 111 + const all = storage.loadPolicies(); 112 + expect(all).toHaveLength(2); 113 + }); 114 + 115 + it("filters by state", () => { 116 + const active = makeStoredPolicy("active"); 117 + const suspended = { ...makeStoredPolicy("suspended"), state: "suspended" as const }; 118 + storage.upsertPolicy(active); 119 + storage.upsertPolicy(suspended); 120 + 121 + const result = storage.loadPolicies(["active"]); 122 + expect(result).toHaveLength(1); 123 + expect(result[0]!.id).toBe("active"); 124 + }); 125 + 126 + it("loadActivePolicies returns only active", () => { 127 + storage.upsertPolicy(makeStoredPolicy("a1")); 128 + storage.upsertPolicy({ ...makeStoredPolicy("t1"), state: "terminated" }); 129 + const active = storage.loadActivePolicies(); 130 + expect(active).toHaveLength(1); 131 + expect(active[0]!.id).toBe("a1"); 132 + }); 133 + }); 134 + 135 + describe("state transitions", () => { 136 + it("transitions active → suspended", () => { 137 + storage.upsertPolicy(makeStoredPolicy("p1")); 138 + const result = storage.transitionState("p1", "suspended"); 139 + expect(result).toBe(true); 140 + 141 + const loaded = storage.getPolicy("p1")!; 142 + expect(loaded.state).toBe("suspended"); 143 + expect(loaded.suspendedAt).not.toBeNull(); 144 + }); 145 + 146 + it("transitions to terminated sets terminatedAt", () => { 147 + storage.upsertPolicy(makeStoredPolicy("p1")); 148 + storage.transitionState("p1", "terminated"); 149 + const loaded = storage.getPolicy("p1")!; 150 + expect(loaded.state).toBe("terminated"); 151 + expect(loaded.terminatedAt).not.toBeNull(); 152 + }); 153 + 154 + it("transitions to active sets activatedAt", () => { 155 + const policy = { ...makeStoredPolicy("p1"), state: "suspended" as const }; 156 + storage.upsertPolicy(policy); 157 + storage.transitionState("p1", "active"); 158 + const loaded = storage.getPolicy("p1")!; 159 + expect(loaded.state).toBe("active"); 160 + expect(loaded.activatedAt).not.toBeNull(); 161 + }); 162 + 163 + it("returns false for non-existent policy", () => { 164 + expect(storage.transitionState("nope", "suspended")).toBe(false); 165 + }); 166 + }); 167 + 168 + describe("deletePolicy", () => { 169 + it("deletes and returns true", () => { 170 + storage.upsertPolicy(makeStoredPolicy("p1")); 171 + expect(storage.deletePolicy("p1")).toBe(true); 172 + expect(storage.getPolicy("p1")).toBeNull(); 173 + }); 174 + 175 + it("returns false for non-existent", () => { 176 + expect(storage.deletePolicy("nope")).toBe(false); 177 + }); 178 + }); 179 + 180 + describe("hasPolicy", () => { 181 + it("returns true for existing", () => { 182 + storage.upsertPolicy(makeStoredPolicy("p1")); 183 + expect(storage.hasPolicy("p1")).toBe(true); 184 + }); 185 + 186 + it("returns false for non-existent", () => { 187 + expect(storage.hasPolicy("nope")).toBe(false); 188 + }); 189 + }); 190 + 191 + describe("count", () => { 192 + it("counts all policies", () => { 193 + storage.upsertPolicy(makeStoredPolicy("p1")); 194 + storage.upsertPolicy(makeStoredPolicy("p2")); 195 + expect(storage.count()).toBe(2); 196 + }); 197 + 198 + it("counts by state", () => { 199 + storage.upsertPolicy(makeStoredPolicy("p1")); 200 + storage.upsertPolicy({ ...makeStoredPolicy("p2"), state: "terminated" }); 201 + expect(storage.count("active")).toBe(1); 202 + expect(storage.count("terminated")).toBe(1); 203 + }); 204 + }); 205 + });
+77
src/policy/types.ts
··· 137 137 }; 138 138 139 139 export const DEFAULT_PRIORITY = 50; 140 + 141 + // ============================================ 142 + // Lifecycle types for persistent policies 143 + // ============================================ 144 + 145 + /** Policy lifecycle state. */ 146 + export type PolicyState = "proposed" | "active" | "suspended" | "terminated" | "purged"; 147 + 148 + /** What kind of policy this is. */ 149 + export type PolicyType = "reciprocal" | "archive" | "config" | "saas" | "group"; 150 + 151 + /** How this policy was created. */ 152 + export type PolicySource = "config" | "file" | "offer" | "manual" | "api"; 153 + 154 + /** Consent status for the replication relationship. */ 155 + export type ConsentStatus = "reciprocal" | "consented" | "unconsented" | "revoked"; 156 + 157 + /** 158 + * A policy with lifecycle metadata for persistence. 159 + * Extends the base Policy with type, state, source, consent, and timestamps. 160 + */ 161 + export interface StoredPolicy extends Policy { 162 + /** What kind of policy this is. */ 163 + type: PolicyType; 164 + /** Current lifecycle state. */ 165 + state: PolicyState; 166 + /** How this policy was created. */ 167 + source: PolicySource; 168 + /** Consent status. */ 169 + consent: ConsentStatus; 170 + /** Who/what created this policy. */ 171 + createdBy: string; 172 + /** The counterparty DID (for reciprocal/P2P policies). */ 173 + counterpartyDid?: string; 174 + /** URI of the local offer record (for P2P policies). */ 175 + localOfferUri?: string; 176 + /** URI of the remote offer record (for P2P policies). */ 177 + remoteOfferUri?: string; 178 + /** When this policy was created. */ 179 + createdAt: string; 180 + /** When this policy was activated. */ 181 + activatedAt: string | null; 182 + /** When this policy was suspended. */ 183 + suspendedAt: string | null; 184 + /** When this policy was terminated. */ 185 + terminatedAt: string | null; 186 + /** When this policy expires (null = never). */ 187 + expiresAt: string | null; 188 + } 189 + 190 + /** 191 + * Create a StoredPolicy from a base Policy with lifecycle metadata. 192 + * `type` is required (no inference from ID prefix). 193 + */ 194 + export function toStoredPolicy( 195 + policy: Policy, 196 + type: PolicyType, 197 + overrides?: Partial<Omit<StoredPolicy, keyof Policy | "type">>, 198 + ): StoredPolicy { 199 + const now = new Date().toISOString(); 200 + return { 201 + ...policy, 202 + type, 203 + state: overrides?.state ?? "active", 204 + source: overrides?.source ?? "manual", 205 + consent: overrides?.consent ?? "unconsented", 206 + createdBy: overrides?.createdBy ?? "system", 207 + counterpartyDid: overrides?.counterpartyDid, 208 + localOfferUri: overrides?.localOfferUri, 209 + remoteOfferUri: overrides?.remoteOfferUri, 210 + createdAt: overrides?.createdAt ?? now, 211 + activatedAt: overrides?.activatedAt ?? (overrides?.state === "active" || !overrides?.state ? now : null), 212 + suspendedAt: overrides?.suspendedAt ?? null, 213 + terminatedAt: overrides?.terminatedAt ?? null, 214 + expiresAt: overrides?.expiresAt ?? null, 215 + }; 216 + }
+24 -29
src/replication/policy-integration.test.ts
··· 19 19 import { RepoManager } from "../repo-manager.js"; 20 20 import type { Config } from "../config.js"; 21 21 import { PolicyEngine } from "../policy/engine.js"; 22 - import { mutualAid, saas } from "../policy/presets.js"; 22 + import { mutualAid, saas, configArchive } from "../policy/presets.js"; 23 23 import type { Policy, PolicySet } from "../policy/types.js"; 24 24 import { 25 25 DEFAULT_REPLICATION, ··· 136 136 ]); 137 137 }); 138 138 139 - it("with PolicyEngine, merges config DIDs and policy explicit DIDs", () => { 139 + it("with PolicyEngine, merges config and policy explicit DIDs", () => { 140 140 const config = testConfig(tmpDir, ["did:plc:config1"]); 141 - const engine = new PolicyEngine( 142 - makePolicySet([ 143 - makePolicy({ 144 - id: "p1", 145 - target: { type: "list", dids: ["did:plc:policy1", "did:plc:policy2"] }, 146 - }), 147 - ]), 148 - ); 141 + const engine = new PolicyEngine(); 142 + // Simulate migration: config DID becomes a config: policy 143 + engine.addPolicy(configArchive("did:plc:config1")); 144 + engine.addPolicy(makePolicy({ 145 + id: "p1", 146 + target: { type: "list", dids: ["did:plc:policy1", "did:plc:policy2"] }, 147 + })); 149 148 150 149 const rm = new ReplicationManager( 151 150 db, ··· 167 166 ]); 168 167 }); 169 168 170 - it("deduplicates DIDs present in both config and policy", () => { 169 + it("deduplicates DIDs present in both config policy and other policy", () => { 171 170 const config = testConfig(tmpDir, ["did:plc:shared", "did:plc:config-only"]); 172 - const engine = new PolicyEngine( 173 - makePolicySet([ 174 - makePolicy({ 175 - id: "p1", 176 - target: { type: "list", dids: ["did:plc:shared", "did:plc:policy-only"] }, 177 - }), 178 - ]), 179 - ); 171 + const engine = new PolicyEngine(); 172 + engine.addPolicy(configArchive("did:plc:shared")); 173 + engine.addPolicy(configArchive("did:plc:config-only")); 174 + engine.addPolicy(makePolicy({ 175 + id: "p1", 176 + target: { type: "list", dids: ["did:plc:shared", "did:plc:policy-only"] }, 177 + })); 180 178 181 179 const rm = new ReplicationManager( 182 180 db, ··· 198 196 ]); 199 197 }); 200 198 201 - it("config DIDs are always included even without matching policy", () => { 199 + it("config: policies are always included via PolicyEngine", () => { 202 200 const config = testConfig(tmpDir, ["did:plc:config-only"]); 203 - // Policy only targets did:plc:policy-only, not did:plc:config-only 204 - const engine = new PolicyEngine( 205 - makePolicySet([ 206 - makePolicy({ 207 - id: "p1", 208 - target: { type: "list", dids: ["did:plc:policy-only"] }, 209 - }), 210 - ]), 211 - ); 201 + const engine = new PolicyEngine(); 202 + engine.addPolicy(configArchive("did:plc:config-only")); 203 + engine.addPolicy(makePolicy({ 204 + id: "p1", 205 + target: { type: "list", dids: ["did:plc:policy-only"] }, 206 + })); 212 207 213 208 const rm = new ReplicationManager( 214 209 db,
+10
src/replication/types.ts
··· 6 6 export const PEER_NSID = "org.p2pds.peer"; 7 7 export const MANIFEST_NSID = "org.p2pds.manifest"; 8 8 export const OFFER_NSID = "org.p2pds.replication.offer"; 9 + export const CONSENT_NSID = "org.p2pds.replication.consent"; 10 + 11 + /** Open consent record — declares that anyone may replicate this account's data. */ 12 + export interface ConsentRecord { 13 + $type: typeof CONSENT_NSID; 14 + /** Scope of consent. Currently only "any" is supported. */ 15 + scope: "any"; 16 + /** ISO 8601 timestamp. */ 17 + createdAt: string; 18 + } 9 19 10 20 /** Peer identity record — binds a DID to an IPFS PeerID. */ 11 21 export interface PeerIdentityRecord {
+1 -1
src/server-startup.test.ts
··· 82 82 expect(body.version).toBeTruthy(); 83 83 }); 84 84 85 - it("dashboard HTML returns 200 with expected content", async () => { 85 + it("app HTML returns 200 with expected content", async () => { 86 86 tmpDir = mkdtempSync(join(tmpdir(), "server-startup-")); 87 87 const config = testConfig(tmpDir); 88 88 handle = await startServer(config);
+2 -2
src/sidecar-process.test.ts
··· 106 106 } 107 107 }); 108 108 109 - it("full sidecar lifecycle: ready → health → dashboard → shutdown", async () => { 109 + it("full sidecar lifecycle: ready → health → app → shutdown", async () => { 110 110 tmpDir = mkdtempSync(join(tmpdir(), "sidecar-test-")); 111 111 112 112 proc = spawn("npx", ["tsx", "src/server.ts"], { ··· 148 148 const healthBody = (await healthRes.json()) as { status: string }; 149 149 expect(healthBody.status).toBe("ok"); 150 150 151 - // 3. Dashboard returns HTML 151 + // 3. App returns HTML 152 152 const dashRes = await fetch(`${ready.url}/`); 153 153 expect(dashRes.status).toBe(200); 154 154 const html = await dashRes.text();
+6 -6
src/xrpc/app-e2e.test.ts
··· 1 1 /** 2 2 * E2E test: two real HTTP servers, replication between them, 3 - * then verify admin dashboard and APIs show accurate live state. 3 + * then verify app and APIs show accurate live state. 4 4 * 5 5 * Node A — "source PDS": has a repo with records, serves getRepo over HTTP. 6 - * Node B — "replicator": replicates Node A's DID, runs the admin dashboard. 6 + * Node B — "replicator": replicates Node A's DID, runs the app. 7 7 */ 8 8 9 9 import { describe, it, expect, beforeEach, afterEach } from "vitest"; ··· 92 92 93 93 const NODE_A_DID = "did:plc:nodea123"; 94 94 95 - describe("Admin E2E: two-node replication + dashboard", () => { 95 + describe("App E2E: two-node replication + app", () => { 96 96 let tmpDir: string; 97 97 let dbA: InstanceType<typeof Database>; 98 98 let dbB: InstanceType<typeof Database>; ··· 125 125 const appA = createApp(configA, firehoseA, undefined, undefined, undefined, undefined, undefined, repoManagerA); 126 126 ({ server: serverA, port: portA } = await startServer(appA)); 127 127 128 - // ---- Node B: replicator with admin dashboard ---- 128 + // ---- Node B: replicator with app ---- 129 129 dbB = new Database(join(tmpDir, "b.db")); 130 130 const configB = makeConfig(join(tmpDir, "b-data"), "did:plc:nodeb456", [ 131 131 NODE_A_DID, ··· 272 272 expect(syncState.status).toBe("synced"); 273 273 }); 274 274 275 - it("dashboard returns HTML with expected structure", async () => { 276 - // Dashboard doesn't require auth header 275 + it("app returns HTML with expected structure", async () => { 276 + // App doesn't require auth header 277 277 const res = await fetch( 278 278 `http://127.0.0.1:${portB}/`, 279 279 );
+12 -12
src/xrpc/app.test.ts
··· 125 125 // getOverview 126 126 // ============================================ 127 127 128 - describe("Admin: getOverview", () => { 128 + describe("App: getOverview", () => { 129 129 let tmpDir: string; 130 130 let db: InstanceType<typeof Database>; 131 131 ··· 241 241 // getDidStatus 242 242 // ============================================ 243 243 244 - describe("Admin: getDidStatus", () => { 244 + describe("App: getDidStatus", () => { 245 245 let tmpDir: string; 246 246 let db: InstanceType<typeof Database>; 247 247 let config: Config; ··· 357 357 // getNetworkStatus 358 358 // ============================================ 359 359 360 - describe("Admin: getNetworkStatus", () => { 360 + describe("App: getNetworkStatus", () => { 361 361 let tmpDir: string; 362 362 let db: InstanceType<typeof Database>; 363 363 ··· 434 434 // getPolicies 435 435 // ============================================ 436 436 437 - describe("Admin: getPolicies", () => { 437 + describe("App: getPolicies", () => { 438 438 let tmpDir: string; 439 439 let db: InstanceType<typeof Database>; 440 440 ··· 591 591 }); 592 592 593 593 // ============================================ 594 - // getDashboard 594 + // getApp 595 595 // ============================================ 596 596 597 - describe("Admin: getDashboard", () => { 597 + describe("App: getApp", () => { 598 598 let tmpDir: string; 599 599 let db: InstanceType<typeof Database>; 600 600 601 601 beforeEach(() => { 602 - tmpDir = mkdtempSync(join(tmpdir(), "admin-dashboard-test-")); 602 + tmpDir = mkdtempSync(join(tmpdir(), "admin-app-test-")); 603 603 db = new Database(join(tmpDir, "test.db")); 604 604 }); 605 605 ··· 608 608 try { rmSync(tmpDir, { recursive: true, force: true }); } catch {} 609 609 }); 610 610 611 - it("returns HTML dashboard with expected structure", async () => { 611 + it("returns HTML app with expected structure", async () => { 612 612 const config = testConfig(tmpDir); 613 613 const repoManager = new RepoManager(db, config); 614 614 repoManager.init(); ··· 639 639 // addDid / removeDid 640 640 // ============================================ 641 641 642 - describe("Admin: addDid / removeDid", () => { 642 + describe("App: addDid / removeDid", () => { 643 643 let tmpDir: string; 644 644 let db: InstanceType<typeof Database>; 645 645 let config: Config; ··· 743 743 const syncStorage = replicationManager.getSyncStorage(); 744 744 expect(syncStorage.isAdminDid(newDid)).toBe(true); 745 745 expect(replicationManager.getReplicateDids()).toContain(newDid); 746 - expect(replicationManager.getDidSource(newDid)).toBe("admin"); 746 + expect(replicationManager.getDidSource(newDid)).toBe("user"); 747 747 }); 748 748 749 749 it("idempotent: re-adding returns already_tracked", async () => { ··· 753 753 expect(res.status).toBe(200); 754 754 const json = await res.json() as Record<string, unknown>; 755 755 expect(json.status).toBe("already_tracked"); 756 - expect(json.source).toBe("admin"); 756 + expect(json.source).toBe("user"); 757 757 }); 758 758 759 759 it("cannot remove a config DID", async () => { ··· 809 809 syncStorage.addAdminDid("did:plc:adminadded1"); 810 810 811 811 expect(replicationManager.getDidSource(configDid)).toBe("config"); 812 - expect(replicationManager.getDidSource("did:plc:adminadded1")).toBe("admin"); 812 + expect(replicationManager.getDidSource("did:plc:adminadded1")).toBe("user"); 813 813 expect(replicationManager.getDidSource("did:plc:unknown")).toBeNull(); 814 814 }); 815 815 });