forked from
tranquil.farm/tranquil-pds
Our Personal Data Server from scratch!
Update TODO, more logging during blob migration
+190
-31
+129
-22
TODO.md
+129
-22
TODO.md
···
2
2
3
3
## Active development
4
4
5
+
### Storage backend abstraction
6
+
Make storage layers swappable via traits.
7
+
8
+
filesystem blob storage
9
+
- [ ] FilesystemBlobStorage implementation
10
+
- [ ] directory structure (content-addressed like blobs/{cid} already used in objsto)
11
+
- [ ] atomic writes (write to temp, rename)
12
+
- [ ] config option to choose backend (env var or config flag)
13
+
- [ ] also traitify BackupStorage (currently hardcoded to objsto)
14
+
15
+
sqlite database backend
16
+
- [ ] abstract db layer behind trait (queries, transactions, migrations)
17
+
- [ ] sqlite implementation matching postgres behavior
18
+
- [ ] handle sqlite's single-writer limitation (connection pooling strategy)
19
+
- [ ] migrations system that works for both
20
+
- [ ] testing: run full test suite against both backends
21
+
- [ ] config option to choose backend (postgres vs sqlite)
22
+
- [ ] document tradeoffs (sqlite for single-user/small, postgres for multi-user/scale)
23
+
5
24
### Plugin system
6
-
Extensible architecture allowing third-party plugins to add functionality. Going with wasm-based rather than scripting language.
25
+
WASM component model plugins. Compile to wasm32-wasip2, sandboxed via wasmtime, capability-gated. Based on zed's extensions.
26
+
27
+
WIT interface
28
+
- [ ] record hooks before/after create, update, delete
29
+
- [ ] blob hooks before/after upload, validate
30
+
- [ ] xrpc hooks before/after (middleware), custom endpoint handler
31
+
- [ ] firehose hook on_commit
32
+
- [ ] host imports http client, kv store, logging, read records
33
+
34
+
wasmtime host
35
+
- [ ] engine with epoch interruption (kill runaway plugins)
36
+
- [ ] plugin manifest (plugin.toml): id, version, capabilities, hooks
37
+
- [ ] capability enforcement at runtime
38
+
- [ ] plugin loader, lifecycle (enable/disable/reload)
39
+
- [ ] resource limits (memory, time)
40
+
- [ ] per-plugin fs sandbox
41
+
42
+
capabilities
43
+
- [ ] http:fetch with domain allowlist
44
+
- [ ] kv:read, kv:write
45
+
- [ ] record:read, blob:read
46
+
- [ ] xrpc:register
47
+
- [ ] firehose:subscribe
48
+
49
+
pds-plugin-api (rust), MVP for plugin system
50
+
- [ ] plugin trait with default impls
51
+
- [ ] register_plugin! macro
52
+
- [ ] typed host import wrappers
53
+
- [ ] publish to crates.io
54
+
- [ ] docs + example
55
+
56
+
pds-plugin-api in golang, nice to have after the fact
57
+
- [ ] wit-bindgen-go bindings
58
+
- [ ] go wrappers
59
+
- [ ] tinygo build instructions
60
+
- [ ] example
61
+
62
+
@pds/plugin-api in typescript, nice to have after the fact
63
+
- [ ] jco/componentize-js bindings
64
+
- [ ] typeScript types
65
+
- [ ] build tooling
66
+
- [ ] example
67
+
68
+
example plugins
69
+
- [ ] content filter
70
+
- [ ] webhook notifier
71
+
- [ ] objsto backup mirror
72
+
- [ ] custom lexicon handler
73
+
- [ ] better audit logger
74
+
75
+
### Misc
76
+
77
+
migration handle preservation
78
+
- [ ] allow users to keep their existing handle during migration (eg. lewis.moe instead of forcing lewis.newpds.com)
79
+
- [ ] UI option to preserve external handle vs create new pds-subdomain handle
80
+
- [ ] handle the DNS verification flow for external handles during migration
81
+
82
+
cross-pds delegation
83
+
when a client (eg. tangled.org) tries to log into a delegated account:
84
+
- [ ] client starts oauth flow to delegated account's pds
85
+
- [ ] delegated pds sees account is externally controlled, launches oauth to controller's pds (delegated pds acts as oauth client)
86
+
- [ ] controller authenticates at their own pds
87
+
- [ ] delegated pds verifies controller perms and scope from its local delegation grants
88
+
- [ ] delegated pds issues session to client within the intersection of controller's granted scope and client's requested scope
89
+
90
+
per-request "act as"
91
+
- [ ] authed as user X, perform action as delegated user Y in single request
92
+
- [ ] approach decision
93
+
- [ ] option 1: `X-Act-As` header with target did, server verifies delegation grant
94
+
- [ ] option 2: token exchange (RFC 8693) for short-lived delegated token
95
+
- [ ] option 3 (lewis fav): extend existing `act` claim to support on-demand minting
96
+
- [ ] something else?
97
+
98
+
### Private/encrypted data
99
+
Records only authorized parties can see and decrypt.
100
+
101
+
research
102
+
- [ ] survey atproto discourse on private data
103
+
- [ ] document bluesky team's likely approach. wait.. are they even gonna do this? whatever
104
+
- [ ] look at matrix/signal for federated e2ee patterns
105
+
106
+
key management
107
+
- [ ] db schema for encryption keys (user_keys, key_grants, key_rotations)
108
+
- [ ] per-user encryption keypair generation (separate from signing keys)
109
+
- [ ] key derivation scheme (per-collection? per-record? both?)
110
+
- [ ] key storage (encrypted at rest, hsm option?)
111
+
- [ ] rotation and revocation flow
112
+
113
+
storage layer
114
+
- [ ] encrypted record format (encrypted cbor blob + metadata)
115
+
- [ ] collection-level vs per-record encryption flag
116
+
- [ ] how encrypted records appear in mst (hash of ciphertext? separate tree?)
117
+
- [ ] blob encryption (same keys? separate?)
118
+
119
+
api surface
120
+
- [ ] xrpc getPublicKey, grantAccess, revokeAccess, listGrants
121
+
- [ ] xrpc getEncryptedRecord (ciphertext for client-side decrypt)
122
+
- [ ] or transparent server-side decrypt if requester has grant?
123
+
- [ ] lexicon for key grant records
7
124
8
-
- [ ] Plugin manifest format (name, version, deps, permissions, hooks)
9
-
- [ ] Plugin loading and lifecycle (enable/disable/hot reload)
10
-
- [ ] WASM host bindings for PDS APIs (database, storage, http, etc.)
11
-
- [ ] Resource limits (memory, cpu time, capability restrictions)
12
-
- [ ] Extension points: request middleware, record lifecycle hooks, custom XRPC endpoints
13
-
- [ ] Extension points: custom lexicons, storage backends, auth providers, notification channels
14
-
- [ ] Extension points: firehose consumers (react to repo events)
15
-
- [ ] Plugin sdk crate with traits and helpers?
16
-
- [ ] Example plugins: cdc, extra logging to 3rd party, content filter, better S3 backup
17
-
- [ ] Plugin registry with signature verification?
125
+
sync/federation
126
+
- [ ] how encrypted records appear on firehose (ciphertext? omitted? placeholder?)
127
+
- [ ] pds-to-pds key exchange protocol
128
+
- [ ] appview behavior (can't index without grants)
129
+
- [ ] relay behavior with encrypted commits
18
130
19
-
### Plugin: Private/encrypted data
20
-
Records that only authorized parties can see and decrypt. Requires key federation between PDSes. Implemented as a plugin using the plugin system above.
131
+
client integration
132
+
- [ ] client-side encryption (pds never sees plaintext) vs server-side with trust
133
+
- [ ] key backup/recovery (lose key = lose data)
21
134
22
-
- [ ] Survey current ATProto discourse on private data
23
-
- [ ] Document Bluesky team's likely approach
24
-
- [ ] Design key management strategy
25
-
- [ ] Per-user encryption keys (separate from signing keys)
26
-
- [ ] Key derivation for per-record or per-collection encryption
27
-
- [ ] Encrypted record storage format
28
-
- [ ] Transparent encryption/decryption in repo operations
29
-
- [ ] Protocol for sharing decryption keys between PDSes
30
-
- [ ] Handle key rotation and revocation
135
+
plugin hooks (once core exists)
136
+
- [ ] on_access_grant_request for custom authorization
137
+
- [ ] on_key_rotation to notify interested parties
31
138
32
139
---
33
140
+4
frontend/src/lib/migration/atproto-client.ts
+4
frontend/src/lib/migration/atproto-client.ts
+14
-5
frontend/src/lib/migration/blob-migration.ts
+14
-5
frontend/src/lib/migration/blob-migration.ts
···
20
20
console.log("[blob-migration] Starting blob migration for", userDid);
21
21
console.log(
22
22
"[blob-migration] Source client:",
23
-
sourceClient ? "available" : "NOT AVAILABLE",
23
+
sourceClient ? `available (baseUrl: ${sourceClient.getBaseUrl()})` : "NOT AVAILABLE",
24
+
);
25
+
console.log(
26
+
"[blob-migration] Local client baseUrl:",
27
+
localClient.getBaseUrl(),
28
+
);
29
+
console.log(
30
+
"[blob-migration] Local client has access token:",
31
+
localClient.getAccessToken() ? "yes" : "NO",
24
32
);
25
33
26
34
onProgress({ currentOperation: "Checking for missing blobs..." });
···
95
103
"contentType:",
96
104
contentType,
97
105
);
98
-
await localClient.uploadBlob(blobData, contentType);
106
+
console.log("[blob-migration] Uploading blob", cid, "to local PDS...");
107
+
const uploadResult = await localClient.uploadBlob(blobData, contentType);
99
108
console.log(
100
-
"[blob-migration] Uploaded blob",
109
+
"[blob-migration] Upload response for",
101
110
cid,
102
-
"with contentType:",
103
-
contentType,
111
+
":",
112
+
JSON.stringify(uploadResult),
104
113
);
105
114
migrated++;
106
115
onProgress({ blobsMigrated: migrated });
+26
-2
frontend/src/lib/migration/flow.svelte.ts
+26
-2
frontend/src/lib/migration/flow.svelte.ts
···
469
469
}
470
470
471
471
async function migrateBlobs(): Promise<void> {
472
-
if (!sourceClient || !localClient) return;
472
+
if (!sourceClient) {
473
+
console.error("[migration] migrateBlobs: sourceClient is null, skipping blob migration");
474
+
migrationLog("migrateBlobs SKIPPED: sourceClient is null");
475
+
setProgress({
476
+
currentOperation: "Warning: Could not migrate blobs - source PDS connection lost",
477
+
});
478
+
return;
479
+
}
480
+
if (!localClient) {
481
+
console.error("[migration] migrateBlobs: localClient is null, skipping blob migration");
482
+
migrationLog("migrateBlobs SKIPPED: localClient is null");
483
+
setProgress({
484
+
currentOperation: "Warning: Could not migrate blobs - local PDS connection lost",
485
+
});
486
+
return;
487
+
}
488
+
489
+
migrationLog("migrateBlobs: Starting blob migration", {
490
+
sourceClientBaseUrl: sourceClient.getBaseUrl(),
491
+
localClientBaseUrl: localClient.getBaseUrl(),
492
+
localClientHasToken: !!localClient.getAccessToken(),
493
+
});
473
494
474
495
const result = await migrateBlobsUtil(
475
496
localClient,
···
482
503
}
483
504
484
505
async function migratePreferences(): Promise<void> {
485
-
if (!sourceClient || !localClient) return;
506
+
if (!sourceClient || !localClient) {
507
+
console.warn("[migration] migratePreferences: client missing, skipping");
508
+
return;
509
+
}
486
510
487
511
try {
488
512
const prefs = await sourceClient.getPreferences();
+10
-1
src/api/error.rs
+10
-1
src/api/error.rs
···
427
427
error: self.error_name(),
428
428
message: self.message(),
429
429
};
430
-
(self.status_code(), Json(body)).into_response()
430
+
let mut response = (self.status_code(), Json(body)).into_response();
431
+
if matches!(self, Self::ExpiredToken(_)) {
432
+
response.headers_mut().insert(
433
+
"WWW-Authenticate",
434
+
"Bearer error=\"invalid_token\", error_description=\"Token has expired\""
435
+
.parse()
436
+
.unwrap(),
437
+
);
438
+
}
439
+
response
431
440
}
432
441
}
433
442
+7
-1
src/lib.rs
+7
-1
src/lib.rs
···
590
590
CorsLayer::new()
591
591
.allow_origin(Any)
592
592
.allow_methods([Method::GET, Method::POST, Method::OPTIONS])
593
-
.allow_headers(Any),
593
+
.allow_headers(Any)
594
+
.expose_headers([
595
+
"WWW-Authenticate".parse().unwrap(),
596
+
"DPoP-Nonce".parse().unwrap(),
597
+
"atproto-repo-rev".parse().unwrap(),
598
+
"atproto-content-labelers".parse().unwrap(),
599
+
]),
594
600
)
595
601
.with_state(state);
596
602