A loose federation of distributed, typed datasets
docs: finalize architectural decisions for atproto integration
Completed key design decisions:
- Lens code storage: external repositories (GitHub/tangled.org)
- Schema evolution: NSID@semver format with lens-based migrations
- Lexicon namespace: ac.foundation.dataset.* with defined types
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
+30
-2
.chainlink/issues.db
.chainlink/issues.db
This is a binary file and will not be displayed.
+8
-1
.planning/decisions/02_lens_code_storage.md
+8
-1
.planning/decisions/02_lens_code_storage.md
···
7
7
8
8
## DECISION
9
9
10
-
...
10
+
Let's go with Option 1, using external repositories. We can actually make this work for
11
+
12
+
* GitHub
13
+
* tangled.org (the native ATProto git repository system)
14
+
15
+
Additionally, we'll want to keep track of metadata for lenses giving the language the referenced code is implemented in.
16
+
17
+
Longer-term, it will also be good to add another Lexicon specification for attestation of `Lens` formal correctness (where possible), as this will enable filtering lens implementations by provability. We'll also want to add our own `verification` records that give attestation of individual atproto DIDs (user identities) as being "trusted" for creating `Lens`es, etc.
11
18
12
19
---
13
20
+12
.planning/decisions/04_schema_evolution.md
+12
.planning/decisions/04_schema_evolution.md
···
5
5
**Blocks**: #50 (Lexicon validation), #39 (Type validation)
6
6
**Priority**: High
7
7
8
+
## DECISION
9
+
10
+
For this, let's take the following approach:
11
+
12
+
1. Let's make the `rkey` for the `ac.foundation.dataset.sampleSchema` records be of type `any`.
13
+
2. Then, we can have our own standard for the `rkey` being of the format `{NSID}@{semver}`, where `{NSID}` gives an NSID for the permanent identifier of this sample schema type.
14
+
* This allows us to bookkeep on the version updates
15
+
* We can make a `ac.foundation.dataset.getLatestSchema` `query` Lexicon that will provide the record for the latest version of a given schema, as well
16
+
3. We can build into the `atdata` SDK that whenever users update their own sample schema types, they can pass in optional `Lens`es between the two versions that give transformations to downgrade / upgrade records, so that there's an easy dev-facing way to auto-update any existing datasets using an older schema and maintain compatibility with older code for newer data.
17
+
18
+
---
19
+
8
20
## Problem Statement
9
21
10
22
We need to define how PackableSample schemas can evolve over time without breaking existing datasets or code. This includes:
+9
-1
.planning/decisions/05_lexicon_namespace.md
+9
-1
.planning/decisions/05_lexicon_namespace.md
···
10
10
We're going to use an org NSID for the steward organization as the base:
11
11
12
12
```
13
-
ac.foundation.data.*
13
+
ac.foundation.dataset.*
14
+
```
15
+
16
+
The choices we have then are
17
+
18
+
```
19
+
ac.foundation.dataset.sampleSchema
20
+
ac.foundation.dataset.record
21
+
ac.foundation.dataset.lens
14
22
```
15
23
16
24
---