A loose federation of distributed, typed datasets
docs: add initial atproto integration planning document
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
+19
.chainlink/issues.db
.chainlink/issues.db
This is a binary file and will not be displayed.
+19
.planning/atproto_integration.md
+19
.planning/atproto_integration.md
···
1
+
# Planning for full atproto integration
2
+
3
+
The overall goal for `atdata` is that the index for datasets is actually present on the atproto distributed repository, with one type of Lexicon schema for actually containing information about `PackableSample` schemas that can be reproduced with code gen, and one type of Lexicon schema designed for the main functionality: records holding the links to the WDS dataset for samples and the msgpack metadata (that can be plugged into the `Dataset` class) as well as a reference to the atproto record containing the schema for the appropriate sample type for the dataset.
4
+
5
+
## Thoughts on functionality
6
+
7
+
* Lexicons
8
+
* Definition of a `PackableSample`-compatible sample type schema, that can be used to reconstitute the code in appropriate languages using code gen toolilng
9
+
* Index records that contain links to the actual WebDataset data, as well as to the records with the corresponding sample schema.
10
+
* `Lenses` between defined sample type schemas across the network.
11
+
* Python library functionality
12
+
* Logging in with the atproto sdk
13
+
* Posting sample schemas and dataset index records to the appropriate lexicons for the user
14
+
* AppView functionality
15
+
* Aggregating index records, making an index of those that is quick to query on
16
+
17
+
## Questions for implementation
18
+
19
+
* What is the best way to store the sample type schemas within atproto Lexicons? I've thought about using JSON schema or protobuf, but want to think through possibilities.