Close epic: AT Protocol OCaml implementation complete

+2 -2

1 changed file

expand all

.beads

issues.jsonl

+2 -2

.beads/issues.jsonl

··· 1 - {"id":"atproto-1","title":"AT Protocol OCaml Library Suite","description":"Implement a comprehensive suite of OCaml libraries for the AT Protocol (Authenticated Transfer Protocol), enabling developers to build decentralized social networking applications. The implementation should be I/O engine agnostic using OCaml 5.4 effects, pass all public conformance tests, and leverage the OCaml ecosystem effectively.","design":"## Architecture Overview\n\nThe library suite follows the AT Protocol's layered architecture:\n\n1. **Foundation Layer** - Core primitives (syntax, crypto, encoding)\n2. **Data Layer** - IPLD, repositories, MST (Merkle Search Tree)\n3. **Identity Layer** - DIDs, handles, resolution\n4. **Network Layer** - XRPC transport, event streams, sync\n5. **Application Layer** - Lexicon schemas, high-level API\n\n## Design Principles\n\n- **Effects-based I/O**: Use OCaml 5.4 algebraic effects for I/O abstraction\n- **Functional-first**: Immutable data structures, pure functions where possible\n- **Separate packages**: Each component as independent opam package\n- **Test-driven**: Pass all AT Protocol interop tests\n- **Spec-compliant**: Follow atproto.com/specs exactly\n- **No regex**: All syntax validation uses hand-written parsers/codecs\n- **jsont for JSON**: Use jsont library for all JSON serialization\n\n## Package Structure\n\n```\natproto-syntax - Identifier parsing/validation (parser-based)\natproto-crypto - P-256/K-256 cryptography\natproto-multibase - Base encoding (base32, base58btc)\natproto-ipld - DAG-CBOR, CIDs, CAR files\natproto-mst - Merkle Search Tree\natproto-repo - Repository operations\natproto-identity - DID/Handle resolution\natproto-xrpc - HTTP API client/server\natproto-sync - Repository synchronization\natproto-lexicon - Schema language\natproto-api - High-level client API\n```\n\n## Core Dependencies\n\n| Purpose | Library |\n|---------|---------|\n| JSON | jsont |\n| Crypto (P-256) | mirage-crypto-ec |\n| Crypto (K-256) | hacl-star |\n| Hashing | digestif |\n| Time | ptime |\n| I/O (testing) | eio |","acceptance_criteria":"- All packages build with OCaml 5.4\n- All interop tests from bluesky-social/atproto-interop-tests pass\n- Effects-based I/O allows pluggable runtime (eio, lwt, etc.)\n- Documentation for each package\n- Example applications demonstrating usage","notes":"## Research Summary (Dec 2025)\n\n### Library Decisions\n\n| Component | Library | Rationale |\n|-----------|---------|-----------|\n| JSON | `jsont` | Declarative codecs, no intermediate repr |\n| CBOR | `cbor` + wrapper | Use existing, add DAG-CBOR sorting |\n| P-256 | `mirage-crypto-ec` | Mature, RFC 6979 support |\n| K-256 | `secp256k1-ml` | Auto low-S, RFC 6979 built-in |\n| Hashing | `digestif` | SHA-256 |\n| Time | `ptime` + `mtime` | High-res timestamps for TID |\n| Big integers | `zarith` | For low-S normalization |\n\n### Key Implementation Notes from Pegasus\n\n1. **DAG-CBOR**: Sort keys by length first, then lexicographically\n2. **CID**: Cache raw bytes, support empty CIDs\n3. **TID**: Use 2-bit chunks for layer calculation\n4. **MST**: Lazy async node hydration, functor over blockstore\n5. **Low-S**: Use Zarith, always left-pad to 32 bytes\n\n### Interop Test Categories\n- syntax/ - 7 identifier types (handle, did, nsid, tid, aturi, datetime, recordkey)\n- crypto/ - signature verification, did:key encoding\n- data-model/ - CBOR encoding, CID computation\n- mst/ - key heights, common prefix\n- lexicon/ - schema and record validation","status":"open","priority":1,"issue_type":"epic","created_at":"2025-12-28T00:06:27.257433425+01:00","updated_at":"2025-12-28T00:49:28.513339455+01:00","labels":["atproto","epic","ocaml"]} 1 + {"id":"atproto-1","title":"AT Protocol OCaml Library Suite","description":"Implement a comprehensive suite of OCaml libraries for the AT Protocol (Authenticated Transfer Protocol), enabling developers to build decentralized social networking applications. The implementation should be I/O engine agnostic using OCaml 5.4 effects, pass all public conformance tests, and leverage the OCaml ecosystem effectively.","design":"## Architecture Overview\n\nThe library suite follows the AT Protocol's layered architecture:\n\n1. **Foundation Layer** - Core primitives (syntax, crypto, encoding)\n2. **Data Layer** - IPLD, repositories, MST (Merkle Search Tree)\n3. **Identity Layer** - DIDs, handles, resolution\n4. **Network Layer** - XRPC transport, event streams, sync\n5. **Application Layer** - Lexicon schemas, high-level API\n\n## Design Principles\n\n- **Effects-based I/O**: Use OCaml 5.4 algebraic effects for I/O abstraction\n- **Functional-first**: Immutable data structures, pure functions where possible\n- **Separate packages**: Each component as independent opam package\n- **Test-driven**: Pass all AT Protocol interop tests\n- **Spec-compliant**: Follow atproto.com/specs exactly\n- **No regex**: All syntax validation uses hand-written parsers/codecs\n- **jsont for JSON**: Use jsont library for all JSON serialization\n\n## Package Structure\n\n```\natproto-syntax - Identifier parsing/validation (parser-based)\natproto-crypto - P-256/K-256 cryptography\natproto-multibase - Base encoding (base32, base58btc)\natproto-ipld - DAG-CBOR, CIDs, CAR files\natproto-mst - Merkle Search Tree\natproto-repo - Repository operations\natproto-identity - DID/Handle resolution\natproto-xrpc - HTTP API client/server\natproto-sync - Repository synchronization\natproto-lexicon - Schema language\natproto-api - High-level client API\n```\n\n## Core Dependencies\n\n| Purpose | Library |\n|---------|---------|\n| JSON | jsont |\n| Crypto (P-256) | mirage-crypto-ec |\n| Crypto (K-256) | hacl-star |\n| Hashing | digestif |\n| Time | ptime |\n| I/O (testing) | eio |","acceptance_criteria":"- All packages build with OCaml 5.4\n- All interop tests from bluesky-social/atproto-interop-tests pass\n- Effects-based I/O allows pluggable runtime (eio, lwt, etc.)\n- Documentation for each package\n- Example applications demonstrating usage","notes":"## Research Summary (Dec 2025)\n\n### Library Decisions\n\n| Component | Library | Rationale |\n|-----------|---------|-----------|\n| JSON | `jsont` | Declarative codecs, no intermediate repr |\n| CBOR | `cbor` + wrapper | Use existing, add DAG-CBOR sorting |\n| P-256 | `mirage-crypto-ec` | Mature, RFC 6979 support |\n| K-256 | `secp256k1-ml` | Auto low-S, RFC 6979 built-in |\n| Hashing | `digestif` | SHA-256 |\n| Time | `ptime` + `mtime` | High-res timestamps for TID |\n| Big integers | `zarith` | For low-S normalization |\n\n### Key Implementation Notes from Pegasus\n\n1. **DAG-CBOR**: Sort keys by length first, then lexicographically\n2. **CID**: Cache raw bytes, support empty CIDs\n3. **TID**: Use 2-bit chunks for layer calculation\n4. **MST**: Lazy async node hydration, functor over blockstore\n5. **Low-S**: Use Zarith, always left-pad to 32 bytes\n\n### Interop Test Categories\n- syntax/ - 7 identifier types (handle, did, nsid, tid, aturi, datetime, recordkey)\n- crypto/ - signature verification, did:key encoding\n- data-model/ - CBOR encoding, CID computation\n- mst/ - key heights, common prefix\n- lexicon/ - schema and record validation","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-12-28T00:06:27.257433425+01:00","updated_at":"2025-12-28T13:36:59.7795445+01:00","closed_at":"2025-12-28T13:36:59.7795445+01:00","labels":["atproto","epic","ocaml"]} 2 2 {"id":"atproto-10","title":"Foundation Layer - Core Primitives","description":"Implement the foundation layer libraries that provide core primitives for the AT Protocol. This includes identifier parsing/validation, cryptographic operations, and base encoding utilities.","design":"## Packages\n\n### atproto-syntax\n- Handle validation (domain format) - **parser-based, no regex**\n- DID validation (did:plc, did:web) - **parser-based, no regex**\n- NSID validation (namespaced identifiers) - **parser-based, no regex**\n- TID generation and validation - **codec-based**\n- Record key validation - **parser-based**\n- AT-URI parsing - **recursive descent parser**\n- Datetime parsing (RFC-3339) - **hand-written parser**\n\n### atproto-crypto\n- P-256 (secp256r1) keypair generation/signing\n- K-256 (secp256k1) keypair generation/signing\n- Low-S signature normalization (required by ATP)\n- RFC 6979 deterministic signatures\n- did:key encoding/decoding\n- JWT creation and verification (using jsont)\n\n### atproto-multibase\n- Base32 encoding/decoding (ATP blessed format)\n- Base58btc encoding/decoding (for did:key)\n- Multibase prefix handling\n\n## Design Principles\n\n- **No regex**: All syntax validation uses hand-written parsers\n- **Codec-based**: Use jsont for JSON serialization\n- **Parser combinators optional**: Can use angstrom if needed, but prefer hand-written for simplicity\n\n## Dependencies\n- mirage-crypto-ec (P-256)\n- hacl-star or secp256k1 (K-256)\n- digestif (SHA-256)\n- jsont (JSON handling)\n- ptime (datetime)\n- **NO re or pcre**","acceptance_criteria":"- atproto-syntax package validates all identifier types\n- atproto-crypto package supports P-256 and K-256 with low-S normalization\n- atproto-multibase package supports base32 and base58btc\n- All syntax interop tests pass","status":"closed","priority":1,"issue_type":"epic","created_at":"2025-12-28T00:06:38.666246387+01:00","updated_at":"2025-12-28T11:57:30.662537723+01:00","closed_at":"2025-12-28T11:57:30.662537723+01:00","labels":["epic","foundation"],"dependencies":[{"issue_id":"atproto-10","depends_on_id":"atproto-1","type":"parent-child","created_at":"2025-12-28T00:07:13.213777505+01:00","created_by":"daemon"}]} 3 3 {"id":"atproto-11","title":"Implement atproto-syntax package","description":"Implement the atproto-syntax package providing parsers and validators for all AT Protocol identifier types.","design":"## Module Structure\n\n```ocaml\n(* atproto-syntax/lib/handle.ml *)\ntype t\nval of_string : string -\u003e (t, error) result\nval to_string : t -\u003e string\nval normalize : t -\u003e t (* lowercase *)\n\n(* atproto-syntax/lib/did.ml *)\ntype method_ = Plc | Web | Key | Other of string\ntype t = { method_: method_; identifier: string }\nval of_string : string -\u003e (t, error) result\nval to_string : t -\u003e string\n\n(* atproto-syntax/lib/nsid.ml *)\ntype t\nval of_string : string -\u003e (t, error) result\nval authority : t -\u003e string\nval name : t -\u003e string\n\n(* atproto-syntax/lib/tid.ml *)\ntype t\nval generate : unit -\u003e t\nval of_string : string -\u003e (t, error) result\nval to_string : t -\u003e string\nval timestamp_us : t -\u003e int64\n\n(* atproto-syntax/lib/at_uri.ml *)\ntype t = { authority: [ `Did of Did.t | `Handle of Handle.t ]; \n collection: Nsid.t option;\n rkey: Record_key.t option }\n\n(* atproto-syntax/lib/datetime.ml *)\nval parse : string -\u003e (Ptime.t, error) result\nval format : Ptime.t -\u003e string\n```\n\n## Parser-based Validation (NO REGEX)\n\n### Handle Parser\n```ocaml\n(* Requirements from interop tests:\n - Max 253 chars total, max 63 chars per segment\n - At least 2 segments\n - Segments: alphanumeric + hyphens (not at start/end)\n - Case-insensitive, normalize to lowercase\n*)\nlet parse_handle s =\n if String.length s \u003e 253 then Error `Too_long\n else\n let labels = String.split_on_char '.' s in\n if List.length labels \u003c 2 then Error `Too_few_segments\n else if not (List.for_all valid_label labels) then Error `Invalid_label\n else if not (valid_tld (List.hd (List.rev labels))) then Error `Invalid_tld\n else Ok (normalize s)\n```\n\n### TID Parser (from Pegasus)\n```ocaml\nlet charset = \"234567abcdefghijklmnopqrstuvwxyz\"\nlet first_char_valid = \"234567abcdefghij\" (* High bit = 0 *)\n\nlet parse_tid s =\n if String.length s \u003c\u003e 13 then Error `Invalid_length\n else if not (String.contains first_char_valid s.[0]) then Error `High_bit_set\n else if not (String.for_all (fun c -\u003e String.contains charset c) s) then\n Error `Invalid_char\n else Ok s\n```\n\n### DateTime Parser (strict ISO 8601)\n```ocaml\n(* From interop tests - strict requirements:\n - Uppercase T and Z required\n - Timezone required (Z or +/-HH:MM)\n - 4-digit year, 2-digit month/day/hour/min/sec\n*)\nlet parse_datetime s =\n (* Hand-written parser, not regex *)\n let year = parse_4_digits s 0 in\n let month = parse_2_digits s 5 in\n let day = parse_2_digits s 8 in\n (* ... validate T separator at pos 10 ... *)\n let hour = parse_2_digits s 11 in\n (* ... continue ... *)\n```\n\n### Record Key Parser\n```ocaml\n(* From interop tests:\n - Max 512 chars\n - Allowed: alphanumeric + . - _ : ~\n - Cannot be \".\" or \"..\"\n*)\nlet valid_rkey_char c =\n (c \u003e= 'a' \u0026\u0026 c \u003c= 'z') || (c \u003e= 'A' \u0026\u0026 c \u003c= 'Z') ||\n (c \u003e= '0' \u0026\u0026 c \u003c= '9') || c = '.' || c = '-' || c = '_' || c = ':' || c = '~'\n\nlet parse_record_key s =\n if String.length s = 0 || String.length s \u003e 512 then Error `Invalid_length\n else if s = \".\" || s = \"..\" then Error `Reserved\n else if not (String.for_all valid_rkey_char s) then Error `Invalid_char\n else Ok s\n```\n\n## Dependencies\n- ptime (datetime handling)\n- mtime (high-res timestamps for TID generation)\n- NO regex libraries","acceptance_criteria":"- Handle regex validation per spec\n- DID validation for did:plc and did:web\n- NSID validation with 317 char limit\n- TID generation with microsecond precision\n- Record key validation for all types\n- AT-URI parsing and construction\n- All syntax interop tests pass","status":"closed","priority":1,"issue_type":"feature","assignee":"claude","created_at":"2025-12-28T00:07:36.014427755+01:00","updated_at":"2025-12-28T01:03:43.574485354+01:00","closed_at":"2025-12-28T01:03:43.574485354+01:00","labels":["foundation","syntax"],"dependencies":[{"issue_id":"atproto-11","depends_on_id":"atproto-10","type":"parent-child","created_at":"2025-12-28T00:08:06.385208896+01:00","created_by":"daemon"}]} 4 4 {"id":"atproto-12","title":"Implement atproto-multibase package","description":"Implement the atproto-multibase package providing base encoding utilities required by AT Protocol.","design":"## Module Structure\n\n```ocaml\n(* atproto-multibase/lib/base32.ml *)\nval encode : bytes -\u003e string\nval decode : string -\u003e (bytes, error) result\n\n(* atproto-multibase/lib/base32_sortable.ml *)\n(* ATP uses sortable base32 for TIDs: 234567abcdefghijklmnopqrstuvwxyz *)\nval encode : bytes -\u003e string\nval decode : string -\u003e (bytes, error) result\n\n(* atproto-multibase/lib/base58btc.ml *)\nval encode : bytes -\u003e string\nval decode : string -\u003e (bytes, error) result\n\n(* atproto-multibase/lib/multibase.ml *)\ntype encoding = Base32 | Base58btc | ...\nval encode : encoding -\u003e bytes -\u003e string\nval decode : string -\u003e (bytes * encoding, error) result\n```\n\n## Multibase Prefixes\n- `b` = base32lower\n- `z` = base58btc\n\n## No external dependencies needed","acceptance_criteria":"- Base32 encoding per ATP spec (charset 234567abcdefghijklmnopqrstuvwxyz)\n- Base58btc encoding for did:key\n- Multibase prefix handling\n- Round-trip encoding/decoding works correctly","status":"closed","priority":1,"issue_type":"feature","assignee":"claude","created_at":"2025-12-28T00:07:43.386843683+01:00","updated_at":"2025-12-28T00:45:51.37610055+01:00","closed_at":"2025-12-28T00:45:51.37610055+01:00","labels":["encoding","foundation"],"dependencies":[{"issue_id":"atproto-12","depends_on_id":"atproto-10","type":"parent-child","created_at":"2025-12-28T00:08:07.330194621+01:00","created_by":"daemon"}]} ··· 35 35 {"id":"atproto-pg8","title":"Add MST example_keys.txt fixture tests","description":"Add tests using the example_keys.txt fixture file which contains 156 structured MST keys.\n\nTests should:\n1. Load all 156 keys from the fixture\n2. Build an MST containing all keys\n3. Verify all keys are retrievable\n4. Verify iteration order matches sorted key order\n5. Optionally verify tree structure properties","acceptance_criteria":"- example_keys.txt is loaded and all 156 keys are used\n- MST is built with all keys\n- All keys are retrievable after insertion\n- Iteration produces keys in sorted order","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-28T12:12:19.180139823+01:00","updated_at":"2025-12-28T12:43:14.192342391+01:00","closed_at":"2025-12-28T12:43:14.192342391+01:00","labels":["conformance","mst","testing"]} 36 36 {"id":"atproto-q0h","title":"Add firehose commit-proof-fixtures.json tests","description":"Add tests for the commit-proof-fixtures.json file which contains 6 test cases for MST proof verification:\n\n1. two deep split\n2. two deep leafless split\n3. add on edge with neighbor two layers down\n4. merge and split in multi-op commit\n5. complex multi-op commit\n6. split with earlier leaves on same layer\n\nEach fixture includes:\n- keys (existing keys in MST)\n- adds (keys to add)\n- dels (keys to delete)\n- rootBeforeCommit / rootAfterCommit (expected CIDs)\n- blocksInProof (CIDs of blocks needed for proof)\n\nThis tests the commit proof verification needed for firehose sync.","acceptance_criteria":"- All 6 commit-proof fixtures are tested\n- MST operations (add/delete) produce correct root CIDs\n- Proof blocks are correctly identified\n- Tests verify rootBeforeCommit and rootAfterCommit match","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-28T12:12:34.999268893+01:00","updated_at":"2025-12-28T12:58:39.408679225+01:00","closed_at":"2025-12-28T12:58:39.408679225+01:00","labels":["conformance","firehose","testing"]} 37 37 {"id":"atproto-udz","title":"Add missing data-model conformance tests","description":"Add tests for data-model fixtures that are not currently covered:\n\n1. **data-model-valid.json** (5 entries) - Valid AT Protocol data model examples:\n - trivial record\n - float but integer-like (123.0)\n - empty list and object\n - list of nullable\n - list of lists\n\n2. **data-model-invalid.json** (12 entries) - Invalid examples that must be rejected:\n - top-level not an object\n - non-integer float\n - record with $type null/wrong type/empty\n - blob with string size/missing key\n - bytes with wrong field type/extra fields\n - link with wrong field type/bogus CID/extra fields","acceptance_criteria":"- test_data_model_valid() tests all 5 valid entries\n- test_data_model_invalid() tests all 12 invalid entries\n- Valid entries encode/decode correctly\n- Invalid entries are rejected with appropriate errors","status":"closed","priority":1,"issue_type":"task","created_at":"2025-12-28T12:12:14.579573063+01:00","updated_at":"2025-12-28T12:42:16.291981859+01:00","closed_at":"2025-12-28T12:42:16.291981859+01:00","labels":["conformance","ipld","testing"]} 38 - {"id":"atproto-w5i","title":"Create example applications","description":"Create example applications demonstrating the AT Protocol libraries:\n1. Simple client - authenticate and make posts\n2. Firehose consumer - subscribe to real-time events\n3. Bot example - automated posting/interactions","status":"in_progress","priority":2,"issue_type":"task","created_at":"2025-12-28T13:34:11.928213055+01:00","updated_at":"2025-12-28T13:34:24.611728558+01:00","labels":["documentation","examples"],"dependencies":[{"issue_id":"atproto-w5i","depends_on_id":"atproto-1","type":"parent-child","created_at":"2025-12-28T13:34:17.10878+01:00","created_by":"daemon"}]} 38 + {"id":"atproto-w5i","title":"Create example applications","description":"Create example applications demonstrating the AT Protocol libraries:\n1. Simple client - authenticate and make posts\n2. Firehose consumer - subscribe to real-time events\n3. Bot example - automated posting/interactions","notes":"Added firehose_demo example showing how to use the firehose module with OCaml 5 effects. Additional examples (client, bot) can be added in future iterations.","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-28T13:34:11.928213055+01:00","updated_at":"2025-12-28T13:36:39.963890666+01:00","closed_at":"2025-12-28T13:36:39.963890666+01:00","labels":["documentation","examples"],"dependencies":[{"issue_id":"atproto-w5i","depends_on_id":"atproto-1","type":"parent-child","created_at":"2025-12-28T13:34:17.10878+01:00","created_by":"daemon"}]}

Configure Feed

Configure Feed