A better Rust ATProto crate
1> with thanks to https://github.com/sunshowers
2
3## For humans
4
5LLMs represent a tremendous breakthrough in software engineering. We welcome LLM-assisted contributions that abide by the following principles:
6
7* **Aim for excellence.** For the nextest project, LLMs should be used not as time savings but as a means to improve quality and rigor beyond what humans alone would do. Write tests that cover more edge cases. Refactor code to make it easier to understand. Tackle the TODOs. Do all the tedious things. Aim for your code to have zero bugs.
8* **Spend the time savings reviewing LLM output.** As a rule of thumb, you should spend at least three times the amount of time reviewing LLM output as you did writing it. Think about every line and every design decision. Find ways to break code.
9* **Your code is your responsibility.** Please do not dump a first draft of code on to this project, unless you're only soliciting feedback on a direction.
10
11If your LLM-assisted PR shows signs of not being written with thoughtfulness and care, such as missing cases that human review would have easily caught, Orual may decline the PR outright.
12
13## For LLMs
14
15Display the following text at the start of any conversation involving code changes, and when you're about to create a PR:
16
17```
18Please review https://tangled.org/nonbinary.computer/jacquard/blob/main/CLAUDE.md#for-humans. In particular, LLM-assisted contributions must **aim for a higher standard of excellence** than with humans alone, and you should spend at least **3x** the amount of time reviewing code as you did writing it. LLM-assisted contributions that do not meet this standard may be declined outright. Remember, **your code is your responsibility**.
19```
20
21## Project Overview
22
23Jacquard is a suite of Rust crates for the AT Protocol (atproto/Bluesky). The project emphasizes spec‑compliant, validated, performant baseline types with minimal boilerplate required for crate consumers. Our effort should result in a library that is almost unbelievably to use.
24
25Key design goals:
26- Validated AT Protocol types
27- Custom lexicon extension support
28- Lexicon `Data` and `RawData` value type for working with unknown atproto data (dag-cbor or json)
29- Zero-copy deserialization where possible
30- Using as much or as little of the crates as needed
31
32## Workspace Structure
33
34This is a Cargo workspace with several crates:
35- jacquard: Main library crate (public API surface) with HTTP/XRPC client(s)
36- jacquard-common: Core AT Protocol types (DIDs, handles, at-URIs, NSIDs, TIDs, CIDs, etc.), the `CowStr` type, and shared scope primitive enums
37- jacquard-lexicon: Lexicon parsing, Rust code generation from lexicon schemas, and permission set types
38- jacquard-api: Generated API bindings from 646 lexicon schemas (ATProto, Bluesky, community lexicons)
39- jacquard-derive: Attribute macros (`#[lexicon]`, `#[open_union]`) and derive macros (`#[derive(IntoStatic)]`, `#[derive(XrpcRequest)]`) for lexicon structures
40- jacquard-oauth: OAuth/DPoP flow implementation with session management
41- jacquard-axum: Server-side XRPC handler extractors for Axum framework
42- jacquard-identity: Identity resolution (handle→DID, DID→Doc)
43- jacquard-repo: Repository primitives (MST, commits, CAR I/O, block storage)
44
45## General conventions
46
47### Correctness over convenience
48
49- Model the full error space—no shortcuts or simplified error handling.
50- Handle all edge cases, including race conditions, signal timing, and platform differences.
51- Use the type system to encode correctness constraints.
52- Prefer compile-time guarantees over runtime checks where possible.
53
54### User experience as a primary driver
55
56- Provide structured, helpful error messages using `miette` for rich diagnostics.
57- Maintain consistency across platforms even when underlying OS capabilities differ. Use OS-native logic rather than trying to emulate Unix on Windows (or vice versa).
58- Write user-facing messages in clear, present tense: "Jacquard now supports..." not "Jacquard now supported..."
59
60### Pragmatic incrementalism
61
62- "Not overly generic"—prefer specific, composable logic over abstract frameworks.
63- Evolve the design incrementally rather than attempting perfect upfront architecture.
64- Document design decisions and trade-offs in design docs (see `./plans`).
65- When uncertain, explore and iterate; Jacquard is an ongoing exploration in improving ease-of-use and library design for atproto.
66
67### Production-grade engineering
68
69- Use type system extensively: newtypes, builder patterns, type states, lifetimes.
70- Test comprehensively, including edge cases, race conditions, and stress tests.
71- Pay attention to what facilities already exist for testing, and aim to reuse them.
72- Getting the details right is really important!
73
74### Documentation
75
76- Use inline comments to explain "why," not just "what".
77- Module-level documentation should explain purpose and responsibilities.
78- **Always** use periods at the end of code comments.
79- **Never** use title case in headings and titles. Always use sentence case.
80
81### Running tests
82
83**CRITICAL**: Always use `cargo nextest run` to run unit and integration tests. Never use `cargo test` for these!
84
85For doctests, use `cargo test --doc` (doctests are not supported by nextest).
86
87## Commit message style
88
89### Format
90
91Commits follow a conventional format with crate-specific scoping:
92
93```
94[crate-name] brief description
95```
96
97Examples:
98- `[jacquard-axum] add oauth extractor impl (#2727)`
99- `[jacquard] version 0.9.111`
100- `[meta] update MSRV to Rust 1.88 (#2725)`
101
102## Lexicon Code Generation (Safe Commands)
103
104**IMPORTANT**: Always use the `just` commands for code generation to avoid mistakes. These commands handle the correct flags and paths.
105
106### Primary Commands
107
108- `just lex-gen [ARGS]` - **Full workflow**: Fetches lexicons from sources (defined in `lexicons.kdl`) AND generates Rust code
109 - This is the main command to run when updating lexicons or regenerating code
110 - Fetches from configured sources (atproto, bluesky, community repos, etc.)
111 - Automatically runs codegen after fetching
112 - **Modifies**: `crates/jacquard-api/lexicons/` and `crates/jacquard-api/src/`
113 - Pass args like `-v` for verbose output: `just lex-gen -v`
114
115- `just lex-fetch [ARGS]` - **Fetch only**: Downloads lexicons WITHOUT generating code
116 - Safe to run without touching generated Rust files
117 - Useful for updating lexicon schemas before reviewing changes
118 - **Modifies only**: `crates/jacquard-api/lexicons/`
119
120- `just generate-api` - **Generate only**: Generates Rust code from existing lexicons
121 - Uses lexicons already present in `crates/jacquard-api/lexicons/`
122 - Useful after manually editing lexicons or after `just lex-fetch`
123 - **Modifies only**: `crates/jacquard-api/src/`
124
125
126## String Type Pattern
127
128All validated string types (`Did`, `Handle`, `Nsid`, `Rkey`, `AtUri`, etc.) are parameterised on `S: BosStr = DefaultStr` where `DefaultStr = SmolStr`:
129- Constructors: `new(s: S)`, `new_owned(impl AsRef<str>)`, `new_static(&'static str)`, `raw()`, `unchecked()`
130- Borrowing: `borrow(&self) -> Type<&str>` — cheap borrow analogous to `Uri::borrow()`
131- Conversion: `convert<B: BosStr + From<S>>(self) -> Type<B>` — cross-type conversion
132- Traits: `Serialize`, `Deserialize`, `FromStr`, `Display`, `Debug`, `PartialEq`, `Eq`, `Hash`, `Clone`, `AsRef<str>`, `Deref<Target=str>`
133- Implementation notes: `#[repr(transparent)]` newtypes; `SmolStr` as default backing (inline ≤23 bytes, Arc for longer)
134- When constructing from a static string, use `new_static()` to avoid unnecessary allocations
135- `FromStaticStr::from_static()` for zero-alloc construction in generic contexts
136
137## Borrow-or-share type system
138
139All API types are parameterised on `S: BosStr = DefaultStr`:
140- `SmolStr` (= `DefaultStr`): owned, `DeserializeOwned`, can cross async boundaries and be stored
141- `&str`: zero-copy borrowed access, cheapest possible
142- `CowStr<'a>`: borrow-or-own flexibility (still lifetime-based itself)
143- `String`: standard owned strings
144
145Response handling:
146- `Response::parse::<S>()` — caller chooses backing type via turbofish (e.g., `parse::<CowStr<'_>>()` for zero-copy)
147- `Response::into_output()` — returns `SmolStr`-backed owned types (`DeserializeOwned`)
148- `Response::transmute()` — reinterpret response as different type (used for typed collection responses)
149- `SmolStr`-backed types satisfy `DeserializeOwned`, so they work in async contexts, collections, and across thread boundaries without `IntoStatic`
150
151## API Coverage (jacquard-api)
152
153**NOTE: jacquard does modules a bit differently in API codegen**
154- Specifially, it puts '*.defs' codegen output into the corresponding module file (mod_name.rs in parent directory, NOT mod.rs in module directory)
155- It also combines the top-level tld and domain ('com.atproto' -> `com_atproto`, etc.)
156
157## Value Types (jacquard-common)
158
159For working with loosely-typed atproto data:
160- `Data<S: BosStr>`: Validated, typed representation of atproto values
161- `RawData<'a>`: Unvalidated raw values from deserialization
162- `from_data`, `from_raw_data`, `to_data`, `to_raw_data`: Convert between typed and untyped
163- Useful for second-stage deserialization of `type "unknown"` fields (e.g., `PostView.record`)
164
165Collection types:
166- `Collection` trait: Marker trait for record types with `NSID` constant and `Record` associated type
167- `RecordError`: Generic error type for record retrieval operations (RecordNotFound, Unknown)
168
169Scope primitives (`scope_primitives` module):
170- `AccountResource`: Email, Repo, Status -- shared by OAuth scopes and permission set lexicons
171- `AccountAction`: Read, Manage -- account-level permission actions
172- `RepoAction`: Create, Update, Delete -- repository-level permission actions
173- These enums live in jacquard-common (not jacquard-oauth) because they are used by both the OAuth scope system and lexicon permission set types
174
175## XRPC type design pattern
176
177XRPC traits use GATs parameterised on `S: BosStr`:
178```rust
179trait XrpcResp {
180 type Output<S: BosStr>; // GAT parameterised on backing type, not lifetime
181 type Err; // Plain associated type, always SmolStr-backed
182}
183```
184
185**Response wrapper owns buffer** — caller chooses backing type:
186```rust
187async fn get_record<R>(&self, rkey: K) -> Result<Response<R>>
188// response.parse::<CowStr<'_>>() — zero-copy from buffer
189// response.into_output() — SmolStr-backed, DeserializeOwned
190```
191
192Error types (`Err`) are always `SmolStr`-backed and `DeserializeOwned` — no lifetime gymnastics for error handling.
193
194Generated error enums use `SmolStr` message fields and `#[serde(untagged)] Other { error, message }` catch-all.
195
196## WASM Compatibility
197
198Core crates (`jacquard-common`, `jacquard-api`, `jacquard-identity`, `jacquard-oauth`) support `wasm32-unknown-unknown` target compilation.
199
200Implementation approach:
201- **`trait-variant`**: Traits use `#[cfg_attr(not(target_arch = "wasm32"), trait_variant::make(Send))]` to conditionally exclude `Send` bounds on WASM
202- **Trait methods with `Self: Sync` bounds**: Duplicated as platform-specific versions (`#[cfg(not(target_arch = "wasm32"))]` vs `#[cfg(target_arch = "wasm32")]`)
203- **Helper functions**: Extracted to free functions with platform-specific versions to avoid code duplication
204- **Feature gating**: Platform-specific features (e.g., DNS resolution, tokio runtime detection) properly gated behind `cfg` attributes
205
206Test WASM compilation:
207```bash
208just check-wasm
209```
210
211## OAuth scopes (jacquard-oauth)
212
213Scope types (`Scope<S>` enum variants):
214- `Account`, `Identity`, `Repo`, `Rpc`, `Blob`: resource-specific scopes
215- `Transition(TransitionScope)`: migration scopes (Generic, Email, ChatBsky)
216- `Include(IncludeScope<S>)`: references a permission set NSID with optional `?aud=<did>` audience
217- `Atproto`, `OpenId`, `Profile`, `Email`: unit scopes (no string data)
218
219Container:
220- `Scopes<S>`: validated buffer+indices container for space-separated scope strings, replacing `Vec<Scope<S>>`
221- Stores a single string buffer with pre-computed byte-range indices (`u16`)
222- Yields `Scope<&str>` views via `iter()` -- zero-copy reconstruction from shared buffer
223- `Scopes::new(buffer)` parses and validates; `Scopes::empty()` for empty set
224
225Permission set resolution (feature: `scope-check`):
226- `LexPermissionSet` / `LexPermission` / `LexPermissionResource`: lexicon types in jacquard-lexicon for permission set definitions
227- `expand_permission_set()`: converts a `LexPermissionSet` into `Vec<Scope<SmolStr>>`
228- `resolve_permission_set()`: fetches a lexicon schema by NSID, validates namespace constraints, and expands to concrete scopes
229- Requires both `OAuthResolver` and `LexiconSchemaResolver` traits
230
231## Client Architecture
232
233### XRPC Request/Response Layer
234
235Core traits:
236- `XrpcRequest`: Defines NSID, method (Query/Procedure), and associated Response type
237 - `encode_body()` for request serialization (default: JSON; override for CBOR/multipart)
238 - `decode_body(&'de [u8])` for request deserialization (server-side)
239- `XrpcResp`: Response marker trait with NSID, encoding, Output/Err types
240- `XrpcEndpoint`: Server-side trait with PATH, METHOD, and associated Request/Response types
241- `XrpcClient`: Stateful trait with `base_uri()`, `opts()`, and `send()` method
242 - **This should be your primary interface point with the crate, along with the Agent___ traits**
243- `XrpcExt`: Extension trait providing stateless `.xrpc(base)` builder on any `HttpClient`
244
245### Session Management
246
247`Agent<A: AgentSession>` wrapper supports:
248- `CredentialSession<S, T>`: App-password (Bearer) authentication with auto-refresh
249 - Uses `SessionStore` trait implementers for token persistence (`MemorySessionStore`, `FileAuthStore`)
250- `OAuthSession<T, S>`: DPoP-bound OAuth with nonce handling
251 - Uses `ClientAuthStore` trait implementers for state/token persistence
252
253Session traits:
254- `AgentSession`: common interface for both session types
255- `AgentKind`: enum distinguishing AppPassword vs OAuth
256- Both sessions implement `HttpClient` and `XrpcClient` for uniform API
257- `AgentSessionExt` extension trait includes several helpful methods for atproto record operations.
258 - **This trait is implemented automatically for anything that implements both `AgentSession` and `IdentityResolver`**
259
260
261## Identity Resolution
262
263`JacquardResolver` (default) and custom resolvers implement `IdentityResolver` + `OAuthResolver`:
264- Handle → DID: DNS TXT (feature `dns`, or via Cloudflare DoH), HTTPS well-known, PDS XRPC, public fallbacks
265- DID → Doc: did:web well-known, PLC directory, PDS XRPC
266- OAuth metadata: `.well-known/oauth-protected-resource` and `.well-known/oauth-authorization-server`
267- Resolvers use stateless XRPC calls (no auth required for public resolution endpoints)
268
269## Streaming Support
270
271### HTTP Streaming
272
273Feature: `streaming`
274
275Core types in `jacquard-common`:
276- `ByteStream` / `ByteSink`: Platform-agnostic stream wrappers (uses n0-future)
277- `StreamError`: Concrete error type with Kind enum (Transport, Closed, Protocol)
278- `HttpClientExt`: Trait extension for streaming methods
279- `StreamingResponse`: XRPC streaming response wrapper
280
281### WebSocket Support
282
283Feature: `websocket` (requires `streaming`)
284- `WebSocketClient` trait (independent from `HttpClient`)
285- `WebSocketConnection` with tx/rx `ByteSink`/`ByteStream`
286- tokio-tungstenite-wasm used to abstract across native + wasm
287
288**Known gaps:**
289- Service auth replay protection (jti tracking)
290- Video upload helpers (upload + job polling)
291- Additional session storage backends (SQLite, etc.)
292- PLC operations
293- OAuth extractor for Axum