jj workspaces over the network
new spec
+616
+54
AGENTS.md
+54
AGENTS.md
···
1
+
# AGENTS
2
+
3
+
Execution guide for building `tandem` from the docs in this repository.
4
+
5
+
## How to read these docs (quick)
6
+
7
+
1. Read `ARCHITECTURE.md` for system boundaries.
8
+
2. Read `docs/exec-plans/active/slice-roadmap.md` and pick the next slice.
9
+
3. Implement via failing integration test first.
10
+
4. Keep any deferred cleanup in `docs/exec-plans/tech-debt-tracker.md`.
11
+
5. When a slice is done, move a completion note into `docs/exec-plans/completed/`.
12
+
13
+
## Working style
14
+
15
+
- Implement **one vertical slice at a time**.
16
+
- Each slice starts with a **failing Rust integration test**.
17
+
- Make the test pass with the smallest correct change.
18
+
- Keep behavior aligned with stock `jj` semantics.
19
+
20
+
## Priority order
21
+
22
+
1. Slice 1: Single-agent round-trip
23
+
2. Slice 2: Two-agent visibility
24
+
3. Slice 3: Concurrent convergence
25
+
4. Slice 4: Promise pipelining
26
+
5. Slice 5: WatchHeads
27
+
6. Slice 6: Git round-trip via server-side `jj`
28
+
7. Slice 7: End-to-end multi-agent
29
+
30
+
## Testing policy
31
+
32
+
- Integration tests are the primary source of truth.
33
+
- Local deterministic tests first; cross-machine tests second.
34
+
- Use `sprites.dev` / `exe.dev` for distributed smoke tests.
35
+
- Keep networked tests opt-in (ignored by default / env-gated).
36
+
37
+
## Debug policy
38
+
39
+
Add structured tracing early so we do not sprinkle debug prints later.
40
+
41
+
Recommended flags:
42
+
43
+
- `--tandem-debug`
44
+
- `--tandem-debug-format pretty|json`
45
+
- `--tandem-debug-file <path>`
46
+
- `--tandem-debug-filter <filter>`
47
+
48
+
Minimum events to emit:
49
+
50
+
- command lifecycle
51
+
- RPC lifecycle
52
+
- object read/write
53
+
- CAS heads success/failure + retries
54
+
- watcher subscribe/notify/reconnect
+85
ARCHITECTURE.md
+85
ARCHITECTURE.md
···
1
+
# ARCHITECTURE
2
+
3
+
`Tandem` = jj workspaces over the network.
4
+
5
+
## Shape
6
+
7
+
Single binary, two modes:
8
+
9
+
- `tandem serve --listen <addr> --repo <path>`
10
+
- `tandem <jj command...>` (client mode)
11
+
12
+
## Core model
13
+
14
+
- Server hosts a **normal jj+git colocated repo**.
15
+
- Client keeps **working copy local**.
16
+
- Client store calls are remote via Cap'n Proto.
17
+
- Clients always read heads from server, so no `workspace update-stale` model.
18
+
19
+
## Responsibilities
20
+
21
+
### Server
22
+
23
+
1. Read/write jj backend + op-store objects (commit/tree/file/symlink/copy/operation/view)
24
+
2. Coordinate op heads with atomic compare-and-swap
25
+
3. Notify watchers on head changes (`watchHeads`)
26
+
27
+
### Client
28
+
29
+
Implements jj traits as RPC stubs:
30
+
31
+
- `Backend`
32
+
- `OpStore`
33
+
- `OpHeadsStore`
34
+
35
+
On CAS failure, client retries using jj’s existing merge flow.
36
+
37
+
## Protocol
38
+
39
+
Cap'n Proto `Store` service (see `docs/design-docs/rpc-protocol.md` for the canonical schema).
40
+
41
+
Core capabilities:
42
+
43
+
- object read/write for backend + op-store data
44
+
- op head reads + atomic updates
45
+
- operation-prefix resolution
46
+
- head watch subscriptions
47
+
- optional snapshot/copy-tracking capabilities
48
+
49
+
No `repoId` in protocol: one server = one repo.
50
+
51
+
## Git compatibility
52
+
53
+
No custom git layer in tandem.
54
+
55
+
Git interop happens on server-hosted repo with stock `jj` commands:
56
+
57
+
- `jj git fetch`
58
+
- `jj git push`
59
+
60
+
## Dependency graph
61
+
62
+
- Slice 1 (round-trip)
63
+
- enables Slice 2 (multi-agent)
64
+
- enables Slice 3 (concurrent merge)
65
+
- enables Slice 4 (pipelining)
66
+
- enables Slice 5 (watchHeads)
67
+
- enables Slice 6 (git round-trip)
68
+
- Slice 7 integrates slices 1-6
69
+
70
+
Critical path: **1 → 2 → 6 → 7**.
71
+
72
+
## Technology choices
73
+
74
+
- Language: Rust
75
+
- Binary: single `tandem`
76
+
- RPC: Cap'n Proto (for promise pipelining)
77
+
- Server storage: normal jj+git colocated repo
78
+
- Serialization: jj-compatible object/op/view bytes
79
+
80
+
## Non-goals (v0.1)
81
+
82
+
- auth / ACL / multi-tenant isolation
83
+
- workflow automation engines
84
+
- web UI / IDE integrations
85
+
- client-side caching
+11
docs/README.md
+11
docs/README.md
···
1
+
# Docs
2
+
3
+
Minimal docs structure for the project:
4
+
5
+
- `../AGENTS.md` — execution/testing/debugging conventions
6
+
- `../ARCHITECTURE.md` — system shape and boundaries
7
+
- `design-docs/` — durable technical decisions
8
+
- `exec-plans/` — active/completed implementation plans
9
+
- `product-specs/` — concise product intent and scope
10
+
11
+
This docs set is now the canonical source of project direction and architecture.
+21
docs/design-docs/core-beliefs.md
+21
docs/design-docs/core-beliefs.md
···
1
+
# Core Beliefs
2
+
3
+
1. **Thin server, smart client integration with jj**
4
+
- Server provides storage + head coordination + notifications.
5
+
- Workflow semantics remain jj-native.
6
+
7
+
2. **Stock jj behavior first**
8
+
- Tandem should feel like jj, not a new VCS.
9
+
10
+
3. **Remote store, local working copy**
11
+
- Fast local file operations with shared global history.
12
+
13
+
4. **No stale-head workflow**
14
+
- Always read latest heads from server.
15
+
16
+
5. **Compatibility over invention**
17
+
- Reuse jj protobuf/object formats where possible.
18
+
- Keep server repo a normal jj+git colocated repo.
19
+
20
+
6. **Integration tests drive slices**
21
+
- Every major claim in SPEC should be proven by an integration test.
+15
docs/design-docs/index.md
+15
docs/design-docs/index.md
···
1
+
# Design Docs Index
2
+
3
+
This folder holds stable technical decisions.
4
+
5
+
## Current docs
6
+
7
+
- [Core beliefs](./core-beliefs.md)
8
+
- [RPC protocol](./rpc-protocol.md)
9
+
- [RPC error model](./rpc-error-model.md)
10
+
11
+
## Add a new design doc when
12
+
13
+
- a decision affects correctness or compatibility
14
+
- a decision changes protocol or storage format
15
+
- a decision introduces operational tradeoffs
+128
docs/design-docs/rpc-error-model.md
+128
docs/design-docs/rpc-error-model.md
···
1
+
# RPC Error Model
2
+
3
+
This document defines Tandem’s error contract between client and server.
4
+
5
+
## Goals
6
+
7
+
- Keep failures mappable to `jj-lib` error types.
8
+
- Separate transport failures from domain/storage failures.
9
+
- Make retries safe and predictable.
10
+
- Keep wire semantics stable for integration tests.
11
+
12
+
## Error classes
13
+
14
+
### 1) Transport/session errors
15
+
16
+
Examples:
17
+
18
+
- connection refused/reset
19
+
- timeout
20
+
- stream canceled
21
+
- server unavailable
22
+
23
+
Behavior:
24
+
25
+
- surfaced by RPC runtime (not domain payload)
26
+
- usually retriable for reads
27
+
- writes may be retried only if idempotency is guaranteed
28
+
29
+
### 2) Domain/storage errors
30
+
31
+
Returned by server logic for request-specific failures.
32
+
33
+
Canonical codes:
34
+
35
+
- `not_found`
36
+
- `invalid_id_length`
37
+
- `invalid_data`
38
+
- `unsupported`
39
+
- `permission_denied` (reserved for future auth)
40
+
- `internal`
41
+
42
+
### 3) Concurrency outcomes (not errors)
43
+
44
+
- `updateOpHeads(...)->ok=false` is **not** an error.
45
+
- It represents normal CAS contention and triggers jj merge/retry flow.
46
+
47
+
## Error envelope (application-level)
48
+
49
+
When a method needs structured failures beyond generic exceptions, use:
50
+
51
+
```text
52
+
RpcError {
53
+
code: <canonical code>,
54
+
message: <human-readable>,
55
+
retriable: <bool>,
56
+
details: {
57
+
object_type?: <string>,
58
+
hash?: <hex>,
59
+
op_id?: <hex>,
60
+
expected_len?: <u32>,
61
+
actual_len?: <u32>
62
+
}
63
+
}
64
+
```
65
+
66
+
Notes:
67
+
68
+
- Do not put secrets/tokens in `message` or `details`.
69
+
- `message` is for operators; clients should branch on `code`.
70
+
71
+
## Mapping to `jj-lib`
72
+
73
+
### Backend mapping
74
+
75
+
- `not_found` + object context -> `BackendError::ObjectNotFound`
76
+
- `invalid_id_length` -> `BackendError::InvalidHashLength`
77
+
- invalid UTF-8 payloads -> `BackendError::InvalidUtf8`
78
+
- read failures -> `BackendError::ReadObject` / `ReadFile`
79
+
- write failures -> `BackendError::WriteObject`
80
+
- unsupported feature -> `BackendError::Unsupported`
81
+
- anything else -> `BackendError::Other`
82
+
83
+
### OpStore mapping
84
+
85
+
- `not_found` -> `OpStoreError::ObjectNotFound`
86
+
- read failures -> `OpStoreError::ReadObject`
87
+
- write failures -> `OpStoreError::WriteObject`
88
+
- other -> `OpStoreError::Other`
89
+
90
+
### OpHeadsStore mapping
91
+
92
+
- get/list failures -> `OpHeadsStoreError::Read`
93
+
- update failures (excluding CAS miss) -> `OpHeadsStoreError::Write`
94
+
- lock failures -> `OpHeadsStoreError::Lock`
95
+
96
+
CAS miss path:
97
+
98
+
- represented by `ok=false` in `updateOpHeads`
99
+
- should not be converted to `OpHeadsStoreError`
100
+
101
+
## Retry policy
102
+
103
+
### Safe to retry automatically
104
+
105
+
- pure reads (`getObject`, `getOperation`, `getView`, `getHeads`)
106
+
- `watchHeads` subscribe after disconnect
107
+
108
+
### Retry with care
109
+
110
+
- writes only if idempotent by content-addressed semantics
111
+
- if write acknowledgment is unknown, client may re-issue same write bytes
112
+
113
+
### Do not blind-retry
114
+
115
+
- `invalid_data`, `invalid_id_length`, `unsupported`
116
+
117
+
## Observability requirements
118
+
119
+
Log on both client and server:
120
+
121
+
- `rpc.method`
122
+
- `rpc.error.code`
123
+
- `retriable`
124
+
- `attempt`
125
+
- `latency_ms`
126
+
- object/op identifiers (short hash prefix)
127
+
128
+
This allows debugging retries/concurrency without ad-hoc logging.
+201
docs/design-docs/rpc-protocol.md
+201
docs/design-docs/rpc-protocol.md
···
1
+
# RPC Protocol (Cap'n Proto)
2
+
3
+
This document defines Tandem’s wire protocol and storage data model for `jj-lib` compatibility.
4
+
5
+
Error semantics are defined in `rpc-error-model.md`.
6
+
7
+
## Goals
8
+
9
+
- Map cleanly to `jj_lib::backend::Backend`, `OpStore`, and `OpHeadsStore`.
10
+
- Preserve jj’s operation/view model and multi-workspace visibility.
11
+
- Keep the server authoritative for shared state while clients keep local working copies.
12
+
- Support low-latency reads and push-based head updates.
13
+
14
+
## Repository scope
15
+
16
+
- One Tandem server serves one repo.
17
+
- No `repoId` is sent in requests.
18
+
- Run multiple servers for multiple repos.
19
+
20
+
## Compatibility contract
21
+
22
+
Clients must call `getRepoInfo()` on connect and verify:
23
+
24
+
- protocol version compatibility
25
+
- jj object/op/view format compatibility
26
+
- expected ID lengths and root IDs
27
+
28
+
If incompatible, client should fail fast with a clear error.
29
+
30
+
## Data model
31
+
32
+
### Backend object kinds
33
+
34
+
- `commit`
35
+
- `tree`
36
+
- `file`
37
+
- `symlink`
38
+
- `copy`
39
+
40
+
### Op-store objects
41
+
42
+
- `operation`
43
+
- `view`
44
+
45
+
### Head state
46
+
47
+
- Current op-head set is stored server-side.
48
+
- Head updates are linearizable via compare-and-swap semantics.
49
+
- A monotonic `headsVersion` is incremented on successful head updates.
50
+
51
+
## Cap'n Proto interface (shape)
52
+
53
+
```capnp
54
+
interface Store {
55
+
getRepoInfo @0 () -> (info :RepoInfo);
56
+
57
+
getObject @1 (kind :ObjectKind, id :Data) -> (data :Data);
58
+
putObject @2 (kind :ObjectKind, data :Data) -> (id :Data, normalizedData :Data);
59
+
60
+
getOperation @3 (id :Data) -> (data :Data);
61
+
putOperation @4 (data :Data) -> (id :Data);
62
+
63
+
getView @5 (id :Data) -> (data :Data);
64
+
putView @6 (data :Data) -> (id :Data);
65
+
66
+
resolveOperationIdPrefix @7 (hexPrefix :Text)
67
+
-> (resolution :PrefixResolution, match :Data);
68
+
69
+
getHeads @8 () -> (heads :List(Data), version :UInt64);
70
+
updateOpHeads @9 (
71
+
oldIds :List(Data),
72
+
newId :Data,
73
+
expectedVersion :UInt64
74
+
) -> (ok :Bool, heads :List(Data), version :UInt64);
75
+
76
+
watchHeads @10 (watcher :HeadWatcher, afterVersion :UInt64)
77
+
-> (cancel :Cancel);
78
+
79
+
getHeadsSnapshot @11 () -> (
80
+
heads :List(Data),
81
+
version :UInt64,
82
+
operations :List(IdBytes),
83
+
views :List(IdBytes)
84
+
);
85
+
86
+
# Optional copy-tracking support (capability-gated)
87
+
getRelatedCopies @12 (copyId :Data) -> (copies :List(Data));
88
+
}
89
+
90
+
interface HeadWatcher {
91
+
notify @0 (version :UInt64, heads :List(Data)) -> ();
92
+
}
93
+
94
+
interface Cancel {
95
+
cancel @0 () -> ();
96
+
}
97
+
98
+
struct IdBytes {
99
+
id @0 :Data;
100
+
data @1 :Data;
101
+
}
102
+
103
+
enum ObjectKind {
104
+
commit @0;
105
+
tree @1;
106
+
file @2;
107
+
symlink @3;
108
+
copy @4;
109
+
}
110
+
111
+
enum PrefixResolution {
112
+
noMatch @0;
113
+
singleMatch @1;
114
+
ambiguous @2;
115
+
}
116
+
117
+
struct RepoInfo {
118
+
protocolMajor @0 :UInt16;
119
+
protocolMinor @1 :UInt16;
120
+
jjVersion @2 :Text;
121
+
122
+
backendName @3 :Text;
123
+
opStoreName @4 :Text;
124
+
125
+
commitIdLength @5 :UInt16;
126
+
changeIdLength @6 :UInt16;
127
+
128
+
rootCommitId @7 :Data;
129
+
rootChangeId @8 :Data;
130
+
emptyTreeId @9 :Data;
131
+
rootOperationId @10 :Data;
132
+
133
+
capabilities @11 :List(Capability);
134
+
}
135
+
136
+
enum Capability {
137
+
watchHeads @0;
138
+
headsSnapshot @1;
139
+
copyTracking @2;
140
+
}
141
+
```
142
+
143
+
## Method semantics
144
+
145
+
### `putObject`
146
+
147
+
- Server computes canonical object ID from bytes.
148
+
- Response returns canonical ID.
149
+
- Writes are idempotent (same object bytes => same ID).
150
+
- `normalizedData` allows commit write normalization; for non-commit objects it may equal input bytes.
151
+
152
+
### `putOperation` / `putView`
153
+
154
+
- Server computes IDs using jj-compatible content hashing.
155
+
- IDs and bytes must remain byte-compatible with jj expectations.
156
+
157
+
### `updateOpHeads`
158
+
159
+
- Logical behavior: remove `oldIds`, add `newId`.
160
+
- `ok=false` means caller must read current heads and retry merge/update flow.
161
+
- This operation is the concurrency correctness boundary.
162
+
163
+
### `watchHeads`
164
+
165
+
- Notifications are monotonic by `version`.
166
+
- Delivery is at-least-once and may coalesce rapid updates.
167
+
- On reconnect, client resubscribes with `afterVersion` and/or calls `getHeads()` to catch up.
168
+
169
+
### `getHeadsSnapshot`
170
+
171
+
- Fast path for dependent read chains (`heads -> operations -> views`).
172
+
- Returns a consistent snapshot tied to one `version`.
173
+
174
+
## Mapping to `jj-lib`
175
+
176
+
### Backend
177
+
178
+
- `read_*` -> `getObject(kind, id)`
179
+
- `write_*` -> `putObject(kind, data)`
180
+
- `get_related_copies` -> `getRelatedCopies` (when `copyTracking` capability exists)
181
+
182
+
### OpStore
183
+
184
+
- `read_operation` -> `getOperation`
185
+
- `write_operation` -> `putOperation`
186
+
- `read_view` -> `getView`
187
+
- `write_view` -> `putView`
188
+
- `resolve_operation_id_prefix` -> `resolveOperationIdPrefix`
189
+
190
+
### OpHeadsStore
191
+
192
+
- `get_op_heads` -> `getHeads`
193
+
- `update_op_heads` -> `updateOpHeads`
194
+
- `lock` -> client-local no-op lock (correctness comes from server-side CAS)
195
+
196
+
## Operational invariants
197
+
198
+
- `wc_commit_ids` in views is preserved exactly (workspace visibility model).
199
+
- Non-root operations must keep valid parent links.
200
+
- Head updates are durable before success responses.
201
+
- Object reads/writes must not require any client-side global cache for correctness.
+55
docs/exec-plans/active/slice-roadmap.md
+55
docs/exec-plans/active/slice-roadmap.md
···
1
+
# Active Execution Plan: Slice Roadmap
2
+
3
+
Canonical vertical-slice execution plan.
4
+
5
+
## Slice 1 — Single-agent round-trip
6
+
7
+
Goal: one client reads/writes via remote server and persists state.
8
+
9
+
Acceptance:
10
+
- `tandem log/new/describe/diff` work
11
+
- restarting client preserves state
12
+
- server-side `jj log` matches
13
+
14
+
## Slice 2 — Two-agent visibility
15
+
16
+
Goal: two workspaces on different machines see each other.
17
+
18
+
Acceptance:
19
+
- agent A and B both see each other's commits and workspaces
20
+
21
+
## Slice 3 — Concurrent convergence
22
+
23
+
Goal: concurrent writes do not lose data.
24
+
25
+
Acceptance:
26
+
- both (or all) concurrent commits survive after CAS contention
27
+
28
+
## Slice 4 — Promise pipelining
29
+
30
+
Goal: dependent reads avoid additive RTT cost.
31
+
32
+
Acceptance:
33
+
- latency benchmark proves pipelining behavior under artificial RPC delay
34
+
35
+
## Slice 5 — WatchHeads
36
+
37
+
Goal: clients receive head updates without polling.
38
+
39
+
Acceptance:
40
+
- callback receives updates quickly
41
+
- reconnect path catches up
42
+
43
+
## Slice 6 — Git round-trip
44
+
45
+
Goal: GitHub <-> server repo <-> clients round-trip via stock `jj git`.
46
+
47
+
Acceptance:
48
+
- fetch and push are successful with expected history/diff
49
+
50
+
## Slice 7 — End-to-end multi-agent
51
+
52
+
Goal: integrated real-repo workflow.
53
+
54
+
Acceptance:
55
+
- two agents collaborate concurrently and ship via server-side `jj git push`
+7
docs/exec-plans/completed/README.md
+7
docs/exec-plans/completed/README.md
+9
docs/exec-plans/tech-debt-tracker.md
+9
docs/exec-plans/tech-debt-tracker.md
···
1
+
# Tech Debt Tracker
2
+
3
+
## Initial items
4
+
5
+
- [ ] Define stable tracing event schema (`command_id`, `rpc_id`, `workspace`, `latency_ms`).
6
+
- [ ] Add redaction rules for logs (paths, tokens, secrets).
7
+
- [ ] Decide reconnect/backoff defaults for `watchHeads`.
8
+
- [ ] Verify object write idempotency contract and error codes.
9
+
- [ ] Add distributed smoke-test harness (`sprites.dev` / `exe.dev`) with env-gated CI step.
+23
docs/product-specs/core-product.md
+23
docs/product-specs/core-product.md
···
1
+
# Core Product Spec
2
+
3
+
## One-liner
4
+
5
+
`tandem` lets multiple agents/machines use jj workspaces against a shared remote jj store, as if they shared a filesystem.
6
+
7
+
## Primary users
8
+
9
+
- AI/code agents collaborating concurrently
10
+
- engineers using multiple machines
11
+
12
+
## Must-have outcomes
13
+
14
+
- same repo state visible across clients
15
+
- no stale workspace workflow
16
+
- safe concurrent writes (no lost updates)
17
+
- server remains plain jj+git compatible
18
+
19
+
## Out of scope (v0.1)
20
+
21
+
- authentication and tenant isolation
22
+
- UI layer
23
+
- policy/workflow automation