jj workspaces over the network
Continue development
+296
+296
README.md
+296
README.md
···
1
+
# Tandem
2
+
3
+
A native forge for [Jujutsu](https://github.com/martinvonz/jj) (jj) with real-time multiplayer collaboration. Tandem syncs jj changes, bookmarks, and user presence across multiple clients using CRDTs.
4
+
5
+
## Overview
6
+
7
+
Tandem provides a server-client architecture where:
8
+
9
+
- The **server** maintains the authoritative Y.Doc (Yrs CRDT document) for each repository
10
+
- **Clients** run a daemon that syncs local changes to the server over WebSocket
11
+
- All data merges automatically without conflicts using Yrs (the Rust port of Yjs)
12
+
13
+
This enables multiple developers to work on the same repository simultaneously with real-time visibility into who is editing what.
14
+
15
+
## Architecture
16
+
17
+
Tandem is organized into three crates:
18
+
19
+
### tandem-core
20
+
21
+
The shared library containing:
22
+
23
+
- **types.rs** - Core data types: `ChangeId`, `TreeHash`, `Change`, `ChangeRecord`, `Bookmark`, `PresenceInfo`
24
+
- **sync.rs** - `ForgeDoc` wrapping a Y.Doc with maps for changes, bookmarks, presence, and subdocuments for lazy content loading
25
+
26
+
### tandem-server
27
+
28
+
An Axum-based HTTP/WebSocket server:
29
+
30
+
- **sync.rs** - WebSocket endpoint at `/sync/:repo_id` handling bidirectional CRDT sync
31
+
- **docs.rs** - `DocManager` loads and persists Y.Doc state to `.yrs` files
32
+
- REST API for repositories, changes, bookmarks, presence, and content
33
+
34
+
### tandem-cli
35
+
36
+
The `jjf` command-line tool and background daemon:
37
+
38
+
- **main.rs** - CLI commands: `init`, `link`, `clone`, `daemon start`
39
+
- **daemon.rs** - Background sync daemon connecting to the forge
40
+
- **presence.rs** - Tracks who is editing which change
41
+
- **offline.rs** - Queues operations when disconnected for later replay
42
+
- **content.rs** - Lazy fetching of tree/blob content from the server
43
+
44
+
## Data Model
45
+
46
+
### Core Types
47
+
48
+
**ChangeId** - A stable 32-byte identifier for a change that persists across rebases. This is jj's native change ID.
49
+
50
+
**Change** - The unit of work containing:
51
+
- `id: ChangeId` - Stable identifier
52
+
- `tree: TreeHash` - Content-addressed 20-byte hash of the tree
53
+
- `parents: Vec<ChangeId>` - Parent changes
54
+
- `description: String` - Commit message
55
+
- `author: Identity` - Name and email
56
+
- `timestamp: DateTime<Utc>`
57
+
58
+
**ChangeRecord** - The CRDT-friendly wrapper stored in Y.Doc:
59
+
- `record_id: Uuid` - Unique key for the Y.Map entry
60
+
- `change_id: ChangeId` - The actual change ID
61
+
- `visible: bool` - False when abandoned/hidden
62
+
- All fields from `Change`
63
+
64
+
The distinction matters: a single `ChangeId` can have multiple `ChangeRecord` entries when a change diverges across clients before syncing. The CRDT merges them all, and the application layer decides how to present divergence.
65
+
66
+
**Bookmark** - A named pointer to a change (like a git branch):
67
+
- `name: String` - Bookmark name
68
+
- `target: ChangeId` - The change it points to
69
+
- `protected: bool` - Whether rules apply
70
+
- `rules: BookmarkRules` - CI/review requirements
71
+
72
+
**PresenceInfo** - Real-time editing status:
73
+
- `user_id: String` - Username
74
+
- `change_id: ChangeId` - Currently edited change
75
+
- `device: String` - Device name
76
+
- `timestamp: DateTime<Utc>` - Last update time (stale after 5 minutes)
77
+
78
+
### Y.Doc Structure
79
+
80
+
Each repository has **one main Y.Doc** for metadata, plus **separate subdocuments** (each its own Y.Doc) for content:
81
+
82
+
```
83
+
ForgeDoc
84
+
├── Main Doc (one per repo)
85
+
│ ├── Y.Map("changes") → {record_id: JSON(ChangeRecord)}
86
+
│ ├── Y.Map("bookmarks") → {name: ChangeId_hex}
87
+
│ └── Y.Map("presence") → {user_id: JSON(PresenceInfo)}
88
+
│
89
+
└── Subdocuments (HashMap<hash, Doc>)
90
+
├── "abc123" → Y.Doc { Y.Map("data") → {content: base64} }
91
+
├── "def456" → Y.Doc { Y.Map("data") → {content: base64} }
92
+
└── ...
93
+
```
94
+
95
+
This design enables:
96
+
1. **Fast metadata sync** - The main doc is small and syncs quickly
97
+
2. **Lazy content loading** - Each content blob is a separate Y.Doc fetched on-demand
98
+
3. **Independent sync** - Subdocuments can sync independently, so you only fetch content you need
99
+
100
+
## Sync Protocol
101
+
102
+
### State Vector Exchange
103
+
104
+
Yrs uses state vectors to track what each peer has seen. The sync flow:
105
+
106
+
1. **Client connects** via WebSocket to `/sync/:repo_id`
107
+
2. **Client sends state vector** - A compact encoding of which updates it has
108
+
3. **Server computes diff** - Encodes only the updates the client lacks
109
+
4. **Server sends update** - Binary Yrs update packet
110
+
5. **Client applies update** - Merges into its local Y.Doc
111
+
6. **Bidirectional sync continues** - Either side can send updates
112
+
113
+
```
114
+
Client Server
115
+
| |
116
+
|-- Binary(state_vector) ------>|
117
+
| | (compute diff)
118
+
|<----- Binary(update) ---------|
119
+
| |
120
+
|-- Binary(local_update) ------>| (from local edit)
121
+
| | (apply, save, broadcast)
122
+
| |
123
+
```
124
+
125
+
### Server-Side Sync (tandem-server/src/sync.rs)
126
+
127
+
The server handles each WebSocket message:
128
+
129
+
1. **Try to apply as update** - If valid Yrs update, apply it
130
+
2. **On success** - Save to disk, broadcast to other clients (excluding sender)
131
+
3. **On failure** - Assume it's a state vector, compute and send diff
132
+
133
+
The `SyncManager` maintains broadcast channels per repository so updates propagate to all connected clients.
134
+
135
+
### Client-Side Sync (tandem-cli/src/daemon.rs)
136
+
137
+
The daemon:
138
+
139
+
1. Connects to the forge WebSocket
140
+
2. Sends its state vector to request initial sync
141
+
3. Receives and applies updates from the server
142
+
4. When local changes happen, sends updates to the server
143
+
5. On disconnect, enters offline mode and queues operations
144
+
145
+
## Conflict Resolution
146
+
147
+
### Why CRDTs Avoid Traditional Conflicts
148
+
149
+
Yrs implements a CRDT (Conflict-free Replicated Data Type). Every operation is designed to commute - the same set of operations applied in any order produces the same result.
150
+
151
+
For Tandem:
152
+
153
+
- **Changes** are keyed by unique `record_id` UUIDs. Two clients creating a record for the same change get two records, not a conflict.
154
+
- **Bookmarks** use last-writer-wins semantics on the Y.Map. Concurrent moves to different targets result in one winning.
155
+
- **Presence** uses last-writer-wins per user. Stale entries are filtered out by timestamp.
156
+
157
+
### Divergence Handling
158
+
159
+
When the same `ChangeId` gets edited on two disconnected clients:
160
+
161
+
1. Each client creates a new `ChangeRecord` with a unique `record_id`
162
+
2. On reconnect, both records sync to all clients
163
+
3. `get_change_records(change_id)` returns multiple records
164
+
4. The application must decide which is canonical (by timestamp, by user, or by user choice)
165
+
166
+
Hidden/abandoned changes are marked with `visible: false` rather than deleted, preserving history.
167
+
168
+
### Edge Cases
169
+
170
+
- **Bookmark races** - If Alice moves `main` to change A while Bob moves it to B, one wins. The CRDT resolves this deterministically but the "loser" may need to re-move the bookmark.
171
+
- **Offline queuing** - Operations made while disconnected are stored in `.jj/forge-queue.json` and replayed on reconnect.
172
+
- **Stale presence** - Presence entries older than 5 minutes are filtered out.
173
+
174
+
## Getting Started
175
+
176
+
### Starting the Server
177
+
178
+
```bash
179
+
cd crates/tandem-server
180
+
DATABASE_URL=sqlite:tandem.db DATA_DIR=./data cargo run
181
+
```
182
+
183
+
The server starts on `http://localhost:3000` with:
184
+
- REST API at `/api/*`
185
+
- WebSocket sync at `/sync/:repo_id`
186
+
- Health check at `/health`
187
+
188
+
### Linking a Repository
189
+
190
+
In an existing jj repository:
191
+
192
+
```bash
193
+
jjf link https://forge.example.com/org/myrepo --token <auth_token>
194
+
```
195
+
196
+
This:
197
+
1. Tests the connection to the forge
198
+
2. Creates `.jj/forge.toml` with the forge URL
199
+
3. Prints instructions to start the daemon
200
+
201
+
### Cloning from Forge
202
+
203
+
```bash
204
+
jjf clone https://forge.example.com/org/myrepo
205
+
```
206
+
207
+
This:
208
+
1. Creates the target directory
209
+
2. Runs `jj init`
210
+
3. Links to the forge
211
+
4. Pulls initial state via WebSocket sync
212
+
5. Saves the Y.Doc state to `.jj/forge-doc.bin`
213
+
214
+
### Running the Daemon
215
+
216
+
```bash
217
+
jjf daemon start
218
+
```
219
+
220
+
The daemon:
221
+
1. Loads forge config from `.jj/forge.toml`
222
+
2. Connects to the forge WebSocket
223
+
3. Syncs bidirectionally in real-time
224
+
4. Tracks presence (which change you're editing)
225
+
5. Warns when someone else is editing the same change
226
+
6. Queues operations when offline, replays on reconnect
227
+
228
+
### Presence Tracking
229
+
230
+
The daemon uses `whoami` to identify the current user and device. When you edit a change, it broadcasts your presence. Other users see warnings like:
231
+
232
+
```
233
+
Warning: This change is currently being edited by alice@laptop
234
+
```
235
+
236
+
Presence entries expire after 5 minutes of inactivity.
237
+
238
+
## Configuration
239
+
240
+
### .jj/forge.toml
241
+
242
+
Created by `jjf link`, stores the forge URL:
243
+
244
+
```toml
245
+
[forge]
246
+
url = "https://forge.example.com/org/myrepo"
247
+
```
248
+
249
+
### .jj/forge-queue.json
250
+
251
+
Stores queued operations when offline:
252
+
253
+
```json
254
+
{
255
+
"operations": [
256
+
{
257
+
"type": "change_updated",
258
+
"record": { ... },
259
+
"timestamp": "2024-01-01T00:00:00Z"
260
+
}
261
+
]
262
+
}
263
+
```
264
+
265
+
### .jj/forge-offline
266
+
267
+
Marker file indicating offline mode. Removed when connection is restored.
268
+
269
+
## API Reference
270
+
271
+
### REST Endpoints
272
+
273
+
| Method | Path | Description |
274
+
|--------|------|-------------|
275
+
| POST | `/api/auth/login` | Authenticate and get token |
276
+
| GET | `/api/auth/me` | Get current user |
277
+
| GET | `/api/repos` | List repositories |
278
+
| POST | `/api/repos` | Create repository |
279
+
| GET | `/api/repos/:id` | Get repository details |
280
+
| GET | `/api/repos/:id/changes` | List changes |
281
+
| GET | `/api/repos/:id/changes/:cid` | Get specific change |
282
+
| GET | `/api/repos/:id/bookmarks` | List bookmarks |
283
+
| POST | `/api/repos/:id/bookmarks` | Move bookmark |
284
+
| GET | `/api/repos/:id/presence` | Get active presence |
285
+
| GET | `/api/repos/:id/content/:hash` | Fetch content by hash |
286
+
287
+
### WebSocket Endpoints
288
+
289
+
| Path | Description |
290
+
|------|-------------|
291
+
| `/sync/:repo_id` | Bidirectional Yrs sync |
292
+
| `/events/:repo_id` | Server-sent events for notifications |
293
+
294
+
## License
295
+
296
+
MIT OR Apache-2.0