jj workspaces over the network
docs: tandem more server
+332
-233
-34
demos/config.tape
-34
demos/config.tape
···
1
-
# Shared VHS configuration for tandem demos
2
-
# Usage: Source config.tape
3
-
4
-
Set Shell "bash"
5
-
Set FontFamily "JetBrains Mono"
6
-
Set FontSize 14
7
-
Set Width 1200
8
-
Set Height 720
9
-
Set Padding 20
10
-
Set Framerate 30
11
-
Set TypingSpeed 40ms
12
-
Set Theme {
13
-
"name": "tandem",
14
-
"black": "#1a1b26",
15
-
"red": "#f7768e",
16
-
"green": "#9ece6a",
17
-
"yellow": "#e0af68",
18
-
"blue": "#7aa2f7",
19
-
"magenta": "#bb9af7",
20
-
"cyan": "#7dcfff",
21
-
"white": "#c0caf5",
22
-
"brightBlack": "#565f89",
23
-
"brightRed": "#f7768e",
24
-
"brightGreen": "#9ece6a",
25
-
"brightYellow": "#e0af68",
26
-
"brightBlue": "#7aa2f7",
27
-
"brightMagenta": "#bb9af7",
28
-
"brightCyan": "#7dcfff",
29
-
"brightWhite": "#c0caf5",
30
-
"background": "#1a1b26",
31
-
"foreground": "#c0caf5",
32
-
"selection": "#33467c",
33
-
"cursor": "#c0caf5"
34
-
}
-8
demos/scripts/agent-a.sh
-8
demos/scripts/agent-a.sh
···
1
-
#!/bin/bash
2
-
set -e
3
-
chmod +x ~/tandem
4
-
~/tandem init --tandem-server=localhost:13013 ~/work
5
-
mkdir -p ~/work/src
6
-
echo 'pub fn authenticate(token: &str) -> bool { !token.is_empty() }' > ~/work/src/auth.rs
7
-
echo 'pub mod auth;' > ~/work/src/lib.rs
8
-
cd ~/work && ~/tandem --config=fsmonitor.backend=none new -m 'feat: add auth module'
-7
demos/scripts/agent-b.sh
-7
demos/scripts/agent-b.sh
···
1
-
#!/bin/bash
2
-
set -e
3
-
chmod +x ~/tandem
4
-
~/tandem init --tandem-server=localhost:13013 --workspace=agent-b ~/work
5
-
mkdir -p ~/work/src
6
-
echo 'pub fn handle_request(path: &str) -> u16 { 200 }' > ~/work/src/api.rs
7
-
cd ~/work && ~/tandem --config=fsmonitor.backend=none new -m 'feat: add API routes'
-8
demos/scripts/server-start.sh
-8
demos/scripts/server-start.sh
-174
demos/tandem-exe-dev.tape
-174
demos/tandem-exe-dev.tape
···
1
-
Source demos/config.tape
2
-
Output demos/tandem-exe-dev.gif
3
-
4
-
# ============================================================================
5
-
# tandem: distributed jj workspaces across 3 VMs on exe.dev
6
-
#
7
-
# Two AI agents on separate VMs collaborating on code through a shared
8
-
# tandem server. Each agent sees the other's commits instantly.
9
-
# ============================================================================
10
-
11
-
Sleep 1s
12
-
13
-
# -- Create 3 VMs on exe.dev ------------------------------------------------
14
-
15
-
Type "# Create three exe.dev VMs: server + two agents"
16
-
Enter
17
-
Sleep 1s
18
-
19
-
Type "ssh exe.dev new --name tandem-server"
20
-
Enter
21
-
Wait@30s
22
-
Sleep 2s
23
-
24
-
Type "ssh exe.dev new --name tandem-agent-a"
25
-
Enter
26
-
Wait@30s
27
-
Sleep 2s
28
-
29
-
Type "ssh exe.dev new --name tandem-agent-b"
30
-
Enter
31
-
Wait@30s
32
-
Sleep 2s
33
-
34
-
# -- Copy tandem binary + scripts -------------------------------------------
35
-
36
-
Type "# Copy tandem binary to all VMs"
37
-
Enter
38
-
Sleep 1s
39
-
40
-
Type "BIN=target/x86_64-unknown-linux-musl/release/tandem"
41
-
Enter
42
-
Sleep 300ms
43
-
44
-
Type "scp $BIN tandem-server.exe.xyz:~/tandem"
45
-
Enter
46
-
Wait@60s
47
-
Sleep 1s
48
-
49
-
Type "scp $BIN tandem-agent-a.exe.xyz:~/tandem"
50
-
Enter
51
-
Wait@60s
52
-
Sleep 1s
53
-
54
-
Type "scp $BIN tandem-agent-b.exe.xyz:~/tandem"
55
-
Enter
56
-
Wait@60s
57
-
Sleep 2s
58
-
59
-
# -- Start the tandem server ------------------------------------------------
60
-
61
-
Type "# Start tandem server"
62
-
Enter
63
-
Sleep 1s
64
-
65
-
Type "scp demos/scripts/server-start.sh tandem-server.exe.xyz:/tmp/start.sh"
66
-
Enter
67
-
Wait@15s
68
-
Sleep 500ms
69
-
70
-
Type "ssh tandem-server.exe.xyz bash /tmp/start.sh"
71
-
Enter
72
-
Wait@15s
73
-
Sleep 2s
74
-
75
-
# -- Set up SSH tunnels ------------------------------------------------------
76
-
77
-
Type "# SSH tunnels: bridge raw TCP between VMs via localhost"
78
-
Enter
79
-
Sleep 1s
80
-
81
-
Type "ssh -f -N -L 15555:localhost:5555 tandem-server.exe.xyz"
82
-
Enter
83
-
Wait@10s
84
-
Sleep 1s
85
-
86
-
Type "ssh -f -N -R 13013:localhost:15555 tandem-agent-a.exe.xyz"
87
-
Enter
88
-
Wait@10s
89
-
Sleep 1s
90
-
91
-
Type "ssh -f -N -R 13013:localhost:15555 tandem-agent-b.exe.xyz"
92
-
Enter
93
-
Wait@10s
94
-
Sleep 2s
95
-
96
-
# -- Agent A: write auth module ----------------------------------------------
97
-
98
-
Type "# --- Agent A: write auth module ---"
99
-
Enter
100
-
Sleep 1s
101
-
102
-
Type "scp demos/scripts/agent-a.sh tandem-agent-a.exe.xyz:/tmp/setup.sh"
103
-
Enter
104
-
Wait@15s
105
-
Sleep 500ms
106
-
107
-
Type "ssh tandem-agent-a.exe.xyz bash /tmp/setup.sh"
108
-
Enter
109
-
Wait@30s
110
-
Sleep 2s
111
-
112
-
Type "# Agent A sees their commit"
113
-
Enter
114
-
Sleep 500ms
115
-
116
-
Type "ssh tandem-agent-a.exe.xyz 'cd ~/work && ~/tandem --config=fsmonitor.backend=none log'"
117
-
Enter
118
-
Wait@15s
119
-
Sleep 4s
120
-
121
-
# -- Agent B: see Agent A, then add API routes --------------------------------
122
-
123
-
Type "# --- Agent B: init workspace, see Agent A's work ---"
124
-
Enter
125
-
Sleep 1s
126
-
127
-
Type "scp demos/scripts/agent-b.sh tandem-agent-b.exe.xyz:/tmp/setup.sh"
128
-
Enter
129
-
Wait@15s
130
-
Sleep 500ms
131
-
132
-
Type "ssh tandem-agent-b.exe.xyz bash /tmp/setup.sh"
133
-
Enter
134
-
Wait@30s
135
-
Sleep 2s
136
-
137
-
Type "# Agent B sees both workspaces in the log"
138
-
Enter
139
-
Sleep 500ms
140
-
141
-
Type "ssh tandem-agent-b.exe.xyz 'cd ~/work && ~/tandem --config=fsmonitor.backend=none log'"
142
-
Enter
143
-
Wait@15s
144
-
Sleep 4s
145
-
146
-
Type "# Agent B reads Agent A's auth.rs from the shared store"
147
-
Enter
148
-
Sleep 500ms
149
-
150
-
Type "ssh tandem-agent-b.exe.xyz 'cd ~/work && ~/tandem --config=fsmonitor.backend=none file show -r @-- src/auth.rs'"
151
-
Enter
152
-
Wait@15s
153
-
Sleep 4s
154
-
155
-
# -- Server: everything is there --------------------------------------------
156
-
157
-
Type "# --- Server: all commits from both agents ---"
158
-
Enter
159
-
Sleep 1s
160
-
161
-
Type "ssh tandem-server.exe.xyz 'cd ~/project && ~/tandem --config=fsmonitor.backend=none log --no-graph --ignore-working-copy'"
162
-
Enter
163
-
Wait@15s
164
-
Sleep 4s
165
-
166
-
Type "# Server has everything. Ready for: jj git push"
167
-
Enter
168
-
Sleep 3s
169
-
170
-
# -- Fin ---------------------------------------------------------------------
171
-
172
-
Type "# Two agents, three VMs, one store. That's tandem."
173
-
Enter
174
-
Sleep 4s
+1
docs/design-docs/index.md
+1
docs/design-docs/index.md
+211
docs/design-docs/server-lifecycle.md
+211
docs/design-docs/server-lifecycle.md
···
1
+
# Server Lifecycle (up/down/status/logs)
2
+
3
+
## Motivation
4
+
5
+
Users shouldn't need to understand systemd, launchd, or process management to
6
+
run a tandem server. `tandem up` starts it, `tandem down` stops it,
7
+
`tandem status` tells you if it's running. Same model as Tailscale (`tailscale up`)
8
+
and Caddy (`caddy start`).
9
+
10
+
## API surface
11
+
12
+
```
13
+
tandem up --repo /srv/project --listen 0.0.0.0:13013 # start daemon, return
14
+
tandem down # stop daemon
15
+
tandem status # health check
16
+
tandem status --json # machine-readable
17
+
tandem logs # stream logs from daemon
18
+
tandem logs --level debug # stream at higher verbosity
19
+
```
20
+
21
+
`tandem serve` remains the foreground mode for systemd/docker/debugging:
22
+
23
+
```
24
+
tandem serve --repo /srv/project --listen 0.0.0.0:13013
25
+
tandem serve --log-level debug --log-file /var/log/tandem.log --log-format json
26
+
tandem serve --pidfile /var/run/tandem.pid
27
+
```
28
+
29
+
## Fork model
30
+
31
+
`tandem up` forks itself as a background process. No separate daemon binary.
32
+
33
+
1. `tandem up` validates flags (repo exists, port parseable).
34
+
2. Forks `tandem serve --daemon` with same flags. `--daemon` is internal/hidden.
35
+
3. Parent waits for child to signal readiness (control socket exists + health OK).
36
+
4. Parent prints "tandem running, PID <n>" and exits 0.
37
+
5. If child fails to start within timeout (5s default), parent exits 1 with error.
38
+
39
+
The `--daemon` flag tells `serve` to:
40
+
- Detach from terminal (setsid, close stdin/stdout/stderr).
41
+
- Write PID file to `$XDG_RUNTIME_DIR/tandem/daemon.pid`.
42
+
- Create control socket.
43
+
- Redirect logs to `$XDG_RUNTIME_DIR/tandem/daemon.log` (unless --log-file overrides).
44
+
45
+
Same pattern as Caddy's `caddy start` → `caddy run --environ`.
46
+
47
+
### Already running
48
+
49
+
`tandem up` when a daemon is already running: exit 1 with
50
+
"tandem is already running (PID <n>). Use `tandem down` first."
51
+
52
+
Detected via control socket liveness check, not just PID file existence.
53
+
54
+
## Control socket
55
+
56
+
Path: `$XDG_RUNTIME_DIR/tandem/control.sock` (Linux) or
57
+
`$TMPDIR/tandem/control.sock` (macOS). Override with `--control-socket <path>`.
58
+
59
+
Protocol: HTTP/1.1 over Unix domain socket. Reasons:
60
+
61
+
- Reuse hyper/axum (same stack as the HTTP API feature).
62
+
- Structured request/response with status codes.
63
+
- Easy to curl for debugging: `curl --unix-socket /path/to/control.sock http://localhost/status`
64
+
- No need to invent a framing protocol.
65
+
66
+
### Control endpoints
67
+
68
+
```
69
+
GET /status → { "pid": 1234, "uptime_secs": 3600, "repo": "/srv/project", ... }
70
+
POST /shutdown → 200 OK, daemon begins graceful shutdown
71
+
GET /logs?level=debug → SSE stream of log events (text/event-stream)
72
+
```
73
+
74
+
The control socket is **local-only** (Unix socket permissions). No auth needed.
75
+
76
+
## Log streaming
77
+
78
+
`tandem logs` connects to the control socket's `/logs` SSE endpoint.
79
+
80
+
Key design: the daemon always logs at trace level internally (ring buffer or
81
+
tracing subscriber). `tandem logs --level info` filters server-side before
82
+
streaming. This means you can attach at debug level to a daemon that was
83
+
started with `--log-level info` — the Consul `consul monitor` pattern.
84
+
85
+
Implementation: tracing subscriber that fans out to:
86
+
1. File/stderr (at configured --log-level).
87
+
2. Zero or more SSE clients (each with independent level filter).
88
+
89
+
Log format over SSE:
90
+
91
+
```
92
+
data: {"ts":"2026-02-19T18:00:00Z","level":"info","target":"tandem::server","msg":"client connected","fields":{"addr":"10.0.0.5:44312"}}
93
+
```
94
+
95
+
`tandem logs` renders these as human-readable lines by default.
96
+
`tandem logs --json` passes the raw JSON through.
97
+
98
+
### No daemon running
99
+
100
+
`tandem logs` when no daemon is running: exit 1 with
101
+
"no tandem daemon running. Start one with `tandem up`."
102
+
103
+
## Status output
104
+
105
+
`tandem status` (human-readable):
106
+
107
+
```
108
+
tandem is running
109
+
PID: 1234
110
+
Uptime: 2h 15m
111
+
Repo: /srv/project
112
+
Listen: 0.0.0.0:13013
113
+
Version: 0.3.0
114
+
```
115
+
116
+
`tandem status --json`:
117
+
118
+
```json
119
+
{
120
+
"running": true,
121
+
"pid": 1234,
122
+
"uptime_secs": 8100,
123
+
"repo": "/srv/project",
124
+
"listen": "0.0.0.0:13013",
125
+
"version": "0.3.0",
126
+
"workspaces": 3
127
+
}
128
+
```
129
+
130
+
Exit codes: 0 = running, 1 = not running / unreachable.
131
+
132
+
When not running:
133
+
134
+
```
135
+
tandem is not running
136
+
```
137
+
138
+
## Signal handling
139
+
140
+
- **SIGTERM**: graceful shutdown. Drain in-flight RPCs (5s timeout), close
141
+
sockets, remove PID file and control socket, exit 0.
142
+
- **SIGINT** (Ctrl+C): same as SIGTERM. Already needed for foreground `tandem serve`.
143
+
- **SIGHUP**: reserved for future config reload. Currently ignored.
144
+
- **Second SIGTERM/SIGINT**: immediate exit.
145
+
146
+
## Relationship to tandem serve
147
+
148
+
| | `tandem serve` | `tandem up` |
149
+
|---|---|---|
150
+
| Foreground | yes | no |
151
+
| Logs to stderr | yes (default) | no (logs to file) |
152
+
| Control socket | yes | yes |
153
+
| PID file | opt-in (--pidfile) | auto-managed |
154
+
| systemd/docker | yes | not needed |
155
+
| Human operator | debugging | normal use |
156
+
157
+
Both modes create the control socket. `tandem down` / `tandem status` /
158
+
`tandem logs` work against either mode.
159
+
160
+
## Flags summary
161
+
162
+
### tandem serve (existing + new)
163
+
164
+
```
165
+
--listen <addr> Cap'n Proto listen address (required)
166
+
--repo <path> Repository path (required)
167
+
--log-level <level> trace|debug|info|warn|error (default: info)
168
+
--log-file <path> Log to file instead of stderr
169
+
--log-format <fmt> text|json (default: text)
170
+
--pidfile <path> Write PID file (opt-in)
171
+
--control-socket <path> Override control socket path
172
+
--daemon Internal flag, set by `tandem up`
173
+
```
174
+
175
+
### tandem up
176
+
177
+
```
178
+
--listen <addr> Cap'n Proto listen address (required)
179
+
--repo <path> Repository path (required)
180
+
--log-level <level> Daemon log level (default: info)
181
+
--log-file <path> Daemon log file (default: $XDG_RUNTIME_DIR/tandem/daemon.log)
182
+
```
183
+
184
+
### tandem down
185
+
186
+
No flags. Finds daemon via control socket.
187
+
188
+
### tandem status
189
+
190
+
```
191
+
--json Machine-readable output
192
+
```
193
+
194
+
### tandem logs
195
+
196
+
```
197
+
--level <level> Filter level (default: info)
198
+
--json Raw JSON output
199
+
```
200
+
201
+
## Open questions
202
+
203
+
1. **Multiple daemons.** Current design assumes one daemon per user (single
204
+
control socket path). Should we support named instances for serving multiple
205
+
repos? Could use `--name <n>` with per-name socket paths. Punt until needed.
206
+
207
+
2. **Log retention.** How large should the daemon log file grow? Rotation
208
+
policy? Probably punt to logrotate / the OS for now.
209
+
210
+
3. **macOS launchd.** Should `tandem up` optionally install a launchd plist
211
+
for auto-restart? Probably not — keep it simple, add later if needed.
+109
docs/exec-plans/active/server-lifecycle.md
+109
docs/exec-plans/active/server-lifecycle.md
···
1
+
# Execution Plan: Server Lifecycle
2
+
3
+
**Design doc:** `docs/design-docs/server-lifecycle.md`
4
+
5
+
Implements `tandem up/down/status/logs` — daemon management without systemd.
6
+
7
+
## Slice 10 — Signal handling and graceful shutdown
8
+
9
+
**Goal:** `tandem serve` handles SIGTERM/SIGINT cleanly. Prerequisite for
10
+
everything else — daemon mode needs reliable shutdown.
11
+
12
+
**Work:**
13
+
14
+
- Install signal handler (tokio::signal) in `tandem serve`.
15
+
- On SIGTERM/SIGINT: stop accepting new connections, drain in-flight RPCs
16
+
(5s timeout), close listeners, exit 0.
17
+
- Second signal: immediate exit.
18
+
- Add `--log-level` and `--log-format` flags to `tandem serve`.
19
+
20
+
**Acceptance:**
21
+
22
+
- `tandem serve` + SIGINT exits 0 (not 130).
23
+
- In-flight `getObject` call during shutdown completes (not dropped).
24
+
- `--log-level debug` produces debug output to stderr.
25
+
- Existing slice 1-7 tests still pass.
26
+
27
+
**Test:** `tests/slice10_graceful_shutdown.rs`
28
+
- Start server, connect client, send SIGTERM, verify clean exit.
29
+
- Start server, begin slow read, send SIGTERM, verify read completes.
30
+
31
+
## Slice 11 — Control socket and tandem status
32
+
33
+
**Goal:** `tandem serve` opens a control socket. `tandem status` queries it.
34
+
35
+
**Work:**
36
+
37
+
- Add HTTP-over-Unix-socket listener to `tandem serve` (axum + hyper-unix).
38
+
- Implement `GET /status` on control socket.
39
+
- Socket path: `$XDG_RUNTIME_DIR/tandem/control.sock` (Linux),
40
+
`$TMPDIR/tandem/control.sock` (macOS). Override with `--control-socket`.
41
+
- Implement `tandem status` command: connect to control socket, print output.
42
+
- Implement `tandem status --json`.
43
+
- Exit code 0 = running, 1 = not running.
44
+
45
+
**Acceptance:**
46
+
47
+
- `tandem serve` creates control socket.
48
+
- `tandem status` prints human-readable output while server runs.
49
+
- `tandem status --json` returns valid JSON with pid, uptime, repo, listen fields.
50
+
- `tandem status` exits 1 when no server is running.
51
+
- Control socket is cleaned up on server exit (from slice 10).
52
+
53
+
**Test:** `tests/slice11_control_socket.rs`
54
+
- Start server, run `tandem status --json`, parse output, verify fields.
55
+
- No server running, run `tandem status`, verify exit code 1.
56
+
57
+
## Slice 12 — tandem up and tandem down
58
+
59
+
**Goal:** `tandem up` starts a background daemon. `tandem down` stops it.
60
+
61
+
**Work:**
62
+
63
+
- Implement `--daemon` internal flag on `tandem serve` (detach, redirect
64
+
stdio, write PID file).
65
+
- Implement `tandem up`: validate flags, fork `tandem serve --daemon`, wait
66
+
for control socket readiness, print PID, exit 0.
67
+
- Implement `tandem down`: connect to control socket, `POST /shutdown`,
68
+
wait for process exit.
69
+
- `tandem up` when already running: exit 1 with message.
70
+
- PID file at `$XDG_RUNTIME_DIR/tandem/daemon.pid`.
71
+
72
+
**Acceptance:**
73
+
74
+
- `tandem up --repo ... --listen ...` returns immediately, daemon is running.
75
+
- `tandem status` shows running after `tandem up`.
76
+
- `tandem down` stops daemon, `tandem status` shows not running.
77
+
- `tandem up` twice: second invocation errors with "already running".
78
+
- PID file and control socket cleaned up after `tandem down`.
79
+
80
+
**Test:** `tests/slice12_up_down.rs`
81
+
- `tandem up`, verify status, connect client, read object, `tandem down`, verify stopped.
82
+
- `tandem up` twice, verify error.
83
+
84
+
## Slice 13 — tandem logs (streaming)
85
+
86
+
**Goal:** `tandem logs` streams log output from a running daemon.
87
+
88
+
**Work:**
89
+
90
+
- Add tracing subscriber that fans out to: file/stderr + SSE clients.
91
+
- Implement `GET /logs?level=<level>` on control socket (SSE stream).
92
+
- Implement `tandem logs` command: connect to SSE endpoint, print lines.
93
+
- `--level` flag on `tandem logs` (default: info).
94
+
- `--json` flag for raw JSON log lines.
95
+
- Client can request higher verbosity than daemon's file log level.
96
+
97
+
**Acceptance:**
98
+
99
+
- `tandem logs` prints log lines as events happen.
100
+
- `tandem logs --level debug` shows debug events even if daemon was started
101
+
with `--log-level info`.
102
+
- `tandem logs --json` outputs one JSON object per line.
103
+
- `tandem logs` exits cleanly when daemon shuts down.
104
+
- `tandem logs` with no daemon: exit 1 with helpful message.
105
+
106
+
**Test:** `tests/slice13_log_streaming.rs`
107
+
- Start daemon, connect client, trigger activity, verify `tandem logs` output
108
+
contains expected event.
109
+
- Verify `--level debug` produces more output than `--level warn`.
+3
-1
docs/exec-plans/tech-debt-tracker.md
+3
-1
docs/exec-plans/tech-debt-tracker.md
···
29
29
- Add redaction rules for logs (paths, tokens, secrets)
30
30
- Decide reconnect/backoff defaults for `watchHeads`
31
31
- Verify object write idempotency contract and error codes
32
-
- Clean shutdown for server (Ctrl+C signal handling)
32
+
- Clean shutdown for server (Ctrl+C signal handling) — now part of server lifecycle feature, see `docs/exec-plans/active/server-lifecycle.md` slice 10
33
33
- Add distributed smoke-test harness (`sprites.dev` / `exe.dev`) with env-gated CI step
34
+
- Control socket protocol design — finalize HTTP-over-Unix-socket vs alternatives, see `docs/design-docs/server-lifecycle.md`
35
+
- Capnp token auth handshake design — how to validate bearer token during capnp connection setup
34
36
35
37
### P3 (performance, not correctness)
36
38
+8
-1
docs/product-specs/core-product.md
+8
-1
docs/product-specs/core-product.md
···
18
18
19
19
## Out of scope
20
20
21
-
- authentication and tenant isolation
21
+
- multi-tenant isolation and user/role auth model
22
22
- UI layer
23
23
- policy/workflow automation
24
+
25
+
## Planned
26
+
27
+
- **Server lifecycle management** — `tandem up/down/status/logs` for daemon
28
+
management without systemd. See `docs/design-docs/server-lifecycle.md`.
29
+
- **Token auth** — bearer token on Cap'n Proto port. Single shared secret,
30
+
no user/role model. Required for servers on public networks.