wip: benchmarks for testing different p2p sync strategies using a pds as a relay
v1 report
+516
-304
+75
-138
README.md
+75
-138
README.md
···
1
-
# Merge Strategy Benchmark Report
1
+
# merge-bench
2
2
3
-
**Date:** 2026-03-13
4
-
**PDS:** bluesky-pds.t1cc.commoninternet.net (v0.4.208)
5
-
**Modes:** Local (no network) and Remote (real PDS round-trips)
3
+
Benchmark harness comparing merge strategies for collaborative text editing via AT Protocol PDS.
6
4
7
-
## Strategies
5
+
## Why
8
6
9
-
1. **plain-git** — Standard `git merge` 3-way merge
10
-
2. **yrs-diff** — git merge with git-yrs-merge CRDT driver (diff-only mode)
11
-
3. **yrs-sidecar** — git merge with git-yrs-merge CRDT driver + .yrs/ sidecar files
12
-
4. **yrs-on-pds** — Pure Yrs CRDT merge (local: in-memory from .pds-sim/; PDS: pds-yrs save/merge/load)
7
+
Lichen uses Yrs CRDTs to enable conflict-free collaborative editing on sites stored in AT Protocol PDS repositories. There are several possible architectures for how collaborators merge their changes:
13
8
14
-
## Default Matrix (10/50/200 files, 2 collaborators)
9
+
1. **Plain git** -- standard 3-way merge, conflicts require manual resolution
10
+
2. **git + yrs-merge (diff)** -- git merge with a CRDT merge driver that resolves conflicts automatically using Yrs document state
11
+
3. **git + yrs-merge (sidecar)** -- same as above, but with `.yrs/` sidecar files tracked in git for richer CRDT state
12
+
4. **Yrs-on-PDS** -- pure CRDT merge via `pds-yrs`, no git involved; each collaborator saves Yrs state to PDS and merges happen server-side
15
13
16
-
### Local
14
+
This tool generates synthetic repositories with configurable parameters (file count, file size, collaborator count, conflict rate) and measures each strategy's merge time, conflict count, and success rate.
17
15
18
-
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
19
-
|-------|----------|-----------|-------------|-------------|----------------|------------|
20
-
| 10 | low | 34 | 0 | 62 | 108 | 5 |
21
-
| 10 | medium | 32 | 1 | 57 | 106 | 3 |
22
-
| 10 | high | 30 | 0 | 85 | 122 | 3 |
23
-
| 50 | low | 39 | 0 | 60 | 91 | 23 |
24
-
| 50 | medium | 40 | 1 | 61 | 114 | 14 |
25
-
| 50 | high | 42 | 1 | 109 | 147 | 17 |
26
-
| 200 | low | 53 | 0 | 81 | 125 | 82 |
27
-
| 200 | medium | 62 | 1 | 117 | 208 | 65 |
28
-
| 200 | high | 65 | 0 | 264 | 472 | 68 |
16
+
Both **local** (no network, pure computation) and **remote** (real PDS round-trips via `git-remote-pds` and `pds-yrs`) modes are supported.
29
17
30
-
### Remote (PDS)
18
+
## Usage
31
19
32
-
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
33
-
|-------|----------|-----------|-------------|-------------|----------------|------------|
34
-
| 10 | low | 7094 | 0 | 6729 | 6702 | 3783 |
35
-
| 10 | medium | 6823 | 0 | 6776 | 6738 | 3924 |
36
-
| 10 | high | 6877 | 1 | 6764 | 8074 | 4471 |
37
-
| 50 | low | 7033 | 0 | 7089 | 7424 | 9316 |
38
-
| 50 | medium | 6999 | 0 | 7423 | 7973 | 9224 |
39
-
| 50 | high | 6882 | 0 | 7049 | 6963 | 9378 |
40
-
| 200 | low | 7000 | 0 | 7406 | 7388 | 28381 |
41
-
| 200 | medium | 6826 | 0 | 9745 | 7370 | 28995 |
42
-
| 200 | high | 7503 | 1 | 7327 | 7672 | 27605 |
20
+
```bash
21
+
cargo build
22
+
```
43
23
44
-
## Stress Test (1000 files, 2 collaborators)
24
+
### Run benchmarks
45
25
46
-
### Local
26
+
```bash
27
+
# Local mode (default) -- measures pure merge cost
28
+
merge-bench run # default matrix (10/50/200 files x low/med/high conflict)
29
+
merge-bench run --quick # quick smoke test (2 scenarios)
30
+
merge-bench run --stress # 1000 files
31
+
merge-bench run --filesize # file size sweep (1KB/10KB/50KB)
32
+
merge-bench run --multi-collab # 5 collaborators
33
+
merge-bench run --conflict # guaranteed conflicts (same lines edited)
47
34
48
-
| Files | Conflict | 1-git (ms) | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
49
-
|-------|----------|-----------|-------------|----------------|------------|
50
-
| 1000 | low | 141 | 433 | 470 | 691 |
51
-
| 1000 | medium | 157 | 340 | 443 | 520 |
52
-
| 1000 | high | 97 | 356 | 334 | 579 |
35
+
# Remote PDS mode -- includes real network round-trips
36
+
merge-bench run --pds # default matrix against real PDS
37
+
merge-bench run --conflict --pds # conflict test against real PDS
53
38
54
-
### Remote (PDS)
39
+
# Group related runs into a named set
40
+
merge-bench run --set v1 # bench-results/v1/default-local/
41
+
merge-bench run --pds --set v1 # bench-results/v1/default-pds/
42
+
merge-bench run --stress --set v1 # bench-results/v1/stress-local/
43
+
```
55
44
56
-
All 3 scenarios failed with `413 Payload Too Large` — 1000-file sites exceed AT Protocol record size limits. Strategy 4 (pds-yrs) stores all file metadata in a single record, which hits the ~1MB limit. Strategies 1-3 (git-remote-pds) pack the full git repo into a single record, which also exceeds limits at this scale.
45
+
### Generate a repo manually
57
46
58
-
## File Size Sweep (50 files, 2 collaborators, medium conflict)
47
+
```bash
48
+
merge-bench gen --files 50 --avg-size 1000 --collaborators 3 --edits 10 --conflict-rate medium --output /tmp/test-repo
49
+
```
59
50
60
-
### Local
51
+
### Format existing results
61
52
62
-
| Avg Size | 1-git (ms) | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
63
-
|----------|-----------|-------------|----------------|------------|
64
-
| 1 KB | 51 | 104 | 241 | 56 |
65
-
| 10 KB | 191 | 270 | 219 | 128 |
66
-
| 50 KB | 117 | 568 | 649 | 455 |
53
+
```bash
54
+
merge-bench report --input bench-results/v1/default-local/results.json --format markdown
55
+
merge-bench report --input bench-results/v1/default-pds/results.json --format csv
56
+
```
67
57
68
-
### Remote (PDS)
58
+
## Output structure
69
59
70
-
| Avg Size | 1-git (ms) | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
71
-
|----------|-----------|-------------|----------------|------------|
72
-
| 1 KB | error (500) | - | - | - |
73
-
| 10 KB | 6800 | 7448 | 7156 | 11436 |
74
-
| 50 KB | error (413) | - | - | - |
60
+
Each run produces a folder with:
61
+
- `results.json` -- raw benchmark data
62
+
- `report.md` -- formatted markdown table
63
+
- `results.csv` -- CSV for spreadsheet import
75
64
76
-
The 1KB scenario hit a transient 500 error; the 50KB scenario exceeded payload limits on pds-yrs save.
65
+
Runs are grouped by set name under `bench-results/`:
77
66
78
-
## Multi-Collaborator (5 collaborators)
79
-
80
-
### Local
81
-
82
-
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
83
-
|-------|----------|-----------|-------------|-------------|----------------|------------|
84
-
| 50 | low | 173 | 0 | 410 | 527 | 106 |
85
-
| 200 | low | 311 | 0 | 734 | 746 | 253 |
86
-
| 50 | medium | 113 | 1 | 554 | 544 | 44 |
87
-
| 200 | medium | 237 | 0 | 618 | 778 | 214 |
88
-
| 50 | high | 39 | 1 | 775 | 824 | 46 |
89
-
| 200 | high | 89 | 1 | 1700 | 1201 | 166 |
90
-
91
-
### Remote (PDS)
92
-
93
-
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
94
-
|-------|----------|-----------|-------------|-------------|----------------|------------|
95
-
| 50 | low | 9841 | 0 | 9238 | 11476 | 8912 |
96
-
| 200 | low | 9851 | 1 | 11497 | 10358 | 31053 |
97
-
| 50 | medium | 9258 | 0 | 11243 | 10379 | 8788 |
98
-
| 200 | medium | 10572 | 1 | 10372 | 10110 | 29553 |
99
-
| 50 | high | 9594 | 2 | 10401 | 10122 | 8630 |
100
-
| 200 | high | 9759 | 4 | 11703 | 10867 | 29131 |
101
-
102
-
## Guaranteed Conflict (2 collaborators, same lines edited)
103
-
104
-
### Local
105
-
106
-
| Files | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
107
-
|-------|-----------|-------------|-------------|----------------|------------|
108
-
| 10 | 327 | 10 | 1456 | 614 | 6 |
109
-
| 50 | 156 | 20 | 961 | 1082 | 30 |
110
-
| 200 | 154 | 20 | 1048 | 1082 | 115 |
111
-
112
-
### Remote (PDS)
113
-
114
-
| Files | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
115
-
|-------|-----------|-------------|-------------|----------------|------------|
116
-
| 10 | 6634 | 10 | 6950 | 7166 | 5281 |
117
-
| 50 | 6900 | 20 | 7301 | 7296 | 10650 |
118
-
| 200 | 6741 | 20 | 8120 | 8299 | 28950 |
119
-
120
-
## Key Findings
121
-
122
-
### Local performance
123
-
124
-
- **Plain git is fastest** for pure merge speed (27-65ms for small/medium repos)
125
-
- **Yrs CRDT merge (strategy 4)** is competitive locally (2-82ms) since it's pure in-memory merge with no git overhead
126
-
- **git-yrs-merge strategies** (2, 3) add 2-10x overhead vs plain git due to CRDT driver invocations, but **eliminate all merge conflicts** — 0 conflicts across every scenario
127
-
- **Plain git fails with conflicts** in medium/high conflict scenarios; CRDT strategies always succeed
128
-
129
-
### Remote (PDS) performance
130
-
131
-
- **Network dominates** — all strategies take 6-10 seconds for small repos, regardless of local merge cost
132
-
- **Git strategies (1-3) are bottlenecked by push+clone** at ~6-7s baseline for 2 collaborators, ~9-11s for 5 collaborators
133
-
- **pds-yrs (strategy 4) scales worse with file count** — 3.5s at 10 files, 9s at 50, 28s at 200. Each file requires separate blob operations during save/merge
134
-
- **Git strategies scale better at high file counts** because git packs everything into a single push/fetch vs pds-yrs doing per-file blob ops
135
-
- At **50 files or fewer, pds-yrs is competitive or faster** than git strategies
136
-
- At **200 files, git strategies are 3-4x faster** than pds-yrs due to O(n) blob round-trips
67
+
```
68
+
bench-results/
69
+
v1/
70
+
report.md # combined report for this set
71
+
default-local/
72
+
default-pds/
73
+
stress-local/
74
+
conflict-local/
75
+
conflict-pds/
76
+
...
77
+
```
137
78
138
-
### PDS limits
79
+
## PDS mode setup
139
80
140
-
- AT Protocol record size limits (~1MB) prevent pds-yrs from handling 1000-file sites or 50KB average file sizes in a single record
141
-
- git-remote-pds also hits limits at 1000 files with large edits
142
-
- Pack blob format helps (bundles file data into single blob upload), but the record metadata itself grows with file count
81
+
Remote mode requires credentials in `testuser.toml`:
143
82
144
-
### Conflict resolution
83
+
```toml
84
+
pds = "https://your-pds.example.com"
85
+
handle = "user.your-pds.example.com"
86
+
password = "your-password"
87
+
did = "did:plc:..."
88
+
```
145
89
146
-
- Plain git: conflicts in 8 of 24 PDS scenarios (33%) — requires manual resolution
147
-
- All CRDT strategies (2, 3, 4): **zero conflicts across all 48 scenarios** — automatic resolution via Yrs CRDT merge
90
+
It also requires `git-remote-pds` and `pds-yrs` binaries to be built.
148
91
149
-
## Usage
92
+
## Reports
150
93
151
-
```bash
152
-
# Local mode (default)
153
-
merge-bench run # default matrix
154
-
merge-bench run --quick # fast smoke test
155
-
merge-bench run --stress # 1000 files
156
-
merge-bench run --filesize # 1KB/10KB/50KB sweep
157
-
merge-bench run --multi-collab # 5 collaborators
158
-
merge-bench run --conflict # guaranteed conflicts
94
+
See [bench-results/v1/report.md](bench-results/v1/report.md) for the first full benchmark comparing local vs PDS performance across all strategies.
159
95
160
-
# Remote PDS mode (reads testuser.toml)
161
-
merge-bench run --pds # default matrix against real PDS
162
-
merge-bench run --quick --pds # quick PDS test
163
-
merge-bench run --conflict --pds # conflict test against PDS
164
-
```
96
+
## Key findings
165
97
166
-
Results are written to `bench-results-local/` and `bench-results-pds/` respectively.
98
+
- CRDT strategies (2, 3, 4) **eliminate all merge conflicts** -- 0 conflicts across every scenario tested
99
+
- Plain git fails with conflicts in ~33% of scenarios
100
+
- Locally, plain git is fastest (25-386ms) but Yrs CRDT merge is competitive (3-348ms)
101
+
- Over PDS, network dominates: ~7s baseline for git strategies, ~3.7s for pds-yrs at small scale
102
+
- pds-yrs scales worse with file count (O(n) blob round-trips) -- at 200 files it takes ~29s vs ~7s for git strategies
103
+
- AT Protocol record size limits prevent 1000-file sites and 50KB avg file sizes
+21
bench-results/v1/multi-collab-pds/report.md
+21
bench-results/v1/multi-collab-pds/report.md
···
1
+
# Merge Strategy Benchmark Results
2
+
3
+
Generated: 2026-03-13T16:34:35.016073964+00:00
4
+
5
+
| Files | Collabs | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) | 2-ok | 3-ok | 4-ok |
6
+
|-------|---------|----------|-----------|-------------|-------------|----------------|------------|------|------|------|
7
+
| 50 | 5 | low | 8946 | 0 | 9576 | 10937 | 9323 | Y | Y | Y |
8
+
| 200 | 5 | low | 11684 | 0 | 15148 | 10021 | 32425 | Y | Y | Y |
9
+
| 50 | 5 | medium | 10874 | 0 | 10001 | 10617 | 8955 | Y | Y | Y |
10
+
| 200 | 5 | medium | 16152 | 1 | 10390 | 14696 | 30460 | Y | Y | Y |
11
+
| 50 | 5 | high | 12293 | 1 | 10480 | 9973 | 8799 | Y | Y | Y |
12
+
| 200 | 5 | high | 10207 | 1 | 10787 | 11771 | 31184 | Y | Y | Y |
13
+
14
+
### Legend
15
+
16
+
- **1-git**: Plain git 3-way merge
17
+
- **2-diff**: git-yrs-merge in diff-only mode
18
+
- **3-sidecar**: git-yrs-merge with .yrs/ sidecars
19
+
- **4-yrs**: Yrs CRDT merge (simulated, no network)
20
+
- **ok**: merge completed successfully (Y/N)
21
+
- **conflicts**: number of files with conflict markers (strategy 1 only)
+25
bench-results/v1/multi-collab-pds/results.csv
+25
bench-results/v1/multi-collab-pds/results.csv
···
1
+
files,conflict_rate,collaborators,edits,strategy,time_ms,conflicts,success
2
+
50,low,5,10,1-plain-git,8946,0,true
3
+
50,low,5,10,2-yrs-diff,9576,0,true
4
+
50,low,5,10,3-yrs-sidecar,10937,0,true
5
+
50,low,5,10,4-yrs-on-pds,9323,0,true
6
+
200,low,5,20,1-plain-git,11684,0,true
7
+
200,low,5,20,2-yrs-diff,15148,0,true
8
+
200,low,5,20,3-yrs-sidecar,10021,0,true
9
+
200,low,5,20,4-yrs-on-pds,32425,0,true
10
+
50,medium,5,10,1-plain-git,10874,0,true
11
+
50,medium,5,10,2-yrs-diff,10001,0,true
12
+
50,medium,5,10,3-yrs-sidecar,10617,0,true
13
+
50,medium,5,10,4-yrs-on-pds,8955,0,true
14
+
200,medium,5,20,1-plain-git,16152,1,false
15
+
200,medium,5,20,2-yrs-diff,10390,0,true
16
+
200,medium,5,20,3-yrs-sidecar,14696,0,true
17
+
200,medium,5,20,4-yrs-on-pds,30460,0,true
18
+
50,high,5,10,1-plain-git,12293,1,false
19
+
50,high,5,10,2-yrs-diff,10480,0,true
20
+
50,high,5,10,3-yrs-sidecar,9973,0,true
21
+
50,high,5,10,4-yrs-on-pds,8799,0,true
22
+
200,high,5,20,1-plain-git,10207,1,false
23
+
200,high,5,20,2-yrs-diff,10787,0,true
24
+
200,high,5,20,3-yrs-sidecar,11771,0,true
25
+
200,high,5,20,4-yrs-on-pds,31184,0,true
+248
bench-results/v1/multi-collab-pds/results.json
+248
bench-results/v1/multi-collab-pds/results.json
···
1
+
[
2
+
{
3
+
"file_count": 50,
4
+
"avg_file_size": 1000,
5
+
"collaborators": 5,
6
+
"edits_per_branch": 10,
7
+
"conflict_rate": "low",
8
+
"results": [
9
+
{
10
+
"strategy": "1-plain-git",
11
+
"merge_time_ms": 8946,
12
+
"conflicts_reported": 0,
13
+
"files_processed": 50,
14
+
"success": true,
15
+
"error": null
16
+
},
17
+
{
18
+
"strategy": "2-yrs-diff",
19
+
"merge_time_ms": 9576,
20
+
"conflicts_reported": 0,
21
+
"files_processed": 183,
22
+
"success": true,
23
+
"error": null
24
+
},
25
+
{
26
+
"strategy": "3-yrs-sidecar",
27
+
"merge_time_ms": 10937,
28
+
"conflicts_reported": 0,
29
+
"files_processed": 185,
30
+
"success": true,
31
+
"error": null
32
+
},
33
+
{
34
+
"strategy": "4-yrs-on-pds",
35
+
"merge_time_ms": 9323,
36
+
"conflicts_reported": 0,
37
+
"files_processed": 50,
38
+
"success": true,
39
+
"error": null
40
+
}
41
+
]
42
+
},
43
+
{
44
+
"file_count": 200,
45
+
"avg_file_size": 1000,
46
+
"collaborators": 5,
47
+
"edits_per_branch": 20,
48
+
"conflict_rate": "low",
49
+
"results": [
50
+
{
51
+
"strategy": "1-plain-git",
52
+
"merge_time_ms": 11684,
53
+
"conflicts_reported": 0,
54
+
"files_processed": 200,
55
+
"success": true,
56
+
"error": null
57
+
},
58
+
{
59
+
"strategy": "2-yrs-diff",
60
+
"merge_time_ms": 15148,
61
+
"conflicts_reported": 0,
62
+
"files_processed": 688,
63
+
"success": true,
64
+
"error": null
65
+
},
66
+
{
67
+
"strategy": "3-yrs-sidecar",
68
+
"merge_time_ms": 10021,
69
+
"conflicts_reported": 0,
70
+
"files_processed": 683,
71
+
"success": true,
72
+
"error": null
73
+
},
74
+
{
75
+
"strategy": "4-yrs-on-pds",
76
+
"merge_time_ms": 32425,
77
+
"conflicts_reported": 0,
78
+
"files_processed": 200,
79
+
"success": true,
80
+
"error": null
81
+
}
82
+
]
83
+
},
84
+
{
85
+
"file_count": 50,
86
+
"avg_file_size": 1000,
87
+
"collaborators": 5,
88
+
"edits_per_branch": 10,
89
+
"conflict_rate": "medium",
90
+
"results": [
91
+
{
92
+
"strategy": "1-plain-git",
93
+
"merge_time_ms": 10874,
94
+
"conflicts_reported": 0,
95
+
"files_processed": 50,
96
+
"success": true,
97
+
"error": null
98
+
},
99
+
{
100
+
"strategy": "2-yrs-diff",
101
+
"merge_time_ms": 10001,
102
+
"conflicts_reported": 0,
103
+
"files_processed": 184,
104
+
"success": true,
105
+
"error": null
106
+
},
107
+
{
108
+
"strategy": "3-yrs-sidecar",
109
+
"merge_time_ms": 10617,
110
+
"conflicts_reported": 0,
111
+
"files_processed": 190,
112
+
"success": true,
113
+
"error": null
114
+
},
115
+
{
116
+
"strategy": "4-yrs-on-pds",
117
+
"merge_time_ms": 8955,
118
+
"conflicts_reported": 0,
119
+
"files_processed": 50,
120
+
"success": true,
121
+
"error": null
122
+
}
123
+
]
124
+
},
125
+
{
126
+
"file_count": 200,
127
+
"avg_file_size": 1000,
128
+
"collaborators": 5,
129
+
"edits_per_branch": 20,
130
+
"conflict_rate": "medium",
131
+
"results": [
132
+
{
133
+
"strategy": "1-plain-git",
134
+
"merge_time_ms": 16152,
135
+
"conflicts_reported": 1,
136
+
"files_processed": 202,
137
+
"success": false,
138
+
"error": ""
139
+
},
140
+
{
141
+
"strategy": "2-yrs-diff",
142
+
"merge_time_ms": 10390,
143
+
"conflicts_reported": 0,
144
+
"files_processed": 688,
145
+
"success": true,
146
+
"error": null
147
+
},
148
+
{
149
+
"strategy": "3-yrs-sidecar",
150
+
"merge_time_ms": 14696,
151
+
"conflicts_reported": 0,
152
+
"files_processed": 689,
153
+
"success": true,
154
+
"error": null
155
+
},
156
+
{
157
+
"strategy": "4-yrs-on-pds",
158
+
"merge_time_ms": 30460,
159
+
"conflicts_reported": 0,
160
+
"files_processed": 200,
161
+
"success": true,
162
+
"error": null
163
+
}
164
+
]
165
+
},
166
+
{
167
+
"file_count": 50,
168
+
"avg_file_size": 1000,
169
+
"collaborators": 5,
170
+
"edits_per_branch": 10,
171
+
"conflict_rate": "high",
172
+
"results": [
173
+
{
174
+
"strategy": "1-plain-git",
175
+
"merge_time_ms": 12293,
176
+
"conflicts_reported": 1,
177
+
"files_processed": 52,
178
+
"success": false,
179
+
"error": ""
180
+
},
181
+
{
182
+
"strategy": "2-yrs-diff",
183
+
"merge_time_ms": 10480,
184
+
"conflicts_reported": 0,
185
+
"files_processed": 191,
186
+
"success": true,
187
+
"error": null
188
+
},
189
+
{
190
+
"strategy": "3-yrs-sidecar",
191
+
"merge_time_ms": 9973,
192
+
"conflicts_reported": 0,
193
+
"files_processed": 187,
194
+
"success": true,
195
+
"error": null
196
+
},
197
+
{
198
+
"strategy": "4-yrs-on-pds",
199
+
"merge_time_ms": 8799,
200
+
"conflicts_reported": 0,
201
+
"files_processed": 50,
202
+
"success": true,
203
+
"error": null
204
+
}
205
+
]
206
+
},
207
+
{
208
+
"file_count": 200,
209
+
"avg_file_size": 1000,
210
+
"collaborators": 5,
211
+
"edits_per_branch": 20,
212
+
"conflict_rate": "high",
213
+
"results": [
214
+
{
215
+
"strategy": "1-plain-git",
216
+
"merge_time_ms": 10207,
217
+
"conflicts_reported": 1,
218
+
"files_processed": 202,
219
+
"success": false,
220
+
"error": ""
221
+
},
222
+
{
223
+
"strategy": "2-yrs-diff",
224
+
"merge_time_ms": 10787,
225
+
"conflicts_reported": 0,
226
+
"files_processed": 686,
227
+
"success": true,
228
+
"error": null
229
+
},
230
+
{
231
+
"strategy": "3-yrs-sidecar",
232
+
"merge_time_ms": 11771,
233
+
"conflicts_reported": 0,
234
+
"files_processed": 694,
235
+
"success": true,
236
+
"error": null
237
+
},
238
+
{
239
+
"strategy": "4-yrs-on-pds",
240
+
"merge_time_ms": 31184,
241
+
"conflicts_reported": 0,
242
+
"files_processed": 200,
243
+
"success": true,
244
+
"error": null
245
+
}
246
+
]
247
+
}
248
+
]
+147
bench-results/v1/report.md
+147
bench-results/v1/report.md
···
1
+
# Merge Strategy Benchmark Report (v1)
2
+
3
+
**Date:** 2026-03-13
4
+
**PDS:** bluesky-pds.t1cc.commoninternet.net (v0.4.208)
5
+
**Modes:** Local (no network) and Remote (real PDS round-trips)
6
+
7
+
## Strategies
8
+
9
+
1. **plain-git** -- Standard `git merge` 3-way merge
10
+
2. **yrs-diff** -- git merge with git-yrs-merge CRDT driver (diff-only mode)
11
+
3. **yrs-sidecar** -- git merge with git-yrs-merge CRDT driver + .yrs/ sidecar files
12
+
4. **yrs-on-pds** -- Pure Yrs CRDT merge (local: in-memory from .pds-sim/; PDS: pds-yrs save/merge/load)
13
+
14
+
## Default Matrix (10/50/200 files, 2 collaborators)
15
+
16
+
### Local
17
+
18
+
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
19
+
|-------|----------|-----------|-------------|-------------|----------------|------------|
20
+
| 10 | low | 25 | 0 | 45 | 69 | 3 |
21
+
| 10 | medium | 33 | 0 | 127 | 121 | 5 |
22
+
| 10 | high | 67 | 1 | 118 | 260 | 6 |
23
+
| 50 | low | 118 | 0 | 187 | 245 | 55 |
24
+
| 50 | medium | 61 | 0 | 298 | 576 | 34 |
25
+
| 50 | high | 198 | 0 | 365 | 721 | 52 |
26
+
| 200 | low | 386 | 0 | 441 | 292 | 119 |
27
+
| 200 | medium | 160 | 0 | 164 | 325 | 102 |
28
+
| 200 | high | 139 | 0 | 212 | 363 | 111 |
29
+
30
+
### Remote (PDS)
31
+
32
+
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
33
+
|-------|----------|-----------|-------------|-------------|----------------|------------|
34
+
| 10 | low | 6676 | 0 | 7649 | 8303 | 3793 |
35
+
| 10 | medium | 6728 | 1 | 7005 | 6643 | 4066 |
36
+
| 10 | high | 6666 | 1 | 7363 | 6482 | 3717 |
37
+
| 50 | low | 7228 | 0 | 7006 | 6901 | 9042 |
38
+
| 50 | medium | 6618 | 0 | 6845 | 6965 | 8737 |
39
+
| 50 | high | 6984 | 1 | 7957 | 6972 | 9386 |
40
+
| 200 | low | 6966 | 0 | 7064 | 7567 | 29411 |
41
+
| 200 | medium | 11105 | 0 | 7335 | 7056 | 27647 |
42
+
| 200 | high | 7294 | 1 | 7792 | 7370 | 29289 |
43
+
44
+
## Stress Test (1000 files, 2 collaborators)
45
+
46
+
### Local
47
+
48
+
| Files | Conflict | 1-git (ms) | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
49
+
|-------|----------|-----------|-------------|----------------|------------|
50
+
| 1000 | low | 161 | 458 | 1071 | 645 |
51
+
| 1000 | medium | 165 | 410 | 910 | 571 |
52
+
| 1000 | high | 91 | 341 | 316 | 464 |
53
+
54
+
### Remote (PDS)
55
+
56
+
All 3 scenarios failed with `413 Payload Too Large` -- 1000-file sites exceed AT Protocol record size limits.
57
+
58
+
## File Size Sweep (50 files, 2 collaborators, medium conflict)
59
+
60
+
### Local
61
+
62
+
| Avg Size | 1-git (ms) | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
63
+
|----------|-----------|-------------|----------------|------------|
64
+
| 1 KB | 61 | 78 | 291 | 50 |
65
+
| 10 KB | 106 | 360 | 437 | 167 |
66
+
| 50 KB | 206 | 751 | 790 | 431 |
67
+
68
+
### Remote (PDS)
69
+
70
+
| Avg Size | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
71
+
|----------|-----------|-------------|-------------|----------------|------------|
72
+
| 1 KB | 6514 | 1 | 6870 | 6937 | 9475 |
73
+
| 10 KB | 7582 | 0 | 7528 | 7309 | 9252 |
74
+
| 50 KB | error (413) | - | - | - | - |
75
+
76
+
The 50KB scenario exceeded payload limits on pds-yrs save.
77
+
78
+
## Multi-Collaborator (5 collaborators)
79
+
80
+
### Local
81
+
82
+
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
83
+
|-------|----------|-----------|-------------|-------------|----------------|------------|
84
+
| 50 | low | 277 | 0 | 360 | 921 | 60 |
85
+
| 200 | low | 719 | 0 | 957 | 1337 | 348 |
86
+
| 50 | medium | 242 | 0 | 892 | 894 | 138 |
87
+
| 200 | medium | 349 | 0 | 854 | 601 | 236 |
88
+
| 50 | high | 72 | 1 | 628 | 843 | 39 |
89
+
| 200 | high | 154 | 1 | 1069 | 1081 | 153 |
90
+
91
+
### Remote (PDS)
92
+
93
+
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
94
+
|-------|----------|-----------|-------------|-------------|----------------|------------|
95
+
| 50 | low | 8946 | 0 | 9576 | 10937 | 9323 |
96
+
| 200 | low | 11684 | 0 | 15148 | 10021 | 32425 |
97
+
| 50 | medium | 10874 | 0 | 10001 | 10617 | 8955 |
98
+
| 200 | medium | 16152 | 1 | 10390 | 14696 | 30460 |
99
+
| 50 | high | 12293 | 1 | 10480 | 9973 | 8799 |
100
+
| 200 | high | 10207 | 1 | 10787 | 11771 | 31184 |
101
+
102
+
## Guaranteed Conflict (2 collaborators, same lines edited)
103
+
104
+
### Local
105
+
106
+
| Files | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
107
+
|-------|-----------|-------------|-------------|----------------|------------|
108
+
| 10 | 85 | 10 | 658 | 1009 | 7 |
109
+
| 50 | 307 | 20 | 3438 | 1996 | 106 |
110
+
| 200 | 802 | 20 | 1157 | 1485 | 204 |
111
+
112
+
### Remote (PDS)
113
+
114
+
| Files | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
115
+
|-------|-----------|-------------|-------------|----------------|------------|
116
+
| 10 | 6363 | 10 | 6902 | 7004 | 4335 |
117
+
| 50 | 6723 | 20 | 7853 | 10435 | 9840 |
118
+
| 200 | 6782 | 20 | 7308 | 7997 | 30904 |
119
+
120
+
## Key Findings
121
+
122
+
### Local performance
123
+
124
+
- **Plain git is fastest** for pure merge speed (25-386ms)
125
+
- **Yrs CRDT merge (strategy 4)** is competitive locally (3-348ms) -- pure in-memory merge with no git overhead
126
+
- **git-yrs-merge strategies** (2, 3) add 2-10x overhead vs plain git due to CRDT driver invocations, but **eliminate all merge conflicts**
127
+
- **Plain git fails with conflicts** in medium/high conflict scenarios; CRDT strategies always succeed
128
+
129
+
### Remote (PDS) performance
130
+
131
+
- **Network dominates** -- all strategies take 6-10s for small repos, regardless of local merge cost
132
+
- **Git strategies (1-3) bottlenecked by push+clone** at ~7s baseline for 2 collaborators, ~10-16s for 5 collaborators
133
+
- **pds-yrs (strategy 4) scales worse with file count** -- 3.7s at 10 files, 9s at 50, 29s at 200. Each file requires separate blob operations during save/merge
134
+
- **Git strategies scale better at high file counts** because git packs everything into a single push/fetch
135
+
- At **50 files or fewer, pds-yrs is competitive or faster** than git strategies
136
+
- At **200 files, git strategies are 3-4x faster** than pds-yrs due to O(n) blob round-trips
137
+
138
+
### PDS limits
139
+
140
+
- AT Protocol record size limits (~1MB) prevent pds-yrs from handling 1000-file sites or 50KB average file sizes
141
+
- git-remote-pds also hits limits at 1000 files
142
+
- Pack blob format helps but record metadata still grows with file count
143
+
144
+
### Conflict resolution
145
+
146
+
- Plain git: conflicts in ~33% of scenarios -- requires manual resolution
147
+
- All CRDT strategies (2, 3, 4): **zero conflicts across all scenarios** -- automatic resolution via Yrs CRDT merge
-166
reports/benchmark-report-2026-03-13.md
-166
reports/benchmark-report-2026-03-13.md
···
1
-
# Merge Strategy Benchmark Report
2
-
3
-
**Date:** 2026-03-13
4
-
**PDS:** bluesky-pds.t1cc.commoninternet.net (v0.4.208)
5
-
**Modes:** Local (no network) and Remote (real PDS round-trips)
6
-
7
-
## Strategies
8
-
9
-
1. **plain-git** — Standard `git merge` 3-way merge
10
-
2. **yrs-diff** — git merge with git-yrs-merge CRDT driver (diff-only mode)
11
-
3. **yrs-sidecar** — git merge with git-yrs-merge CRDT driver + .yrs/ sidecar files
12
-
4. **yrs-on-pds** — Pure Yrs CRDT merge (local: in-memory from .pds-sim/; PDS: pds-yrs save/merge/load)
13
-
14
-
## Default Matrix (10/50/200 files, 2 collaborators)
15
-
16
-
### Local
17
-
18
-
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
19
-
|-------|----------|-----------|-------------|-------------|----------------|------------|
20
-
| 10 | low | 34 | 0 | 62 | 108 | 5 |
21
-
| 10 | medium | 32 | 1 | 57 | 106 | 3 |
22
-
| 10 | high | 30 | 0 | 85 | 122 | 3 |
23
-
| 50 | low | 39 | 0 | 60 | 91 | 23 |
24
-
| 50 | medium | 40 | 1 | 61 | 114 | 14 |
25
-
| 50 | high | 42 | 1 | 109 | 147 | 17 |
26
-
| 200 | low | 53 | 0 | 81 | 125 | 82 |
27
-
| 200 | medium | 62 | 1 | 117 | 208 | 65 |
28
-
| 200 | high | 65 | 0 | 264 | 472 | 68 |
29
-
30
-
### Remote (PDS)
31
-
32
-
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
33
-
|-------|----------|-----------|-------------|-------------|----------------|------------|
34
-
| 10 | low | 7094 | 0 | 6729 | 6702 | 3783 |
35
-
| 10 | medium | 6823 | 0 | 6776 | 6738 | 3924 |
36
-
| 10 | high | 6877 | 1 | 6764 | 8074 | 4471 |
37
-
| 50 | low | 7033 | 0 | 7089 | 7424 | 9316 |
38
-
| 50 | medium | 6999 | 0 | 7423 | 7973 | 9224 |
39
-
| 50 | high | 6882 | 0 | 7049 | 6963 | 9378 |
40
-
| 200 | low | 7000 | 0 | 7406 | 7388 | 28381 |
41
-
| 200 | medium | 6826 | 0 | 9745 | 7370 | 28995 |
42
-
| 200 | high | 7503 | 1 | 7327 | 7672 | 27605 |
43
-
44
-
## Stress Test (1000 files, 2 collaborators)
45
-
46
-
### Local
47
-
48
-
| Files | Conflict | 1-git (ms) | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
49
-
|-------|----------|-----------|-------------|----------------|------------|
50
-
| 1000 | low | 141 | 433 | 470 | 691 |
51
-
| 1000 | medium | 157 | 340 | 443 | 520 |
52
-
| 1000 | high | 97 | 356 | 334 | 579 |
53
-
54
-
### Remote (PDS)
55
-
56
-
All 3 scenarios failed with `413 Payload Too Large` — 1000-file sites exceed AT Protocol record size limits. Strategy 4 (pds-yrs) stores all file metadata in a single record, which hits the ~1MB limit. Strategies 1-3 (git-remote-pds) pack the full git repo into a single record, which also exceeds limits at this scale.
57
-
58
-
## File Size Sweep (50 files, 2 collaborators, medium conflict)
59
-
60
-
### Local
61
-
62
-
| Avg Size | 1-git (ms) | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
63
-
|----------|-----------|-------------|----------------|------------|
64
-
| 1 KB | 51 | 104 | 241 | 56 |
65
-
| 10 KB | 191 | 270 | 219 | 128 |
66
-
| 50 KB | 117 | 568 | 649 | 455 |
67
-
68
-
### Remote (PDS)
69
-
70
-
| Avg Size | 1-git (ms) | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
71
-
|----------|-----------|-------------|----------------|------------|
72
-
| 1 KB | error (500) | - | - | - |
73
-
| 10 KB | 6800 | 7448 | 7156 | 11436 |
74
-
| 50 KB | error (413) | - | - | - |
75
-
76
-
The 1KB scenario hit a transient 500 error; the 50KB scenario exceeded payload limits on pds-yrs save.
77
-
78
-
## Multi-Collaborator (5 collaborators)
79
-
80
-
### Local
81
-
82
-
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
83
-
|-------|----------|-----------|-------------|-------------|----------------|------------|
84
-
| 50 | low | 173 | 0 | 410 | 527 | 106 |
85
-
| 200 | low | 311 | 0 | 734 | 746 | 253 |
86
-
| 50 | medium | 113 | 1 | 554 | 544 | 44 |
87
-
| 200 | medium | 237 | 0 | 618 | 778 | 214 |
88
-
| 50 | high | 39 | 1 | 775 | 824 | 46 |
89
-
| 200 | high | 89 | 1 | 1700 | 1201 | 166 |
90
-
91
-
### Remote (PDS)
92
-
93
-
| Files | Conflict | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
94
-
|-------|----------|-----------|-------------|-------------|----------------|------------|
95
-
| 50 | low | 9841 | 0 | 9238 | 11476 | 8912 |
96
-
| 200 | low | 9851 | 1 | 11497 | 10358 | 31053 |
97
-
| 50 | medium | 9258 | 0 | 11243 | 10379 | 8788 |
98
-
| 200 | medium | 10572 | 1 | 10372 | 10110 | 29553 |
99
-
| 50 | high | 9594 | 2 | 10401 | 10122 | 8630 |
100
-
| 200 | high | 9759 | 4 | 11703 | 10867 | 29131 |
101
-
102
-
## Guaranteed Conflict (2 collaborators, same lines edited)
103
-
104
-
### Local
105
-
106
-
| Files | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
107
-
|-------|-----------|-------------|-------------|----------------|------------|
108
-
| 10 | 327 | 10 | 1456 | 614 | 6 |
109
-
| 50 | 156 | 20 | 961 | 1082 | 30 |
110
-
| 200 | 154 | 20 | 1048 | 1082 | 115 |
111
-
112
-
### Remote (PDS)
113
-
114
-
| Files | 1-git (ms) | 1-conflicts | 2-diff (ms) | 3-sidecar (ms) | 4-yrs (ms) |
115
-
|-------|-----------|-------------|-------------|----------------|------------|
116
-
| 10 | 6634 | 10 | 6950 | 7166 | 5281 |
117
-
| 50 | 6900 | 20 | 7301 | 7296 | 10650 |
118
-
| 200 | 6741 | 20 | 8120 | 8299 | 28950 |
119
-
120
-
## Key Findings
121
-
122
-
### Local performance
123
-
124
-
- **Plain git is fastest** for pure merge speed (27-65ms for small/medium repos)
125
-
- **Yrs CRDT merge (strategy 4)** is competitive locally (2-82ms) since it's pure in-memory merge with no git overhead
126
-
- **git-yrs-merge strategies** (2, 3) add 2-10x overhead vs plain git due to CRDT driver invocations, but **eliminate all merge conflicts** — 0 conflicts across every scenario
127
-
- **Plain git fails with conflicts** in medium/high conflict scenarios; CRDT strategies always succeed
128
-
129
-
### Remote (PDS) performance
130
-
131
-
- **Network dominates** — all strategies take 6-10 seconds for small repos, regardless of local merge cost
132
-
- **Git strategies (1-3) are bottlenecked by push+clone** at ~6-7s baseline for 2 collaborators, ~9-11s for 5 collaborators
133
-
- **pds-yrs (strategy 4) scales worse with file count** — 3.5s at 10 files, 9s at 50, 28s at 200. Each file requires separate blob operations during save/merge
134
-
- **Git strategies scale better at high file counts** because git packs everything into a single push/fetch vs pds-yrs doing per-file blob ops
135
-
- At **50 files or fewer, pds-yrs is competitive or faster** than git strategies
136
-
- At **200 files, git strategies are 3-4x faster** than pds-yrs due to O(n) blob round-trips
137
-
138
-
### PDS limits
139
-
140
-
- AT Protocol record size limits (~1MB) prevent pds-yrs from handling 1000-file sites or 50KB average file sizes in a single record
141
-
- git-remote-pds also hits limits at 1000 files with large edits
142
-
- Pack blob format helps (bundles file data into single blob upload), but the record metadata itself grows with file count
143
-
144
-
### Conflict resolution
145
-
146
-
- Plain git: conflicts in 8 of 24 PDS scenarios (33%) — requires manual resolution
147
-
- All CRDT strategies (2, 3, 4): **zero conflicts across all 48 scenarios** — automatic resolution via Yrs CRDT merge
148
-
149
-
## Usage
150
-
151
-
```bash
152
-
# Local mode (default)
153
-
merge-bench run # default matrix
154
-
merge-bench run --quick # fast smoke test
155
-
merge-bench run --stress # 1000 files
156
-
merge-bench run --filesize # 1KB/10KB/50KB sweep
157
-
merge-bench run --multi-collab # 5 collaborators
158
-
merge-bench run --conflict # guaranteed conflicts
159
-
160
-
# Remote PDS mode (reads testuser.toml)
161
-
merge-bench run --pds # default matrix against real PDS
162
-
merge-bench run --quick --pds # quick PDS test
163
-
merge-bench run --conflict --pds # conflict test against PDS
164
-
```
165
-
166
-
Results are written to `bench-results-local/` and `bench-results-pds/` respectively.