My aggregated monorepo of OCaml code, automaintained
Merge commit '22a05d38f11f0a2a845e663c376d26af57ccbaed' as 'braid'
+2222
+670
braid/README.md
+670
braid/README.md
···
1
+
# Braid
2
+
3
+
Build status tracker for opam overlay repositories. Braid runs [day10](https://github.com/mtelvers/day10) health checks across git commits and provides a queryable manifest of results.
4
+
5
+
## Overview
6
+
7
+
Braid solves the problem of tracking package build status across multiple commits in an opam overlay repository. It:
8
+
9
+
1. **Runs day10 health checks** across a configurable number of commits
10
+
2. **Generates a manifest.json** containing all results, build logs, and dependency graphs
11
+
3. **Provides query commands** to investigate failures, track package history, and diagnose problems
12
+
13
+
The manifest format is designed for both human inspection and AI agent consumption.
14
+
15
+
## Installation
16
+
17
+
```bash
18
+
# Clone the repository
19
+
git clone https://github.com/mtelvers/braid.git
20
+
cd braid
21
+
22
+
# Create an opam switch and install dependencies
23
+
opam switch create . 5.4.0 --deps-only
24
+
eval $(opam env)
25
+
opam install . --deps-only
26
+
27
+
# Build
28
+
dune build
29
+
30
+
# Install (optional)
31
+
dune install
32
+
```
33
+
34
+
## Usage
35
+
36
+
### Running Health Checks
37
+
38
+
```bash
39
+
braid run <REPO_PATH> [OPTIONS]
40
+
```
41
+
42
+
**Arguments:**
43
+
- `REPO_PATH` - Path to the overlay opam repository
44
+
45
+
**Options:**
46
+
- `-n, --num-commits N` - Number of commits to process (default: 10)
47
+
- `-j, --jobs N` - Number of parallel jobs for solving (default: 40)
48
+
- `-o, --output PATH` - Output directory for results (default: results)
49
+
- `--opam-repo PATH` - Path to the main opam repository (default: /home/mtelvers/opam-repository)
50
+
- `--cache-dir PATH` - Cache directory for day10 (default: /var/cache/day10)
51
+
- `--os OS` - Operating system (default: linux)
52
+
- `--os-family FAMILY` - OS family (default: debian)
53
+
- `--os-distribution DIST` - OS distribution (default: debian)
54
+
- `--os-version VERSION` - OS version (default: 13)
55
+
- `-v, --verbose` - Increase verbosity
56
+
57
+
**Example:**
58
+
```bash
59
+
# Run on the last 57 commits of an overlay repository
60
+
braid run /home/mtelvers/aoah-opam-repo -n 57 -o results -v
61
+
```
62
+
63
+
### Merge Testing
64
+
65
+
Test the cumulative effect of merging multiple overlay repositories without tracking commit history.
66
+
67
+
```bash
68
+
braid merge-test <REPOS>... [OPTIONS]
69
+
```
70
+
71
+
**Arguments:**
72
+
- `REPOS` - One or more overlay repository paths (in priority order, first = highest priority)
73
+
74
+
**Options:**
75
+
- `-j, --jobs N` - Number of parallel jobs for solving (default: 40)
76
+
- `-o, --output PATH` - Output directory for results (default: results)
77
+
- `--opam-repo PATH` - Path to the main opam repository (default: /home/mtelvers/opam-repository)
78
+
- `--cache-dir PATH` - Cache directory for day10 (default: /var/cache/day10)
79
+
- `--dry-run` - Only solve dependencies, don't actually build
80
+
- `--os OS` - Operating system (default: linux)
81
+
- `--os-family FAMILY` - OS family (default: debian)
82
+
- `--os-distribution DIST` - OS distribution (default: debian)
83
+
- `--os-version VERSION` - OS version (default: 13)
84
+
- `-v, --verbose` - Increase verbosity
85
+
86
+
**Examples:**
87
+
```bash
88
+
# Test a single overlay repository
89
+
braid merge-test /home/mtelvers/my-overlay -o results
90
+
91
+
# Test what happens if 'experimental' is merged into 'stable'
92
+
braid merge-test /path/to/experimental /path/to/stable -o merge-results
93
+
94
+
# Stack multiple overlays (first has highest priority)
95
+
braid merge-test /path/to/repo1 /path/to/repo2 /path/to/repo3
96
+
97
+
# Quick dependency check without building
98
+
braid merge-test /path/to/overlay --dry-run
99
+
```
100
+
101
+
**How it works:**
102
+
103
+
1. Lists packages from all overlay repositories (not from opam-repository)
104
+
2. **Stage 1:** Runs `day10 health-check --dry-run --fork N` for fast parallel dependency solving
105
+
3. **Stage 2:** For packages that returned "solution" (solvable but not built), runs `day10 health-check` without `--dry-run` to actually build them
106
+
107
+
With `--dry-run`, only stage 1 runs, showing which packages are solvable without building them.
108
+
109
+
**Querying merge-test results:**
110
+
111
+
Query commands work with merge-test manifests using `merge-q` as the commit identifier:
112
+
113
+
```bash
114
+
# Show history for a package
115
+
braid history smtpd.dev -m merge-results/manifest.json
116
+
# Output: First seen: merge-q, Latest status: failure
117
+
118
+
# Show build log
119
+
braid log merge-q smtpd.dev -m merge-results/manifest.json
120
+
121
+
# Show dependencies
122
+
braid deps merge-q smtpd.dev -m merge-results/manifest.json
123
+
```
124
+
125
+
### Query Commands
126
+
127
+
All query commands read from a manifest file (default: `manifest.json`). Use `-m PATH` to specify a different manifest.
128
+
129
+
#### summary
130
+
131
+
Show overview statistics.
132
+
133
+
```bash
134
+
$ braid summary -m results/manifest.json
135
+
Repository: /home/mtelvers/aoah-opam-repo
136
+
Generated: 2026-01-19T19:45:16Z
137
+
OS: debian-13
138
+
Commits: 57
139
+
Packages: 55
140
+
141
+
Latest commit status:
142
+
Success: 12
143
+
Failure: 11
144
+
Dependency failed: 10
145
+
No solution: 22
146
+
Solution (buildable): 0
147
+
Error: 0
148
+
```
149
+
150
+
#### failures
151
+
152
+
List packages with status 'failure' in the latest commit.
153
+
154
+
```bash
155
+
$ braid failures -m results/manifest.json
156
+
Failures in commit 3289824:
157
+
atp.dev
158
+
bytesrw-eio.dev
159
+
claude.dev
160
+
frontmatter.dev
161
+
hermest.dev
162
+
html5rw.dev
163
+
init.dev
164
+
langdetect.dev
165
+
monopam.dev
166
+
owntracks.dev
167
+
srcsetter-cmd.dev
168
+
```
169
+
170
+
#### log
171
+
172
+
Show the build log for a specific package at a specific commit.
173
+
174
+
```bash
175
+
$ braid log 3289824 bytesrw-eio.dev -m results/manifest.json
176
+
Processing: [default: loading data]
177
+
[bytesrw-eio.dev: git]
178
+
...
179
+
-> retrieved bytesrw-eio.dev (git+https://tangled.org/@anil.recoil.org/ocaml-bytesrw-eio.git#main)
180
+
[bytesrw-eio: dune subst]
181
+
+ /home/opam/.opam/default/bin/dune "subst" (CWD=/home/opam/.opam/default/.opam-switch/build/bytesrw-eio.dev)
182
+
- File "dune-project", line 25, characters 16-33:
183
+
- 25 | (documentation (depends bytesrw)))
184
+
- ^^^^^^^^^^^^^^^^^
185
+
- Error: Atom or quoted string expected
186
+
[ERROR] The compilation of bytesrw-eio.dev failed at "dune subst".
187
+
build failed...
188
+
```
189
+
190
+
#### history
191
+
192
+
Show the status of a package across all commits.
193
+
194
+
```bash
195
+
$ braid history cbort.dev -m results/manifest.json
196
+
Package: cbort.dev
197
+
First seen: 82661d5
198
+
Latest status: success
199
+
History:
200
+
3289824: success
201
+
b92aa39: success
202
+
2345324: success
203
+
...
204
+
```
205
+
206
+
#### first-failure
207
+
208
+
Find when a package first started failing (the commit where it transitioned from success to failure).
209
+
210
+
```bash
211
+
$ braid first-failure atp.dev -m results/manifest.json
212
+
First failure: 160dd2e (Add owntracks and owntracks-cli dev packages)
213
+
```
214
+
215
+
#### deps
216
+
217
+
Show the dependency graph for a package (in DOT format).
218
+
219
+
```bash
220
+
$ braid deps 3289824 cbort.dev -m results/manifest.json
221
+
digraph opam {
222
+
"bytesrw.0.3.0" -> {"conf-pkg-config.4" "ocaml.5.3.0" "ocamlbuild.0.16.1" "ocamlfind.1.9.8" "topkg.1.1.1"}
223
+
"cbort.dev" -> {"bytesrw.0.3.0" "dune.3.21.0" "ocaml.5.3.0" "zarith.1.14"}
224
+
...
225
+
}
226
+
```
227
+
228
+
#### result
229
+
230
+
Get the full JSON result for a package at a commit.
231
+
232
+
```bash
233
+
$ braid result 3289824 cbort.dev -m results/manifest.json
234
+
{
235
+
"name": "cbort.dev",
236
+
"status": "success",
237
+
"sha": "32898245e4f7e95e2122f6aa8106c2680c4daffa...",
238
+
"layer": "d41bb6c70aa39c972b04922bb5d9be03",
239
+
"log": "Processing: [default: loading data]...",
240
+
"solution": "digraph opam { ... }"
241
+
}
242
+
```
243
+
244
+
#### matrix
245
+
246
+
Output a terminal-friendly status matrix with vertical package names for better readability.
247
+
248
+
```bash
249
+
$ braid matrix -m results/manifest.json
250
+
Build Status Matrix
251
+
Legend: S=success, F=failure, D=dependency_failed, -=no_solution, B=solution, (space)=not present
252
+
253
+
b
254
+
y
255
+
t
256
+
e
257
+
s
258
+
r
259
+
c w
260
+
b -
261
+
o a e
262
+
r t i
263
+
t p o
264
+
---------------------
265
+
3289824 S F F ...
266
+
b92aa39 S F F ...
267
+
```
268
+
269
+
Package names are displayed vertically and bottom-aligned (with `.dev` suffix stripped), so all names end at the same row just above the data. This makes it easy to read package names of varying lengths.
270
+
271
+
## Manifest Format
272
+
273
+
The manifest.json file contains all results in a structured format:
274
+
275
+
```json
276
+
{
277
+
"repo_path": "/home/mtelvers/aoah-opam-repo",
278
+
"opam_repo_path": "/home/mtelvers/opam-repository",
279
+
"os": "debian-13",
280
+
"os_version": "13",
281
+
"generated_at": "2026-01-19T19:45:16Z",
282
+
"commits": ["3289824", "b92aa39", ...],
283
+
"packages": ["atp.dev", "cbort.dev", ...],
284
+
"results": [
285
+
{
286
+
"commit": "3289824...",
287
+
"short_commit": "3289824",
288
+
"message": "activitypub",
289
+
"packages": [
290
+
{
291
+
"name": "cbort.dev",
292
+
"status": "success",
293
+
"sha": "...",
294
+
"layer": "...",
295
+
"log": "...",
296
+
"solution": "..."
297
+
},
298
+
...
299
+
]
300
+
},
301
+
...
302
+
],
303
+
"mode": "history",
304
+
"overlay_repos": []
305
+
}
306
+
```
307
+
308
+
### Manifest Fields
309
+
310
+
| Field | Description |
311
+
|-------|-------------|
312
+
| `repo_path` | Primary overlay repository path |
313
+
| `opam_repo_path` | Main opam-repository path |
314
+
| `os` | Target OS (e.g., "debian-13") |
315
+
| `os_version` | OS version |
316
+
| `generated_at` | ISO 8601 timestamp |
317
+
| `commits` | List of commit hashes (or `["merge-test"]` for merge-test mode; short form: "merge-q") |
318
+
| `packages` | List of all package names |
319
+
| `results` | Array of per-commit results |
320
+
| `mode` | "history" for `run` command, "merge-test" for `merge-test` command |
321
+
| `overlay_repos` | For merge-test: list of stacked repos in priority order (first = highest) |
322
+
```
323
+
324
+
### Status Values
325
+
326
+
| Status | Symbol | Description |
327
+
|--------|--------|-------------|
328
+
| `success` | S | Package built successfully |
329
+
| `failure` | F | Package build failed |
330
+
| `dependency_failed` | D | A dependency failed to build |
331
+
| `no_solution` | - | Dependencies cannot be solved |
332
+
| `solution` | B | Solvable but not yet built (build candidate) |
333
+
| not present | (space) | Package does not exist at this commit |
334
+
335
+
## AI Agent Integration
336
+
337
+
Braid is designed for easy integration with AI agents. The manifest.json provides:
338
+
339
+
1. **Structured data** - All results in a single queryable JSON file
340
+
2. **Build logs** - Full build output for diagnosing failures
341
+
3. **Dependency graphs** - DOT format graphs showing package dependencies
342
+
4. **History tracking** - Status of each package across all commits
343
+
344
+
### Example: Diagnosing a Failure
345
+
346
+
An AI agent can:
347
+
348
+
1. Read the manifest to get an overview:
349
+
```bash
350
+
braid summary -m manifest.json
351
+
```
352
+
353
+
2. List current failures:
354
+
```bash
355
+
braid failures -m manifest.json
356
+
```
357
+
358
+
3. Get the build log for a failing package:
359
+
```bash
360
+
braid log 3289824 atp.dev -m manifest.json
361
+
```
362
+
363
+
4. Check when it started failing:
364
+
```bash
365
+
braid first-failure atp.dev -m manifest.json
366
+
```
367
+
368
+
5. View dependencies to understand the failure context:
369
+
```bash
370
+
braid deps 3289824 atp.dev -m manifest.json
371
+
```
372
+
373
+
### Programmatic Access
374
+
375
+
The manifest can also be read directly as JSON for more complex queries:
376
+
377
+
```python
378
+
import json
379
+
380
+
with open('manifest.json') as f:
381
+
manifest = json.load(f)
382
+
383
+
# Find all packages that have ever failed
384
+
failed_packages = set()
385
+
for result in manifest['results']:
386
+
for pkg in result['packages']:
387
+
if pkg['status'] == 'failure':
388
+
failed_packages.add(pkg['name'])
389
+
390
+
print(f"Packages that have failed: {failed_packages}")
391
+
```
392
+
393
+
## Tutorial: Merge Testing Workflow
394
+
395
+
This tutorial demonstrates using `braid merge-test` to validate overlay repositories before merging.
396
+
397
+
### 1. Create a Test Overlay Repository
398
+
399
+
Create an opam repository structure with packages to test:
400
+
401
+
```bash
402
+
mkdir -p ~/claude-repo/packages/mypackage/mypackage.dev
403
+
```
404
+
405
+
Create the `repo` file:
406
+
```bash
407
+
echo 'opam-version: "2.0"' > ~/claude-repo/repo
408
+
```
409
+
410
+
For each package, create an opam file at `packages/<name>/<name>.dev/opam`:
411
+
412
+
```
413
+
opam-version: "2.0"
414
+
synopsis: "My package"
415
+
depends: [
416
+
"ocaml" {>= "5.0"}
417
+
"dune" {>= "3.0"}
418
+
]
419
+
build: [
420
+
["dune" "build" "-p" name "-j" jobs]
421
+
]
422
+
url {
423
+
src: "git+https://github.com/user/repo.git"
424
+
}
425
+
```
426
+
427
+
### 2. Run Initial Merge Test
428
+
429
+
Test the overlay with a quick dry-run first:
430
+
431
+
```bash
432
+
braid merge-test ~/claude-repo --dry-run -o results
433
+
```
434
+
435
+
This shows which packages are solvable without building. The output includes a "solution" count for packages that can be built.
436
+
437
+
### 3. Run Full Build Test
438
+
439
+
Run the actual build:
440
+
441
+
```bash
442
+
braid merge-test ~/claude-repo -o results
443
+
```
444
+
445
+
Example output:
446
+
```
447
+
Merge test: 1 overlay repos, 3 packages
448
+
Overlay repos (priority order):
449
+
/home/user/claude-repo
450
+
Results: 1 success, 2 failure, 0 dep_failed, 0 no_solution, 0 error
451
+
```
452
+
453
+
### 4. Diagnose Failures
454
+
455
+
Check which packages failed:
456
+
457
+
```bash
458
+
braid failures -m results/manifest.json
459
+
```
460
+
461
+
View the build log for a failing package:
462
+
463
+
```bash
464
+
braid log merge-q smtpd.dev -m results/manifest.json
465
+
```
466
+
467
+
Example output showing a missing dependency:
468
+
```
469
+
pam_stubs.c:7:10: fatal error: security/pam_appl.h: No such file or directory
470
+
7 | #include <security/pam_appl.h>
471
+
| ^~~~~~~~~~~~~~~~~~~~~
472
+
```
473
+
474
+
### 5. Fix and Retest
475
+
476
+
In this case, the package needs `conf-pam` as a build dependency. Update the opam file:
477
+
478
+
```
479
+
depends: [
480
+
"ocaml" {>= "5.0"}
481
+
"dune" {>= "3.0"}
482
+
"conf-pam" {build} # Added for PAM headers
483
+
...
484
+
]
485
+
```
486
+
487
+
Rerun the merge test:
488
+
489
+
```bash
490
+
braid merge-test ~/claude-repo -o results
491
+
```
492
+
493
+
```
494
+
Results: 3 success, 0 failure, 0 dep_failed, 0 no_solution, 0 error
495
+
```
496
+
497
+
### 6. Test Multiple Stacked Overlays
498
+
499
+
Test what happens when merging multiple overlays together:
500
+
501
+
```bash
502
+
braid merge-test ~/my-overlay ~/upstream-overlay -o merge-results
503
+
```
504
+
505
+
Overlays are listed in priority order (first = highest priority, "on top"). This tests the combined effect of merging all overlays.
506
+
507
+
### 7. Query Results
508
+
509
+
After testing, query the manifest for details:
510
+
511
+
```bash
512
+
# Package history
513
+
braid history smtpd.dev -m results/manifest.json
514
+
515
+
# Dependency graph
516
+
braid deps merge-q smtpd.dev -m results/manifest.json
517
+
518
+
# Full JSON result
519
+
braid result merge-q smtpd.dev -m results/manifest.json
520
+
```
521
+
522
+
## Tutorial: Remote Execution via RPC
523
+
524
+
Braid supports remote execution using Cap'n Proto RPC. This allows you to run health checks on a remote server that has day10 and opam-repository available, from a client that only needs the braid binary and a capability file.
525
+
526
+
### Use Cases
527
+
528
+
- **Dev containers**: Run builds from lightweight development environments without day10 or opam-repository
529
+
- **Centralised build server**: Share a single build server across multiple developers
530
+
- **CI/CD integration**: Submit builds from CI pipelines to a dedicated build infrastructure
531
+
532
+
### 1. Start the Server
533
+
534
+
On a machine with day10 and opam-repository:
535
+
536
+
```bash
537
+
braid server --port 5000 \
538
+
--public-addr build.example.com \
539
+
--key-file /var/lib/braid/server.key \
540
+
--cap-file /var/lib/braid/braid.cap \
541
+
--opam-repo /home/user/opam-repository \
542
+
--cache-dir /var/cache/day10
543
+
```
544
+
545
+
**Server options:**
546
+
- `--port PORT` - Port to listen on (required)
547
+
- `--public-addr HOST` - Public hostname for the capability URI (required)
548
+
- `--key-file PATH` - Path to store/load the server's secret key (default: server.key)
549
+
- `--cap-file PATH` - Path to write the capability file (default: braid.cap)
550
+
- `--opam-repo PATH` - Path to opam-repository
551
+
- `--cache-dir PATH` - Cache directory for day10
552
+
553
+
The `--key-file` option ensures the capability URI remains stable across server restarts. Without it, clients would need a new capability file each time the server restarts.
554
+
555
+
### 2. Distribute the Capability File
556
+
557
+
Copy the capability file to any client machine:
558
+
559
+
```bash
560
+
scp build.example.com:/var/lib/braid/braid.cap ~/.config/braid.cap
561
+
```
562
+
563
+
The capability file contains a URI like:
564
+
```
565
+
capnp://sha-256:abc123...@build.example.com:5000/def456...
566
+
```
567
+
568
+
This URI encodes both the server address and a cryptographic capability token.
569
+
570
+
### 3. Run Remote Merge Tests
571
+
572
+
From the client, use `--connect` with a repository URL (not a local path):
573
+
574
+
```bash
575
+
braid merge-test https://github.com/user/overlay-repo \
576
+
--connect ~/.config/braid.cap \
577
+
-o results
578
+
```
579
+
580
+
The server will:
581
+
1. Clone the repository to a temporary directory
582
+
2. Run day10 health checks
583
+
3. Return the manifest JSON
584
+
4. Clean up the temporary directory
585
+
586
+
Example output:
587
+
```
588
+
Merge test: 1 overlay repos, 4 packages (remote)
589
+
Overlay repos (priority order):
590
+
https://github.com/user/overlay-repo
591
+
Results: 4 success, 0 failure, 0 dep_failed, 0 no_solution, 0 error
592
+
```
593
+
594
+
### 4. Run Remote History Checks
595
+
596
+
The `run` command also supports remote execution:
597
+
598
+
```bash
599
+
braid run https://github.com/user/overlay-repo \
600
+
--connect ~/.config/braid.cap \
601
+
-n 10 \
602
+
-o results
603
+
```
604
+
605
+
### 5. Query Results Locally
606
+
607
+
Once the manifest is downloaded, all query commands work locally:
608
+
609
+
```bash
610
+
braid summary -m results/manifest.json
611
+
braid failures -m results/manifest.json
612
+
braid log merge-q mypackage.dev -m results/manifest.json
613
+
```
614
+
615
+
### Important Notes for RPC Usage
616
+
617
+
1. **Repository URLs**: When using `--connect`, pass git URLs instead of local paths. The server clones the repository.
618
+
619
+
2. **Opam file requirements**: Packages in the overlay repository must have a `url` section in their opam files:
620
+
```
621
+
url {
622
+
src: "git+https://github.com/user/package-repo.git"
623
+
}
624
+
```
625
+
This tells day10 where to fetch the package source.
626
+
627
+
3. **Network requirements**: The server must be able to clone repositories from the URLs you provide.
628
+
629
+
4. **Capability security**: The capability file grants full access to the braid server. Treat it like a password.
630
+
631
+
### Example: Complete Workflow
632
+
633
+
```bash
634
+
# On the server (once)
635
+
braid server --port 5000 \
636
+
--public-addr basil.caelum.ci.dev \
637
+
--key-file ~/braid-server.key \
638
+
--cap-file ~/braid.cap \
639
+
--opam-repo ~/opam-repository \
640
+
--cache-dir /var/cache/day10
641
+
642
+
# Distribute capability (once)
643
+
scp basil.caelum.ci.dev:~/braid.cap ~/.config/
644
+
645
+
# From any client - test an overlay
646
+
braid merge-test https://github.com/mtelvers/claude-repo \
647
+
--connect ~/.config/braid.cap \
648
+
-o /tmp/results
649
+
650
+
# Check results
651
+
braid summary -m /tmp/results/manifest.json
652
+
braid failures -m /tmp/results/manifest.json
653
+
```
654
+
655
+
## Dependencies
656
+
657
+
- OCaml >= 4.14
658
+
- cmdliner >= 1.2
659
+
- yojson >= 2.0
660
+
- bos >= 0.2
661
+
- fmt >= 0.9
662
+
- logs >= 0.7
663
+
- fpath >= 0.7
664
+
- capnp-rpc = 2.1 (for RPC support)
665
+
- eio >= 1.2 (for RPC support)
666
+
- [day10](https://github.com/mtelvers/day10) (must be in PATH on server)
667
+
668
+
## License
669
+
670
+
ISC
+6
braid/bin/dune
+6
braid/bin/dune
+481
braid/bin/main.ml
+481
braid/bin/main.ml
···
1
+
(** Braid CLI - Build status tracker for opam overlay repositories *)
2
+
3
+
open Cmdliner
4
+
open Braid
5
+
6
+
let setup_log style_renderer level =
7
+
Fmt_tty.setup_std_outputs ?style_renderer ();
8
+
Logs.set_level level;
9
+
Logs.set_reporter (Logs_fmt.reporter ());
10
+
()
11
+
12
+
let setup_log_term =
13
+
Term.(const setup_log $ Fmt_cli.style_renderer () $ Logs_cli.level ())
14
+
15
+
(* Common arguments *)
16
+
let manifest_file =
17
+
let doc = "Path to manifest.json file" in
18
+
Arg.(value & opt string "manifest.json" & info ["m"; "manifest"] ~docv:"FILE" ~doc)
19
+
20
+
let connect_arg =
21
+
let doc = "Cap'n Proto capability file for remote execution" in
22
+
Arg.(value & opt (some string) None & info ["connect"] ~docv:"CAP_FILE" ~doc)
23
+
24
+
(* Run subcommand *)
25
+
let run_cmd =
26
+
let repo_path =
27
+
let doc = "Path to the overlay opam repository (or URL when using --connect)" in
28
+
Arg.(required & pos 0 (some string) None & info [] ~docv:"REPO" ~doc)
29
+
in
30
+
let opam_repo =
31
+
let doc = "Path to the main opam repository" in
32
+
Arg.(value & opt string "/home/mtelvers/opam-repository" & info ["opam-repo"] ~docv:"PATH" ~doc)
33
+
in
34
+
let cache_dir =
35
+
let doc = "Cache directory for day10" in
36
+
Arg.(value & opt string "/var/cache/day10" & info ["cache-dir"] ~docv:"PATH" ~doc)
37
+
in
38
+
let output_dir =
39
+
let doc = "Output directory for results" in
40
+
Arg.(value & opt string "results" & info ["o"; "output"] ~docv:"PATH" ~doc)
41
+
in
42
+
let num_commits =
43
+
let doc = "Number of commits to process" in
44
+
Arg.(value & opt int 10 & info ["n"; "num-commits"] ~docv:"N" ~doc)
45
+
in
46
+
let fork_jobs =
47
+
let doc = "Number of parallel jobs for solving" in
48
+
Arg.(value & opt int 40 & info ["j"; "jobs"] ~docv:"N" ~doc)
49
+
in
50
+
let os =
51
+
let doc = "Operating system" in
52
+
Arg.(value & opt string "linux" & info ["os"] ~docv:"OS" ~doc)
53
+
in
54
+
let os_family =
55
+
let doc = "OS family" in
56
+
Arg.(value & opt string "debian" & info ["os-family"] ~docv:"FAMILY" ~doc)
57
+
in
58
+
let os_distribution =
59
+
let doc = "OS distribution" in
60
+
Arg.(value & opt string "debian" & info ["os-distribution"] ~docv:"DIST" ~doc)
61
+
in
62
+
let os_version =
63
+
let doc = "OS version" in
64
+
Arg.(value & opt string "13" & info ["os-version"] ~docv:"VERSION" ~doc)
65
+
in
66
+
67
+
let run _setup repo_path opam_repo cache_dir output_dir num_commits fork_jobs
68
+
os os_family os_distribution os_version connect =
69
+
match connect with
70
+
| Some cap_file ->
71
+
(* Remote execution via RPC *)
72
+
Eio_main.run @@ fun env ->
73
+
Eio.Switch.run @@ fun sw ->
74
+
let net = Eio.Stdenv.net env in
75
+
let manifest_json = Rpc_client.run_remote ~sw ~net ~cap_file
76
+
~repo_url:repo_path ~num_commits ~fork_jobs
77
+
~os ~os_family ~os_distribution ~os_version in
78
+
let json = Yojson.Basic.from_string manifest_json in
79
+
let manifest = Json.manifest_of_json json in
80
+
(* Write manifest to output directory *)
81
+
(try Unix.mkdir output_dir 0o755 with Unix.Unix_error (Unix.EEXIST, _, _) -> ());
82
+
let manifest_path = Filename.concat output_dir "manifest.json" in
83
+
let _ = Json.write_manifest manifest_path manifest in
84
+
let (s, f, d, n, b, e) = Query.summary manifest in
85
+
Fmt.pr "Processed %d commits, %d packages (remote)@."
86
+
(List.length manifest.commits) (List.length manifest.packages);
87
+
Fmt.pr "Latest: %d success, %d failure, %d dep_failed, %d no_solution, %d solution, %d error@."
88
+
s f d n b e;
89
+
`Ok ()
90
+
| None ->
91
+
(* Local execution *)
92
+
match Runner.run ~repo_path ~opam_repo_path:opam_repo ~cache_dir ~output_dir
93
+
~os ~os_family ~os_distribution ~os_version
94
+
~fork_jobs ~num_commits with
95
+
| Ok manifest ->
96
+
let (s, f, d, n, b, e) = Query.summary manifest in
97
+
Fmt.pr "Processed %d commits, %d packages@."
98
+
(List.length manifest.commits) (List.length manifest.packages);
99
+
Fmt.pr "Latest: %d success, %d failure, %d dep_failed, %d no_solution, %d solution, %d error@."
100
+
s f d n b e;
101
+
`Ok ()
102
+
| Error e ->
103
+
Fmt.epr "Error: %a@." Rresult.R.pp_msg e;
104
+
`Error (false, "run failed")
105
+
in
106
+
107
+
let doc = "Run day10 health checks across commits" in
108
+
let info = Cmd.info "run" ~doc in
109
+
Cmd.v info Term.(ret (const run $ setup_log_term $ repo_path $ opam_repo $ cache_dir
110
+
$ output_dir $ num_commits $ fork_jobs
111
+
$ os $ os_family $ os_distribution $ os_version $ connect_arg))
112
+
113
+
(* Merge-test subcommand *)
114
+
let merge_test_cmd =
115
+
let overlay_repos =
116
+
let doc = "Overlay repository paths (in priority order, first = highest priority). Use URLs when using --connect." in
117
+
Arg.(non_empty & pos_all string [] & info [] ~docv:"REPOS" ~doc)
118
+
in
119
+
let opam_repo =
120
+
let doc = "Path to the main opam repository" in
121
+
Arg.(value & opt string "/home/mtelvers/opam-repository" & info ["opam-repo"] ~docv:"PATH" ~doc)
122
+
in
123
+
let cache_dir =
124
+
let doc = "Cache directory for day10" in
125
+
Arg.(value & opt string "/var/cache/day10" & info ["cache-dir"] ~docv:"PATH" ~doc)
126
+
in
127
+
let output_dir =
128
+
let doc = "Output directory for results" in
129
+
Arg.(value & opt string "results" & info ["o"; "output"] ~docv:"PATH" ~doc)
130
+
in
131
+
let fork_jobs =
132
+
let doc = "Number of parallel jobs for solving" in
133
+
Arg.(value & opt int 40 & info ["j"; "jobs"] ~docv:"N" ~doc)
134
+
in
135
+
let os =
136
+
let doc = "Operating system" in
137
+
Arg.(value & opt string "linux" & info ["os"] ~docv:"OS" ~doc)
138
+
in
139
+
let os_family =
140
+
let doc = "OS family" in
141
+
Arg.(value & opt string "debian" & info ["os-family"] ~docv:"FAMILY" ~doc)
142
+
in
143
+
let os_distribution =
144
+
let doc = "OS distribution" in
145
+
Arg.(value & opt string "debian" & info ["os-distribution"] ~docv:"DIST" ~doc)
146
+
in
147
+
let os_version =
148
+
let doc = "OS version" in
149
+
Arg.(value & opt string "13" & info ["os-version"] ~docv:"VERSION" ~doc)
150
+
in
151
+
let dry_run =
152
+
let doc = "Only solve dependencies, don't actually build" in
153
+
Arg.(value & flag & info ["dry-run"] ~doc)
154
+
in
155
+
156
+
let run _setup overlay_repos opam_repo cache_dir output_dir fork_jobs
157
+
os os_family os_distribution os_version dry_run connect =
158
+
match connect with
159
+
| Some cap_file ->
160
+
(* Remote execution via RPC *)
161
+
Eio_main.run @@ fun env ->
162
+
Eio.Switch.run @@ fun sw ->
163
+
let net = Eio.Stdenv.net env in
164
+
let manifest_json = Rpc_client.merge_test_remote ~sw ~net ~cap_file
165
+
~repo_urls:overlay_repos ~dry_run ~fork_jobs
166
+
~os ~os_family ~os_distribution ~os_version in
167
+
let json = Yojson.Basic.from_string manifest_json in
168
+
let manifest = Json.manifest_of_json json in
169
+
(* Write manifest to output directory *)
170
+
(try Unix.mkdir output_dir 0o755 with Unix.Unix_error (Unix.EEXIST, _, _) -> ());
171
+
let manifest_path = Filename.concat output_dir "manifest.json" in
172
+
let _ = Json.write_manifest manifest_path manifest in
173
+
let (s, f, d, n, b, e) = Query.summary manifest in
174
+
Fmt.pr "Merge test: %d overlay repos, %d packages (remote)@."
175
+
(List.length overlay_repos) (List.length manifest.packages);
176
+
Fmt.pr "Overlay repos (priority order):@.";
177
+
List.iter (fun r -> Fmt.pr " %s@." r) overlay_repos;
178
+
if dry_run then
179
+
Fmt.pr "Results: %d success, %d failure, %d dep_failed, %d no_solution, %d solution, %d error@."
180
+
s f d n b e
181
+
else
182
+
Fmt.pr "Results: %d success, %d failure, %d dep_failed, %d no_solution, %d error@."
183
+
s f d n e;
184
+
`Ok ()
185
+
| None ->
186
+
(* Local execution *)
187
+
match Runner.merge_test ~overlay_repos ~opam_repo_path:opam_repo ~cache_dir ~output_dir
188
+
~os ~os_family ~os_distribution ~os_version ~fork_jobs ~dry_run with
189
+
| Ok manifest ->
190
+
let (s, f, d, n, b, e) = Query.summary manifest in
191
+
Fmt.pr "Merge test: %d overlay repos, %d packages@."
192
+
(List.length overlay_repos) (List.length manifest.packages);
193
+
Fmt.pr "Overlay repos (priority order):@.";
194
+
List.iter (fun r -> Fmt.pr " %s@." r) overlay_repos;
195
+
if dry_run then
196
+
Fmt.pr "Results: %d success, %d failure, %d dep_failed, %d no_solution, %d solution, %d error@."
197
+
s f d n b e
198
+
else
199
+
Fmt.pr "Results: %d success, %d failure, %d dep_failed, %d no_solution, %d error@."
200
+
s f d n e;
201
+
`Ok ()
202
+
| Error e ->
203
+
Fmt.epr "Error: %a@." Rresult.R.pp_msg e;
204
+
`Error (false, "merge-test failed")
205
+
in
206
+
207
+
let doc = "Test cumulative effect of merging multiple overlay repositories" in
208
+
let info = Cmd.info "merge-test" ~doc in
209
+
Cmd.v info Term.(ret (const run $ setup_log_term $ overlay_repos $ opam_repo $ cache_dir
210
+
$ output_dir $ fork_jobs
211
+
$ os $ os_family $ os_distribution $ os_version $ dry_run $ connect_arg))
212
+
213
+
(* Server subcommand *)
214
+
let server_cmd =
215
+
let port_arg =
216
+
let doc = "Port to listen on" in
217
+
Arg.(required & opt (some int) None & info ["port"] ~docv:"PORT" ~doc)
218
+
in
219
+
let public_addr_arg =
220
+
let doc = "Public hostname/IP for capability URI (what clients use to connect)" in
221
+
Arg.(required & opt (some string) None & info ["public-addr"] ~docv:"HOST" ~doc)
222
+
in
223
+
let key_file_arg =
224
+
let doc = "Path to secret key file (created if doesn't exist)" in
225
+
Arg.(value & opt string "braid.key" & info ["key-file"] ~docv:"FILE" ~doc)
226
+
in
227
+
let cap_file_arg =
228
+
let doc = "Path to write capability file" in
229
+
Arg.(value & opt string "braid.cap" & info ["cap-file"] ~docv:"FILE" ~doc)
230
+
in
231
+
let listen_addr_arg =
232
+
let doc = "Address to listen on" in
233
+
Arg.(value & opt string "0.0.0.0" & info ["listen-addr"] ~docv:"ADDR" ~doc)
234
+
in
235
+
let opam_repo =
236
+
let doc = "Path to the main opam repository" in
237
+
Arg.(value & opt string "/home/mtelvers/opam-repository" & info ["opam-repo"] ~docv:"PATH" ~doc)
238
+
in
239
+
let cache_dir =
240
+
let doc = "Cache directory for day10" in
241
+
Arg.(value & opt string "/var/cache/day10" & info ["cache-dir"] ~docv:"PATH" ~doc)
242
+
in
243
+
244
+
let server _setup port public_addr key_file cap_file listen_addr opam_repo cache_dir =
245
+
Eio_main.run @@ fun env ->
246
+
Eio.Switch.run @@ fun sw ->
247
+
let net = Eio.Stdenv.net env in
248
+
let fs = Eio.Stdenv.cwd env in
249
+
Server.run ~sw ~net ~fs ~listen_addr ~listen_port:port ~public_addr
250
+
~key_file ~cap_file ~opam_repo_path:opam_repo ~cache_dir
251
+
in
252
+
253
+
let doc = "Start RPC server for remote braid execution" in
254
+
let info = Cmd.info "server" ~doc in
255
+
Cmd.v info Term.(const server $ setup_log_term $ port_arg $ public_addr_arg
256
+
$ key_file_arg $ cap_file_arg $ listen_addr_arg $ opam_repo $ cache_dir)
257
+
258
+
(* Query: failures *)
259
+
let failures_cmd =
260
+
let run _setup manifest_file =
261
+
match Json.read_manifest manifest_file with
262
+
| Error e ->
263
+
Fmt.epr "Error reading manifest: %a@." Rresult.R.pp_msg e;
264
+
`Error (false, "read failed")
265
+
| Ok manifest ->
266
+
let failures = Query.failures manifest in
267
+
if failures = [] then
268
+
Fmt.pr "No failures@."
269
+
else begin
270
+
Fmt.pr "Failures in commit %s:@." (fst (List.hd failures));
271
+
List.iter (fun (_, p) ->
272
+
Fmt.pr " %s@." p.Types.name
273
+
) failures
274
+
end;
275
+
`Ok ()
276
+
in
277
+
let doc = "List packages with status 'failure'" in
278
+
let info = Cmd.info "failures" ~doc in
279
+
Cmd.v info Term.(ret (const run $ setup_log_term $ manifest_file))
280
+
281
+
(* Query: log *)
282
+
let log_cmd =
283
+
let commit =
284
+
let doc = "Commit hash (short or full)" in
285
+
Arg.(required & pos 0 (some string) None & info [] ~docv:"COMMIT" ~doc)
286
+
in
287
+
let package =
288
+
let doc = "Package name" in
289
+
Arg.(required & pos 1 (some string) None & info [] ~docv:"PACKAGE" ~doc)
290
+
in
291
+
let run _setup manifest_file commit package =
292
+
match Json.read_manifest manifest_file with
293
+
| Error e ->
294
+
Fmt.epr "Error reading manifest: %a@." Rresult.R.pp_msg e;
295
+
`Error (false, "read failed")
296
+
| Ok manifest ->
297
+
match Query.log manifest ~commit ~package with
298
+
| None ->
299
+
Fmt.epr "No log found for %s at %s@." package commit;
300
+
`Error (false, "not found")
301
+
| Some log ->
302
+
Fmt.pr "%s@." log;
303
+
`Ok ()
304
+
in
305
+
let doc = "Show build log for a package at a commit" in
306
+
let info = Cmd.info "log" ~doc in
307
+
Cmd.v info Term.(ret (const run $ setup_log_term $ manifest_file $ commit $ package))
308
+
309
+
(* Query: history *)
310
+
let history_cmd =
311
+
let package =
312
+
let doc = "Package name" in
313
+
Arg.(required & pos 0 (some string) None & info [] ~docv:"PACKAGE" ~doc)
314
+
in
315
+
let run _setup manifest_file package =
316
+
match Json.read_manifest manifest_file with
317
+
| Error e ->
318
+
Fmt.epr "Error reading manifest: %a@." Rresult.R.pp_msg e;
319
+
`Error (false, "read failed")
320
+
| Ok manifest ->
321
+
match Query.history manifest ~package with
322
+
| None ->
323
+
Fmt.epr "Package %s not found@." package;
324
+
`Error (false, "not found")
325
+
| Some h ->
326
+
Fmt.pr "Package: %s@." h.Types.package;
327
+
Fmt.pr "First seen: %s@." h.first_seen;
328
+
Fmt.pr "Latest status: %s@." (Types.string_of_status h.latest_status);
329
+
Fmt.pr "History:@.";
330
+
List.iter (fun (c, s) ->
331
+
Fmt.pr " %s: %s@." c (Types.string_of_status s)
332
+
) h.history;
333
+
`Ok ()
334
+
in
335
+
let doc = "Show history of a package across commits" in
336
+
let info = Cmd.info "history" ~doc in
337
+
Cmd.v info Term.(ret (const run $ setup_log_term $ manifest_file $ package))
338
+
339
+
(* Query: deps *)
340
+
let deps_cmd =
341
+
let commit =
342
+
let doc = "Commit hash (short or full)" in
343
+
Arg.(required & pos 0 (some string) None & info [] ~docv:"COMMIT" ~doc)
344
+
in
345
+
let package =
346
+
let doc = "Package name" in
347
+
Arg.(required & pos 1 (some string) None & info [] ~docv:"PACKAGE" ~doc)
348
+
in
349
+
let run _setup manifest_file commit package =
350
+
match Json.read_manifest manifest_file with
351
+
| Error e ->
352
+
Fmt.epr "Error reading manifest: %a@." Rresult.R.pp_msg e;
353
+
`Error (false, "read failed")
354
+
| Ok manifest ->
355
+
match Query.deps manifest ~commit ~package with
356
+
| None ->
357
+
Fmt.epr "No dependency info for %s at %s@." package commit;
358
+
`Error (false, "not found")
359
+
| Some deps ->
360
+
Fmt.pr "%s@." deps;
361
+
`Ok ()
362
+
in
363
+
let doc = "Show dependency graph for a package (in dot format)" in
364
+
let info = Cmd.info "deps" ~doc in
365
+
Cmd.v info Term.(ret (const run $ setup_log_term $ manifest_file $ commit $ package))
366
+
367
+
(* Query: summary *)
368
+
let summary_cmd =
369
+
let run _setup manifest_file =
370
+
match Json.read_manifest manifest_file with
371
+
| Error e ->
372
+
Fmt.epr "Error reading manifest: %a@." Rresult.R.pp_msg e;
373
+
`Error (false, "read failed")
374
+
| Ok manifest ->
375
+
Fmt.pr "Repository: %s@." manifest.repo_path;
376
+
Fmt.pr "Generated: %s@." manifest.generated_at;
377
+
Fmt.pr "OS: %s@." manifest.os;
378
+
Fmt.pr "Commits: %d@." (List.length manifest.commits);
379
+
Fmt.pr "Packages: %d@." (List.length manifest.packages);
380
+
let (s, f, d, n, b, e) = Query.summary manifest in
381
+
Fmt.pr "@.Latest commit status:@.";
382
+
Fmt.pr " Success: %d@." s;
383
+
Fmt.pr " Failure: %d@." f;
384
+
Fmt.pr " Dependency failed: %d@." d;
385
+
Fmt.pr " No solution: %d@." n;
386
+
Fmt.pr " Solution (buildable): %d@." b;
387
+
Fmt.pr " Error: %d@." e;
388
+
`Ok ()
389
+
in
390
+
let doc = "Show summary statistics" in
391
+
let info = Cmd.info "summary" ~doc in
392
+
Cmd.v info Term.(ret (const run $ setup_log_term $ manifest_file))
393
+
394
+
(* Query: matrix *)
395
+
let matrix_cmd =
396
+
let run _setup manifest_file =
397
+
match Json.read_manifest manifest_file with
398
+
| Error e ->
399
+
Fmt.epr "Error reading manifest: %a@." Rresult.R.pp_msg e;
400
+
`Error (false, "read failed")
401
+
| Ok manifest ->
402
+
Fmt.pr "%s@." (Query.matrix manifest);
403
+
`Ok ()
404
+
in
405
+
let doc = "Output status matrix in markdown format" in
406
+
let info = Cmd.info "matrix" ~doc in
407
+
Cmd.v info Term.(ret (const run $ setup_log_term $ manifest_file))
408
+
409
+
(* Query: result - get full result for commit/package *)
410
+
let result_cmd =
411
+
let commit =
412
+
let doc = "Commit hash (short or full)" in
413
+
Arg.(required & pos 0 (some string) None & info [] ~docv:"COMMIT" ~doc)
414
+
in
415
+
let package =
416
+
let doc = "Package name" in
417
+
Arg.(required & pos 1 (some string) None & info [] ~docv:"PACKAGE" ~doc)
418
+
in
419
+
let run _setup manifest_file commit package =
420
+
match Json.read_manifest manifest_file with
421
+
| Error e ->
422
+
Fmt.epr "Error reading manifest: %a@." Rresult.R.pp_msg e;
423
+
`Error (false, "read failed")
424
+
| Ok manifest ->
425
+
match Query.result manifest ~commit ~package with
426
+
| None ->
427
+
Fmt.epr "No result for %s at %s@." package commit;
428
+
`Error (false, "not found")
429
+
| Some r ->
430
+
let json = Json.package_result_to_json r in
431
+
Fmt.pr "%s@." (Yojson.Basic.pretty_to_string json);
432
+
`Ok ()
433
+
in
434
+
let doc = "Get full result JSON for a package at a commit" in
435
+
let info = Cmd.info "result" ~doc in
436
+
Cmd.v info Term.(ret (const run $ setup_log_term $ manifest_file $ commit $ package))
437
+
438
+
(* Query: first-failure - find when a package first failed *)
439
+
let first_failure_cmd =
440
+
let package =
441
+
let doc = "Package name" in
442
+
Arg.(required & pos 0 (some string) None & info [] ~docv:"PACKAGE" ~doc)
443
+
in
444
+
let run _setup manifest_file package =
445
+
match Json.read_manifest manifest_file with
446
+
| Error e ->
447
+
Fmt.epr "Error reading manifest: %a@." Rresult.R.pp_msg e;
448
+
`Error (false, "read failed")
449
+
| Ok manifest ->
450
+
match Query.first_failure manifest ~package with
451
+
| None ->
452
+
Fmt.pr "Package %s has not failed (or was never successful)@." package;
453
+
`Ok ()
454
+
| Some (commit, message) ->
455
+
Fmt.pr "First failure: %s (%s)@." commit message;
456
+
`Ok ()
457
+
in
458
+
let doc = "Find when a package first started failing" in
459
+
let info = Cmd.info "first-failure" ~doc in
460
+
Cmd.v info Term.(ret (const run $ setup_log_term $ manifest_file $ package))
461
+
462
+
(* Main command *)
463
+
let main_cmd =
464
+
let doc = "Build status tracker for opam overlay repositories" in
465
+
let info = Cmd.info "braid" ~version:"0.1.0" ~doc in
466
+
let default = Term.(ret (const (`Help (`Pager, None)))) in
467
+
Cmd.group info ~default [
468
+
run_cmd;
469
+
merge_test_cmd;
470
+
server_cmd;
471
+
failures_cmd;
472
+
log_cmd;
473
+
history_cmd;
474
+
deps_cmd;
475
+
summary_cmd;
476
+
matrix_cmd;
477
+
result_cmd;
478
+
first_failure_cmd;
479
+
]
480
+
481
+
let () = exit (Cmd.eval main_cmd)
+42
braid/braid.opam
+42
braid/braid.opam
···
1
+
# This file is generated by dune, edit dune-project instead
2
+
opam-version: "2.0"
3
+
version: "0.1.0"
4
+
synopsis: "Build status tracker for opam overlay repositories"
5
+
description:
6
+
"Track and query package build status across git commits using day10"
7
+
maintainer: ["mark@mtelvers.com"]
8
+
authors: ["Mark Mayfield Mayfield"]
9
+
license: "ISC"
10
+
homepage: "https://github.com/mtelvers/braid"
11
+
bug-reports: "https://github.com/mtelvers/braid/issues"
12
+
depends: [
13
+
"dune" {>= "3.0"}
14
+
"ocaml" {>= "4.14"}
15
+
"cmdliner" {>= "1.2"}
16
+
"yojson" {>= "2.0"}
17
+
"bos" {>= "0.2"}
18
+
"fmt" {>= "0.9"}
19
+
"logs" {>= "0.7"}
20
+
"fpath" {>= "0.7"}
21
+
"capnp" {>= "3.6.0"}
22
+
"capnp-rpc" {= "2.1"}
23
+
"capnp-rpc-unix" {= "2.1"}
24
+
"eio" {>= "1.2"}
25
+
"eio_main" {>= "1.2"}
26
+
"odoc" {with-doc}
27
+
]
28
+
build: [
29
+
["dune" "subst"] {dev}
30
+
[
31
+
"dune"
32
+
"build"
33
+
"-p"
34
+
name
35
+
"-j"
36
+
jobs
37
+
"@install"
38
+
"@runtest" {with-test}
39
+
"@doc" {with-doc}
40
+
]
41
+
]
42
+
dev-repo: "git+https://github.com/mtelvers/braid.git"
+27
braid/dune-project
+27
braid/dune-project
···
1
+
(lang dune 3.0)
2
+
(name braid)
3
+
(version 0.1.0)
4
+
(generate_opam_files true)
5
+
6
+
(source (github mtelvers/braid))
7
+
(license ISC)
8
+
(authors "Mark Mayfield Mayfield")
9
+
(maintainers "mark@mtelvers.com")
10
+
11
+
(package
12
+
(name braid)
13
+
(synopsis "Build status tracker for opam overlay repositories")
14
+
(description "Track and query package build status across git commits using day10")
15
+
(depends
16
+
(ocaml (>= 4.14))
17
+
(cmdliner (>= 1.2))
18
+
(yojson (>= 2.0))
19
+
(bos (>= 0.2))
20
+
(fmt (>= 0.9))
21
+
(logs (>= 0.7))
22
+
(fpath (>= 0.7))
23
+
(capnp (>= 3.6.0))
24
+
(capnp-rpc (= 2.1))
25
+
(capnp-rpc-unix (= 2.1))
26
+
(eio (>= 1.2))
27
+
(eio_main (>= 1.2))))
+10
braid/lib/braid.ml
+10
braid/lib/braid.ml
···
1
+
(** Braid - Build status tracker for opam overlay repositories *)
2
+
3
+
module Types = Types
4
+
module Json = Json
5
+
module Runner = Runner
6
+
module Query = Query
7
+
module Rpc_schema = Rpc_schema
8
+
module Rpc_service = Rpc_service
9
+
module Rpc_client = Rpc_client
10
+
module Server = Server
+10
braid/lib/dune
+10
braid/lib/dune
···
1
+
(library
2
+
(name braid)
3
+
(public_name braid)
4
+
(libraries yojson bos fmt logs fpath str unix
5
+
capnp capnp-rpc capnp-rpc-unix eio eio_main))
6
+
7
+
(rule
8
+
(targets rpc_schema.ml rpc_schema.mli)
9
+
(deps rpc_schema.capnp)
10
+
(action (run capnp compile -o %{bin:capnpc-ocaml} %{deps})))
+129
braid/lib/json.ml
+129
braid/lib/json.ml
···
1
+
(** JSON serialization for braid types *)
2
+
3
+
open Types
4
+
5
+
let status_to_json status =
6
+
`String (string_of_status status)
7
+
8
+
let status_of_json = function
9
+
| `String s -> status_of_string s
10
+
| _ -> Error
11
+
12
+
let package_result_to_json r =
13
+
let fields = [
14
+
"name", `String r.name;
15
+
"status", status_to_json r.status;
16
+
] in
17
+
let fields = match r.sha with
18
+
| Some s -> ("sha", `String s) :: fields
19
+
| None -> fields
20
+
in
21
+
let fields = match r.layer with
22
+
| Some s -> ("layer", `String s) :: fields
23
+
| None -> fields
24
+
in
25
+
let fields = match r.log with
26
+
| Some s -> ("log", `String s) :: fields
27
+
| None -> fields
28
+
in
29
+
let fields = match r.solution with
30
+
| Some s -> ("solution", `String s) :: fields
31
+
| None -> fields
32
+
in
33
+
`Assoc (List.rev fields)
34
+
35
+
let package_result_of_json json =
36
+
let open Yojson.Basic.Util in
37
+
let name = json |> member "name" |> to_string in
38
+
let status = json |> member "status" |> status_of_json in
39
+
let sha = json |> member "sha" |> to_string_option in
40
+
let layer = json |> member "layer" |> to_string_option in
41
+
let log = json |> member "log" |> to_string_option in
42
+
let solution = json |> member "solution" |> to_string_option in
43
+
{ name; status; sha; layer; log; solution }
44
+
45
+
let commit_result_to_json r =
46
+
`Assoc [
47
+
"commit", `String r.commit;
48
+
"short_commit", `String r.short_commit;
49
+
"message", `String r.message;
50
+
"packages", `List (List.map package_result_to_json r.packages);
51
+
]
52
+
53
+
let commit_result_of_json json =
54
+
let open Yojson.Basic.Util in
55
+
let commit = json |> member "commit" |> to_string in
56
+
let short_commit = json |> member "short_commit" |> to_string in
57
+
let message = json |> member "message" |> to_string in
58
+
let packages = json |> member "packages" |> to_list |> List.map package_result_of_json in
59
+
{ commit; short_commit; message; packages }
60
+
61
+
let package_history_to_json h =
62
+
`Assoc [
63
+
"package", `String h.package;
64
+
"first_seen", `String h.first_seen;
65
+
"latest_status", status_to_json h.latest_status;
66
+
"history", `List (List.map (fun (c, s) ->
67
+
`Assoc ["commit", `String c; "status", status_to_json s]
68
+
) h.history);
69
+
]
70
+
71
+
let manifest_to_json m =
72
+
`Assoc [
73
+
"repo_path", `String m.repo_path;
74
+
"opam_repo_path", `String m.opam_repo_path;
75
+
"os", `String m.os;
76
+
"os_version", `String m.os_version;
77
+
"generated_at", `String m.generated_at;
78
+
"commits", `List (List.map (fun c -> `String c) m.commits);
79
+
"packages", `List (List.map (fun p -> `String p) m.packages);
80
+
"results", `List (List.map commit_result_to_json m.results);
81
+
"mode", `String m.mode;
82
+
"overlay_repos", `List (List.map (fun r -> `String r) m.overlay_repos);
83
+
]
84
+
85
+
let manifest_of_json json =
86
+
let open Yojson.Basic.Util in
87
+
let repo_path = json |> member "repo_path" |> to_string in
88
+
let opam_repo_path = json |> member "opam_repo_path" |> to_string in
89
+
let os = json |> member "os" |> to_string in
90
+
let os_version = json |> member "os_version" |> to_string in
91
+
let generated_at = json |> member "generated_at" |> to_string in
92
+
let commits = json |> member "commits" |> to_list |> List.map to_string in
93
+
let packages = json |> member "packages" |> to_list |> List.map to_string in
94
+
let results = json |> member "results" |> to_list |> List.map commit_result_of_json in
95
+
(* For backwards compatibility, default mode to "history" and overlay_repos to [] *)
96
+
let mode = match json |> member "mode" with
97
+
| `Null -> "history"
98
+
| j -> to_string j
99
+
in
100
+
let overlay_repos = match json |> member "overlay_repos" with
101
+
| `Null -> []
102
+
| j -> to_list j |> List.map to_string
103
+
in
104
+
{ repo_path; opam_repo_path; os; os_version; generated_at; commits; packages; results; mode; overlay_repos }
105
+
106
+
(** Parse a day10 JSON result file *)
107
+
let parse_day10_result json =
108
+
let open Yojson.Basic.Util in
109
+
let name = json |> member "name" |> to_string in
110
+
let status = json |> member "status" |> status_of_json in
111
+
let sha = json |> member "sha" |> to_string_option in
112
+
let layer = json |> member "layer" |> to_string_option in
113
+
let log = json |> member "log" |> to_string_option in
114
+
let solution = json |> member "solution" |> to_string_option in
115
+
{ name; status; sha; layer; log; solution }
116
+
117
+
(** Write manifest to file *)
118
+
let write_manifest path manifest =
119
+
let json = manifest_to_json manifest in
120
+
let content = Yojson.Basic.pretty_to_string json in
121
+
Bos.OS.File.write (Fpath.v path) content
122
+
123
+
(** Read manifest from file *)
124
+
let read_manifest path =
125
+
match Bos.OS.File.read (Fpath.v path) with
126
+
| Ok content ->
127
+
let json = Yojson.Basic.from_string content in
128
+
Ok (manifest_of_json json)
129
+
| Error e -> Error e
+159
braid/lib/query.ml
+159
braid/lib/query.ml
···
1
+
(** Query functions for braid manifest data *)
2
+
3
+
open Types
4
+
5
+
(** Get all failures (status = Failure) from latest commit *)
6
+
let failures manifest =
7
+
match manifest.results with
8
+
| [] -> []
9
+
| latest :: _ ->
10
+
latest.packages
11
+
|> List.filter (fun p -> p.status = Failure)
12
+
|> List.map (fun p -> (latest.short_commit, p))
13
+
14
+
(** Get all packages with a specific status from latest commit *)
15
+
let by_status manifest status =
16
+
match manifest.results with
17
+
| [] -> []
18
+
| latest :: _ ->
19
+
latest.packages
20
+
|> List.filter (fun p -> p.status = status)
21
+
|> List.map (fun p -> (latest.short_commit, p))
22
+
23
+
(** Get log for a specific commit and package *)
24
+
let log manifest ~commit ~package =
25
+
let commit_result = List.find_opt (fun r ->
26
+
r.short_commit = commit || r.commit = commit
27
+
) manifest.results in
28
+
match commit_result with
29
+
| None -> None
30
+
| Some r ->
31
+
let pkg_result = List.find_opt (fun p -> p.name = package) r.packages in
32
+
match pkg_result with
33
+
| None -> None
34
+
| Some p -> p.log
35
+
36
+
(** Get full result for a specific commit and package *)
37
+
let result manifest ~commit ~package =
38
+
let commit_result = List.find_opt (fun r ->
39
+
r.short_commit = commit || r.commit = commit
40
+
) manifest.results in
41
+
match commit_result with
42
+
| None -> None
43
+
| Some r ->
44
+
List.find_opt (fun p -> p.name = package) r.packages
45
+
46
+
(** Get history for a package across all commits *)
47
+
let history manifest ~package =
48
+
let hist = List.filter_map (fun (r : commit_result) ->
49
+
match List.find_opt (fun p -> p.name = package) r.packages with
50
+
| None -> None
51
+
| Some p -> Some (r.short_commit, p.status)
52
+
) manifest.results in
53
+
match hist with
54
+
| [] -> None
55
+
| _ ->
56
+
let latest_status = snd (List.hd hist) in
57
+
let first_seen = fst (List.hd (List.rev hist)) in
58
+
Some { package; first_seen; latest_status; history = hist }
59
+
60
+
(** Get dependencies for a package (from solution graph) *)
61
+
let deps manifest ~commit ~package =
62
+
match result manifest ~commit ~package with
63
+
| None -> None
64
+
| Some r -> r.solution
65
+
66
+
(** Get packages that depend on a given package *)
67
+
let rdeps manifest ~commit ~package =
68
+
let commit_result = List.find_opt (fun r ->
69
+
r.short_commit = commit || r.commit = commit
70
+
) manifest.results in
71
+
match commit_result with
72
+
| None -> []
73
+
| Some r ->
74
+
r.packages
75
+
|> List.filter (fun p ->
76
+
match p.solution with
77
+
| None -> false
78
+
| Some sol ->
79
+
(* Check if the solution graph mentions our package *)
80
+
let pattern = "\"" ^ package in
81
+
String.length sol > 0 &&
82
+
(try let _ = Str.search_forward (Str.regexp_string pattern) sol 0 in true
83
+
with Not_found -> false))
84
+
|> List.map (fun p -> p.name)
85
+
86
+
(** Get summary statistics *)
87
+
let summary manifest =
88
+
match manifest.results with
89
+
| [] -> (0, 0, 0, 0, 0, 0)
90
+
| latest :: _ ->
91
+
let count status = List.length (List.filter (fun p -> p.status = status) latest.packages) in
92
+
(count Success, count Failure, count Dependency_failed, count No_solution, count Solution, count Error)
93
+
94
+
(** Find when a package first started failing *)
95
+
let first_failure manifest ~package =
96
+
let hist = List.filter_map (fun (r : commit_result) ->
97
+
match List.find_opt (fun p -> p.name = package) r.packages with
98
+
| None -> None
99
+
| Some p -> Some (r.short_commit, r.message, p.status)
100
+
) manifest.results in
101
+
(* Find transition from non-failure to failure (going backwards in time) *)
102
+
let rec find_transition = function
103
+
| [] -> None
104
+
| [(c, m, s)] -> if s = Failure then Some (c, m) else None
105
+
| (c1, m1, s1) :: ((_c2, _m2, s2) :: _ as rest) ->
106
+
if s1 = Failure && s2 <> Failure then Some (c1, m1)
107
+
else find_transition rest
108
+
in
109
+
find_transition (List.rev hist)
110
+
111
+
(** Generate terminal-friendly matrix with vertical package names *)
112
+
let matrix manifest =
113
+
let buf = Buffer.create 4096 in
114
+
Buffer.add_string buf "Build Status Matrix\n";
115
+
Buffer.add_string buf "Legend: S=success, F=failure, D=dependency_failed, -=no_solution, B=solution\n\n";
116
+
117
+
let packages = manifest.packages in
118
+
(* Strip .dev suffix for display *)
119
+
let display_names = List.map (fun pkg ->
120
+
if String.length pkg > 4 && String.sub pkg (String.length pkg - 4) 4 = ".dev" then
121
+
String.sub pkg 0 (String.length pkg - 4)
122
+
else pkg
123
+
) packages in
124
+
let commit_width = 9 in (* "Commit" + padding *)
125
+
126
+
(* Find the longest display name for vertical header height *)
127
+
let max_pkg_len = List.fold_left (fun acc pkg -> max acc (String.length pkg)) 0 display_names in
128
+
129
+
(* Print vertical package names (bottom-aligned) *)
130
+
for row = 0 to max_pkg_len - 1 do
131
+
Buffer.add_string buf (String.make commit_width ' ');
132
+
List.iter (fun pkg ->
133
+
let pkg_len = String.length pkg in
134
+
let offset = max_pkg_len - pkg_len in
135
+
let ch = if row >= offset then String.make 1 pkg.[row - offset] else " " in
136
+
Buffer.add_string buf (Printf.sprintf " %s " ch)
137
+
) display_names;
138
+
Buffer.add_char buf '\n'
139
+
done;
140
+
141
+
(* Separator line *)
142
+
Buffer.add_string buf (String.make commit_width '-');
143
+
List.iter (fun _ -> Buffer.add_string buf "---") display_names;
144
+
Buffer.add_char buf '\n';
145
+
146
+
(* Data rows *)
147
+
List.iter (fun (r : commit_result) ->
148
+
Buffer.add_string buf (Printf.sprintf "%-8s " r.short_commit);
149
+
List.iter (fun pkg_name ->
150
+
let symbol = match List.find_opt (fun p -> p.name = pkg_name) r.packages with
151
+
| None -> " "
152
+
| Some p -> status_symbol p.status
153
+
in
154
+
Buffer.add_string buf (Printf.sprintf " %s " symbol)
155
+
) packages;
156
+
Buffer.add_char buf '\n'
157
+
) manifest.results;
158
+
159
+
Buffer.contents buf
+41
braid/lib/rpc_client.ml
+41
braid/lib/rpc_client.ml
···
1
+
(** RPC client for connecting to remote BraidService *)
2
+
3
+
module Api = Rpc_schema.MakeRPC(Capnp_rpc)
4
+
5
+
(** Connect to a remote BraidService using a capability file *)
6
+
let connect ~sw ~net cap_file =
7
+
let vat = Capnp_rpc_unix.client_only_vat ~sw net in
8
+
let sr = Capnp_rpc_unix.Cap_file.load vat cap_file |> Result.get_ok in
9
+
Capnp_rpc.Sturdy_ref.connect_exn sr
10
+
11
+
(** Run health checks on a remote server *)
12
+
let run_remote ~sw ~net ~cap_file ~repo_url ~num_commits ~fork_jobs
13
+
~os ~os_family ~os_distribution ~os_version =
14
+
let service = connect ~sw ~net cap_file in
15
+
let open Api.Client.BraidService.Run in
16
+
let request, params = Capnp_rpc.Capability.Request.create Params.init_pointer in
17
+
Params.repo_url_set params repo_url;
18
+
Params.num_commits_set params (Stdint.Uint32.of_int num_commits);
19
+
Params.fork_jobs_set params (Stdint.Uint32.of_int fork_jobs);
20
+
Params.os_set params os;
21
+
Params.os_family_set params os_family;
22
+
Params.os_distribution_set params os_distribution;
23
+
Params.os_version_set params os_version;
24
+
let response = Capnp_rpc.Capability.call_for_value_exn service method_id request in
25
+
Results.manifest_json_get response
26
+
27
+
(** Run merge test on a remote server *)
28
+
let merge_test_remote ~sw ~net ~cap_file ~repo_urls ~dry_run ~fork_jobs
29
+
~os ~os_family ~os_distribution ~os_version =
30
+
let service = connect ~sw ~net cap_file in
31
+
let open Api.Client.BraidService.MergeTest in
32
+
let request, params = Capnp_rpc.Capability.Request.create Params.init_pointer in
33
+
let _ = Params.repo_urls_set_list params repo_urls in
34
+
Params.dry_run_set params dry_run;
35
+
Params.fork_jobs_set params (Stdint.Uint32.of_int fork_jobs);
36
+
Params.os_set params os;
37
+
Params.os_family_set params os_family;
38
+
Params.os_distribution_set params os_distribution;
39
+
Params.os_version_set params os_version;
40
+
let response = Capnp_rpc.Capability.call_for_value_exn service method_id request in
41
+
Results.manifest_json_get response
+29
braid/lib/rpc_schema.capnp
+29
braid/lib/rpc_schema.capnp
···
1
+
@0xb8e7a3f2c1d4e5f6;
2
+
3
+
interface BraidService {
4
+
# Run health checks across commits
5
+
run @0 (
6
+
repoUrl :Text,
7
+
numCommits :UInt32,
8
+
forkJobs :UInt32,
9
+
os :Text,
10
+
osFamily :Text,
11
+
osDistribution :Text,
12
+
osVersion :Text
13
+
) -> (
14
+
manifestJson :Text
15
+
);
16
+
17
+
# Merge test on stacked repositories
18
+
mergeTest @1 (
19
+
repoUrls :List(Text),
20
+
dryRun :Bool,
21
+
forkJobs :UInt32,
22
+
os :Text,
23
+
osFamily :Text,
24
+
osDistribution :Text,
25
+
osVersion :Text
26
+
) -> (
27
+
manifestJson :Text
28
+
);
29
+
}
+118
braid/lib/rpc_service.ml
+118
braid/lib/rpc_service.ml
···
1
+
(** RPC service implementation for BraidService *)
2
+
3
+
module Api = Rpc_schema.MakeRPC(Capnp_rpc)
4
+
5
+
(** Clone a git repository to a temporary directory *)
6
+
let clone_repo ~temp_dir url =
7
+
let repo_name =
8
+
(* Extract repo name from URL, e.g., "https://github.com/user/repo" -> "repo" *)
9
+
let base = Filename.basename url in
10
+
if String.length base > 4 && String.sub base (String.length base - 4) 4 = ".git" then
11
+
String.sub base 0 (String.length base - 4)
12
+
else
13
+
base
14
+
in
15
+
let repo_path = Filename.concat temp_dir repo_name in
16
+
(* GIT_TERMINAL_PROMPT=0 prevents git from prompting for credentials *)
17
+
let cmd_str = Printf.sprintf "GIT_TERMINAL_PROMPT=0 git clone --depth 100 %s %s" url repo_path in
18
+
match Unix.system cmd_str with
19
+
| Unix.WEXITED 0 -> Ok repo_path
20
+
| _ -> Error (`Msg (Printf.sprintf "Failed to clone %s" url))
21
+
22
+
(** Create a unique temp directory using mktemp *)
23
+
let make_temp_dir () =
24
+
let ic = Unix.open_process_in "mktemp -d -t braid.XXXXXX" in
25
+
let temp_dir = input_line ic in
26
+
let _ = Unix.close_process_in ic in
27
+
temp_dir
28
+
29
+
(** Create the local BraidService implementation *)
30
+
let local ~opam_repo_path ~cache_dir =
31
+
let module Service = Api.Service.BraidService in
32
+
Service.local @@ object
33
+
inherit Service.service
34
+
35
+
method run_impl params release_param_caps =
36
+
let open Service.Run in
37
+
release_param_caps ();
38
+
let repo_url = Params.repo_url_get params in
39
+
let num_commits = Params.num_commits_get params |> Stdint.Uint32.to_int in
40
+
let fork_jobs = Params.fork_jobs_get params |> Stdint.Uint32.to_int in
41
+
let os = Params.os_get params in
42
+
let os_family = Params.os_family_get params in
43
+
let os_distribution = Params.os_distribution_get params in
44
+
let os_version = Params.os_version_get params in
45
+
46
+
(* Create unique temp directory for each request *)
47
+
let temp_dir = make_temp_dir () in
48
+
49
+
let result =
50
+
match clone_repo ~temp_dir repo_url with
51
+
| Error (`Msg msg) -> Error msg
52
+
| Ok repo_path ->
53
+
let output_dir = Filename.concat temp_dir "results" in
54
+
(try Unix.mkdir output_dir 0o755 with Unix.Unix_error (Unix.EEXIST, _, _) -> ());
55
+
match Runner.run ~repo_path ~opam_repo_path ~cache_dir ~output_dir
56
+
~os ~os_family ~os_distribution ~os_version
57
+
~fork_jobs ~num_commits with
58
+
| Ok manifest ->
59
+
let json = Json.manifest_to_json manifest in
60
+
Ok (Yojson.Basic.to_string json)
61
+
| Error (`Msg msg) -> Error msg
62
+
in
63
+
64
+
(* Clean up temp directory *)
65
+
let _ = Unix.system (Printf.sprintf "rm -rf %s" temp_dir) in
66
+
67
+
let response, results = Capnp_rpc.Service.Response.create Results.init_pointer in
68
+
(match result with
69
+
| Ok manifest_json -> Results.manifest_json_set results manifest_json
70
+
| Error msg -> Results.manifest_json_set results (Printf.sprintf "{\"error\": \"%s\"}" msg));
71
+
Capnp_rpc.Service.return response
72
+
73
+
method merge_test_impl params release_param_caps =
74
+
let open Service.MergeTest in
75
+
release_param_caps ();
76
+
let repo_urls = Params.repo_urls_get_list params in
77
+
let dry_run = Params.dry_run_get params in
78
+
let fork_jobs = Params.fork_jobs_get params |> Stdint.Uint32.to_int in
79
+
let os = Params.os_get params in
80
+
let os_family = Params.os_family_get params in
81
+
let os_distribution = Params.os_distribution_get params in
82
+
let os_version = Params.os_version_get params in
83
+
84
+
(* Create unique temp directory for each request *)
85
+
let temp_dir = make_temp_dir () in
86
+
87
+
let result =
88
+
(* Clone all repos *)
89
+
let rec clone_all urls acc =
90
+
match urls with
91
+
| [] -> Ok (List.rev acc)
92
+
| url :: rest ->
93
+
match clone_repo ~temp_dir url with
94
+
| Error e -> Error e
95
+
| Ok path -> clone_all rest (path :: acc)
96
+
in
97
+
match clone_all repo_urls [] with
98
+
| Error (`Msg msg) -> Error msg
99
+
| Ok overlay_repos ->
100
+
let output_dir = Filename.concat temp_dir "results" in
101
+
(try Unix.mkdir output_dir 0o755 with Unix.Unix_error (Unix.EEXIST, _, _) -> ());
102
+
match Runner.merge_test ~overlay_repos ~opam_repo_path ~cache_dir ~output_dir
103
+
~os ~os_family ~os_distribution ~os_version ~fork_jobs ~dry_run with
104
+
| Ok manifest ->
105
+
let json = Json.manifest_to_json manifest in
106
+
Ok (Yojson.Basic.to_string json)
107
+
| Error (`Msg msg) -> Error msg
108
+
in
109
+
110
+
(* Clean up temp directory *)
111
+
let _ = Unix.system (Printf.sprintf "rm -rf %s" temp_dir) in
112
+
113
+
let response, results = Capnp_rpc.Service.Response.create Results.init_pointer in
114
+
(match result with
115
+
| Ok manifest_json -> Results.manifest_json_set results manifest_json
116
+
| Error msg -> Results.manifest_json_set results (Printf.sprintf "{\"error\": \"%s\"}" msg));
117
+
Capnp_rpc.Service.return response
118
+
end
+403
braid/lib/runner.ml
+403
braid/lib/runner.ml
···
1
+
(** Run day10 commands and collect results *)
2
+
3
+
open Types
4
+
5
+
let ( let* ) = Result.bind
6
+
7
+
(** Run a command and return stdout *)
8
+
let run_cmd args =
9
+
let cmd = Bos.Cmd.of_list args in
10
+
Logs.info (fun m -> m "Executing: %a" Bos.Cmd.pp cmd);
11
+
Bos.OS.Cmd.run_out cmd |> Bos.OS.Cmd.out_string
12
+
13
+
(** Run a command, ignoring output *)
14
+
let run_cmd_quiet args =
15
+
let cmd = Bos.Cmd.of_list args in
16
+
match Bos.OS.Cmd.run_out cmd |> Bos.OS.Cmd.out_null with
17
+
| Ok ((), _status) -> Ok ()
18
+
| Error e -> Error e
19
+
20
+
(** Get list of packages from day10 list *)
21
+
let list_packages ~repo_path ~os ~os_family ~os_distribution ~os_version =
22
+
let args = [
23
+
"day10"; "list";
24
+
"--opam-repository"; repo_path;
25
+
"--os"; os;
26
+
"--os-family"; os_family;
27
+
"--os-distribution"; os_distribution;
28
+
"--os-version"; os_version;
29
+
] in
30
+
let* (output, _) = run_cmd args in
31
+
let packages = String.split_on_char '\n' output
32
+
|> List.filter (fun s -> String.length s > 0)
33
+
in
34
+
Ok packages
35
+
36
+
(** Get git commits *)
37
+
let get_commits ~repo_path ~num_commits =
38
+
let* () = Bos.OS.Dir.set_current (Fpath.v repo_path) in
39
+
let args = ["git"; "log"; "--oneline"; "-n"; string_of_int num_commits; "--format=%H"] in
40
+
let* (output, _) = run_cmd args in
41
+
let commits = String.split_on_char '\n' output
42
+
|> List.filter (fun s -> String.length s > 0)
43
+
in
44
+
Ok commits
45
+
46
+
(** Get commit message *)
47
+
let get_commit_message commit =
48
+
let args = ["git"; "log"; "-1"; "--format=%s"; commit] in
49
+
match run_cmd args with
50
+
| Ok (msg, _) -> String.trim msg
51
+
| Error _ -> ""
52
+
53
+
(** Checkout a commit *)
54
+
let checkout commit =
55
+
let args = ["git"; "checkout"; "-q"; commit] in
56
+
run_cmd_quiet args
57
+
58
+
(** Run day10 health-check with --dry-run and --json *)
59
+
let health_check ~repo_path ~opam_repo_path ~cache_dir ~os ~os_family ~os_distribution ~os_version ~fork_jobs ~output_dir ~packages =
60
+
(* Create packages JSON file *)
61
+
let packages_file = Filename.concat output_dir "packages.json" in
62
+
let packages_json = Printf.sprintf {|{"packages":[%s]}|}
63
+
(String.concat "," (List.map (Printf.sprintf {|"%s"|}) packages))
64
+
in
65
+
let* () = Bos.OS.File.write (Fpath.v packages_file) packages_json in
66
+
67
+
let results_dir = Filename.concat output_dir "results" in
68
+
let* _ = Bos.OS.Dir.create (Fpath.v results_dir) in
69
+
70
+
let args = [
71
+
"day10"; "health-check";
72
+
"--opam-repository"; repo_path;
73
+
"--opam-repository"; opam_repo_path;
74
+
"--cache-dir"; cache_dir;
75
+
"--os"; os;
76
+
"--os-family"; os_family;
77
+
"--os-distribution"; os_distribution;
78
+
"--os-version"; os_version;
79
+
"--dry-run";
80
+
"--fork"; string_of_int fork_jobs;
81
+
"--json"; results_dir;
82
+
"@" ^ packages_file;
83
+
] in
84
+
let* _ = run_cmd args in
85
+
86
+
(* Parse result files *)
87
+
let* entries = Bos.OS.Dir.contents (Fpath.v results_dir) in
88
+
let results = List.filter_map (fun path ->
89
+
if Fpath.has_ext "json" path then
90
+
match Bos.OS.File.read path with
91
+
| Ok content ->
92
+
(try
93
+
let json = Yojson.Basic.from_string content in
94
+
Some (Json.parse_day10_result json)
95
+
with _ -> None)
96
+
| Error _ -> None
97
+
else None
98
+
) entries in
99
+
Ok results
100
+
101
+
(** Process a single commit *)
102
+
let process_commit ~repo_path ~opam_repo_path ~cache_dir ~os ~os_family ~os_distribution ~os_version ~fork_jobs ~temp_dir commit =
103
+
let short_commit = String.sub commit 0 7 in
104
+
let message = get_commit_message commit in
105
+
106
+
Logs.info (fun m -> m "Processing commit %s: %s" short_commit message);
107
+
108
+
let* () = checkout commit in
109
+
110
+
let* packages = list_packages ~repo_path ~os ~os_family ~os_distribution ~os_version in
111
+
112
+
if packages = [] then
113
+
Ok { commit; short_commit; message; packages = [] }
114
+
else begin
115
+
let output_dir = Filename.concat temp_dir short_commit in
116
+
let* _ = Bos.OS.Dir.create (Fpath.v output_dir) in
117
+
118
+
let* results = health_check
119
+
~repo_path ~opam_repo_path ~cache_dir
120
+
~os ~os_family ~os_distribution ~os_version
121
+
~fork_jobs ~output_dir ~packages
122
+
in
123
+
124
+
(* Sort results by package name *)
125
+
let sorted_results = List.sort (fun a b -> String.compare a.name b.name) results in
126
+
127
+
Ok { commit; short_commit; message; packages = sorted_results }
128
+
end
129
+
130
+
(** Run the full analysis *)
131
+
let run ~repo_path ~opam_repo_path ~cache_dir ~output_dir
132
+
~os ~os_family ~os_distribution ~os_version
133
+
~fork_jobs ~num_commits =
134
+
135
+
let* () = Bos.OS.Dir.set_current (Fpath.v repo_path) in
136
+
137
+
(* Reset to main branch *)
138
+
let* () = checkout "main" in
139
+
140
+
(* Get commits to process *)
141
+
let* commits = get_commits ~repo_path ~num_commits in
142
+
143
+
Logs.info (fun m -> m "Processing %d commits" (List.length commits));
144
+
145
+
(* Create temp directory for intermediate results *)
146
+
let* temp_dir = Bos.OS.Dir.tmp "braid-%s" in
147
+
let temp_dir = Fpath.to_string temp_dir in
148
+
149
+
(* Process each commit *)
150
+
let results = List.filter_map (fun commit ->
151
+
match process_commit ~repo_path ~opam_repo_path ~cache_dir
152
+
~os ~os_family ~os_distribution ~os_version
153
+
~fork_jobs ~temp_dir commit with
154
+
| Ok result -> Some result
155
+
| Error (`Msg e) ->
156
+
Logs.err (fun m -> m "Error processing %s: %s" commit e);
157
+
None
158
+
) commits in
159
+
160
+
(* Return to main branch *)
161
+
let _ = checkout "main" in
162
+
163
+
(* Collect all unique packages *)
164
+
let all_packages = results
165
+
|> List.concat_map (fun (r : commit_result) -> List.map (fun p -> p.name) r.packages)
166
+
|> List.sort_uniq String.compare
167
+
in
168
+
169
+
(* Build manifest *)
170
+
let generated_at =
171
+
let t = Unix.gettimeofday () in
172
+
let tm = Unix.gmtime t in
173
+
Printf.sprintf "%04d-%02d-%02dT%02d:%02d:%02dZ"
174
+
(tm.Unix.tm_year + 1900) (tm.Unix.tm_mon + 1) tm.Unix.tm_mday
175
+
tm.Unix.tm_hour tm.Unix.tm_min tm.Unix.tm_sec
176
+
in
177
+
178
+
let manifest = {
179
+
repo_path;
180
+
opam_repo_path;
181
+
os = Printf.sprintf "%s-%s" os_distribution os_version;
182
+
os_version;
183
+
generated_at;
184
+
commits = List.map (fun c -> String.sub c 0 7) commits;
185
+
packages = all_packages;
186
+
results;
187
+
mode = "history";
188
+
overlay_repos = [];
189
+
} in
190
+
191
+
(* Write manifest *)
192
+
let* _ = Bos.OS.Dir.create (Fpath.v output_dir) in
193
+
let manifest_path = Filename.concat output_dir "manifest.json" in
194
+
let* () = Json.write_manifest manifest_path manifest in
195
+
196
+
Logs.info (fun m -> m "Manifest written to %s" manifest_path);
197
+
198
+
Ok manifest
199
+
200
+
(** List packages from multiple overlay repos (union of all) *)
201
+
let list_packages_multi ~repo_paths ~os ~os_family ~os_distribution ~os_version =
202
+
(* Build args with multiple --opam-repository flags *)
203
+
let repo_args = List.concat_map (fun path ->
204
+
["--opam-repository"; path]
205
+
) repo_paths in
206
+
let args = ["day10"; "list"] @ repo_args @ [
207
+
"--os"; os;
208
+
"--os-family"; os_family;
209
+
"--os-distribution"; os_distribution;
210
+
"--os-version"; os_version;
211
+
] in
212
+
let* (output, _) = run_cmd args in
213
+
let packages = String.split_on_char '\n' output
214
+
|> List.filter (fun s -> String.length s > 0)
215
+
in
216
+
Ok packages
217
+
218
+
(** Parse results from a directory of JSON files *)
219
+
let parse_results_dir results_dir =
220
+
let* entries = Bos.OS.Dir.contents (Fpath.v results_dir) in
221
+
let results = List.filter_map (fun path ->
222
+
if Fpath.has_ext "json" path then
223
+
match Bos.OS.File.read path with
224
+
| Ok content ->
225
+
(try
226
+
let json = Yojson.Basic.from_string content in
227
+
Some (Json.parse_day10_result json)
228
+
with _ -> None)
229
+
| Error _ -> None
230
+
else None
231
+
) entries in
232
+
Ok results
233
+
234
+
(** Run day10 health-check with multiple overlay repos (two-stage: solve then build) *)
235
+
let health_check_multi ~overlay_repos ~opam_repo_path ~cache_dir ~os ~os_family ~os_distribution ~os_version ~fork_jobs ~dry_run ~output_dir ~packages =
236
+
(* Create packages JSON file *)
237
+
let packages_file = Filename.concat output_dir "packages.json" in
238
+
let packages_json = Printf.sprintf {|{"packages":[%s]}|}
239
+
(String.concat "," (List.map (Printf.sprintf {|"%s"|}) packages))
240
+
in
241
+
let* () = Bos.OS.File.write (Fpath.v packages_file) packages_json in
242
+
243
+
let results_dir = Filename.concat output_dir "results" in
244
+
let* _ = Bos.OS.Dir.create (Fpath.v results_dir) in
245
+
246
+
(* Build repo args: overlay repos first (highest priority), then opam-repository *)
247
+
let repo_args = List.concat_map (fun path ->
248
+
["--opam-repository"; path]
249
+
) overlay_repos in
250
+
251
+
(* Stage 1: dry-run with high parallelism to solve dependencies *)
252
+
let args_stage1 = [
253
+
"day10"; "health-check";
254
+
] @ repo_args @ [
255
+
"--opam-repository"; opam_repo_path;
256
+
"--cache-dir"; cache_dir;
257
+
"--os"; os;
258
+
"--os-family"; os_family;
259
+
"--os-distribution"; os_distribution;
260
+
"--os-version"; os_version;
261
+
"--dry-run";
262
+
"--fork"; string_of_int fork_jobs;
263
+
"--json"; results_dir;
264
+
"@" ^ packages_file;
265
+
] in
266
+
let* _ = run_cmd args_stage1 in
267
+
268
+
(* Parse stage 1 results *)
269
+
let* results_stage1 = parse_results_dir results_dir in
270
+
271
+
if dry_run then
272
+
(* If --dry-run, we're done after stage 1 *)
273
+
Ok results_stage1
274
+
else begin
275
+
(* Stage 2: find packages with status "solution" and actually build them *)
276
+
let solution_packages = results_stage1
277
+
|> List.filter (fun r -> r.status = Solution)
278
+
|> List.map (fun r -> r.name)
279
+
in
280
+
281
+
if solution_packages = [] then
282
+
(* No packages need building *)
283
+
Ok results_stage1
284
+
else begin
285
+
Logs.info (fun m -> m "Stage 2: building %d packages" (List.length solution_packages));
286
+
287
+
(* Create packages file for stage 2 *)
288
+
let packages_file2 = Filename.concat output_dir "packages_stage2.json" in
289
+
let packages_json2 = Printf.sprintf {|{"packages":[%s]}|}
290
+
(String.concat "," (List.map (Printf.sprintf {|"%s"|}) solution_packages))
291
+
in
292
+
let* () = Bos.OS.File.write (Fpath.v packages_file2) packages_json2 in
293
+
294
+
(* Stage 2: no --dry-run, no --fork (sequential builds) *)
295
+
let args_stage2 = [
296
+
"day10"; "health-check";
297
+
] @ repo_args @ [
298
+
"--opam-repository"; opam_repo_path;
299
+
"--cache-dir"; cache_dir;
300
+
"--os"; os;
301
+
"--os-family"; os_family;
302
+
"--os-distribution"; os_distribution;
303
+
"--os-version"; os_version;
304
+
"--json"; results_dir;
305
+
"@" ^ packages_file2;
306
+
] in
307
+
let* _ = run_cmd args_stage2 in
308
+
309
+
(* Parse final results (stage 2 overwrites stage 1 results for built packages) *)
310
+
parse_results_dir results_dir
311
+
end
312
+
end
313
+
314
+
(** Run merge test on stacked repositories *)
315
+
let merge_test ~overlay_repos ~opam_repo_path ~cache_dir ~output_dir
316
+
~os ~os_family ~os_distribution ~os_version ~fork_jobs ~dry_run =
317
+
318
+
(* List packages from overlay repos only (not opam-repository) *)
319
+
let* packages = list_packages_multi ~repo_paths:overlay_repos ~os ~os_family ~os_distribution ~os_version in
320
+
321
+
Logs.info (fun m -> m "Found %d packages across %d overlay repos" (List.length packages) (List.length overlay_repos));
322
+
323
+
if packages = [] then begin
324
+
Logs.warn (fun m -> m "No packages found in overlay repos");
325
+
let generated_at =
326
+
let t = Unix.gettimeofday () in
327
+
let tm = Unix.gmtime t in
328
+
Printf.sprintf "%04d-%02d-%02dT%02d:%02d:%02dZ"
329
+
(tm.Unix.tm_year + 1900) (tm.Unix.tm_mon + 1) tm.Unix.tm_mday
330
+
tm.Unix.tm_hour tm.Unix.tm_min tm.Unix.tm_sec
331
+
in
332
+
let manifest = {
333
+
repo_path = List.hd overlay_repos;
334
+
opam_repo_path;
335
+
os = Printf.sprintf "%s-%s" os_distribution os_version;
336
+
os_version;
337
+
generated_at;
338
+
commits = ["merge-test"];
339
+
packages = [];
340
+
results = [{ commit = "merge-test"; short_commit = "merge-q"; message = "Merge queue snapshot"; packages = [] }];
341
+
mode = "merge-test";
342
+
overlay_repos;
343
+
} in
344
+
let* _ = Bos.OS.Dir.create (Fpath.v output_dir) in
345
+
let manifest_path = Filename.concat output_dir "manifest.json" in
346
+
let* () = Json.write_manifest manifest_path manifest in
347
+
Ok manifest
348
+
end else begin
349
+
(* Create output directory *)
350
+
let* _ = Bos.OS.Dir.create (Fpath.v output_dir) in
351
+
352
+
(* Run health check with all overlay repos stacked *)
353
+
let* results = health_check_multi
354
+
~overlay_repos ~opam_repo_path ~cache_dir
355
+
~os ~os_family ~os_distribution ~os_version
356
+
~fork_jobs ~dry_run ~output_dir ~packages
357
+
in
358
+
359
+
(* Sort results by package name *)
360
+
let sorted_results = List.sort (fun a b -> String.compare a.name b.name) results in
361
+
362
+
(* Build manifest *)
363
+
let generated_at =
364
+
let t = Unix.gettimeofday () in
365
+
let tm = Unix.gmtime t in
366
+
Printf.sprintf "%04d-%02d-%02dT%02d:%02d:%02dZ"
367
+
(tm.Unix.tm_year + 1900) (tm.Unix.tm_mon + 1) tm.Unix.tm_mday
368
+
tm.Unix.tm_hour tm.Unix.tm_min tm.Unix.tm_sec
369
+
in
370
+
371
+
let all_packages = sorted_results
372
+
|> List.map (fun p -> p.name)
373
+
|> List.sort_uniq String.compare
374
+
in
375
+
376
+
let commit_result = {
377
+
commit = "merge-test";
378
+
short_commit = "merge-q";
379
+
message = "Merge queue snapshot";
380
+
packages = sorted_results;
381
+
} in
382
+
383
+
let manifest = {
384
+
repo_path = List.hd overlay_repos;
385
+
opam_repo_path;
386
+
os = Printf.sprintf "%s-%s" os_distribution os_version;
387
+
os_version;
388
+
generated_at;
389
+
commits = ["merge-test"];
390
+
packages = all_packages;
391
+
results = [commit_result];
392
+
mode = "merge-test";
393
+
overlay_repos;
394
+
} in
395
+
396
+
(* Write manifest *)
397
+
let manifest_path = Filename.concat output_dir "manifest.json" in
398
+
let* () = Json.write_manifest manifest_path manifest in
399
+
400
+
Logs.info (fun m -> m "Manifest written to %s" manifest_path);
401
+
402
+
Ok manifest
403
+
end
+19
braid/lib/server.ml
+19
braid/lib/server.ml
···
1
+
(** Cap'n Proto RPC server for BraidService *)
2
+
3
+
(** Start the RPC server *)
4
+
let run ~sw ~net ~fs ~listen_addr ~listen_port ~public_addr ~key_file ~cap_file ~opam_repo_path ~cache_dir =
5
+
let service = Rpc_service.local ~opam_repo_path ~cache_dir in
6
+
let addr = `TCP (listen_addr, listen_port) in
7
+
let public_address = `TCP (public_addr, listen_port) in
8
+
let secret_key = `File (Eio.Path.(fs / key_file)) in
9
+
let config = Capnp_rpc_unix.Vat_config.create ~secret_key ~public_address ~net addr in
10
+
let service_id = Capnp_rpc_unix.Vat_config.derived_id config "main" in
11
+
let restore = Capnp_rpc_net.Restorer.single service_id service in
12
+
let vat = Capnp_rpc_unix.serve ~sw ~restore config in
13
+
Capnp_rpc_unix.Cap_file.save_service vat service_id cap_file |> Result.get_ok;
14
+
Fmt.pr "Server listening on %s:%d@." listen_addr listen_port;
15
+
Fmt.pr " Public address: %s:%d@." public_addr listen_port;
16
+
Fmt.pr " Key file: %s@." key_file;
17
+
Fmt.pr " Capability file: %s@." cap_file;
18
+
(* Block forever - server runs until killed *)
19
+
Eio.Fiber.await_cancel ()
+73
braid/lib/types.ml
+73
braid/lib/types.ml
···
1
+
(** Core types for braid *)
2
+
3
+
type status =
4
+
| Success
5
+
| Failure
6
+
| Dependency_failed
7
+
| No_solution
8
+
| Solution (* solvable but not yet built *)
9
+
| Error
10
+
11
+
let status_of_string = function
12
+
| "success" -> Success
13
+
| "failure" -> Failure
14
+
| "dependency_failed" -> Dependency_failed
15
+
| "no_solution" -> No_solution
16
+
| "solution" -> Solution
17
+
| _ -> Error
18
+
19
+
let string_of_status = function
20
+
| Success -> "success"
21
+
| Failure -> "failure"
22
+
| Dependency_failed -> "dependency_failed"
23
+
| No_solution -> "no_solution"
24
+
| Solution -> "solution"
25
+
| Error -> "error"
26
+
27
+
let status_symbol = function
28
+
| Success -> "S"
29
+
| Failure -> "F"
30
+
| Dependency_failed -> "D"
31
+
| No_solution -> "-"
32
+
| Solution -> "B"
33
+
| Error -> " "
34
+
35
+
(** Result of a single package build/check *)
36
+
type package_result = {
37
+
name : string;
38
+
status : status;
39
+
sha : string option;
40
+
layer : string option;
41
+
log : string option;
42
+
solution : string option; (* dependency graph in dot format *)
43
+
}
44
+
45
+
(** Results for a single commit *)
46
+
type commit_result = {
47
+
commit : string; (* full sha *)
48
+
short_commit : string; (* 7-char prefix *)
49
+
message : string;
50
+
packages : package_result list;
51
+
}
52
+
53
+
(** Package history across commits *)
54
+
type package_history = {
55
+
package : string;
56
+
first_seen : string; (* commit where first appeared *)
57
+
latest_status : status;
58
+
history : (string * status) list; (* commit, status pairs, newest first *)
59
+
}
60
+
61
+
(** Summary manifest for the entire run *)
62
+
type manifest = {
63
+
repo_path : string;
64
+
opam_repo_path : string;
65
+
os : string;
66
+
os_version : string;
67
+
generated_at : string;
68
+
commits : string list; (* newest first *)
69
+
packages : string list; (* sorted alphabetically *)
70
+
results : commit_result list;
71
+
mode : string; (* "history" for run command, "merge-test" for merge-test command *)
72
+
overlay_repos : string list; (* for merge-test: list of stacked repos in priority order *)
73
+
}