forked from
anil.recoil.org/monopam-myspace
My aggregated monorepo of OCaml code, automaintained
Walkthough
+486
+486
day10/docs/WALKTHROUGH.md
+486
day10/docs/WALKTHROUGH.md
···
1
+
# day10: Code Walkthrough
2
+
3
+
This document walks through the changes to day10 since commit `e76b3395` —
4
+
roughly 3,600 lines of new code across 16 commits. The work transforms day10
5
+
from a batch builder into something closer to a queryable build service with
6
+
history tracking, failure recovery, and incremental cascade rebuilds.
7
+
8
+
The base URL for source links is:
9
+
`https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10`
10
+
11
+
---
12
+
13
+
## Table of Contents
14
+
15
+
1. [New Library Modules](#1-new-library-modules)
16
+
2. [Build Failure Classification & History Recording](#2-build-failure-classification--history-recording)
17
+
3. [Status Generation](#3-status-generation)
18
+
4. [DAG Executor](#4-dag-executor)
19
+
5. [CLI Commands](#5-cli-commands)
20
+
6. [Infrastructure: Logging, Races, and Robustness](#6-infrastructure-logging-races-and-robustness)
21
+
7. [Utility Consolidation](#7-utility-consolidation)
22
+
8. [On-Disk Structure](#8-on-disk-structure)
23
+
24
+
---
25
+
26
+
## 1. New Library Modules
27
+
28
+
Four new modules were added to `day10_lib`:
29
+
30
+
### History ([lib/history.ml](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/history.ml))
31
+
32
+
Per-package append-only build history, stored as JSONL files at
33
+
`packages/{pkg}/history.jsonl`. Each entry records a single build attempt:
34
+
35
+
```ocaml
36
+
type entry = {
37
+
ts : string; (* ISO 8601 timestamp *)
38
+
run : string; (* Run identifier, e.g. "2026-03-09-125444" *)
39
+
build_hash : string; (* Content-addressed layer hash *)
40
+
status : string; (* "success" or "failure" *)
41
+
category : string; (* e.g. "success", "build_failure", "dependency_failure" *)
42
+
compiler : string; (* OCaml version *)
43
+
blessed : bool; (* Whether this is the canonical build *)
44
+
error : string option; (* Error description *)
45
+
failed_dep : string option; (* Package name of the failed dependency *)
46
+
failed_dep_hash : string option; (* Build hash of the failed dependency *)
47
+
}
48
+
```
49
+
50
+
Key functions:
51
+
52
+
- [`append`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/history.ml#L83) —
53
+
Appends an entry with `Unix.lockf F_LOCK` file locking, safe for concurrent
54
+
forked processes writing to the same package's history.
55
+
56
+
- [`read_latest`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/history.ml#L114) —
57
+
Returns the most recent entry per `build_hash`, deduplicating across runs.
58
+
Used by the status command and status index generation.
59
+
60
+
- [`compact`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/history.ml#L147) —
61
+
Compresses old history by collapsing consecutive same-status, same-hash
62
+
entries older than `max_age_days` down to first + last. Keeps the file from
63
+
growing without bound.
64
+
65
+
### Status Index ([lib/status_index.ml](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/status_index.ml))
66
+
67
+
Global build status snapshot, written to `status.json` after each run:
68
+
69
+
```ocaml
70
+
type t = {
71
+
generated : string;
72
+
run_id : string;
73
+
blessed_totals : (string * int) list; (* category -> count for blessed builds *)
74
+
non_blessed_totals : (string * int) list; (* category -> count for non-blessed *)
75
+
changes : change list; (* status transitions since last run *)
76
+
new_packages : string list;
77
+
}
78
+
```
79
+
80
+
[`generate`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/status_index.ml#L130)
81
+
scans all package directories, reads their history files, tallies totals by
82
+
category, and detects changes by comparing entries from the current `run_id`
83
+
against entries from previous runs.
84
+
85
+
### GC ([lib/gc.ml](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/gc.ml))
86
+
87
+
Garbage collection for the build cache. Three levels:
88
+
89
+
- [`gc_layers`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/gc.ml#L66) —
90
+
Deletes `build-*`, `doc-*`, `jtw-*` layer directories not referenced by any
91
+
current solution. Protected layers (`base/`, `packages/`, `solutions/`,
92
+
`logs/`, tool layers) are
93
+
[never deleted](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/gc.ml#L53).
94
+
95
+
- [`gc_universes`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/gc.ml#L165) —
96
+
Deletes empty universe directories not referenced by any doc output.
97
+
Conservative: preserves universes that still contain package documentation.
98
+
99
+
- [`gc_logs`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/gc.ml#L230) —
100
+
Compacts per-package history files and manages old run directories (archive
101
+
or delete based on `keep_runs` threshold).
102
+
103
+
### Notify ([lib/notify.ml](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/lib/notify.ml))
104
+
105
+
Simple notification dispatch to Slack, Zulip, Telegram, Email, or stdout.
106
+
Each channel reads its configuration from environment variables. Used by the
107
+
`notify` CLI command.
108
+
109
+
---
110
+
111
+
## 2. Build Failure Classification & History Recording
112
+
113
+
When a build completes, two things happen before moving on:
114
+
115
+
### Classification ([bin/main.ml#L341](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L341))
116
+
117
+
`classify_build_failure` scans the build log for known error patterns and
118
+
assigns a category:
119
+
120
+
```ocaml
121
+
let classify_build_failure build_log_path =
122
+
let log_content = try Os.read_from_file build_log_path with _ -> "" in
123
+
let transient_patterns = [
124
+
"No space left on device"; "Connection timed out";
125
+
"Could not resolve host"; ...
126
+
] in
127
+
let depext_patterns = [
128
+
"Unable to locate package"; "is not available";
129
+
"unmet dependencies"; ...
130
+
] in
131
+
if matches_any transient_patterns log_content then
132
+
("failure", "transient_failure", Some "Transient infrastructure failure")
133
+
else if matches_any depext_patterns log_content then
134
+
("failure", "depext_unavailable", Some "Missing system dependency")
135
+
else
136
+
("failure", "build_failure", None)
137
+
```
138
+
139
+
Pattern matching uses
140
+
[`contains_substring_ci`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L308) —
141
+
a pure-OCaml case-insensitive substring search, replacing the earlier
142
+
`Str.regexp` approach which had thread-safety issues (the `Str` module uses
143
+
global mutable state).
144
+
145
+
### Recording ([bin/main.ml#L286](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L286))
146
+
147
+
`record_build_result` appends a history entry, deduplicating within a run via
148
+
an in-memory hashtable:
149
+
150
+
```ocaml
151
+
let recorded_this_run : (string, bool) Hashtbl.t = Hashtbl.create 1024
152
+
153
+
let record_build_result ~packages_dir ~run_id ~pkg_str ~build_hash
154
+
~status ~category ~compiler ~blessed ~error ~failed_dep ~failed_dep_hash =
155
+
let key = Printf.sprintf "%s:%s" pkg_str build_hash in
156
+
if not (Hashtbl.mem recorded_this_run key) then begin
157
+
Hashtbl.replace recorded_this_run key true;
158
+
let entry = { ts = ...; run = run_id; build_hash; status;
159
+
category; compiler; blessed; error;
160
+
failed_dep; failed_dep_hash } in
161
+
History.append ~packages_dir ~pkg_str entry
162
+
end
163
+
```
164
+
165
+
### Root-cause tracking ([bin/main.ml#L1746](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L1746))
166
+
167
+
When a package fails due to a dependency failure, `find_root_failure` walks the
168
+
dependency graph transitively to find the actual build that broke:
169
+
170
+
```ocaml
171
+
let rec find_root_failure solution pkg_hashes pkg visited =
172
+
(* ... walks dep graph to find the package that actually failed to build,
173
+
rather than recording the immediate dep that was skipped *)
174
+
```
175
+
176
+
The result is stored in the history entry's `failed_dep` and `failed_dep_hash`
177
+
fields, so queries can immediately show *why* a package failed without
178
+
re-walking the graph.
179
+
180
+
---
181
+
182
+
## 3. Status Generation
183
+
184
+
[`print_batch_summary`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L1568)
185
+
runs after every batch, rerun, and cascade. It:
186
+
187
+
1. Iterates all solutions and their packages
188
+
2. Records build results to per-package history
189
+
3. Classifies failures by scanning build logs
190
+
4. Walks dep graphs for root-cause attribution
191
+
5. Generates `status.json` via `Status_index.generate`
192
+
193
+
This is the integration point where the build pipeline meets the
194
+
history/status system.
195
+
196
+
---
197
+
198
+
## 4. DAG Executor
199
+
200
+
The biggest single addition. When `--fork N` is passed to `batch`, instead of
201
+
building packages sequentially per-solution, day10 builds a *global* DAG of
202
+
all unique build layers across all solutions and executes them in parallel.
203
+
204
+
### Build nodes ([bin/main.ml#L1147](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L1147))
205
+
206
+
```ocaml
207
+
type build_node = {
208
+
pkg : OpamPackage.t;
209
+
build_hash : string; (* "build-{hash}" *)
210
+
ordered_deps : OpamPackage.t list;
211
+
dep_build_hashes : string list; (* build hashes of deps, in order *)
212
+
}
213
+
```
214
+
215
+
### DAG construction ([bin/main.ml#L1157](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L1157))
216
+
217
+
`build_global_dag` walks all solutions, computes build hashes for each
218
+
package, deduplicates by hash (since the same package with the same deps
219
+
produces the same hash regardless of which solution requested it), then
220
+
topologically sorts the result.
221
+
222
+
### Execution ([bin/main.ml#L1219](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L1219))
223
+
224
+
`execute_dag` manages the parallel build loop. Its internal structure was
225
+
refactored into four composable helpers:
226
+
227
+
```ocaml
228
+
let execute_dag ~np ~on_complete ?on_cascade ~cache_dir ~os_key nodes build_one =
229
+
(* ... setup ... *)
230
+
231
+
(* Mark a node as cascade-failed and notify callbacks *)
232
+
let cascade_fail ~failed_hash ~failed_dep_hash = ... in
233
+
234
+
(* Recursively propagate failure to all transitive dependents *)
235
+
let rec propagate_failure failed_dep_hash = ... in
236
+
237
+
(* After a node completes, promote or cascade-fail its dependents *)
238
+
let promote_dependents hash = ... in
239
+
240
+
(* Record a node's completion and promote its dependents *)
241
+
let complete_node hash success = ... in
242
+
```
243
+
244
+
The main loop pops nodes from a ready queue, checks for cached layers (instant
245
+
resolution without forking), forks workers for uncached layers, and reaps
246
+
completed children. Cache hits and failures are handled identically via
247
+
`complete_node`, which calls `promote_dependents` to either enqueue dependents
248
+
or cascade-fail them.
249
+
250
+
**Cascade failure propagation**: When a build fails, `promote_dependents`
251
+
checks each reverse dependent's deps. If any dep failed, the dependent is
252
+
immediately marked as failed via `cascade_fail` (no fork, no build). Then
253
+
`propagate_failure` recursively walks the reverse-dependency graph, marking
254
+
all transitive dependents as failed.
255
+
256
+
**The `on_cascade` callback**: Passed by the caller, it fires for every
257
+
cascade-failed node with the `failed_hash` and `failed_dep_hash`. The
258
+
[batch caller](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L1960)
259
+
uses this to:
260
+
261
+
1. Log the cascade:
262
+
`dag: CASCADE lwt-ssl.1.2.0 (build-abc) — dep lwt.6.1.1 (build-def) failed`
263
+
2. Write a skeleton layer via
264
+
[`Util.write_skeleton_layer`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/util.ml#L239)
265
+
266
+
**Forked children use `Unix._exit`**
267
+
([line 1317](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L1317))
268
+
instead of `exit` to avoid running OCaml's `at_exit` handlers, which could
269
+
flush the parent's partially-written stdio buffers.
270
+
271
+
---
272
+
273
+
## 5. CLI Commands
274
+
275
+
All commands are wired up via cmdliner at the bottom of `main.ml`. All support
276
+
`--format json` for machine consumption.
277
+
278
+
### status ([bin/main.ml#L2409](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L2409))
279
+
280
+
Displays the build status overview from `status.json`. With `--details`,
281
+
breaks down failures by category with package lists.
282
+
283
+
### query ([bin/main.ml#L2569](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L2569))
284
+
285
+
Shows detailed build information for a specific package: current status, build
286
+
hash, compiler version, history of status changes across runs. With `--log`,
287
+
displays the build log inline.
288
+
289
+
### failures ([bin/main.ml#L2648](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L2648))
290
+
291
+
Lists all packages with failing builds. Supports `--blessed-only` to filter
292
+
to canonical builds and `--category` to filter by failure type
293
+
(`build_failure`, `dependency_failure`, `transient_failure`, etc.).
294
+
295
+
### changes ([bin/main.ml#L2696](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L2696))
296
+
297
+
Shows status transitions since the last run (e.g. `success → failure`). Reads
298
+
the `changes_since_last` field from `status.json`.
299
+
300
+
### disk ([bin/main.ml#L2743](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L2743))
301
+
302
+
Reports disk usage breakdown: base image, build layers, doc layers, packages
303
+
metadata, logs, solutions.
304
+
305
+
### rerun ([bin/main.ml#L2910](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L2910))
306
+
307
+
Retries a failed build. Accepts a build hash or package name. The key design
308
+
choice: reruns use the opam files stored in the layer's own
309
+
`opam-repository/` directory, so **no external opam-repository path is
310
+
needed**. The rebuild uses the exact same opam files as the original build.
311
+
312
+
The [`rerun_build_layer`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L2842)
313
+
function reads the skeleton layer's `layer.json` for the dep list, points the
314
+
solver at the layer's embedded `opam-repository/`, and rebuilds.
315
+
316
+
With `--cascade`, after the rerun succeeds, it finds all packages that
317
+
recorded a `dependency_failure` pointing at this build hash and reruns
318
+
them too.
319
+
320
+
### cascade ([bin/main.ml#L3107](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L3107))
321
+
322
+
Bulk cascade: scans all packages for `dependency_failure` entries, checks if
323
+
the failing dependency now succeeds, and reruns everything that's unblocked.
324
+
Supports `--blessed-first`, `--dry-run`, and `--fork N`.
325
+
326
+
[`find_cascade_targets`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L2811)
327
+
does the scan: for each package with a `dependency_failure` history entry, it
328
+
checks whether the `failed_dep_hash` layer now has `exit_status = 0`.
329
+
330
+
### rdeps ([bin/main.ml#L3007](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L3007))
331
+
332
+
Finds reverse dependencies of a package by scanning cached solutions. With
333
+
`--failing`, filters to rdeps that are currently failing.
334
+
335
+
### gc ([bin/main.ml#L3226](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L3226))
336
+
337
+
Garbage collects logs, old run directories, and compacts history files.
338
+
Supports `--archive`, `--keep-runs`, `--stable-threshold`, `--dry-run`.
339
+
340
+
### universe ([bin/main.ml#L3276](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L3276))
341
+
342
+
Looks up packages in a universe by hash (or prefix). Without arguments, lists
343
+
all universes with package counts.
344
+
345
+
### log ([bin/main.ml#L3374](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L3374))
346
+
347
+
Displays the build or doc log for a specific layer hash, with metadata.
348
+
349
+
### notify ([bin/main.ml#L3084](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/main.ml#L3084))
350
+
351
+
Sends a message to an external channel (Slack, Zulip, Telegram, Email,
352
+
stdout).
353
+
354
+
---
355
+
356
+
## 6. Infrastructure: Logging, Races, and Robustness
357
+
358
+
### Per-process logging ([bin/os.ml](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/os.ml))
359
+
360
+
`Os.log` writes timestamped messages to `logs/{pid}.log`. This is critical for
361
+
`--fork` mode where multiple processes are running concurrently — each gets
362
+
its own log file keyed by PID. The `sudo` and `exec` functions log their
363
+
commands and exit codes.
364
+
365
+
### Container command logging ([bin/linux.ml](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/linux.ml))
366
+
367
+
The `build` function logs the full container command (runc argv) before
368
+
execution, so build failures can be reproduced.
369
+
370
+
### mkdir race fix ([bin/os.ml](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/os.ml#L74))
371
+
372
+
```ocaml
373
+
let rec mkdir ?(parents = false) dir =
374
+
if not (Sys.file_exists dir) then (
375
+
(if parents then ...);
376
+
try Sys.mkdir dir 0o755
377
+
with Sys_error _ when Sys.file_exists dir && Sys.is_directory dir -> ())
378
+
```
379
+
380
+
The original `Sys.mkdir` would raise `Sys_error` if another forked worker
381
+
created the directory between the `file_exists` check and the `mkdir` call.
382
+
This race caused crashes when multiple workers built different variants of the
383
+
same package (e.g. 8 workers building `lwt.6.1.1` concurrently), which then
384
+
cascaded to ~370 dependent packages. The fix catches the error and verifies
385
+
the directory exists.
386
+
387
+
### EINTR handling in fork functions
388
+
389
+
The `fork`, `fork_with_progress`, and `fork_map` functions now catch
390
+
`Unix.Unix_error(EINTR, _, _)` from `waitpid`, which occurs when system calls
391
+
are interrupted by signals. Previously, this could crash the parent process.
392
+
393
+
### Base image hash invalidation ([bin/linux.ml](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/linux.ml))
394
+
395
+
`base_hash` computes a hash from the OS distribution, version, architecture,
396
+
and opam-build source hash. `layer_hash` now includes the base hash, so when
397
+
the base container image changes (new OS version, updated opam-build), all
398
+
cached layers are automatically invalidated.
399
+
400
+
---
401
+
402
+
## 7. Utility Consolidation
403
+
404
+
Three helpers were extracted into
405
+
[`bin/util.ml`](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/util.ml)
406
+
to eliminate code that was duplicated 2-3 times across the build, skeleton,
407
+
and DAG paths:
408
+
409
+
### populate_opam_repository ([util.ml#L217](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/util.ml#L217))
410
+
411
+
Copies opam files for a list of packages into an opam-repository directory,
412
+
searching the configured repositories in order. Copies both the `opam` file
413
+
and the optional `files/` subdirectory. Previously inlined in three places.
414
+
415
+
### write_skeleton_layer ([util.ml#L239](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/util.ml#L239))
416
+
417
+
Creates a skeleton layer for a cascade-failed package:
418
+
419
+
```ocaml
420
+
let write_skeleton_layer ~cache_dir ~os_key ~opam_repositories
421
+
~layer_name ~pkg ~ordered_deps ~dep_build_hashes =
422
+
let layer_dir = Path.(cache_dir / os_key / layer_name) in
423
+
if not (Sys.file_exists layer_dir) then begin
424
+
Os.mkdir ~parents:true layer_dir;
425
+
save_layer_info Path.(layer_dir / "layer.json")
426
+
pkg ordered_deps dep_build_hashes (-1);
427
+
let opam_repo = create_opam_repository layer_dir in
428
+
populate_opam_repository ~opam_repo ~opam_repositories
429
+
(pkg :: ordered_deps);
430
+
ensure_package_layer_symlink ~cache_dir ~os_key
431
+
~pkg_str:(OpamPackage.to_string pkg) ~layer_name
432
+
end
433
+
```
434
+
435
+
The `exit_status = -1` sentinel marks the layer as "never built". The embedded
436
+
`opam-repository/` means `rerun` and `cascade` can rebuild without needing an
437
+
external opam-repository path. Previously inlined in two places (sequential
438
+
build path and DAG `on_cascade` callback).
439
+
440
+
### wait_for_layer_json ([util.ml#L251](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/bin/util.ml#L251))
441
+
442
+
Polls for a `layer.json` file to appear (created by a parallel worker), with
443
+
0.5s intervals for up to 5 minutes. Also updates the file's timestamp.
444
+
Previously inlined in three places (build, doc, and jtw layer functions).
445
+
446
+
---
447
+
448
+
## 8. On-Disk Structure
449
+
450
+
The [on-disk structure doc](https://tangled.org/jon.recoil.org/monopam-myspace/blob/main/day10/docs/ON_DISK_STRUCTURE.md)
451
+
describes the full layout. The key design properties:
452
+
453
+
**Content-addressable layers.** Build hashes are computed from the base image
454
+
hash plus the opam file contents of every dependency. Identical dependency sets
455
+
always produce the same hash, enabling cross-solution cache reuse. When the
456
+
base image changes, all hashes change and stale layers are naturally
457
+
invalidated.
458
+
459
+
**Self-contained layers.** Each layer directory contains everything needed to
460
+
rebuild it: `layer.json` (metadata, dep list, dep hashes), `opam-repository/`
461
+
(exact opam files), `build.log`, and `fs/` (filesystem delta). The `rerun` and
462
+
`cascade` commands exploit this: they read the skeleton layer's embedded opam
463
+
files rather than requiring an external opam-repository path.
464
+
465
+
**Append-only history.** Per-package `history.jsonl` files provide a full
466
+
audit trail of build results across runs. The `compact` operation prevents
467
+
unbounded growth by collapsing old consecutive same-status entries.
468
+
469
+
**Atomic writes.** `status.json` and compacted history files are written via
470
+
temp file + rename. History appends use file locking. Layer directories are
471
+
created with exclusive locking via `create_directory_exclusively`.
472
+
473
+
```
474
+
<cache-dir>/
475
+
├── solutions/<opam-repo-sha>/<pkg>.json
476
+
├── logs/{<pid>.log, runs/<run-id>/{summary.json, build/, docs/}}
477
+
└── <os-key>/
478
+
├── build-config.json
479
+
├── status.json
480
+
├── base/{Dockerfile, fs/, build.log, base.hash}
481
+
├── build-<hash>/{layer.json, build.log, fs/, opam-repository/}
482
+
├── doc-<hash>/{layer.json, odoc-voodoo-all.log}
483
+
├── jtw-<hash>/{layer.json, jtw.log}
484
+
├── universes/<hash>.json
485
+
└── packages/<pkg>/{history.jsonl, build-<hash>→, blessed-build→}
486
+
```