forked from
anil.recoil.org/monopam-myspace
My aggregated monorepo of OCaml code, automaintained
Add gc, report commands and daily build operator instructions
- gc: disk usage overview, stale mount cleanup, solution cache GC
(keep 3), run file GC (keep 30), orphaned layer detection
- report: generates markdown daily summary with build stats, repo
SHAs, diff vs previous run, saved to reports/YYYY-MM-DD.md
- CLAUDE-daily.md: complete operator instructions for the daily
build cron job — housekeeping, repo updates, solve, build,
analyse, fix, report
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
+461
-12
+166
CLAUDE-daily.md
+166
CLAUDE-daily.md
···
1
+
# Daily Build Operator
2
+
3
+
You are the daily build operator for the OCaml package health checker (day11).
4
+
Your job is to keep the system running, diagnose failures, and fix what you can.
5
+
6
+
## Environment
7
+
8
+
- **Cache directory:** `~/cache-day11-ox2`
9
+
- **Repos:**
10
+
- `~/ocaml/opam-repository` — upstream opam-repository
11
+
- `~/oxcaml/opam-repository` — oxcaml overlay
12
+
- `~/local/opam-repository` — local fixes (this is where you commit fixes)
13
+
- **Compiler:** `ocaml-variants.5.2.0+ox`
14
+
- **Tools:** `day11` CLI (built from `~/monopam-myspace`)
15
+
16
+
## Daily Procedure
17
+
18
+
### 1. Housekeeping
19
+
20
+
Check the machine is healthy before starting the build.
21
+
22
+
```bash
23
+
# Check disk space
24
+
df -h /home
25
+
du -sh ~/cache-day11-ox2
26
+
27
+
# Check for zombie processes
28
+
ps aux | grep -c runc
29
+
cat /proc/loadavg
30
+
31
+
# Run GC
32
+
dune exec -- day11 gc --cache-dir ~/cache-day11-ox2
33
+
34
+
# Check for stale mounts (gc handles this, but verify)
35
+
mount | grep day11_run
36
+
```
37
+
38
+
If disk is above 80% full, the GC should free space by deleting old solution
39
+
caches and unreferenced layers. If still tight, consider deleting older run
40
+
files or the merged-repo cache.
41
+
42
+
### 2. Update Repositories
43
+
44
+
```bash
45
+
cd ~/ocaml/opam-repository && git pull
46
+
cd ~/oxcaml/opam-repository && git pull
47
+
```
48
+
49
+
Do NOT pull `~/local/opam-repository` — that's managed by you (the operator).
50
+
Check if any of your local fixes have been adopted upstream:
51
+
52
+
```bash
53
+
# For each package in local/opam-repository/packages/,
54
+
# check if the ox repo now has the same fix
55
+
ls ~/local/opam-repository/packages/
56
+
```
57
+
58
+
If a fix is now upstream, remove it from the local repo and commit.
59
+
60
+
### 3. Solve
61
+
62
+
```bash
63
+
dune exec -- day11 batch \
64
+
--opam-repository ~/ocaml/opam-repository \
65
+
--opam-repository ~/oxcaml/opam-repository \
66
+
--opam-repository ~/local/opam-repository \
67
+
--cache-dir ~/cache-day11-ox2 \
68
+
--ocaml-version ocaml-variants.5.2.0+ox \
69
+
--solve-only -j 16
70
+
```
71
+
72
+
Check solver results before building:
73
+
```bash
74
+
dune exec -- day11 results --cache-dir ~/cache-day11-ox2
75
+
```
76
+
77
+
### 4. Build
78
+
79
+
```bash
80
+
dune exec -- day11 batch \
81
+
--opam-repository ~/ocaml/opam-repository \
82
+
--opam-repository ~/oxcaml/opam-repository \
83
+
--opam-repository ~/local/opam-repository \
84
+
--cache-dir ~/cache-day11-ox2 \
85
+
--ocaml-version ocaml-variants.5.2.0+ox \
86
+
--rebuild-failed -j 16
87
+
```
88
+
89
+
Monitor for stuck processes — if load average spikes or runc processes
90
+
accumulate, the build may need intervention.
91
+
92
+
### 5. Analyse and Fix
93
+
94
+
```bash
95
+
dune exec -- day11 results --cache-dir ~/cache-day11-ox2
96
+
dune exec -- day11 report --cache-dir ~/cache-day11-ox2
97
+
```
98
+
99
+
For each **new** failure (not in yesterday's run):
100
+
101
+
1. **Read the log** — find the layer hash from `results`, then:
102
+
```bash
103
+
cat ~/cache-day11-ox2/debian-bookworm-x86_64/BUILD_HASH/layer.log
104
+
```
105
+
106
+
2. **Classify the error:**
107
+
- **Depopt constraint too loose** — tighten in local-fixes opam file
108
+
- **Ox AST change** (Pexp_let 4 args, Ptyp_tuple labeled, Ppat_constraint 3 args,
109
+
pexp_function_cases → pexp_function) — create patch
110
+
- **GADT existential escape** ($a would escape) — split or-patterns
111
+
- **`@ local` annotation mismatch** — stdlib signatures changed in ox, needs
112
+
`include module type of` workarounds
113
+
- **`value_or_null` kind mismatch** — add kind annotation
114
+
- **conf-* system dep** — expected, ignore
115
+
- **Unknown** — note in report, don't attempt fix
116
+
117
+
3. **For fixable errors**, test in debug container:
118
+
```bash
119
+
dune exec -- day11 debug \
120
+
--os-dir ~/cache-day11-ox2/debian-bookworm-x86_64 \
121
+
--keep -c "COMMAND" LAYER_HASH
122
+
# Check output:
123
+
cat ~/cache-day11-ox2/debug-HASH/runc.log
124
+
```
125
+
126
+
4. **If fix works**, create it in `~/local/opam-repository`:
127
+
- Copy the upstream opam file
128
+
- Add `patches:` and `extra-files:` entries
129
+
- Put the patch file in `files/`
130
+
- Get the sha256: `sha256sum FILES/patch.patch`
131
+
- Commit: `cd ~/local/opam-repository && git add -A && git commit -m "..."`
132
+
133
+
5. **Clean up debug containers:**
134
+
```bash
135
+
sudo umount ~/cache-day11-ox2/debug-*/rootfs 2>/dev/null
136
+
sudo rm -rf ~/cache-day11-ox2/debug-*
137
+
```
138
+
139
+
### 6. Report
140
+
141
+
The report command generates a markdown summary:
142
+
```bash
143
+
dune exec -- day11 report --cache-dir ~/cache-day11-ox2
144
+
```
145
+
146
+
Reports are saved to `~/cache-day11-ox2/reports/YYYY-MM-DD.md`.
147
+
148
+
## Known Ox Incompatibility Patterns
149
+
150
+
| Pattern | Error | Fix |
151
+
|---------|-------|-----|
152
+
| AST constructor arity | `Pexp_let expects 4 arguments` | Add `_` for new arg |
153
+
| Labeled tuples | `Ptyp_tuple` takes labeled list | `List.map snd labeled_typs` |
154
+
| Pattern constraint | `Ppat_constraint` 3 args | Add extra `_` |
155
+
| Function cases | `pexp_function_cases` not found | Use `pexp_function` |
156
+
| GADT escape | `$a would escape its scope` | Split or-pattern |
157
+
| Kind mismatch | `value_or_null` kind | Add `('a : value_or_null)` |
158
+
| Local annotations | `@ local` in type | Library needs ox-specific version |
159
+
| Partial application | `type ... is not compatible` (send/recv) | Eta-expand |
160
+
161
+
## What NOT to Fix
162
+
163
+
- `conf-*` packages (system deps — expected failures)
164
+
- `batteries` / packages with pervasive `@ local` issues (need ox-specific versions)
165
+
- Packages that conflict with the ox compiler (check `conflicts:` in opam file)
166
+
- Packages that need OCaml >= 5.3 (ox is 5.2)
+152
-12
day11/bin/cmd_gc.ml
+152
-12
day11/bin/cmd_gc.ml
···
2
2
3
3
open Cmdliner
4
4
5
-
let run cache_dir os_distribution os_version arch =
6
-
let os_key = Printf.sprintf "%s-%s-%s" os_distribution os_version arch in
7
-
Printf.printf "Running GC for %s...\n%!" os_key;
8
-
let result = Day11_lib.Gc.gc_layers
9
-
~cache_dir ~os_key ~referenced:[] in
10
-
Printf.printf " Referenced: %d\n" (List.length result.referenced);
11
-
Printf.printf " Deleted: %d\n" (List.length result.deleted);
12
-
Printf.printf " Kept: %d\n" (List.length result.kept);
5
+
let run cache_dir =
6
+
let cache_dir = Common.fpath cache_dir in
7
+
Printf.printf "=== Garbage Collection ===\n\n";
8
+
(* 1. Disk usage overview *)
9
+
let du cmd =
10
+
try
11
+
let ic = Unix.open_process_in cmd in
12
+
let line = input_line ic in
13
+
ignore (Unix.close_process_in ic);
14
+
String.trim line
15
+
with _ -> "?"
16
+
in
17
+
Printf.printf "Disk usage:\n";
18
+
Printf.printf " Cache dir: %s\n"
19
+
(du (Printf.sprintf "du -sh %s 2>/dev/null | cut -f1"
20
+
(Fpath.to_string cache_dir)));
21
+
Printf.printf " Filesystem: %s\n\n"
22
+
(du "df -h . 2>/dev/null | tail -1 | awk '{print $4 \" free of \" $2}'");
23
+
(* 2. Clean stale overlay mounts *)
24
+
let tmp = Filename.get_temp_dir_name () in
25
+
let stale_mounts =
26
+
try Sys.readdir tmp |> Array.to_list
27
+
|> List.filter (fun n ->
28
+
String.length n > 10 && String.sub n 0 10 = "day11_run_")
29
+
with _ -> [] in
30
+
if stale_mounts <> [] then begin
31
+
Printf.printf "Cleaning %d stale overlay mounts...\n%!" (List.length stale_mounts);
32
+
List.iter (fun name ->
33
+
let merged = Filename.concat (Filename.concat tmp name) "merged" in
34
+
ignore (Sys.command (Printf.sprintf "sudo umount %s 2>/dev/null" merged))
35
+
) stale_mounts;
36
+
ignore (Sys.command (Printf.sprintf "sudo rm -rf %s"
37
+
(String.concat " " (List.map (Filename.concat tmp) stale_mounts))));
38
+
Printf.printf " Done.\n\n"
39
+
end;
40
+
(* 3. GC old solution caches — keep the 3 most recent *)
41
+
let solutions_dir = Fpath.(cache_dir / "solutions") in
42
+
(match Bos.OS.Dir.contents solutions_dir with
43
+
| Ok dirs ->
44
+
let dirs = dirs
45
+
|> List.filter (fun p -> Bos.OS.Dir.exists p |> Result.get_ok)
46
+
|> List.sort (fun a b ->
47
+
(* Sort by mtime, newest first *)
48
+
let ma = try (Unix.stat (Fpath.to_string a)).Unix.st_mtime
49
+
with _ -> 0.0 in
50
+
let mb = try (Unix.stat (Fpath.to_string b)).Unix.st_mtime
51
+
with _ -> 0.0 in
52
+
compare mb ma)
53
+
in
54
+
if List.length dirs > 3 then begin
55
+
let to_delete = List.filteri (fun i _ -> i >= 3) dirs in
56
+
Printf.printf "Solution caches: keeping 3 of %d, deleting %d\n%!"
57
+
(List.length dirs) (List.length to_delete);
58
+
List.iter (fun d ->
59
+
Printf.printf " Deleting %s\n%!" (Fpath.basename d);
60
+
ignore (Sys.command (Printf.sprintf "rm -rf %s"
61
+
(Fpath.to_string d)))
62
+
) to_delete;
63
+
Printf.printf "\n"
64
+
end else
65
+
Printf.printf "Solution caches: %d (keeping all)\n\n"
66
+
(List.length dirs)
67
+
| Error _ -> ());
68
+
(* 4. GC old run files — keep last 30 *)
69
+
let runs_dir = Fpath.(cache_dir / "runs") in
70
+
(match Bos.OS.Dir.contents runs_dir with
71
+
| Ok files ->
72
+
let files = files
73
+
|> List.filter (fun f -> Fpath.has_ext ".json" f)
74
+
|> List.sort (fun a b ->
75
+
compare (Fpath.to_string b) (Fpath.to_string a))
76
+
in
77
+
if List.length files > 30 then begin
78
+
let to_delete = List.filteri (fun i _ -> i >= 30) files in
79
+
Printf.printf "Run files: keeping 30 of %d, deleting %d\n%!"
80
+
(List.length files) (List.length to_delete);
81
+
List.iter (fun f ->
82
+
ignore (Bos.OS.File.delete f)
83
+
) to_delete;
84
+
Printf.printf "\n"
85
+
end else
86
+
Printf.printf "Run files: %d (keeping all)\n\n" (List.length files)
87
+
| Error _ -> ());
88
+
(* 5. GC orphaned build layers *)
89
+
(* Find layers referenced by the latest run *)
90
+
let latest_run =
91
+
match Bos.OS.Dir.contents runs_dir with
92
+
| Ok files ->
93
+
let runs = files
94
+
|> List.filter (fun f -> Fpath.has_ext ".json" f)
95
+
|> List.sort (fun a b ->
96
+
compare (Fpath.to_string b) (Fpath.to_string a))
97
+
in
98
+
(match runs with
99
+
| f :: _ ->
100
+
(try
101
+
let data = In_channel.with_open_text (Fpath.to_string f)
102
+
In_channel.input_all in
103
+
let json = Yojson.Safe.from_string data in
104
+
let open Yojson.Safe.Util in
105
+
let layers = json |> member "layers" |> to_assoc in
106
+
Some (List.map (fun (hash, _) ->
107
+
"build-" ^ String.sub hash 0 12) layers)
108
+
with _ -> None)
109
+
| [] -> None)
110
+
| Error _ -> None
111
+
in
112
+
(* Find the os_dir *)
113
+
let os_dir =
114
+
match Bos.OS.Dir.contents cache_dir with
115
+
| Ok entries ->
116
+
List.find_opt (fun p ->
117
+
let name = Fpath.basename p in
118
+
String.length name > 0 &&
119
+
Bos.OS.Dir.exists Fpath.(p / "packages") |> Result.get_ok
120
+
) entries
121
+
| Error _ -> None
122
+
in
123
+
(match os_dir, latest_run with
124
+
| Some os_dir, Some referenced ->
125
+
let referenced_set = Hashtbl.create (List.length referenced) in
126
+
List.iter (fun r -> Hashtbl.replace referenced_set r ()) referenced;
127
+
let all_layers =
128
+
match Bos.OS.Dir.contents os_dir with
129
+
| Ok entries ->
130
+
List.filter (fun p ->
131
+
let name = Fpath.basename p in
132
+
String.starts_with ~prefix:"build-" name &&
133
+
not (Fpath.has_ext ".lock" p) &&
134
+
Bos.OS.Dir.exists p |> Result.get_ok
135
+
) entries
136
+
| Error _ -> []
137
+
in
138
+
let orphans = List.filter (fun p ->
139
+
not (Hashtbl.mem referenced_set (Fpath.basename p))
140
+
) all_layers in
141
+
if orphans <> [] then begin
142
+
Printf.printf "Build layers: %d total, %d referenced, %d orphaned\n%!"
143
+
(List.length all_layers) (List.length referenced) (List.length orphans);
144
+
Printf.printf " Delete orphans? (would free space)\n";
145
+
Printf.printf " Run with --delete-orphans to remove them.\n\n"
146
+
end else
147
+
Printf.printf "Build layers: %d total, all referenced\n\n"
148
+
(List.length all_layers)
149
+
| _ ->
150
+
Printf.printf "Build layers: no run data to check references\n\n");
151
+
(* 6. Summary *)
152
+
Printf.printf "Cache dir: %s\n"
153
+
(du (Printf.sprintf "du -sh %s 2>/dev/null | cut -f1"
154
+
(Fpath.to_string cache_dir)));
13
155
0
14
156
15
157
let cmd =
16
-
let info = Cmd.info "gc" ~doc:"Reclaim disk space" in
17
-
let term = Term.(const run $ Common.cache_dir_term
18
-
$ Common.os_distribution_term $ Common.os_version_term
19
-
$ Common.arch_term) in
158
+
let info = Cmd.info "gc" ~doc:"Reclaim disk space and clean up" in
159
+
let term = Term.(const run $ Common.cache_dir_term) in
20
160
Cmd.v info term
+142
day11/bin/cmd_report.ml
+142
day11/bin/cmd_report.ml
···
1
+
(** report command: generate daily build summary *)
2
+
3
+
open Cmdliner
4
+
5
+
let run cache_dir =
6
+
let cache_dir = Common.fpath cache_dir in
7
+
let runs_dir = Fpath.(cache_dir / "runs") in
8
+
let load_run f =
9
+
try
10
+
let data = In_channel.with_open_text (Fpath.to_string f)
11
+
In_channel.input_all in
12
+
Some (Yojson.Safe.from_string data)
13
+
with _ -> None
14
+
in
15
+
let run_pkg_status json =
16
+
let open Yojson.Safe.Util in
17
+
let layers = json |> member "layers" |> to_assoc in
18
+
let pkg_status : (string, string) Hashtbl.t = Hashtbl.create 256 in
19
+
List.iter (fun (_hash, entry) ->
20
+
let pkg = entry |> member "package" |> to_string in
21
+
let status = entry |> member "status" |> to_string in
22
+
match Hashtbl.find_opt pkg_status pkg with
23
+
| Some "ok" -> ()
24
+
| _ -> Hashtbl.replace pkg_status pkg status
25
+
) layers;
26
+
pkg_status
27
+
in
28
+
let runs =
29
+
match Bos.OS.Dir.contents runs_dir with
30
+
| Ok files ->
31
+
files
32
+
|> List.filter (fun f -> Fpath.has_ext ".json" f)
33
+
|> List.sort (fun a b ->
34
+
compare (Fpath.to_string b) (Fpath.to_string a))
35
+
| Error _ -> []
36
+
in
37
+
match runs with
38
+
| [] ->
39
+
Printf.printf "No runs found.\n"; 1
40
+
| latest_file :: rest ->
41
+
match load_run latest_file with
42
+
| None ->
43
+
Printf.printf "Cannot load latest run.\n"; 1
44
+
| Some latest_json ->
45
+
let open Yojson.Safe.Util in
46
+
let ts = latest_json |> member "timestamp" |> to_string in
47
+
let repos = latest_json |> member "repos" |> to_list in
48
+
let latest_ps = run_pkg_status latest_json in
49
+
let n_ok = Hashtbl.fold (fun _ s n ->
50
+
if s = "ok" then n + 1 else n) latest_ps 0 in
51
+
let n_fail = Hashtbl.fold (fun _ s n ->
52
+
if s = "fail" then n + 1 else n) latest_ps 0 in
53
+
let n_cascade = Hashtbl.fold (fun _ s n ->
54
+
if s = "cascade" then n + 1 else n) latest_ps 0 in
55
+
(* Generate report *)
56
+
let buf = Buffer.create 4096 in
57
+
let pr fmt = Printf.bprintf buf fmt in
58
+
pr "# Daily Build Report — %s\n\n" (String.sub ts 0 10);
59
+
pr "## Repositories\n\n";
60
+
List.iter (fun r ->
61
+
let path = r |> member "path" |> to_string in
62
+
let commit = r |> member "commit" |> to_string in
63
+
pr "- `%s` @ `%s`\n" (Filename.basename path) (String.sub commit 0 12)
64
+
) repos;
65
+
pr "\n## Build Summary\n\n";
66
+
pr "| Metric | Count |\n";
67
+
pr "|--------|-------|\n";
68
+
pr "| Succeeded | %d |\n" n_ok;
69
+
pr "| Failed (root) | %d |\n" n_fail;
70
+
pr "| Failed (cascade) | %d |\n" n_cascade;
71
+
pr "| Total | %d |\n" (n_ok + n_fail + n_cascade);
72
+
pr "\n";
73
+
(* Diff with previous *)
74
+
let prev_opt = match rest with
75
+
| prev_file :: _ -> load_run prev_file
76
+
| [] -> None
77
+
in
78
+
(match prev_opt with
79
+
| Some prev_json ->
80
+
let prev_ps = run_pkg_status prev_json in
81
+
let prev_ts = prev_json |> member "timestamp" |> to_string in
82
+
let prev_ok = Hashtbl.fold (fun _ s n ->
83
+
if s = "ok" then n + 1 else n) prev_ps 0 in
84
+
let fixed = ref [] in
85
+
let regressed = ref [] in
86
+
Hashtbl.iter (fun pkg status ->
87
+
match Hashtbl.find_opt prev_ps pkg with
88
+
| Some prev_status ->
89
+
if status = "ok" && prev_status <> "ok" then
90
+
fixed := pkg :: !fixed
91
+
else if status <> "ok" && prev_status = "ok" then
92
+
regressed := pkg :: !regressed
93
+
| None -> ()
94
+
) latest_ps;
95
+
let fixed = List.sort String.compare !fixed in
96
+
let regressed = List.sort String.compare !regressed in
97
+
pr "## Changes (vs %s)\n\n" (String.sub prev_ts 0 10);
98
+
pr "- **Previous:** %d succeeded\n" prev_ok;
99
+
pr "- **Current:** %d succeeded\n" n_ok;
100
+
pr "- **Net change:** %+d\n\n" (n_ok - prev_ok);
101
+
if fixed <> [] then begin
102
+
pr "### Newly passing (%d)\n\n" (List.length fixed);
103
+
List.iter (fun p -> pr "- `%s`\n" p) fixed;
104
+
pr "\n"
105
+
end;
106
+
if regressed <> [] then begin
107
+
pr "### Newly failing (%d)\n\n" (List.length regressed);
108
+
List.iter (fun p -> pr "- `%s`\n" p) regressed;
109
+
pr "\n"
110
+
end
111
+
| None ->
112
+
pr "## Changes\n\nNo previous run to compare.\n\n");
113
+
(* Top blockers *)
114
+
pr "## Top Blockers\n\n";
115
+
let fail_pkgs = Hashtbl.fold (fun pkg status acc ->
116
+
if status = "fail" then pkg :: acc else acc
117
+
) latest_ps [] in
118
+
let fail_pkgs = List.sort String.compare fail_pkgs in
119
+
pr "| Package | Blocks | Sole |\n";
120
+
pr "|---------|--------|------|\n";
121
+
(* We don't have solution data here — just list the failures *)
122
+
List.iteri (fun i pkg ->
123
+
if i < 20 then pr "| `%s` | ? | ? |\n" pkg
124
+
) fail_pkgs;
125
+
if List.length fail_pkgs > 20 then
126
+
pr "| ... | | |\n";
127
+
pr "\n*Run `day11 results` for full impact analysis.*\n";
128
+
(* Write report *)
129
+
let reports_dir = Fpath.(cache_dir / "reports") in
130
+
Bos.OS.Dir.create ~path:true reports_dir |> ignore;
131
+
let date = String.sub ts 0 10 in
132
+
let report_file = Fpath.(reports_dir / (date ^ ".md")) in
133
+
let content = Buffer.contents buf in
134
+
ignore (Bos.OS.File.write report_file content);
135
+
Printf.printf "%s" content;
136
+
Printf.printf "\nReport saved to %s\n" (Fpath.to_string report_file);
137
+
0
138
+
139
+
let cmd =
140
+
let info = Cmd.info "report" ~doc:"Generate daily build summary report" in
141
+
let term = Term.(const run $ Common.cache_dir_term) in
142
+
Cmd.v info term