My aggregated monorepo of OCaml code, automaintained
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

docs: implementation plan for day10 --local-repo option

8-task plan covering Local_repo module, Config/CLI changes,
jtw_tools.ml and doc_tools.ml local pin support, linux.ml
bind mounts, and integration testing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

+532
+532
docs/plans/2026-02-21-day10-local-repo-plan.md
··· 1 + # day10 --local-repo Implementation Plan 2 + 3 + > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. 4 + 5 + **Goal:** Add `--local-repo PATH` CLI option to day10, allowing local directories to be bind-mounted into build containers and their opam packages pinned from the local path instead of from a git URL. 6 + 7 + **Architecture:** A new `local_repos` field in Config.t holds paths from repeatable `--local-repo` CLI args. A `Local_repo` module discovers `*.opam` packages in local paths and computes git-state-based cache hashes. Build scripts in `jtw_tools.ml` and `doc_tools.ml` check for local overrides when generating `opam pin` commands. `linux.ml` adds bind mounts for any local repos used by a given layer build. 8 + 9 + **Tech Stack:** OCaml, cmdliner (CLI), runc (containers), opam (package pinning) 10 + 11 + **Design doc:** `docs/plans/2026-02-21-day10-local-repo-design.md` 12 + 13 + --- 14 + 15 + ### Task 1: Add Local_repo module 16 + 17 + **Files:** 18 + - Create: `day10/bin/local_repo.ml` 19 + 20 + **Step 1: Create the module** 21 + 22 + ```ocaml 23 + (** Local repository package discovery and hashing for --local-repo. *) 24 + 25 + (** Discover opam package names in a local directory. 26 + Scans for *.opam files at the root (not recursive). *) 27 + let discover_packages path = 28 + let entries = Sys.readdir path in 29 + Array.to_list entries 30 + |> List.filter_map (fun name -> 31 + match Filename.extension name with 32 + | ".opam" -> Some (Filename.remove_extension name) 33 + | _ -> None) 34 + 35 + (** Compute a cache hash for a local repository path. 36 + Uses git HEAD + dirty state if available, otherwise hashes opam file contents. *) 37 + let repo_hash path = 38 + let git_dir = Filename.concat path ".git" in 39 + if Sys.file_exists git_dir then begin 40 + (* Git repo: use HEAD sha + dirty flag *) 41 + let head = 42 + let ic = Unix.open_process_in (Printf.sprintf "git -C %s rev-parse HEAD 2>/dev/null" (Filename.quote path)) in 43 + let line = try input_line ic with End_of_file -> "unknown" in 44 + let _ = Unix.close_process_in ic in 45 + String.trim line 46 + in 47 + let dirty = 48 + let r = Sys.command (Printf.sprintf "git -C %s diff --quiet 2>/dev/null" (Filename.quote path)) in 49 + r <> 0 50 + in 51 + if dirty then begin 52 + let ic = Unix.open_process_in (Printf.sprintf "git -C %s diff 2>/dev/null | md5sum" (Filename.quote path)) in 53 + let diff_hash = try input_line ic |> String.split_on_char ' ' |> List.hd with End_of_file -> "unknown" in 54 + let _ = Unix.close_process_in ic in 55 + Printf.sprintf "local:%s|%s|dirty-%s" path head diff_hash 56 + end else 57 + Printf.sprintf "local:%s|%s" path head 58 + end else begin 59 + (* Not a git repo: hash opam file contents *) 60 + let packages = discover_packages path in 61 + let contents = List.map (fun pkg -> 62 + let opam_file = Filename.concat path (pkg ^ ".opam") in 63 + In_channel.with_open_text opam_file In_channel.input_all 64 + ) packages in 65 + let combined = String.concat "|" ("local" :: path :: contents) in 66 + "local:" ^ (Digest.string combined |> Digest.to_hex) 67 + end 68 + 69 + (** Find the local repo (if any) that provides at least one of the given package names. 70 + Returns [Some (path, matching_packages)] or [None]. *) 71 + let find_for_packages ~local_repos packages = 72 + List.find_map (fun repo_path -> 73 + let available = discover_packages repo_path in 74 + let matches = List.filter (fun pkg -> List.mem pkg available) packages in 75 + if matches <> [] then Some (repo_path, matches) 76 + else None 77 + ) local_repos 78 + 79 + (** Validate all local repos at startup. 80 + Returns [Ok ()] or [Error msg]. *) 81 + let validate local_repos = 82 + let errors = List.filter_map (fun path -> 83 + if not (Sys.file_exists path) then 84 + Some (Printf.sprintf "--local-repo: directory does not exist: %s" path) 85 + else if not (Sys.is_directory path) then 86 + Some (Printf.sprintf "--local-repo: not a directory: %s" path) 87 + else 88 + let pkgs = discover_packages path in 89 + if pkgs = [] then begin 90 + Printf.eprintf "Warning: --local-repo %s contains no *.opam files\n%!" path; 91 + None 92 + end else 93 + None 94 + ) local_repos in 95 + (* Check for duplicate packages across repos *) 96 + let all_pkgs = List.concat_map (fun path -> 97 + List.map (fun pkg -> (pkg, path)) (discover_packages path) 98 + ) local_repos in 99 + let dup_errors = List.filter_map (fun (pkg, path) -> 100 + let others = List.filter (fun (p, pa) -> p = pkg && pa <> path) all_pkgs in 101 + if others <> [] then 102 + Some (Printf.sprintf "--local-repo: package %s found in multiple repos: %s and %s" pkg path (snd (List.hd others))) 103 + else None 104 + ) all_pkgs in 105 + (* Deduplicate error messages *) 106 + let dup_errors = List.sort_uniq String.compare dup_errors in 107 + match errors @ dup_errors with 108 + | [] -> Ok () 109 + | errs -> Error (String.concat "\n" errs) 110 + ``` 111 + 112 + **Step 2: Verify it compiles** 113 + 114 + Run: `cd /home/jons-agent/workspace/mono && dune build day10/bin/local_repo.ml 2>&1 | head -20` 115 + 116 + If there's no dune rule for this file yet, that's expected — we'll add it to the dune file in the next task. 117 + 118 + **Step 3: Commit** 119 + 120 + ```bash 121 + git add day10/bin/local_repo.ml 122 + git commit -m "feat(day10): add Local_repo module for package discovery and hashing" 123 + ``` 124 + 125 + --- 126 + 127 + ### Task 2: Add local_repos to Config.t and CLI 128 + 129 + **Files:** 130 + - Modify: `day10/bin/config.ml:1` (add field) 131 + - Modify: `day10/bin/main.ml:1435-1441` (add CLI term) 132 + - Modify: `day10/bin/main.ml:1521-1554` (wire into ci command Config) 133 + - Modify: `day10/bin/main.ml:1566-1570` (wire into run/health-check command Config) 134 + - Modify: `day10/bin/main.ml:1675-1678` (wire into batch command Config) 135 + 136 + **Step 1: Add field to Config.t** 137 + 138 + In `day10/bin/config.ml`, add after line 21 (`jtw_tools_branch : string;`): 139 + 140 + ```ocaml 141 + local_repos : string list; 142 + ``` 143 + 144 + **Step 2: Add CLI term in main.ml** 145 + 146 + After `jtw_tools_branch_term` (around line 1441), add: 147 + 148 + ```ocaml 149 + let local_repo_term = 150 + let doc = "Use local directory for opam pins instead of git (repeatable). \ 151 + Scans for *.opam files and pins matching packages from the local path." in 152 + Arg.(value & opt_all string [] & info [ "local-repo" ] ~docv:"PATH" ~doc) 153 + ``` 154 + 155 + **Step 3: Wire into all commands that use Config.t** 156 + 157 + Each command constructor (`ci`, `health-check`/`run`, `batch`, `list`) needs: 158 + - `local_repo_term` added to the `$` chain 159 + - `local_repos` parameter added to the `const` function 160 + - `local_repos` field set in the Config.t record (use `[]` for commands that don't support it like `list`) 161 + 162 + For `run`/`health-check` (line ~1566): 163 + ```ocaml 164 + const (fun dir ocaml_version ... jtw_tools_branch local_repos ... -> 165 + ... 166 + { ...; jtw_tools_branch; local_repos; ... } 167 + ``` 168 + 169 + For `ci` (line ~1521): set `local_repos = [];` (ci doesn't use jtw) 170 + 171 + For `list` (line ~1578): set `local_repos = [];` 172 + 173 + **Step 4: Add validation call** 174 + 175 + In each command handler that uses local repos (`run_health_check_multi`, `run_batch`), add validation early: 176 + 177 + ```ocaml 178 + let () = match Local_repo.validate config.local_repos with 179 + | Ok () -> () 180 + | Error msg -> failwith msg 181 + in 182 + ``` 183 + 184 + **Step 5: Build and fix any compilation errors** 185 + 186 + Run: `cd /home/jons-agent/workspace/mono && dune build day10/ 2>&1 | head -40` 187 + 188 + Fix any missing field errors in Config.t record constructions throughout the codebase. 189 + 190 + **Step 6: Commit** 191 + 192 + ```bash 193 + git add day10/bin/config.ml day10/bin/main.ml 194 + git commit -m "feat(day10): add --local-repo CLI option and config field" 195 + ``` 196 + 197 + --- 198 + 199 + ### Task 3: Update jtw_tools.ml to use local repos 200 + 201 + **Files:** 202 + - Modify: `day10/bin/jtw_tools.ml:10-13` (layer_hash) 203 + - Modify: `day10/bin/jtw_tools.ml:28-48` (build_script) 204 + 205 + **Step 1: Update layer_hash to include local repo state** 206 + 207 + Replace the `layer_hash` function: 208 + 209 + ```ocaml 210 + let layer_hash ~(config : Config.t) ~(ocaml_version : OpamPackage.t) = 211 + let version = OpamPackage.Version.to_string (OpamPackage.version ocaml_version) in 212 + let jtw_packages = [ "js_top_worker"; "js_top_worker-rpc"; "js_top_worker-bin"; 213 + "js_top_worker-web"; "js_top_worker_rpc_def" ] in 214 + let source_component = match Local_repo.find_for_packages ~local_repos:config.local_repos jtw_packages with 215 + | Some (path, _) -> Local_repo.repo_hash path 216 + | None -> config.jtw_tools_repo ^ "|" ^ config.jtw_tools_branch 217 + in 218 + let components = [ "jtw-tools"; version; source_component ] in 219 + String.concat "|" components |> Digest.string |> Digest.to_hex 220 + ``` 221 + 222 + **Step 2: Update build_script to pin from local path when available** 223 + 224 + Replace the `build_script` function: 225 + 226 + ```ocaml 227 + let jtw_packages = [ "js_top_worker"; "js_top_worker-rpc"; "js_top_worker-bin"; 228 + "js_top_worker-web"; "js_top_worker_rpc_def" ] 229 + 230 + let build_script ~(config : Config.t) ~(ocaml_version : OpamPackage.t) = 231 + let version = OpamPackage.Version.to_string (OpamPackage.version ocaml_version) in 232 + let pin_cmds = match Local_repo.find_for_packages ~local_repos:config.local_repos jtw_packages with 233 + | Some (_, matched) -> 234 + (* Pin matched packages from local mount, rest from git *) 235 + let local_mount = "/home/opam/local/js_top_worker" in 236 + let from_local = List.map (fun pkg -> 237 + Printf.sprintf "opam pin add -yn %s %s" pkg local_mount 238 + ) matched in 239 + let from_git = List.filter_map (fun pkg -> 240 + if List.mem pkg matched then None 241 + else Some (Printf.sprintf "opam pin add -yn %s git+%s#%s" pkg config.jtw_tools_repo config.jtw_tools_branch) 242 + ) jtw_packages in 243 + from_local @ from_git 244 + | None -> 245 + List.map (fun pkg -> 246 + Printf.sprintf "opam pin add -yn %s git+%s#%s" pkg config.jtw_tools_repo config.jtw_tools_branch 247 + ) jtw_packages 248 + in 249 + String.concat " && " 250 + ([ Printf.sprintf "opam install -y ocaml-base-compiler.%s" version ] 251 + @ pin_cmds 252 + @ [ "opam install -y js_of_ocaml js_top_worker-bin js_top_worker-web"; 253 + "eval $(opam env) && which js_of_ocaml && which jtw"; 254 + "eval $(opam env) && jtw opam -o /home/opam/jtw-tools-output stdlib" ]) 255 + ``` 256 + 257 + **Step 3: Add a function to get the local repo mount (if any)** 258 + 259 + ```ocaml 260 + (** Return the local repo path to bind-mount for jtw, if any. *) 261 + let local_repo_mount ~(config : Config.t) = 262 + match Local_repo.find_for_packages ~local_repos:config.local_repos jtw_packages with 263 + | Some (path, _) -> Some (path, "/home/opam/local/js_top_worker") 264 + | None -> None 265 + ``` 266 + 267 + **Step 4: Build and test** 268 + 269 + Run: `cd /home/jons-agent/workspace/mono && dune build day10/ 2>&1 | head -40` 270 + 271 + **Step 5: Commit** 272 + 273 + ```bash 274 + git add day10/bin/jtw_tools.ml 275 + git commit -m "feat(day10): jtw_tools uses local repo for pins when available" 276 + ``` 277 + 278 + --- 279 + 280 + ### Task 4: Update doc_tools.ml to use local repos 281 + 282 + **Files:** 283 + - Modify: `day10/bin/doc_tools.ml:16-18` (driver_layer_hash) 284 + - Modify: `day10/bin/doc_tools.ml:31-50` (driver_build_script) 285 + - Modify: `day10/bin/doc_tools.ml:73-76` (odoc_layer_hash) 286 + - Modify: `day10/bin/doc_tools.ml:89-103` (odoc_build_script) 287 + 288 + **Step 1: Define the package lists and add local_repo_mount helper** 289 + 290 + Add near the top of doc_tools.ml: 291 + 292 + ```ocaml 293 + let driver_packages = [ "odoc"; "odoc-parser"; "odoc-md"; "sherlodoc"; "odoc-driver" ] 294 + let odoc_packages = [ "odoc"; "odoc-parser" ] 295 + 296 + (** Return the local repo path to bind-mount for doc tools, if any. *) 297 + let local_repo_mount ~(config : Config.t) = 298 + match Local_repo.find_for_packages ~local_repos:config.local_repos driver_packages with 299 + | Some (path, _) -> Some (path, "/home/opam/local/odoc") 300 + | None -> None 301 + ``` 302 + 303 + **Step 2: Update driver_layer_hash** 304 + 305 + ```ocaml 306 + let driver_layer_hash ~(config : Config.t) = 307 + let source_component = match Local_repo.find_for_packages ~local_repos:config.local_repos driver_packages with 308 + | Some (path, _) -> Local_repo.repo_hash path 309 + | None -> config.doc_tools_repo ^ "|" ^ config.doc_tools_branch 310 + in 311 + let components = [ "driver"; source_component ] in 312 + String.concat "|" components |> Digest.string |> Digest.to_hex 313 + ``` 314 + 315 + **Step 3: Update driver_build_script** 316 + 317 + Same pattern as jtw_tools: pin matched packages from `/home/opam/local/odoc`, pin unmatched from git. 318 + 319 + ```ocaml 320 + let driver_build_script ~(config : Config.t) = 321 + let repo = config.doc_tools_repo in 322 + let branch = config.doc_tools_branch in 323 + let pin_cmds = match Local_repo.find_for_packages ~local_repos:config.local_repos driver_packages with 324 + | Some (_, matched) -> 325 + let local_mount = "/home/opam/local/odoc" in 326 + let from_local = List.map (fun pkg -> 327 + Printf.sprintf "opam pin add -yn %s %s" pkg local_mount 328 + ) matched in 329 + let from_git = List.filter_map (fun pkg -> 330 + if List.mem pkg matched then None 331 + else Some (Printf.sprintf "opam pin add -yn %s %s#%s" pkg repo branch) 332 + ) driver_packages in 333 + from_local @ from_git 334 + | None -> 335 + List.map (fun pkg -> 336 + Printf.sprintf "opam pin add -yn %s %s#%s" pkg repo branch 337 + ) driver_packages 338 + in 339 + String.concat " && " 340 + ([ "opam install -y ocaml-base-compiler.5.2.1" ] 341 + @ pin_cmds 342 + @ [ "opam install -y odoc-driver odoc-md sherlodoc"; 343 + "eval $(opam env) && sherlodoc js > /home/opam/sherlodoc.js"; 344 + "which odoc_driver_voodoo && which sherlodoc" ]) 345 + ``` 346 + 347 + **Step 4: Update odoc_layer_hash and odoc_build_script** 348 + 349 + Same pattern. `odoc_layer_hash` includes local repo state. `odoc_build_script` pins from local mount for matched packages. 350 + 351 + **Step 5: Build and test** 352 + 353 + Run: `cd /home/jons-agent/workspace/mono && dune build day10/ 2>&1 | head -40` 354 + 355 + **Step 6: Commit** 356 + 357 + ```bash 358 + git add day10/bin/doc_tools.ml 359 + git commit -m "feat(day10): doc_tools uses local repo for pins when available" 360 + ``` 361 + 362 + --- 363 + 364 + ### Task 5: Add bind mounts in linux.ml 365 + 366 + **Files:** 367 + - Modify: `day10/bin/linux.ml:274-337` (build_doc_tools_layer — add optional mounts param) 368 + - Modify: `day10/bin/linux.ml:342-365` (ensure_driver_layer — pass local mount) 369 + - Modify: `day10/bin/linux.ml:370-394` (ensure_odoc_layer — pass local mount) 370 + - Modify: `day10/bin/linux.ml:629-653` (ensure_jtw_tools_layer — pass local mount) 371 + 372 + **Step 1: Add optional extra_mounts parameter to build_doc_tools_layer** 373 + 374 + Change the signature from: 375 + ```ocaml 376 + let build_doc_tools_layer ~t ~temp_dir ~build_script build_log = 377 + ``` 378 + to: 379 + ```ocaml 380 + let build_doc_tools_layer ~t ~temp_dir ~build_script ?(extra_mounts=[]) build_log = 381 + ``` 382 + 383 + Add `extra_mounts` to the mounts list at line 293: 384 + ```ocaml 385 + let mounts = 386 + [ 387 + { Mount.ty = "bind"; src = Path.(temp_dir / "opam-repository"); 388 + dst = "/home/opam/.opam/repo/default"; options = [ "rbind"; "rprivate" ] }; 389 + { ty = "bind"; src = etc_hosts; dst = "/etc/hosts"; options = [ "ro"; "rbind"; "rprivate" ] }; 390 + ] @ extra_mounts 391 + in 392 + ``` 393 + 394 + **Step 2: Update ensure_jtw_tools_layer to pass local mount** 395 + 396 + In `ensure_jtw_tools_layer`, inside `write_layer`, before calling `build_doc_tools_layer`: 397 + 398 + ```ocaml 399 + let extra_mounts = match Jtw_tools.local_repo_mount ~config with 400 + | Some (src, dst) -> 401 + [ { Mount.ty = "bind"; src; dst; options = [ "ro"; "rbind"; "rprivate" ] } ] 402 + | None -> [] 403 + in 404 + let r = build_doc_tools_layer ~t ~temp_dir ~build_script ~extra_mounts build_log in 405 + ``` 406 + 407 + **Step 3: Update ensure_driver_layer to pass local mount** 408 + 409 + Same pattern: 410 + ```ocaml 411 + let extra_mounts = match Doc_tools.local_repo_mount ~config with 412 + | Some (src, dst) -> 413 + [ { Mount.ty = "bind"; src; dst; options = [ "ro"; "rbind"; "rprivate" ] } ] 414 + | None -> [] 415 + in 416 + let r = build_doc_tools_layer ~t ~temp_dir ~build_script ~extra_mounts build_log in 417 + ``` 418 + 419 + **Step 4: Update ensure_odoc_layer to pass local mount** 420 + 421 + Same pattern as ensure_driver_layer. 422 + 423 + **Step 5: Build and test** 424 + 425 + Run: `cd /home/jons-agent/workspace/mono && dune build day10/ 2>&1 | head -40` 426 + 427 + **Step 6: Commit** 428 + 429 + ```bash 430 + git add day10/bin/linux.ml 431 + git commit -m "feat(day10): bind-mount local repos into tool layer containers" 432 + ``` 433 + 434 + --- 435 + 436 + ### Task 6: Update dune file and integration test 437 + 438 + **Files:** 439 + - Modify: `day10/bin/dune` (add local_repo module if needed) 440 + 441 + **Step 1: Check dune file and add local_repo if needed** 442 + 443 + If `day10/bin/dune` uses `(modules ...)` explicitly, add `local_repo`. If it uses implicit module discovery (no `(modules ...)` stanza), it should be picked up automatically. 444 + 445 + **Step 2: Full build** 446 + 447 + Run: `cd /home/jons-agent/workspace/mono && dune build day10/ 2>&1 | head -40` 448 + 449 + **Step 3: Verify --help shows the new option** 450 + 451 + Run: `cd /home/jons-agent/workspace/mono && dune exec day10/bin/main.exe -- run --help 2>&1 | grep -A2 local-repo` 452 + 453 + Expected: the `--local-repo` option appears in help output. 454 + 455 + **Step 4: Smoke test (dry run)** 456 + 457 + Run: `cd /home/jons-agent/workspace/mono && dune exec day10/bin/main.exe -- run yojson --with-jtw --local-repo /home/jons-agent/workspace/mono/js_top_worker --dry-run 2>&1 | head -40` 458 + 459 + Expected: Should show that it detects the local repo packages and doesn't error out. 460 + 461 + **Step 5: Commit** 462 + 463 + ```bash 464 + git add day10/bin/dune 465 + git commit -m "feat(day10): wire up local_repo module in build" 466 + ``` 467 + 468 + --- 469 + 470 + ### Task 7: Also handle run_jtw_in_container (per-package jtw builds) 471 + 472 + **Files:** 473 + - Modify: `day10/bin/linux.ml:656-729` (run_jtw_in_container) 474 + 475 + The jtw per-package builds also run in containers. If the user is using `--local-repo` for jtw, the per-package container might also need the local source mounted (in case `jtw opam` needs to resolve package metadata from the pinned source). 476 + 477 + **Step 1: Add local repo bind mount to run_jtw_in_container** 478 + 479 + In the mounts list at line 700-703, add: 480 + 481 + ```ocaml 482 + let local_mounts = match Jtw_tools.local_repo_mount ~config with 483 + | Some (src, dst) -> 484 + [ { Mount.ty = "bind"; src; dst; options = [ "ro"; "rbind"; "rprivate" ] } ] 485 + | None -> [] 486 + in 487 + let mounts = [ 488 + { Mount.ty = "bind"; src = jtw_output_host; dst = "/home/opam/jtw-output"; options = [ "rw"; "rbind"; "rprivate" ] }; 489 + { ty = "bind"; src = etc_hosts; dst = "/etc/hosts"; options = [ "ro"; "rbind"; "rprivate" ] }; 490 + ] @ local_mounts in 491 + ``` 492 + 493 + **Step 2: Build** 494 + 495 + Run: `cd /home/jons-agent/workspace/mono && dune build day10/ 2>&1 | head -40` 496 + 497 + **Step 3: Commit** 498 + 499 + ```bash 500 + git add day10/bin/linux.ml 501 + git commit -m "feat(day10): mount local repos in per-package jtw containers too" 502 + ``` 503 + 504 + --- 505 + 506 + ### Task 8: Final build, cleanup, and commit 507 + 508 + **Step 1: Full build** 509 + 510 + Run: `cd /home/jons-agent/workspace/mono && dune build day10/ 2>&1` 511 + 512 + **Step 2: Review all changes** 513 + 514 + Run: `cd /home/jons-agent/workspace/mono && git diff HEAD~7 --stat` 515 + 516 + Verify only the expected files were changed. 517 + 518 + **Step 3: If possible, run a real test** 519 + 520 + ```bash 521 + dune exec day10/bin/main.exe -- run yojson \ 522 + --with-jtw \ 523 + --local-repo /home/jons-agent/workspace/mono/js_top_worker \ 524 + --cache-dir /tmp/day10-test \ 525 + --opam-repository /path/to/opam-repository 526 + ``` 527 + 528 + This requires the base layer and opam repository to exist. If the test environment supports it, verify: 529 + - The local repo is detected and packages discovered 530 + - The layer hash changes when local repo content changes 531 + - The container bind-mounts the local repo 532 + - `opam pin` uses the local path inside the container