forked from
anil.recoil.org/monopam-myspace
My aggregated monorepo of OCaml code, automaintained
docs: implementation plan for per-package findlib_index.json
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+312
+312
docs/plans/2026-02-22-per-package-findlib-index.md
+312
docs/plans/2026-02-22-per-package-findlib-index.md
···
1
+
# Per-package findlib_index.json Implementation Plan
2
+
3
+
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
4
+
5
+
**Goal:** Write `p/<pkg>/<ver>/findlib_index.json` for each target package so `.mld` files can use one tag (`@x-ocaml.universe ../../p/yojson/3.0.0`) and everything is discoverable.
6
+
7
+
**Architecture:** day10's `assemble_jtw_output` already collects all the information (compiler hash, all META paths with content hashes). We add a step that writes a per-package findlib_index.json with paths relative to `p/<pkg>/<ver>/`. On the x-ocaml side, we add a pre-fetch of the findlib_index to extract the `compiler` field and auto-construct the worker URL.
8
+
9
+
**Tech Stack:** OCaml (day10 batch, x-ocaml.js/js_of_ocaml, findlibish)
10
+
11
+
**Design doc:** `docs/plans/2026-02-22-per-package-findlib-index-design.md`
12
+
13
+
---
14
+
15
+
### Task 1: Write per-package findlib_index.json in assemble_jtw_output
16
+
17
+
**Files:**
18
+
- Modify: `day10/bin/jtw_gen.ml:360-372` (after existing findlib_index write)
19
+
20
+
**Step 1: Add per-package findlib_index generation**
21
+
22
+
After the existing `u/<hash>/findlib_index.json` write (line 371), add code to write
23
+
`p/<pkg>/<ver>/findlib_index.json` for each target package. The target package is
24
+
`_target_pkg` (currently ignored at line 260).
25
+
26
+
The meta_files paths need to be rebased from `u/<hash>/` to `p/<pkg>/<ver>/`.
27
+
Currently paths look like `../../p/seq/base/85ab.../lib/seq/META` (relative to
28
+
`u/<hash>/`). From `p/<pkg>/<ver>/`, the same file is at
29
+
`../../../p/seq/base/85ab.../lib/seq/META` — one more `../` because we're one
30
+
level deeper.
31
+
32
+
Exception: the target package's own META path is currently
33
+
`../../p/<pkg>/<ver>/<hash>/lib/<name>/META`. From `p/<pkg>/<ver>/`, this becomes
34
+
just `<hash>/lib/<name>/META`.
35
+
36
+
In `day10/bin/jtw_gen.ml`, after line 372 (`Os.write_to_file ... findlib_index`), add:
37
+
38
+
```ocaml
39
+
(* Write per-package findlib_index for the target package.
40
+
This provides a stable, human-readable entry point at
41
+
p/<pkg>/<ver>/findlib_index.json that contains all the same
42
+
information as the universe index but with paths relative to
43
+
p/<pkg>/<ver>/. *)
44
+
let target_name = OpamPackage.name_to_string _target_pkg in
45
+
let target_version = OpamPackage.version_to_string _target_pkg in
46
+
let target_prefix = "../../p/" ^ target_name ^ "/" ^ target_version ^ "/" in
47
+
let pkg_metas = List.map (fun meta_path ->
48
+
if String.length meta_path > String.length target_prefix
49
+
&& String.sub meta_path 0 (String.length target_prefix) = target_prefix then
50
+
(* This package's own META — strip the ../../p/<pkg>/<ver>/ prefix *)
51
+
String.sub meta_path (String.length target_prefix)
52
+
(String.length meta_path - String.length target_prefix)
53
+
else
54
+
(* Other package's META — add one more ../ since we're at p/<pkg>/<ver>/
55
+
instead of u/<hash>/ *)
56
+
"../" ^ meta_path
57
+
) sorted_metas in
58
+
let pkg_findlib_index = generate_findlib_index ~compiler:compiler_json pkg_metas in
59
+
let pkg_index_dir = Path.(jtw_output / "p" / target_name / target_version) in
60
+
Os.mkdir ~parents:true pkg_index_dir;
61
+
Os.write_to_file Path.(pkg_index_dir / "findlib_index.json") pkg_findlib_index
62
+
```
63
+
64
+
Also rename `_target_pkg` to `target_pkg` at line 260:
65
+
66
+
```ocaml
67
+
List.iter (fun (target_pkg, solution, _ocaml_version, worker_output_dir) ->
68
+
```
69
+
70
+
**Step 2: Build and verify**
71
+
72
+
Run:
73
+
```bash
74
+
cd /home/jons-agent/workspace/mono/day10 && eval $(opam env --switch=default) && dune build --root . bin/main.exe
75
+
```
76
+
Expected: builds without errors.
77
+
78
+
**Step 3: Run batch and check output**
79
+
80
+
Run the day10 batch for both yojson packages, then inspect:
81
+
```bash
82
+
cat /home/jons-agent/day10-multi-demo/p/yojson/3.0.0/findlib_index.json | python3 -m json.tool
83
+
cat /home/jons-agent/day10-multi-demo/p/yojson/2.2.2/findlib_index.json | python3 -m json.tool
84
+
```
85
+
86
+
Expected: JSON with `compiler` field (version + content_hash) and `meta_files` array
87
+
where the target package's own META has a short relative path (just `<hash>/lib/...`)
88
+
and all deps have `../../../p/...` paths.
89
+
90
+
**Step 4: Commit**
91
+
92
+
```bash
93
+
git add day10/bin/jtw_gen.ml
94
+
git commit -m "day10: write per-package findlib_index.json at p/<pkg>/<ver>/"
95
+
```
96
+
97
+
---
98
+
99
+
### Task 2: Auto-construct worker URL from findlib_index compiler field in x-ocaml.js
100
+
101
+
**Files:**
102
+
- Modify: `x-ocaml/src/x_ocaml.ml:26-64`
103
+
104
+
**Step 1: Understand the current flow**
105
+
106
+
Currently in `x_ocaml.ml`:
107
+
1. `worker_url` is computed synchronously (lines 26-39) from `src-worker` attr or `x-ocaml-worker` meta
108
+
2. `findlib_index` URL is computed (lines 56-62) from `x-ocaml-universe` meta
109
+
3. `Backend.make` is called with both (line 64)
110
+
111
+
The problem: the worker URL must be known before creating the backend, but the
112
+
compiler info is inside the findlib_index which hasn't been fetched yet.
113
+
114
+
**Step 2: Add synchronous pre-fetch of findlib_index**
115
+
116
+
The worker runs in a Web Worker, created synchronously. We need the worker URL
117
+
before creating it. Solution: do a synchronous XHR fetch of findlib_index.json
118
+
from x_ocaml.ml (main thread) to extract the compiler field, then construct the
119
+
worker URL from it.
120
+
121
+
This is a small blocking fetch of a single JSON file (~200 bytes). It only happens
122
+
when there's a `x-ocaml-universe` meta tag and no explicit `x-ocaml-worker` override.
123
+
124
+
In `x-ocaml/src/x_ocaml.ml`, replace lines 26-64 with:
125
+
126
+
```ocaml
127
+
(** Synchronous XHR fetch — used only for the small findlib_index.json pre-fetch.
128
+
Returns None if the fetch fails. *)
129
+
let sync_fetch_text url =
130
+
let xhr = Jv.new' (Jv.get Jv.global "XMLHttpRequest") [||] in
131
+
Jv.call xhr "open" [| Jv.of_string "GET"; Jv.of_string url; Jv.of_bool false |] |> ignore;
132
+
(try Jv.call xhr "send" [||] |> ignore with _ -> ());
133
+
let status = Jv.to_int (Jv.get xhr "status") in
134
+
if status = 200 then
135
+
Some (Jv.to_string (Jv.get xhr "responseText"))
136
+
else
137
+
None
138
+
139
+
let findlib_index_url =
140
+
match read_meta "x-ocaml-universe" with
141
+
| None -> None
142
+
| Some base ->
143
+
let base = if String.length base > 0 && base.[String.length base - 1] = '/'
144
+
then base else base ^ "/" in
145
+
Some (resolve_url (base ^ "findlib_index.json"))
146
+
147
+
(** Pre-fetch findlib_index.json to extract compiler info for worker URL.
148
+
Returns (worker_url_from_compiler, findlib_index_url) *)
149
+
let compiler_from_findlib_index =
150
+
match findlib_index_url with
151
+
| None -> None
152
+
| Some url ->
153
+
match sync_fetch_text url with
154
+
| None -> None
155
+
| Some text ->
156
+
(try
157
+
let json = Yojson.Safe.from_string text in
158
+
let open Yojson.Safe.Util in
159
+
let compiler = json |> member "compiler" in
160
+
let version = compiler |> member "version" |> to_string in
161
+
let content_hash = compiler |> member "content_hash" |> to_string in
162
+
Some (version, content_hash)
163
+
with _ -> None)
164
+
165
+
let worker_url =
166
+
(* Priority: 1) explicit x-ocaml-worker meta, 2) compiler from findlib_index, 3) src-worker attr *)
167
+
match read_meta "x-ocaml-worker" with
168
+
| Some url -> url
169
+
| None ->
170
+
match compiler_from_findlib_index with
171
+
| Some (version, content_hash) ->
172
+
(* Construct worker URL relative to findlib_index's base.
173
+
findlib_index is at e.g. p/yojson/3.0.0/findlib_index.json
174
+
worker is at compiler/<ver>/<hash>/worker.js
175
+
From p/yojson/3.0.0/, that's ../../../compiler/<ver>/<hash>/worker.js *)
176
+
let fi_url = Option.get findlib_index_url in
177
+
let base_dir = match String.rindex_opt fi_url '/' with
178
+
| Some i -> String.sub fi_url 0 (i + 1)
179
+
| None -> fi_url
180
+
in
181
+
base_dir ^ "../../../compiler/" ^ version ^ "/" ^ content_hash ^ "/worker.js"
182
+
| None ->
183
+
match current_attribute "src-worker" with
184
+
| None ->
185
+
if backend_name = "builtin" then
186
+
failwith "x-ocaml script missing src-worker attribute"
187
+
else ""
188
+
| Some url -> Jstr.to_string url
189
+
190
+
let findlib_requires =
191
+
match read_meta "x-ocaml-packages" with
192
+
| None -> None
193
+
| Some s ->
194
+
let pkgs = List.filter (fun s -> s <> "")
195
+
(List.map String.trim (String.split_on_char ',' s)) in
196
+
if pkgs = [] then None else Some pkgs
197
+
198
+
let backend = Backend.make ~backend:backend_name ?extra_load ?findlib_requires
199
+
?findlib_index:findlib_index_url worker_url
200
+
```
201
+
202
+
Note: this requires `yojson` to be available in x-ocaml's compilation. Check
203
+
whether it's already a dependency — if not, we need a lighter JSON parser or
204
+
use Jv directly.
205
+
206
+
**Step 3: Check x-ocaml dependencies for yojson**
207
+
208
+
```bash
209
+
grep -r "yojson\|Yojson" x-ocaml/src/dune x-ocaml/dune 2>/dev/null
210
+
```
211
+
212
+
If yojson is NOT available, replace the OCaml JSON parsing with Jv calls:
213
+
214
+
```ocaml
215
+
let compiler_from_findlib_index =
216
+
match findlib_index_url with
217
+
| None -> None
218
+
| Some url ->
219
+
match sync_fetch_text url with
220
+
| None -> None
221
+
| Some text ->
222
+
(try
223
+
let json = Jv.call (Jv.get Jv.global "JSON") "parse" [| Jv.of_string text |] in
224
+
let compiler = Jv.get json "compiler" in
225
+
if Jv.is_none compiler then None
226
+
else
227
+
let version = Jv.to_string (Jv.get compiler "version") in
228
+
let content_hash = Jv.to_string (Jv.get compiler "content_hash") in
229
+
Some (version, content_hash)
230
+
with _ -> None)
231
+
```
232
+
233
+
This uses the browser's built-in JSON.parse — no OCaml JSON library needed.
234
+
235
+
**Step 4: Handle the worker URL base path correctly**
236
+
237
+
The worker URL construction assumes a known relationship between the
238
+
findlib_index URL and the compiler directory. The `../../../` prefix works when
239
+
findlib_index is at `p/<pkg>/<ver>/findlib_index.json` (3 levels deep from root).
240
+
241
+
But a more robust approach: resolve relative to the page's base URL or the
242
+
findlib_index URL. The `resolve_url` helper already handles this. We should
243
+
construct the relative path from the findlib_index's directory:
244
+
245
+
The findlib_index is fetched from an absolute URL like
246
+
`https://host/site/p/yojson/3.0.0/findlib_index.json`.
247
+
The compiler dir is `https://host/site/compiler/5.4.1/<hash>/worker.js`.
248
+
Relative from the findlib_index's parent dir: `../../../compiler/5.4.1/<hash>/worker.js`.
249
+
250
+
This is correct for the day10 layout. For other layouts, the explicit
251
+
`@x-ocaml.worker` override handles it.
252
+
253
+
**Step 5: Build x-ocaml**
254
+
255
+
```bash
256
+
cd /home/jons-agent/workspace/mono && dune build x-ocaml/src/x_ocaml.bc.js
257
+
```
258
+
Expected: builds without errors.
259
+
260
+
**Step 6: Commit**
261
+
262
+
```bash
263
+
git add x-ocaml/src/x_ocaml.ml
264
+
git commit -m "x-ocaml: auto-discover worker URL from findlib_index compiler field"
265
+
```
266
+
267
+
---
268
+
269
+
### Task 3: Update demo and test with per-package findlib_index
270
+
271
+
**Files:**
272
+
- Modify: demo HTML files in `/home/jons-agent/day10-multi-demo/`
273
+
- Modify: test runner `/home/jons-agent/day10-multi-demo/test/run_tests.js`
274
+
275
+
**Step 1: Rebuild day10 batch to generate per-package findlib_index files**
276
+
277
+
Run the day10 batch for both yojson packages. This regenerates the output
278
+
including the new `p/<pkg>/<ver>/findlib_index.json` files.
279
+
280
+
**Step 2: Verify per-package findlib_index files exist**
281
+
282
+
```bash
283
+
cat /home/jons-agent/day10-multi-demo/p/yojson/3.0.0/findlib_index.json | python3 -m json.tool
284
+
cat /home/jons-agent/day10-multi-demo/p/yojson/2.2.2/findlib_index.json | python3 -m json.tool
285
+
```
286
+
287
+
**Step 3: Create test .mld-style demo pages**
288
+
289
+
Create demo pages that use only `@x-ocaml.universe` (via meta tag) with NO
290
+
explicit `x-ocaml-worker` — the worker URL should be auto-discovered from the
291
+
findlib_index compiler field.
292
+
293
+
Update `demo_yojson3.html`:
294
+
- Remove `<meta name="x-ocaml-worker" ...>`
295
+
- Change `<meta name="x-ocaml-universe" ...>` to point at `p/yojson/3.0.0`
296
+
297
+
Update `demo_yojson2.html` similarly for `p/yojson/2.2.2`.
298
+
299
+
Deploy rebuilt x-ocaml.js to `_x-ocaml/`.
300
+
301
+
**Step 4: Run Playwright tests**
302
+
303
+
```bash
304
+
cd /home/jons-agent/day10-multi-demo/test && node run_tests.js
305
+
```
306
+
Expected: 6/6 tests pass.
307
+
308
+
**Step 5: Commit**
309
+
310
+
```bash
311
+
git commit -m "demo: use per-package findlib_index for worker auto-discovery"
312
+
```