forked from
anil.recoil.org/monopam-myspace
My aggregated monorepo of OCaml code, automaintained
docs: design for per-package findlib_index.json entry points
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+154
+154
docs/plans/2026-02-22-per-package-findlib-index-design.md
+154
docs/plans/2026-02-22-per-package-findlib-index-design.md
···
1
+
# Design: Per-package findlib_index.json
2
+
3
+
## Problem
4
+
5
+
day10 currently produces universe-level entry points at
6
+
`u/<universe-hash>/findlib_index.json`. The universe hash is an opaque
7
+
content address computed from build hashes — an `.mld` author cannot
8
+
predict it, so they must either hard-code it after a build or use the
9
+
separate `@x-ocaml.worker` and `@x-ocaml.universe` tags with explicit
10
+
hashes.
11
+
12
+
What we want: a stable, human-readable entry point at
13
+
`p/<pkg>/<ver>/findlib_index.json` so an `.mld` file can say:
14
+
15
+
```
16
+
@x-ocaml.universe ../../p/yojson/3.0.0
17
+
```
18
+
19
+
and everything is discoverable from that one URL — which worker to load,
20
+
where every library's META/cmi/cma.js files are, all with content hashes
21
+
for immutable caching.
22
+
23
+
## Design
24
+
25
+
### Output layout (unchanged paths + one new file)
26
+
27
+
```
28
+
compiler/<ver>/<worker-hash>/
29
+
worker.js
30
+
lib/ocaml/stdlib.cmi, stdlib.cma.js, ...
31
+
32
+
p/<pkg>/<ver>/<content-hash>/
33
+
lib/<findlib-name>/META, *.cmi, *.cma.js, dynamic_cmis.json
34
+
35
+
p/<pkg>/<ver>/findlib_index.json <-- NEW
36
+
37
+
u/<universe-hash>/findlib_index.json (kept for backward compat)
38
+
```
39
+
40
+
### Per-package findlib_index.json format
41
+
42
+
`p/yojson/3.0.0/findlib_index.json`:
43
+
44
+
```json
45
+
{
46
+
"compiler": {
47
+
"version": "5.4.1",
48
+
"content_hash": "1f7e4fa461714841"
49
+
},
50
+
"meta_files": [
51
+
"4c22b3794e61bc3f/lib/yojson/META",
52
+
"../../p/seq/base/85ab713df4290233/lib/seq/META",
53
+
"../../p/dune/3.21.1/5f2666121f6b317c/lib/dune/META",
54
+
"../../compiler/5.4.1/1f7e4fa461714841/lib/ocaml/stdlib/META"
55
+
]
56
+
}
57
+
```
58
+
59
+
All paths are relative to this file's location (`p/yojson/3.0.0/`).
60
+
61
+
- **`compiler`** — version + content_hash. x-ocaml uses this to
62
+
construct `compiler/<ver>/<hash>/worker.js` when no explicit
63
+
`@x-ocaml.worker` override is present.
64
+
- **`meta_files`** — flat list of every META path needed for this
65
+
package's complete transitive dependency set. Every path contains a
66
+
content hash, so all referenced files are immutable.
67
+
- No `universes` field — the flat list means one fetch gives the client
68
+
everything. No recursive lookups needed.
69
+
70
+
### Why flat, not recursive
71
+
72
+
The `jtw opam` multiverse mode uses per-package `findlib_index.json`
73
+
with a `universes` field linking to deps, requiring the client to chase
74
+
the graph. For a server like ocaml.org this means N+1 fetches (one per
75
+
transitive dep). A flat list means exactly one fetch to discover
76
+
everything. The client (findlibish.ml) already handles flat meta_files
77
+
lists — no changes needed there.
78
+
79
+
### Immutability and consistency
80
+
81
+
Every path in `meta_files` includes a content hash. If ocaml.org
82
+
rebuilds a package, the content hash changes and a new
83
+
`p/<pkg>/<ver>/<new-hash>/` directory appears. The findlib_index.json is
84
+
then updated atomically to point at the new hashes. This is the only
85
+
mutable file — everything it references is immutable and CDN-cacheable.
86
+
87
+
Without content hashes in the paths, a client could fetch the index,
88
+
then fetch a META file that has been replaced by a rebuild, leading to
89
+
inconsistent state. Content hashes eliminate this race.
90
+
91
+
### .mld usage
92
+
93
+
```
94
+
{0 Yojson Tutorial}
95
+
96
+
@x-ocaml.universe ../../p/yojson/3.0.0
97
+
98
+
{@ocaml[
99
+
#require "yojson"
100
+
]}
101
+
102
+
{@ocaml[
103
+
let json = `Assoc [("hello", `String "world")]
104
+
let () = print_endline (Yojson.Safe.pretty_to_string json)
105
+
]}
106
+
```
107
+
108
+
One tag. x-ocaml.js fetches `../../p/yojson/3.0.0/findlib_index.json`,
109
+
reads `compiler` to load the correct per-solution worker, and resolves
110
+
all META paths for findlibish to consume.
111
+
112
+
### Changes needed
113
+
114
+
#### 1. day10 `assemble_jtw_output` (jtw_gen.ml)
115
+
116
+
After assembling p/ directories and writing u/<hash>/findlib_index.json,
117
+
also write `p/<pkg>/<ver>/findlib_index.json` for each target package.
118
+
The content is the compiler info + the same meta_files list already
119
+
computed for the universe index, but with paths adjusted to be relative
120
+
to `p/<pkg>/<ver>/` instead of `u/<hash>/`.
121
+
122
+
#### 2. x-ocaml.js (x_ocaml.ml)
123
+
124
+
Currently the worker URL comes from `@x-ocaml.worker` meta tag or
125
+
`src-worker` attribute. Add: if findlib_index.json has a `compiler`
126
+
field and no explicit `@x-ocaml.worker` override, construct the worker
127
+
URL as `<base>/compiler/<ver>/<hash>/worker.js` where `<base>` is
128
+
resolved relative to the findlib_index URL.
129
+
130
+
This requires x-ocaml.js to fetch the findlib_index *before* creating
131
+
the worker (currently it creates the worker first and passes the
132
+
findlib_index URL to it). The flow becomes:
133
+
134
+
1. Fetch findlib_index.json
135
+
2. If it has `compiler`, construct worker URL from it
136
+
3. Create worker with that URL
137
+
4. Pass findlib_index URL to worker for META loading
138
+
139
+
#### 3. findlibish.ml — no changes
140
+
141
+
Already handles flat `meta_files` lists. The `universes` field is
142
+
optional and simply won't be present.
143
+
144
+
#### 4. u/<hash>/ directories — kept
145
+
146
+
Continue generating them for backward compatibility with existing
147
+
deployments. No changes to existing format.
148
+
149
+
## Non-goals
150
+
151
+
- Changing the `jtw opam` multiverse format (that's a different use case
152
+
for local development)
153
+
- Removing the u/<hash>/ universe directories
154
+
- Changing how per-package p/<pkg>/<ver>/<hash>/ artifacts are generated