this repo has no description
chore: stage pending changes before monopam sync
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
+387
+387
docs/widget-plan.md
+387
docs/widget-plan.md
···
1
+
# Widget Support Plan of Record
2
+
3
+
*Created: 2026-02-12*
4
+
5
+
## Goals
6
+
7
+
Add Jupyter-style interactive widget support to js_top_worker, enabling OCaml
8
+
code running in the toplevel to create interactive UI elements (sliders, buttons,
9
+
dropdowns, etc.) that communicate bidirectionally with the frontend.
10
+
11
+
## Design Principles
12
+
13
+
1. **Zero new dependencies in the worker** - Every dependency compiled into the
14
+
worker can conflict with libraries the user wants to load at runtime. Widget
15
+
support must not add any new OCaml dependencies to the worker.
16
+
17
+
2. **Broad OCaml version compatibility** - The project currently targets OCaml
18
+
>= 4.04. New features must not raise this floor unnecessarily.
19
+
20
+
3. **Build on the message protocol, not RPC** - The worker already uses the
21
+
message-based protocol (`message.ml` / `js_top_worker_client_msg.ml`). Widget
22
+
communication extends this protocol rather than the legacy JSON-RPC layer.
23
+
24
+
4. **Remove, don't accumulate** - The legacy rpclib-based communication layer
25
+
should be removed as part of this work, reducing the dependency footprint.
26
+
27
+
## Why Not CBOR?
28
+
29
+
The architecture document previously listed CBOR as a planned transport format.
30
+
After investigation, we've decided against it:
31
+
32
+
- **Dependency risk**: Even the lightweight `cbor` opam package brings in
33
+
`ocplib-endian`. Any dependency in the worker namespace can conflict with
34
+
user-loaded libraries.
35
+
- **Unnecessary complexity**: The existing message protocol uses `Js_of_ocaml`'s
36
+
native JSON handling (`Json.output` / `Json.unsafe_input`), which has zero
37
+
additional dependencies.
38
+
- **Binary data via Typed Arrays**: For binary payloads (images, etc.),
39
+
`js_of_ocaml`'s `Typed_array` module provides native browser typed array
40
+
support without any extra libraries.
41
+
- **JSON is the browser's native format** - No encoding/decoding overhead when
42
+
passing structured data via `postMessage`.
43
+
44
+
## Communication Architecture
45
+
46
+
### Current State (Two Parallel Layers)
47
+
48
+
```
49
+
1. Legacy RPC (to be removed):
50
+
Client (js_top_worker_client.ml) <-> JSON-RPC <-> Server (Toplevel_api_gen)
51
+
Dependencies: rpclib, rpclib-lwt, rpclib.json, ppx_deriving_rpc
52
+
53
+
2. Message protocol (to be extended):
54
+
Client (js_top_worker_client_msg.ml) <-> JSON messages <-> Worker (worker.ml)
55
+
Dependencies: js_of_ocaml (already required)
56
+
```
57
+
58
+
### Target State
59
+
60
+
```
61
+
Client (js_top_worker_client_msg.ml) <-> JSON messages <-> Worker (worker.ml)
62
+
| |
63
+
|-- Request/Response (existing: eval, complete, errors, ...) |
64
+
|-- Push messages (existing: output_at streaming) |
65
+
|-- Widget messages (NEW: comm_open, comm_update, comm_msg, ...) |
66
+
```
67
+
68
+
All communication uses the existing `message.ml` protocol extended with widget
69
+
message types. No new serialization libraries.
70
+
71
+
## Widget Protocol Design
72
+
73
+
### Message Types (Worker -> Client)
74
+
75
+
```
76
+
CommOpen { comm_id; target; state } -- Widget created by OCaml code
77
+
CommUpdate { comm_id; state } -- Widget state changed
78
+
CommClose { comm_id } -- Widget destroyed
79
+
```
80
+
81
+
### Message Types (Client -> Worker)
82
+
83
+
```
84
+
CommMsg { comm_id; data } -- Frontend event (click, value change)
85
+
CommClose { comm_id } -- Frontend closed widget
86
+
```
87
+
88
+
### Widget State Format
89
+
90
+
Widget state is a JSON object with well-known keys, following the Jupyter widget
91
+
convention where practical:
92
+
93
+
```json
94
+
{
95
+
"widget_type": "slider",
96
+
"value": 50,
97
+
"min": 0,
98
+
"max": 100,
99
+
"step": 1,
100
+
"description": "Threshold",
101
+
"disabled": false
102
+
}
103
+
```
104
+
105
+
The `widget_type` field replaces Jupyter's `_model_module` / `_model_name` /
106
+
`_view_module` / `_view_name` quartet, since we don't need the npm module
107
+
indirection - our widget renderers are built into the client.
108
+
109
+
### Alignment with Jupyter Protocol
110
+
111
+
We adopt the **concepts** from the Jupyter widget protocol but simplify the
112
+
implementation:
113
+
114
+
| Jupyter Concept | Our Equivalent |
115
+
|-----------------|----------------|
116
+
| comm_open | CommOpen message |
117
+
| comm_msg method:"update" | CommUpdate message |
118
+
| comm_msg method:"custom" | CommMsg message |
119
+
| comm_close | CommClose message |
120
+
| _model_module + _model_name | widget_type string |
121
+
| buffer_paths (binary) | Typed_array via js_of_ocaml |
122
+
| Display message | CommOpen includes display flag |
123
+
124
+
We do **not** implement:
125
+
- echo_update (single frontend, no multi-client sync needed)
126
+
- request_state / request_states (state is authoritative in worker)
127
+
- Version negotiation (internal protocol, not cross-system)
128
+
129
+
## OCaml Widget API
130
+
131
+
User-facing API available as an OCaml library in the toplevel:
132
+
133
+
```ocaml
134
+
module Widget : sig
135
+
type t
136
+
137
+
(** Create a widget. Returns it and displays it. *)
138
+
val slider : ?min:int -> ?max:int -> ?step:int ->
139
+
?description:string -> int -> t
140
+
141
+
val button : ?style:string -> string -> t
142
+
143
+
val text : ?placeholder:string -> ?description:string -> string -> t
144
+
145
+
val dropdown : ?description:string -> options:string list -> string -> t
146
+
147
+
val checkbox : ?description:string -> bool -> t
148
+
149
+
val html : string -> t
150
+
151
+
(** Read current value *)
152
+
val get : t -> Yojson.Safe.t (* or a simpler JSON type *)
153
+
154
+
(** Update widget state *)
155
+
val set : t -> string -> Yojson.Safe.t -> unit
156
+
157
+
(** Register event handler *)
158
+
val on_change : t -> (Yojson.Safe.t -> unit) -> unit
159
+
val on_click : t -> (unit -> unit) -> unit
160
+
161
+
(** Display / close *)
162
+
val display : t -> unit
163
+
val close : t -> unit
164
+
end
165
+
```
166
+
167
+
**Important**: This API library (`widget` or similar) runs *inside* the toplevel
168
+
and must have minimal dependencies. It communicates with the frontend by pushing
169
+
messages through the same channel as `Mime_printer`.
170
+
171
+
## Code Removal Plan
172
+
173
+
### Files to Remove
174
+
175
+
| File | Reason |
176
+
|------|--------|
177
+
| `idl/transport.ml`, `transport.mli` | JSON-RPC transport wrapper |
178
+
| `idl/js_top_worker_client.ml`, `.mli` | RPC-based Lwt client |
179
+
| `idl/js_top_worker_client_fut.ml` | RPC-based Fut client |
180
+
| `idl/_old/` directory | Historical RPC reference code |
181
+
182
+
### Dependencies to Remove
183
+
184
+
| Package | Used By |
185
+
|---------|---------|
186
+
| `rpclib` | transport.ml, RPC clients |
187
+
| `rpclib-lwt` | impl.ml (IdlM module) |
188
+
| `rpclib.json` | transport.ml, RPC clients |
189
+
| `ppx_deriving_rpc` | toplevel_api.ml code generation |
190
+
| `xmlm` | transitive via rpclib |
191
+
| `cmdliner` | transitive via rpclib |
192
+
193
+
### Files to Refactor
194
+
195
+
| File | Change |
196
+
|------|--------|
197
+
| `lib/impl.ml` | Remove `IdlM` / `Rpc_lwt` usage, use own types |
198
+
| `idl/toplevel_api.ml` | Keep type definitions, remove RPC IDL machinery |
199
+
| `idl/toplevel_api_gen.ml` | Replace with hand-written types (no ppx_deriving_rpc) |
200
+
| `idl/dune` | Remove rpclib library deps and ppx rules |
201
+
| `lib/dune` | Remove rpclib-lwt dep |
202
+
| `test/node/*.ml` | Migrate from RPC Server/Client to message protocol |
203
+
| `test/browser/client_test.ml` | Use message-based client |
204
+
| `example/unix_worker.ml` | Use message protocol over Unix socket |
205
+
| `example/unix_client.ml` | Use message protocol over Unix socket |
206
+
207
+
### Impact on `toplevel_api_gen.ml`
208
+
209
+
The generated file is 92k+ lines (from ppx_deriving_rpc). The types it defines
210
+
are used extensively in `impl.ml` and `worker.ml`. The plan:
211
+
212
+
1. Extract the **type definitions** into a new lightweight module (no ppx)
213
+
2. Hand-write any needed serialization for the message protocol
214
+
3. Remove `ppx_deriving_rpc` dependency entirely
215
+
4. Delete `toplevel_api_gen.ml`
216
+
217
+
## Testing Strategy
218
+
219
+
### Existing Test Infrastructure
220
+
221
+
| Backend | Location | Framework |
222
+
|---------|----------|-----------|
223
+
| Unit tests | `test/libtest/` | ppx_expect |
224
+
| Node.js | `test/node/` | js_of_ocaml + Node |
225
+
| Unix (cram) | `test/cram/` | Cram tests with unix_worker |
226
+
| Browser | `test/browser/` | Playwright + Chromium |
227
+
228
+
### Widget Testing Approach
229
+
230
+
#### 1. Unit Tests (`test/libtest/`)
231
+
232
+
- Widget state management logic
233
+
- Message serialization/deserialization for widget messages
234
+
- CommManager state tracking (open/update/close lifecycle)
235
+
236
+
#### 2. Node.js Tests (`test/node/`)
237
+
238
+
- Widget creation produces correct CommOpen messages
239
+
- Widget state updates produce CommUpdate messages
240
+
- Event handler registration and dispatch
241
+
- Widget close cleanup
242
+
- Multiple simultaneous widgets
243
+
- Widget interaction with regular exec output (mime_vals + widgets)
244
+
245
+
#### 3. Cram Tests (`test/cram/`)
246
+
247
+
- Unix worker handles widget messages over socket
248
+
- Widget lifecycle via command-line client
249
+
- Widget messages interleaved with regular eval output
250
+
251
+
#### 4. Browser Tests (`test/browser/`)
252
+
253
+
- **End-to-end widget rendering**: OCaml creates widget -> message sent ->
254
+
client renders DOM element -> user interaction -> event sent back -> OCaml
255
+
handler fires
256
+
- **Widget types**: Test each widget type (slider, button, text, dropdown,
257
+
checkbox, html)
258
+
- **State synchronization**: Frontend changes propagated to worker and back
259
+
- **Multiple widgets**: Several widgets active simultaneously
260
+
- **Widget cleanup**: Closing widgets removes DOM elements
261
+
- **Integration with existing features**: Widgets alongside code completion,
262
+
error reporting, MIME output
263
+
264
+
### Test Utilities
265
+
266
+
A shared test helper module for widget testing:
267
+
268
+
```ocaml
269
+
(* test/test_widget_helpers.ml *)
270
+
val assert_comm_open : worker_msg -> comm_id:string -> widget_type:string -> unit
271
+
val assert_comm_update : worker_msg -> comm_id:string -> key:string -> unit
272
+
val assert_comm_close : worker_msg -> comm_id:string -> unit
273
+
val simulate_event : comm_id:string -> data:string -> client_msg
274
+
```
275
+
276
+
## Example Widgets
277
+
278
+
### Priority 1: Core Widgets (Implement First)
279
+
280
+
These are the most commonly used Jupyter widgets and cover the fundamental
281
+
interaction patterns:
282
+
283
+
| Widget | State | Events | Jupyter Equivalent |
284
+
|--------|-------|--------|--------------------|
285
+
| IntSlider | value, min, max, step | on_change(int) | IntSlider |
286
+
| Button | description, style | on_click | Button |
287
+
| Text | value, placeholder | on_change(string) | Text |
288
+
| Dropdown | value, options | on_change(string) | Dropdown |
289
+
| Checkbox | value, description | on_change(bool) | Checkbox |
290
+
| HTML | value (html string) | none | HTML |
291
+
292
+
### Priority 2: Composition Widgets
293
+
294
+
| Widget | Purpose | Jupyter Equivalent |
295
+
|--------|---------|-------------------|
296
+
| HBox / VBox | Layout containers | HBox / VBox |
297
+
| Output | Capture stdout/display | Output |
298
+
| FloatSlider | Decimal slider | FloatSlider |
299
+
300
+
### Priority 3: Domain-Specific Widgets
301
+
302
+
| Widget | Purpose | Inspired By |
303
+
|--------|---------|-------------|
304
+
| Plot | Simple 2D charts (SVG) | bqplot (simplified) |
305
+
| Table | Data grid display | ipydatagrid (read-only) |
306
+
| Image | Display image bytes | Image widget |
307
+
308
+
### Example: Interactive Slider
309
+
310
+
```ocaml
311
+
(* User code in toplevel *)
312
+
let threshold = Widget.slider ~min:0 ~max:100 ~description:"Threshold" 50;;
313
+
314
+
Widget.on_change threshold (fun v ->
315
+
let n = Widget.Int.of_json v in
316
+
Printf.printf "Threshold changed to: %d\n" n
317
+
);;
318
+
```
319
+
320
+
### Example: Button with Output
321
+
322
+
```ocaml
323
+
let count = ref 0;;
324
+
let label = Widget.html (Printf.sprintf "<b>Count: %d</b>" !count);;
325
+
let btn = Widget.button "Increment";;
326
+
327
+
Widget.on_click btn (fun () ->
328
+
incr count;
329
+
Widget.set label "value"
330
+
(`String (Printf.sprintf "<b>Count: %d</b>" !count))
331
+
);;
332
+
```
333
+
334
+
### Example: Linked Widgets
335
+
336
+
```ocaml
337
+
let slider = Widget.slider ~min:0 ~max:255 ~description:"Red" 128;;
338
+
let preview = Widget.html {|<div style="width:50px;height:50px"></div>|};;
339
+
340
+
Widget.on_change slider (fun v ->
341
+
let r = Widget.Int.of_json v in
342
+
Widget.set preview "value"
343
+
(`String (Printf.sprintf
344
+
{|<div style="width:50px;height:50px;background:rgb(%d,0,0)"></div>|} r))
345
+
);;
346
+
```
347
+
348
+
## Implementation Phases
349
+
350
+
### Phase 1: RPC Removal & Type Cleanup
351
+
352
+
Remove the legacy RPC layer and establish clean type definitions.
353
+
354
+
### Phase 2: Widget Message Protocol
355
+
356
+
Extend `message.ml` and `worker.ml` with widget message types.
357
+
358
+
### Phase 3: Widget Manager (Worker Side)
359
+
360
+
Implement the comm manager that tracks widget state and routes events.
361
+
362
+
### Phase 4: OCaml Widget API
363
+
364
+
Create the user-facing `Widget` module available in the toplevel.
365
+
366
+
### Phase 5: JavaScript Widget Renderer
367
+
368
+
Implement widget rendering in the JavaScript client.
369
+
370
+
### Phase 6: Testing & Examples
371
+
372
+
Full test coverage across all backends, example widgets, documentation.
373
+
374
+
## Open Questions
375
+
376
+
1. **JSON representation in OCaml API**: Use `Yojson.Safe.t`? A custom minimal
377
+
JSON type? Raw `Js.Unsafe.any`? (Yojson adds a dependency; custom type is
378
+
more work; Unsafe.any is untyped.)
379
+
380
+
2. **Widget library loading**: Should the Widget module be preloaded in the
381
+
worker, or loaded on demand via `#require "widget"`?
382
+
383
+
3. **Layout model**: How much of Jupyter's CSS-based layout model to support?
384
+
Full flexbox control per-widget, or simpler HBox/VBox only?
385
+
386
+
4. **Persistence**: Should widget state survive cell re-execution? Jupyter
387
+
widgets are destroyed and recreated; we could do the same or preserve state.