Monorepo for Aesthetic.Computer
aesthetic.computer
notepat-remote: Ableton-ish orange palette + loud unfocused TAP ME overlay; add usb-midi-gadget plan
+318
-48
+1
-1
ac-m4l/AC-NotepatRemote.amxd.json
+1
-1
ac-m4l/AC-NotepatRemote.amxd.json
+228
plans/usb-midi-gadget.md
+228
plans/usb-midi-gadget.md
···
1
+
# USB MIDI Gadget for ac-native → Ableton direct
2
+
3
+
> Status: **plan / unscoped**. Lets the ac-native ThinkPad present itself
4
+
> to the MacBook (running Ableton) as a USB MIDI controller, bypassing the
5
+
> session-server WebSocket relay entirely for the lowest possible
6
+
> ThinkPad→Ableton latency.
7
+
8
+
## Motivation
9
+
10
+
Current path: `notepat.mjs (ThinkPad) → UDP → session-server.aesthetic.computer:10010 → WS fanout → notepat-remote.amxd (Mac) → Operator`. That's a round trip over public internet. Observed latency is fine for async use but **40-100ms** is unavoidable even on good wifi.
11
+
12
+
Direct USB MIDI gets us:
13
+
14
+
- **~1-3ms** ThinkPad key → MacBook MIDI-in (USB controller polling + kernel hop)
15
+
- No internet dependency, no session-server required
16
+
- No WS subscriptions, no handle/machineId negotiation
17
+
- Ableton sees it as a native MIDI controller — appears in MIDI input list, can be mapped, recorded, etc.
18
+
19
+
The session-server relay can stay as a **networked** backup / for multi-ThinkPad scenarios (one ThinkPad playing into multiple remote DAWs), but the USB path becomes the default for a local setup.
20
+
21
+
## Architecture
22
+
23
+
```
24
+
┌───────────────────────────┐ USB-C ┌──────────────────────┐
25
+
│ ThinkPad (ac-native OS) │ ═════════▶ │ MacBook (Ableton) │
26
+
│ Linux g_midi gadget │ │ CoreMIDI auto-mount │
27
+
│ /dev/snd/midiCxDy │ │ "Linux USB MIDI" │
28
+
│ ↑ notepat.mjs writes │ │ → MIDI input list │
29
+
│ raw MIDI bytes │ │ → Ableton track │
30
+
└───────────────────────────┘ └──────────────────────┘
31
+
```
32
+
33
+
`notepat.mjs` on ac-native gains a new output sink alongside the existing UDP→session-server path:
34
+
35
+
```
36
+
keypress → playSoundKey → [ UDP relay | USB MIDI | USB audio synth | ... ]
37
+
```
38
+
39
+
Configurable at runtime via `/mnt/config.json` (new key `usbMidiGadget: true`). Both paths can be on simultaneously — it's fire-and-forget MIDI data, the ingest endpoints are independent.
40
+
41
+
## Hardware prerequisites
42
+
43
+
### ThinkPad 11e Yoga Gen 6
44
+
45
+
The USB-C port must support **device/OTG mode** in its controller. Most 8th-gen Intel and newer support this through the Thunderbolt PHY, but it needs to be enabled in:
46
+
47
+
1. **BIOS/UEFI**: look for "USB-C device mode", "Thunderbolt configuration", or similar. On Lenovo ThinkPads this is usually under "Config → USB" or "Thunderbolt(TM) 3".
48
+
2. **Kernel**: `CONFIG_USB_GADGET=y`, `CONFIG_USB_GADGETFS=y`, `CONFIG_USB_CONFIGFS=y`, `CONFIG_USB_CONFIGFS_F_MIDI=y`. Verify with `zcat /proc/config.gz | grep GADGET` on the running kernel.
49
+
3. **UDC driver**: the "USB Device Controller" needs a loaded driver. On Intel platforms this is often `dwc3` or `xhci` with gadget support. Check `ls /sys/class/udc/` — there should be at least one entry when device mode is active.
50
+
51
+
**If the ThinkPad's USB-C is host-only**, the fallback is a USB device such as a **Raspberry Pi Zero W / 2 W** or **BeagleBone** tethered to the ThinkPad via network/serial, acting as the gadget. Out of scope for this plan.
52
+
53
+
### MacBook
54
+
55
+
No setup required. macOS auto-detects USB MIDI devices on plug-in and exposes them through CoreMIDI. They show up in:
56
+
57
+
- Audio MIDI Setup → Window → Show MIDI Studio
58
+
- Ableton → Preferences → Link Tempo MIDI → MIDI Ports → check "Track" for the gadget
59
+
60
+
### Cable
61
+
62
+
USB-C to USB-C (or USB-C to USB-A with correct gadget direction). Thunderbolt cables work but are overkill — a standard USB 2.0 data cable is plenty for MIDI (31.25 kbps).
63
+
64
+
## Linux gadget setup (ac-native)
65
+
66
+
### Option A — legacy `g_midi` driver (simplest)
67
+
68
+
```bash
69
+
modprobe g_midi \
70
+
iProduct="AC Notepat" \
71
+
iManufacturer="Aesthetic Computer" \
72
+
id="AC_NOTEPAT"
73
+
```
74
+
75
+
Creates `/dev/snd/midiC<N>D0` and shows up on the Mac as "AC Notepat".
76
+
77
+
### Option B — configfs composite gadget (more control)
78
+
79
+
Scripted at boot time so it survives reboots and can be composed with
80
+
other gadget functions later (e.g. serial console).
81
+
82
+
```bash
83
+
#!/bin/sh
84
+
# /usr/local/bin/ac-usb-gadget-start
85
+
86
+
set -e
87
+
GADGET=/sys/kernel/config/usb_gadget/ac_notepat
88
+
mkdir -p "$GADGET"
89
+
90
+
echo 0x1d6b > "$GADGET/idVendor" # Linux Foundation
91
+
echo 0x0104 > "$GADGET/idProduct" # Multifunction Composite Gadget
92
+
echo 0x0100 > "$GADGET/bcdDevice"
93
+
echo 0x0200 > "$GADGET/bcdUSB"
94
+
95
+
mkdir -p "$GADGET/strings/0x409"
96
+
echo "ACNP-$(cat /etc/machine-id | cut -c1-8)" > "$GADGET/strings/0x409/serialnumber"
97
+
echo "Aesthetic Computer" > "$GADGET/strings/0x409/manufacturer"
98
+
echo "AC Notepat" > "$GADGET/strings/0x409/product"
99
+
100
+
mkdir -p "$GADGET/configs/c.1/strings/0x409"
101
+
echo "MIDI config" > "$GADGET/configs/c.1/strings/0x409/configuration"
102
+
echo 250 > "$GADGET/configs/c.1/MaxPower"
103
+
104
+
mkdir -p "$GADGET/functions/midi.usb0"
105
+
echo 1 > "$GADGET/functions/midi.usb0/in_ports"
106
+
echo 1 > "$GADGET/functions/midi.usb0/out_ports"
107
+
echo 64 > "$GADGET/functions/midi.usb0/buflen" # small buffer = low latency
108
+
echo 32 > "$GADGET/functions/midi.usb0/qlen"
109
+
110
+
ln -s "$GADGET/functions/midi.usb0" "$GADGET/configs/c.1/"
111
+
112
+
# Bind to the first available UDC. ls /sys/class/udc shows the devices.
113
+
UDC=$(ls /sys/class/udc | head -n1)
114
+
echo "$UDC" > "$GADGET/UDC"
115
+
```
116
+
117
+
Run at boot via an ac-native init hook. A matching teardown script writes empty to `UDC` and `rmdir`s the tree.
118
+
119
+
### Verifying on the Mac
120
+
121
+
With the cable plugged in:
122
+
123
+
```bash
124
+
system_profiler SPUSBDataType | grep -A 4 "AC Notepat"
125
+
# Should list it as a USB device.
126
+
127
+
# In Audio MIDI Setup app: "AC Notepat" appears with a MIDI icon.
128
+
```
129
+
130
+
In Ableton → Preferences → Link Tempo MIDI → MIDI Ports, the new input appears. Enable "Track" to make it a note source, or "Remote" to map its controls.
131
+
132
+
## ac-native integration
133
+
134
+
### New code in `fedac/native/src/`
135
+
136
+
`usb-midi.c` (new file):
137
+
138
+
```c
139
+
int usb_midi_open(const char *device); // opens /dev/snd/midiC0D0
140
+
void usb_midi_close(int fd);
141
+
int usb_midi_send_note_on(int fd, int channel, int pitch, int velocity);
142
+
int usb_midi_send_note_off(int fd, int channel, int pitch);
143
+
```
144
+
145
+
Uses raw ALSA MIDI bytes:
146
+
147
+
- note-on: `0x90 | channel, pitch, velocity` (3 bytes)
148
+
- note-off: `0x80 | channel, pitch, 0`
149
+
150
+
`write(fd, buf, 3)` on the MIDI char device. Non-blocking mode preferred so a stalled receiver doesn't wedge the audio thread.
151
+
152
+
### JS bindings (`fedac/native/src/js-bindings.c`)
153
+
154
+
Mirror the existing `system.udp.sendMidi` helpers:
155
+
156
+
```js
157
+
system.usbGadget.open() // → bool
158
+
system.usbGadget.sendMidi(event, note, vel, ch)
159
+
system.usbGadget.close()
160
+
system.usbGadget.status // { connected, device, bytesSent }
161
+
```
162
+
163
+
### notepat.mjs wiring
164
+
165
+
At the existing `sendUdpMidiEvent(…)` call site (around `fedac/native/pieces/notepat.mjs:1847-1850`), also emit USB MIDI when enabled:
166
+
167
+
```js
168
+
const usbMidiGadgetEnabled =
169
+
cfg.usbMidiGadget === true || cfg.usbMidiGadget === "true";
170
+
171
+
function sendMidiEvent(system, event, midiNote, velocity, channel = 0) {
172
+
// Existing: UDP relay
173
+
if (udpMidiBroadcast && system?.udp?.connected) {
174
+
system.udp.sendMidi(event, midiNote, velocity, channel, "notepat");
175
+
}
176
+
// NEW: USB gadget direct to Mac
177
+
if (usbMidiGadgetEnabled && system?.usbGadget?.status?.connected) {
178
+
system.usbGadget.sendMidi(event, midiNote, velocity, channel);
179
+
}
180
+
// Existing telemetry counters
181
+
udpMidiSentCount += 1;
182
+
…
183
+
}
184
+
```
185
+
186
+
### Prompt command
187
+
188
+
Add `usb midi on/off/status` to `prompt.mjs`, parallel to the existing `midi relay on/off` command. Persists the flag to `/mnt/config.json`.
189
+
190
+
## Expected latency budget
191
+
192
+
| Stage | Cost |
193
+
|---|---|
194
+
| ThinkPad key press → notepat handler | <1 ms |
195
+
| JS → C `sendMidi()` binding | <0.1 ms |
196
+
| `write()` → kernel USB gadget | <0.2 ms |
197
+
| USB bus traversal (2× full-speed frame ≈ 1 ms polling interval) | ~1-2 ms |
198
+
| MacBook USB host → CoreMIDI callback | <0.5 ms |
199
+
| Ableton MIDI-in → track → instrument | <1 ms |
200
+
| **Audio buffer output** (Live at 64 samples / 48 kHz) | ~1.3 ms |
201
+
| **Total keypress → audible** | **~5-7 ms** |
202
+
203
+
Versus current session-server path (~40-100 ms), that's **~10-15× faster**.
204
+
205
+
## Out of scope / follow-ups
206
+
207
+
- **Bi-directional MIDI**: this plan is one-way ThinkPad→Mac. Receiving MIDI from Ableton back to ac-native (e.g., for playback-synced visuals) needs a `read()` loop on the same gadget endpoint. Easy to add.
208
+
- **MIDI clock sync**: ac-native can emit `0xF8` every 24 PPQ to sync Ableton's transport.
209
+
- **Multi-channel**: currently all notes fire on channel 0. Exposing channel selection in notepat's UI is trivial once the gadget is up.
210
+
- **Power**: the ThinkPad draws from its own battery; USB in device mode doesn't supply power to the Mac. Bus power is one-directional.
211
+
- **SysEx / MPE**: raw `write()` handles arbitrary MIDI bytes — both just work.
212
+
213
+
## Test plan
214
+
215
+
1. On the ThinkPad: `dmesg | grep gadget` after `g_midi` modprobe — expect "using random self ethernet address" + MIDI gadget init lines.
216
+
2. `aconnect -l` lists the gadget port.
217
+
3. `amidi -l` shows the device with its hw:N,M address.
218
+
4. `amidi -p hw:N,M -S '90 3C 7F'` plays a C4 note-on — verify by ear through Ableton after enabling the MIDI input.
219
+
5. Wire into `notepat.mjs`, rebuild ac-native, flash to ThinkPad.
220
+
6. Plug into Mac. Enable in Live's MIDI preferences.
221
+
7. Press keys in ac-native notepat — notes should fire instantly on the Mac.
222
+
8. A/B against the session-server relay path (`midi relay on`) to confirm the latency improvement.
223
+
224
+
## Why not this?
225
+
226
+
- If the ThinkPad's USB-C can't do device mode (BIOS or silicon limitation), fall back to a USB-serial adapter or a Pi Zero bridge.
227
+
- If you're primarily using ac-native remotely (not physically next to the Mac), keep the session-server path — USB obviously requires a cable.
228
+
- The M4L `notepat-remote.amxd` device is still useful either way: on USB MIDI the AC-native-sourced notes land on *any* MIDI track directly, no device needed; on network relay the device is required to subscribe + route.
+89
-47
system/public/aesthetic.computer/disks/notepat-remote.mjs
+89
-47
system/public/aesthetic.computer/disks/notepat-remote.mjs
···
267
267
const W = screen.width;
268
268
const H = screen.height;
269
269
270
-
// Attract mode: blink brightness between 0.4 and 1.
271
-
const blinkPhase = (sin(frame * 0.12) + 1) / 2; // 0..1
272
-
const attractPulse = focused ? 1 : 0.5 + blinkPhase * 0.5;
270
+
// ── Palette ──────────────────────────────────────────────────────────
271
+
// Ableton-tasteful: graphite background, amber/orange accent (Live's
272
+
// MIDI color), muted grays. Unfocused = deeply dimmed + angry red blink
273
+
// so you can't miss it.
274
+
const blinkPhase = (sin(frame * 0.14) + 1) / 2; // 0..1 sinusoidal
275
+
const blinkOn = blinkPhase > 0.5;
273
276
274
-
const accent = focused
275
-
? [floor(130 + blinkPhase * 40), 255, floor(170 + blinkPhase * 50)] // lime
276
-
: [255, floor(70 + blinkPhase * 100), floor(70 + blinkPhase * 40)]; // red
277
-
const dim = [130, 140, 170];
278
-
const fg = [220, 225, 255];
279
-
const bgBase = focused ? [10, 20, 18] : [22, 10, 10];
280
-
wipe(
281
-
floor(bgBase[0] * attractPulse),
282
-
floor(bgBase[1] * attractPulse),
283
-
floor(bgBase[2] * attractPulse),
284
-
);
277
+
// Focused theme (always-on)
278
+
const focusedBg = [16, 18, 22];
279
+
const focusedAccent = [255, 156, 60]; // Live orange
280
+
const focusedAccentBright = [255, 196, 110];
281
+
const focusedFg = [212, 216, 224];
282
+
const focusedDim = [110, 116, 130];
283
+
const focusedKeyWhite = [38, 42, 50];
284
+
const focusedKeyBlack = [22, 24, 30];
285
+
const focusedOutline = [70, 76, 88];
286
+
287
+
// Unfocused theme — dark + red; flashes for attention.
288
+
const unfocusedBg = blinkOn ? [48, 12, 12] : [22, 6, 6];
289
+
const unfocusedAccent = blinkOn ? [255, 70, 70] : [180, 45, 45];
290
+
const unfocusedFg = [180, 150, 150];
291
+
const unfocusedDim = [100, 70, 70];
292
+
const unfocusedKeyWhite = [40, 18, 18];
293
+
const unfocusedKeyBlack = [24, 10, 10];
294
+
const unfocusedOutline = [80, 30, 30];
295
+
296
+
const bgBase = focused ? focusedBg : unfocusedBg;
297
+
const accent = focused ? focusedAccent : unfocusedAccent;
298
+
const accentBright = focused ? focusedAccentBright : unfocusedAccent;
299
+
const fg = focused ? focusedFg : unfocusedFg;
300
+
const dim = focused ? focusedDim : unfocusedDim;
301
+
const keyWhite = focused ? focusedKeyWhite : unfocusedKeyWhite;
302
+
const keyBlack = focused ? focusedKeyBlack : unfocusedKeyBlack;
303
+
const outline = focused ? focusedOutline : unfocusedOutline;
304
+
305
+
wipe(...bgBase);
285
306
286
307
const sinceNote = frame - lastNoteFrame;
287
-
// Flash overlay on recent note-on.
288
-
if (lastNote && sinceNote < 10) {
308
+
if (lastNote && sinceNote < 10 && focused) {
289
309
const f = 1 - sinceNote / 10;
290
-
const alpha = floor(45 * f);
291
-
ink(...accent, alpha).box(0, 0, W, H, "fill");
310
+
ink(...accentBright, floor(40 * f)).box(0, 0, W, H, "fill");
292
311
}
293
312
294
-
// ── Header row: piece name + ws state
313
+
// ── Header row: piece name + ws state ─────────────────────────────────
295
314
let y = 2;
296
315
ink(...accent).write("notepat-remote", { x: 4, y });
297
316
const wsColor =
298
-
wsState === "open" ? [140, 255, 180] :
299
-
wsState === "connecting" ? [255, 220, 100] :
300
-
wsState === "error" || wsState === "closed" ? [255, 120, 120] : dim;
317
+
!focused ? dim :
318
+
wsState === "open" ? [120, 220, 140] :
319
+
wsState === "connecting" ? [255, 200, 90] :
320
+
wsState === "error" || wsState === "closed" ? [255, 100, 100] : dim;
301
321
ink(...wsColor).write(wsState, { x: W - wsState.length * 6 - 4, y });
302
322
y += 10;
303
323
304
-
// ── Status row: ACTIVE / TAP ME + octave + last note
324
+
// ── Status row: ACTIVE + octave + last note ──────────────────────────
305
325
if (focused) {
306
326
ink(...accent).box(4, y + 1, 6, 6, "fill");
307
327
ink(...fg).write("ACTIVE", { x: 14, y });
308
-
} else {
309
-
const blinkOn = blinkPhase > 0.35;
310
-
ink(blinkOn ? 255 : 110, blinkOn ? 40 : 20, blinkOn ? 40 : 20)
311
-
.write("TAP ME!", { x: 4, y });
312
328
}
313
329
ink(...dim).write(`oct ${baseOctave}`, { x: 70, y });
314
-
// Right side: compact last-note readout
315
330
if (lastNote) {
316
331
const noteFresh = sinceNote < 30;
317
332
const pn = pitchName(lastNote.pitch);
318
-
const noteColor = noteFresh ? accent : fg;
333
+
const noteColor = noteFresh ? accent : dim;
319
334
const srcTag =
320
335
lastNote.source === "relay" ? "@" + (lastNote.handle || "?") :
321
336
lastNote.source === "tap" ? "tap" : "kbd";
···
324
339
}
325
340
y += 10;
326
341
327
-
// ── Button grid area: two 4×3 octave blocks side-by-side
342
+
// ── Button grid area: two 4×3 octave blocks side-by-side ─────────────
328
343
const gridTop = y + 2;
329
-
const gridBottom = H - 4; // leave 4px safe margin at bottom
344
+
const gridBottom = H - 4;
330
345
const gap = 6;
331
346
const blockW = floor((W - 8 - gap) / 2);
332
347
const blockH = gridBottom - gridTop;
···
337
352
for (let octIdx = 0; octIdx < OCTAVE_GRIDS.length; octIdx += 1) {
338
353
const grid = OCTAVE_GRIDS[octIdx];
339
354
const blockX = 4 + octIdx * (blockW + gap);
340
-
// Octave number label in the corner of each block
341
355
const octNum = baseOctave + octIdx;
342
356
ink(...dim).write(`o${octNum}`, { x: blockX + 1, y: gridTop - 1 });
343
357
···
364
378
const black = isBlackKey(pitch);
365
379
366
380
let fill;
367
-
if (held) {
381
+
if (held && focused) {
368
382
fill = accent;
369
-
} else if (recentFlash) {
383
+
} else if (recentFlash && focused) {
370
384
const f = 1 - sinceNote / 18;
371
385
fill = [
372
-
floor(bgBase[0] + (accent[0] - bgBase[0]) * f * 0.6),
373
-
floor(bgBase[1] + (accent[1] - bgBase[1]) * f * 0.6),
374
-
floor(bgBase[2] + (accent[2] - bgBase[2]) * f * 0.6),
386
+
floor(bgBase[0] + (accent[0] - bgBase[0]) * f * 0.5),
387
+
floor(bgBase[1] + (accent[1] - bgBase[1]) * f * 0.5),
388
+
floor(bgBase[2] + (accent[2] - bgBase[2]) * f * 0.5),
375
389
];
376
-
} else if (black) {
377
-
fill = focused ? [22, 32, 28] : [48, 18, 18];
378
390
} else {
379
-
fill = focused ? [42, 52, 48] : [70, 28, 28];
391
+
fill = black ? keyBlack : keyWhite;
380
392
}
381
393
ink(...fill).box(b.x, b.y, b.w, b.h, "fill");
382
-
ink(...(held ? accent : [80, 90, 100])).box(b.x, b.y, b.w, b.h, "outline");
394
+
ink(...(held && focused ? accent : outline))
395
+
.box(b.x, b.y, b.w, b.h, "outline");
383
396
384
-
// Letter centered in cell
385
397
const labelX = b.x + floor(b.w / 2) - 2;
386
398
const labelY = b.y + floor(b.h / 2) - 4;
387
-
const labelColor = held
388
-
? [10, 20, 10]
389
-
: black
390
-
? [200, 210, 220]
391
-
: fg;
399
+
const labelColor =
400
+
held && focused ? [10, 16, 10] :
401
+
black ? [200, 210, 220] : fg;
392
402
ink(...labelColor).write(key.toUpperCase(), { x: labelX, y: labelY });
393
403
}
394
404
}
405
+
}
406
+
407
+
// ── Unfocused overlay: big blinking "TAP ME!" over everything ────────
408
+
if (!focused) {
409
+
// Semi-opaque scrim so the grid visibly darkens
410
+
ink(4, 0, 0, 160).box(0, 0, W, H, "fill");
411
+
412
+
// Thick pulsing red border that can't be missed
413
+
const borderAlpha = floor(140 + blinkPhase * 115);
414
+
for (let i = 0; i < 3; i += 1) {
415
+
ink(255, 40, 40, borderAlpha).box(i, i, W - i * 2, H - i * 2, "outline");
416
+
}
417
+
418
+
// Huge "TAP ME!" centered in the device
419
+
const msg = "TAP ME!";
420
+
const msgSize = 2; // AC text scaling
421
+
const charW = 6 * msgSize;
422
+
const charH = 10 * msgSize;
423
+
const msgW = msg.length * charW;
424
+
const msgX = floor((W - msgW) / 2);
425
+
const msgY = floor((H - charH) / 2) - 4;
426
+
// Drop shadow
427
+
ink(0, 0, 0, 180).write(msg, { x: msgX + 2, y: msgY + 2, size: msgSize });
428
+
ink(255, blinkOn ? 80 : 40, blinkOn ? 80 : 40)
429
+
.write(msg, { x: msgX, y: msgY, size: msgSize });
430
+
431
+
// Subtitle — smaller, italic-y hint
432
+
const sub = "click me to play";
433
+
const subX = floor((W - sub.length * 6) / 2);
434
+
const subY = msgY + charH + 4;
435
+
ink(blinkOn ? 220 : 140, 120, 120)
436
+
.write(sub, { x: subX, y: subY });
395
437
}
396
438
}
397
439
system/public/m4l/notepat-remote.amxd
system/public/m4l/notepat-remote.amxd
This is a binary file and will not be displayed.