Monorepo for Aesthetic.Computer
aesthetic.computer
1# Stample ↔ Pixel Store ↔ Share Pipeline Unification Plan
2
3## Problem
4
5The pixel↔audio conversion pipeline is fragmented across three pieces with duplicated code:
6
71. **`stample.mjs`** — has the canonical `encodeSampleToBitmap()`, `decodeBitmapToSample()`, `imageToBuffer()`, and `loadPaintingCode()` / `loadSystemPainting()` functions, but they're all **local/unexported**
82. **`clock.mjs`** — copy-pasted `decodeBitmapToSample()` and `imageToBuffer()` (comment: "adapted from stample.mjs")
93. **`notepat.mjs`** — can play stample samples but has **no way to load paintings as samples** (no `stample p` param handler, no painting→sample pipeline, no decode functions)
10
11### What Should Work But Doesn't
12
13- `notepat:stample` sets wave type to stample, but only plays whatever is in `store["stample:sample"]`
14- `notepat:stample p` or `notepat:stample $roz` — no param handler exists to load a painting or KidLisp as a sample
15- The `picture` buffer in notepat accumulates note colors but is never connected to audio
16- KidLisp visualization in notepat renders to the screen but never feeds into the sample buffer
17
18---
19
20## Architecture
21
22### Current Flow (Fragmented)
23
24```
25stample.mjs notepat.mjs clock.mjs
26┌─────────────────────┐ ┌──────────────────┐ ┌──────────────────┐
27│ encodeSampleToBitmap│ │ store.retrieve │ │ decodeBitmapTo │
28│ decodeBitmapToSample│ │ ("stample:sample")│ │ Sample (COPY) │
29│ imageToBuffer │ │ → registerSample │ │ imageToBuffer │
30│ loadPaintingCode │ │ │ │ (COPY) │
31│ loadSystemPainting │ │ No painting→audio│ └──────────────────┘
32│ │ │ No KidLisp→audio │
33│ $code → pixels → │ │ No #code loading │
34│ decode → register │ └──────────────────┘
35└─────────────────────┘
36```
37
38### Target Flow (Unified)
39
40```
41lib/pixel-sample.mjs (NEW shared module)
42┌─────────────────────────────────────────────────────┐
43│ encodeSampleToBitmap(data, width) │
44│ decodeBitmapToSample(bitmap, meta) │
45│ imageToBuffer(image) │
46│ loadPaintingAsAudio(source, {sound, preload, store}) │
47│ → handles: #code, $kidlisp, "p"/painting, URL │
48│ → returns {sampleData, sampleId, bitmap, meta} │
49└─────────────────────────────────────────────────────┘
50 ↓ imported by
51 stample.mjs notepat.mjs clock.mjs
52```
53
54---
55
56## Step-by-Step Plan
57
58### Step 1: Extract shared module `lib/pixel-sample.mjs`
59
60Create `/system/public/aesthetic.computer/lib/pixel-sample.mjs` with:
61
62```js
63// 3 audio samples per pixel (R, G, B channels)
64export function encodeSampleToBitmap(data, width = 256) { ... }
65export function decodeBitmapToSample(bitmap, meta) { ... }
66export async function imageToBuffer(image) { ... }
67```
68
69These are verbatim copies from `stample.mjs` lines 1300-1537, now exported.
70
71### Step 2: Extract `loadPaintingAsAudio()` into the shared module
72
73Generalize `loadPaintingCode()` and `loadSystemPainting()` into a single function:
74
75```js
76/**
77 * Load any pixel source as a playable audio sample.
78 * @param {string|object} source - "#code", "$kidlisp", "p"/"painting", or a bitmap object
79 * @param {object} opts - { sound, preload, store, system, get, painting, kidlisp }
80 * @returns {{ sampleData, sampleId, bitmap, meta } | null}
81 */
82export async function loadPaintingAsAudio(source, opts) {
83 const sampleId = "stample:bitmap";
84
85 if (typeof source === "string") {
86 if (source.startsWith("$")) {
87 // KidLisp: render to pixel buffer, decode to audio
88 // Use opts.painting() + opts.kidlisp() to render
89 } else if (source.startsWith("#") || source.startsWith("%23")) {
90 // Painting code: fetch from /media/paintings/ or /api/painting-code
91 // Reuse loadPaintingCode logic
92 } else if (source === "p" || source === "painting") {
93 // System painting: from nopaint buffer or store
94 // Reuse loadSystemPainting logic
95 }
96 } else if (source?.pixels) {
97 // Direct bitmap object
98 }
99
100 if (!bitmap?.pixels?.length) return null;
101
102 const totalPixels = bitmap.width * bitmap.height;
103 const meta = { sampleLength: totalPixels * 3, sampleRate: opts.sound?.sampleRate || 48000 };
104 const sampleData = decodeBitmapToSample(bitmap, meta);
105
106 if (sampleData?.length) {
107 opts.sound?.registerSample?.(sampleId, sampleData, meta.sampleRate);
108 }
109
110 return { sampleData, sampleId, bitmap, meta };
111}
112```
113
114### Step 3: Update `stample.mjs` to import from shared module
115
116Replace local functions with imports:
117
118```js
119import {
120 encodeSampleToBitmap,
121 decodeBitmapToSample,
122 imageToBuffer,
123 loadPaintingAsAudio,
124} from "../lib/pixel-sample.mjs";
125```
126
127Remove the local copies (~200 lines). Update `loadPaintingCode` and `loadSystemPainting` callers to use `loadPaintingAsAudio()`. The KidLisp sim loop still calls `decodeBitmapToSample` directly for live frame-by-frame updates.
128
129### Step 4: Update `clock.mjs` to import from shared module
130
131Replace the copy-pasted functions:
132
133```js
134import { decodeBitmapToSample, imageToBuffer } from "../lib/pixel-sample.mjs";
135```
136
137Remove local copies (~70 lines).
138
139### Step 5: Add painting→sample pipeline to `notepat.mjs`
140
141**5a. Import the shared module:**
142
143```js
144import {
145 decodeBitmapToSample,
146 imageToBuffer,
147 loadPaintingAsAudio,
148} from "../lib/pixel-sample.mjs";
149```
150
151**5b. Handle `stample` params in boot:**
152
153After the existing `params[0] === "piano"` / `params[0] === "twinkle"` checks (~line 1252), add:
154
155```js
156// Handle stample with painting source: notepat:stample:p, notepat:stample:#code, notepat:stample:$kidlisp
157if (wave === "stample" || requestedWave === "stample" || requestedWave === "sample") {
158 const stampleSource = colon[1]; // e.g. "p", "#abc123", "$roz"
159 if (stampleSource) {
160 const result = await loadPaintingAsAudio(stampleSource, {
161 sound, preload, store, system, get, painting, kidlisp,
162 });
163 if (result) {
164 stampleSampleId = result.sampleId;
165 stampleSampleData = result.sampleData;
166 stampleSampleRate = result.meta.sampleRate;
167 }
168 }
169}
170```
171
172This makes these paths work:
173- `notepat:stample:p` — load system painting as sample
174- `notepat:stample:#abc123` — load painting by code as sample
175- `notepat:stample:$roz` — load KidLisp as sample
176
177**5c. Connect KidLisp visualization to sample buffer (live):**
178
179When `kidlispBgEnabled` is true and wave is `stample`, feed the kidlisp pixel buffer into the sample system each frame (in `sim`), similar to stample.mjs lines 514-597:
180
181```js
182// In sim(), after the kidlisp background update:
183if (kidlispBgEnabled && (wave === "stample" || wave === "sample") && kidlispBackground) {
184 // Render kidlisp to a small buffer and decode as audio
185 const bufferSize = 128;
186 const lispPainting = painting(bufferSize, bufferSize, (paintApi) => {
187 paintApi.kidlisp(0, 0, bufferSize, bufferSize, kidlispBackground);
188 });
189 if (lispPainting?.pixels?.length) {
190 const totalPixels = bufferSize * bufferSize;
191 const meta = { sampleLength: totalPixels * 3, sampleRate: sound?.sampleRate || 48000 };
192 const decoded = decodeBitmapToSample(lispPainting, meta);
193 if (decoded?.length) {
194 sound.updateSample?.("stample:bitmap", decoded, meta.sampleRate);
195 stampleSampleId = "stample:bitmap";
196 stampleSampleData = decoded;
197 }
198 }
199}
200```
201
202**5d. Connect `picture` buffer to sample (optional/future):**
203
204The `picture` buffer accumulates note-color stamps. It could optionally feed the stample system, making the visual history playable:
205
206```js
207// When picture buffer is updated and wave is stample, re-encode to audio
208if (wave === "stample" && picture?.pixels?.length) {
209 const meta = { sampleLength: picture.width * picture.height * 3, sampleRate: 48000 };
210 const decoded = decodeBitmapToSample(picture, meta);
211 if (decoded?.length) {
212 sound.updateSample?.("stample:bitmap", decoded, meta.sampleRate);
213 }
214}
215```
216
217This is a stretch goal — the picture buffer is low-res and would produce very short samples. Worth experimenting with but not essential for initial unification.
218
219### Step 6: Store pipeline consistency
220
221Ensure all three pieces use the same store keys:
222- `"stample:sample"` — raw audio sample data (Float32Array + sampleRate)
223- `"stample:bitmap"` — pixel buffer + meta (width, height, pixels, sampleLength, sampleRate)
224
225Notepat currently only reads `"stample:sample"`. After unification, it should also read/write `"stample:bitmap"` when loading from painting sources.
226
227---
228
229## Files Modified
230
231| File | Change |
232|------|--------|
233| `lib/pixel-sample.mjs` | **NEW** — shared encode/decode/load functions |
234| `disks/stample.mjs` | Replace local functions with imports (~200 lines removed) |
235| `disks/clock.mjs` | Replace copy-pasted functions with imports (~70 lines removed) |
236| `disks/notepat.mjs` | Add import, painting param handling, optional live kidlisp→sample |
237
238## Verification
239
2401. **`stample p`** — loads system painting as sample, plays with pitch control
2412. **`stample #abc123`** — loads painting by code, plays as before
2423. **`stample $roz`** — KidLisp renders live to audio, plays as before
2434. **`notepat:stample:p`** — loads painting into notepat's stample mode
2445. **`notepat:stample:$roz`** — KidLisp feeds sample buffer in notepat
2456. **`notepat:stample`** — fallback to stored sample (existing behavior preserved)
2467. **`clock.mjs`** stample features — unchanged behavior, now using shared module
2478. **Tab through wavetypes** — stample mode still cycles correctly
2489. **Recording in stample** — mic → encode → store still works
249
250## Risk Assessment
251
252- **Low risk**: Extracting functions to shared module is mechanical (identical code)
253- **Medium risk**: `loadPaintingAsAudio` generalization — need to ensure all edge cases from both `loadPaintingCode` and `loadSystemPainting` are covered
254- **Low risk**: notepat param handling — additive change, doesn't modify existing paths
255- **Higher risk**: Live KidLisp→sample in notepat sim loop — performance-sensitive, may need throttling (stample does it every frame at 128x128)