Stample ↔ Pixel Store ↔ Share Pipeline Unification Plan#

Problem#

The pixel↔audio conversion pipeline is fragmented across three pieces with duplicated code:

1. stample.mjs — has the canonical encodeSampleToBitmap(), decodeBitmapToSample(), imageToBuffer(), and loadPaintingCode() / loadSystemPainting() functions, but they're all local/unexported
2. clock.mjs — copy-pasted decodeBitmapToSample() and imageToBuffer() (comment: "adapted from stample.mjs")
3. notepat.mjs — can play stample samples but has no way to load paintings as samples (no stample p param handler, no painting→sample pipeline, no decode functions)

What Should Work But Doesn't#

• notepat:stample sets wave type to stample, but only plays whatever is in store["stample:sample"]
• notepat:stample p or notepat:stample $roz — no param handler exists to load a painting or KidLisp as a sample
• The picture buffer in notepat accumulates note colors but is never connected to audio
• KidLisp visualization in notepat renders to the screen but never feeds into the sample buffer

Architecture#

Current Flow (Fragmented)#

stample.mjs                          notepat.mjs                    clock.mjs
┌─────────────────────┐              ┌──────────────────┐           ┌──────────────────┐
│ encodeSampleToBitmap│              │ store.retrieve   │           │ decodeBitmapTo   │
│ decodeBitmapToSample│              │ ("stample:sample")│          │ Sample (COPY)    │
│ imageToBuffer       │              │ → registerSample │           │ imageToBuffer    │
│ loadPaintingCode    │              │                  │           │ (COPY)           │
│ loadSystemPainting  │              │ No painting→audio│           └──────────────────┘
│                     │              │ No KidLisp→audio │
│ $code → pixels →    │              │ No #code loading │
│   decode → register │              └──────────────────┘
└─────────────────────┘

Target Flow (Unified)#

lib/pixel-sample.mjs (NEW shared module)
┌─────────────────────────────────────────────────────┐
│ encodeSampleToBitmap(data, width)                    │
│ decodeBitmapToSample(bitmap, meta)                   │
│ imageToBuffer(image)                                 │
│ loadPaintingAsAudio(source, {sound, preload, store}) │
│   → handles: #code, $kidlisp, "p"/painting, URL     │
│   → returns {sampleData, sampleId, bitmap, meta}     │
└─────────────────────────────────────────────────────┘
         ↓ imported by
   stample.mjs    notepat.mjs    clock.mjs

Step-by-Step Plan#

Step 1: Extract shared module lib/pixel-sample.mjs#

Create /system/public/aesthetic.computer/lib/pixel-sample.mjs with:

// 3 audio samples per pixel (R, G, B channels)
export function encodeSampleToBitmap(data, width = 256) { ... }
export function decodeBitmapToSample(bitmap, meta) { ... }
export async function imageToBuffer(image) { ... }

These are verbatim copies from stample.mjs lines 1300-1537, now exported.

Step 2: Extract loadPaintingAsAudio() into the shared module#

Generalize loadPaintingCode() and loadSystemPainting() into a single function:

/**
 * Load any pixel source as a playable audio sample.
 * @param {string|object} source - "#code", "$kidlisp", "p"/"painting", or a bitmap object
 * @param {object} opts - { sound, preload, store, system, get, painting, kidlisp }
 * @returns {{ sampleData, sampleId, bitmap, meta } | null}
 */
export async function loadPaintingAsAudio(source, opts) {
  const sampleId = "stample:bitmap";

  if (typeof source === "string") {
    if (source.startsWith("$")) {
      // KidLisp: render to pixel buffer, decode to audio
      // Use opts.painting() + opts.kidlisp() to render
    } else if (source.startsWith("#") || source.startsWith("%23")) {
      // Painting code: fetch from /media/paintings/ or /api/painting-code
      // Reuse loadPaintingCode logic
    } else if (source === "p" || source === "painting") {
      // System painting: from nopaint buffer or store
      // Reuse loadSystemPainting logic
    }
  } else if (source?.pixels) {
    // Direct bitmap object
  }

  if (!bitmap?.pixels?.length) return null;

  const totalPixels = bitmap.width * bitmap.height;
  const meta = { sampleLength: totalPixels * 3, sampleRate: opts.sound?.sampleRate || 48000 };
  const sampleData = decodeBitmapToSample(bitmap, meta);

  if (sampleData?.length) {
    opts.sound?.registerSample?.(sampleId, sampleData, meta.sampleRate);
  }

  return { sampleData, sampleId, bitmap, meta };
}

Step 3: Update stample.mjs to import from shared module#

Replace local functions with imports:

import {
  encodeSampleToBitmap,
  decodeBitmapToSample,
  imageToBuffer,
  loadPaintingAsAudio,
} from "../lib/pixel-sample.mjs";

Remove the local copies (~200 lines). Update loadPaintingCode and loadSystemPainting callers to use loadPaintingAsAudio(). The KidLisp sim loop still calls decodeBitmapToSample directly for live frame-by-frame updates.

Step 4: Update clock.mjs to import from shared module#

Replace the copy-pasted functions:

import { decodeBitmapToSample, imageToBuffer } from "../lib/pixel-sample.mjs";

Remove local copies (~70 lines).

Step 5: Add painting→sample pipeline to notepat.mjs#

5a. Import the shared module:

import {
  decodeBitmapToSample,
  imageToBuffer,
  loadPaintingAsAudio,
} from "../lib/pixel-sample.mjs";

5b. Handle stample params in boot:

After the existing params[0] === "piano" / params[0] === "twinkle" checks (~line 1252), add:

// Handle stample with painting source: notepat:stample:p, notepat:stample:#code, notepat:stample:$kidlisp
if (wave === "stample" || requestedWave === "stample" || requestedWave === "sample") {
  const stampleSource = colon[1]; // e.g. "p", "#abc123", "$roz"
  if (stampleSource) {
    const result = await loadPaintingAsAudio(stampleSource, {
      sound, preload, store, system, get, painting, kidlisp,
    });
    if (result) {
      stampleSampleId = result.sampleId;
      stampleSampleData = result.sampleData;
      stampleSampleRate = result.meta.sampleRate;
    }
  }
}

This makes these paths work:

• notepat:stample:p — load system painting as sample
• notepat:stample:#abc123 — load painting by code as sample
• notepat:stample:$roz — load KidLisp as sample

5c. Connect KidLisp visualization to sample buffer (live):

When kidlispBgEnabled is true and wave is stample, feed the kidlisp pixel buffer into the sample system each frame (in sim), similar to stample.mjs lines 514-597:

// In sim(), after the kidlisp background update:
if (kidlispBgEnabled && (wave === "stample" || wave === "sample") && kidlispBackground) {
  // Render kidlisp to a small buffer and decode as audio
  const bufferSize = 128;
  const lispPainting = painting(bufferSize, bufferSize, (paintApi) => {
    paintApi.kidlisp(0, 0, bufferSize, bufferSize, kidlispBackground);
  });
  if (lispPainting?.pixels?.length) {
    const totalPixels = bufferSize * bufferSize;
    const meta = { sampleLength: totalPixels * 3, sampleRate: sound?.sampleRate || 48000 };
    const decoded = decodeBitmapToSample(lispPainting, meta);
    if (decoded?.length) {
      sound.updateSample?.("stample:bitmap", decoded, meta.sampleRate);
      stampleSampleId = "stample:bitmap";
      stampleSampleData = decoded;
    }
  }
}

5d. Connect picture buffer to sample (optional/future):

The picture buffer accumulates note-color stamps. It could optionally feed the stample system, making the visual history playable:

// When picture buffer is updated and wave is stample, re-encode to audio
if (wave === "stample" && picture?.pixels?.length) {
  const meta = { sampleLength: picture.width * picture.height * 3, sampleRate: 48000 };
  const decoded = decodeBitmapToSample(picture, meta);
  if (decoded?.length) {
    sound.updateSample?.("stample:bitmap", decoded, meta.sampleRate);
  }
}

This is a stretch goal — the picture buffer is low-res and would produce very short samples. Worth experimenting with but not essential for initial unification.

Step 6: Store pipeline consistency#

Ensure all three pieces use the same store keys:

• "stample:sample" — raw audio sample data (Float32Array + sampleRate)
• "stample:bitmap" — pixel buffer + meta (width, height, pixels, sampleLength, sampleRate)

Notepat currently only reads "stample:sample". After unification, it should also read/write "stample:bitmap" when loading from painting sources.

Files Modified#

Verification#

1. stample p — loads system painting as sample, plays with pitch control
2. stample #abc123 — loads painting by code, plays as before
3. stample $roz — KidLisp renders live to audio, plays as before
4. notepat:stample:p — loads painting into notepat's stample mode
5. notepat:stample:$roz — KidLisp feeds sample buffer in notepat
6. notepat:stample — fallback to stored sample (existing behavior preserved)
7. clock.mjs stample features — unchanged behavior, now using shared module
8. Tab through wavetypes — stample mode still cycles correctly
9. Recording in stample — mic → encode → store still works

Risk Assessment#

• Low risk: Extracting functions to shared module is mechanical (identical code)
• Medium risk: loadPaintingAsAudio generalization — need to ensure all edge cases from both loadPaintingCode and loadSystemPainting are covered
• Low risk: notepat param handling — additive change, doesn't modify existing paths
• Higher risk: Live KidLisp→sample in notepat sim loop — performance-sensitive, may need throttling (stample does it every frame at 128x128)