Monorepo for Aesthetic.Computer
aesthetic.computer
fix: remove SDL/Mesa/DRI libs from initramfs — causing boot crash
SDL3/GBM/Mesa DRI libs were copied into initramfs unconditionally.
Their transitive deps or constructors crash on hardware without
working DRI. Since ac-native uses dlopen (no link-time SDL dep),
the binary runs fine without these libs and falls back to DRM.
Also updated cross-platform-samples.md with finalized architecture:
- Samples share # sigil with paintings (same PNG format)
- Separate samples MongoDB collection
- Shared code namespace (cross-collection uniqueness for #)
- Encoding version (v:1) on every record
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
+155
-141
+11
-19
fedac/native/docker-build.sh
+11
-19
fedac/native/docker-build.sh
···
216
216
[ -f "$REAL" ] && cp -L "$REAL" "$IROOT/lib64/$BASENAME"
217
217
done
218
218
219
-
# SDL3 + Mesa/GPU libs
220
-
for lib in libSDL3.so.0 \
221
-
libgbm.so.1 libEGL.so.1 libEGL_mesa.so.0 libGLESv2.so.2 \
222
-
libGL.so.1 libGLX_mesa.so.0 libGLdispatch.so.0 libglapi.so.0 \
223
-
libdrm_intel.so.1 libdrm_amdgpu.so.1; do
224
-
REAL=$(readlink -f "/lib64/$lib" 2>/dev/null)
225
-
[ -f "$REAL" ] && cp -L "$REAL" "$IROOT/lib64/$lib"
226
-
done
227
-
228
-
# Mesa DRI drivers
229
-
if [ -d /lib64/dri ]; then
230
-
mkdir -p "$IROOT/lib64/dri"
231
-
for drv in /lib64/dri/i915_dri.so /lib64/dri/iris_dri.so /lib64/dri/kms_swrast_dri.so /lib64/dri/swrast_dri.so; do
232
-
[ -f "$drv" ] && cp -L "$drv" "$IROOT/lib64/dri/"
233
-
done
234
-
fi
235
-
# Gallium megadriver
236
-
GALLIUM=$(readlink -f /lib64/libgallium-*.so 2>/dev/null || true)
237
-
[ -f "$GALLIUM" ] && cp -L "$GALLIUM" "$IROOT/lib64/"
219
+
# SDL3 + Mesa/GPU libs — only include if SDL dlopen will be attempted.
220
+
# These libs and their transitive deps (LLVM, etc.) can crash on hardware
221
+
# without working DRI drivers. Since ac-native uses dlopen (not link-time),
222
+
# the binary runs fine without them and falls back to DRM dumb buffers.
223
+
# TODO: Re-enable when SDL3 dlopen fallback is proven stable on all targets.
224
+
# for lib in libSDL3.so.0 libgbm.so.1 libEGL.so.1 libEGL_mesa.so.0 \
225
+
# libGLESv2.so.2 libGL.so.1 libGLX_mesa.so.0 libGLdispatch.so.0 \
226
+
# libglapi.so.0 libdrm_intel.so.1 libdrm_amdgpu.so.1; do
227
+
# REAL=$(readlink -f "/lib64/$lib" 2>/dev/null)
228
+
# [ -f "$REAL" ] && cp -L "$REAL" "$IROOT/lib64/$lib"
229
+
# done
238
230
239
231
# Transitive deps — resolve everything in lib64
240
232
log " Resolving transitive dependencies..."
+144
-122
plans/cross-platform-samples.md
+144
-122
plans/cross-platform-samples.md
···
7
7
infrastructure: upload, CDN, short codes (`#k3d`), @handle ownership, track-media
8
8
API, and cross-platform rendering.
9
9
10
-
**We don't need a new storage system. Samples should be stored as paintings.**
11
-
12
-
Record audio → encode as bitmap → save as painting → get a short code → share.
13
-
Anyone loads the painting back as audio on web or native.
10
+
**Samples use the same `#` sigil and PNG storage format as paintings, but live in
11
+
a separate `samples` MongoDB collection with audio-specific metadata.**
14
12
15
13
---
16
14
17
-
## Current State
15
+
## Architecture Decision: Shared Sigil, Separate Collection
18
16
19
-
### Web (stample.mjs + pixel-sample.mjs)
20
-
- `encodeSampleToBitmap(data, width)` — float32[] → RGB pixels (3 samples/pixel)
21
-
- `decodeBitmapToSample(bitmap, meta)` — RGB pixels → float32[]
22
-
- `loadPaintingAsAudio(source, opts)` — load a painting code/object as playable audio
23
-
- Paintings upload via `track-media.mjs` → DO Spaces + MongoDB + short code
24
-
- Download via `/media/@handle/painting/slug.png` or `/media/paintings/CODE.png`
17
+
### Why `#` (not a new sigil like `^`)
18
+
- Samples and paintings share the same PNG storage format
19
+
- A painting can BE a sample (load any image as audio via stample)
20
+
- A sample IS a painting (the pixel-encoded waveform is visible art)
21
+
- `%` and `&` are URL-unfriendly (`%` is URL escape, `&` is query separator)
22
+
- Keeps the sigil set small: `@` people, `$` code, `#` media, `*` time
23
+
24
+
### Why separate collection (not a flag on paintings)
25
+
- Clean querying: "list my samples" vs "list my paintings" without filters
26
+
- Separate indexes optimized for audio metadata (duration, sampleRate, etc.)
27
+
- Separate counts/quotas per media type
28
+
- Future-proof: audio-specific features (waveform preview, BPM detection)
25
29
26
-
### Native (ac-native)
27
-
- Samples stored as raw float32 PCM (rate + length + data) at `/mnt/ac-sample.raw`
28
-
- `audio_sample_save()` / `audio_sample_load()` in C
29
-
- `sound.sample.saveTo(path)` / `sound.sample.loadFrom(path)` JS bindings
30
-
- `sound.sample.getData()` → Float32Array, `sound.sample.loadData(f32, rate)` → load
31
-
- `/mnt/samples/` directory with `manifest.json` for local sample library
32
-
- No pixel-sample encoding yet, no painting upload capability
30
+
### Shared code namespace
31
+
- `#` codes must be unique across BOTH `paintings` AND `samples` collections
32
+
- `generateUniqueCode()` checks both collections before assigning
33
+
- Other sigils (`$` KidLisp, `*` clock, tapes) have independent namespaces
34
+
- The code resolver checks both collections to find which type a `#code` refers to
33
35
34
36
---
35
37
36
-
## Plan
38
+
## MongoDB Schema
37
39
38
-
### Phase 1: Native Pixel-Sample Bridge
40
+
### Collection: `samples`
41
+
```js
42
+
{
43
+
_id: ObjectId,
44
+
user: "auth0|63effeeb...", // owner (null for guest)
45
+
slug: "kick-drum", // user-friendly name
46
+
code: "k3d", // unique short code (shared with paintings)
47
+
when: ISODate, // upload timestamp
48
+
49
+
// Sample-specific metadata
50
+
v: 1, // pixel-sample encoding version
51
+
sampleRate: 48000,
52
+
sampleLength: 240000, // exact sample count
53
+
duration: 5.0, // seconds
54
+
channels: 1,
55
+
source: "native", // or "web"
56
+
57
+
// Standard media fields
58
+
ext: "png", // storage format
59
+
width: 256, // image dimensions
60
+
height: 313,
61
+
}
62
+
```
39
63
40
-
**Add pixel-sample encoding/decoding to native JS pieces.**
64
+
### Indexes
65
+
```js
66
+
// Unique code (shared namespace with paintings — enforced at generation time)
67
+
await samples.createIndex({ code: 1 }, { unique: true, sparse: true });
41
68
42
-
Since native pieces run in QuickJS (no DOM, no Canvas), implement a pure-JS
43
-
version of the encode/decode that works without browser APIs:
69
+
// User queries: "list my samples"
70
+
await samples.createIndex({ user: 1 });
44
71
45
-
**File: `fedac/native/pieces/lib/pixel-sample-native.mjs`**
46
-
- `encodeSampleToBitmap(float32Array, width)` — same algorithm as web version
47
-
(float32 → 8-bit RGB, 3 samples per pixel)
48
-
- `decodeBitmapToSample(rgbaArray, width, height, sampleLength)` — reverse
49
-
- These are pure math — no DOM needed
72
+
// Audio-specific queries
73
+
await samples.createIndex({ duration: 1 }); // sort by length
74
+
await samples.createIndex({ "source": 1 }); // native vs web
75
+
await samples.createIndex({ v: 1 }); // encoding version (for migration)
76
+
await samples.createIndex({ when: -1 }); // recent first
77
+
await samples.createIndex({ user: 1, slug: 1 }, { unique: true }); // per-user slugs
78
+
```
50
79
51
-
**Modify: `fedac/native/pieces/samples.mjs`**
52
-
- When saving, also encode sample as bitmap PNG
53
-
- Store both `.raw` (for instant native reload) and `.png` (for upload/sharing)
54
-
55
-
**Requires:** A way to write PNG from native. Options:
56
-
1. Use the existing `graph.c` framebuffer snapshot capability
57
-
2. Add a minimal PNG writer in C (stb_image_write.h is ~1KB)
58
-
3. Encode as BMP (simpler header, no compression) — paintings can be any image format
59
-
60
-
### Phase 2: Native Painting Upload
61
-
62
-
**Let native devices upload paintings to the AC cloud.**
80
+
### Encoding Version Contract
81
+
```
82
+
v1 (current — pixel-sample.mjs):
83
+
- 3 audio samples per pixel (R, G, B channels)
84
+
- Float range -1.0..+1.0 mapped to 0..255
85
+
- A channel: 255 (opaque)
86
+
- Width: 256px (configurable)
87
+
- Height: ceil(sampleLength / 3 / width)
88
+
- Mono only
89
+
- The `v` field MUST be stored on every record so decoders know which algorithm to use
90
+
```
63
91
64
-
**Add C binding: `system.uploadPainting(localPath, handle, token)`**
65
-
- POST to `/api/track-media` with painting metadata
66
-
- GET presigned upload URL
67
-
- PUT the image file to DO Spaces
68
-
- Returns short code on success
92
+
---
69
93
70
-
**Or simpler: POST the bitmap directly.**
71
-
- Encode sample as pixel data in JS
72
-
- Use `system.fetchPost()` to send base64-encoded bitmap to a new
73
-
`/api/sample-painting` endpoint that:
74
-
1. Decodes the base64 bitmap
75
-
2. Renders it as PNG via sharp/canvas
76
-
3. Uploads to DO Spaces via existing painting pipeline
77
-
4. Returns a short code
94
+
## Implementation Plan
78
95
79
-
**Update samples.mjs:** Add `u` (upload) key that encodes the current sample
80
-
as a painting bitmap and uploads it. Shows the short code on success.
96
+
### Phase 1: Backend — track-media + code generation
97
+
**Files to modify:**
81
98
82
-
### Phase 3: Native Painting Download (Load Remote Samples)
99
+
1. **`system/netlify/functions/track-media.mjs`**
100
+
- Add `mediaType: "sample"` branch for PNG uploads with sample metadata
101
+
- Route to `samples` collection instead of `paintings`
102
+
- Store `v`, `sampleRate`, `sampleLength`, `duration`, `channels`, `source`
83
103
84
-
**Let native devices load samples from painting codes.**
104
+
2. **`system/backend/generate-short-code.mjs`**
105
+
- `generateUniqueCode()` accepts optional `siblingCollections` array
106
+
- For `#` codes: checks both `paintings` AND `samples` before assigning
107
+
- Other types unchanged (single collection check)
85
108
86
-
**Flow:**
87
-
1. User types a painting code (e.g., `#k3d`) in samples.mjs
88
-
2. `system.fetch("/api/painting-code?code=k3d")` → get slug + handle
89
-
3. `system.fetchBinary("https://aesthetic.computer/media/paintings/k3d.png", "/tmp/sample.png")`
90
-
4. Decode PNG → extract RGB pixels → `decodeBitmapToSample()` → load into audio
109
+
3. **`system/netlify/functions/painting-code.mjs`** (or equivalent resolver)
110
+
- When resolving `#code`, check `paintings` first, then `samples`
111
+
- Return `{ type: "painting" | "sample", ...record }` so the client knows
91
112
92
-
**Requires:** PNG decoding in native. Options:
93
-
1. Add `stb_image.h` to the C build (single-header PNG decoder, tiny)
94
-
2. Decode in JS using a pure-JS PNG decoder
95
-
3. Use the existing `graph.c` image loading if it supports PNG
113
+
4. **New: `system/netlify/functions/list-samples.mjs`** (or extend existing)
114
+
- `GET /api/samples/@handle` → list user's samples
115
+
- `GET /api/samples/@handle/:slug` → get specific sample
116
+
- Returns CDN URLs + metadata
96
117
97
-
### Phase 4: Unified samples.mjs (Web + Native)
118
+
### Phase 2: Native pixel-sample bridge
119
+
**Files to create/modify:**
98
120
99
-
**Create a web `samples.mjs` piece that mirrors the native one.**
121
+
1. **`fedac/native/pieces/lib/pixel-sample-native.mjs`**
122
+
- Pure-JS encode/decode (no DOM, no Canvas — works in QuickJS)
123
+
- Same algorithm as web `pixel-sample.mjs`
124
+
- `encodeSampleToBitmap(float32Array, width)` → RGBA pixel array
125
+
- `decodeBitmapToSample(rgbaArray, width, height, sampleLength)` → float32[]
100
126
101
-
**File: `system/public/aesthetic.computer/disks/samples.mjs`**
127
+
2. **PNG write from native**
128
+
- Option A: Add `stb_image_write.h` (single-header, ~1KB) for PNG encoding in C
129
+
- Option B: Minimal PNG writer in JS (deflate + PNG header — ~100 lines)
130
+
- Option C: BMP format (simpler, no compression, server converts to PNG on upload)
102
131
103
-
Uses the existing painting infrastructure:
104
-
- List user's paintings that are tagged as samples (metadata flag)
105
-
- Record new sample via `Microphone` API
106
-
- Encode → upload as painting via `track-media`
107
-
- Browse + play back via `loadPaintingAsAudio()`
108
-
- Share via short code
109
-
- Compatible with native — same painting, same code, playable on both
132
+
### Phase 3: Upload from native
133
+
**Flow:**
134
+
```
135
+
Record audio → float32[] → encodeSampleToBitmap → PNG bytes
136
+
→ POST /api/track-media { ext:"png", mediaType:"sample", sampleMeta:{v:1,...} }
137
+
→ GET presigned URL → PUT PNG to Spaces → #code returned
138
+
```
110
139
111
-
### Phase 5: notepat Cross-Platform Samples
140
+
**Files:**
141
+
- `fedac/native/pieces/samples.mjs` — add upload key (`u`)
142
+
- `fedac/native/src/js-bindings.c` — `system.uploadMedia()` binding if needed
143
+
- Or use existing `system.fetch()` + `system.fetchPost()` for the API calls
112
144
113
-
**Both web and native notepat can load sample paintings.**
145
+
### Phase 4: Download to native
146
+
**Flow:**
147
+
```
148
+
Type #code → resolve via /api/painting-code → get CDN URL
149
+
→ fetchBinary(url, /tmp/sample.png) → decode PNG → decodeBitmapToSample
150
+
→ sound.sample.loadData(float32, rate) → play
151
+
```
114
152
115
-
**Native notepat.mjs:**
116
-
- Add `stample CODE` command or sample bank that loads from painting codes
117
-
- After recording, offer to save as painting (upload to cloud)
118
-
- Saved samples show their short code in the status bar
153
+
**Requires:** PNG decoding in native
154
+
- Option A: `stb_image.h` (single-header PNG decoder)
155
+
- Option B: Decode in JS (pure-JS PNG inflate)
156
+
- Option C: Server endpoint that returns raw PCM (avoid client-side PNG decode)
119
157
120
-
**Web notepat.mjs:**
121
-
- Already has `stample.mjs` as a sibling piece
122
-
- Add sample mode that uses `loadPaintingAsAudio()` to load from codes
123
-
- Shared sample format means a sample recorded on bare metal can be
124
-
played in the browser and vice versa
158
+
### Phase 5: Unified experience
159
+
- **notepat.mjs**: type `#code` to load a sample from cloud into sample bank
160
+
- **samples.mjs**: browse/record/upload/download on web and native
161
+
- **stample.mjs**: already works, becomes the web sample player
162
+
- **Gallery**: shows speaker icon on `#` codes that are samples
163
+
- **Profile**: separate "samples" tab alongside "paintings"
125
164
126
165
---
127
166
···
134
173
| |
135
174
encodeSampleToBitmap encodeSampleToBitmap
136
175
| |
137
-
RGB pixels RGB pixels
176
+
PNG bytes PNG bytes (via Canvas)
138
177
| |
139
-
upload as painting upload as painting
178
+
POST /api/track-media POST /api/track-media
179
+
{ mediaType:"sample" } { mediaType:"sample" }
140
180
| |
141
181
+----→ DO Spaces ←------+
142
-
+ MongoDB
143
-
+ short code (#k3d)
182
+
+ MongoDB "samples"
183
+
+ short code (#abc)
144
184
|
145
185
+------------+------------+
146
186
| |
···
153
193
audio playback audio playback
154
194
```
155
195
156
-
## Why Paintings, Not WAV
157
-
158
-
1. **Infrastructure exists** — upload, CDN, short codes, @handle, MongoDB, all done
159
-
2. **Visual** — you can SEE the sample as an image, share it as art
160
-
3. **Compact** — 8-bit RGB is ~4x smaller than float32 WAV for the same data
161
-
4. **Cross-platform** — PNG/image works everywhere, WAV needs special handling
162
-
5. **Social** — paintings are already the shareable unit in AC, samples become paintings
163
-
6. **stample.mjs already does this** — proven format, just extend it
164
-
165
-
## Storage Format
166
-
167
-
```
168
-
Painting PNG (256px wide, height varies):
169
-
R channel: sample[i*3+0] → 8-bit (mapped from -1..1 to 0..255)
170
-
G channel: sample[i*3+1]
171
-
B channel: sample[i*3+2]
172
-
A channel: 255 (opaque)
196
+
## Storage Math
173
197
174
-
Metadata (in MongoDB painting record):
175
-
type: "sample"
176
-
sampleLength: number (exact sample count)
177
-
sampleRate: 48000
178
-
durationSecs: number
179
-
source: "native" | "web"
180
-
```
181
-
182
-
A 5-second 48kHz sample = 240,000 samples = 80,000 pixels = 256×313 PNG ≈ 100KB.
198
+
| Duration | Samples @48kHz | Pixels (3/px) | Image (256w) | PNG size |
199
+
|----------|---------------|---------------|--------------|----------|
200
+
| 1 sec | 48,000 | 16,000 | 256×63 | ~20 KB |
201
+
| 5 sec | 240,000 | 80,000 | 256×313 | ~100 KB |
202
+
| 10 sec | 480,000 | 160,000 | 256×625 | ~200 KB |
183
203
184
204
## Key Files
185
205
186
206
| Existing | Purpose |
187
207
|----------|---------|
188
-
| `system/public/aesthetic.computer/lib/pixel-sample.mjs` | Encode/decode samples↔paintings |
208
+
| `system/public/aesthetic.computer/lib/pixel-sample.mjs` | Encode/decode samples↔bitmaps |
189
209
| `system/public/aesthetic.computer/disks/stample.mjs` | Web sample-painting player |
190
-
| `system/netlify/functions/track-media.mjs` | Painting upload API |
210
+
| `system/netlify/functions/track-media.mjs` | Media upload API |
191
211
| `system/netlify/functions/painting-code.mjs` | Short code → slug resolver |
212
+
| `system/backend/generate-short-code.mjs` | Unique code generation |
192
213
| `system/netlify/functions/presigned-url.js` | CDN upload/download URLs |
193
214
194
215
| New/Modified | Purpose |
195
216
|-------------|---------|
196
-
| `fedac/native/pieces/lib/pixel-sample-native.mjs` | Pure-JS encode/decode for native |
197
-
| `fedac/native/pieces/samples.mjs` | Add upload/download/code support |
217
+
| `fedac/native/pieces/lib/pixel-sample-native.mjs` | Pure-JS encode/decode for QuickJS |
218
+
| `fedac/native/pieces/samples.mjs` | Native sample browser + upload/download |
198
219
| `system/public/aesthetic.computer/disks/samples.mjs` | Web sample browser piece |
199
-
| `fedac/native/src/js-bindings.c` | PNG read/write bindings if needed |
220
+
| `system/netlify/functions/track-media.mjs` | Add sample branch |
221
+
| `system/backend/generate-short-code.mjs` | Cross-collection check for # codes |