Monorepo for Aesthetic.Computer
aesthetic.computer
plan: paper outline — Sampling in AC (every sound is a painting)
Outlines a paper tracing sampling from notepat microphone capture through
stample pixel-sample encoding to cross-platform painting-based storage.
Connects to Attali's composition era and existing platter papers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
+187
+187
plans/paper-sampling-in-ac.md
+187
plans/paper-sampling-in-ac.md
···
1
+
# Paper: Sampling in Aesthetic Computer
2
+
3
+
## Working Title
4
+
**"Every Sound is a Painting: Sampling as Visual-Auditory Practice in Aesthetic Computer"**
5
+
6
+
## Abstract Sketch
7
+
8
+
Aesthetic Computer treats audio samples as first-class visual objects by encoding
9
+
them as RGB pixel data in shareable "paintings." This paper traces the evolution
10
+
of sampling in AC — from the initial microphone capture in `notepat.mjs` through
11
+
`stample.mjs`'s pixel-sample encoding to a cross-platform architecture where bare-
12
+
metal devices and web browsers share samples through the same painting infrastructure.
13
+
We argue that collapsing the boundary between visual and auditory media enables new
14
+
creative workflows: a recorded sound becomes an image you can see, share as a short
15
+
code, and play back on any AC device. The system extends Attali's concept of
16
+
"composition" (where everyone creates) to the sample itself — every user's sound
17
+
becomes a painting in a shared library.
18
+
19
+
---
20
+
21
+
## Paper Structure
22
+
23
+
### 1. Introduction: The Sample Problem
24
+
25
+
- Digital audio workstations treat samples as opaque binary blobs (WAV, AIFF)
26
+
- Sharing samples requires file transfer, format negotiation, storage accounts
27
+
- Disconnect between visual creative tools and audio creative tools
28
+
- AC's premise: what if a sample was just another painting?
29
+
30
+
### 2. Background and Related Work
31
+
32
+
- **Attali's "Noise"** (already cited in PLORK paper): recording → repetition →
33
+
composition. AC pushes into composition where everyone records and shares.
34
+
- **Laptop orchestra tradition** (PLORK paper context): PLOrk, SLOrk, CLOrk.
35
+
Sample sharing in ensemble contexts.
36
+
- **Visual-auditory encoding precedents**: spectrograms as visual representation,
37
+
Jerobeam Fenderson's oscilloscope art, bytebeat/pixel-to-audio experiments
38
+
- **Painting as data format**: the tradition of steganography, data art,
39
+
Manfred Mohr's algorithmic marks containing their own instructions
40
+
41
+
### 3. Sampling in AC: Current Architecture
42
+
43
+
#### 3.1 notepat.mjs — The Instrument
44
+
45
+
- 7 waveform types including "sample" mode
46
+
- Home key: record from microphone
47
+
- End key: arm per-key sample recording (different sample per note)
48
+
- Sample bank: `sampleBank[key]` with individual float32 buffers
49
+
- Auto-save to `/mnt/ac-sample.raw` on native boot media
50
+
- Auto-load on next boot — the instrument remembers its voice
51
+
52
+
Reference: `fedac/native/pieces/notepat.mjs` lines 16-28, 385-430, 590-600
53
+
54
+
#### 3.2 stample.mjs — The Sampler as Painting
55
+
56
+
- `encodeSampleToBitmap(data, width)`: float32 audio → 8-bit RGB pixels
57
+
- 3 samples per pixel (R, G, B channels)
58
+
- -1.0..+1.0 mapped to 0..255
59
+
- 256px wide, height varies with duration
60
+
- 5 seconds @ 48kHz = 240,000 samples = 80,000 pixels = 256×313 PNG ~100KB
61
+
- `decodeBitmapToSample(bitmap, meta)`: reverse process
62
+
- `loadPaintingAsAudio(source, opts)`: load ANY painting as audio
63
+
- Painting code (`#k3d`) → resolve via API → download PNG → decode → play
64
+
- KidLisp source (`$roz`) → render to bitmap → decode → play
65
+
- System painting → current canvas → decode → play
66
+
67
+
Reference: `system/public/aesthetic.computer/lib/pixel-sample.mjs`
68
+
69
+
#### 3.3 The Painting Infrastructure
70
+
71
+
- Upload: `track-media.mjs` → presigned URL → DO Spaces CDN → MongoDB record
72
+
- Short codes: every painting gets a code like `#k3d`
73
+
- URL addressable: `aesthetic.computer/painting/#k3d`
74
+
- Owned by @handle: `@jeffrey/painting/slug`
75
+
- Social: shareable, embeddable, browsable
76
+
- Already handles images of any size and format
77
+
78
+
### 4. The Bridge: Samples as Paintings Across Platforms
79
+
80
+
#### 4.1 The Problem of Two Worlds
81
+
82
+
- Native (ac-native): QuickJS, ALSA, direct hardware, no DOM, no Canvas
83
+
- Web (browser): Web Audio API, Canvas, full network stack
84
+
- Same pieces need to work on both platforms
85
+
- Samples recorded on bare metal should be playable in browsers and vice versa
86
+
87
+
#### 4.2 The Solution: Paint the Sound
88
+
89
+
- Phase 1: Add pixel-sample encoding to native (pure JS, no DOM needed)
90
+
- Phase 2: Native uploads encoded bitmaps as paintings via existing API
91
+
- Phase 3: Native downloads paintings by code and decodes to audio
92
+
- Phase 4: Unified `samples.mjs` piece on both platforms
93
+
- Phase 5: notepat gains cloud sample support on both platforms
94
+
95
+
#### 4.3 Data Flow
96
+
97
+
```
98
+
Record sound → float32 array → encode as RGB pixels → upload as painting
99
+
→ get short code (#abc) → share
100
+
↓
101
+
Download painting → decode RGB pixels → float32 array → play as audio
102
+
```
103
+
104
+
### 5. Design Philosophy
105
+
106
+
#### 5.1 Why Paintings, Not WAV Files
107
+
108
+
1. **Infrastructure reuse**: no new storage backend, API, or auth flow
109
+
2. **Visibility**: you can SEE your sample as an image
110
+
3. **Compactness**: 8-bit RGB is 4× smaller than float32
111
+
4. **Universality**: PNG renders everywhere, WAV needs specialized tools
112
+
5. **Social integration**: paintings are already AC's shareable unit
113
+
6. **Artistic potential**: the visual appearance of a sample IS its visual identity
114
+
115
+
#### 5.2 The Encoding as Aesthetic Choice
116
+
117
+
The RGB encoding is lossy (32-bit float → 8-bit per channel) but this is
118
+
a feature, not a bug. The quantization introduces a subtle lo-fi character
119
+
that is consistent and reversible-within-tolerance. Like vinyl's warmth or
120
+
cassette's hiss, the encoding IS the medium.
121
+
122
+
The visual appearance of an encoded sample reveals its structure:
123
+
- Sine waves produce smooth color gradients
124
+
- Percussion creates sharp color boundaries
125
+
- Noise produces visual static
126
+
- Speech shows rhythmic color patterns
127
+
128
+
A user browsing a gallery of sample-paintings can develop visual intuition
129
+
for how sounds look — and vice versa.
130
+
131
+
#### 5.3 Every User's Sound Becomes a Painting
132
+
133
+
Following Attali's trajectory: in the "composition" era, the distinction
134
+
between producer and consumer dissolves. In AC, every user who records a
135
+
sample creates a painting. That painting enters the shared library with a
136
+
short code. Other users can load it, play it, modify it, re-record it.
137
+
138
+
The sample is not a file in a folder. It is a painting in a gallery.
139
+
140
+
### 6. Implementation Status and Future Work
141
+
142
+
#### 6.1 What Exists Today
143
+
- stample.mjs (web): full pixel-sample pipeline
144
+
- notepat.mjs (native): microphone recording, per-key sample bank
145
+
- samples.mjs (native): local sample library browser
146
+
- pixel-sample.mjs: encode/decode library
147
+
- Painting upload/download/code infrastructure
148
+
149
+
#### 6.2 What's In Progress
150
+
- Native pixel-sample encoding (pure JS)
151
+
- Native painting upload from device
152
+
- Cross-platform samples.mjs
153
+
- Cloud sample sync (local-first, cloud optional)
154
+
155
+
#### 6.3 Future Directions
156
+
- **Collaborative sampling**: multiple devices record simultaneously, merge into one painting
157
+
- **Live sample streaming**: one device records, others play in real time via UDP
158
+
- **Generative samples**: KidLisp programs that generate sample-paintings algorithmically
159
+
- **Sample DNA**: track lineage when samples are re-recorded/modified (like git for sound)
160
+
- **Physical prints**: print sample-paintings on paper, scan them back as audio
161
+
162
+
### 7. Conclusion
163
+
164
+
By treating samples as paintings, AC removes an artificial boundary between
165
+
visual and auditory creative practice. A sound has a color. A painting has
166
+
a voice. The infrastructure is the same. The creative act is unified.
167
+
168
+
---
169
+
170
+
## References to Include
171
+
172
+
- Attali, Jacques. "Noise: The Political Economy of Music." 1985.
173
+
- Blackwell et al. "Live Coding: A User's Manual." 2022.
174
+
- Turchet, Luca. "Elk Audio OS." 2021.
175
+
- Existing AC papers: arxiv-ac, arxiv-plork, arxiv-notepat
176
+
- Collins, Nick. "Introduction to Computer Music." (on laptop orchestras)
177
+
- Roads, Curtis. "Microsound." (on granular/sample-based synthesis)
178
+
- Wishart, Trevor. "On Sonic Art." (on the materiality of sound)
179
+
180
+
## Placement
181
+
182
+
This paper fits in the platter as a companion to the notepat paper,
183
+
expanding from "the instrument" to "the sample as creative object."
184
+
It connects to the PLORK paper's discussion of distributed music-making
185
+
and to the main AC paper's architecture sections.
186
+
187
+
Directory: `papers/arxiv-sampling/`