Monorepo for Aesthetic.Computer
aesthetic.computer
1# Paper: Sampling in Aesthetic Computer
2
3## Working Title
4**"Every Sound is a Painting: Sampling as Visual-Auditory Practice in Aesthetic Computer"**
5
6## Abstract Sketch
7
8Aesthetic Computer treats audio samples as first-class visual objects by encoding
9them as RGB pixel data in shareable "paintings." This paper traces the evolution
10of sampling in AC — from the initial microphone capture in `notepat.mjs` through
11`stample.mjs`'s pixel-sample encoding to a cross-platform architecture where bare-
12metal devices and web browsers share samples through the same painting infrastructure.
13We argue that collapsing the boundary between visual and auditory media enables new
14creative workflows: a recorded sound becomes an image you can see, share as a short
15code, and play back on any AC device. The system extends Attali's concept of
16"composition" (where everyone creates) to the sample itself — every user's sound
17becomes a painting in a shared library.
18
19---
20
21## Paper Structure
22
23### 1. Introduction: The Sample Problem
24
25- Digital audio workstations treat samples as opaque binary blobs (WAV, AIFF)
26- Sharing samples requires file transfer, format negotiation, storage accounts
27- Disconnect between visual creative tools and audio creative tools
28- AC's premise: what if a sample was just another painting?
29
30### 2. Background and Related Work
31
32- **Attali's "Noise"** (already cited in PLORK paper): recording → repetition →
33 composition. AC pushes into composition where everyone records and shares.
34- **Laptop orchestra tradition** (PLORK paper context): PLOrk, SLOrk, CLOrk.
35 Sample sharing in ensemble contexts.
36- **Visual-auditory encoding precedents**: spectrograms as visual representation,
37 Jerobeam Fenderson's oscilloscope art, bytebeat/pixel-to-audio experiments
38- **Painting as data format**: the tradition of steganography, data art,
39 Manfred Mohr's algorithmic marks containing their own instructions
40
41### 3. Sampling in AC: Current Architecture
42
43#### 3.1 notepat.mjs — The Instrument
44
45- 7 waveform types including "sample" mode
46- Home key: record from microphone
47- End key: arm per-key sample recording (different sample per note)
48- Sample bank: `sampleBank[key]` with individual float32 buffers
49- Auto-save to `/mnt/ac-sample.raw` on native boot media
50- Auto-load on next boot — the instrument remembers its voice
51
52Reference: `fedac/native/pieces/notepat.mjs` lines 16-28, 385-430, 590-600
53
54#### 3.2 stample.mjs — The Sampler as Painting
55
56- `encodeSampleToBitmap(data, width)`: float32 audio → 8-bit RGB pixels
57 - 3 samples per pixel (R, G, B channels)
58 - -1.0..+1.0 mapped to 0..255
59 - 256px wide, height varies with duration
60 - 5 seconds @ 48kHz = 240,000 samples = 80,000 pixels = 256×313 PNG ~100KB
61- `decodeBitmapToSample(bitmap, meta)`: reverse process
62- `loadPaintingAsAudio(source, opts)`: load ANY painting as audio
63 - Painting code (`#k3d`) → resolve via API → download PNG → decode → play
64 - KidLisp source (`$roz`) → render to bitmap → decode → play
65 - System painting → current canvas → decode → play
66
67Reference: `system/public/aesthetic.computer/lib/pixel-sample.mjs`
68
69#### 3.3 The Painting Infrastructure
70
71- Upload: `track-media.mjs` → presigned URL → DO Spaces CDN → MongoDB record
72- Short codes: every painting gets a code like `#k3d`
73- URL addressable: `aesthetic.computer/painting/#k3d`
74- Owned by @handle: `@jeffrey/painting/slug`
75- Social: shareable, embeddable, browsable
76- Already handles images of any size and format
77
78### 4. The Bridge: Samples as Paintings Across Platforms
79
80#### 4.1 The Problem of Two Worlds
81
82- Native (ac-native): QuickJS, ALSA, direct hardware, no DOM, no Canvas
83- Web (browser): Web Audio API, Canvas, full network stack
84- Same pieces need to work on both platforms
85- Samples recorded on bare metal should be playable in browsers and vice versa
86
87#### 4.2 The Solution: Paint the Sound
88
89- Phase 1: Add pixel-sample encoding to native (pure JS, no DOM needed)
90- Phase 2: Native uploads encoded bitmaps as paintings via existing API
91- Phase 3: Native downloads paintings by code and decodes to audio
92- Phase 4: Unified `samples.mjs` piece on both platforms
93- Phase 5: notepat gains cloud sample support on both platforms
94
95#### 4.3 Data Flow
96
97```
98Record sound → float32 array → encode as RGB pixels → upload as painting
99 → get short code (#abc) → share
100 ↓
101Download painting → decode RGB pixels → float32 array → play as audio
102```
103
104### 5. Design Philosophy
105
106#### 5.1 Why Paintings, Not WAV Files
107
1081. **Infrastructure reuse**: no new storage backend, API, or auth flow
1092. **Visibility**: you can SEE your sample as an image
1103. **Compactness**: 8-bit RGB is 4× smaller than float32
1114. **Universality**: PNG renders everywhere, WAV needs specialized tools
1125. **Social integration**: paintings are already AC's shareable unit
1136. **Artistic potential**: the visual appearance of a sample IS its visual identity
114
115#### 5.2 The Encoding as Aesthetic Choice
116
117The RGB encoding is lossy (32-bit float → 8-bit per channel) but this is
118a feature, not a bug. The quantization introduces a subtle lo-fi character
119that is consistent and reversible-within-tolerance. Like vinyl's warmth or
120cassette's hiss, the encoding IS the medium.
121
122The visual appearance of an encoded sample reveals its structure:
123- Sine waves produce smooth color gradients
124- Percussion creates sharp color boundaries
125- Noise produces visual static
126- Speech shows rhythmic color patterns
127
128A user browsing a gallery of sample-paintings can develop visual intuition
129for how sounds look — and vice versa.
130
131#### 5.3 Every User's Sound Becomes a Painting
132
133Following Attali's trajectory: in the "composition" era, the distinction
134between producer and consumer dissolves. In AC, every user who records a
135sample creates a painting. That painting enters the shared library with a
136short code. Other users can load it, play it, modify it, re-record it.
137
138The sample is not a file in a folder. It is a painting in a gallery.
139
140### 6. Implementation Status and Future Work
141
142#### 6.1 What Exists Today
143- stample.mjs (web): full pixel-sample pipeline
144- notepat.mjs (native): microphone recording, per-key sample bank
145- samples.mjs (native): local sample library browser
146- pixel-sample.mjs: encode/decode library
147- Painting upload/download/code infrastructure
148
149#### 6.2 What's In Progress
150- Native pixel-sample encoding (pure JS)
151- Native painting upload from device
152- Cross-platform samples.mjs
153- Cloud sample sync (local-first, cloud optional)
154
155#### 6.3 Future Directions
156- **Collaborative sampling**: multiple devices record simultaneously, merge into one painting
157- **Live sample streaming**: one device records, others play in real time via UDP
158- **Generative samples**: KidLisp programs that generate sample-paintings algorithmically
159- **Sample DNA**: track lineage when samples are re-recorded/modified (like git for sound)
160- **Physical prints**: print sample-paintings on paper, scan them back as audio
161
162### 7. Conclusion
163
164By treating samples as paintings, AC removes an artificial boundary between
165visual and auditory creative practice. A sound has a color. A painting has
166a voice. The infrastructure is the same. The creative act is unified.
167
168---
169
170## References to Include
171
172- Attali, Jacques. "Noise: The Political Economy of Music." 1985.
173- Blackwell et al. "Live Coding: A User's Manual." 2022.
174- Turchet, Luca. "Elk Audio OS." 2021.
175- Existing AC papers: arxiv-ac, arxiv-plork, arxiv-notepat
176- Collins, Nick. "Introduction to Computer Music." (on laptop orchestras)
177- Roads, Curtis. "Microsound." (on granular/sample-based synthesis)
178- Wishart, Trevor. "On Sonic Art." (on the materiality of sound)
179
180## Placement
181
182This paper fits in the platter as a companion to the notepat paper,
183expanding from "the instrument" to "the sample as creative object."
184It connects to the PLORK paper's discussion of distributed music-making
185and to the main AC paper's architecture sections.
186
187Directory: `papers/arxiv-sampling/`