Paper: Sampling in Aesthetic Computer#

Working Title#

"Every Sound is a Painting: Sampling as Visual-Auditory Practice in Aesthetic Computer"

Abstract Sketch#

Aesthetic Computer treats audio samples as first-class visual objects by encoding them as RGB pixel data in shareable "paintings." This paper traces the evolution of sampling in AC — from the initial microphone capture in notepat.mjs through stample.mjs's pixel-sample encoding to a cross-platform architecture where bare- metal devices and web browsers share samples through the same painting infrastructure. We argue that collapsing the boundary between visual and auditory media enables new creative workflows: a recorded sound becomes an image you can see, share as a short code, and play back on any AC device. The system extends Attali's concept of "composition" (where everyone creates) to the sample itself — every user's sound becomes a painting in a shared library.

Paper Structure#

1. Introduction: The Sample Problem#

Digital audio workstations treat samples as opaque binary blobs (WAV, AIFF)
Sharing samples requires file transfer, format negotiation, storage accounts
Disconnect between visual creative tools and audio creative tools
AC's premise: what if a sample was just another painting?

Attali's "Noise" (already cited in PLORK paper): recording → repetition → composition. AC pushes into composition where everyone records and shares.
Laptop orchestra tradition (PLORK paper context): PLOrk, SLOrk, CLOrk. Sample sharing in ensemble contexts.
Visual-auditory encoding precedents: spectrograms as visual representation, Jerobeam Fenderson's oscilloscope art, bytebeat/pixel-to-audio experiments
Painting as data format: the tradition of steganography, data art, Manfred Mohr's algorithmic marks containing their own instructions

3. Sampling in AC: Current Architecture#

3.1 notepat.mjs — The Instrument#

7 waveform types including "sample" mode
Home key: record from microphone
End key: arm per-key sample recording (different sample per note)
Sample bank: sampleBank[key] with individual float32 buffers
Auto-save to /mnt/ac-sample.raw on native boot media
Auto-load on next boot — the instrument remembers its voice

Reference: fedac/native/pieces/notepat.mjs lines 16-28, 385-430, 590-600

3.2 stample.mjs — The Sampler as Painting#

encodeSampleToBitmap(data, width): float32 audio → 8-bit RGB pixels
- 3 samples per pixel (R, G, B channels)
- -1.0..+1.0 mapped to 0..255
- 256px wide, height varies with duration
- 5 seconds @ 48kHz = 240,000 samples = 80,000 pixels = 256×313 PNG ~100KB
decodeBitmapToSample(bitmap, meta): reverse process
loadPaintingAsAudio(source, opts): load ANY painting as audio
- Painting code (#k3d) → resolve via API → download PNG → decode → play
- KidLisp source ($roz) → render to bitmap → decode → play
- System painting → current canvas → decode → play

Reference: system/public/aesthetic.computer/lib/pixel-sample.mjs

3.3 The Painting Infrastructure#

Upload: track-media.mjs → presigned URL → DO Spaces CDN → MongoDB record
Short codes: every painting gets a code like #k3d
URL addressable: aesthetic.computer/painting/#k3d
Owned by @handle: @jeffrey/painting/slug
Social: shareable, embeddable, browsable
Already handles images of any size and format

4. The Bridge: Samples as Paintings Across Platforms#

4.1 The Problem of Two Worlds#

Native (ac-native): QuickJS, ALSA, direct hardware, no DOM, no Canvas
Web (browser): Web Audio API, Canvas, full network stack
Same pieces need to work on both platforms
Samples recorded on bare metal should be playable in browsers and vice versa

4.2 The Solution: Paint the Sound#

Phase 1: Add pixel-sample encoding to native (pure JS, no DOM needed)
Phase 2: Native uploads encoded bitmaps as paintings via existing API
Phase 3: Native downloads paintings by code and decodes to audio
Phase 4: Unified samples.mjs piece on both platforms
Phase 5: notepat gains cloud sample support on both platforms

4.3 Data Flow#

Record sound → float32 array → encode as RGB pixels → upload as painting
              → get short code (#abc) → share
                              ↓
Download painting → decode RGB pixels → float32 array → play as audio

5. Design Philosophy#

5.1 Why Paintings, Not WAV Files#

Infrastructure reuse: no new storage backend, API, or auth flow
Visibility: you can SEE your sample as an image
Compactness: 8-bit RGB is 4× smaller than float32
Universality: PNG renders everywhere, WAV needs specialized tools
Social integration: paintings are already AC's shareable unit
Artistic potential: the visual appearance of a sample IS its visual identity

5.2 The Encoding as Aesthetic Choice#

The RGB encoding is lossy (32-bit float → 8-bit per channel) but this is a feature, not a bug. The quantization introduces a subtle lo-fi character that is consistent and reversible-within-tolerance. Like vinyl's warmth or cassette's hiss, the encoding IS the medium.

The visual appearance of an encoded sample reveals its structure:

Sine waves produce smooth color gradients
Percussion creates sharp color boundaries
Noise produces visual static
Speech shows rhythmic color patterns

A user browsing a gallery of sample-paintings can develop visual intuition for how sounds look — and vice versa.

5.3 Every User's Sound Becomes a Painting#

Following Attali's trajectory: in the "composition" era, the distinction between producer and consumer dissolves. In AC, every user who records a sample creates a painting. That painting enters the shared library with a short code. Other users can load it, play it, modify it, re-record it.

The sample is not a file in a folder. It is a painting in a gallery.

6. Implementation Status and Future Work#

6.1 What Exists Today#

stample.mjs (web): full pixel-sample pipeline
notepat.mjs (native): microphone recording, per-key sample bank
samples.mjs (native): local sample library browser
pixel-sample.mjs: encode/decode library
Painting upload/download/code infrastructure

6.2 What's In Progress#

Native pixel-sample encoding (pure JS)
Native painting upload from device
Cross-platform samples.mjs
Cloud sample sync (local-first, cloud optional)

6.3 Future Directions#

Collaborative sampling: multiple devices record simultaneously, merge into one painting
Live sample streaming: one device records, others play in real time via UDP
Generative samples: KidLisp programs that generate sample-paintings algorithmically
Sample DNA: track lineage when samples are re-recorded/modified (like git for sound)
Physical prints: print sample-paintings on paper, scan them back as audio

7. Conclusion#

By treating samples as paintings, AC removes an artificial boundary between visual and auditory creative practice. A sound has a color. A painting has a voice. The infrastructure is the same. The creative act is unified.

References to Include#

Attali, Jacques. "Noise: The Political Economy of Music." 1985.
Blackwell et al. "Live Coding: A User's Manual." 2022.
Turchet, Luca. "Elk Audio OS." 2021.
Existing AC papers: arxiv-ac, arxiv-plork, arxiv-notepat
Collins, Nick. "Introduction to Computer Music." (on laptop orchestras)
Roads, Curtis. "Microsound." (on granular/sample-based synthesis)
Wishart, Trevor. "On Sonic Art." (on the materiality of sound)

Placement#

This paper fits in the platter as a companion to the notepat paper, expanding from "the instrument" to "the sample as creative object." It connects to the PLORK paper's discussion of distributed music-making and to the main AC paper's architecture sections.

Directory: papers/arxiv-sampling/

Configure Feed