A nightstand noise generator based on M5Stack Atom Echo and integrating with Home Assistant
1# firmware/
2
3Rust firmware for the M5Stack Atom Echo–based nightstand sound machines. Built on **`esp-idf-svc` (std)** + the C ESP-IDF underneath. Targets the **`esp` Xtensa Rust toolchain** installed via `espup`.
4
5For the *what* and *why* (architecture, MQTT contract, operating modes, signal chain), see [`../reference/`](../reference/). This README is the *how* — current state, dev workflow, and the gotchas we've already paid for.
6
7## Status
8
9**Milestone v0.3.x — OTA-capable online firmware.** v0.3 adds end-to-end OTA: a two-slot partition layout, an HA `update` entity with installed/latest versions and a live progress bar, the `make ota-publish` workflow, and `esp_ota_mark_app_valid_cancel_rollback` boot validation so a broken firmware reverts to the previous slot. v0.2 (online mode: WiFi + MQTT + HA Discovery, round-tripped short-press) and v0.1 (offline button + audio + NVS + LED) are the foundation underneath. One binary works on every unit — identity is derived at runtime from the STA MAC.
10
11| Subsystem | State |
12| --- | --- |
13| Cargo build + flash | ✅ working |
14| GPIO input (button G39) | ✅ working — debounced, active-low |
15| I2S TX (16-bit / 44.1 kHz / stereo, Philips) | ✅ working into external MAX98357A |
16| Continuous pink noise generator (Paul Kellet IIR) | ✅ xorshift32 white → pink filter, volume-scaled |
17| Button state machine (short / long / double) | ✅ working |
18| NVS persistence (volume + direction + playing) | ✅ working |
19| RGB LED (SK6812 on G27 via RMT) | ✅ working — (net, audio) base + OTA/Error overrides + press-flash |
20| WiFi | ✅ working — STA, hostname `nightstand-<mac_hex>`, auto-reconnect |
21| MQTT client + HA discovery | ✅ working — LWT, retained discovery on (re)connect, `cmd/play`/`cmd/volume`/`cmd/update` subscribed |
22| OTA updates | ✅ working — `esp_https_ota` chunked download with HA progress bar, two-slot rollback |
23| Hardware: external MAX98357A + GRS 3FR-4 speaker | ✅ assembled on breadboard |
24
25## Module layout
26
27```
28firmware/src/
29├── main.rs — entry; spawns tasks; coordinator loop routes ButtonEvent → audio + outbound
30├── events.rs — ButtonEvent / AudioCommand / LedSignal / OutboundEvent / StateSnapshot
31├── state.rs — VOLUME_PRESETS, VolumeDirection, yo-yo cycle math, snap_to_preset_index
32├── nvs.rs — typed NVS wrapper for volume_index, volume_direction, was_playing
33├── audio.rs — I2S + xorshift white noise + Paul Kellet pink filter + audio task
34├── button.rs — 5-state button FSM + button task (owns G39 PinDriver)
35├── led.rs — SK6812 RMT driver + LED task; (net, audio) base + OTA/Error overrides
36├── network.rs — WiFi + MQTT state machine; gatekeeper for online/offline routing
37├── discovery.rs — HA Discovery JSON payloads (built via format!() — no serde_json)
38├── ota.rs — esp_https_ota wrapper with chunked progress + mark_app_valid
39├── channels.rs — FreeRTOS-queue-backed Sender/Receiver (std::sync::mpsc is broken on esp-idf)
40└── secrets.rs — toml-cfg config struct sourced from cfg.toml at compile time
41```
42
43The deliberate factoring: each task owns its peripherals exclusively; cross-task communication is via FreeRTOS-queue channels carrying typed events. The network task is the gatekeeper — it decides whether short presses toggle audio locally (offline) or wait for HA to publish back via MQTT (online), and it owns the OTA install path. Long-press cycles volume locally in both modes; double is a pure-MQTT gesture.
44
45## Build & flash
46
47Project uses direnv (`.envrc` at the project root). Open a terminal in any subdir of the project and the env loads automatically. There's also a `Makefile` at the repo root and in `firmware/` that exports the same env so you can build without direnv (CI, fresh clones, etc.).
48
49```bash
50# from anywhere in the repo:
51make firmware # cargo build --release
52make firmware-flash # headless: build + write bootloader + partition table + app
53make firmware-flash-monitor # interactive: build + flash + serial monitor (needs TTY)
54make firmware-monitor # serial monitor only
55make firmware-check # cargo check
56make firmware-ota-publish # build + save .bin to OTA dir + publish latest_version (see OTA below)
57make firmware-clean
58```
59
60Or directly with cargo:
61
62```bash
63cd firmware
64cargo build --release
65espflash flash --port /dev/ttyUSB0 \
66 --bootloader target/xtensa-esp32-espidf/release/bootloader.bin \
67 --partition-table partitions.csv \
68 --erase-parts otadata \
69 target/xtensa-esp32-espidf/release/sound-machine
70```
71
72Incremental builds are ~5 s. First-time builds are ~20 min — they download and compile ESP-IDF (~500 MB) plus all the Rust deps.
73
74### `cfg.toml` — compile-time config
75
76Before the first build, copy [`cfg.toml.example`](cfg.toml.example) to `cfg.toml` and fill in real values:
77
78```toml
79[sound-machine]
80wifi_ssid = "your-wifi-ssid"
81wifi_password = "your-wifi-password"
82mqtt_url = "mqtt://your-broker.lan:1883"
83ota_url_base = "http://firmware.example.lan/sound-machine"
84```
85
86`cfg.toml` is gitignored. The firmware checks for empty values at boot and refuses to proceed with an empty `wifi_ssid`/`wifi_password`/`mqtt_url`, so you can't accidentally flash a no-config binary. An empty `ota_url_base` only blocks OTA installs (with a warning); the rest of the firmware still runs.
87
88**Compile-time means wired-flash** — changing any of these values requires a USB reflash of every device that needs the new value. WiFi/MQTT/OTA host changes are infrequent enough to be worth the inconvenience for the simpler model.
89
90### `.envrc.private` — host-only secrets
91
92The publish flow needs three host-side variables — `OTA_LOCAL_DIR` (where to copy the .bin), `OTA_URL_BASE` (matches `cfg.toml`), and `MQTT_URL`. Copy [`../.envrc.private.example`](../.envrc.private.example) to `../.envrc.private` and fill in real values; direnv loads it automatically. `.envrc.private` is gitignored.
93
94### Monitoring without re-flashing
95
96To see logs from an already-running device:
97
98```bash
99espflash monitor --port /dev/ttyUSB0
100```
101
102`Ctrl+C` exits the monitor; the device keeps running. Anything written via `log::info!()` etc. shows up here, prefixed with the crate name and a millisecond timestamp.
103
104## OTA workflow
105
106Once a device is on v0.3.x, the next version goes out over WiFi. Two parts: the publish, and the install.
107
108### Publish (your dev machine)
109
110```bash
111# Cargo.toml: bump version
112make firmware-ota-publish
113```
114
115That target:
1161. Builds a release binary
1172. `espflash save-image`s it to `$OTA_LOCAL_DIR/sound-machine-<version>.bin`
1183. `mosquitto_pub`s the new version retained to `sound-machine/firmware/latest`
119
120Both nightstands' HA update cards light up within a couple seconds.
121
122### Install (HA UI)
123
124Click "Install" on the device's Firmware card. The device:
1251. LED switches to a magenta pulse
1262. Streams the binary into the inactive OTA slot, publishing `update_percentage` every 5 % (HA renders a progress bar)
1273. Reboots into the new slot
1284. Reconnects to WiFi + MQTT
1295. Calls `esp_ota_mark_app_valid_cancel_rollback` — confirms the new firmware works and disables the bootloader's pending-rollback timer
1306. Republishes `installed_version` matching `latest_version`; HA's card flips back to "Up to date"
131
132If the new firmware fails to reach MQTT (panic, WiFi misconfig, etc.) the bootloader rolls back to the previous slot on the next reset, and the device comes up on the old version. HA notices `installed_version` reverted and shows "Update available" again.
133
134### Bootstrap (the one-time wired flash)
135
136Going from a pre-v0.3 partition layout (single-slot) to v0.3.x's two-slot layout requires a wired `make firmware-flash` once per device. The `flash` target writes the new bootloader, partition table, and otadata in addition to the app. After that, OTA is the path forever; if you ever need wired access (changing `cfg.toml`, debugging OTA breakage in the OTA path itself), USB still works.
137
138### Webserver expectations
139
140`$OTA_LOCAL_DIR` should be served as static HTTP at `$OTA_URL_BASE`. Plain HTTP is fine and intentional — the trust boundary is the LAN, same as MQTT. ESP-IDF will refuse plain HTTP for OTA *unless* `CONFIG_ESP_HTTPS_OTA_ALLOW_HTTP=y` is set in `sdkconfig.defaults` (it is).
141
142## Toolchain (one-time setup)
143
144Already done on Chris's vega; documented here for re-setup or a second machine.
145
146```bash
147# 1. Rust 1.88+ stable (espup needs it)
148rustup update stable
149
150# 2. espup CLI itself
151cargo install espup
152
153# 3. The Xtensa fork of Rust + LLVM + GCC for ESP32
154espup install --targets esp32 # writes ~/export-esp.sh — see "Home dir cleanup" below
155
156# 4. Flasher + linker shim
157cargo install espflash
158cargo install ldproxy
159
160# 5. System packages (Ubuntu/Debian)
161sudo apt install ninja-build # ESP-IDF's CMake build needs it
162sudo apt install libxml2 # see "libxml2 shim" gotcha below
163```
164
165### Home dir cleanup
166
167`espup install` writes `~/export-esp.sh`. We don't keep that — the two `export` lines it contains are inlined into the project's [`.envrc`](../.envrc) so the home dir stays clean. If you re-run `espup install` (e.g., for a toolchain update), copy the regenerated `~/export-esp.sh` contents into `.envrc` and delete the home-dir file. Or pass `--export-file ./.envrc-esp` to `espup install` and source that explicitly.
168
169## Known gotchas (so we don't relearn these)
170
171These all bit us during initial setup. Some are environmental, some are esp-rs ecosystem quirks.
172
173### 1. `libxml2.so.2` missing on Ubuntu Questing (and any libxml2 ≥ 2.14 distro)
174
175ESP-IDF's bundled esp-clang is linked against the older `libxml2.so.2` SONAME, but Ubuntu Questing ships libxml2 with SONAME `.so.16` (upstream bumped it in 2.14). First `cargo build` fails with:
176
177```
178clang: error while loading shared libraries: libxml2.so.2: cannot open shared object file
179```
180
181**Fix:** project-local symlink in `firmware/.lib/libxml2.so.2 → /usr/lib/x86_64-linux-gnu/libxml2.so.16`, exposed via `LD_LIBRARY_PATH` in [`.envrc`](../.envrc). The clang/libxml2 ABI surface is small and stable enough that the SONAME alias works. The `.lib/` dir is gitignored.
182
183### 2. Default Cargo features pulled in `embassy` and broke linking
184
185The vanilla `esp-idf-template` enables `embassy` in default features, which adds `embassy-executor` to the dep graph. Without an actual embassy executor wired up, the linker errors with `undefined reference to __pender`. We don't use embassy in v1, so [`Cargo.toml`](Cargo.toml) drops it from `default = [...]`. Re-add it (and properly wire an executor) if/when we want async.
186
187### 3. `I2sDriver::new_std_tx` pin order is not what intuition expects
188
189Signature in `esp-idf-hal 0.45.2`:
190
191```rust
192new_std_tx(i2s, config, bclk, dout, mclk, ws)
193```
194
195That's **`dout` before `ws`**, not `bclk → ws → dout` as you'd guess from reading "Bclk, Ws, Data" in audio convention. Getting them swapped produces:
196- "Static" instead of a tone (data on the LRCK line, clock on the data line)
197- A persistent quiet squeal during silence (WS clock ticking on the data input)
198- The amp running warm (sustained switching, no real audio output)
199
200The compiler accepts both orderings since the pin types are generic. Always check the signature against the version of `esp-idf-hal` you're on. Symptom: garbage output where you'd expect tone.
201
202### 4. DMA replays stale data when starved
203
204ESP-IDF's I2S driver, once enabled, will keep clocking out whatever's in its DMA buffer. If you write data once and then stop, the buffer keeps getting replayed → an "idling" device produces a quiet repeating noise loop.
205
206**Fix in `main.rs`:**
207- Pre-fill DMA with zeros via `i2s.preload_data(&silence)` *before* `tx_enable()`
208- In the main loop, write a silence chunk every iteration. `i2s.write` blocks for ~23 ms while DMA consumes it, which is also a perfectly fine button-poll cadence — no `sleep` needed.
209
210This keeps the amp's input continuously fed, even when "doing nothing."
211
212### 5. `std::sync::mpsc` and `crossbeam-channel` are broken on esp-idf-rs
213
214Both libraries assume the GNU/newlib `pthread_mutex_t` ABI (40 bytes, zero-initializer). ESP-IDF's pthread layer uses a 4-byte handle with `0xffffffff` as the lazy-init sentinel. Sending across a channel under contention triggers `pthread_mutex_lock` on a misaligned struct → `LoadProhibited` exception. Symptom: `<channel_type>::try_recv` or `::send` panics in the bowels of mpsc.
215
216**Fix:** [`channels.rs`](src/channels.rs) wraps `esp_idf_svc::hal::task::queue::Queue` (the FreeRTOS native queue, ISR-safe, byte-copy semantics) with `Sender`/`Receiver` types that mimic the mpsc API. `T: Copy` is required (FreeRTOS queues memcpy values) — usually fine for typed events. Use these in place of `std::sync::mpsc::channel` everywhere.
217
218### 6. ESP-IDF's `CONFIG_PARTITION_TABLE_FILENAME` resolves relative to the CMake source dir, not your project
219
220The ESP-IDF Kconfig docs imply the path is relative to your project. In an esp-idf-rs build, the CMake "project" is the embuild-generated directory under `target/`, so a relative path silently fails. The Makefile generates a small sdkconfig overlay at build time with the *absolute* path to `partitions.csv`, chained via `ESP_IDF_SDKCONFIG_DEFAULTS`. The committed `sdkconfig.defaults` only carries the boolean (`CONFIG_PARTITION_TABLE_CUSTOM=y`).
221
222### 7. `esp_https_ota` rejects plain HTTP unless an opt-in flag is set
223
224Despite working fine at the protocol level, `esp_https_ota` validates the URL scheme and refuses plain HTTP unless `CONFIG_ESP_HTTPS_OTA_ALLOW_HTTP=y` is in sdkconfig. With the flag missing, `esp_https_ota_begin` returns immediately with an error and never opens a socket — webserver logs show no GET requests. The flag is set in `sdkconfig.defaults`.
225
226## Layout
227
228```
229firmware/
230├── Cargo.toml # deps + features
231├── rust-toolchain.toml # pins to the `esp` channel
232├── .cargo/config.toml # target = xtensa-esp32-espidf, ldproxy, runner
233├── build.rs # embuild bootstrap
234├── Makefile # build/flash/ota-publish entry points
235├── sdkconfig.defaults # ESP-IDF kconfig knobs (4MB flash, two-OTA layout, allow-HTTP-OTA)
236├── partitions.csv # custom two-slot OTA partition table
237├── cfg.toml.example # template for compile-time config
238├── cfg.toml # real values (gitignored)
239├── src/
240│ └── *.rs # see "Module layout" above
241├── .lib/ # libxml2 shim (gitignored)
242├── target/ # cargo build output (gitignored)
243│ └── gen/ # generated sdkconfig overlay for absolute partitions.csv path
244└── .embuild/ # embuild-managed ESP-IDF clone (gitignored)
245```
246
247## Hardware-pinout cheat sheet (Atom Echo)
248
249| Pin | Function (this firmware) | Notes |
250| --- | --- | --- |
251| G39 | Button input | Active-low, on-board pull-up, input-only pin |
252| G26 | I2S BCLK → MAX98357A | GROVE yellow |
253| G21 | I2S DOUT → MAX98357A | Side header |
254| G32 | I2S WS/LRCK → MAX98357A | GROVE white |
255| G27 | SK6812 RGB LED data | On-board, top-face button cap |
256| G19 | (reserved — onboard NS4168 BCLK) | Driven silent: no I2S routed here |
257| G22 | (reserved — onboard NS4168 DOUT) | Driven silent: no I2S routed here |
258| G33 | (reserved — onboard NS4168 LRCK / mic DATA) | Driven silent: no I2S routed here |
259| G25 | Side header / spare | Free |
260
261Authoritative pinout: [`../reference/atom-echo/pinmap.md`](../reference/atom-echo/pinmap.md).
262
263## Sources
264
265- [ESP-IDF documentation (v5.3)](https://docs.espressif.com/projects/esp-idf/en/v5.3.3/esp32/)
266- [`esp-idf-svc` on docs.rs](https://docs.rs/esp-idf-svc/0.51.0/esp_idf_svc/)
267- [`esp-idf-hal` on docs.rs](https://docs.rs/esp-idf-hal/0.45.2/esp_idf_hal/)
268- [esp-rs / Embedded Rust on Espressif (book)](https://docs.esp-rs.org/book/)
269- [`espup` README — env setup options](https://github.com/esp-rs/espup#environment-variables-setup)
270- [Project design docs](../reference/) — operating modes, MQTT contract, signal chain, pin map