Monorepo for Aesthetic.Computer
aesthetic.computer
boot report
+129
+129
fedac/native/BOOT-REGRESSION-REPORT.md
+129
fedac/native/BOOT-REGRESSION-REPORT.md
···
1
+
# AC Native OS Boot Regression Report
2
+
3
+
**Date:** 2026-03-21
4
+
**Affected Machine:** ThinkPad Yoga 11e Gen 5 (Intel Gemini Lake, UHD 600/605)
5
+
**Symptom:** "no DRM device" — `/dev/dri/card0` never appears
6
+
**Last Working Build:** ~March 19, 2026 (devcontainer builds)
7
+
**Broken Since:** March 20, 2026 (oven builds began)
8
+
9
+
---
10
+
11
+
## Root Cause Analysis
12
+
13
+
The boot regression has **three compounding causes**, all introduced between March 19-21:
14
+
15
+
### 1. Firmware Loading Changed (HIGH IMPACT)
16
+
17
+
**Before (working):**
18
+
```
19
+
CONFIG_EXTRA_FIRMWARE="iwlwifi-9260-th-b0-jf-b0-46.ucode iwlwifi-cc-a0-77.ucode ..."
20
+
CONFIG_EXTRA_FIRMWARE_DIR="/workspaces/aesthetic-computer/fedac/native/build/firmware"
21
+
```
22
+
WiFi firmware was **compiled directly into the kernel binary**. While this was for WiFi, the kernel's firmware loading infrastructure was tested and working at early boot.
23
+
24
+
**After (broken):**
25
+
```
26
+
CONFIG_EXTRA_FIRMWARE=""
27
+
CONFIG_EXTRA_FIRMWARE_DIR=""
28
+
```
29
+
Changed to load firmware from initramfs at runtime. This was done to fix the oven build (the oven doesn't have the firmware at the devcontainer path). But it may have broken early firmware loading for i915 GPU, which also needs firmware blobs (DMC, GuC, HuC) before it can probe the GPU.
30
+
31
+
**Why this matters for the Yoga 11e:** Gemini Lake GPUs (GLK) require `glk_dmc_ver1_04.bin` firmware to initialize the display. If the firmware loader can't find it early enough, i915 fails to probe and `/dev/dri/card0` is never created.
32
+
33
+
### 2. Init Script Complexity (MEDIUM IMPACT)
34
+
35
+
**Before (working):** 35 lines
36
+
```sh
37
+
#!/bin/sh
38
+
mount -t devtmpfs devtmpfs /dev
39
+
mkdir -p /dev/pts /tmp /run
40
+
# ... basic mounts ...
41
+
# Wait for GPU (up to 1 second)
42
+
while [ ! -e /dev/dri/card0 ] && [ $i -lt 100 ]; do
43
+
usleep 10000; i=$((i+1))
44
+
done
45
+
exec /ac-native /piece.mjs
46
+
```
47
+
48
+
**After (broken):** 163+ lines with:
49
+
- USB device detection and logging
50
+
- Crash recovery loop (5 retries)
51
+
- Claude credentials restoration
52
+
- seatd/cage Wayland setup
53
+
- zram swap with error checking
54
+
- cpufreq governor setting
55
+
- Diagnostic dump on failure
56
+
57
+
The added complexity introduces more failure points. Commands like `mkdir`, `sleep`, `mount` need to work — and when busybox is missing or shared libs are broken, the init fails silently.
58
+
59
+
### 3. Busybox vs Dynamic Binaries (MEDIUM IMPACT)
60
+
61
+
**Before (working):** Statically-linked busybox provided `/bin/sh` and all shell utilities. Zero shared library dependencies for basic commands.
62
+
63
+
**After (broken, local builds):** The devcontainer's `build-and-flash-initramfs.sh` copies individual Fedora binaries (bash, coreutils) which are dynamically linked and need `libc.so.6`, `libtinfo.so.6`, `libffi.so.8`, etc. If any lib is missing, basic commands like `sleep` and `mkdir` fail.
64
+
65
+
**After (broken, oven builds):** The oven correctly uses busybox (statically linked), but the binary (`ac-native`) is compiled with `musl-gcc` while shared libraries are glibc — they're incompatible. Later fixed to use `gcc`, but the initramfs was still missing the glibc dynamic linker initially.
66
+
67
+
---
68
+
69
+
## Timeline of Changes
70
+
71
+
| Date | Commit | Change | Impact |
72
+
|------|--------|--------|--------|
73
+
| Mar 19 | `1358c1f28` | Last known working build | Baseline |
74
+
| Mar 20 | `1d8a56758` | Added `ac-os pull` command | Minor |
75
+
| Mar 20 | `efeec75dc` | Oven build race condition fix | Started oven builds |
76
+
| Mar 20 | `11bb20c2f` | Cleared CONFIG_EXTRA_FIRMWARE | **BROKE firmware loading** |
77
+
| Mar 21 | `c847d82c2` | Added apt/Ubuntu package install | Fixed oven deps |
78
+
| Mar 21 | `8dc9ead1b` | musl linker fix attempt | Partial fix |
79
+
| Mar 21 | `d804c9776` | Switched to gcc from musl-gcc | Fixed linking |
80
+
| Mar 21 | `c4398bccd` | Added AMD/Nvidia/simpledrm GPU drivers | Added complexity |
81
+
| Mar 21 | `1df5dba77` | Created /dev/console in initramfs | Partial fix |
82
+
| Mar 21 | `97ec7cf23` | Added crash recovery to init | Added complexity |
83
+
| Mar 21 | `f9c4999da` | Fixed zstd firmware decompression | Fixed firmware format |
84
+
| Mar 21 | Multiple | Various init script changes | More complexity |
85
+
86
+
---
87
+
88
+
## The Fix
89
+
90
+
### Option A: Revert to Simple (Recommended)
91
+
1. **Restore `CONFIG_EXTRA_FIRMWARE`** with i915 GLK firmware added alongside WiFi firmware
92
+
2. **Revert init to the simple 35-line version** that was working
93
+
3. **Ensure busybox is always used** as `/bin/sh` (statically linked)
94
+
4. Add features back incrementally, testing on the Yoga 11e each time
95
+
96
+
### Option B: Fix Forward
97
+
1. Add `CONFIG_FW_LOADER_COMPRESS_ZSTD=y` to kernel config so firmware can load from compressed initramfs files
98
+
2. Ensure initramfs firmware is at `/lib/firmware/` accessible before i915 probes
99
+
3. Keep crash recovery but make it more robust (no shared lib deps in crash path)
100
+
4. Test with QEMU smoke test before every flash
101
+
102
+
### Option C: Hybrid
103
+
1. Build i915 firmware INTO the kernel (`CONFIG_EXTRA_FIRMWARE`) for reliability
104
+
2. Load other firmware (WiFi) from initramfs (less critical, happens after boot)
105
+
3. Keep simple init for boot, add features as post-boot services
106
+
107
+
---
108
+
109
+
## Oven vs Devcontainer Build Differences
110
+
111
+
| Aspect | Devcontainer (Fedora) | Oven (Ubuntu) |
112
+
|--------|----------------------|---------------|
113
+
| Compiler | gcc (glibc) | gcc (glibc) — was musl-gcc |
114
+
| Busybox | Not installed by default | Installed, statically linked |
115
+
| Shared libs | Fedora .so files | Ubuntu .so files |
116
+
| Mesa/GPU | Full Mesa stack available | Mesa installed via apt |
117
+
| Firefox | Available | Not available |
118
+
| Claude Code | Available (225MB) | Available |
119
+
| Kernel compile | 2 cores, ~15 min | 8 cores, ~2 min |
120
+
| Initramfs size | ~287MB (with Firefox+Claude) | ~200MB (no Firefox) |
121
+
122
+
---
123
+
124
+
## Recommended Next Steps
125
+
126
+
1. **Immediate:** Flash a build from the `1358c1f28` commit (last known working) to confirm the Yoga 11e still works with old code
127
+
2. **Then:** Apply Option C (hybrid) — bake GPU firmware in, load WiFi firmware from initramfs
128
+
3. **Then:** Add features back one at a time, testing each on the Yoga 11e
129
+
4. **Long-term:** QEMU smoke test in oven pipeline catches boot failures before upload