A fork of https://github.com/crosspoint-reader/crosspoint-reader
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

fix: boot looping when opening large XTC files (#1648)

Opening XTC files with a high page count (e.g. *The Magic Mountain* at
4,187 pages) causes an immediate `abort()` crash and reboot loop. The
device becomes unusable until the book is removed from the SD card.

**Crash log:**
```
abort() was called at 0x4214a5fb on core 0
```

### Root cause

During `XtcParser::open()`, the parser calls
`m_pageTable.resize(pageCount)` to load the entire page table into RAM.
Each `PageInfo` entry is 16 bytes, so:

- 4,187 pages x 16 bytes = **66,992 bytes (~65KB)** as a single
contiguous heap allocation

On the ESP32-C3 with ~380KB total RAM (no PSRAM), this allocation fails
after firmware, fonts, and the activity system are already loaded.
Because the firmware is compiled with `-fno-exceptions`, the failed
`new` inside `std::vector::resize()` calls `abort()` instead of
throwing.

This affects any XTC file with roughly 3,000+ pages, depending on heap
state at the time of loading.

## Solution

Replace the bulk page table allocation with on-demand reads from the SD
card. Instead of loading all page table entries into a vector at file
open, we now:

1. Read only the **first** page table entry at open time (to get default
page dimensions)
2. Read a **single** 16-byte entry from the SD card each time a page is
loaded

This reduces page table memory usage from `pageCount * 16` bytes to
**zero bytes**, regardless of how many pages the file contains.

### Changes

| File | What changed |
|------|-------------|
| `XtcParser.h` | Removed `std::vector<PageInfo> m_pageTable`. Added
`readPageTableEntry()` for on-demand reads. |
| `XtcParser.cpp` | Replaced `readPageTable()` with
`readFirstPageInfo()`. Updated `getPageInfo()`, `loadPage()`, and
`loadPageStreaming()` to seek and read individual entries from the file.
|

## Trade-offs

### Performance

Each page turn now requires one additional SD card seek + 16-byte read
to look up the page table entry before reading the page data itself.

- SD card sequential read latency: ~0.1-0.5ms for a 16-byte read
- E-ink full display refresh: ~1,000-2,000ms

I personally can't see any performance difference while reading and the
trade off of not boot looping seems to make this well worth it.

### Memory

| Metric | Before | After |
|--------|--------|-------|
| Page table RAM (4,187 pages) | ~65KB | 0 bytes |
| Page table RAM (1,000 pages) | ~16KB | 0 bytes |
| Page table RAM (max 65,535 pages) | ~1MB (impossible) | 0 bytes |

authored by

Justin Mitchell and committed by
GitHub
fedcb2f5 a888978f

+143 -59
+1 -1
lib/Xtc/Xtc.cpp
··· 103 103 return parser->hasChapters(); 104 104 } 105 105 106 - const std::vector<xtc::ChapterInfo>& Xtc::getChapters() const { 106 + const std::vector<xtc::ChapterInfo>& Xtc::getChapters() { 107 107 static const std::vector<xtc::ChapterInfo> kEmpty; 108 108 if (!loaded || !parser) { 109 109 return kEmpty;
+1 -1
lib/Xtc/Xtc.h
··· 58 58 std::string getTitle() const; 59 59 std::string getAuthor() const; 60 60 bool hasChapters() const; 61 - const std::vector<xtc::ChapterInfo>& getChapters() const; 61 + const std::vector<xtc::ChapterInfo>& getChapters(); 62 62 63 63 // Cover image support (for sleep screen) 64 64 std::string getCoverBmpPath() const;
+128 -53
lib/Xtc/Xtc/XtcParser.cpp
··· 21 21 m_defaultHeight(DISPLAY_HEIGHT), 22 22 m_bitDepth(1), 23 23 m_hasChapters(false), 24 + m_chaptersLoaded(false), 24 25 m_lastError(XtcError::OK) { 25 26 memset(&m_header, 0, sizeof(m_header)); 26 27 } ··· 32 33 if (m_isOpen) { 33 34 close(); 34 35 } 36 + 37 + m_filepath = filepath; 35 38 36 39 // Open file 37 40 if (!Storage.openFileForRead("XTC", filepath, m_file)) { ··· 64 67 m_file.close(); 65 68 return m_lastError; 66 69 } 70 + // Trim excess capacity from metadata strings 71 + m_title.shrink_to_fit(); 72 + m_author.shrink_to_fit(); 67 73 } 68 74 69 - // Read page table 70 - m_lastError = readPageTable(); 75 + // Read first page info for default dimensions (no bulk page table allocation) 76 + m_lastError = readFirstPageInfo(); 71 77 if (m_lastError != XtcError::OK) { 72 - LOG_DBG("XTC", "Failed to read page table: %s", errorToString(m_lastError)); 78 + LOG_DBG("XTC", "Failed to read first page info: %s", errorToString(m_lastError)); 73 79 // Explicit close() required: member variable persists beyond function scope 74 80 m_file.close(); 75 81 return m_lastError; 76 82 } 77 83 78 - // Read chapters if present 79 - m_lastError = readChapters(); 80 - if (m_lastError != XtcError::OK) { 81 - LOG_DBG("XTC", "Failed to read chapters: %s", errorToString(m_lastError)); 82 - // Explicit close() required: member variable persists beyond function scope 83 - m_file.close(); 84 - return m_lastError; 85 - } 84 + // Defer chapter parsing until actually needed (lazy load). 85 + // Chapter strings can use significant heap; keeping them out of memory 86 + // during rendering leaves more room for the page bitmap buffer. 87 + m_hasChapters = (m_header.hasChapters == 1); 88 + m_chaptersLoaded = false; 89 + 90 + // Close the source file to free its internal SdFat buffers. 91 + // It will be reopened on-demand for page table lookups and bitmap reads. 92 + m_file.close(); 86 93 87 94 m_isOpen = true; 88 95 LOG_DBG("XTC", "Opened file: %s (%u pages, %dx%d)", filepath, m_header.pageCount, m_defaultWidth, m_defaultHeight); ··· 90 97 } 91 98 92 99 void XtcParser::close() { 93 - if (m_isOpen) { 94 - // Explicit close() required: member variable persists beyond function scope 95 - m_file.close(); 96 - m_isOpen = false; 97 - } 98 - m_pageTable.clear(); 100 + closeFile(); 101 + m_isOpen = false; 102 + m_chaptersLoaded = false; 99 103 m_chapters.clear(); 100 104 m_title.clear(); 105 + m_author.clear(); 101 106 m_hasChapters = false; 102 107 memset(&m_header, 0, sizeof(m_header)); 103 108 } 104 109 110 + bool XtcParser::ensureFileOpen() { 111 + if (m_file.isOpen()) { 112 + return true; 113 + } 114 + return Storage.openFileForRead("XTC", m_filepath.c_str(), m_file); 115 + } 116 + 117 + void XtcParser::closeFile() { 118 + if (m_file.isOpen()) { 119 + m_file.close(); 120 + } 121 + } 122 + 105 123 XtcError XtcParser::readHeader() { 106 124 // Read first 56 bytes of header 107 125 size_t bytesRead = m_file.read(reinterpret_cast<uint8_t*>(&m_header), sizeof(XtcHeader)); ··· 169 187 return XtcError::OK; 170 188 } 171 189 172 - XtcError XtcParser::readPageTable() { 190 + XtcError XtcParser::readFirstPageInfo() { 173 191 if (m_header.pageTableOffset == 0) { 174 192 LOG_DBG("XTC", "Page table offset is 0, cannot read"); 175 193 return XtcError::CORRUPTED_HEADER; 176 194 } 177 195 178 - // Seek to page table 196 + // Verify the file is large enough to contain the full page table 197 + const uint64_t fileSize = m_file.size(); 198 + const uint64_t pageTableSize = static_cast<uint64_t>(m_header.pageCount) * sizeof(PageTableEntry); 199 + if (m_header.pageTableOffset < sizeof(XtcHeader) || m_header.pageTableOffset > fileSize || 200 + pageTableSize > fileSize - m_header.pageTableOffset) { 201 + LOG_DBG("XTC", "Page table exceeds file bounds"); 202 + return XtcError::CORRUPTED_HEADER; 203 + } 204 + 205 + // Read only the first entry to get default page dimensions 206 + // All other entries are read on-demand via readPageTableEntry() 207 + // This avoids allocating pageCount * 16 bytes (e.g. 65KB for 4000+ pages) 208 + PageTableEntry entry; 179 209 if (!m_file.seek(m_header.pageTableOffset)) { 180 210 LOG_DBG("XTC", "Failed to seek to page table at %llu", m_header.pageTableOffset); 181 211 return XtcError::READ_ERROR; 182 212 } 213 + size_t bytesRead = m_file.read(reinterpret_cast<uint8_t*>(&entry), sizeof(PageTableEntry)); 214 + if (bytesRead != sizeof(PageTableEntry)) { 215 + LOG_DBG("XTC", "Failed to read first page table entry"); 216 + return XtcError::READ_ERROR; 217 + } 183 218 184 - m_pageTable.resize(m_header.pageCount); 219 + m_defaultWidth = entry.width; 220 + m_defaultHeight = entry.height; 221 + 222 + LOG_DBG("XTC", "Page table validated: %u pages, default %dx%d", m_header.pageCount, m_defaultWidth, m_defaultHeight); 223 + return XtcError::OK; 224 + } 225 + 226 + bool XtcParser::readPageTableEntry(uint32_t pageIndex, PageInfo& info) { 227 + if (pageIndex >= m_header.pageCount) { 228 + return false; 229 + } 185 230 186 - // Read page table entries 187 - for (uint16_t i = 0; i < m_header.pageCount; i++) { 188 - PageTableEntry entry; 189 - size_t bytesRead = m_file.read(reinterpret_cast<uint8_t*>(&entry), sizeof(PageTableEntry)); 190 - if (bytesRead != sizeof(PageTableEntry)) { 191 - LOG_DBG("XTC", "Failed to read page table entry %u", i); 192 - return XtcError::READ_ERROR; 193 - } 231 + if (!ensureFileOpen()) { 232 + LOG_DBG("XTC", "Failed to reopen file for page table read"); 233 + return false; 234 + } 194 235 195 - m_pageTable[i].offset = static_cast<uint32_t>(entry.dataOffset); 196 - m_pageTable[i].size = entry.dataSize; 197 - m_pageTable[i].width = entry.width; 198 - m_pageTable[i].height = entry.height; 199 - m_pageTable[i].bitDepth = m_bitDepth; 236 + // Seek to the specific page table entry on the SD card 237 + const uint64_t entryOffset = m_header.pageTableOffset + static_cast<uint64_t>(pageIndex) * sizeof(PageTableEntry); 238 + if (!m_file.seek(entryOffset)) { 239 + LOG_DBG("XTC", "Failed to seek to page table entry %lu at %llu", pageIndex, entryOffset); 240 + return false; 241 + } 200 242 201 - // Update default dimensions from first page 202 - if (i == 0) { 203 - m_defaultWidth = entry.width; 204 - m_defaultHeight = entry.height; 205 - } 243 + PageTableEntry entry; 244 + size_t bytesRead = m_file.read(reinterpret_cast<uint8_t*>(&entry), sizeof(PageTableEntry)); 245 + if (bytesRead != sizeof(PageTableEntry)) { 246 + LOG_DBG("XTC", "Failed to read page table entry %lu", pageIndex); 247 + return false; 206 248 } 207 249 208 - LOG_DBG("XTC", "Read %u page table entries", m_header.pageCount); 209 - return XtcError::OK; 250 + info.offset = static_cast<uint32_t>(entry.dataOffset); 251 + info.size = entry.dataSize; 252 + info.width = entry.width; 253 + info.height = entry.height; 254 + info.bitDepth = m_bitDepth; 255 + return true; 210 256 } 211 257 212 258 XtcError XtcParser::readChapters() { 213 - m_hasChapters = false; 214 259 m_chapters.clear(); 260 + 261 + if (!ensureFileOpen()) { 262 + return XtcError::READ_ERROR; 263 + } 215 264 216 265 uint8_t hasChaptersFlag = 0; 217 266 if (!m_file.seek(0x0B)) { ··· 242 291 return XtcError::OK; 243 292 } 244 293 245 - uint64_t maxOffset = 0; 246 - if (m_header.pageTableOffset > chapterOffset) { 294 + // Clamp maxOffset to fileSize so bogus header values can't inflate chapterCount 295 + uint64_t maxOffset = fileSize; 296 + if (m_header.pageTableOffset > chapterOffset && m_header.pageTableOffset <= fileSize) { 247 297 maxOffset = m_header.pageTableOffset; 248 - } else if (m_header.dataOffset > chapterOffset) { 298 + } else if (m_header.dataOffset > chapterOffset && m_header.dataOffset <= fileSize) { 249 299 maxOffset = m_header.dataOffset; 250 - } else { 251 - maxOffset = fileSize; 252 300 } 253 301 254 302 if (maxOffset <= chapterOffset) { ··· 266 314 return XtcError::READ_ERROR; 267 315 } 268 316 317 + m_chapters.reserve(chapterCount); 269 318 std::vector<uint8_t> chapterBuf(chapterSize); 270 319 for (size_t i = 0; i < chapterCount; i++) { 271 320 if (m_file.read(chapterBuf.data(), chapterSize) != chapterSize) { ··· 315 364 return XtcError::OK; 316 365 } 317 366 318 - bool XtcParser::getPageInfo(uint32_t pageIndex, PageInfo& info) const { 319 - if (pageIndex >= m_pageTable.size()) { 320 - return false; 367 + const std::vector<ChapterInfo>& XtcParser::getChapters() { 368 + // Lazy load chapters on first access 369 + if (!m_chaptersLoaded && m_hasChapters) { 370 + const XtcError err = readChapters(); 371 + if (err != XtcError::OK) { 372 + LOG_ERR("XTC", "Failed to lazy-load chapters: %s", errorToString(err)); 373 + m_hasChapters = false; 374 + m_chapters.clear(); 375 + } 376 + m_chaptersLoaded = true; 377 + // Close file after chapter read to free buffers for rendering 378 + closeFile(); 321 379 } 322 - info = m_pageTable[pageIndex]; 323 - return true; 380 + return m_chapters; 324 381 } 325 382 383 + bool XtcParser::getPageInfo(uint32_t pageIndex, PageInfo& info) { return readPageTableEntry(pageIndex, info); } 384 + 326 385 size_t XtcParser::loadPage(uint32_t pageIndex, uint8_t* buffer, size_t bufferSize) { 327 386 if (!m_isOpen) { 328 387 m_lastError = XtcError::FILE_NOT_FOUND; ··· 334 393 return 0; 335 394 } 336 395 337 - const PageInfo& page = m_pageTable[pageIndex]; 396 + PageInfo page; 397 + if (!readPageTableEntry(pageIndex, page)) { 398 + m_lastError = XtcError::READ_ERROR; 399 + return 0; 400 + } 401 + 402 + if (!ensureFileOpen()) { 403 + m_lastError = XtcError::FILE_NOT_FOUND; 404 + return 0; 405 + } 338 406 339 407 // Seek to page data 340 408 if (!m_file.seek(page.offset)) { ··· 402 470 return XtcError::PAGE_OUT_OF_RANGE; 403 471 } 404 472 405 - const PageInfo& page = m_pageTable[pageIndex]; 473 + PageInfo page; 474 + if (!readPageTableEntry(pageIndex, page)) { 475 + return XtcError::READ_ERROR; 476 + } 477 + 478 + if (!ensureFileOpen()) { 479 + return XtcError::FILE_NOT_FOUND; 480 + } 406 481 407 482 // Seek to page data 408 483 if (!m_file.seek(page.offset)) {
+13 -4
lib/Xtc/Xtc/XtcParser.h
··· 23 23 * 24 24 * Reads XTC files from SD card and extracts page data. 25 25 * Designed for ESP32-C3's limited RAM (~380KB) using streaming. 26 + * 27 + * The source file is kept closed between reads to free heap for rendering. 28 + * It is reopened on-demand for page table lookups and bitmap data reads. 26 29 */ 27 30 class XtcParser { 28 31 public: ··· 42 45 uint8_t getBitDepth() const { return m_bitDepth; } // 1 = XTC/XTG, 2 = XTCH/XTH 43 46 44 47 // Page information 45 - bool getPageInfo(uint32_t pageIndex, PageInfo& info) const; 48 + bool getPageInfo(uint32_t pageIndex, PageInfo& info); 46 49 47 50 /** 48 51 * Load page bitmap (raw 1-bit data, skipping XTG header) ··· 72 75 std::string getAuthor() const { return m_author; } 73 76 74 77 bool hasChapters() const { return m_hasChapters; } 75 - const std::vector<ChapterInfo>& getChapters() const { return m_chapters; } 78 + const std::vector<ChapterInfo>& getChapters(); 76 79 77 80 // Validation 78 81 static bool isValidXtcFile(const char* filepath); ··· 82 85 83 86 private: 84 87 FsFile m_file; 88 + std::string m_filepath; 85 89 bool m_isOpen; 86 90 XtcHeader m_header; 87 - std::vector<PageInfo> m_pageTable; 88 91 std::vector<ChapterInfo> m_chapters; 89 92 std::string m_title; 90 93 std::string m_author; ··· 92 95 uint16_t m_defaultHeight; 93 96 uint8_t m_bitDepth; // 1 = XTC/XTG (1-bit), 2 = XTCH/XTH (2-bit) 94 97 bool m_hasChapters; 98 + bool m_chaptersLoaded; 95 99 XtcError m_lastError; 96 100 97 101 // Internal helper functions 98 102 XtcError readHeader(); 99 - XtcError readPageTable(); 103 + XtcError readFirstPageInfo(); 100 104 XtcError readTitle(); 101 105 XtcError readAuthor(); 102 106 XtcError readChapters(); 107 + bool readPageTableEntry(uint32_t pageIndex, PageInfo& info); 108 + 109 + // File handle management — reopen on demand, close after use 110 + bool ensureFileOpen(); 111 + void closeFile(); 103 112 }; 104 113 105 114 } // namespace xtc