nano
forked niri forked from okami.mom/niri
Star 1
Fork 0
Watch

fix(discord-embeddings): stop poison rows from perpetually breaking embedding sync #65

Merged callie.on-her.computer
targeting main from deleted fork:fix/embedding-poison-rows
Merged opened by and assigned to callie.on-her.computer
targeting main from deleted fork:fix/embedding-poison-rows

the bug #

4 test-fixture rows with non-numeric ids (msg_dm_1 etc) got seeded into the prod DB on Aug 2. The embed write path does BigInt(row.message_id), which throws on those. Because the whole batch runs in one transaction, one throw rolled back the entire batch - so nothing persisted, the rows stayed pending forever, and the backfill worker re-selected them every cycle. An infinite loop that (once the key was disabled) also burned ~$10 in API spend by subdividing the failing batch down to one call per pending message.

three fixes #

  1. per-row robustness - malformed rows are quarantined (logged + meta recorded so they aren't re-selected) instead of throwing and rolling back everyone else's embeddings. Per-REQUEST failures (auth/quota) bail hard; per-ROW failures skip just that row.
  2. circuit breaker - fatal errors (401/403/429/quota) halt the worker instead of retrying forever, so a dead key can't spin into unbounded spend.
  3. real test isolation - initDb() takes an explicit path override + resolves NANO_DB_PATH lazily at call time instead of freezing HOME at import. The old test set process.env after import (a no-op) and wrote fixtures into the real DB - the original source of the poison rows. Fixtures now use realistic numeric snowflake ids so a future leak can't poison anything.

tsc clean, state.test.ts passes. The 4 poison rows were also deleted from the live DB out-of-band.

Commits 0