turss reindex re-embedded the whole captured set one article at a time —
load+parse (CPU) → embed (Metal) → upsert (redb), each awaited in series — so
the GPU idled during every parse and write, and each embed call carried only one
article's handful of chunks. On a large store the full rebuild took hours.
Refactor index_article into three shared, blocking-pool stage helpers
(load_and_chunk / embed_chunks / upsert_pairs); the per-fetch path is now a
thin composition of them with identical behavior.
Replace the sequential reindex loop with a pipeline that overlaps the three stages across articles:
- a parse-ahead producer streams
(id, chunks)over a bounded channel so CPU parsing runs ahead of the GPU, - a batch-embed driver accumulates whole articles up to a soft chunk cap (REINDEX_BATCH_CHUNKS=64) and embeds each batch in one call, splitting the vectors back per article, and
- upserts run on a bounded JoinSet so redb writes overlap the next batch's embed.
Correctness is preserved: a batch-embed error (or length mismatch) falls back to per-article embedding so one bad article can't sink the batch; per-article upsert failures count only themselves; the model stamp is still advanced only on a clean pass; progress is reported from a single drain point, ending at (total, total).
Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com