bound the queue · microcosm.blue/microcosm-rs@b345084

Constellation, Spacedust, Slingshot, UFOs: atproto crates and services for microcosm

bound the queue

helpfully, jetstream got stuck for a bunch of hours, so there was a lot of firehose to backfill and find out the actual performance limit and how much the queue can grow.

the queue grew to millions before the raspi finally ran out of memory and we died with a failed allocation. i bounded it at 100k to let it finish catching up, and that worked.

2048 is still seems fairly arbitrary, but it's not obviously too wildly high or low.

phil 1 year ago b345084d b4d170e7

+1 -1

1 changed file

expand all

link_aggregator

src

consumer

mod.rs

+1 -1

link_aggregator/src/consumer/mod.rs

··· 42 42 thread::spawn(move || consume_jsonl_file(f, sender)), 43 43 ) 44 44 } else { 45 - let (sender, receiver) = flume::unbounded(); // eek 45 + let (sender, receiver) = flume::bounded(2048); // eek 46 46 let cursor = store.get_cursor().unwrap(); 47 47 ( 48 48 receiver,

Configure Feed

Configure Feed