ptr.pet ptr.pet Hydrant is using 37+ gigs of RAM when I'm trying to backfill, causing OOM. I assume this is unexpected.
My settings (running it in Docker):
HYDRANT_DATABASE_PATH: /data/hydrant.db
HYDRANT_EPHEMERAL: "true"
HYDRANT_EPHEMERAL_TTL: "60min"
HYDRANT_FULL_NETWORK: "true"
TEAM TM8FP6001T
i did some fixing around that might help, can you pull latest and try again? you might also want to decrease BACKFILL_CONCURRENCY_LIMIT from 64 (default in full network) to a lower value? the other option would be to play around with the database config but i dont think thatll help
After testing from latest: it seems to have made a small difference? Thanks! Let me give some numbers:
This still seems higher than I expected, but idk, maybe just inherent to the problem space. I can decrease the limit, but I'm curious what you think the expected memory usage should be. Given the blog post and the machine I'm using. I wouldn't expect to have to decrease it.
Peaks at about 12 gigs used, but doesn't seem to just be growing without bound. That might be where the difference is, for the recent commits you made.
Here's a graph of about an hour of memory usage: https://docs.google.com/spreadsheets/d/1TiNi3yk9whvt0hGh1Pn5CzP-ZXe10CohwEFpSW72SjU/edit?usp=sharing
(these numbers collected without a websocket consumer, which is why memory usage is stable)
One part of the code that seems like it would cause intermittent memory issues is that the entire CAR repo is held in memory, rather than streamed. Idk if that's the culprit here but it seems relevant.
The extreme memory ballooning (the 37G+ I initial reported) is only reproduceable when I run some other tasks on the same machine, including postgres and some custom code consuming the websocket. I thought this might be disk utilization as you hinted at, but that wasn't it.
The other explanation is there is some sort of memory leak involved with consumers of the websocket, or DID resolvers - as that is what one of the containers is doing.
I investigated this and found that:
So I think there is some sort of memory leak with a slow websocket consumer - something to do with the iterator not letting fjall compact? I'm not familiar with this kind of database though.
Even with 8 backfill workers instead of 64, I still see memory usage hit 19 gigs in 16 mins - when there is a websocket consumer.
Claude seems to have given me a fix. I don't want to give you a low quality PR (per AGENTS.md), but I also don't know Rust, so let me just describe what it's done at a high level: instead of holding the websocket catch up iterator open the entire time, it batches 128 items, then does blocking sends, then repeats. This way the iterator is not held open by a slow websocket consumer, and compaction can proceed.
This fixed my memory issue so far. No more ballooning, only 7-8 gigs used even with 64 workers.
Update: it eventually OOM'd again, hitting 35G. So this is maybe an improvement but not everything? It takes longer to get to the first OOM now but still does.
Once things got going long enough, it OOM'd multiple times, indicated by breaks on this graph. Note the Y-axis is not memory usage, I don't have that.
thanks for testing it so much! this is very useful. just to make sure though, so the way you are reproducing this is, running full network backfill, and then streaming the events from the beginning always? does it switch to live tailing? if it switches to live tailing it shouldnt be touching db at all (other than to write the events)
there isnt an easy way to know if it switches to live tailing but if you try just calling it without a cursor it should just be live tailing, im wondering if that changes anything
i pushed a test-13-fix branch including your fix and some others. compaction is definitely the issue here. ill look into it deeper later since i cant right now (not home for a week) so i let codex take the reins, im hoping at least this helps somewhat, if not, then there is something deeper with the memory buildup, i assume the ephemeral gc might be part of that also due to it writing a lot of tombstones and such...
so the way you are reproducing this is, running full network backfill, and then streaming the events from the beginning always?
Uh actually I now realize that I am streaming without a cursor at the beginning (since I thought that would mean non-live events), and only apply a cursor if the websocket drops or the consumer restarts.
if you try just calling it without a cursor it should just be live tailing, im wondering if that changes anything
I guess that's what I've been doing, up until hydrant restarts due to OOM, then I start using a cursor.
Let me test with properly doing backfill from cursor=0...
It's been running for 3 hours now without issue! This might be the fix. Thanks. It's using about 3 gigs of mem right now, although of course it's varies, up to 8 gigs maybe. I'll update if I have any further issues.
For clarity, I am running your new branch, with cursor=0 at the start, although I have used other cursors since then.
I can see from my disk usage graph that compaction is happening. If you look at the image I posted earlier, ignoring the OOMs, you can see it only ever increases. In my graph now by contrast, there are clear decreases in disk usage when compaction happens.
Closing this as it seems to just work now. Will re-open if it reappears after the deadlock is fixed. Thanks!
you can close this, i already pushed the fixes from the branch to main, sorry i forgot to tell that
can you share the specs of the machine this is running on, and filesystem / do you have transparent compression? it might be that it just cant compact l0 fast enough and so it balloons memory usage, though that should have just backpressured, so i suspect there is something else going on there