Fast and robust atproto CAR file processing in rust
where it's at
+12
-1
+12
-1
readme.md
+12
-1
readme.md
···
1
1
# repo-stream
2
2
3
-
an AsyncRead for atproto MSTs in CAR files
3
+
a futures atproto record stream from CAR file
4
+
5
+
current notes
6
+
7
+
- just buffering all the blocks is 2.5x faster than interleaving optimistic walking
8
+
- at least, this is true on huge CARs with the current (stream-unfriendly) pds export behaviour
9
+
10
+
- transform function is a little tricky because we can't *know* if a block is a record or a node until we actually walk the tree to it (after they're all buffered in memory anyway).
11
+
- still might as well benchmark a test with optimistic block probing+transform on the way in
12
+
13
+
14
+
original ideas:
4
15
5
16
- tries to walk and emit the MST *while streaming in the CAR*
6
17
- drops intermediate mst blocks after reading to reduce total memory