Fast and robust atproto CAR file processing in rust
update benches
we're back (at least with mimalloc)
+13
-13
+2
-2
benches/huge-car.rs
+2
-2
benches/huge-car.rs
···
1
1
extern crate repo_stream;
2
-
use repo_stream::Driver;
2
+
use repo_stream::{Driver, Step};
3
3
use std::path::{Path, PathBuf};
4
4
5
5
use criterion::{Criterion, criterion_group, criterion_main};
···
38
38
};
39
39
40
40
let mut n = 0;
41
-
while let Some(pairs) = driver.next_chunk(256).await.unwrap() {
41
+
while let Step::Value(pairs) = driver.next_chunk(256).await.unwrap() {
42
42
n += pairs.len();
43
43
}
44
44
n
+11
-11
readme.md
+11
-11
readme.md
···
79
79
80
80
current car processing times (records processed into their length usize, phil's dev machine):
81
81
82
-
- 450MiB CAR file (huge): `1.3s`
82
+
- 450MiB CAR file (huge): `1.4s`
83
83
- 128MiB (huge): `350ms`
84
-
- 5.0MiB: `6.8ms`
85
-
- 279KiB: `160us`
86
-
- 3.4KiB: `5.1us`
87
-
- empty: `690ns`
84
+
- 5.0MiB: `7.0ms`
85
+
- 279KiB: `170us`
86
+
- 3.4KiB: `5.3us`
87
+
- empty: `720ns`
88
88
89
89
it's a little faster with `mimalloc`
90
90
···
94
94
static GLOBAL: MiMalloc = MiMalloc;
95
95
```
96
96
97
-
- 450MiB CAR file: `1.2s` (-8%)
98
-
- 128MiB: `300ms` (-14%)
99
-
- 5.0MiB: `6.0ms` (-11%)
100
-
- 279KiB: `150us` (-7%)
101
-
- 3.4KiB: `4.7us` (-8%)
102
-
- empty: `670ns` (-4%)
97
+
- 450MiB CAR file: `1.1s` (-15%)
98
+
- 128MiB: `300ms` (-15%)
99
+
- 5.0MiB: `5.5ms` (-21%)
100
+
- 279KiB: `140us` (-17%)
101
+
- 3.4KiB: `4.3us` (-18%)
102
+
- empty: `610ns` (-16%)
103
103
104
104
processing CARs requires buffering blocks, so it can consume a lot of memory. repo-stream's in-memory driver has minimal memory overhead, but there are two ways to make it work with less mem (you can do either or both!)
105
105