microcosm.blue / star
STreaming ARchives: stricter, verifiable, deterministic, highly compressible alternatives to CAR files for atproto repositories.
atproto car
9
fork

Configure Feed

Select the types of activity you want to include in your feed.

star / star-lite / compression-summary.md
at main 2.1 kB view raw view code
phil put the compression stuff out of the readme
bdb3843a

STAR-lite compression results#

Ratios are STAR/CAR (lower is better). "raw" baseline = uncompressed CAR; "coder" baseline = CAR compressed with the same coder.

overall#

N=4866, raw CAR=2.18 GiB, raw STAR=1.62 GiB.

setting mean (raw) med (raw) wt (raw) mean (coder) med (coder) wt (coder)
raw 0.668 0.678 0.746 — — —
gzip 0.292 0.232 0.215 0.568 0.556 0.552
zstd --fast 1 0.333 0.295 0.286 0.632 0.635 0.671
zstd 3 0.281 0.224 0.195 0.566 0.551 0.553
zstd 9 0.276 0.218 0.183 0.562 0.544 0.542

< 10 KiB#

N=2168, raw CAR=6.31 MiB, raw STAR=4.04 MiB.

setting mean (raw) med (raw) wt (raw) mean (coder) med (coder) wt (coder)
raw 0.614 0.611 0.640 — — —
gzip 0.395 0.370 0.301 0.624 0.640 0.585
zstd --fast 1 0.413 0.402 0.342 0.653 0.663 0.628
zstd 3 0.383 0.363 0.295 0.621 0.630 0.578
zstd 9 0.381 0.361 0.292 0.622 0.634 0.578

10 KiB – 1 MiB#

N=2346, raw CAR=379.92 MiB, raw STAR=276.84 MiB.

setting mean (raw) med (raw) wt (raw) mean (coder) med (coder) wt (coder)
raw 0.706 0.710 0.729 — — —
gzip 0.209 0.210 0.208 0.517 0.520 0.526
zstd --fast 1 0.266 0.270 0.270 0.607 0.615 0.629
zstd 3 0.199 0.198 0.193 0.519 0.521 0.529
zstd 9 0.193 0.191 0.184 0.511 0.513 0.519

1 MiB – 100 MiB#

N=352, raw CAR=1.80 GiB, raw STAR=1.35 GiB.

setting mean (raw) med (raw) wt (raw) mean (coder) med (coder) wt (coder)
raw 0.749 0.746 0.750 — — —
gzip 0.216 0.216 0.216 0.556 0.557 0.557
zstd --fast 1 0.288 0.290 0.289 0.676 0.675 0.680
zstd 3 0.192 0.194 0.195 0.547 0.543 0.558
zstd 9 0.180 0.182 0.183 0.534 0.532 0.547