STreaming ARchives: stricter, verifiable, deterministic, highly compressible alternatives to CAR files for atproto repositories.
atproto
car
oh yeah gzip
+2
+2
star-lite/compression-summary.md
+2
star-lite/compression-summary.md
···
17
17
18
18
**dictionaries**: **not recommended**. zstd has dictionary training, but it performed poorly on our large sample of real atproto repositories: it mainly benefits very-small files, and while 74% of our sample is under 100KiB (in CAR format), those files combined take up less than 3% of the total disk space. Using dictionaries actually **compresses worse** for repos over 1MiB, which represents 84% of the input sample when weighted by disk usage.
19
19
20
+
**gzip**: a fine option for web servers compressing at the transport layer and maximally compatible: compressess to within about 10% of zstd 3.
21
+
20
22
21
23
## Results
22
24