STreaming ARchives: stricter, verifiable, deterministic, highly compressible alternatives to CAR files for atproto repositories.
atproto
car
verifying the tree
+8
-2
+8
-2
readme.md
+8
-2
readme.md
···
65
65
66
66
## problems
67
67
68
-
It might be difficult to convert a STAR to stream-friendly (preorder traversal) CAR format, since the CID of each MST node block can only be computed after visiting all of its children.
68
+
It might be difficult to convert a STAR to stream-friendly (preorder traversal) CAR format, since the CID of each MST node block can only be computed after visiting all of its children. STAR (could)[https://bsky.app/profile/bad-example.com/post/3mcv4zxwtgs2w] require that MST nodes above a certain depth store their own CIDs which would be sad but pragmatic.
69
+
70
+
Similarly, the validity of the commit signature cannot be known until the root node's CID is calculated. a parser might emit an entire repo-worth of keys-record pairs only to find out at the very end that none of it was valid. The same boring pragmatic fix as the last problem can probably also address this: small near-leaf subtrees need to be buffered for validation; as trees get larger this looks more like streaming. (see `verifying the whole tree` below)
69
71
70
-
STAR (could)[https://bsky.app/profile/bad-example.com/post/3mcv4zxwtgs2w] require that MST nodes above a certain depth store their own CIDs which would be sad but pragmatic.
71
72
72
73
## format
73
74
···
135
136
TODO
136
137
- (but basically do what the repo-spec does but apply it across the whole stream)
137
138
- (but also actually run some tests and measure how much this decreases file sizes post-normal-file-compression)
139
+
140
+
141
+
### verifying the whole tree
142
+
143
+
A STAR reader must compute CIDs for all MST nodes as they are encountered, so that parent node CIDs can be computed, until eventually the root node's CID is known and can be compared againt the commit object's `data` hash link. If the root node's CID does not match, the commit's signature is not valid for the archive.