this repo has no description
oops I forgot to commit half the docs
+61
+19
docs/cartool.rst
+19
docs/cartool.rst
···
1
+
.. _cartool:
2
+
3
+
cartool
4
+
=======
5
+
6
+
``cartool`` aims to be a CLI swiss-army-knife for analysing and modifying atproto repos stored inside CAR files.
7
+
8
+
.. code-block:: text
9
+
10
+
USAGE: cartool COMMAND [args...]
11
+
12
+
Available commands:
13
+
info <car_path> : print CAR header and repo info
14
+
list <car_path> : list all records in the CAR (values as CIDs)
15
+
dump <car_path> : dump all records in the CAR (values as JSON)
16
+
dump_record <car_path> <key> : dump a single record, keyed on ('collection/rkey')
17
+
compact <car_in> <car_out> : rewrite the whole CAR, dropping any duplicated or unreferenced blocks
18
+
diff <car_a> <car_b> : list the record diff between two CAR files
19
+
+42
docs/overview.rst
+42
docs/overview.rst
···
1
+
.. _overview:
2
+
3
+
Library Overview
4
+
================
5
+
6
+
If you have some `atproto repository <https://atproto.com/specs/repository>`_ data, and you want to operate on it with Python, you've come to the right place [1]_. The APIs offered here are rather low-level, but I'm planning on adding higher-level helper utilities in the future.
7
+
8
+
.. [1] Maybe also check out `arroba <https://github.com/snarfed/arroba>`_!
9
+
10
+
=============
11
+
Block Storage
12
+
=============
13
+
14
+
The foundations of repos are content-addressed Blocks of data, as in the `IPLD <https://ipld.io/docs/motivation/benefits-of-content-addressing/>`_ data model. The abstract :meth:`~atmst.blockstore.BlockStore` interface facilitates access to blocks, agnostic of the underlying storage medium. The following implementations are available:
15
+
16
+
* :meth:`~atmst.blockstore.MemoryBlockStore` - stores blocks in memory only (inside a dict)
17
+
18
+
* :meth:`~atmst.blockstore.car_file.ReadOnlyCARBlockStore` - accesses the contents of a CAR file.
19
+
20
+
* :meth:`~atmst.blockstore.SqliteBlockStore` - accesses blocks stored in a table of an sqlite database.
21
+
22
+
Finally, the :meth:`~atmst.blockstore.OverlayBlockStore` class allows you to layer one BlockStore over another, with writes going to the top layer only. This is useful in several scenarios, for example, reading blocks from two CAR files at once so that you can diff them, or for staging modifications in memory ready to be committed to persistent storage.
23
+
24
+
===================
25
+
Merkle Search Trees
26
+
===================
27
+
28
+
With a BlockStore, we can read and write content-addressed blocks of data. Content-addressing is cool, but sometimes you want mutability. The `Merkle Search Tree <https://inria.hal.science/hal-02303490/document>`_ data structure builds on top of content-addressed Block storage, providing a mutable map of keys onto values. In atproto, the keys are arbitrary strings (under certain constraints), and the values are "records".
29
+
30
+
Everything is still immutable under the hood, so modifying an MST results in a new root hash.
31
+
32
+
:py:mod:`atmst` doesn't have a dedicated class to represent an MST (yet?), instead we just reference the root node by CID.
33
+
34
+
=====
35
+
Nodes
36
+
=====
37
+
38
+
An MST is comprised of one or more Nodes. :py:mod:`atmst` represents Nodes using :meth:`~atmst.mst.node.MSTNode`, an immutable dataclass.
39
+
40
+
Nodes are ultimately stored in a BlockStore, serialised as `DAG-CBOR <https://ipld.io/docs/codecs/known/dag-cbor/>`_, and the :meth:`~atmst.mst.node_store.NodeStore` class facilitates this. A NodeStore also maintains an LRU cache, mapping CIDs to MSTNode objects, to reduce the impact of BlockStore read latency, hash verification, and deserialisation overheads.
41
+
42
+
The :meth:`~atmst.mst.node_wrangler.NodeWrangler` class facilitates modifications to MSTs, and the :meth:`~atmst.mst.node_walker.NodeWalker` class facilitates access to MSTs, which the :meth:`~atmst.mst.diff.mst_diff` method makes use of.