Pure OCaml B-tree implementation for persistent storage
ocaml-linkedin: apply dune fmt
Pure formatting changes from `dune fmt`: doc comment placement moves
from above the binding to below it for `type`s, multi-line `match`
expressions collapse onto one line where they fit, and infix operator
applications pick up spaces (`Soup.($?)` -> `Soup.( $? )`). No
semantic changes.
+110
-43
+103
-43
README.md
+103
-43
README.md
···
11
11
Two B-tree variants following the
12
12
[SQLite file format specification](https://www.sqlite.org/fileformat.html):
13
13
14
-
- **Table B-tree** (B+tree) — data lives only in leaves, interior nodes
14
+
- **Table B-tree** (B+tree) -- data lives only in leaves, interior nodes
15
15
hold rowid keys and child pointers. Optimised for sequential scans.
16
-
- **Index B-tree** — keys stored in both interior and leaf nodes. Optimised
16
+
- **Index B-tree** -- keys stored in both interior and leaf nodes. Optimised
17
17
for point lookups.
18
18
19
19
### Features
···
23
23
- File-backed or purely in-memory operation
24
24
- Configurable page cache
25
25
26
-
### Design choices
26
+
## Installation
27
27
28
-
The SQLite file format is an implementation choice, not a limitation. It
29
-
brings free tooling (`sqlite3` CLI, DB Browser, etc.) and a
30
-
[spec](https://www.sqlite.org/fileformat.html) with 20+ years of
31
-
battle-testing. The user-facing API is generic — persistent ordered map
32
-
(`Table`) and persistent ordered set (`Index`).
28
+
Install with opam:
29
+
30
+
```sh
31
+
$ opam install btree
32
+
```
33
33
34
-
What the format does **not** give you (compared to LMDB/sanakirja):
34
+
If opam cannot find the package, it may not yet be released in the public
35
+
`opam-repository`. Add the overlay repository, then install it:
35
36
36
-
| Feature | SQLite format | LMDB / sanakirja |
37
-
|---------|--------------|------------------|
38
-
| Concurrency | In-place updates + WAL/journal | Copy-on-write (lock-free readers) |
39
-
| Range scans | Via parent traversal | Leaf sibling pointers |
40
-
| Crash safety | Rollback journal or WAL | Atomic root pointer swap |
37
+
```sh
38
+
$ opam repo add samoht https://tangled.org/gazagnaire.org/opam-overlay.git
39
+
$ opam update
40
+
$ opam install btree
41
+
```
41
42
42
-
These are deliberate tradeoffs — compatibility and tooling vs. raw
43
-
throughput on concurrent workloads.
43
+
## Usage
44
44
45
-
## Installation
45
+
### Table: insert, find, iterate
46
46
47
-
```
48
-
opam install btree
47
+
```ocaml
48
+
let demo () =
49
+
let pager = Btree.Pager.mem ~page_size:4096 () in
50
+
let table = Btree.Table.v pager in
51
+
Btree.Table.insert table ~rowid:1L "Hello";
52
+
Btree.Table.insert table ~rowid:2L "World";
53
+
(match Btree.Table.find table 1L with
54
+
| Some v -> Fmt.pr "1 -> %s@." v
55
+
| None -> ());
56
+
Btree.Table.iter table (fun rowid data ->
57
+
Printf.printf "%Ld: %s\n" rowid data)
49
58
```
50
59
51
-
## Usage
60
+
### Index: add, membership, range scan
52
61
53
62
```ocaml
54
-
(* File-backed table B-tree *)
55
-
let pager = Btree.Pager.v ~page_size:4096 file in
56
-
let table = Btree.Table.v pager in
57
-
Btree.Table.insert table ~rowid:1L "Hello";
58
-
Btree.Table.insert table ~rowid:2L "World";
59
-
Btree.Table.find table 1L (* Some "Hello" *)
63
+
let index_demo () =
64
+
let pager = Btree.Pager.mem ~page_size:4096 () in
65
+
let index = Btree.Index.v pager in
66
+
Btree.Index.insert index "key";
67
+
Btree.Index.mem index "key"
68
+
```
60
69
61
-
(* In-memory index B-tree *)
62
-
let pager = Btree.Pager.mem ~page_size:4096 () in
63
-
let index = Btree.Index.v pager in
64
-
Btree.Index.insert index "key";
65
-
Btree.Index.mem index "key" (* true *)
70
+
### File-backed
71
+
72
+
Point the pager at an Eio file and call `Pager.sync` to persist. The result
73
+
is a file that `sqlite3` can open:
66
74
67
-
(* Iterate in key order *)
68
-
Btree.Table.iter table (fun rowid data ->
69
-
Printf.printf "%Ld: %s\n" rowid data)
75
+
```ocaml
76
+
let persist ~fs =
77
+
Eio.Path.with_open_out
78
+
~create:(`If_missing 0o644) Eio.Path.(fs / "demo.db")
79
+
(fun file ->
80
+
let pager = Btree.Pager.v ~page_size:4096 file in
81
+
let table = Btree.Table.v pager in
82
+
Btree.Table.insert table ~rowid:1L "persisted";
83
+
Btree.Pager.sync pager)
70
84
```
71
85
72
-
## Modules
86
+
Reopen by constructing a pager on the same file and calling
87
+
`Btree.Table.open_ pager ~root_page`; the root page number comes from
88
+
`Btree.Table.save_root` (or `1` for a fresh file).
89
+
90
+
## API
91
+
92
+
### Table
93
+
94
+
- `v pager` / `open_ pager ~root_page`
95
+
- `insert t ~rowid data` / `find t rowid` / `delete t rowid`
96
+
- `iter t f` / `fold t ~init ~f`
97
+
- `save_root t` / `restore_root t root` -- persist the root for reopen
98
+
99
+
### Index
100
+
101
+
- `v pager` / `open_ pager ~root_page`
102
+
- `insert t key` / `mem t key`
103
+
- Iteration and range scans mirror `Table`
104
+
105
+
### Pager
106
+
107
+
- `Pager.v ~page_size file` -- file-backed
108
+
- `Pager.mem ~page_size ()` -- in-memory, `sync` is a no-op
109
+
- `Pager.sync t` -- flush dirty pages
110
+
- `Pager.snapshot t` / `rollback t snap` -- capture and restore for
111
+
aborting a logical transaction
112
+
113
+
## Internals
114
+
115
+
### Modules
73
116
74
117
| Module | Purpose |
75
118
|--------|---------|
···
80
123
| `Cell` | Cell encoding/decoding (table leaf, interior, index) |
81
124
| `Record` | SQLite record format (serial types, column values) |
82
125
| `Varint` | Variable-length integer encoding |
83
-
84
-
## File format
85
126
86
127
### Page header (8 bytes leaf, 12 bytes interior)
87
128
···
107
148
local = if K <= X then K else M
108
149
```
109
150
151
+
### Design choices
152
+
153
+
The SQLite file format is an implementation choice, not a limitation. It
154
+
brings free tooling (`sqlite3` CLI, DB Browser, etc.) and a
155
+
[spec](https://www.sqlite.org/fileformat.html) with 20+ years of
156
+
battle-testing. The user-facing API is generic -- persistent ordered map
157
+
(`Table`) and persistent ordered set (`Index`).
158
+
159
+
What the format does **not** give you (compared to LMDB/sanakirja):
160
+
161
+
| Feature | SQLite format | LMDB / sanakirja |
162
+
|---------|--------------|------------------|
163
+
| Concurrency | In-place updates + WAL/journal | Copy-on-write (lock-free readers) |
164
+
| Range scans | Via parent traversal | Leaf sibling pointers |
165
+
| Crash safety | Rollback journal or WAL | Atomic root pointer swap |
166
+
167
+
These are deliberate tradeoffs -- compatibility and tooling vs. raw
168
+
throughput on concurrent workloads.
169
+
110
170
## Related work
111
171
112
-
- [SQLite file format](https://www.sqlite.org/fileformat.html) — the specification this library implements
113
-
- [ocaml-sqlite](../ocaml-sqlite/) — database layer built on top of this library (KV API, named tables, schema)
114
-
- [LMDB](http://www.lmdb.tech/doc/) — C B+tree with memory-mapped COW (different tradeoffs)
115
-
- [sanakirja](https://docs.rs/sanakirja/) — Rust COW B-tree, used by Pijul
116
-
- [bbolt](https://github.com/etcd-io/bbolt) — Go B+tree, used by etcd
117
-
- [Limbo](https://github.com/tursodatabase/limbo) — Rust SQLite reimplementation
172
+
- [SQLite file format](https://www.sqlite.org/fileformat.html) -- the specification this library implements
173
+
- [ocaml-sqlite](../ocaml-sqlite/) -- database layer built on top of this library (KV API, named tables, schema)
174
+
- [LMDB](http://www.lmdb.tech/doc/) -- C B+tree with memory-mapped COW (different tradeoffs)
175
+
- [sanakirja](https://docs.rs/sanakirja/) -- Rust COW B-tree, used by Pijul
176
+
- [bbolt](https://github.com/etcd-io/bbolt) -- Go B+tree, used by etcd
177
+
- [Limbo](https://github.com/tursodatabase/limbo) -- Rust SQLite reimplementation
118
178
119
179
## Licence
120
180
+1
btree.opam
+1
btree.opam
+4
dune
+4
dune