an efficient binary archive format
update spec
+45
-49
+45
-49
SPEC.md
+45
-49
SPEC.md
···
1
-
# Bindle File Format (.bndl)
1
+
# Bindle File Format
2
2
3
-
Bindle is a simple append-only binary archive format. It features a trailing index to support efficient writes and memory-mapped reads.
3
+
Bindle is an append-only binary archive format designed for efficient writes, logical updates via shadowing, and memory-mapped reads.
4
4
5
5
---
6
6
7
7
## 1. High-Level Layout
8
8
9
-
The file contains an 8 byte signature, followed by the data and then the metadata map at the end of the file.
9
+
A Bindle file consists of a fixed header, a data payload area, a trailing index, and a fixed-size footer.
10
10
11
11
| Offset | Component | Description |
12
12
| :--- | :--- | :--- |
13
-
| `0x00` | **Header** | 8-byte magic identification string. |
14
-
| `0x08` | **Data Payload** | Sequential blobs of raw or compressed data. |
15
-
| `Variable` | **Index** | A sequence of metadata entries and filenames. |
16
-
| `EOF - 16` | **Footer** | Pointer to the index and file count. |
13
+
| `0x00` | **Header** | 8-byte magic identification string |
14
+
| `0x08` | **Data Payload** | Sequential blobs of raw or compressed data |
15
+
| `Variable` | **Index** | Sequence of `Entry` headers and filenames |
16
+
| `EOF - 16` | **Footer** | 16-byte tail containing the Index pointer and count |
17
17
18
18
---
19
19
20
-
## 2. Components
20
+
## 2. Component Details
21
21
22
22
### 2.1 Header
23
-
Every Bindle file MUST begin with the following 8 bytes:
24
-
`42 49 4e 44 4c 30 30 31` (ASCII: `BINDL001`)
23
+
Every Bindle file MUST begin with exactly 8 bytes:
24
+
`42 49 4e 44 4c 30 30 31` (ASCII: `BINDL001`).
25
25
26
26
### 2.2 Data Segment
27
-
Data blobs are stored starting at offset `0x08`.
28
-
- Each blob SHOULD be aligned to an **8-byte boundary** to ensure optimal performance when memory-mapping the file.
29
-
- Data can be stored as-is (Raw) or compressed using **Zstandard (zstd)**.
27
+
Data blobs begin at offset `0x08`.
28
+
- **Alignment:** Every data blob MUST be padded with null bytes to an **8-byte boundary**.
29
+
- **Compression:** Blobs may be raw or compressed via Zstd.
30
+
- **Shadowing:** New versions of existing files are simply appended to the end of the data segment. The file remains append-only until a vacuum operation is performed.
30
31
31
-
### 2.3 Index Entry (`Entry`)
32
-
The index consists of a series of entries. Each entry is a fixed-size header followed immediately by a variable-length UTF-8 filename.
32
+
### 2.3 Index Entry
33
+
The index is a series of entries. Each entry consists of a fixed metadata block followed by a variable-length filename.
33
34
34
35
| Field | Size | Type | Description |
35
36
| :--- | :--- | :--- | :--- |
36
-
| `offset` | 8 bytes | u64 | Absolute file offset to start of data. |
37
-
| `c_size` | 8 bytes | u64 | Compressed size on disk. |
38
-
| `u_size` | 8 bytes | u64 | Original uncompressed size. |
39
-
| `crc32` | 4 bytes | u32 | Checksum of the stored data. |
40
-
| `name_len` | 2 bytes | u16 | Length of the following filename string. |
41
-
| `comp_type` | 1 byte | u8 | `0` = Raw, `1` = Zstd. |
42
-
| `reserved` | 1 byte | u8 | Alignment padding. |
43
-
| `filename` | variable | utf8 | Filename string
37
+
| `offset` | 8 bytes | u64 | Absolute file offset to the data blob |
38
+
| `c_size` | 8 bytes | u64 | Compressed size on disk |
39
+
| `u_size` | 8 bytes | u64 | Original uncompressed size |
40
+
| `crc32` | 4 bytes | u32 | Checksum of the stored data |
41
+
| `name_len` | 2 bytes | u16 | Length of the filename string |
42
+
| `comp_type` | 1 byte | u8 | `0` = Raw, `1` = Zstandard |
43
+
| `reserved` | 1 byte | u8 | Alignment padding |
44
+
| `filename` | Variable | UTF-8 | The entry name |
44
45
45
-
**Padding:** After the filename string, the file MUST be padded with null bytes until the next 8-byte boundary is reached.
46
+
**Padding:** After the filename, the file MUST be padded with null bytes (`\0`) to the next 8-byte boundary before the next entry begins.
46
47
47
48
### 2.4 Footer
48
-
The last 16 bytes of the file contain the lookup information required to parse the archive.
49
+
The last 16 bytes of the file are used to locate the index. Both fields are stored in little-endian format.
49
50
50
51
| Field | Size | Type | Description |
51
52
| :--- | :--- | :--- | :--- |
52
-
| `index_offset` | 8 bytes | u64 | Absolute offset to the start of the Index. |
53
-
| `entry_count` | 8 bytes | u32 | Total number of entries in the file. |
53
+
| `index_offset` | 8 bytes | u64 | Absolute offset to the start of the index |
54
+
| `entry_count` | 8 bytes | u64 | Total number of unique entries in the index |
54
55
55
56
---
56
57
57
-
## 3. Implementation Guidelines
58
+
## 3. Operational Logic
58
59
59
-
### 3.1 Reading Logic
60
-
To read a Bindle file:
61
-
1. Validate the file size (must be at least 28 bytes).
62
-
2. Read the first 8 bytes and the last 8 bytes to verify the `BINDL001` magic.
63
-
3. Read the `index_offset` from the footer (EOF - 16).
64
-
4. Seek to `index_offset` and iterate `entry_count` times to populate an in-memory map of files.
60
+
### 3.1 Shadowing & Atomic Updates
61
+
To "update" a file or add new ones:
62
+
1. Append new data starting at the current `index_offset`.
63
+
2. Write a new Index. If a filename is repeated, the index points to the **newest** data offset.
64
+
3. Write a new Footer.
65
+
4. Old data remains in the file (unreferenced) until a vacuum occurs.
65
66
66
-
### 3.2 Writing Logic (Atomic Updates)
67
-
To maintain an append-only structure:
68
-
1. Seek to the `index_offset` found in the current footer (effectively overwriting the old index).
69
-
2. Append new data blobs.
70
-
3. Write a new Index containing all previous entries plus the new ones.
71
-
4. Write a new Footer.
72
-
5. Flush/Sync the file to disk.
73
-
74
-
### 3.3 Constraints
75
-
- **Unique Keys:** Duplicate filenames are not permitted.
76
-
- **Null Bytes:** Filenames MUST NOT contain internal null bytes (`\0`).
77
-
- **Maximum Size:** File offsets are 64-bit, supporting archives up to 16 Exabytes.
67
+
### 3.2 Vacuuming
68
+
To reclaim space used by shadowed data:
69
+
1. Create a temporary file and write the `BINDL001` header.
70
+
2. Iterate through the **live** index entries only.
71
+
3. Copy the referenced data blobs to the new file, updating their offsets in a new in-memory index.
72
+
4. Write the new Index and Footer to the temporary file.
73
+
5. Atomically replace the old file with the new one.
78
74
79
75
---
80
76
81
77
## 4. Design Rationale
82
-
- **Trailing Index:** Allows files to be "updated" or added by simply appending to the end of the file and writing a new index.
83
-
- **Alignment:** 8-byte alignment ensures that `u64` fields can be read directly from a memory-mapped pointer without unaligned access penalties on modern CPUs.
84
-
- **Zero-Copy:** Raw entries can be used directly as slices from memory without decompression or copying.
78
+
- **Trailing Index:** Enables "single-pass" appending. You don't need to shift existing data to grow the index.
79
+
- **8-Byte Alignment:** Ensures that all 64-bit integers in the metadata and footer are naturally aligned, preventing performance penalties on architectures that dislike unaligned reads.
80
+
- **Zero-Copy Potential:** Raw (uncompressed) data blobs can be used directly as memory slices via `mmap` without intermediate buffers.