this repo has no description
Rewrite table index section
+33
-30
+33
-30
rfds/001_xks_storage_engine.md
+33
-30
rfds/001_xks_storage_engine.md
···
41
41
All blocks have an address consisting of the `{index, checksum}` of the block.
42
42
43
43
The `index` is the index of the block within the file.
44
-
The byte-offset of the block within the file is therefore `index * block_size`.
44
+
The byte offset of the block within the file is therefore `index * block_size`.
45
45
46
46
The `checksum` is a crypto-safe checksum of the block's contents.
47
47
This is currently a 128-bit truncation of the SHA256 hash of the contents,
···
182
182
183
183
The trailer consists of:
184
184
185
-
- A list of block addresses (`index` and `checksum`) for the blocks in the table
185
+
- An array of block addresses (`index` and `checksum`) for the blocks in the table
186
186
- The fixed-length metadata needed to decode the table
187
187
188
188
#### Metadata
···
213
213
index (8 bytes) | checksum (16 bytes)
214
214
```
215
215
216
-
By storing the list of block addresses entirely within the last block,
216
+
By storing the array of block addresses entirely within the last block,
217
217
we avoid having to build a linked list of blocks for the trailer.
218
218
We can fit roughly 20,000 24-byte addresses within a 512 KiB block,
219
219
so this approach limits us to a table size of around 10 GiB: *far* larger than we will ever need.
220
220
221
+
### Table index
222
+
223
+
The table index is an index used to look up subs within a table.
224
+
Each sub has an entry in the index consisting of its key range and its byte offset within the table.
225
+
226
+
The index entries are written contiguously in the following format:
227
+
228
+
```
229
+
# Index entry
230
+
start_key (variable) | start_key_version (8 bytes) | sub_offset (4 bytes) | sub_size (4 bytes)
231
+
```
232
+
233
+
The `start_key` and `start_key_version` are the key/version of the first pair in the sub.
234
+
235
+
The `sub_offset` and `sub_size` are the byte offset and byte size of the sub within the table.
236
+
237
+
Because keys are variable in length,
238
+
the index entries are followed by an array of fixed-length slots (one per entry):
239
+
240
+
```
241
+
# Entry slot
242
+
offset (4 bytes) | key_size (4 bytes)
243
+
```
244
+
245
+
The `offset` is the byte offset of the entry within the table.
246
+
The `key_size` is the length of the key.
247
+
248
+
After decoding the table metadata we obtain the location and size of this slot array.
249
+
We can then binary search the index by looking up the keys for slots in the array.
250
+
221
251
### Subtables
222
252
223
253
Each sub consists of key/value pairs in sorted order followed by their indexes and then a trailer indicating the number of pairs.
···
262
292
4. Binary search the pairs.
263
293
Using the offsets, we can look up any pair's key at the byte range `[index, index + key_length + 8)`.
264
294
Note that we *include* the version (8 bytes) in the key when binary searching because this is a multiversion LSM.
265
-
266
-
### Table Index
267
-
268
-
The table index is an index of all subs within the table and their corresponding key ranges.
269
-
The purpose of this is to avoid loading the entire (large) table into memory to perform a binary search.
270
-
We can binary search the index first to find the right sub and then search the sub for a pair.
271
-
272
-
The index consists of sub keys/addresses stored contiguously and then their offsets stored contiguously.
273
-
274
-
```
275
-
# Sub entry:
276
-
key (variable) | key_version (8 bytes) | sub_offset (8 bytes) | sub_size (4 bytes)
277
-
278
-
# Entry slot:
279
-
offset (4 bytes)
280
-
```
281
-
282
-
The `key` and `key_version` are the key/version of the first key in the sub.
283
-
284
-
The `sub_offset` is the byte offset of the sub *within the table*,
285
-
and the `sub_size` is the byte length of the sub.
286
-
287
-
The `offset` stores the byte offset of each entry within the index.
288
-
289
-
The length of the `key` is stored implicitly as the difference between slot `offset`s.
290
-
291
-
The sub entries can be binary searched with an algorithm similar to that of the subs detailed above.