this repo has no description
Add merge section
+38
+38
rfds/001_xks_storage_engine.md
+38
rfds/001_xks_storage_engine.md
···
301
301
```
302
302
303
303
The `key` and `value` are binaries, and the `version` is the integer database version.
304
+
305
+
306
+
## Merge
307
+
308
+
A k-way merge takes multiple sorted input streams (tables and memtables) and produces a single sorted output stream.
309
+
This operation is used for both reads *and* writes.
310
+
311
+
To perform a merge, a priority queue of iterators is used, with one iterator for each memtable and each level of tables.
312
+
The priority queue is keyed by database keys, so the next iterator in the queue contains the next key.
313
+
This queue is implemented as a tree using `:gb_trees`,
314
+
but a more efficient data structure may be future work (loser trees are known to be optimal).
315
+
316
+
### Memtable Iterator
317
+
318
+
The memtable iterator stores a reference to the memtable (an `:ets` table) and the current key.
319
+
Iteration is performed by calling `:ets.next()`.
320
+
321
+
### Level Iterator
322
+
323
+
The level iterator is used to iterate over the keys of an entire level.
324
+
After each subtable is exhausted, the iterator moves to the next subtable using the index from the table.
325
+
After each table is exhausted, the iterator moves to the next table by looking it up in the manifest.
326
+
327
+
Note that all manifest operations must take place at a particular manifest epoch for consistency.
328
+
This is handled by the reader process.
329
+
330
+
The level iterator consists of the following:
331
+
332
+
```
333
+
# Level Iterator
334
+
{level, table, subtable, key_pos}
335
+
```
336
+
337
+
The `level` is the (integer) level of the LSM being iterated.
338
+
339
+
The `table` and `subtable` hold decoded metadata and references needed to decode the the table/subtable being iterated.
340
+
341
+
The `key_pos` is the position of the key within the subtable.