this repo has no description
Problems
+37
-1
+37
-1
rfds/0005_generational_read_version.md
+37
-1
rfds/0005_generational_read_version.md
···
1
1
# Generational Read Version
2
2
3
3
In FoundationDB, transactions are not allowed to span generations.
4
-
During recovery, the current database version is advanced by 100 million (100 seconds).
4
+
During recovery, the database fast-forwards by 100 million versions (100 seconds).
5
5
Because the mvcc window is only 5 million (5 seconds),
6
6
this effectively kills any transactions that were in-progress during the recovery;
7
7
they will fail with `transaction too old`.
8
+
9
+
## Why fast-forward?
8
10
9
11
There are a couple of reasons for this design.
10
12
···
41
43
a new transaction with a read version from the new generation's Sequencer could attempt to read uncommitted mutations from the *last* generation.
42
44
That is, if we did not fast-forward, the generations' version ranges could *overlap* and there would be confusion about which mutations are being read.
43
45
The fast-forward fixes this because it effectively skips over those versions in the "history" of the database.
46
+
47
+
### Why 100M?
48
+
49
+
As a side note, it's not clear to me why FDB fast-forwards by 100M versions,
50
+
as it appears 5M versions (or whatever the MVCC window is set to) would be sufficient.
51
+
It's tempting to chalk this up to superstition (100M "just to be safe"),
52
+
but this would be very uncharacteristic of FDB and its heavy focus on correctness testing and validation.
53
+
54
+
## Problems with fast-forward
55
+
56
+
The fast-forward approach is simple and elegant.
57
+
If you are operating under the assumption that FDB versions are not supposed to correlate with real time,
58
+
skipping 100M versions here and there is not a big deal.
59
+
And this is the assumption FDB was designed under:
60
+
FDB versions correlate with real time *internally* to provide a real-time-ish MVCC window,
61
+
but this is an implementation detail that should not be relied upon by *users*.
62
+
63
+
You can imagine how that went.
64
+
65
+
### Versionstamp
66
+
67
+
Over time, versions worked their way into FDB's userspace APIs,
68
+
most prominently with the "versionstamp" feature that enables the client to inject the commit version into a key or value.
69
+
Versionstamps are used internally and externally (in layers/apps) to write keys into the database at a particular "time".
70
+
71
+
A good example is the implementation of the "automatic idempotency" feature.
72
+
The authors wanted to easily clean up old idempotency keys,
73
+
so they wrote them with versionstamped ids such that they can simply be range-cleared below a particular timestamp.
74
+
This would be trivial, except that due to recoveries versionstamps *do not correlate with real-time*,
75
+
so the authors had track recoveries and perform adjustments to the watermark to compensate.
76
+
77
+
At some point a Timekeeper role was added to the database to write the current versionstamp into the meta keyspace every 10 seconds.
78
+
Those keys are used as a reverse lookup table (real time -> versionstamp) internally within the database (IIRC for something backup-related).
79
+
This is, frankly, a pretty good hint that something has gone off the rails.