Every OAuth mint or refresh rewrites the whole pds.grants file and fsyncs it (crates/didbot-pds/src/durable.rs:3461-3463). Spent refresh-token digests stay in it until the login ends, 14 days out (crates/didbot-pds/src/oauth.rs:427), at about 196 bytes each. So each refresh costs time proportional to every refresh before it.
Measured on loopback at 73ddde8 (2 pinned CPUs; about double for a t3.small):
| pds.grants size | refresh p50 | CPU per refresh | throughput |
|---|---|---|---|
| 0.3 MB | 45 ms | 8 ms | 68/s |
| 1.3 MB | 79 ms | 18 ms | |
| 3.2 MB | 118 ms | 26 ms | 26/s |
15,000 refreshes wrote 25 GB to the device. At 10x, with about 15k refreshes in 14 days, the file is about 3 MB and each refresh is about 50 ms of t3 CPU plus 3 MB of fsync'd writes. In the worst case of 2,700 sessions refreshing every 5 minutes for 14 days the file would reach about 2 GB. Today it is a few hundred KB (2–8 ms).
Options: append grant changes to the WAL, or as their own log, and compact them at checkpoint; or keep only the last N spent digests per token family, enough to detect reuse.
Fixed in #1237. Each change appends one synced line (the existing
KeptJSON) topds.grants. The file is written whole once the lines outgrow it, at boot, and at a clean stop. Spent digests still stay until the login ends, so the whole file and memory still grow with refreshes over 14 days. Only the cost of each refresh changed.