tranquil-pds
Our Personal Data Server from scratch! tranquil.farm
pds rust database fun oauth atproto
Star 547
Fork 39
Watch

Align blob management with spec #671

Merged jola.dev
targeting main from jola.dev/tranquil-pds:johanna/record-blobs
Merged opened by and assigned to jola.dev
targeting main from jola.dev/tranquil-pds:johanna/record-blobs

History #

Tranquil blobs used to just be completely separate from records, meaning a record could be created with a reference to a blob that didn't exist (outside of imports). Without references, an uploaded blob could never be deleted. Oh, and a blob that had been taken down could still be referenced by new records. ListBlobs returned all blobs that had ever been uploaded by you, minus the ones that were identical to the ones someone else had, rather than the blobs that were actively referenced. Which also meant that if you uploaded a blob someone else had uploaded first, it would skip it, and then when that person deleted their blob, yours was gone too. Oh, and you could fetch any blob for any user, whether they had uploaded it or not. Probably missing other stuff too. Just giving some context on what motivates this massive PR.

As part of this story, I can't skip over the fact that I attempted to fix some of the things above, but without understanding what the spec and ecosystem actually expects here. So we had a previous blob related migration that I'm also cleaning up in this PR.

In reality, a blob only exists through records that reference it. A blob should be uploaded before creating the record that references it, but it can't be fetched until it's referenced. It should also not show up in listBlobs. The data model should let you determine whether a blob is referenced or not, which then also enables garbage collection.

What this does #

Most of the code is in one big commit, since it's all kinda connected, but once I got to iterating on it I split the rest out into commits.

Data model #

This changes the data model for blobs in Tranquil. Blobs are back to being keyed by cid only. The old record_blobs table is dropped and then resurrected in a clean state, now to own the relationship between blobs and records. Record blobs track cid+record ID, and the latter is already related to a user, so with this PR we can now track all these nice relationships.

To fix the data model this PR includes postgres and metastore migrations, and a blocking backfill job that runs on startup that scans the whole repo and recreates all relationships. Looking at the rough sizes of Tranquil instances in the wild, this should take <10s on startup, and means post migration we can trust the data to be correct. Note that corrupted records are skipped and logged and might need to be addressed separately, but that should be unlikely.

Behavior changes #

  • listBlobs will only return "live" blobs for the user.
  • getBlob will only return the blob if it's live for the user (no longer based on it just existing).
  • Creating a record for a blob that has been taken down now returns 400.
  • listMissingBlobs etc should now show accurate counts of live blobs.

Code changes #

The import path still allows creating records before blobs exist, just like the reference PDS, but the code has been simplified to use the shared code paths (insert_record_blobs deleted).. Also cleaning up the "blob ownership" stuff meant lots of deletions, blob_ops.rs shrinks a lot. I've replaced the per user operations in BlobRepository and replaced with record referencing ones.

Otherwise the rest is hopefully straightforward, it's all just tracking record references everywhere and validating blobs are referenced before returning them, that kind of thing.

Addresses #151

Next up #

Next PR will be blob garbage collection. Blarbage collection if you will.

Commits 0