Caveat: I'm still working on this so I might be getting some point here wrong, but I want to share for visibility.

Currently the relationship between blobs, records, and users, is defined in a way that prevents us from correctly implementing blob garbage collection, and means that the PDS leaks unlinked blobs on getBlobs etc.

For postgres: a blob is identified by cid and user. This means we can correctly represent that two different users have the same blob, which until recently would risk data loss. However, we're not tracking the relationship between records and blobs.

For metastore: a blob is identified by cid, but also has a metadata record that links to user. Blobs themselves are reference counted, but by user, not record.

Quick note on record_blobs. This table looks like it would be used for blob<>record relationships, but it's actually only used for imports.

Blob docs https://atproto.com/specs/blob

Problem #

On listBlobs tranquil returns all blobs associated with a user, even ones that are not referenced by a record. This does not match the spec, which expects it to return referenced blobs only. Additionally, because we don't know if a blob is referenced, we can't know whether we can garbage collect.

Migrating this is complicated by the fact that we also have a blob repair job that I added to fix a subset of this design issue.

Proposal #

We drop record_blobs and replace it with a new record_blobs that contains cid and record id. The existing migration paths can be simplified to use the same blob upload code as regular operations, this should delete some code. We can clean up the blobs table in a plain pg/metastore migration, and then we need a proper backfill job that blocks startup and runs before other jobs that lists all records, finds all blobs, and builds the record_blobs table. Like the blob repair job it stores a marker to remember that it has run.

listBlobs and other operations will use recordBlobs to determine the blobs to list. getBlob on an unreferenced blob returns nothing. We support uploading blobs and later referencing, or creating the record first and uploading blob later. The spec says that blobs must be uploaded first, but then makes a concession for imports, so it's safer to just allow both. It also matches the reference PDS.

With that migration done implementing garbage collection is possible.

Next steps #

I'm working on a few PRs to do this

Make metastore migrations be a list, so you can have more than one.
Main migration, all the required code changes.
Implement GC and maybe some deferred cleanup.