Keep using Photos.app like you always do. Attic quietly backs up your originals and edits to an S3 bucket you control. One-way, append-only.
Add architecture and metadata reference docs
architecture.md covers the system overview, Photos.sqlite reader,
backup pipeline, ladder protocol, verification, and design boundaries.
metadata.md documents every field in the per-asset JSON uploaded to S3.
+230
+122
docs/architecture.md
+122
docs/architecture.md
···
1
+
# Architecture
2
+
3
+
Attic reads the macOS Photos library, exports original files via a companion Swift tool, and uploads them to S3 with rich metadata. A local manifest tracks progress so runs are incremental.
4
+
5
+
## System overview
6
+
7
+
```
8
+
Photos.sqlite ──→ reader.ts ──→ PhotoAsset[] ──→ backup pipeline
9
+
(read-only) │
10
+
├─→ ladder (Swift subprocess)
11
+
│ exports originals to staging/
12
+
│
13
+
├─→ S3 upload (original + metadata JSON)
14
+
│
15
+
└─→ manifest.json (local progress tracker)
16
+
```
17
+
18
+
Attic never modifies Photos.sqlite. The database is opened read-only.
19
+
20
+
## Reading the Photos library
21
+
22
+
`reader.ts` queries Photos.sqlite in two stages:
23
+
24
+
**Main query** — a single SELECT joining `ZASSET` and `ZADDITIONALASSETATTRIBUTES` returns core fields: UUID, filename, date, dimensions, GPS, file size, UTI, favorite status, cloud state. Trashed assets are excluded.
25
+
26
+
**Enrichment queries** — seven independent queries each build a `Map` keyed by asset primary key (Z_PK). During row mapping, each asset is enriched from these maps with a default of `null` or `[]` if no match exists.
27
+
28
+
| Query | Source tables | Returns |
29
+
|-------|--------------|---------|
30
+
| Descriptions | `ZASSETDESCRIPTION` → `ZADDITIONALASSETATTRIBUTES` | `Map<number, string>` |
31
+
| Albums | `ZGENERICALBUM` → `Z_33ASSETS` | `Map<number, AlbumRef[]>` |
32
+
| Keywords | `ZKEYWORD` → `Z_1KEYWORDS` → `ZADDITIONALASSETATTRIBUTES` | `Map<number, string[]>` |
33
+
| People | `ZPERSON` → `ZDETECTEDFACE` | `Map<number, PersonRef[]>` |
34
+
| Edits | `ZUNMANAGEDADJUSTMENT` → `ZADDITIONALASSETATTRIBUTES` | `Map<number, EditInfo>` |
35
+
| Rendered resources | `ZINTERNALRESOURCE` (resource type 1) | `Set<number>` |
36
+
37
+
All enrichment queries go through `safeQuery()`, which catches "no such table" errors silently and logs other failures. This makes the reader resilient across macOS versions where table schemas may differ.
38
+
39
+
People are deduplicated per asset — a person appears at most once even if detected in multiple face regions.
40
+
41
+
### Edit detection
42
+
43
+
An asset is considered edited (`hasEdit: true`) only when two conditions are met:
44
+
45
+
1. An entry exists in `ZUNMANAGEDADJUSTMENT` (an edit was performed)
46
+
2. An entry exists in `ZINTERNALRESOURCE` with resource type 1 (a rendered file was produced)
47
+
48
+
This distinguishes visual edits from metadata-only adjustments that don't produce a visible render. When `hasEdit` is false, `editedAt` and `editor` are both null.
49
+
50
+
## The backup pipeline
51
+
52
+
`backup.ts` orchestrates the full flow: filter → batch → export → upload → manifest.
53
+
54
+
### 1. Filter
55
+
56
+
Assets are filtered against the manifest to find pending work. Optional filters narrow by type (`--type photo|video`) or count (`--limit N`). Dry run mode stops here.
57
+
58
+
### 2. Batch and export
59
+
60
+
Pending assets are processed in batches (default 50). Each batch is sent to **ladder**, a companion Swift binary that uses PhotoKit to export original files. Communication is via JSON over stdin/stdout:
61
+
62
+
```
63
+
attic → stdin: { "uuids": ["UUID/L0/001", ...], "stagingDir": "/path" }
64
+
ladder → stdout: { "results": [...], "errors": [...] }
65
+
```
66
+
67
+
Each result includes the file path, size, and SHA-256 hash. PhotoKit identifiers use the `UUID/L0/001` format; attic strips the suffix before further processing. Ladder output is validated at the trust boundary with `assertExportBatchResult()`.
68
+
69
+
### 3. Upload
70
+
71
+
For each exported file, attic:
72
+
73
+
1. Reads the staged file from disk
74
+
2. Uploads the original to `originals/{year}/{month}/{uuid}.{ext}`
75
+
3. Builds and uploads a metadata JSON to `metadata/assets/{uuid}.json` (see `docs/metadata.md`)
76
+
4. Updates the in-memory manifest
77
+
5. Cleans up the staged file
78
+
79
+
S3 keys are built from UUID and extension, both validated with regex (`/^[A-Za-z0-9._-]+$/` and `/^[a-z0-9]+$/`) to prevent path traversal. Extensions are resolved from the asset's UTI via a lookup table, falling back to the filename extension.
80
+
81
+
### 4. Manifest
82
+
83
+
The manifest is a JSON file at `~/.attic/manifest.json` mapping UUID to `{ s3Key, checksum, backedUpAt }`. It's saved periodically during backup (every 50 assets by default) and always at the end. Writes are atomic: write to `.tmp`, then rename.
84
+
85
+
The manifest can be reconstructed from S3 via `verify --rebuild-manifest`, which reads every `metadata/assets/*.json` file and validates UUID format, S3 key pattern, and checksum format before accepting an entry.
86
+
87
+
## Verification
88
+
89
+
`verify.ts` checks backup integrity in two modes:
90
+
91
+
- **Quick** (default) — HEAD each S3 key in the manifest, confirm it exists
92
+
- **Deep** — download each object, compute SHA-256, compare to the manifest checksum
93
+
94
+
Both modes use a bounded concurrency pool (default 50 workers). Errors are capped at 1,000 to prevent unbounded memory growth.
95
+
96
+
## Credentials
97
+
98
+
S3 credentials are stored in the macOS Keychain under service names `attic-s3-access-key` and `attic-s3-secret-key`. They are read at runtime via `security find-generic-password` — never stored in env vars, config files, or code.
99
+
100
+
## Interfaces and testability
101
+
102
+
All external dependencies are behind interfaces:
103
+
104
+
| Interface | Real implementation | Mock |
105
+
|-----------|-------------------|------|
106
+
| `S3Provider` | AWS SDK client for Scaleway | In-memory `Map<string, Uint8Array>` |
107
+
| `Exporter` | Ladder subprocess | Returns pre-configured assets from a `Map` |
108
+
| `ManifestStore` | File-based JSON with atomic writes | Same implementation, pointed at a temp dir |
109
+
| `PhotosDbReader` | SQLite reader for Photos.sqlite | In-memory SQLite with test fixtures |
110
+
111
+
Tests never hit external services, credentials, or the real Photos library.
112
+
113
+
## What attic doesn't do
114
+
115
+
- **Modify Photos.sqlite** — read-only access, always
116
+
- **Download from iCloud** — relies on Photos having local copies of originals
117
+
- **Delete from S3** — the backup is append-only; there is no prune or cleanup command
118
+
- **Back up thumbnails** — only original files and metadata
119
+
- **Back up adjustment plists** — Apple's edit recipes are not portable
120
+
- **Back up rendered edits** — detecting edits is implemented (Phase 1); exporting and uploading rendered versions is planned (Phase 2/3, see `docs/plans/2026-03-13-feat-backup-rendered-edits-plan.md`)
121
+
- **Handle slo-mo or Live Photos specially** — these have unique resource types that need dedicated investigation
122
+
- **Run on non-macOS** — depends on Photos.sqlite, Keychain, and PhotoKit via ladder
+108
docs/metadata.md
+108
docs/metadata.md
···
1
+
# Asset Metadata
2
+
3
+
Each backed-up asset gets a companion JSON file uploaded to S3 at `metadata/assets/{uuid}.json`. This file makes the backup browsable and searchable without access to Apple Photos or the original Photos.sqlite database.
4
+
5
+
## Example
6
+
7
+
```json
8
+
{
9
+
"uuid": "8A3B1C2D-4E5F-6789-ABCD-EF0123456789",
10
+
"originalFilename": "IMG_4231.HEIC",
11
+
"dateCreated": "2024-07-14T16:23:41.000Z",
12
+
"width": 4032,
13
+
"height": 3024,
14
+
"latitude": 52.0907,
15
+
"longitude": 4.3386,
16
+
"fileSize": 3158112,
17
+
"type": "public.heic",
18
+
"favorite": true,
19
+
"title": "Sunset at the beach",
20
+
"description": "A beautiful sunset over the ocean",
21
+
"albums": [
22
+
{ "uuid": "album-uuid-1", "title": "Vacation 2024" },
23
+
{ "uuid": "album-uuid-2", "title": "Favorites" }
24
+
],
25
+
"keywords": ["sunset", "ocean"],
26
+
"people": [
27
+
{ "uuid": "person-uuid-1", "displayName": "Alice" }
28
+
],
29
+
"hasEdit": true,
30
+
"editedAt": "2024-07-14T18:45:00.000Z",
31
+
"editor": "com.apple.photo",
32
+
"s3Key": "originals/2024/07/8A3B1C2D-4E5F-6789-ABCD-EF0123456789.heic",
33
+
"checksum": "sha256:a1b2c3d4e5f6...",
34
+
"backedUpAt": "2026-03-13T10:30:00.000Z"
35
+
}
36
+
```
37
+
38
+
## Fields
39
+
40
+
### Asset identification
41
+
42
+
| Field | Type | Description |
43
+
|-------|------|-------------|
44
+
| `uuid` | string | Photos library UUID, unique per asset |
45
+
| `originalFilename` | string | Filename as imported (e.g. `IMG_4231.HEIC`) |
46
+
47
+
### Date and dimensions
48
+
49
+
| Field | Type | Description |
50
+
|-------|------|-------------|
51
+
| `dateCreated` | string \| null | ISO 8601 timestamp from Photos.sqlite (CoreData epoch converted) |
52
+
| `width` | number | Pixel width |
53
+
| `height` | number | Pixel height |
54
+
55
+
### Location
56
+
57
+
| Field | Type | Description |
58
+
|-------|------|-------------|
59
+
| `latitude` | number \| null | GPS latitude, null if no location data |
60
+
| `longitude` | number \| null | GPS longitude, null if no location data |
61
+
62
+
### File info
63
+
64
+
| Field | Type | Description |
65
+
|-------|------|-------------|
66
+
| `fileSize` | number \| null | Original file size in bytes |
67
+
| `type` | string \| null | Uniform Type Identifier (e.g. `public.heic`, `com.apple.quicktime-movie`) |
68
+
| `favorite` | boolean | Whether the asset is marked as a favorite in Photos |
69
+
70
+
### Enrichment
71
+
72
+
These fields come from auxiliary tables in Photos.sqlite via separate enrichment queries. All degrade gracefully — if the source table is missing (older macOS versions), the field returns its empty default.
73
+
74
+
| Field | Type | Default | Source table |
75
+
|-------|------|---------|--------------|
76
+
| `title` | string \| null | null | `ZADDITIONALASSETATTRIBUTES.ZTITLE` |
77
+
| `description` | string \| null | null | `ZASSETDESCRIPTION.ZLONGDESCRIPTION` |
78
+
| `albums` | AlbumRef[] | [] | `ZGENERICALBUM` via `Z_33ASSETS` join |
79
+
| `keywords` | string[] | [] | `ZKEYWORD` via `Z_1KEYWORDS` join |
80
+
| `people` | PersonRef[] | [] | `ZPERSON` via `ZDETECTEDFACE` join |
81
+
82
+
An `AlbumRef` contains `uuid` and `title`. A `PersonRef` contains `uuid` and `displayName`. People are deduplicated per asset (a person appears at most once even if detected in multiple face regions).
83
+
84
+
### Edit detection
85
+
86
+
| Field | Type | Description |
87
+
|-------|------|-------------|
88
+
| `hasEdit` | boolean | True only when both an adjustment record and a rendered resource exist |
89
+
| `editedAt` | string \| null | ISO 8601 timestamp of the edit, null when `hasEdit` is false |
90
+
| `editor` | string \| null | Bundle ID of the editing app (e.g. `com.apple.photo`, `com.pixelmator.photomator`), null when `hasEdit` is false |
91
+
92
+
`hasEdit` requires two conditions: an entry in `ZUNMANAGEDADJUSTMENT` (the edit happened) AND an entry in `ZINTERNALRESOURCE` with resource type 1 (a rendered file exists). Metadata-only adjustments that don't produce a visible render are excluded.
93
+
94
+
### Backup tracking
95
+
96
+
| Field | Type | Description |
97
+
|-------|------|-------------|
98
+
| `s3Key` | string | S3 object key where the original file is stored (e.g. `originals/2024/07/{uuid}.heic`) |
99
+
| `checksum` | string | SHA-256 hash of the uploaded file, prefixed with `sha256:` |
100
+
| `backedUpAt` | string | ISO 8601 timestamp of when this asset was uploaded |
101
+
102
+
## What's not included
103
+
104
+
- **Adjustment plists** — Apple's non-destructive edit recipes are not portable outside Photos
105
+
- **Thumbnail data** — Not useful for a full backup
106
+
- **iCloud sync state** — Only relevant at backup time, not for the archived copy
107
+
- **Face region coordinates** — Only person identity is stored, not bounding boxes
108
+
- **Slo-mo / Live Photo markers** — Deferred to a future phase