Keep using Photos.app like you always do. Attic quietly backs up your originals and edits to an S3 bucket you control. One-way, append-only.
Fix code review findings and add example config
- Extract consolidated keychain module (load + store)
- Use Cliffy EnumType for --type flag instead of unsafe cast
- Set restrictive file permissions on config (0o700 dir, 0o600 file)
- Add Array.isArray guard in config validation
- Add bucket/region validation to init prompts
- Extract s3ConnectionFromConfig helper to reduce duplication
- Remove dead configDir export and redundant error check
- Add config.example.json and gitignore ~/.attic/
+1008
-305
+3
.gitignore
+3
.gitignore
+6
-3
CLAUDE.md
+6
-3
CLAUDE.md
···
1
1
# Attic
2
2
3
-
Deno/TypeScript CLI for backing up iCloud Photos to S3-compatible storage. Part of the photo-cloud system (companion: [ladder](https://github.com/tijs/ladder)).
3
+
Deno/TypeScript CLI for backing up iCloud Photos to S3-compatible storage. Part
4
+
of the photo-cloud system (companion: [ladder](https://github.com/tijs/ladder)).
4
5
5
6
## Commands
6
7
···
26
27
27
28
## Reference Docs
28
29
29
-
- [Architecture](docs/architecture.md) — pipeline, reader, ladder protocol, manifest, interfaces, design boundaries
30
+
- [Architecture](docs/architecture.md) — pipeline, reader, ladder protocol,
31
+
manifest, interfaces, design boundaries
30
32
- [Asset Metadata](docs/metadata.md) — per-asset JSON schema uploaded to S3
31
33
32
34
## Conventions
33
35
34
36
- Files should stay under 500 lines
35
37
- Use `AssetKind.PHOTO` / `AssetKind.VIDEO` constants, not magic numbers
36
-
- S3 keys and UUIDs are validated with regex before interpolation (path traversal prevention)
38
+
- S3 keys and UUIDs are validated with regex before interpolation (path
39
+
traversal prevention)
37
40
- `removeStagedFile()` constrains deletion to the staging directory
+51
-28
README.md
+51
-28
README.md
···
6
6
7
7
Back up your iCloud Photos library to S3-compatible storage.
8
8
9
-
Attic reads the Photos.sqlite database directly, exports originals via a companion Swift tool called [ladder](https://github.com/tijs/ladder), and uploads them to an S3-compatible bucket. A local manifest tracks what has already been backed up so subsequent runs only upload new assets.
9
+
Attic reads the Photos.sqlite database directly, exports originals via a
10
+
companion Swift tool called [ladder](https://github.com/tijs/ladder), and
11
+
uploads them to an S3-compatible bucket. A local manifest tracks what has
12
+
already been backed up so subsequent runs only upload new assets.
10
13
11
-
Works with any S3-compatible provider. EU-friendly options include [Scaleway](https://www.scaleway.com/en/object-storage/), [Hetzner](https://www.hetzner.com/storage/object-storage), and [OVH](https://www.ovhcloud.com/en/public-cloud/object-storage/).
14
+
Works with any S3-compatible provider. EU-friendly options include
15
+
[Scaleway](https://www.scaleway.com/en/object-storage/),
16
+
[Hetzner](https://www.hetzner.com/storage/object-storage), and
17
+
[OVH](https://www.ovhcloud.com/en/public-cloud/object-storage/).
12
18
13
19
## Prerequisites
14
20
15
21
- [Deno](https://deno.land/) (v2+)
16
-
- The [ladder](https://github.com/tijs/ladder) binary. Ladder is a separate Swift tool that uses PhotoKit to export original photo/video files from the Photos library.
22
+
- The [ladder](https://github.com/tijs/ladder) binary. Ladder is a separate
23
+
Swift tool that uses PhotoKit to export original photo/video files from the
24
+
Photos library.
17
25
- An S3-compatible storage bucket and API credentials
18
26
- macOS (Photos.sqlite access and Keychain are macOS-only)
19
27
···
25
33
deno task init
26
34
```
27
35
28
-
This prompts for your S3 endpoint, region, bucket name, and credentials. Config is saved to `~/.attic/config.json` and credentials are stored in the macOS Keychain.
36
+
This prompts for your S3 endpoint, region, bucket name, and credentials. Config
37
+
is saved to `~/.attic/config.json` and credentials are stored in the macOS
38
+
Keychain.
29
39
30
-
Build the ladder binary (see [ladder](https://github.com/tijs/ladder) for details):
40
+
Build the ladder binary (see [ladder](https://github.com/tijs/ladder) for
41
+
details):
31
42
32
43
```bash
33
44
git clone https://github.com/tijs/ladder.git
···
49
60
50
61
### scan
51
62
52
-
Scan the Photos library and print statistics (asset counts, sizes, types, local vs iCloud-only).
63
+
Scan the Photos library and print statistics (asset counts, sizes, types, local
64
+
vs iCloud-only).
53
65
54
66
```bash
55
67
deno task scan
···
57
69
58
70
### status
59
71
60
-
Compare the Photos database against the local backup manifest to show how many assets are backed up vs pending.
72
+
Compare the Photos database against the local backup manifest to show how many
73
+
assets are backed up vs pending.
61
74
62
75
```bash
63
76
deno task status
···
71
84
deno task backup
72
85
```
73
86
74
-
| Flag | Description |
75
-
|---|---|
76
-
| `--dry-run` | Show what would be uploaded without uploading |
77
-
| `--limit N` | Back up at most N assets |
78
-
| `--batch-size N` | Assets per ladder export batch (default: 50) |
79
-
| `--type photo\|video` | Only back up photos or videos |
80
-
| `--bucket NAME` | Override bucket from config |
81
-
| `--ladder PATH` | Path to the ladder binary (or set `LADDER_PATH` env var) |
82
-
| `--db PATH` | Path to Photos.sqlite |
87
+
| Flag | Description |
88
+
| --------------------- | -------------------------------------------------------- |
89
+
| `--dry-run` | Show what would be uploaded without uploading |
90
+
| `--limit N` | Back up at most N assets |
91
+
| `--batch-size N` | Assets per ladder export batch (default: 50) |
92
+
| `--type photo\|video` | Only back up photos or videos |
93
+
| `--bucket NAME` | Override bucket from config |
94
+
| `--ladder PATH` | Path to the ladder binary (or set `LADDER_PATH` env var) |
95
+
| `--db PATH` | Path to Photos.sqlite |
83
96
84
97
### verify
85
98
···
89
102
deno task verify
90
103
```
91
104
92
-
| Flag | Description |
93
-
|---|---|
94
-
| `--deep` | Download each object and re-verify SHA-256 checksum (slow) |
95
-
| `--rebuild-manifest` | Reconstruct the local manifest from S3 metadata files |
96
-
| `--bucket NAME` | Override bucket from config |
105
+
| Flag | Description |
106
+
| -------------------- | ---------------------------------------------------------- |
107
+
| `--deep` | Download each object and re-verify SHA-256 checksum (slow) |
108
+
| `--rebuild-manifest` | Reconstruct the local manifest from S3 metadata files |
109
+
| `--bucket NAME` | Override bucket from config |
97
110
98
111
## Configuration
99
112
···
112
125
}
113
126
```
114
127
115
-
The `keychain` section is optional and defaults to the service names shown above. Credentials are always stored in the macOS Keychain, never in config files or environment variables.
128
+
The `keychain` section is optional and defaults to the service names shown
129
+
above. Credentials are always stored in the macOS Keychain, never in config
130
+
files or environment variables.
116
131
117
-
`scan` and `status` work without config (they only read Photos.sqlite). `backup` and `verify` require config and will tell you to run `attic init` if it's missing.
132
+
`scan` and `status` work without config (they only read Photos.sqlite). `backup`
133
+
and `verify` require config and will tell you to run `attic init` if it's
134
+
missing.
118
135
119
136
## Testing
120
137
···
122
139
deno task test
123
140
```
124
141
125
-
Tests use dependency injection with mock implementations for the S3 client and exporter, so no external services or credentials are needed.
142
+
Tests use dependency injection with mock implementations for the S3 client and
143
+
exporter, so no external services or credentials are needed.
126
144
127
145
## Documentation
128
146
129
-
- [Architecture](docs/architecture.md) -- How attic works: the backup pipeline, Photos.sqlite reader, ladder protocol, manifest lifecycle, and design boundaries
130
-
- [Asset Metadata](docs/metadata.md) -- Schema reference for the per-asset JSON uploaded to S3
147
+
- [Architecture](docs/architecture.md) -- How attic works: the backup pipeline,
148
+
Photos.sqlite reader, ladder protocol, manifest lifecycle, and design
149
+
boundaries
150
+
- [Asset Metadata](docs/metadata.md) -- Schema reference for the per-asset JSON
151
+
uploaded to S3
131
152
132
153
## Future Plans
133
154
134
-
- **Scheduled backups via launchd** -- A LaunchAgent plist to run backups daily on a dedicated Mac
135
-
- **Rendered edit backup** -- Detect and upload edited versions alongside originals (see `docs/plans/`)
155
+
- **Scheduled backups via launchd** -- A LaunchAgent plist to run backups daily
156
+
on a dedicated Mac
157
+
- **Rendered edit backup** -- Detect and upload edited versions alongside
158
+
originals (see `docs/plans/`)
+31
-23
cli/mod.ts
+31
-23
cli/mod.ts
···
1
-
import { Command } from "@cliffy/command";
2
-
import { requireConfig } from "./src/config/config.ts";
1
+
import { Command, EnumType } from "@cliffy/command";
2
+
import { type AtticConfig, requireConfig } from "./src/config/config.ts";
3
+
import type { S3ConnectionConfig } from "./src/storage/s3-client.ts";
4
+
5
+
const assetType = new EnumType(["photo", "video"]);
6
+
7
+
function s3ConnectionFromConfig(config: AtticConfig): S3ConnectionConfig {
8
+
return {
9
+
endpoint: config.endpoint,
10
+
region: config.region,
11
+
pathStyle: config.pathStyle,
12
+
};
13
+
}
3
14
4
15
const main = new Command()
5
16
.name("attic")
···
47
58
.option("--batch-size <n:integer>", "Assets per ladder batch", {
48
59
default: 50,
49
60
})
50
-
.option("--type <type:string>", "Only back up photos or videos")
61
+
.type("asset-type", assetType)
62
+
.option("--type <type:asset-type>", "Only back up photos or videos")
51
63
.option("--bucket <name:string>", "Override bucket from config")
52
64
.option("--ladder <path:string>", "Path to ladder binary")
53
65
.option("--db <path:string>", "Path to Photos.sqlite")
···
55
67
dryRun?: boolean;
56
68
limit?: number;
57
69
batchSize: number;
58
-
type?: string;
70
+
type?: "photo" | "video";
59
71
bucket?: string;
60
72
ladder?: string;
61
73
db?: string;
···
63
75
const { openPhotosDb } = await import("./src/photos-db/reader.ts");
64
76
const { runBackup } = await import("./src/commands/backup.ts");
65
77
const { createManifestStore } = await import("./src/manifest/manifest.ts");
66
-
const { createS3Provider, loadKeychainCredentials } = await import(
67
-
"./src/storage/s3-client.ts"
78
+
const { createS3Provider } = await import("./src/storage/s3-client.ts");
79
+
const { loadKeychainCredentials } = await import(
80
+
"./src/keychain/keychain.ts"
68
81
);
69
82
const { createLadderExporter } = await import("./src/export/exporter.ts");
70
83
···
82
95
const s3 = createS3Provider(
83
96
credentials,
84
97
options.bucket ?? config.bucket,
85
-
{
86
-
endpoint: config.endpoint,
87
-
region: config.region,
88
-
pathStyle: config.pathStyle,
89
-
},
98
+
s3ConnectionFromConfig(config),
90
99
);
91
100
92
101
const ladderPath = options.ladder ??
···
94
103
"ladder";
95
104
const exporter = createLadderExporter(ladderPath);
96
105
97
-
const typeFilter = options.type as "photo" | "video" | undefined;
98
-
99
106
await runBackup(assets, manifest, manifestStore, exporter, s3, {
100
107
batchSize: options.batchSize,
101
108
limit: options.limit ?? 0,
102
-
type: typeFilter ?? null,
109
+
type: options.type ?? null,
103
110
dryRun: options.dryRun ?? false,
104
111
});
105
112
} finally {
···
125
132
const { runVerify } = await import("./src/commands/verify.ts");
126
133
const { rebuildManifest } = await import("./src/commands/rebuild.ts");
127
134
const { createManifestStore } = await import("./src/manifest/manifest.ts");
128
-
const { createS3Provider, loadKeychainCredentials } = await import(
129
-
"./src/storage/s3-client.ts"
135
+
const { createS3Provider } = await import("./src/storage/s3-client.ts");
136
+
const { loadKeychainCredentials } = await import(
137
+
"./src/keychain/keychain.ts"
130
138
);
131
139
132
140
const config = requireConfig();
···
138
146
const s3 = createS3Provider(
139
147
credentials,
140
148
options.bucket ?? config.bucket,
141
-
{
142
-
endpoint: config.endpoint,
143
-
region: config.region,
144
-
pathStyle: config.pathStyle,
145
-
},
149
+
s3ConnectionFromConfig(config),
146
150
);
147
151
const manifestStore = createManifestStore();
148
152
···
190
194
// Config validation error
191
195
if (msg.startsWith("Config:")) {
192
196
console.error(msg);
193
-
console.error('Run "attic init" to reconfigure, or edit ~/.attic/config.json.\n');
197
+
console.error(
198
+
'Run "attic init" to reconfigure, or edit ~/.attic/config.json.\n',
199
+
);
194
200
return;
195
201
}
196
202
···
223
229
}
224
230
225
231
// Photos.sqlite not found
226
-
if (msg.includes("Photos.sqlite") || msg.includes("unable to open database")) {
232
+
if (
233
+
msg.includes("Photos.sqlite") || msg.includes("unable to open database")
234
+
) {
227
235
console.error("Could not open Photos database.");
228
236
console.error(
229
237
"Make sure Photos is set up on this Mac and the database exists.\n",
+14
-26
cli/src/commands/init.ts
+14
-26
cli/src/commands/init.ts
···
5
5
loadConfig,
6
6
writeConfig,
7
7
} from "../config/config.ts";
8
+
import { storeKeychainCredential } from "../keychain/keychain.ts";
9
+
10
+
const BUCKET_PATTERN = /^[a-z0-9][a-z0-9.\-]{1,61}[a-z0-9]$/;
8
11
9
12
const EU_PROVIDER_EXAMPLES = [
10
13
" Scaleway (EU): https://s3.fr-par.scw.cloud",
···
47
50
const region = await Input.prompt({
48
51
message: "Region",
49
52
hint: "e.g. fr-par, eu-central-1, fsn1",
53
+
validate: (v) => {
54
+
if (v.trim() === "") return "Region is required";
55
+
return true;
56
+
},
50
57
});
51
58
52
59
const bucket = await Input.prompt({
53
60
message: "Bucket name",
61
+
validate: (v) => {
62
+
if (v.trim() === "") return "Bucket name is required";
63
+
if (!BUCKET_PATTERN.test(v)) {
64
+
return "Use lowercase letters, numbers, dots, and hyphens (3-63 chars)";
65
+
}
66
+
return true;
67
+
},
54
68
});
55
69
56
70
const pathStyle = await Confirm.prompt({
···
101
115
'\n Setup complete. Run "attic scan" to see your Photos library.\n',
102
116
);
103
117
}
104
-
105
-
async function storeKeychainCredential(
106
-
service: string,
107
-
value: string,
108
-
): Promise<void> {
109
-
const cmd = new Deno.Command("security", {
110
-
args: [
111
-
"add-generic-password",
112
-
"-U",
113
-
"-s",
114
-
service,
115
-
"-a",
116
-
"attic",
117
-
"-w",
118
-
value,
119
-
],
120
-
stderr: "piped",
121
-
});
122
-
const { code, stderr } = await cmd.output();
123
-
if (code !== 0) {
124
-
const err = new TextDecoder().decode(stderr);
125
-
throw new Error(
126
-
`Failed to store credential in Keychain for service "${service}": ${err.trim()}`,
127
-
);
128
-
}
129
-
}
+1
-2
cli/src/config/config.test.ts
+1
-2
cli/src/config/config.test.ts
···
57
57
58
58
Deno.test("validateConfig rejects missing region", () => {
59
59
assertThrows(
60
-
() =>
61
-
validateConfig({ endpoint: "https://s3.example.com", bucket: "bbb" }),
60
+
() => validateConfig({ endpoint: "https://s3.example.com", bucket: "bbb" }),
62
61
Error,
63
62
'"region" is required',
64
63
);
+9
-9
cli/src/config/config.ts
+9
-9
cli/src/config/config.ts
···
45
45
const obj = raw as Record<string, unknown>;
46
46
47
47
if (typeof obj.endpoint !== "string" || obj.endpoint === "") {
48
-
throw new Error('Config: "endpoint" is required (e.g. "https://s3.fr-par.scw.cloud")');
48
+
throw new Error(
49
+
'Config: "endpoint" is required (e.g. "https://s3.fr-par.scw.cloud")',
50
+
);
49
51
}
50
52
if (!obj.endpoint.startsWith("https://")) {
51
53
throw new Error('Config: "endpoint" must start with https://');
···
67
69
68
70
const pathStyle = obj.pathStyle !== undefined ? Boolean(obj.pathStyle) : true;
69
71
70
-
const keychain = typeof obj.keychain === "object" && obj.keychain !== null
72
+
const keychain = typeof obj.keychain === "object" && obj.keychain !== null &&
73
+
!Array.isArray(obj.keychain)
71
74
? obj.keychain as Record<string, unknown>
72
75
: {};
73
76
···
95
98
config: AtticConfig,
96
99
dir: string = CONFIG_DIR,
97
100
): void {
98
-
Deno.mkdirSync(dir, { recursive: true });
101
+
Deno.mkdirSync(dir, { recursive: true, mode: 0o700 });
99
102
const path = join(dir, "config.json");
100
103
Deno.writeTextFileSync(
101
104
path,
102
105
JSON.stringify(config, null, 2) + "\n",
106
+
{ mode: 0o600 },
103
107
);
104
-
}
105
-
106
-
/** Resolve the default config directory path. */
107
-
export function configDir(): string {
108
-
return CONFIG_DIR;
109
108
}
110
109
111
110
/** Resolve the default config file path. */
···
120
119
export function requireConfig(dir?: string): AtticConfig {
121
120
const config = loadConfig(dir);
122
121
if (config === null) {
122
+
const path = dir ? join(dir, "config.json") : configPath();
123
123
throw new Error(
124
-
`No config file found at ${configPath()}\n` +
124
+
`No config file found at ${path}\n` +
125
125
'Run "attic init" to set up your S3 connection, or create the file manually.',
126
126
);
127
127
}
+65
cli/src/keychain/keychain.ts
+65
cli/src/keychain/keychain.ts
···
1
+
const ACCOUNT = "attic";
2
+
3
+
export interface KeychainCredentials {
4
+
accessKeyId: string;
5
+
secretAccessKey: string;
6
+
}
7
+
8
+
/** Read S3 credentials from macOS Keychain. */
9
+
export async function loadKeychainCredentials(
10
+
accessKeyService = "attic-s3-access-key",
11
+
secretKeyService = "attic-s3-secret-key",
12
+
): Promise<KeychainCredentials> {
13
+
const accessKeyId = await keychainGet(accessKeyService);
14
+
const secretAccessKey = await keychainGet(secretKeyService);
15
+
return { accessKeyId, secretAccessKey };
16
+
}
17
+
18
+
/** Store a credential in macOS Keychain. Uses -U to update if it already exists. */
19
+
export async function storeKeychainCredential(
20
+
service: string,
21
+
value: string,
22
+
): Promise<void> {
23
+
const cmd = new Deno.Command("security", {
24
+
args: [
25
+
"add-generic-password",
26
+
"-U",
27
+
"-s",
28
+
service,
29
+
"-a",
30
+
ACCOUNT,
31
+
"-w",
32
+
value,
33
+
],
34
+
stderr: "piped",
35
+
});
36
+
const { code, stderr } = await cmd.output();
37
+
if (code !== 0) {
38
+
const err = new TextDecoder().decode(stderr);
39
+
throw new Error(
40
+
`Failed to store credential in Keychain for service "${service}": ${err.trim()}`,
41
+
);
42
+
}
43
+
}
44
+
45
+
async function keychainGet(service: string): Promise<string> {
46
+
const cmd = new Deno.Command("security", {
47
+
args: [
48
+
"find-generic-password",
49
+
"-s",
50
+
service,
51
+
"-w",
52
+
],
53
+
stdout: "piped",
54
+
stderr: "piped",
55
+
});
56
+
const { code, stdout, stderr } = await cmd.output();
57
+
if (code !== 0) {
58
+
const err = new TextDecoder().decode(stderr);
59
+
throw new Error(
60
+
`Failed to read keychain item "${service}": ${err.trim()}. ` +
61
+
`Store it with: security add-generic-password -s ${service} -a ${ACCOUNT} -w "<value>"`,
62
+
);
63
+
}
64
+
return new TextDecoder().decode(stdout).trim();
65
+
}
+8
-2
cli/src/photos-db/reader.test.ts
+8
-2
cli/src/photos-db/reader.test.ts
···
305
305
INSERT INTO ZASSET VALUES (1, 'uuid-adj-only', 'IMG.HEIC', '/dir', ${coreDataTs},
306
306
${AssetKind.PHOTO}, 'public.heic', 4032, 3024, NULL, NULL, 0, ${CloudLocalState.LOCAL}, 0);
307
307
INSERT INTO ZADDITIONALASSETATTRIBUTES VALUES (1, 1, 1000, 'IMG.HEIC', 'hash', NULL, 1);
308
-
INSERT INTO ZUNMANAGEDADJUSTMENT VALUES (1, ${coreDataTs + 100}, 'com.apple.photo');
308
+
INSERT INTO ZUNMANAGEDADJUSTMENT VALUES (1, ${
309
+
coreDataTs + 100
310
+
}, 'com.apple.photo');
309
311
`);
310
312
// No ZINTERNALRESOURCE row — adjustment exists but no rendered file
311
313
db.close();
···
316
318
reader.close();
317
319
318
320
assertEquals(assets.length, 1);
319
-
assertEquals(assets[0].hasEdit, false, "no rendered resource = hasEdit false");
321
+
assertEquals(
322
+
assets[0].hasEdit,
323
+
false,
324
+
"no rendered resource = hasEdit false",
325
+
);
320
326
assertEquals(assets[0].editedAt, null);
321
327
assertEquals(assets[0].editor, null);
322
328
} finally {
+2
-42
cli/src/storage/s3-client.ts
+2
-42
cli/src/storage/s3-client.ts
···
29
29
listObjects(prefix: string): AsyncIterable<S3Object>;
30
30
}
31
31
32
-
export interface S3Credentials {
33
-
accessKeyId: string;
34
-
secretAccessKey: string;
35
-
}
36
-
37
32
export interface S3ConnectionConfig {
38
33
endpoint: string;
39
34
region: string;
40
35
pathStyle: boolean;
41
36
}
42
37
43
-
/** Read S3 credentials from macOS Keychain. */
44
-
export async function loadKeychainCredentials(
45
-
accessKeyService = "attic-s3-access-key",
46
-
secretKeyService = "attic-s3-secret-key",
47
-
): Promise<S3Credentials> {
48
-
const accessKeyId = await keychainGet(accessKeyService);
49
-
const secretAccessKey = await keychainGet(secretKeyService);
50
-
return { accessKeyId, secretAccessKey };
51
-
}
52
-
53
-
async function keychainGet(service: string): Promise<string> {
54
-
const cmd = new Deno.Command("security", {
55
-
args: [
56
-
"find-generic-password",
57
-
"-s",
58
-
service,
59
-
"-w",
60
-
],
61
-
stdout: "piped",
62
-
stderr: "piped",
63
-
});
64
-
const { code, stdout, stderr } = await cmd.output();
65
-
if (code !== 0) {
66
-
const err = new TextDecoder().decode(stderr);
67
-
throw new Error(
68
-
`Failed to read keychain item "${service}": ${err.trim()}. ` +
69
-
`Store it with: security add-generic-password -s ${service} -a attic -w "<value>"`,
70
-
);
71
-
}
72
-
return new TextDecoder().decode(stdout).trim();
73
-
}
74
-
75
38
export function createS3Provider(
76
-
credentials: S3Credentials,
39
+
credentials: { accessKeyId: string; secretAccessKey: string },
77
40
bucket: string,
78
41
connection: S3ConnectionConfig,
79
42
): S3Provider {
···
123
86
etag: result.ETag ?? null,
124
87
};
125
88
} catch (error: unknown) {
126
-
if (
127
-
error instanceof Error && "name" in error &&
128
-
error.name === "NotFound"
129
-
) {
89
+
if (error instanceof Error && error.name === "NotFound") {
130
90
return null;
131
91
}
132
92
throw error;
+10
config.example.json
+10
config.example.json
+84
-39
docs/architecture.md
+84
-39
docs/architecture.md
···
1
1
# Architecture
2
2
3
-
Attic reads the macOS Photos library, exports original files via a companion Swift tool, and uploads them to S3 with rich metadata. A local manifest tracks progress so runs are incremental.
3
+
Attic reads the macOS Photos library, exports original files via a companion
4
+
Swift tool, and uploads them to S3 with rich metadata. A local manifest tracks
5
+
progress so runs are incremental.
4
6
5
7
## System overview
6
8
···
21
23
22
24
`reader.ts` queries Photos.sqlite in two stages:
23
25
24
-
**Main query** — a single SELECT joining `ZASSET` and `ZADDITIONALASSETATTRIBUTES` returns core fields: UUID, filename, date, dimensions, GPS, file size, UTI, favorite status, cloud state. Trashed assets are excluded.
26
+
**Main query** — a single SELECT joining `ZASSET` and
27
+
`ZADDITIONALASSETATTRIBUTES` returns core fields: UUID, filename, date,
28
+
dimensions, GPS, file size, UTI, favorite status, cloud state. Trashed assets
29
+
are excluded.
25
30
26
-
**Enrichment queries** — seven independent queries each build a `Map` keyed by asset primary key (Z_PK). During row mapping, each asset is enriched from these maps with a default of `null` or `[]` if no match exists.
31
+
**Enrichment queries** — seven independent queries each build a `Map` keyed by
32
+
asset primary key (Z_PK). During row mapping, each asset is enriched from these
33
+
maps with a default of `null` or `[]` if no match exists.
27
34
28
-
| Query | Source tables | Returns |
29
-
|-------|--------------|---------|
30
-
| Descriptions | `ZASSETDESCRIPTION` → `ZADDITIONALASSETATTRIBUTES` | `Map<number, string>` |
31
-
| Albums | `ZGENERICALBUM` → `Z_33ASSETS` | `Map<number, AlbumRef[]>` |
32
-
| Keywords | `ZKEYWORD` → `Z_1KEYWORDS` → `ZADDITIONALASSETATTRIBUTES` | `Map<number, string[]>` |
33
-
| People | `ZPERSON` → `ZDETECTEDFACE` | `Map<number, PersonRef[]>` |
34
-
| Edits | `ZUNMANAGEDADJUSTMENT` → `ZADDITIONALASSETATTRIBUTES` | `Map<number, EditInfo>` |
35
-
| Rendered resources | `ZINTERNALRESOURCE` (resource type 1) | `Set<number>` |
35
+
| Query | Source tables | Returns |
36
+
| ------------------ | --------------------------------------------------------- | -------------------------- |
37
+
| Descriptions | `ZASSETDESCRIPTION` → `ZADDITIONALASSETATTRIBUTES` | `Map<number, string>` |
38
+
| Albums | `ZGENERICALBUM` → `Z_33ASSETS` | `Map<number, AlbumRef[]>` |
39
+
| Keywords | `ZKEYWORD` → `Z_1KEYWORDS` → `ZADDITIONALASSETATTRIBUTES` | `Map<number, string[]>` |
40
+
| People | `ZPERSON` → `ZDETECTEDFACE` | `Map<number, PersonRef[]>` |
41
+
| Edits | `ZUNMANAGEDADJUSTMENT` → `ZADDITIONALASSETATTRIBUTES` | `Map<number, EditInfo>` |
42
+
| Rendered resources | `ZINTERNALRESOURCE` (resource type 1) | `Set<number>` |
36
43
37
-
All enrichment queries go through `safeQuery()`, which catches "no such table" errors silently and logs other failures. This makes the reader resilient across macOS versions where table schemas may differ.
44
+
All enrichment queries go through `safeQuery()`, which catches "no such table"
45
+
errors silently and logs other failures. This makes the reader resilient across
46
+
macOS versions where table schemas may differ.
38
47
39
-
People are deduplicated per asset — a person appears at most once even if detected in multiple face regions.
48
+
People are deduplicated per asset — a person appears at most once even if
49
+
detected in multiple face regions.
40
50
41
51
### Edit detection
42
52
43
-
An asset is considered edited (`hasEdit: true`) only when two conditions are met:
53
+
An asset is considered edited (`hasEdit: true`) only when two conditions are
54
+
met:
44
55
45
56
1. An entry exists in `ZUNMANAGEDADJUSTMENT` (an edit was performed)
46
-
2. An entry exists in `ZINTERNALRESOURCE` with resource type 1 (a rendered file was produced)
57
+
2. An entry exists in `ZINTERNALRESOURCE` with resource type 1 (a rendered file
58
+
was produced)
47
59
48
-
This distinguishes visual edits from metadata-only adjustments that don't produce a visible render. When `hasEdit` is false, `editedAt` and `editor` are both null.
60
+
This distinguishes visual edits from metadata-only adjustments that don't
61
+
produce a visible render. When `hasEdit` is false, `editedAt` and `editor` are
62
+
both null.
49
63
50
64
## The backup pipeline
51
65
52
-
`backup.ts` orchestrates the full flow: filter → batch → export → upload → manifest.
66
+
`backup.ts` orchestrates the full flow: filter → batch → export → upload →
67
+
manifest.
53
68
54
69
### 1. Filter
55
70
56
-
Assets are filtered against the manifest to find pending work. Optional filters narrow by type (`--type photo|video`) or count (`--limit N`). Dry run mode stops here.
71
+
Assets are filtered against the manifest to find pending work. Optional filters
72
+
narrow by type (`--type photo|video`) or count (`--limit N`). Dry run mode stops
73
+
here.
57
74
58
75
### 2. Batch and export
59
76
60
-
Pending assets are processed in batches (default 50). Each batch is sent to **ladder**, a companion Swift binary that uses PhotoKit to export original files. Communication is via JSON over stdin/stdout:
77
+
Pending assets are processed in batches (default 50). Each batch is sent to
78
+
**ladder**, a companion Swift binary that uses PhotoKit to export original
79
+
files. Communication is via JSON over stdin/stdout:
61
80
62
81
```
63
82
attic → stdin: { "uuids": ["UUID/L0/001", ...], "stagingDir": "/path" }
64
83
ladder → stdout: { "results": [...], "errors": [...] }
65
84
```
66
85
67
-
Each result includes the file path, size, and SHA-256 hash. PhotoKit identifiers use the `UUID/L0/001` format; attic strips the suffix before further processing. Ladder output is validated at the trust boundary with `assertExportBatchResult()`.
86
+
Each result includes the file path, size, and SHA-256 hash. PhotoKit identifiers
87
+
use the `UUID/L0/001` format; attic strips the suffix before further processing.
88
+
Ladder output is validated at the trust boundary with
89
+
`assertExportBatchResult()`.
68
90
69
91
### 3. Upload
70
92
···
72
94
73
95
1. Reads the staged file from disk
74
96
2. Uploads the original to `originals/{year}/{month}/{uuid}.{ext}`
75
-
3. Builds and uploads a metadata JSON to `metadata/assets/{uuid}.json` (see `docs/metadata.md`)
97
+
3. Builds and uploads a metadata JSON to `metadata/assets/{uuid}.json` (see
98
+
`docs/metadata.md`)
76
99
4. Updates the in-memory manifest
77
100
5. Cleans up the staged file
78
101
79
-
S3 keys are built from UUID and extension, both validated with regex (`/^[A-Za-z0-9._-]+$/` and `/^[a-z0-9]+$/`) to prevent path traversal. Extensions are resolved from the asset's UTI via a lookup table, falling back to the filename extension.
102
+
S3 keys are built from UUID and extension, both validated with regex
103
+
(`/^[A-Za-z0-9._-]+$/` and `/^[a-z0-9]+$/`) to prevent path traversal.
104
+
Extensions are resolved from the asset's UTI via a lookup table, falling back to
105
+
the filename extension.
80
106
81
107
### 4. Manifest
82
108
83
-
The manifest is a JSON file at `~/.attic/manifest.json` mapping UUID to `{ s3Key, checksum, backedUpAt }`. It's saved periodically during backup (every 50 assets by default) and always at the end. Writes are atomic: write to `.tmp`, then rename.
109
+
The manifest is a JSON file at `~/.attic/manifest.json` mapping UUID to
110
+
`{ s3Key, checksum, backedUpAt }`. It's saved periodically during backup (every
111
+
50 assets by default) and always at the end. Writes are atomic: write to `.tmp`,
112
+
then rename.
84
113
85
-
The manifest can be reconstructed from S3 via `verify --rebuild-manifest`, which reads every `metadata/assets/*.json` file and validates UUID format, S3 key pattern, and checksum format before accepting an entry.
114
+
The manifest can be reconstructed from S3 via `verify --rebuild-manifest`, which
115
+
reads every `metadata/assets/*.json` file and validates UUID format, S3 key
116
+
pattern, and checksum format before accepting an entry.
86
117
87
118
## Verification
88
119
89
120
`verify.ts` checks backup integrity in two modes:
90
121
91
122
- **Quick** (default) — HEAD each S3 key in the manifest, confirm it exists
92
-
- **Deep** — download each object, compute SHA-256, compare to the manifest checksum
123
+
- **Deep** — download each object, compute SHA-256, compare to the manifest
124
+
checksum
93
125
94
-
Both modes use a bounded concurrency pool (default 50 workers). Errors are capped at 1,000 to prevent unbounded memory growth.
126
+
Both modes use a bounded concurrency pool (default 50 workers). Errors are
127
+
capped at 1,000 to prevent unbounded memory growth.
95
128
96
129
## Configuration
97
130
98
-
Attic reads its configuration from `~/.attic/config.json`. The config file specifies the S3 endpoint, region, bucket, path-style preference, and Keychain service names. It's created by `attic init` or manually.
131
+
Attic reads its configuration from `~/.attic/config.json`. The config file
132
+
specifies the S3 endpoint, region, bucket, path-style preference, and Keychain
133
+
service names. It's created by `attic init` or manually.
99
134
100
-
`scan` and `status` work without config (they only read Photos.sqlite). `backup` and `verify` require config and fail fast with a clear message if it's missing or invalid.
135
+
`scan` and `status` work without config (they only read Photos.sqlite). `backup`
136
+
and `verify` require config and fail fast with a clear message if it's missing
137
+
or invalid.
101
138
102
139
## Credentials
103
140
104
-
S3 credentials are stored in the macOS Keychain under configurable service names (defaults: `attic-s3-access-key` and `attic-s3-secret-key`). They are read at runtime via `security find-generic-password` — never stored in env vars, config files, or code.
141
+
S3 credentials are stored in the macOS Keychain under configurable service names
142
+
(defaults: `attic-s3-access-key` and `attic-s3-secret-key`). They are read at
143
+
runtime via `security find-generic-password` — never stored in env vars, config
144
+
files, or code.
105
145
106
146
## Interfaces and testability
107
147
108
148
All external dependencies are behind interfaces:
109
149
110
-
| Interface | Real implementation | Mock |
111
-
|-----------|-------------------|------|
112
-
| `S3Provider` | AWS SDK client for any S3-compatible endpoint | In-memory `Map<string, Uint8Array>` |
113
-
| `Exporter` | Ladder subprocess | Returns pre-configured assets from a `Map` |
114
-
| `ManifestStore` | File-based JSON with atomic writes | Same implementation, pointed at a temp dir |
115
-
| `PhotosDbReader` | SQLite reader for Photos.sqlite | In-memory SQLite with test fixtures |
150
+
| Interface | Real implementation | Mock |
151
+
| ---------------- | --------------------------------------------- | ------------------------------------------ |
152
+
| `S3Provider` | AWS SDK client for any S3-compatible endpoint | In-memory `Map<string, Uint8Array>` |
153
+
| `Exporter` | Ladder subprocess | Returns pre-configured assets from a `Map` |
154
+
| `ManifestStore` | File-based JSON with atomic writes | Same implementation, pointed at a temp dir |
155
+
| `PhotosDbReader` | SQLite reader for Photos.sqlite | In-memory SQLite with test fixtures |
116
156
117
157
Tests never hit external services, credentials, or the real Photos library.
118
158
···
120
160
121
161
- **Modify Photos.sqlite** — read-only access, always
122
162
- **Download from iCloud** — relies on Photos having local copies of originals
123
-
- **Delete from S3** — the backup is append-only; there is no prune or cleanup command
163
+
- **Delete from S3** — the backup is append-only; there is no prune or cleanup
164
+
command
124
165
- **Back up thumbnails** — only original files and metadata
125
166
- **Back up adjustment plists** — Apple's edit recipes are not portable
126
-
- **Back up rendered edits** — detecting edits is implemented (Phase 1); exporting and uploading rendered versions is planned (Phase 2/3, see `docs/plans/2026-03-13-feat-backup-rendered-edits-plan.md`)
127
-
- **Handle slo-mo or Live Photos specially** — these have unique resource types that need dedicated investigation
128
-
- **Run on non-macOS** — depends on Photos.sqlite, Keychain, and PhotoKit via ladder
167
+
- **Back up rendered edits** — detecting edits is implemented (Phase 1);
168
+
exporting and uploading rendered versions is planned (Phase 2/3, see
169
+
`docs/plans/2026-03-13-feat-backup-rendered-edits-plan.md`)
170
+
- **Handle slo-mo or Live Photos specially** — these have unique resource types
171
+
that need dedicated investigation
172
+
- **Run on non-macOS** — depends on Photos.sqlite, Keychain, and PhotoKit via
173
+
ladder
+54
-25
docs/brainstorms/2026-03-13-edited-assets-backup-brainstorm.md
+54
-25
docs/brainstorms/2026-03-13-edited-assets-backup-brainstorm.md
···
1
1
# Back Up Rendered Edits Alongside Originals
2
2
3
-
**Date:** 2026-03-13
4
-
**Status:** Ready for planning
3
+
**Date:** 2026-03-13 **Status:** Ready for planning
5
4
6
5
## What We're Building
7
6
8
-
Extend the backup pipeline to detect edited photos/videos and upload the rendered (fullsize JPEG) version alongside the original. Also detect edits made to already-backed-up assets and upload their rendered versions retroactively.
7
+
Extend the backup pipeline to detect edited photos/videos and upload the
8
+
rendered (fullsize JPEG) version alongside the original. Also detect edits made
9
+
to already-backed-up assets and upload their rendered versions retroactively.
9
10
10
11
## Why This Approach
11
12
12
-
The backup should be self-contained and viewable without Apple Photos. Currently only originals are backed up. Apple Photos edits are non-destructive (the original is always preserved), but the "finished" version a user actually wants to see requires either Apple Photos or the adjustment plist to re-render. Backing up the rendered version makes the backup independently useful.
13
+
The backup should be self-contained and viewable without Apple Photos. Currently
14
+
only originals are backed up. Apple Photos edits are non-destructive (the
15
+
original is always preserved), but the "finished" version a user actually wants
16
+
to see requires either Apple Photos or the adjustment plist to re-render.
17
+
Backing up the rendered version makes the backup independently useful.
13
18
14
19
## Current State
15
20
16
-
| Metric | Value |
17
-
|--------|-------|
18
-
| Total assets | 37,289 |
19
-
| Edited assets | 1,312 (3.5%) |
21
+
| Metric | Value |
22
+
| -------------------------------------------------- | --------------- |
23
+
| Total assets | 37,289 |
24
+
| Edited assets | 1,312 (3.5%) |
20
25
| Rendered versions (fullsize JPEG, resource type 1) | 1,672 resources |
21
-
| Locally available renders | 1,480 |
22
-
| Original size (edited subset) | ~6.2 GB |
23
-
| Rendered size (edited subset) | ~13.7 GB |
26
+
| Locally available renders | 1,480 |
27
+
| Original size (edited subset) | ~6.2 GB |
28
+
| Rendered size (edited subset) | ~13.7 GB |
24
29
25
-
Edit sources: Apple Photos (1,181), slo-mo (47), Google Photos (42), Markup (11), Adobe Lens (9), Snapseed (3).
30
+
Edit sources: Apple Photos (1,181), slo-mo (47), Google Photos (42), Markup
31
+
(11), Adobe Lens (9), Snapseed (3).
26
32
27
33
## Key Decisions
28
34
29
-
1. **Back up rendered versions** (not just adjustment plists). The fullsize JPEG is what users actually see. Plists are Apple-internal and not portable.
35
+
1. **Back up rendered versions** (not just adjustment plists). The fullsize JPEG
36
+
is what users actually see. Plists are Apple-internal and not portable.
30
37
31
38
2. **Sibling key with `_edited` suffix** in S3:
32
39
```
···
35
42
metadata/assets/{uuid}.json # includes edit metadata
36
43
```
37
44
38
-
3. **Same pass as originals**. When processing a batch, detect edits and upload both files together. No separate command needed.
45
+
3. **Same pass as originals**. When processing a batch, detect edits and upload
46
+
both files together. No separate command needed.
39
47
40
-
4. **Re-scan already-backed-up assets for new edits**. Compare adjustment timestamps against the manifest's `backedUpAt` to detect photos edited after their initial backup.
48
+
4. **Re-scan already-backed-up assets for new edits**. Compare adjustment
49
+
timestamps against the manifest's `backedUpAt` to detect photos edited after
50
+
their initial backup.
41
51
42
52
## Data Sources in Photos.sqlite
43
53
44
54
### Edit detection
45
-
- `ZUNMANAGEDADJUSTMENT` joined via `ZADDITIONALASSETATTRIBUTES` tells us an asset has been edited
55
+
56
+
- `ZUNMANAGEDADJUSTMENT` joined via `ZADDITIONALASSETATTRIBUTES` tells us an
57
+
asset has been edited
46
58
- `ZADJUSTMENTTIMESTAMP` tells us when the edit happened
47
-
- `ZADJUSTMENTFORMATIDENTIFIER` tells us which editor (com.apple.photo, com.adobe.lens, etc.)
59
+
- `ZADJUSTMENTFORMATIDENTIFIER` tells us which editor (com.apple.photo,
60
+
com.adobe.lens, etc.)
48
61
49
62
### Rendered file location
50
-
- `ZINTERNALRESOURCE` with `ZRESOURCETYPE = 1` (fullsize JPEG) points to the rendered version
63
+
64
+
- `ZINTERNALRESOURCE` with `ZRESOURCETYPE = 1` (fullsize JPEG) points to the
65
+
rendered version
51
66
- `ZLOCALAVAILABILITY = 1` means the file is on disk
52
67
- `ZDATALENGTH` gives the file size
53
-
- The actual file lives in the Photos Library package, path derivable from `ZDATASTORECLASSID` + fingerprint
68
+
- The actual file lives in the Photos Library package, path derivable from
69
+
`ZDATASTORECLASSID` + fingerprint
54
70
55
71
### Export via ladder
72
+
56
73
- The current exporter uses PhotoKit ID `{uuid}/L0/001` for originals
57
-
- Rendered versions may need a different resource variant or direct file copy from the library package
74
+
- Rendered versions may need a different resource variant or direct file copy
75
+
from the library package
58
76
59
77
## Scope
60
78
61
79
### In scope
80
+
62
81
- Detect which assets have edits (via ZUNMANAGEDADJUSTMENT)
63
-
- Add edit metadata to PhotoAsset and the S3 metadata JSON (hasEdit, editedAt, editor)
82
+
- Add edit metadata to PhotoAsset and the S3 metadata JSON (hasEdit, editedAt,
83
+
editor)
64
84
- Export and upload rendered fullsize JPEG alongside original
65
85
- Re-scan manifest for assets edited after backup
66
86
- Track edit backup state in manifest (so renders are not re-uploaded)
67
87
68
88
### Out of scope
89
+
69
90
- Backing up adjustment plists (edit recipes)
70
91
- Handling slo-mo video rendering (complex, different pipeline)
71
92
- Re-rendering from adjustment data outside Apple Photos
···
73
94
74
95
## Resolved Questions
75
96
76
-
1. **Manifest schema**: Extend the existing manifest entry with optional `editS3Key`, `editChecksum`, `editBackedUpAt` fields. No separate entries, no schema break.
97
+
1. **Manifest schema**: Extend the existing manifest entry with optional
98
+
`editS3Key`, `editChecksum`, `editBackedUpAt` fields. No separate entries, no
99
+
schema break.
77
100
78
-
2. **Re-edit handling**: Always upload the latest render. Compare adjustment timestamp against `editBackedUpAt` to detect re-edits. The backup should reflect the current state of the edit.
101
+
2. **Re-edit handling**: Always upload the latest render. Compare adjustment
102
+
timestamp against `editBackedUpAt` to detect re-edits. The backup should
103
+
reflect the current state of the edit.
79
104
80
105
## Open Questions
81
106
82
-
1. **How does ladder/PhotoKit export the rendered version?** The current `/L0/001` suffix gets the original. Need to investigate what identifier or API call retrieves the fullsize rendered JPEG. May need a ladder change.
107
+
1. **How does ladder/PhotoKit export the rendered version?** The current
108
+
`/L0/001` suffix gets the original. Need to investigate what identifier or
109
+
API call retrieves the fullsize rendered JPEG. May need a ladder change.
83
110
84
-
2. **What about iCloud-only rendered versions?** 1,480 of 1,672 renders are local. The remaining ~200 may need to be downloaded first, same as iCloud-only originals. Is there an existing mechanism for this?
111
+
2. **What about iCloud-only rendered versions?** 1,480 of 1,672 renders are
112
+
local. The remaining ~200 may need to be downloaded first, same as
113
+
iCloud-only originals. Is there an existing mechanism for this?
+55
-27
docs/brainstorms/2026-03-13-ux-open-source-readiness-brainstorm.md
+55
-27
docs/brainstorms/2026-03-13-ux-open-source-readiness-brainstorm.md
···
1
1
# UX and Open-Source Readiness
2
2
3
-
**Date:** 2026-03-13
4
-
**Status:** Ready for planning
3
+
**Date:** 2026-03-13 **Status:** Ready for planning
5
4
6
5
## What We're Building
7
6
8
-
Make attic friendly to use for technical Mac users and ready to open source. Replace hardcoded Scaleway configuration with a generic S3-compatible config layer, add an interactive `attic init` command, adopt Cliffy for polished CLI output, and improve error messages throughout.
7
+
Make attic friendly to use for technical Mac users and ready to open source.
8
+
Replace hardcoded Scaleway configuration with a generic S3-compatible config
9
+
layer, add an interactive `attic init` command, adopt Cliffy for polished CLI
10
+
output, and improve error messages throughout.
9
11
10
12
## Why This Approach
11
13
12
-
Attic currently works well as a personal tool but has Scaleway details baked into the code (endpoint, region, keychain service names, type names). To open source it, the tool needs to work with any S3-compatible provider out of the box. The UX should feel polished — good help text, colored output, and clear error messages that tell you what to do next.
14
+
Attic currently works well as a personal tool but has Scaleway details baked
15
+
into the code (endpoint, region, keychain service names, type names). To open
16
+
source it, the tool needs to work with any S3-compatible provider out of the
17
+
box. The UX should feel polished — good help text, colored output, and clear
18
+
error messages that tell you what to do next.
13
19
14
20
## Current State
15
21
16
-
| Area | Current | Target |
17
-
|------|---------|--------|
18
-
| S3 endpoint/region | Hardcoded Scaleway constants | Config file, any S3-compatible provider |
19
-
| Bucket name | Hardcoded default, `--bucket` flag | Config file, CLI override |
20
-
| Credentials | Keychain with hardcoded service names | Keychain with configurable service names |
21
-
| Config file | None | `~/.attic/config.json` |
22
-
| First-run setup | Manual (read README, set keychain, run) | `attic init` interactive prompts |
23
-
| CLI framework | Hand-rolled arg parsing | Cliffy (subcommands, typed flags, help, color) |
24
-
| Error messages | Raw exceptions in some paths | Friendly messages with suggested fixes |
25
-
| path style | Hardcoded `true` | Config option, default `true` |
26
-
| Provider docs | Scaleway-specific | Provider-neutral with EU-focused examples |
22
+
| Area | Current | Target |
23
+
| ------------------ | --------------------------------------- | ---------------------------------------------- |
24
+
| S3 endpoint/region | Hardcoded Scaleway constants | Config file, any S3-compatible provider |
25
+
| Bucket name | Hardcoded default, `--bucket` flag | Config file, CLI override |
26
+
| Credentials | Keychain with hardcoded service names | Keychain with configurable service names |
27
+
| Config file | None | `~/.attic/config.json` |
28
+
| First-run setup | Manual (read README, set keychain, run) | `attic init` interactive prompts |
29
+
| CLI framework | Hand-rolled arg parsing | Cliffy (subcommands, typed flags, help, color) |
30
+
| Error messages | Raw exceptions in some paths | Friendly messages with suggested fixes |
31
+
| path style | Hardcoded `true` | Config option, default `true` |
32
+
| Provider docs | Scaleway-specific | Provider-neutral with EU-focused examples |
27
33
28
34
## Key Decisions
29
35
30
-
1. **Config file at `~/.attic/config.json`** — primary configuration source. CLI flags override. No env var fallback (keep it simple, macOS-only tool).
36
+
1. **Config file at `~/.attic/config.json`** — primary configuration source. CLI
37
+
flags override. No env var fallback (keep it simple, macOS-only tool).
31
38
32
-
2. **Interactive `attic init`** — asks for S3 endpoint, region, bucket, and keychain service names step by step. Writes config.json. Can offer provider suggestions (Scaleway, Hetzner, OVH as EU options).
39
+
2. **Interactive `attic init`** — asks for S3 endpoint, region, bucket, and
40
+
keychain service names step by step. Writes config.json. Can offer provider
41
+
suggestions (Scaleway, Hetzner, OVH as EU options).
33
42
34
-
3. **Keychain with configurable service names** — stay macOS Keychain-only (security principle from CLAUDE.md), but let config.json specify the service names instead of hardcoding `attic-s3-access-key` / `attic-s3-secret-key`.
43
+
3. **Keychain with configurable service names** — stay macOS Keychain-only
44
+
(security principle from CLAUDE.md), but let config.json specify the service
45
+
names instead of hardcoding `attic-s3-access-key` / `attic-s3-secret-key`.
35
46
36
-
4. **Cliffy for CLI** — replace hand-rolled arg parsing with Cliffy. Gets us subcommands, typed flags, auto-generated help, colored output, and shell completions.
47
+
4. **Cliffy for CLI** — replace hand-rolled arg parsing with Cliffy. Gets us
48
+
subcommands, typed flags, auto-generated help, colored output, and shell
49
+
completions.
37
50
38
-
5. **`forcePathStyle` as config option** — default `true` (works with most S3-compatible providers). AWS users can set to `false`.
51
+
5. **`forcePathStyle` as config option** — default `true` (works with most
52
+
S3-compatible providers). AWS users can set to `false`.
39
53
40
-
6. **EU-focused provider examples** — highlight Scaleway, Hetzner, OVH as EU data sovereignty options in docs and init prompts. Mention AWS/Backblaze as alternatives. Position attic as a good choice for keeping your photos in the EU.
54
+
6. **EU-focused provider examples** — highlight Scaleway, Hetzner, OVH as EU
55
+
data sovereignty options in docs and init prompts. Mention AWS/Backblaze as
56
+
alternatives. Position attic as a good choice for keeping your photos in the
57
+
EU.
41
58
42
-
7. **Top-level error boundary** — catch unhandled errors in mod.ts, present friendly messages instead of stack traces. Pattern: detect known error types (keychain missing, network timeout, S3 access denied) and print actionable guidance.
59
+
7. **Top-level error boundary** — catch unhandled errors in mod.ts, present
60
+
friendly messages instead of stack traces. Pattern: detect known error types
61
+
(keychain missing, network timeout, S3 access denied) and print actionable
62
+
guidance.
43
63
44
64
## Config File Schema
45
65
···
59
79
## Scope
60
80
61
81
### In scope
82
+
62
83
- Config file (`~/.attic/config.json`) with validation
63
84
- `attic init` interactive setup command
64
85
- Cliffy migration for all commands (scan, status, backup, verify)
···
69
90
- EU-focused provider examples in docs and init
70
91
71
92
### Out of scope
93
+
72
94
- Env var credential fallback (keep Keychain-only)
73
95
- Non-macOS support
74
-
- Provider presets in init (just ask for endpoint/region directly, with examples)
96
+
- Provider presets in init (just ask for endpoint/region directly, with
97
+
examples)
75
98
- Web UI or GUI
76
99
- Auto-detection of Photos.sqlite path across macOS versions
77
100
78
101
## Resolved Questions
79
102
80
103
1. **Audience**: Technical Mac users comfortable with terminal and S3 setup.
81
-
2. **Config approach**: Config file at `~/.attic/config.json`, CLI flags override.
104
+
2. **Config approach**: Config file at `~/.attic/config.json`, CLI flags
105
+
override.
82
106
3. **Init style**: Interactive prompts, writes config at the end.
83
107
4. **Credentials**: Keychain-only with configurable service names in config.
84
-
5. **Provider presentation**: EU-focused examples (Scaleway, Hetzner, OVH), others mentioned as alternatives.
108
+
5. **Provider presentation**: EU-focused examples (Scaleway, Hetzner, OVH),
109
+
others mentioned as alternatives.
85
110
6. **Path style**: Config option `pathStyle`, default `true`.
86
111
7. **CLI framework**: Cliffy.
87
-
8. **Init stores credentials directly**: `attic init` prompts for access key and secret key and runs `security add-generic-password` automatically.
88
-
9. **Validate config when S3 is needed**: scan/status only need Photos.sqlite — they work without config. backup/verify validate config and fail fast with a clear message if missing or incomplete.
112
+
8. **Init stores credentials directly**: `attic init` prompts for access key and
113
+
secret key and runs `security add-generic-password` automatically.
114
+
9. **Validate config when S3 is needed**: scan/status only need Photos.sqlite —
115
+
they work without config. backup/verify validate config and fail fast with a
116
+
clear message if missing or incomplete.
+52
-39
docs/metadata.md
+52
-39
docs/metadata.md
···
1
1
# Asset Metadata
2
2
3
-
Each backed-up asset gets a companion JSON file uploaded to S3 at `metadata/assets/{uuid}.json`. This file makes the backup browsable and searchable without access to Apple Photos or the original Photos.sqlite database.
3
+
Each backed-up asset gets a companion JSON file uploaded to S3 at
4
+
`metadata/assets/{uuid}.json`. This file makes the backup browsable and
5
+
searchable without access to Apple Photos or the original Photos.sqlite
6
+
database.
4
7
5
8
## Example
6
9
···
39
42
40
43
### Asset identification
41
44
42
-
| Field | Type | Description |
43
-
|-------|------|-------------|
44
-
| `uuid` | string | Photos library UUID, unique per asset |
45
+
| Field | Type | Description |
46
+
| ------------------ | ------ | ------------------------------------------- |
47
+
| `uuid` | string | Photos library UUID, unique per asset |
45
48
| `originalFilename` | string | Filename as imported (e.g. `IMG_4231.HEIC`) |
46
49
47
50
### Date and dimensions
48
51
49
-
| Field | Type | Description |
50
-
|-------|------|-------------|
52
+
| Field | Type | Description |
53
+
| ------------- | -------------- | ---------------------------------------------------------------- |
51
54
| `dateCreated` | string \| null | ISO 8601 timestamp from Photos.sqlite (CoreData epoch converted) |
52
-
| `width` | number | Pixel width |
53
-
| `height` | number | Pixel height |
55
+
| `width` | number | Pixel width |
56
+
| `height` | number | Pixel height |
54
57
55
58
### Location
56
59
57
-
| Field | Type | Description |
58
-
|-------|------|-------------|
59
-
| `latitude` | number \| null | GPS latitude, null if no location data |
60
+
| Field | Type | Description |
61
+
| ----------- | -------------- | --------------------------------------- |
62
+
| `latitude` | number \| null | GPS latitude, null if no location data |
60
63
| `longitude` | number \| null | GPS longitude, null if no location data |
61
64
62
65
### File info
63
66
64
-
| Field | Type | Description |
65
-
|-------|------|-------------|
66
-
| `fileSize` | number \| null | Original file size in bytes |
67
-
| `type` | string \| null | Uniform Type Identifier (e.g. `public.heic`, `com.apple.quicktime-movie`) |
68
-
| `favorite` | boolean | Whether the asset is marked as a favorite in Photos |
67
+
| Field | Type | Description |
68
+
| ---------- | -------------- | ------------------------------------------------------------------------- |
69
+
| `fileSize` | number \| null | Original file size in bytes |
70
+
| `type` | string \| null | Uniform Type Identifier (e.g. `public.heic`, `com.apple.quicktime-movie`) |
71
+
| `favorite` | boolean | Whether the asset is marked as a favorite in Photos |
69
72
70
73
### Enrichment
71
74
72
-
These fields come from auxiliary tables in Photos.sqlite via separate enrichment queries. All degrade gracefully — if the source table is missing (older macOS versions), the field returns its empty default.
75
+
These fields come from auxiliary tables in Photos.sqlite via separate enrichment
76
+
queries. All degrade gracefully — if the source table is missing (older macOS
77
+
versions), the field returns its empty default.
73
78
74
-
| Field | Type | Default | Source table |
75
-
|-------|------|---------|--------------|
76
-
| `title` | string \| null | null | `ZADDITIONALASSETATTRIBUTES.ZTITLE` |
77
-
| `description` | string \| null | null | `ZASSETDESCRIPTION.ZLONGDESCRIPTION` |
78
-
| `albums` | AlbumRef[] | [] | `ZGENERICALBUM` via `Z_33ASSETS` join |
79
-
| `keywords` | string[] | [] | `ZKEYWORD` via `Z_1KEYWORDS` join |
80
-
| `people` | PersonRef[] | [] | `ZPERSON` via `ZDETECTEDFACE` join |
79
+
| Field | Type | Default | Source table |
80
+
| ------------- | -------------- | ------- | ------------------------------------- |
81
+
| `title` | string \| null | null | `ZADDITIONALASSETATTRIBUTES.ZTITLE` |
82
+
| `description` | string \| null | null | `ZASSETDESCRIPTION.ZLONGDESCRIPTION` |
83
+
| `albums` | AlbumRef[] | [] | `ZGENERICALBUM` via `Z_33ASSETS` join |
84
+
| `keywords` | string[] | [] | `ZKEYWORD` via `Z_1KEYWORDS` join |
85
+
| `people` | PersonRef[] | [] | `ZPERSON` via `ZDETECTEDFACE` join |
81
86
82
-
An `AlbumRef` contains `uuid` and `title`. A `PersonRef` contains `uuid` and `displayName`. People are deduplicated per asset (a person appears at most once even if detected in multiple face regions).
87
+
An `AlbumRef` contains `uuid` and `title`. A `PersonRef` contains `uuid` and
88
+
`displayName`. People are deduplicated per asset (a person appears at most once
89
+
even if detected in multiple face regions).
83
90
84
91
### Edit detection
85
92
86
-
| Field | Type | Description |
87
-
|-------|------|-------------|
88
-
| `hasEdit` | boolean | True only when both an adjustment record and a rendered resource exist |
89
-
| `editedAt` | string \| null | ISO 8601 timestamp of the edit, null when `hasEdit` is false |
90
-
| `editor` | string \| null | Bundle ID of the editing app (e.g. `com.apple.photo`, `com.pixelmator.photomator`), null when `hasEdit` is false |
93
+
| Field | Type | Description |
94
+
| ---------- | -------------- | ---------------------------------------------------------------------------------------------------------------- |
95
+
| `hasEdit` | boolean | True only when both an adjustment record and a rendered resource exist |
96
+
| `editedAt` | string \| null | ISO 8601 timestamp of the edit, null when `hasEdit` is false |
97
+
| `editor` | string \| null | Bundle ID of the editing app (e.g. `com.apple.photo`, `com.pixelmator.photomator`), null when `hasEdit` is false |
91
98
92
-
`hasEdit` requires two conditions: an entry in `ZUNMANAGEDADJUSTMENT` (the edit happened) AND an entry in `ZINTERNALRESOURCE` with resource type 1 (a rendered file exists). Metadata-only adjustments that don't produce a visible render are excluded.
99
+
`hasEdit` requires two conditions: an entry in `ZUNMANAGEDADJUSTMENT` (the edit
100
+
happened) AND an entry in `ZINTERNALRESOURCE` with resource type 1 (a rendered
101
+
file exists). Metadata-only adjustments that don't produce a visible render are
102
+
excluded.
93
103
94
104
### Backup tracking
95
105
96
-
| Field | Type | Description |
97
-
|-------|------|-------------|
98
-
| `s3Key` | string | S3 object key where the original file is stored (e.g. `originals/2024/07/{uuid}.heic`) |
99
-
| `checksum` | string | SHA-256 hash of the uploaded file, prefixed with `sha256:` |
100
-
| `backedUpAt` | string | ISO 8601 timestamp of when this asset was uploaded |
106
+
| Field | Type | Description |
107
+
| ------------ | ------ | -------------------------------------------------------------------------------------- |
108
+
| `s3Key` | string | S3 object key where the original file is stored (e.g. `originals/2024/07/{uuid}.heic`) |
109
+
| `checksum` | string | SHA-256 hash of the uploaded file, prefixed with `sha256:` |
110
+
| `backedUpAt` | string | ISO 8601 timestamp of when this asset was uploaded |
101
111
102
112
## What's not included
103
113
104
-
- **Adjustment plists** — Apple's non-destructive edit recipes are not portable outside Photos
114
+
- **Adjustment plists** — Apple's non-destructive edit recipes are not portable
115
+
outside Photos
105
116
- **Thumbnail data** — Not useful for a full backup
106
-
- **iCloud sync state** — Only relevant at backup time, not for the archived copy
107
-
- **Face region coordinates** — Only person identity is stored, not bounding boxes
117
+
- **iCloud sync state** — Only relevant at backup time, not for the archived
118
+
copy
119
+
- **Face region coordinates** — Only person identity is stored, not bounding
120
+
boxes
108
121
- **Slo-mo / Live Photo markers** — Deferred to a future phase
+457
docs/plans/2026-03-13-feat-backup-rendered-edits-plan.md
+457
docs/plans/2026-03-13-feat-backup-rendered-edits-plan.md
···
1
+
---
2
+
title: "feat: Back Up Rendered Edits Alongside Originals"
3
+
type: feat
4
+
status: active
5
+
date: 2026-03-13
6
+
brainstorm: docs/brainstorms/2026-03-13-edited-assets-backup-brainstorm.md
7
+
---
8
+
9
+
# Back Up Rendered Edits Alongside Originals
10
+
11
+
## Overview
12
+
13
+
Extend the backup pipeline to detect edited photos/videos and upload the
14
+
rendered (fullsize) version alongside the original to S3. Also detect edits on
15
+
already-backed-up assets and re-edits that produce newer renders. This makes the
16
+
backup self-contained and viewable without Apple Photos.
17
+
18
+
**Scope:** 1,312 edited assets out of 37,289 total (~3.5%), adding ~13.7 GB of
19
+
rendered files.
20
+
21
+
## Problem Statement
22
+
23
+
The backup currently stores only originals. Apple Photos edits are
24
+
non-destructive — the original is preserved, but the "finished" version requires
25
+
Apple Photos (or the adjustment plist) to re-render. If Photos.app is lost, the
26
+
user has the raw originals but not the edited versions they actually curated.
27
+
28
+
## Proposed Solution
29
+
30
+
Three-phase implementation that progresses from detection (metadata-only)
31
+
through ladder protocol changes to full rendered file backup.
32
+
33
+
## Design Decisions
34
+
35
+
| Decision | Choice | Rationale |
36
+
| --------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------- |
37
+
| S3 key layout | `originals/{y}/{m}/{uuid}_edited.{ext}` sibling | Easy to discover, clearly paired |
38
+
| Manifest schema | Extend existing entry with optional edit fields | No schema break, simple comparison |
39
+
| Rendered file extension | From the rendered resource's UTI, not the original's | A RAW `.orf` edited in Photos renders as HEIC |
40
+
| Re-edit handling | Overwrite same `_edited` S3 key | S3 bucket versioning provides history if needed |
41
+
| Edit revert | Leave S3 file, clear manifest edit fields | Safe, cheap, no data loss |
42
+
| Slo-mo / Live Photos | Defer to follow-up | 47 slo-mo + Live Photos have special resource types |
43
+
| Partial failure (original OK, rendered fails) | Back up original, retry rendered next run | Progressive backup, no data loss |
44
+
| Visual vs metadata-only edits | Check for rendered resource existence in ZINTERNALRESOURCE | Not all adjustments produce a visible render |
45
+
46
+
## Implementation Phases
47
+
48
+
### Phase 1: Edit Detection and Metadata (TypeScript only)
49
+
50
+
Add edit awareness to the reader and metadata JSON without exporting rendered
51
+
files. This is independently useful and requires no ladder changes.
52
+
53
+
#### 1.1 Extend `PhotoAsset` type — `shared/types.ts`
54
+
55
+
Add three fields:
56
+
57
+
```typescript
58
+
hasEdit: boolean;
59
+
editedAt: Date | null;
60
+
editor: string | null;
61
+
```
62
+
63
+
Export nothing new from `shared/mod.ts` (these are primitive types on the
64
+
existing interface).
65
+
66
+
#### 1.2 Add edit enrichment query — `cli/src/photos-db/reader.ts`
67
+
68
+
New `buildEditMap()` following the existing enrichment pattern:
69
+
70
+
```sql
71
+
SELECT aa.ZASSET, ua.ZADJUSTMENTTIMESTAMP, ua.ZADJUSTMENTFORMATIDENTIFIER
72
+
FROM ZADDITIONALASSETATTRIBUTES aa
73
+
JOIN ZUNMANAGEDADJUSTMENT ua ON aa.ZUNMANAGEDADJUSTMENT = ua.Z_PK
74
+
WHERE aa.ZUNMANAGEDADJUSTMENT IS NOT NULL
75
+
```
76
+
77
+
Returns `Map<number, { editedAt: Date; editor: string }>` keyed by asset Z_PK.
78
+
Uses `safeQuery` for schema resilience. `ZADJUSTMENTTIMESTAMP` is CoreData
79
+
format — convert with existing `coreDataTimestampToDate()`.
80
+
81
+
Update `rowToAsset()` to merge edit data:
82
+
83
+
- `hasEdit`: `editMap.has(pk)`
84
+
- `editedAt`: from map or `null`
85
+
- `editor`: from `ZADJUSTMENTFORMATIDENTIFIER` or `null`
86
+
87
+
#### 1.3 Add rendered resource metadata query — `cli/src/photos-db/reader.ts`
88
+
89
+
Detect whether a rendered file actually exists (distinguishes visual edits from
90
+
metadata-only adjustments):
91
+
92
+
```sql
93
+
SELECT ir.ZASSET, ir.ZDATALENGTH, ir.ZCOMPACTUTI, ir.ZLOCALAVAILABILITY
94
+
FROM ZINTERNALRESOURCE ir
95
+
WHERE ir.ZRESOURCETYPE = 1
96
+
AND ir.ZTRASHEDSTATE = 0
97
+
AND ir.ZVERSION != 0
98
+
```
99
+
100
+
Returns `Map<number, { renderSize: number; renderLocallyAvailable: boolean }>`.
101
+
An asset `hasEdit = true` only if it appears in BOTH the adjustment map AND the
102
+
rendered resource map. This prevents trying to export rendered versions that
103
+
don't exist.
104
+
105
+
#### 1.4 Update metadata JSON — `cli/src/commands/backup.ts`
106
+
107
+
Add to `AssetMetadata`:
108
+
109
+
```typescript
110
+
hasEdit: boolean;
111
+
editedAt: string | null;
112
+
editor: string | null;
113
+
```
114
+
115
+
Simple pass-through from `PhotoAsset` in `buildMetadataJson()`.
116
+
117
+
#### 1.5 Update scan report — `cli/src/commands/scan.ts`
118
+
119
+
Add a line to `printScanReport()`:
120
+
121
+
```
122
+
Edited: 1,312 (rendered: 1,180 local, 132 iCloud-only)
123
+
```
124
+
125
+
#### 1.6 Tests — `cli/src/photos-db/reader.test.ts`
126
+
127
+
- Extend `createTestDb()` with `ZUNMANAGEDADJUSTMENT` and `ZINTERNALRESOURCE`
128
+
tables
129
+
- Assert `hasEdit`, `editedAt`, `editor` on photo with edit data
130
+
- Assert `hasEdit: false` on video without edit data
131
+
- Schema resilience: missing adjustment tables return `hasEdit: false`
132
+
133
+
**Files modified:**
134
+
135
+
| File | Change | ~Lines |
136
+
| ---------------------------------- | ------------------------------------------------------------- | ------ |
137
+
| `shared/types.ts` | +3 fields on PhotoAsset | +3 |
138
+
| `cli/src/photos-db/reader.ts` | +buildEditMap, +buildRenderedResourceMap, merge in rowToAsset | +50 |
139
+
| `cli/src/commands/backup.ts` | +3 fields in AssetMetadata + buildMetadataJson | +6 |
140
+
| `cli/src/commands/scan.ts` | Edit stats in report | +10 |
141
+
| `cli/src/photos-db/reader.test.ts` | Edit enrichment fixtures + assertions | +40 |
142
+
143
+
**Verification:** `deno task check && deno task test && deno task scan` — scan
144
+
output shows edit counts.
145
+
146
+
---
147
+
148
+
### Phase 2: Ladder Protocol Extension (Swift)
149
+
150
+
Extend the ladder binary to export rendered versions via PhotoKit's
151
+
`PHAssetResource` API.
152
+
153
+
#### 2.1 Extend `ExportRequest` JSON schema — ladder
154
+
155
+
Current input format:
156
+
157
+
```json
158
+
{ "uuids": ["UUID/L0/001"], "stagingDir": "/path" }
159
+
```
160
+
161
+
New format — add optional `variant` field per UUID:
162
+
163
+
```json
164
+
{
165
+
"requests": [
166
+
{ "id": "UUID/L0/001", "variant": "original" },
167
+
{ "id": "UUID/L0/001", "variant": "rendered" }
168
+
],
169
+
"stagingDir": "/path"
170
+
}
171
+
```
172
+
173
+
If `variant` is omitted or `"original"`, use current behavior (`.photo` /
174
+
`.video` resource type). If `"rendered"`, use `.fullSizePhoto` /
175
+
`.fullSizeVideo`.
176
+
177
+
#### 2.2 Extend `ExportResult` JSON — ladder
178
+
179
+
Add `variant` to each result:
180
+
181
+
```json
182
+
{
183
+
"uuid": "UUID",
184
+
"variant": "rendered",
185
+
"path": "/staging/UUID_rendered.heic",
186
+
"size": 3158112,
187
+
"sha256": "abc123"
188
+
}
189
+
```
190
+
191
+
The filename in `path` should reflect the actual rendered format (HEIC, JPEG,
192
+
MOV).
193
+
194
+
#### 2.3 PhotoKit resource selection — ladder Swift code
195
+
196
+
For rendered exports, use:
197
+
198
+
```swift
199
+
let resources = PHAssetResource.assetResources(for: asset)
200
+
let rendered = resources.first { $0.type == .fullSizePhoto }
201
+
?? resources.first { $0.type == .fullSizeVideo }
202
+
```
203
+
204
+
If no rendered resource exists, return an error result for that UUID+variant
205
+
(not a crash).
206
+
207
+
#### 2.4 Backward compatibility
208
+
209
+
If ladder receives the old `uuids` format (no `requests` field), fall back to
210
+
current behavior. This allows the TypeScript CLI to be deployed independently of
211
+
the ladder upgrade.
212
+
213
+
#### 2.5 Tests — ladder
214
+
215
+
- Export an edited photo: should return both original and rendered with correct
216
+
variants
217
+
- Export an unedited photo with `variant: "rendered"`: should return an error
218
+
(no fullsize resource)
219
+
- Old-format `uuids` input: should still work
220
+
221
+
**Verification:** Run ladder manually against a known edited asset, confirm both
222
+
variants export correctly.
223
+
224
+
---
225
+
226
+
### Phase 3: Full Rendered Backup Pipeline (TypeScript)
227
+
228
+
Wire the ladder protocol changes into the backup pipeline.
229
+
230
+
#### 3.1 S3 path helper — `shared/s3-paths.ts`
231
+
232
+
New exported function:
233
+
234
+
```typescript
235
+
export function editedKey(
236
+
uuid: string,
237
+
dateCreated: Date | null,
238
+
ext: string,
239
+
): string;
240
+
```
241
+
242
+
Same structure as `originalKey()` but appends `_edited` before the extension.
243
+
Same UUID/extension regex validation. Tests follow existing patterns in
244
+
`s3-paths.test.ts`.
245
+
246
+
#### 3.2 Update Exporter interface — `cli/src/export/exporter.ts`
247
+
248
+
Extend `ExportedAsset` with variant:
249
+
250
+
```typescript
251
+
interface ExportedAsset {
252
+
uuid: string;
253
+
variant: "original" | "rendered";
254
+
path: string;
255
+
size: number;
256
+
sha256: string;
257
+
}
258
+
```
259
+
260
+
Update `exportBatch()` signature:
261
+
262
+
```typescript
263
+
exportBatch(
264
+
requests: Array<{ uuid: string; variant: "original" | "rendered" }>,
265
+
): Promise<ExportBatchResult>;
266
+
```
267
+
268
+
Update `createLadderExporter()` to send the new JSON format and parse variant
269
+
from results.
270
+
271
+
#### 3.3 Update mock exporter — `cli/src/export/exporter.mock.ts`
272
+
273
+
Mirror the interface changes. Mock data includes variant-tagged assets.
274
+
275
+
#### 3.4 Extend manifest — `cli/src/manifest/manifest.ts`
276
+
277
+
Add optional fields to `ManifestEntry`:
278
+
279
+
```typescript
280
+
interface ManifestEntry {
281
+
uuid: string;
282
+
s3Key: string;
283
+
checksum: string;
284
+
backedUpAt: string;
285
+
editS3Key?: string;
286
+
editChecksum?: string;
287
+
editBackedUpAt?: string;
288
+
}
289
+
```
290
+
291
+
New helpers:
292
+
293
+
```typescript
294
+
function needsEditBackup(
295
+
manifest: Manifest,
296
+
uuid: string,
297
+
editedAt: Date | null,
298
+
): boolean;
299
+
function markEditBackedUp(
300
+
manifest: Manifest,
301
+
uuid: string,
302
+
checksum: string,
303
+
s3Key: string,
304
+
): void;
305
+
```
306
+
307
+
`needsEditBackup()` returns `true` if:
308
+
309
+
- Asset has `hasEdit` and a rendered resource, AND
310
+
- No `editBackedUpAt` in manifest, OR `editedAt > editBackedUpAt`
311
+
312
+
#### 3.5 Update backup pipeline — `cli/src/commands/backup.ts`
313
+
314
+
The backup loop changes to handle three categories per batch:
315
+
316
+
1. **New assets** (not in manifest): export original + rendered (if edited),
317
+
upload both
318
+
2. **Edit-pending assets** (in manifest, but `needsEditBackup()` is true):
319
+
export rendered only, upload, update manifest
320
+
3. **Fully backed up** (in manifest, no pending edit): skip
321
+
322
+
The filtering step becomes:
323
+
324
+
```typescript
325
+
const newAssets = assets.filter((a) => !isBackedUp(manifest, a.uuid));
326
+
const editPending = assets.filter((a) =>
327
+
isBackedUp(manifest, a.uuid) && needsEditBackup(manifest, a.uuid, a.editedAt)
328
+
);
329
+
```
330
+
331
+
For each new asset with `hasEdit`:
332
+
333
+
- Build export requests:
334
+
`[{uuid, variant: "original"}, {uuid, variant: "rendered"}]`
335
+
- Upload original to `originalKey()`, rendered to `editedKey()`
336
+
- `markBackedUp()` + `markEditBackedUp()`
337
+
338
+
For each edit-pending asset:
339
+
340
+
- Build export request: `[{uuid, variant: "rendered"}]`
341
+
- Upload to `editedKey()`, update metadata JSON
342
+
- `markEditBackedUp()`
343
+
344
+
**Partial failure handling:** If original exports OK but rendered fails, upload
345
+
original and mark it in manifest. The edit will be retried on the next run
346
+
(detected by `needsEditBackup()`).
347
+
348
+
#### 3.6 Update verify command — `cli/src/commands/verify.ts`
349
+
350
+
When `entry.editS3Key` is present, also verify it with HEAD (quick mode) or
351
+
checksum (deep mode). Report edit verification separately:
352
+
353
+
```
354
+
Checked 100/100 OK: 98 Missing: 1 Corrupted: 0 Edits OK: 45 Edits Missing: 1
355
+
```
356
+
357
+
#### 3.7 Update status command — `cli/src/commands/status.ts`
358
+
359
+
Add edit backup progress:
360
+
361
+
```
362
+
Backed up: 35,000 (originals)
363
+
Edits backed up: 1,100 / 1,312
364
+
```
365
+
366
+
#### 3.8 Handle edit reverts
367
+
368
+
When `hasEdit` is `false` but manifest has `editS3Key`:
369
+
370
+
- Clear `editS3Key`, `editChecksum`, `editBackedUpAt` from manifest
371
+
- Re-upload metadata JSON (reflecting `hasEdit: false`)
372
+
- Leave the S3 file in place (orphaned, but cheap and safe)
373
+
374
+
#### 3.9 Tests
375
+
376
+
**backup.test.ts:**
377
+
378
+
- New asset with edit: both original + rendered uploaded, manifest has both keys
379
+
- Already-backed-up asset gains edit: only rendered uploaded, manifest updated
380
+
- Re-edited asset: rendered re-uploaded, manifest timestamp updated
381
+
- Edit reverted: manifest edit fields cleared, metadata re-uploaded
382
+
- Partial failure: original succeeds, rendered fails — original in manifest,
383
+
edit retried
384
+
385
+
**manifest.test.ts:**
386
+
387
+
- `needsEditBackup()` returns true when no edit backed up
388
+
- `needsEditBackup()` returns true when editedAt > editBackedUpAt
389
+
- `needsEditBackup()` returns false when edit already current
390
+
- `markEditBackedUp()` sets edit fields
391
+
392
+
**s3-paths.test.ts:**
393
+
394
+
- `editedKey()` generates correct path
395
+
- `editedKey()` rejects unsafe UUID/extension
396
+
397
+
**Files modified:**
398
+
399
+
| File | Change | ~Lines |
400
+
| ----------------------------------- | --------------------------------------------------------------- | ------ |
401
+
| `shared/s3-paths.ts` | +editedKey() | +15 |
402
+
| `shared/s3-paths.test.ts` | editedKey tests | +20 |
403
+
| `cli/src/export/exporter.ts` | Variant support in interface + ladder exporter | +30 |
404
+
| `cli/src/export/exporter.mock.ts` | Mirror variant changes | +15 |
405
+
| `cli/src/manifest/manifest.ts` | +ManifestEntry edit fields, +needsEditBackup, +markEditBackedUp | +30 |
406
+
| `cli/src/manifest/manifest.test.ts` | Edit-aware manifest tests | +30 |
407
+
| `cli/src/commands/backup.ts` | Edit-aware pipeline + revert handling | +60 |
408
+
| `cli/src/commands/backup.test.ts` | Edit backup scenarios | +80 |
409
+
| `cli/src/commands/verify.ts` | Verify editS3Key | +15 |
410
+
| `cli/src/commands/status.ts` | Edit backup stats | +10 |
411
+
412
+
## Out of Scope (Future Work)
413
+
414
+
- **Slo-mo videos**: 47 assets with special resource types — needs dedicated
415
+
investigation
416
+
- **Live Photos**: Paired still + video resources; edit may affect one or both
417
+
- **Adjustment plist backup**: The non-destructive recipe; low portability
418
+
outside Apple Photos
419
+
- **Thumbnail/preview backup**: Other resource types (3, 14) not needed for a
420
+
"viewable backup"
421
+
- **S3 cleanup of orphaned edit files**: Could add a `prune` command later
422
+
423
+
## Acceptance Criteria
424
+
425
+
### Phase 1
426
+
427
+
- [ ] `deno task scan` shows edit count and rendered resource availability
428
+
- [ ] Metadata JSON for edited assets includes `hasEdit`, `editedAt`, `editor`
429
+
- [ ] All existing tests pass; new tests cover edit enrichment + schema
430
+
resilience
431
+
432
+
### Phase 2
433
+
434
+
- [ ] Ladder exports both original and rendered for an edited asset
435
+
- [ ] Ladder handles missing rendered resource gracefully (error, not crash)
436
+
- [ ] Old-format input still works (backward compat)
437
+
438
+
### Phase 3
439
+
440
+
- [ ] Edited assets get `_edited.{ext}` sibling uploaded to S3
441
+
- [ ] Already-backed-up assets with new edits get rendered version uploaded
442
+
- [ ] Re-edits detected and re-uploaded
443
+
- [ ] Reverted edits clear manifest edit fields
444
+
- [ ] Verify checks both original and edit S3 keys
445
+
- [ ] Status shows edit backup progress
446
+
- [ ] Partial failures (original OK, rendered fails) handled gracefully
447
+
448
+
## References
449
+
450
+
- Brainstorm: `docs/brainstorms/2026-03-13-edited-assets-backup-brainstorm.md`
451
+
- Current reader enrichment pattern: `cli/src/photos-db/reader.ts:130-201`
452
+
- Current S3 path helpers: `shared/s3-paths.ts`
453
+
- Current manifest schema: `cli/src/manifest/manifest.ts:5-15`
454
+
- Current backup pipeline: `cli/src/commands/backup.ts:61-243`
455
+
- Current exporter: `cli/src/export/exporter.ts`
456
+
- PhotoKit resource types: `PHAssetResourceType.fullSizePhoto` (type 3),
457
+
`.fullSizeVideo` (type 5)
+106
-40
docs/plans/2026-03-13-feat-ux-open-source-readiness-plan.md
+106
-40
docs/plans/2026-03-13-feat-ux-open-source-readiness-plan.md
···
9
9
10
10
## Overview
11
11
12
-
Replace hardcoded Scaleway configuration with a generic S3-compatible config layer, add an interactive `attic init` command, migrate CLI to Cliffy, and improve error messages. Makes attic usable with any S3-compatible provider and ready to open source.
12
+
Replace hardcoded Scaleway configuration with a generic S3-compatible config
13
+
layer, add an interactive `attic init` command, migrate CLI to Cliffy, and
14
+
improve error messages. Makes attic usable with any S3-compatible provider and
15
+
ready to open source.
13
16
14
17
## Problem Statement
15
18
16
-
Attic has Scaleway details baked into the code — endpoint, region, keychain service names, type names. A user who wants to use Hetzner, OVH, or AWS must fork and edit constants. The CLI uses hand-rolled arg parsing (130+ lines in `cli/mod.ts`) with no help generation, no colored output, and no shell completions. Error messages are raw exceptions in some paths.
19
+
Attic has Scaleway details baked into the code — endpoint, region, keychain
20
+
service names, type names. A user who wants to use Hetzner, OVH, or AWS must
21
+
fork and edit constants. The CLI uses hand-rolled arg parsing (130+ lines in
22
+
`cli/mod.ts`) with no help generation, no colored output, and no shell
23
+
completions. Error messages are raw exceptions in some paths.
17
24
18
25
## Proposed Solution
19
26
···
32
39
Add config file support at `~/.attic/config.json`.
33
40
34
41
**Files to modify:**
42
+
35
43
- New: `cli/src/config/config.ts` (~80 lines)
36
44
- New: `cli/src/config/config.test.ts` (~60 lines)
37
45
···
69
77
const CONFIG_PATH = join(CONFIG_DIR, "config.json");
70
78
71
79
/** Load and validate config. Returns null if file doesn't exist. */
72
-
export function loadConfig(): AtticConfig | null
80
+
export function loadConfig(): AtticConfig | null;
73
81
74
82
/** Validate config fields, throw with specific message on missing/invalid. */
75
-
export function validateConfig(raw: unknown): AtticConfig
83
+
export function validateConfig(raw: unknown): AtticConfig;
76
84
77
85
/** Write config to disk, creating ~/.attic/ if needed. */
78
-
export function writeConfig(config: AtticConfig): void
86
+
export function writeConfig(config: AtticConfig): void;
79
87
```
80
88
81
89
**Validation rules:**
90
+
82
91
- `endpoint` — required, must start with `https://`
83
92
- `region` — required, non-empty string
84
-
- `bucket` — required, non-empty string, validated against `/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/`
93
+
- `bucket` — required, non-empty string, validated against
94
+
`/^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/`
85
95
- `pathStyle` — optional, defaults to `true`
86
96
- `keychain.accessKeyService` — optional, defaults to `"attic-s3-access-key"`
87
97
- `keychain.secretKeyService` — optional, defaults to `"attic-s3-secret-key"`
88
98
89
99
**Tests:**
100
+
90
101
- Valid config round-trips through write/load
91
102
- Missing required fields throw descriptive errors
92
103
- Optional fields get defaults
···
99
110
Remove Scaleway-specific naming. Parameterize the S3 client.
100
111
101
112
**Files to modify:**
113
+
102
114
- `cli/src/storage/s3-client.ts` (~20 lines changed)
103
115
- `cli/mod.ts` (~15 lines changed)
104
-
- `deno.json` (~2 lines changed — remove `--allow-net=s3.fr-par.scw.cloud`, use broader net permission)
116
+
- `deno.json` (~2 lines changed — remove `--allow-net=s3.fr-par.scw.cloud`, use
117
+
broader net permission)
105
118
106
119
**Changes:**
107
120
···
131
144
): S3Provider
132
145
```
133
146
134
-
- `loadKeychainCredentials()` accepts service names as parameters instead of hardcoding them
147
+
- `loadKeychainCredentials()` accepts service names as parameters instead of
148
+
hardcoding them
135
149
- Delete `SCALEWAY_ENDPOINT` and `SCALEWAY_REGION` constants
136
-
- `cli/mod.ts` reads config and passes connection details to `createS3Provider()`
137
-
- `deno.json` tasks: replace `--allow-net=s3.fr-par.scw.cloud` with `--allow-net` (endpoint is now configurable)
150
+
- `cli/mod.ts` reads config and passes connection details to
151
+
`createS3Provider()`
152
+
- `deno.json` tasks: replace `--allow-net=s3.fr-par.scw.cloud` with
153
+
`--allow-net` (endpoint is now configurable)
138
154
139
-
**Migration for existing users:** scan/status continue working without config (they don't need S3). backup/verify check for config and fail fast:
155
+
**Migration for existing users:** scan/status continue working without config
156
+
(they don't need S3). backup/verify check for config and fail fast:
140
157
141
158
```
142
159
Error: No config file found at ~/.attic/config.json
···
149
166
Replace hand-rolled arg parsing with Cliffy.
150
167
151
168
**Dependencies to add (JSR):**
169
+
152
170
- `@cliffy/command@1.0.0`
153
171
- `@cliffy/prompt@1.0.0`
154
172
- `@cliffy/ansi@1.0.0`
155
173
156
174
**Files to modify:**
175
+
157
176
- `cli/mod.ts` — full rewrite (~120 lines, replaces 252 lines)
158
177
- `cli/deno.json` — add Cliffy imports
159
178
···
201
220
```
202
221
203
222
**What this gives us:**
223
+
204
224
- Auto-generated `--help` for every command
205
225
- Typed flags with validation (`:integer`, `:string`)
206
226
- Unknown flag detection
207
227
- Version flag (`--version`)
208
-
- Shell completions via `main.command("completions", ...).action(completeCommand)`
228
+
- Shell completions via
229
+
`main.command("completions", ...).action(completeCommand)`
209
230
210
231
**What we delete:**
232
+
211
233
- `parseBackupFlags()` (~55 lines)
212
234
- `parseVerifyFlags()` (~30 lines)
213
235
- `requireArg()`, `parsePositiveInt()` (~15 lines)
···
218
240
Add `attic init` command with interactive prompts.
219
241
220
242
**Files to modify:**
243
+
221
244
- New: `cli/src/commands/init.ts` (~120 lines)
222
245
- `cli/mod.ts` — wire up init command
223
246
···
261
284
262
285
```typescript
263
286
// cli/src/commands/init.ts
264
-
import { Input, Confirm, Secret } from "@cliffy/prompt";
287
+
import { Confirm, Input, Secret } from "@cliffy/prompt";
265
288
import { colors } from "@cliffy/ansi";
266
289
267
290
export async function runInit(): Promise<void> {
···
275
298
const endpoint = await Input.prompt({ message: "Endpoint URL", hint: "..." });
276
299
const region = await Input.prompt({ message: "Region" });
277
300
const bucket = await Input.prompt({ message: "Bucket name" });
278
-
const pathStyle = await Confirm.prompt({ message: "Use path-style URLs?", default: true });
301
+
const pathStyle = await Confirm.prompt({
302
+
message: "Use path-style URLs?",
303
+
default: true,
304
+
});
279
305
280
306
const accessKey = await Input.prompt({ message: "Access key" });
281
307
const secretKey = await Secret.prompt({ message: "Secret key" });
282
308
283
309
// Write config
284
-
writeConfig({ endpoint, region, bucket, pathStyle, keychain: {
285
-
accessKeyService: "attic-s3-access-key",
286
-
secretKeyService: "attic-s3-secret-key",
287
-
}});
310
+
writeConfig({
311
+
endpoint,
312
+
region,
313
+
bucket,
314
+
pathStyle,
315
+
keychain: {
316
+
accessKeyService: "attic-s3-access-key",
317
+
secretKeyService: "attic-s3-secret-key",
318
+
},
319
+
});
288
320
289
321
// Store credentials with -U flag (update if exists)
290
322
await storeKeychainCredential("attic-s3-access-key", accessKey);
291
323
await storeKeychainCredential("attic-s3-secret-key", secretKey);
292
324
}
293
325
294
-
async function storeKeychainCredential(service: string, value: string): Promise<void> {
326
+
async function storeKeychainCredential(
327
+
service: string,
328
+
value: string,
329
+
): Promise<void> {
295
330
// Try update first, fall back to add
296
331
const update = new Deno.Command("security", {
297
-
args: ["add-generic-password", "-U", "-s", service, "-a", "attic", "-w", value],
332
+
args: [
333
+
"add-generic-password",
334
+
"-U",
335
+
"-s",
336
+
service,
337
+
"-a",
338
+
"attic",
339
+
"-w",
340
+
value,
341
+
],
298
342
stderr: "piped",
299
343
});
300
344
const { code } = await update.output();
301
345
if (code !== 0) {
302
-
throw new Error(`Failed to store credential in Keychain for service "${service}"`);
346
+
throw new Error(
347
+
`Failed to store credential in Keychain for service "${service}"`,
348
+
);
303
349
}
304
350
}
305
351
```
306
352
307
-
**Keychain idempotency:** Use `security add-generic-password -U` which updates an existing entry or creates a new one. No need to delete-then-add.
353
+
**Keychain idempotency:** Use `security add-generic-password -U` which updates
354
+
an existing entry or creates a new one. No need to delete-then-add.
308
355
309
-
**No test file for init** — it's pure I/O (prompts + Keychain + file writes). The config validation is tested in Phase 1. Keychain interaction is tested manually.
356
+
**No test file for init** — it's pure I/O (prompts + Keychain + file writes).
357
+
The config validation is tested in Phase 1. Keychain interaction is tested
358
+
manually.
310
359
311
360
### Phase 5: Error Boundary
312
361
313
362
Add a top-level error handler in `cli/mod.ts`.
314
363
315
364
**Files to modify:**
365
+
316
366
- `cli/mod.ts` (~40 lines added)
317
367
318
368
**Implementation:**
···
335
385
const msg = error.message;
336
386
337
387
// Keychain not found
338
-
if (msg.includes("find-generic-password") || msg.includes("SecKeychainSearchCopyNext")) {
388
+
if (
389
+
msg.includes("find-generic-password") ||
390
+
msg.includes("SecKeychainSearchCopyNext")
391
+
) {
339
392
console.error("Could not read credentials from macOS Keychain.");
340
393
console.error('Run "attic init" to set up your credentials.\n');
341
394
return;
···
350
403
351
404
// S3 access denied
352
405
if (msg.includes("AccessDenied") || msg.includes("403")) {
353
-
console.error("S3 access denied. Check your credentials and bucket permissions.");
406
+
console.error(
407
+
"S3 access denied. Check your credentials and bucket permissions.",
408
+
);
354
409
console.error("Your credentials are stored in macOS Keychain.");
355
410
console.error('Run "attic init" to update them.\n');
356
411
return;
···
358
413
359
414
// S3 bucket not found
360
415
if (msg.includes("NoSuchBucket") || msg.includes("404")) {
361
-
console.error(`S3 bucket not found. Check the bucket name in ~/.attic/config.json`);
416
+
console.error(
417
+
`S3 bucket not found. Check the bucket name in ~/.attic/config.json`,
418
+
);
362
419
return;
363
420
}
364
421
365
422
// Network error
366
-
if (msg.includes("ECONNREFUSED") || msg.includes("ETIMEDOUT") || msg.includes("fetch failed")) {
367
-
console.error("Could not connect to S3 endpoint. Check your network and endpoint URL.");
423
+
if (
424
+
msg.includes("ECONNREFUSED") || msg.includes("ETIMEDOUT") ||
425
+
msg.includes("fetch failed")
426
+
) {
427
+
console.error(
428
+
"Could not connect to S3 endpoint. Check your network and endpoint URL.",
429
+
);
368
430
return;
369
431
}
370
432
371
433
// Photos.sqlite not found
372
434
if (msg.includes("Photos.sqlite") || msg.includes("no such file")) {
373
435
console.error("Could not open Photos database.");
374
-
console.error("Make sure Photos is set up on this Mac and the database exists.");
436
+
console.error(
437
+
"Make sure Photos is set up on this Mac and the database exists.",
438
+
);
375
439
return;
376
440
}
377
441
···
382
446
383
447
## Files Summary
384
448
385
-
| Phase | File | Change |
386
-
|-------|------|--------|
387
-
| 1 | `cli/src/config/config.ts` | New — config load/validate/write |
388
-
| 1 | `cli/src/config/config.test.ts` | New — config tests |
389
-
| 2 | `cli/src/storage/s3-client.ts` | Rename types, parameterize |
390
-
| 2 | `cli/mod.ts` | Read config, pass to S3 |
391
-
| 2 | `deno.json` | Broader net permission |
392
-
| 3 | `cli/mod.ts` | Rewrite with Cliffy |
393
-
| 3 | `cli/deno.json` | Add Cliffy imports |
394
-
| 4 | `cli/src/commands/init.ts` | New — interactive setup |
395
-
| 5 | `cli/mod.ts` | Error boundary wrapper |
449
+
| Phase | File | Change |
450
+
| ----- | ------------------------------- | -------------------------------- |
451
+
| 1 | `cli/src/config/config.ts` | New — config load/validate/write |
452
+
| 1 | `cli/src/config/config.test.ts` | New — config tests |
453
+
| 2 | `cli/src/storage/s3-client.ts` | Rename types, parameterize |
454
+
| 2 | `cli/mod.ts` | Read config, pass to S3 |
455
+
| 2 | `deno.json` | Broader net permission |
456
+
| 3 | `cli/mod.ts` | Rewrite with Cliffy |
457
+
| 3 | `cli/deno.json` | Add Cliffy imports |
458
+
| 4 | `cli/src/commands/init.ts` | New — interactive setup |
459
+
| 5 | `cli/mod.ts` | Error boundary wrapper |
396
460
397
461
## Verification
398
462
399
463
After each phase:
464
+
400
465
1. `deno task check` — type checking
401
466
2. `deno task test` — all tests pass
402
467
3. `deno task lint` — no lint errors
403
468
404
469
Integration test (after all phases):
470
+
405
471
1. Run `attic init` with a test bucket
406
472
2. Run `attic scan` — works without config
407
473
3. Run `attic backup --dry-run` — reads config, validates, shows plan