Keep using Photos.app like you always do. Attic quietly backs up your originals and edits to an S3 bucket you control. One-way, append-only.
docs: refresh README, CLAUDE.md, and architecture for 1.0.0-beta.7
Covers adaptive iCloud-lane throttling (AIMDController), retry queue,
unavailable-asset store, network pause/resume, staging reuse, the shift
from aws-sdk-swift to URLSessionS3Client + aws-signer-v4, and the
AtticCore-as-SPM-library story for the future menu bar app.
+185
-68
+41
-14
CLAUDE.md
+41
-14
CLAUDE.md
···
27
27
28
28
Swift package with three targets:
29
29
30
-
- `AtticCore` — shared library: S3 provider, manifest, config, keychain,
31
-
metadata, backup/verify/refresh pipelines. Used by both CLI and menu bar app.
32
-
- `AtticCLI` — executable: ArgumentParser commands, terminal renderer
33
-
- `AtticCoreTests` — tests using Swift Testing framework
30
+
- `AtticCore` — shared library (public product): S3 client, manifest, config,
31
+
keychain, metadata, backup/verify/refresh pipelines, AIMD concurrency
32
+
controller, retry queue, unavailable-asset store, network monitor, viewer
33
+
data store, thumbnailing. Designed for reuse by the CLI and a future macOS
34
+
menu bar app — both consume `AtticCore` as an SPM library.
35
+
- `AtticCLI` — executable: ArgumentParser commands, terminal dashboard,
36
+
Hummingbird-based viewer server, `LadderKitExportProvider` bridge.
37
+
- `AtticCoreTests` — tests using Swift Testing framework (178 tests).
34
38
35
-
Dependencies: `aws-sdk-swift` (AWSS3), `swift-argument-parser`, `LadderKit`
36
-
(path dependency from `../ladder`).
39
+
Dependencies: `aws-signer-v4` (SigV4 signing for `URLSessionS3Client` — no full
40
+
AWS SDK), `swift-argument-parser`, `Hummingbird` (viewer), and `LadderKit`
41
+
pinned to `0.5.1+` via `https://github.com/tijs/ladder.git`.
37
42
38
43
Platform: macOS 14+, Swift 6.x, Apple Silicon only.
39
44
40
45
## Architecture
41
46
42
47
The backup pipeline:
43
-
`Photos Library → LadderKit (PhotoKit + enrichment) → AssetInfo[] → BackupPipeline → S3 upload → manifest update`
48
+
`Photos Library → LadderKit (PhotoKit + enrichment) → AssetInfo[] → BackupPipeline → (adaptive export + upload) → manifest update`
44
49
45
50
- **LadderKit** provides `PhotoLibrary` (PhotoKit), `PhotosDatabase`
46
-
(Photos.sqlite enrichment), and `PhotoExporter` (export with SHA-256 hashing +
47
-
AppleScript fallback for iCloud-only assets). Called directly as a library.
51
+
(Photos.sqlite enrichment), `PhotoExporter` (export + SHA-256 + AppleScript
52
+
fallback), `LocalAvailabilityProviding` (local vs. iCloud split), and the
53
+
`AdaptiveConcurrencyControlling` protocol.
54
+
- **AIMDController** (`Sources/AtticCore/AIMDController.swift`) is attic's
55
+
implementation of the adaptive controller. Observation-only: the exporter
56
+
polls `currentLimit()` and reports `ExportOutcome` via `record(_:)`. Policy
57
+
is additive-increase / multiplicative-decrease over a sliding 20-outcome
58
+
window. Permanent failures (`.permanentlyUnavailable`) are ignored — not a
59
+
lane-health signal. Local-only assets run at full `maxConcurrency`; iCloud
60
+
assets are gated by the controller.
61
+
- **RetryQueue** (`retry-queue.json` on S3) — UUIDs of assets that failed in a
62
+
previous run, with attempts/classification/first-seen/last-seen/message per
63
+
entry. Merged across runs; unattempted entries are preserved when `--limit`
64
+
cuts a run short. Retried first on the next run.
65
+
- **UnavailableStore** (`unavailable-assets.json` on S3) — shared-album or
66
+
otherwise permanently-unreachable assets. Never auto-cleared — skip-forever.
48
67
- **S3 key format** — originals: `originals/{year}/{month}/{uuid}.{ext}`,
49
-
metadata: `metadata/assets/{uuid}.json`
68
+
metadata: `metadata/assets/{uuid}.json`.
50
69
- **Manifest** (`manifest.json` on S3) — maps UUID →
51
-
`{ s3Key, checksum, backedUpAt }`. S3 is the single source of truth. Saved to
52
-
S3 every 50 assets during backup.
70
+
`{ s3Key, checksum, backedUpAt }`. S3 is the single source of truth. Saved
71
+
at batch boundaries during backup.
72
+
- **Network pause/resume** — `NetworkMonitoring` (backed by `NWPath`) detects
73
+
network loss mid-upload; the upload loop waits and resumes from the retry
74
+
set.
75
+
- **Staging reuse** — `StagingReclaim` finds previously-exported files on disk
76
+
and reuses them instead of re-exporting (saves PhotoKit round-trips on
77
+
resume).
53
78
54
79
All external dependencies are behind protocols (`S3Providing`, `ManifestStoring`,
55
-
`ConfigProviding`, `KeychainProviding`, `ExportProviding`) for testability.
80
+
`ConfigProviding`, `KeychainProviding`, `ExportProviding`, `NetworkMonitoring`,
81
+
`ThumbnailProviding`) for testability.
56
82
57
83
## CLI Commands
58
84
···
70
96
71
97
Tests use mock implementations — never external services or credentials:
72
98
73
-
- `MockS3Provider` — in-memory `[String: Data]`
99
+
- `MockS3Provider` — in-memory actor-backed `[String: Data]` (ships in `AtticCore`)
74
100
- `MockExportProvider` — returns canned export results
75
101
- `TimeoutExportProvider` — simulates batch timeouts + deferred retry
102
+
- `MockNetworkMonitor` — simulates network up/down for pause-resume tests
76
103
77
104
Uses Swift Testing framework (`@Test`, `#expect`, `@Suite`).
78
105
+45
-14
README.md
+45
-14
README.md
···
84
84
85
85
### status
86
86
87
-
Compare the Photos library against the S3 manifest to show how many assets are
88
-
backed up vs pending.
87
+
Compare the Photos library against the S3 manifest. Shows assets backed up vs
88
+
pending, broken down by local-cache vs iCloud-only lane, and a retry-queue
89
+
summary (count, max attempts, oldest first-failed timestamp).
89
90
90
91
```bash
91
92
attic status
···
107
108
| `--type photo\|video` | Only back up photos or videos |
108
109
109
110
During a backup, a live-updating terminal dashboard shows progress, speed,
110
-
current file, and elapsed time. Non-TTY output (pipes, CI) falls back to
111
-
line-by-line progress.
111
+
current file, elapsed time, and the adaptive iCloud-lane concurrency limit.
112
+
Non-TTY output (pipes, CI) falls back to line-by-line progress.
113
+
114
+
Attic is crash- and network-resilient:
115
+
116
+
- **Adaptive iCloud throttling** — local-cache and iCloud-only exports run
117
+
in separate lanes. The iCloud lane uses an AIMD controller (attic's
118
+
`AIMDController` implementing LadderKit's `AdaptiveConcurrencyControlling`)
119
+
to back off when Photos.app or iCloud pushes back, and to ramp up on a
120
+
clean lane.
121
+
- **Retry queue** — transient failures are remembered on S3
122
+
(`retry-queue.json`) and retried first on the next run, carrying
123
+
attempts/first-seen/last-message for each UUID.
124
+
- **Permanent-unavailable store** — shared-album assets whose derivative has
125
+
gone server-side (Photos.app error `-1728`) are recorded in
126
+
`unavailable-assets.json` and skipped on subsequent runs.
127
+
- **Network pause/resume** — loss of connectivity pauses uploads instead of
128
+
failing them. The manifest is saved before waiting so a long outage doesn't
129
+
lose progress.
130
+
- **Staging reuse** — exported files left behind by an aborted run are
131
+
re-used on the next run instead of re-exported from Photos.app.
112
132
113
133
### verify
114
134
···
197
217
198
218
The project is a Swift package with three targets:
199
219
200
-
- **AtticCore** — shared library: S3 provider, manifest, config, keychain,
201
-
metadata, backup/verify/refresh pipelines. Designed for reuse by both the CLI
202
-
and a planned macOS menu bar app.
203
-
- **AtticCLI** — executable: ArgumentParser commands, terminal renderer
204
-
- **AtticCoreTests** — tests using Swift Testing framework
220
+
- **AtticCore** — shared library (public SPM product): S3 client
221
+
(`URLSessionS3Client`, SigV4 via `aws-signer-v4` — no full AWS SDK),
222
+
manifest, config, keychain, metadata, backup/verify/refresh pipelines,
223
+
`AIMDController` (adaptive concurrency), `RetryQueue`, `UnavailableStore`,
224
+
`NWPathNetworkMonitor`, viewer data store, and thumbnailing. Consumed by
225
+
the CLI and designed for reuse by a future macOS menu bar app.
226
+
- **AtticCLI** — executable: ArgumentParser commands, terminal dashboard,
227
+
Hummingbird-based viewer server, `LadderKitExportProvider` bridge.
228
+
- **AtticCoreTests** — 178 tests using the Swift Testing framework.
205
229
206
230
All external dependencies are behind protocols (`S3Providing`, `ManifestStoring`,
207
-
`ConfigProviding`, `KeychainProviding`, `ExportProviding`) for testability.
231
+
`ConfigProviding`, `KeychainProviding`, `ExportProviding`, `NetworkMonitoring`,
232
+
`ThumbnailProviding`) for testability.
208
233
209
234
## Development
210
235
···
220
245
221
246
## Dependencies
222
247
223
-
- [LadderKit](https://github.com/tijs/ladder) — PhotoKit access, Photos.sqlite
224
-
enrichment, and photo export with AppleScript fallback
225
-
- [aws-sdk-swift](https://github.com/awslabs/aws-sdk-swift) — S3 client
248
+
- [LadderKit](https://github.com/tijs/ladder) (≥ 0.5.1) — PhotoKit access,
249
+
Photos.sqlite enrichment, photo export with AppleScript fallback,
250
+
local/iCloud lane partitioning, and the
251
+
`AdaptiveConcurrencyControlling` protocol.
252
+
- [aws-signer-v4](https://github.com/adam-fowler/aws-signer-v4) — SigV4
253
+
request signing. Attic ships a URLSession-based S3 client instead of the
254
+
full AWS SDK (smaller binary, fewer transitive deps).
226
255
- [swift-argument-parser](https://github.com/apple/swift-argument-parser) —
227
-
CLI command parsing
256
+
CLI command parsing.
257
+
- [Hummingbird](https://github.com/hummingbird-project/hummingbird) — HTTP
258
+
server for `attic viewer`.
228
259
229
260
## Documentation
230
261
+99
-40
docs/architecture.md
+99
-40
docs/architecture.md
···
63
63
64
64
## The backup pipeline
65
65
66
-
`BackupPipeline.swift` orchestrates the full flow: filter → batch → export →
67
-
upload → manifest.
66
+
`BackupPipeline.swift` orchestrates the full flow: filter → staging reuse →
67
+
adaptive export → upload (with network pause/resume) → manifest.
68
+
69
+
Internally `runBackup` decomposes into `filterPending`, `exportBatchWithFallback`,
70
+
the upload loop (`BackupUpload.swift`), and `finalizeBackup`. Each reads
71
+
top-to-bottom — no hidden state threaded through shared mutables.
68
72
69
73
### 1. Filter
70
74
71
-
Assets are filtered against the manifest to find pending work. Optional filters
72
-
narrow by type (`--type photo|video`) or count (`--limit N`). Dry run mode stops
73
-
here.
75
+
`filterPending` combines four inputs:
74
76
75
-
### 2. Batch and export
77
+
1. Library assets from LadderKit.
78
+
2. Current manifest (already-backed-up UUIDs → skip).
79
+
3. Retry queue (previous run's failures → retry *first* on this run).
80
+
4. Unavailable store (permanently-unreachable assets → skip forever).
76
81
77
-
Pending assets are processed in batches (default 50). Each batch is exported via
78
-
LadderKit's `PhotoExporter`, which uses PhotoKit to export original files. When
79
-
PhotoKit can't find an asset (typically iCloud-only with Optimize Storage),
80
-
LadderKit falls back to AppleScript via Photos.app, which handles the iCloud
81
-
download transparently.
82
+
Optional filters narrow by type (`--type photo|video`) or count (`--limit N`).
83
+
Dry run stops here.
82
84
83
-
Each export result includes the file path, size, and SHA-256 hash.
85
+
### 2. Staging reuse
84
86
85
-
### 3. Upload
87
+
`StagingReclaim` scans the staging directory for files from a prior aborted run
88
+
and matches them against pending UUIDs. Reclaimed files skip the PhotoKit
89
+
export round-trip — a meaningful speedup on resume.
86
90
87
-
For each exported file, attic:
91
+
### 3. Adaptive export
88
92
89
-
1. Uploads the original to `originals/{year}/{month}/{uuid}.{ext}` using
90
-
memory-mapped I/O to avoid loading entire files into heap
91
-
2. Builds and uploads a metadata JSON to `metadata/assets/{uuid}.json` (see
92
-
`docs/metadata.md`)
93
-
3. Updates the in-memory manifest
94
-
4. Cleans up the staged file
93
+
Pending assets are processed in batches (default 50). Each batch is passed to
94
+
LadderKit's `PhotoExporter`, which:
95
+
96
+
- Partitions the batch into **local** (cached originals) and **iCloud**
97
+
(Optimize Storage) lanes via `LocalAvailabilityProviding`
98
+
(`PhotosDatabaseLocalAvailability` reads `ZINTERNALRESOURCE.ZLOCALAVAILABILITY`).
99
+
- Runs the local lane at full `maxConcurrency`.
100
+
- Runs the iCloud lane at a limit polled from attic's `AIMDController`
101
+
(observation-only; implements LadderKit's `AdaptiveConcurrencyControlling`).
102
+
The controller maintains a sliding 20-outcome window: >30% transient failure
103
+
rate → halve the limit; ≤5% → +1. Permanent failures (`-1728` asset
104
+
unavailable, shared-album tombstones) are ignored as lane-health signals.
105
+
- Falls back to AppleScript via Photos.app when PhotoKit can't find an asset.
106
+
`-1728` errors are classified `.permanentlyUnavailable` and recorded in the
107
+
unavailable store.
108
+
109
+
Each export result includes the file path, size, and SHA-256 hash (computed
110
+
inline during the streaming write — no second pass).
111
+
112
+
### 4. Upload
113
+
114
+
`BackupUpload.swift` runs bounded-concurrency uploads with retry. For each
115
+
exported file, attic:
116
+
117
+
1. Uploads the original to `originals/{year}/{month}/{uuid}.{ext}` via
118
+
`URLSessionS3Client.putObject(key:fileURL:contentType:)`, streaming from
119
+
disk (no memory load).
120
+
2. Builds and uploads metadata JSON to `metadata/assets/{uuid}.json` (see
121
+
`docs/metadata.md`).
122
+
3. Updates the in-memory manifest.
123
+
4. Cleans up the staged file.
95
124
96
125
S3 keys are built from UUID and extension, both validated with regex
97
126
(`/^[A-Za-z0-9._-]+$/` and `/^[a-z0-9]+$/`) to prevent path traversal.
98
-
Extensions are resolved from the asset's UTI via a lookup table, falling back to
99
-
the filename extension.
127
+
Extensions are resolved from the asset's UTI via a lookup table, falling back
128
+
to the filename extension.
100
129
101
-
### 4. Manifest
130
+
**Network pause/resume**: on a network-down error, the upload loop drains the
131
+
current pass, queues the failed inputs, waits for `NetworkMonitoring` to
132
+
report recovery (with timeout), then restarts the pass. Capped by
133
+
`maxPauseRetries`. The manifest is saved before each pause so a long outage
134
+
doesn't lose progress.
102
135
103
-
The manifest is stored on S3 at `manifest.json` in the bucket root, mapping UUID
104
-
to `{ s3Key, checksum, backedUpAt }`. S3 is the single source of truth — there
105
-
is no local manifest file. This enables cross-machine and cross-app (CLI ↔ menu
106
-
bar app) continuity.
136
+
**Retries**: S3 requests use exponential backoff on transient errors
137
+
(timeouts, `ECONNRESET`, etc.). Per-request timeouts scale with body size so
138
+
large video uploads don't trip a dead-connection check.
139
+
140
+
### 5. Manifest
141
+
142
+
The manifest is stored on S3 at `manifest.json` in the bucket root, mapping
143
+
UUID to `{ s3Key, checksum, backedUpAt }`. S3 is the single source of truth —
144
+
no local manifest. This enables cross-machine and cross-app (CLI ↔ future
145
+
menu bar app) continuity.
107
146
108
-
On backup start, the manifest is downloaded from S3. It's saved back to S3
109
-
periodically (every 50 assets by default) for crash resilience, and always at
147
+
On backup start, the manifest is downloaded from S3. It's saved at **batch
148
+
boundaries** (and before network pauses) for crash resilience, and always at
110
149
the end of a run.
111
150
112
151
**Migration**: existing local manifests at `~/.attic/manifest.json` (from the
113
-
Deno CLI) are automatically uploaded to S3 on first run via
152
+
earlier Deno CLI) are uploaded to S3 on first run via
114
153
`loadManifestWithMigration()`.
115
154
116
-
The manifest can be reconstructed from S3 via `attic rebuild`, which reads every
117
-
`metadata/assets/*.json` file and validates UUID format, S3 key pattern, and
118
-
checksum format before accepting an entry.
155
+
The manifest can be reconstructed from S3 via `attic rebuild`, which reads
156
+
every `metadata/assets/*.json` file and validates UUID, S3 key, and checksum
157
+
format before accepting an entry.
158
+
159
+
### 6. Retry queue and unavailable store
160
+
161
+
Two auxiliary JSON files persist failure state across runs:
162
+
163
+
- **`retry-queue.json`** — transient failures. Each entry carries
164
+
`classification`, `attempts`, `firstFailedAt`, `lastFailedAt`, `lastMessage`.
165
+
Merge semantics preserve `firstFailedAt` and increment `attempts`, so the
166
+
UI can surface "stuck for 3 days". Entries attempted-and-succeeded on a
167
+
later run are removed; entries never attempted (e.g. when `--limit` cut
168
+
the run short) survive with their full history.
169
+
- **`unavailable-assets.json`** — `.permanentlyUnavailable` assets
170
+
(typically shared-album derivatives gone server-side). Never auto-cleared.
171
+
Retrying is pointless; only user action (via a future command) clears.
119
172
120
173
## Verification
121
174
···
146
199
147
200
All external dependencies are behind protocols:
148
201
149
-
| Protocol | Real implementation | Mock |
150
-
| ------------------- | ------------------------- | ------------------------------------------ |
151
-
| `S3Providing` | `AWSS3Client` | `MockS3Provider` (in-memory actor) |
152
-
| `ExportProviding` | `LadderKitExportProvider` | `MockExportProvider` / `TimeoutExportProvider` |
153
-
| `ManifestStoring` | `S3ManifestStore` | Uses MockS3Provider |
154
-
| `ConfigProviding` | `FileConfigProvider` | Direct struct construction in tests |
155
-
| `KeychainProviding` | `SecurityKeychain` | Direct struct construction in tests |
202
+
| Protocol | Real implementation | Mock |
203
+
| --------------------------------- | ---------------------------------------- | ---------------------------------------------- |
204
+
| `S3Providing` | `URLSessionS3Client` (SigV4 via aws-signer-v4; no AWS SDK) | `MockS3Provider` (in-memory actor, ships in AtticCore) |
205
+
| `ExportProviding` | `LadderKitExportProvider` (in AtticCLI) | `MockExportProvider` / `TimeoutExportProvider` |
206
+
| `ManifestStoring` | `S3ManifestStore` | Uses `MockS3Provider` |
207
+
| `ConfigProviding` | `FileConfigProvider` | Direct struct construction in tests |
208
+
| `KeychainProviding` | `SecurityKeychain` | Direct struct construction in tests |
209
+
| `NetworkMonitoring` | `NWPathNetworkMonitor` (Network framework) | `MockNetworkMonitor` |
210
+
| `AdaptiveConcurrencyControlling` (LadderKit) | `AIMDController` (AtticCore) | Any stub conforming to the protocol |
211
+
| `LocalAvailabilityProviding` (LadderKit) | `PhotosDatabaseLocalAvailability` | Any stub conforming to the protocol |
212
+
| `ThumbnailProviding` | `ThumbnailService` (viewer thumbnails) | — |
156
213
157
214
Tests never hit external services, credentials, or the real Photos library.
215
+
AtticCore ships `MockS3Provider` as a public type so the menu bar app can
216
+
wire fake S3 for SwiftUI previews without duplicating test infrastructure.
158
217
159
218
## What attic doesn't do
160
219