commits
Covers adaptive iCloud-lane throttling (AIMDController), retry queue,
unavailable-asset store, network pause/resume, staging reuse, the shift
from aws-sdk-swift to URLSessionS3Client + aws-signer-v4, and the
AtticCore-as-SPM-library story for the future menu bar app.
Local path dep was only for development. For releases, pin to the
published ladder tag so CI can build from a clean checkout.
Security
- attic init fails closed if stdin isn't a TTY (can't disable echo), so
piped secrets can't leak to the screen or a tee'd log.
- Viewer CSP is scoped to the configured S3 endpoint host instead of a
hardcoded *.amazonaws.com. Works correctly for R2/MinIO/Backblaze.
- Viewer presigned-URL lifetime cut from 4h to 1h.
- Reject bucket names containing a dot when pathStyle=false (virtual-hosted
TLS certs only cover one label — saves a confusing connect error).
- Staging dir created with 0o700.
Polish
- ViewerDataStore parses year from ISO8601 prefix instead of allocating a
formatter per asset.
- httpMaximumConnectionsPerHost bumped from 6 → 32 so bounded concurrency
isn't re-serialized at the socket layer.
- Per-asset metadata JSON uploads drop .prettyPrinted (~40% smaller).
Manifest/config/retry-queue stay pretty-printed.
- BackupOptions: drop saveInterval; manifest saves at batch boundaries.
- Rename AdaptiveConcurrency.swift → AIMDController.swift and
BackupConstants.swift → DateFormatting.swift.
- Simplify concurrencyChanged plumbing.
Bumps version to 1.0.0-beta.7.
- RetryQueue: drop legacy failedUUIDs decoder, custom Codable, and
failedUUIDs initializer. Synthesized via compiler-generated Codable now.
- BackupPipeline: check classification == .permanentlyUnavailable directly
(not legacy unavailable bool). Drop normalizeUUID — ladder now returns
bare UUIDs at source.
- BackupPipeline: extract filterPending, exportBatchWithFallback, and
finalizeBackup. Drop dead ExportProviderError.isPermission catch
(permission is pre-flight only, never raised during exportBatch).
- BackupUpload: convert network-pause retry from recursion to loop.
- LadderKitExportProvider: translate AppleScriptError into
ExportProviderError.permissionDenied / .timeout.
When --limit cut a run short, a successful result would wipe the entire
retry queue — including UUIDs that were never attempted in that run.
Their attempts/firstFailedAt history was lost and they lost their
retry-first priority on the next run.
RetryQueue.merged now takes the attempted set. Prior entries whose UUID
isn't in that set are carried forward unchanged. Attempted UUIDs drop
(success) or merge (failure) as before. Unlimited `attic backup` runs
still clear the queue fully on full success, because `attempted` covers
the whole pending list.
The status command now reads local-availability from Photos.sqlite and
the retry queue from disk, then surfaces them as two new rows:
- "Lanes" under Backup: N local / M iCloud pending, so users can see
how much of the pending work runs in the fast lane vs. the throttled
iCloud lane.
- "Retries" section: queued count, max attempts, and the date of the
oldest first failure — uses the new RetryEntry schema from beta.6.
Both rows render in rich (ANSI) and plain output.
RetryQueue now stores per-asset entries with classification, attempts,
firstFailedAt, lastFailedAt, and lastMessage. Merging across runs
preserves firstFailedAt and bumps attempts, so the UI can surface how
long an asset has been stuck.
Legacy `failedUUIDs: [String]` payloads decode transparently — existing
retry-queue files are upgraded on next write.
Also normalize PhotoKit's full-path identifiers (UUID/L0/001) to bare
UUIDs when appending to report.errors, so the retry-first partitioning
actually matches on the next run.
- BackupCommand builds a PhotosDatabaseLocalAvailability + AIMDController
and passes both to LadderKitExportProvider, so the backup now runs a
fast local lane and a throttled iCloud lane.
- runBackup takes an optional AdaptiveConcurrencyControlling, polls it
between batches, and emits BackupProgressDelegate.concurrencyChanged
when the limit shifts.
- TerminalRenderer shows lane count beside upload speed; the log
delegate prints a single line when concurrency changes.
Moves the iCloud-lane backoff policy out of LadderKit (which now exposes
only the observation-only protocol) and into attic, where "how aggressively
to back off" is an orchestration decision alongside the retry queue and
manifest.
- AIMDController actor conforms to AdaptiveConcurrencyControlling.
Sliding window of the last 20 outcomes; halves on >30% transient-failure
rate, grows by 1 on <=5%, ignores permanent failures. Window clears on
every limit change so stale pre-change outcomes can't immediately re-trip.
- 3-field Config (initial/min/max); thresholds and window size are internal
constants — callers don't tune them today.
- Tests cover backoff, recovery, permanent-failure ignore, floor/ceiling,
and the sliding-window burst-straddle case that a tumbling window would
miss.
Also flips Package.swift's ladder dep from `from: "0.4.0"` to a local path
for development; revert to `from: "0.5.0"` after tagging upstream. Updates
the BackupPipelineTests mock for ExportError's new classification-based
initializer.
BackupPipeline.swift was 536 lines (over the 500-line convention).
Move UploadContext + uploadExported into the existing BackupUpload.swift
alongside uploadSingleAsset. Pure refactor, no behavioral change.
Prep work for adaptive-concurrency additions in the next release.
Shared-album assets whose iCloud derivative has failed server-side are
now detected via LadderKit's new ExportError.unavailable flag, recorded
in ~/.attic/unavailable-assets.json, and skipped on subsequent backups
instead of being retried forever through the AppleScript fallback
(which goes through the same broken shared-stream pipeline).
Bumps LadderKit to 0.4.0.
Failed assets now retry first on the next run via a local queue
(~/.attic/retry-queue.json). Staging directory is persisted at
~/.attic/staging/ so already-exported files (including iCloud downloads)
are reclaimed across runs instead of re-exported, with dedup when
PhotoKit and AppleScript both produce a copy.
Upload path simplified: removed the nested TaskGroup timeout race that
caused a rare malloc heap corruption abort mid-upload. URLSession's
built-in timeoutIntervalForResource provides the per-upload cap.
Also switches date formatting to Date.ISO8601FormatStyle (value-type,
concurrency-safe) and tightens the terminal renderer's locking.
- Prevent concurrent fetchFilters calls from stacking via in-flight guard
- Fix metadataLoading stuck true when server loads fast (no poll needed)
- Clean up loading state when discarding stale generation responses
- Discard stale lightbox fetches on rapid arrow key navigation
Start server immediately and load metadata in background so the browser
opens right away. Filter dropdowns update as assets arrive with a progress
bar showing loading state. Cascading filters: selecting a year shows only
albums from that year and vice versa. Unavailable type options are greyed
out. Clear filters button resets all.
Replace hardcoded UTI/extension/MIME maps with UTType system lookups in
S3Paths, ExportProviding, and ViewerDataStore. Normalize jpeg->jpg for
backward compatibility with existing S3 keys.
Includes all code review fixes: single-pass filterOptions, CSP header,
thread-safe date formatting, consolidated JS filter functions, extracted
URL decoding helper, removed Google Fonts CDN dependency.
P1: Fix actor reentrancy bug in ThumbnailService concurrency limiter
using slot-transfer pattern. Hoist ISO8601DateFormatter to static.
P2: Add UUID validation on /api/assets/:uuid, fix DOM XSS in lightbox
by replacing innerHTML with DOM APIs, clamp pageSize to 200, add O(1)
UUID dictionary index, eliminate double S3 round-trip (HEAD+GET to GET),
consolidate duplicate response types, remove dead yearExtractor code.
P3: Make VideoThumbnailer URL overload internal, add security headers
on HTML response, simplify AssetPage return type.
Add `attic viewer` command that serves a browser-based photo grid via
Hummingbird. Thumbnails are generated lazily on first request (400px
JPEG) with three-tier caching: local disk, S3, generate from original.
Includes viewport-based memory management, infinite scroll, lightbox
with keyboard nav, filtering by year/album/media type/favorites, and
video poster frame extraction via AVAssetImageGenerator.
Disable terminal echo/canonical mode during backup so keypresses don't
inject newlines that push the dashboard down. Show failed asset filenames
and error messages in the completion summary with a retry hint.
Dictionary sort is non-deterministic for equal values, causing
flaky CI on utiBreakdownTopTypes test. Add alphabetical tiebreaker.
- Replace tuple with TypeBreakdown struct for Sendable conformance
- Narrow catch to CLIError.notInitialized to avoid swallowing future cases
- Single dictionary lookup in computeBackupStats
- Hoist NumberFormatter to file-level constant
- Avoid intermediate array in computeS3Info via max(by:)
- Fix typo in test name (s3InfoDerives)
Merge scan into status as a sectioned dashboard showing library stats,
backup progress, S3 manifest info, and UTI breakdown. Supports ANSI
rich output for TTY and plain text for piped/CI. Remove scan command.
Concurrent S3 uploads via bounded TaskGroup (default 6 workers).
Two-tier error handling: network-down errors trigger drain-then-pause
with automatic retry on recovery, server-transient errors retry with
exponential backoff and jitter.
Key changes:
- Typed URLError.Code + S3ClientError matching in RetryPolicy
- Network pause/resume with manifest save before waiting
- Cancellation-safe manifest save via do/catch for CancellationError
- UploadContext parameter object to reduce function parameter count
- maxPauseRetries cap prevents infinite pause/resume on flaky networks
- MockNetworkMonitor moved from production to test target
Test job now has needs: lint so it won't run if linting fails,
providing early exit on formatting/style issues.
- Add .swiftlint.yml and .swiftformat with reasonable defaults
- Add CI workflow running lint + format check + tests on push/PR
- Run swiftlint --fix and swiftformat across entire codebase
- Fix modifier order (nonisolated before private)
- Fix debugPrint → print in all CLI commands (debugPrint wraps
output in quotes, which is wrong for user-facing CLI output)
Verify that S3ManifestStore.load() returns an empty manifest for
httpError(404) and s3Error("NoSuchKey"), and throws on non-404 errors.
These cover the isNotFoundError fix for the new URLSession S3 client.
Drop ~27 transitive packages (Smithy, aws-crt-swift, async-http-client, etc.)
down to ~9 by replacing the full AWS SDK with a minimal S3 client built on
URLSession and adam-fowler/aws-signer-v4. The project only uses 4 S3 operations
against S3-compatible providers — the full SDK was massive overkill.
- URLSessionS3Client: putObject (data + file streaming), getObject, headObject,
listObjectsV2 with pagination, SigV4 request signing
- S3XMLParsing: SAX parsers for ListObjectsV2 responses and S3 error bodies
- Proper payload signing: UNSIGNED-PAYLOAD for uploads, empty-body hash for
GET/HEAD (compatible with all S3 providers)
- S3 error XML parsing wired into checkResponse for better diagnostics
- isNotFoundError updated to match new S3ClientError types
- Host header set from constructed URL (correct for both path-style and
virtual-hosted)
- No force unwraps — all URL construction uses guard/throw
- Remove PowerAssertion.end() to eliminate double-endActivity risk
- Fix hardcoded "15 minutes" in timeout error message to use actual value
- Use guard let self in NWPathNetworkMonitor closure
- Cap stabilization delay to remaining timeout budget
- Document intentional divergence between isTransientUploadError and isTransient
Auto-pause uploads when network drops (NWPathMonitor), auto-resume when
it returns, and exit cleanly after 15-minute timeout with manifest saved.
Prevent idle sleep during backup via ProcessInfo.beginActivity. Terminal
shows pause status with elapsed wait time, excludes pause from speed.
Show a Braille-pattern spinner with status messages during the
preparation phase (config load, manifest download, library scan)
so the CLI no longer appears hung before uploads begin.
Bump version to 1.0.0-alpha.2.
Replace the Deno/TypeScript CLI with a native Swift implementation using
a shared AtticCore library (for future menu bar app reuse). LadderKit is
now called directly as a library instead of shelling out to a subprocess.
Swift implementation includes:
- AtticCore: S3 provider, manifest, config, keychain, metadata,
backup/verify/refresh/rebuild pipelines (all behind protocols)
- AtticCLI: ArgumentParser commands with live-updating terminal dashboard
- 75 tests using Swift Testing framework with mock implementations
Also: include iCloud Shared Photo Library assets in PhotoKit enumeration,
add macOS permission dialog guidance to init and README.
- Check ladder exit code 77 instead of string-matching stderr for
permission errors — more robust across message text changes
- Increase base subprocess timeout from 5 to 10 minutes to match
ladder's per-asset AppleScript timeout, preventing attic from
killing ladder while iCloud downloads are still in progress
Spawn ladder with an empty UUID list before processing any assets to
verify Photos and Automation permissions are available. If the check
fails, abort immediately with actionable instructions instead of
waiting until the first batch.
Detect Automation permission errors from ladder and abort early instead
of retrying every batch. Update init to list all required permissions.
Update docs with AppleScript fallback and Automation permission setup.
Bump to v0.2.5.
The photo-first sort was a workaround for batch timeouts, but
skip-and-defer already handles that. The sort caused unnatural behavior:
--limit 100 would only process tiny photos, never reaching videos.
Natural DB order (roughly chronological) gives a representative mix
per run. Timeout scaling and skip-and-defer handle resilience.
The AWS SDK S3Client keeps HTTP connections alive in its pool. Deno
waits for them to drain before exiting, causing a long hang after
backup completes. Added destroy() to S3Provider interface and call
it in finally blocks for all commands.
Security
- attic init fails closed if stdin isn't a TTY (can't disable echo), so
piped secrets can't leak to the screen or a tee'd log.
- Viewer CSP is scoped to the configured S3 endpoint host instead of a
hardcoded *.amazonaws.com. Works correctly for R2/MinIO/Backblaze.
- Viewer presigned-URL lifetime cut from 4h to 1h.
- Reject bucket names containing a dot when pathStyle=false (virtual-hosted
TLS certs only cover one label — saves a confusing connect error).
- Staging dir created with 0o700.
Polish
- ViewerDataStore parses year from ISO8601 prefix instead of allocating a
formatter per asset.
- httpMaximumConnectionsPerHost bumped from 6 → 32 so bounded concurrency
isn't re-serialized at the socket layer.
- Per-asset metadata JSON uploads drop .prettyPrinted (~40% smaller).
Manifest/config/retry-queue stay pretty-printed.
- BackupOptions: drop saveInterval; manifest saves at batch boundaries.
- Rename AdaptiveConcurrency.swift → AIMDController.swift and
BackupConstants.swift → DateFormatting.swift.
- Simplify concurrencyChanged plumbing.
Bumps version to 1.0.0-beta.7.
- RetryQueue: drop legacy failedUUIDs decoder, custom Codable, and
failedUUIDs initializer. Synthesized via compiler-generated Codable now.
- BackupPipeline: check classification == .permanentlyUnavailable directly
(not legacy unavailable bool). Drop normalizeUUID — ladder now returns
bare UUIDs at source.
- BackupPipeline: extract filterPending, exportBatchWithFallback, and
finalizeBackup. Drop dead ExportProviderError.isPermission catch
(permission is pre-flight only, never raised during exportBatch).
- BackupUpload: convert network-pause retry from recursion to loop.
- LadderKitExportProvider: translate AppleScriptError into
ExportProviderError.permissionDenied / .timeout.
When --limit cut a run short, a successful result would wipe the entire
retry queue — including UUIDs that were never attempted in that run.
Their attempts/firstFailedAt history was lost and they lost their
retry-first priority on the next run.
RetryQueue.merged now takes the attempted set. Prior entries whose UUID
isn't in that set are carried forward unchanged. Attempted UUIDs drop
(success) or merge (failure) as before. Unlimited `attic backup` runs
still clear the queue fully on full success, because `attempted` covers
the whole pending list.
The status command now reads local-availability from Photos.sqlite and
the retry queue from disk, then surfaces them as two new rows:
- "Lanes" under Backup: N local / M iCloud pending, so users can see
how much of the pending work runs in the fast lane vs. the throttled
iCloud lane.
- "Retries" section: queued count, max attempts, and the date of the
oldest first failure — uses the new RetryEntry schema from beta.6.
Both rows render in rich (ANSI) and plain output.
RetryQueue now stores per-asset entries with classification, attempts,
firstFailedAt, lastFailedAt, and lastMessage. Merging across runs
preserves firstFailedAt and bumps attempts, so the UI can surface how
long an asset has been stuck.
Legacy `failedUUIDs: [String]` payloads decode transparently — existing
retry-queue files are upgraded on next write.
Also normalize PhotoKit's full-path identifiers (UUID/L0/001) to bare
UUIDs when appending to report.errors, so the retry-first partitioning
actually matches on the next run.
- BackupCommand builds a PhotosDatabaseLocalAvailability + AIMDController
and passes both to LadderKitExportProvider, so the backup now runs a
fast local lane and a throttled iCloud lane.
- runBackup takes an optional AdaptiveConcurrencyControlling, polls it
between batches, and emits BackupProgressDelegate.concurrencyChanged
when the limit shifts.
- TerminalRenderer shows lane count beside upload speed; the log
delegate prints a single line when concurrency changes.
Moves the iCloud-lane backoff policy out of LadderKit (which now exposes
only the observation-only protocol) and into attic, where "how aggressively
to back off" is an orchestration decision alongside the retry queue and
manifest.
- AIMDController actor conforms to AdaptiveConcurrencyControlling.
Sliding window of the last 20 outcomes; halves on >30% transient-failure
rate, grows by 1 on <=5%, ignores permanent failures. Window clears on
every limit change so stale pre-change outcomes can't immediately re-trip.
- 3-field Config (initial/min/max); thresholds and window size are internal
constants — callers don't tune them today.
- Tests cover backoff, recovery, permanent-failure ignore, floor/ceiling,
and the sliding-window burst-straddle case that a tumbling window would
miss.
Also flips Package.swift's ladder dep from `from: "0.4.0"` to a local path
for development; revert to `from: "0.5.0"` after tagging upstream. Updates
the BackupPipelineTests mock for ExportError's new classification-based
initializer.
Shared-album assets whose iCloud derivative has failed server-side are
now detected via LadderKit's new ExportError.unavailable flag, recorded
in ~/.attic/unavailable-assets.json, and skipped on subsequent backups
instead of being retried forever through the AppleScript fallback
(which goes through the same broken shared-stream pipeline).
Bumps LadderKit to 0.4.0.
Failed assets now retry first on the next run via a local queue
(~/.attic/retry-queue.json). Staging directory is persisted at
~/.attic/staging/ so already-exported files (including iCloud downloads)
are reclaimed across runs instead of re-exported, with dedup when
PhotoKit and AppleScript both produce a copy.
Upload path simplified: removed the nested TaskGroup timeout race that
caused a rare malloc heap corruption abort mid-upload. URLSession's
built-in timeoutIntervalForResource provides the per-upload cap.
Also switches date formatting to Date.ISO8601FormatStyle (value-type,
concurrency-safe) and tightens the terminal renderer's locking.
Start server immediately and load metadata in background so the browser
opens right away. Filter dropdowns update as assets arrive with a progress
bar showing loading state. Cascading filters: selecting a year shows only
albums from that year and vice versa. Unavailable type options are greyed
out. Clear filters button resets all.
Replace hardcoded UTI/extension/MIME maps with UTType system lookups in
S3Paths, ExportProviding, and ViewerDataStore. Normalize jpeg->jpg for
backward compatibility with existing S3 keys.
Includes all code review fixes: single-pass filterOptions, CSP header,
thread-safe date formatting, consolidated JS filter functions, extracted
URL decoding helper, removed Google Fonts CDN dependency.
P1: Fix actor reentrancy bug in ThumbnailService concurrency limiter
using slot-transfer pattern. Hoist ISO8601DateFormatter to static.
P2: Add UUID validation on /api/assets/:uuid, fix DOM XSS in lightbox
by replacing innerHTML with DOM APIs, clamp pageSize to 200, add O(1)
UUID dictionary index, eliminate double S3 round-trip (HEAD+GET to GET),
consolidate duplicate response types, remove dead yearExtractor code.
P3: Make VideoThumbnailer URL overload internal, add security headers
on HTML response, simplify AssetPage return type.
Add `attic viewer` command that serves a browser-based photo grid via
Hummingbird. Thumbnails are generated lazily on first request (400px
JPEG) with three-tier caching: local disk, S3, generate from original.
Includes viewport-based memory management, infinite scroll, lightbox
with keyboard nav, filtering by year/album/media type/favorites, and
video poster frame extraction via AVAssetImageGenerator.
- Replace tuple with TypeBreakdown struct for Sendable conformance
- Narrow catch to CLIError.notInitialized to avoid swallowing future cases
- Single dictionary lookup in computeBackupStats
- Hoist NumberFormatter to file-level constant
- Avoid intermediate array in computeS3Info via max(by:)
- Fix typo in test name (s3InfoDerives)
Concurrent S3 uploads via bounded TaskGroup (default 6 workers).
Two-tier error handling: network-down errors trigger drain-then-pause
with automatic retry on recovery, server-transient errors retry with
exponential backoff and jitter.
Key changes:
- Typed URLError.Code + S3ClientError matching in RetryPolicy
- Network pause/resume with manifest save before waiting
- Cancellation-safe manifest save via do/catch for CancellationError
- UploadContext parameter object to reduce function parameter count
- maxPauseRetries cap prevents infinite pause/resume on flaky networks
- MockNetworkMonitor moved from production to test target
- Add .swiftlint.yml and .swiftformat with reasonable defaults
- Add CI workflow running lint + format check + tests on push/PR
- Run swiftlint --fix and swiftformat across entire codebase
- Fix modifier order (nonisolated before private)
- Fix debugPrint → print in all CLI commands (debugPrint wraps
output in quotes, which is wrong for user-facing CLI output)
Drop ~27 transitive packages (Smithy, aws-crt-swift, async-http-client, etc.)
down to ~9 by replacing the full AWS SDK with a minimal S3 client built on
URLSession and adam-fowler/aws-signer-v4. The project only uses 4 S3 operations
against S3-compatible providers — the full SDK was massive overkill.
- URLSessionS3Client: putObject (data + file streaming), getObject, headObject,
listObjectsV2 with pagination, SigV4 request signing
- S3XMLParsing: SAX parsers for ListObjectsV2 responses and S3 error bodies
- Proper payload signing: UNSIGNED-PAYLOAD for uploads, empty-body hash for
GET/HEAD (compatible with all S3 providers)
- S3 error XML parsing wired into checkResponse for better diagnostics
- isNotFoundError updated to match new S3ClientError types
- Host header set from constructed URL (correct for both path-style and
virtual-hosted)
- No force unwraps — all URL construction uses guard/throw
- Remove PowerAssertion.end() to eliminate double-endActivity risk
- Fix hardcoded "15 minutes" in timeout error message to use actual value
- Use guard let self in NWPathNetworkMonitor closure
- Cap stabilization delay to remaining timeout budget
- Document intentional divergence between isTransientUploadError and isTransient
Show a Braille-pattern spinner with status messages during the
preparation phase (config load, manifest download, library scan)
so the CLI no longer appears hung before uploads begin.
Bump version to 1.0.0-alpha.2.
Replace the Deno/TypeScript CLI with a native Swift implementation using
a shared AtticCore library (for future menu bar app reuse). LadderKit is
now called directly as a library instead of shelling out to a subprocess.
Swift implementation includes:
- AtticCore: S3 provider, manifest, config, keychain, metadata,
backup/verify/refresh/rebuild pipelines (all behind protocols)
- AtticCLI: ArgumentParser commands with live-updating terminal dashboard
- 75 tests using Swift Testing framework with mock implementations
Also: include iCloud Shared Photo Library assets in PhotoKit enumeration,
add macOS permission dialog guidance to init and README.
- Check ladder exit code 77 instead of string-matching stderr for
permission errors — more robust across message text changes
- Increase base subprocess timeout from 5 to 10 minutes to match
ladder's per-asset AppleScript timeout, preventing attic from
killing ladder while iCloud downloads are still in progress
The photo-first sort was a workaround for batch timeouts, but
skip-and-defer already handles that. The sort caused unnatural behavior:
--limit 100 would only process tiny photos, never reaching videos.
Natural DB order (roughly chronological) gives a representative mix
per run. Timeout scaling and skip-and-defer handle resilience.