A container registry that uses the AT Protocol for manifest storage and S3 for blob storage.
atcr.io
docker
container
atproto
go
1# Bring Your Own Storage (BYOS)
2
3## Overview
4
5ATCR supports "Bring Your Own Storage" (BYOS) for blob storage. Users can:
6- Deploy their own hold service with embedded PDS
7- Control access via crew membership in the hold's PDS
8- Keep blob data in their own S3-compatible storage (AWS S3, Storj, Minio, UpCloud, etc.) while manifests stay in their user PDS
9
10## Architecture
11
12```
13┌──────────────────────────────────────────┐
14│ ATCR AppView (API) │
15│ - Manifests → User's PDS │
16│ - Auth & service token management │
17│ - Blob routing via XRPC │
18│ - Profile management │
19└────────────┬─────────────────────────────┘
20 │
21 │ Hold discovery priority:
22 │ 1. io.atcr.sailor.profile.defaultHold (DID)
23 │ 2. io.atcr.hold records (legacy)
24 │ 3. AppView default_hold_did
25 ▼
26┌──────────────────────────────────────────┐
27│ User's PDS │
28│ - io.atcr.sailor.profile (hold DID) │
29│ - io.atcr.manifest (with holdDid) │
30└────────────┬─────────────────────────────┘
31 │
32 │ Service token from user's PDS
33 ▼
34┌──────────────────────────────────────────┐
35│ Hold Service (did:web:hold.example.com) │
36│ ├── Embedded PDS │
37│ │ ├── Captain record (ownership) │
38│ │ └── Crew records (access control) │
39│ ├── XRPC multipart upload endpoints │
40│ └── Storage driver (S3/Storj/etc.) │
41└──────────────────────────────────────────┘
42```
43
44## Hold Service Components
45
46Each hold is a full ATProto actor with:
47- **DID**: `did:web:hold.example.com` (hold's identity)
48- **Embedded PDS**: Stores captain + crew records (shared data)
49- **Storage backend**: S3-compatible (AWS S3, Storj, Minio, UpCloud, etc.)
50- **XRPC endpoints**: Standard ATProto + custom OCI multipart upload
51
52### Records in Hold's PDS
53
54**Captain record** (`io.atcr.hold.captain/self`):
55```json
56{
57 "$type": "io.atcr.hold.captain",
58 "owner": "did:plc:alice123",
59 "public": false,
60 "deployedAt": "2025-10-14T...",
61 "region": "iad",
62 "provider": "fly.io"
63}
64```
65
66**Crew records** (`io.atcr.hold.crew/{rkey}`):
67```json
68{
69 "$type": "io.atcr.hold.crew",
70 "member": "did:plc:bob456",
71 "role": "admin",
72 "permissions": ["blob:read", "blob:write"],
73 "addedAt": "2025-10-14T..."
74}
75```
76
77### Sailor Profile (User's PDS)
78
79Users set their preferred hold in their sailor profile:
80
81```json
82{
83 "$type": "io.atcr.sailor.profile",
84 "defaultHold": "did:web:hold.example.com",
85 "createdAt": "2025-10-02T...",
86 "updatedAt": "2025-10-02T..."
87}
88```
89
90## Deployment
91
92### Configuration
93
94Hold service is configured entirely via environment variables:
95
96```bash
97# Hold identity (REQUIRED)
98HOLD_PUBLIC_URL=https://hold.example.com
99HOLD_OWNER=did:plc:your-did-here
100
101# S3 storage backend (REQUIRED)
102AWS_ACCESS_KEY_ID=your_access_key
103AWS_SECRET_ACCESS_KEY=your_secret_key
104AWS_REGION=us-east-1
105S3_BUCKET=my-blobs
106
107# Access control
108HOLD_PUBLIC=false # Require authentication for reads
109HOLD_ALLOW_ALL_CREW=false # Only explicit crew members can write
110
111# Embedded PDS
112HOLD_DATABASE_PATH=/var/lib/atcr-hold/hold.db
113HOLD_DATABASE_KEY_PATH=/var/lib/atcr-hold/keys
114```
115
116### Running Locally
117
118For local development, use Minio as an S3-compatible storage:
119
120```bash
121# Start Minio (in separate terminal)
122docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"
123
124# Build
125go build -o bin/atcr-hold ./cmd/hold
126
127# Run (with env vars or .env file)
128export HOLD_PUBLIC_URL=http://localhost:8080
129export HOLD_OWNER=did:plc:your-did-here
130export AWS_ACCESS_KEY_ID=minioadmin
131export AWS_SECRET_ACCESS_KEY=minioadmin
132export S3_BUCKET=test
133export S3_ENDPOINT=http://localhost:9000
134export HOLD_DATABASE_PATH=/tmp/atcr-hold/hold.db
135
136./bin/atcr-hold
137```
138
139On first run, the hold service creates:
140- Captain record in embedded PDS (making you the owner)
141- Crew record for owner with all permissions
142- DID document at `/.well-known/did.json`
143
144### Deploy to Fly.io
145
146```bash
147# Create fly.toml
148cat > fly.toml <<EOF
149app = "my-atcr-hold"
150primary_region = "ord"
151
152[env]
153 HOLD_PUBLIC_URL = "https://my-atcr-hold.fly.dev"
154 AWS_REGION = "us-east-1"
155 S3_BUCKET = "my-blobs"
156 HOLD_PUBLIC = "false"
157 HOLD_ALLOW_ALL_CREW = "false"
158
159[http_service]
160 internal_port = 8080
161 force_https = true
162 auto_stop_machines = true
163 auto_start_machines = true
164 min_machines_running = 0
165
166[[vm]]
167 cpu_kind = "shared"
168 cpus = 1
169 memory_mb = 256
170EOF
171
172# Deploy
173fly launch
174fly deploy
175
176# Set secrets
177fly secrets set AWS_ACCESS_KEY_ID=...
178fly secrets set AWS_SECRET_ACCESS_KEY=...
179fly secrets set HOLD_OWNER=did:plc:your-did-here
180```
181
182## Request Flow
183
184### Push with BYOS
185
186```
1871. Client: docker push atcr.io/alice/myapp:latest
188
1892. AppView resolves alice → did:plc:alice123
190
1913. AppView discovers hold DID:
192 - Check alice's sailor profile for defaultHold
193 - Returns: "did:web:alice-storage.fly.dev"
194
1954. AppView gets service token from alice's PDS:
196 GET /xrpc/com.atproto.server.getServiceAuth?aud=did:web:alice-storage.fly.dev
197 Response: { "token": "eyJ..." }
198
1995. AppView initiates multipart upload to hold:
200 POST https://alice-storage.fly.dev/xrpc/io.atcr.hold.initiateUpload
201 Authorization: Bearer {serviceToken}
202 Body: { "digest": "sha256:abc..." }
203 Response: { "uploadId": "xyz" }
204
2056. For each part:
206 - AppView: POST /xrpc/io.atcr.hold.getPartUploadUrl
207 - Hold validates service token, checks crew membership
208 - Hold returns: { "url": "https://s3.../presigned" }
209 - Client uploads directly to S3 presigned URL
210
2117. AppView completes upload:
212 POST /xrpc/io.atcr.hold.completeUpload
213 Body: { "uploadId": "xyz", "digest": "sha256:abc...", "parts": [...] }
214
2158. Manifest stored in alice's PDS:
216 - holdDid: "did:web:alice-storage.fly.dev"
217 - holdEndpoint: "https://alice-storage.fly.dev" (backward compat)
218```
219
220### Pull with BYOS
221
222```
2231. Client: docker pull atcr.io/alice/myapp:latest
224
2252. AppView fetches manifest from alice's PDS
226
2273. Manifest contains:
228 - holdDid: "did:web:alice-storage.fly.dev"
229
2304. AppView caches hold DID for 10 minutes (covers pull operation)
231
2325. Client requests blob: GET /v2/alice/myapp/blobs/sha256:abc123
233
2346. AppView uses cached hold DID from manifest
235
2367. AppView gets service token from alice's PDS
237
2388. AppView calls hold XRPC:
239 GET /xrpc/com.atproto.sync.getBlob?did={userDID}&cid=sha256:abc123
240 Authorization: Bearer {serviceToken}
241 Response: { "url": "https://s3.../presigned-download" }
242
2439. AppView redirects client to presigned S3 URL
244
24510. Client downloads directly from S3
246```
247
248**Key insight:** Pull uses the `holdDid` stored in the manifest, ensuring blobs are fetched from where they were originally pushed.
249
250## Access Control
251
252### Read Access
253
254- **Public hold** (`HOLD_PUBLIC=true`): Anonymous + authenticated users
255- **Private hold** (`HOLD_PUBLIC=false`): Authenticated users with crew membership
256
257### Write Access
258
259- Hold owner (captain) OR crew members only
260- Verified via `io.atcr.hold.crew` records in hold's embedded PDS
261- Service token proves user identity (from user's PDS)
262
263### Authorization Flow
264
265```go
2661. AppView gets service token from user's PDS
2672. AppView sends request to hold with service token
2683. Hold validates service token (checks it's from user's PDS)
2694. Hold extracts user's DID from token
2705. Hold checks crew records in its embedded PDS
2716. If crew member found → allow, else → deny
272```
273
274## Managing Crew Members
275
276### Add Crew Member
277
278Use ATProto client to create crew record in hold's PDS:
279
280```bash
281# Via XRPC (if hold supports it)
282POST https://hold.example.com/xrpc/io.atcr.hold.requestCrew
283Authorization: Bearer {userOAuthToken}
284
285# Or manually via captain's OAuth to hold's PDS
286atproto put-record \
287 --pds https://hold.example.com \
288 --collection io.atcr.hold.crew \
289 --rkey "{memberDID}" \
290 --value '{
291 "$type": "io.atcr.hold.crew",
292 "member": "did:plc:bob456",
293 "role": "admin",
294 "permissions": ["blob:read", "blob:write"]
295 }'
296```
297
298### Remove Crew Member
299
300```bash
301atproto delete-record \
302 --pds https://hold.example.com \
303 --collection io.atcr.hold.crew \
304 --rkey "{memberDID}"
305```
306
307## Storage Backends
308
309Hold service requires S3-compatible storage. Supported providers:
310- **AWS S3** - Amazon Simple Storage Service
311- **Storj** - Decentralized cloud storage (via S3 gateway)
312- **Minio** - High-performance object storage (great for local development)
313- **UpCloud** - European cloud provider
314- **Azure** - Azure Blob Storage (via S3-compatible API)
315- **GCS** - Google Cloud Storage (via S3-compatible API)
316
317## Example: Team Hold
318
319```bash
320# 1. Deploy hold service
321export HOLD_PUBLIC_URL=https://team-hold.fly.dev
322export HOLD_OWNER=did:plc:admin
323export HOLD_PUBLIC=false # Private
324export AWS_ACCESS_KEY_ID=...
325export AWS_SECRET_ACCESS_KEY=...
326export S3_BUCKET=team-blobs
327
328fly deploy
329
330# 2. Hold auto-creates captain + crew records on first run
331
332# 3. Admin adds team members via hold's PDS (requires OAuth)
333# (TODO: Implement crew management UI/CLI)
334
335# 4. Team members set their sailor profile:
336atproto put-record \
337 --collection io.atcr.sailor.profile \
338 --rkey "self" \
339 --value '{
340 "$type": "io.atcr.sailor.profile",
341 "defaultHold": "did:web:team-hold.fly.dev"
342 }'
343
344# 5. Team members can now push/pull using team hold
345```
346
347## Limitations
348
349### Current IAM Challenges
350
351See [EMBEDDED_PDS.md](./EMBEDDED_PDS.md#iam-challenges) for detailed discussion.
352
353**Known issues:**
3541. **RPC permission format**: Service tokens don't work with IP-based DIDs in local dev
3552. **Dynamic hold discovery**: AppView can't dynamically OAuth arbitrary holds from sailor profiles
3563. **Manual profile management**: No UI for updating sailor profile (must use ATProto client)
357
358**Workaround:** Use hostname-based DIDs (`did:web:hold.example.com`) and public holds for now.
359
360## Future Improvements
361
3621. **Crew management UI** - Web interface for adding/removing crew members
3632. **Dynamic OAuth** - Support for arbitrary BYOS holds without pre-configuration
3643. **Hold migration** - Tools for moving blobs between holds
3654. **Storage analytics** - Track usage per user/repository
3665. **Distributed cache** - Redis for hold DID cache in multi-instance deployments
367
368## References
369
370- [EMBEDDED_PDS.md](./EMBEDDED_PDS.md) - Embedded PDS architecture and IAM details
371- [ATProto Lexicon Spec](https://atproto.com/specs/lexicon)
372- [Distribution Storage Drivers](https://distribution.github.io/distribution/storage-drivers/)
373- [S3 Presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html)