A container registry that uses the AT Protocol for manifest storage and S3 for blob storage.
atcr.io
docker
container
atproto
go
rebuild repomgr into a custom repo operator. up to 2x faster
+2701
-777
+148
docs/REPOMGR_MIGRATION.md
+148
docs/REPOMGR_MIGRATION.md
···
1
+
# Incremental Migration: Vendored repomgr → Direct `indigo/repo`
2
+
3
+
## Context
4
+
5
+
The hold PDS uses a vendored copy of indigo's `repomgr` (~1450 lines in `pkg/hold/pds/repomgr.go`). Upstream, repomgr is [soft-deprecated](https://github.com/bluesky-social/indigo/pull/1102#issuecomment-2985956040) — bnewbold recommends using `indigo/repo` directly (as [cocoon](https://github.com/haileyok/cocoon) does). The vendored copy already has custom patches (PutRecord, UpsertRecord, prevData) and will continue to drift. This migration defines a clean interface, then swaps the implementation behind it.
6
+
7
+
## Phase 0: Save plan to docs, remove dead code, define interface ✅
8
+
9
+
**Goal:** Persist this migration plan as a reference doc. Shrink repomgr.go from ~1450 lines to ~750 by removing dead code. Fix import.go encapsulation. Define `RepoOperator` interface so all code accesses repomgr through it. No behavior change.
10
+
11
+
**Completed 2026-02-28.** repomgr.go: 1453 → 871 lines (-40%). import.go: 189 → 88 lines (-53%). New repo_operator.go: 82 lines.
12
+
13
+
### Step 1: Save plan to docs ✅
14
+
- Write this plan to `docs/REPOMGR_MIGRATION.md`
15
+
16
+
### Step 2: Dead code deleted from `repomgr.go` ✅
17
+
- `HandleExternalUserEvent()` + `handleExternalUserEventNoArchive()` + `handleExternalUserEventArchive()`
18
+
- `ImportNewRepo()` + `processNewRepo()` + `walkTree()` + `processOp()` + `stringOrNil()`
19
+
- `CheckRepoSig()`
20
+
- `TakeDownRepo()`, `ResetRepo()`, `VerifyRepo()`
21
+
- `GetProfile()`
22
+
- `CarStore()`
23
+
- `NextTID()` / `nextTID()` (removed entirely — `BatchWrite` uses `rm.clk.Next()` directly)
24
+
- `noArchive` field removed from struct
25
+
- 12 unused imports cleaned up
26
+
27
+
### Step 3: Fixed `import.go` encapsulation ✅
28
+
Added `BulkUpsert()` method to `RepoManager`. Rewrote `ImportFromCAR` to call `p.repomgr.BulkUpsert()` instead of reaching into `p.repomgr.lockUser`, `p.repomgr.cs`, `p.repomgr.kmgr`, `p.repomgr.events`. Removed the 88-line `bulkImportRecords()` private method.
29
+
30
+
### Step 4: Defined `RepoOperator` interface ✅
31
+
32
+
New file: `pkg/hold/pds/repo_operator.go` — interface with 16 methods, compile-time check, shared types (`RepoEvent`, `RepoOp`, `EventKind`, `BulkRecord`).
33
+
34
+
### Step 5: Updated callers to use interface ✅
35
+
- `pkg/hold/pds/server.go` — `repomgr *RepoManager` → `repomgr RepoOperator`, `RepomgrRef()` returns `RepoOperator`
36
+
- Downstream callers (`hold/server.go`, `admin/handlers_relays.go`, tests) unchanged — they go through `RepomgrRef()` which returns the interface
37
+
- `var _ RepoOperator = (*RepoManager)(nil)` compile-time check in `repo_operator.go`
38
+
39
+
### Verification: ✅
40
+
- `make lint` — 0 issues (also fixed pre-existing unchecked error in `events.go`)
41
+
- `make test` — all tests pass
42
+
43
+
---
44
+
45
+
## Phase 1: Test hardening against the interface ✅
46
+
47
+
**Goal:** Write tests against `RepoOperator` that verify current behavior while `RepoManager` is the only implementation. These become the regression safety net when swapping to the new implementation in Phase 3.
48
+
49
+
**Completed 2026-02-28.** New `repo_operator_test.go`: 38 subtests covering all CRUD, read, event emission, error paths, and edge cases. `runRepoOperatorTests(t, setup)` pattern ready for Phase 2's `DirectRepoOperator`.
50
+
51
+
### Gaps covered ✅
52
+
- `CreateRecord` — round-trip, TID 13-char rkey format, no-panic without event handler
53
+
- `UpdateRecord` — CID changes, new data returned, non-existent record error, hydrated events
54
+
- `PutRecord` — explicit rkey, duplicate rkey error, hydrated events
55
+
- `UpsertRecord` — create path (created=true), update path (created=false, CID changes)
56
+
- `DeleteRecord` — delete + verify gone, non-existent rkey error
57
+
- `BatchWrite` — create+delete batch, update write type, auto-rkey (nil Rkey), delete-not-found error, empty write elem error, multi-op event emission with hydration, update hydration
58
+
- `BulkUpsert` — create + re-upsert with changed data, multi-op event emission
59
+
- `GetRecord` — CID match, CID mismatch error, not-found error
60
+
- `GetRecordProof` — head CID + proof blocks, not-found error, no-repo error
61
+
- `GetRepoRoot` — defined CID after init, changes after write
62
+
- `GetRepoRev` — non-empty, changes after write
63
+
- `ReadRepo` — non-empty CAR output, incremental export with `since`
64
+
- `InitNewActor` — empty DID error, zero user error, event emission with hydration
65
+
- Event emission — create/update/delete events verified: prevData, ops, oldRoot, newRoot, rev, since, repoSlice, hydration
66
+
67
+
### Coverage ✅
68
+
All RepoOperator methods 81–100%. Remaining uncovered lines are internal infrastructure error guards (`GetUserRepoRev`, `NewDeltaSession`, `OpenRepo`, `Commit`, `CloseWithRoot`) — not reachable without mocking the carstore.
69
+
70
+
### Files modified ✅
71
+
- `pkg/hold/pds/repo_operator_test.go` — new file, 38 subtests
72
+
73
+
### Verification ✅
74
+
- `make lint` — 0 issues
75
+
- `make test` — all tests pass
76
+
77
+
---
78
+
79
+
## Phase 2: Build new implementation ✅
80
+
81
+
**Goal:** Create `DirectRepoOperator` using `indigo/repo` directly (cocoon pattern).
82
+
83
+
**Completed 2026-02-28.** New `pkg/hold/pds/repo.go`: 548 lines (vs 852 in repomgr.go, ~36% reduction). All 37 subtests pass identically for both implementations. Race detector, shuffled order, and parallel execution all clean.
84
+
85
+
### New file: `pkg/hold/pds/repo.go` ✅
86
+
87
+
Key differences from vendored repomgr:
88
+
- **Single `sync.Mutex`** instead of per-user lock map (`lklk` + `userLocks` + `userLock` struct + reference counting)
89
+
- **No OpenTelemetry tracing** (`otel.Tracer` calls removed)
90
+
- **No `gorm` dependency** (`RepoHead` struct removed, `gorm.io/gorm` dropped from go.mod)
91
+
- **`openWriteSession` / `commitWrite` helpers** extract the repeated 6-step write pattern
92
+
93
+
Core mutation pattern (same as current, just cleaner):
94
+
1. Lock → get rev → open delta session → open repo
95
+
2. Capture `r.DataCid()` for prevData
96
+
3. Perform operation(s)
97
+
4. `r.Commit()` → `ds.CloseWithRoot()` → emit event → unlock
98
+
99
+
### Shared types moved to `repo_operator.go` ✅
100
+
- `KeyManager` interface and `ActorInfo` struct moved from `repomgr.go`
101
+
- Both implementations import from the same location
102
+
103
+
### Test wiring ✅
104
+
- `setupTestDirectRepoOperator` — creates carstore + key manager directly (no `NewHoldPDS`)
105
+
- `runRepoOperatorTests` refactored to accept optional `freshSetup` for `InitNewActor_EventEmission`
106
+
- `TestDirectRepoOperator` runs all 37 subtests identically
107
+
108
+
### Verification ✅
109
+
- `go build ./cmd/hold` — compiles
110
+
- `TestRepoManager` — 37/37 subtests pass
111
+
- `TestDirectRepoOperator` — 37/37 subtests pass
112
+
- `-race -shuffle=on -count=5 -parallel=8` — all clean
113
+
- `make lint` — 0 issues
114
+
- `make test` — all tests pass
115
+
116
+
---
117
+
118
+
## Phase 3: Config flag + switchover
119
+
120
+
**Goal:** Feature flag to select implementation, default old.
121
+
122
+
### Changes:
123
+
- `pkg/hold/config.go` — add `UseDirectRepo bool` to DatabaseConfig
124
+
- `pkg/hold/pds/server.go` — select implementation based on config in `NewHoldPDS`/`NewHoldPDSWithDB`
125
+
- Regenerate example configs
126
+
127
+
### Verification:
128
+
- Deploy with `use_direct_repo: false` (default)
129
+
- Test with `use_direct_repo: true` in staging
130
+
- `make lint && make test`
131
+
132
+
---
133
+
134
+
## Phase 4: Remove vendored repomgr
135
+
136
+
**Goal:** After production validation, delete the old code.
137
+
138
+
- Delete `repomgr.go`
139
+
- Remove config flag, make `DirectRepoOperator` the only implementation
140
+
- Rename `repo_direct.go` → `repo_operator_impl.go`
141
+
- Regenerate example configs
142
+
- `make lint && make test`
143
+
144
+
---
145
+
146
+
## Decision log
147
+
148
+
- **`indigo/repo` over `atproto/repo`**: `atproto/repo` has the MST primitives (`Insert`, `Remove`, `ApplyOp`) but no high-level PDS API (`OpenRepo`, `CreateRecord`, `Commit(signFn)`). Its own doc.go says "does not yet work for implementing a repository host (PDS)." `indigo/repo` is what the reference PDS and cocoon use. The `RepoOperator` interface means we can swap later if `atproto/repo` adds PDS support.
+2
-2
go.mod
+2
-2
go.mod
···
25
25
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674
26
26
github.com/ipfs/go-block-format v0.2.3
27
27
github.com/ipfs/go-cid v0.6.0
28
-
github.com/ipfs/go-datastore v0.9.1
29
28
github.com/ipfs/go-ipfs-blockstore v1.3.1
30
29
github.com/ipfs/go-ipld-cbor v0.2.1
31
30
github.com/ipfs/go-ipld-format v0.6.3
···
50
49
golang.org/x/image v0.36.0
51
50
golang.org/x/sys v0.41.0
52
51
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da
53
-
gorm.io/gorm v1.31.1
54
52
)
55
53
56
54
require (
···
120
118
github.com/ipfs/bbloom v0.0.4 // indirect
121
119
github.com/ipfs/boxo v0.36.0 // indirect
122
120
github.com/ipfs/go-cidutil v0.1.1 // indirect
121
+
github.com/ipfs/go-datastore v0.9.1 // indirect
123
122
github.com/ipfs/go-dsqueue v0.2.0 // indirect
124
123
github.com/ipfs/go-ipfs-ds-help v1.1.1 // indirect
125
124
github.com/ipfs/go-ipfs-util v0.0.3 // indirect
···
213
212
google.golang.org/protobuf v1.36.11 // indirect
214
213
gopkg.in/yaml.v2 v2.4.0 // indirect
215
214
gopkg.in/yaml.v3 v3.0.1 // indirect
215
+
gorm.io/gorm v1.31.1 // indirect
216
216
lukechampine.com/blake3 v1.4.1 // indirect
217
217
)
+4
-3
pkg/hold/pds/events.go
+4
-3
pkg/hold/pds/events.go
···
56
56
Rev string `json:"rev" cborgen:"rev"`
57
57
Since *string `json:"since,omitempty" cborgen:"since,omitempty"`
58
58
PrevData string `json:"prevData,omitempty" cborgen:"prevData,omitempty"` // MST root CID string of previous commit
59
-
Blocks []byte `json:"blocks" cborgen:"blocks"` // CAR slice bytes
59
+
Blocks []byte `json:"blocks" cborgen:"blocks"` // CAR slice bytes
60
60
Ops []*atproto.SyncSubscribeRepos_RepoOp `json:"ops" cborgen:"ops"`
61
61
Time string `json:"time" cborgen:"time"`
62
62
Type string `json:"$type" cborgen:"$type"` // Always "#commit"
···
170
170
}
171
171
}
172
172
173
-
// Migration: add prev_data column if missing (existing databases)
174
-
b.db.Exec("ALTER TABLE firehose_events ADD COLUMN prev_data TEXT")
173
+
// Migration: add prev_data column if missing (existing databases).
174
+
// Intentionally ignore error — fails with "duplicate column" if already present.
175
+
_, _ = b.db.Exec("ALTER TABLE firehose_events ADD COLUMN prev_data TEXT")
175
176
176
177
// Load last sequence number from database
177
178
var lastSeq sql.NullInt64
+4
-104
pkg/hold/pds/import.go
+4
-104
pkg/hold/pds/import.go
···
8
8
9
9
"github.com/bluesky-social/indigo/repo"
10
10
"github.com/ipfs/go-cid"
11
-
"go.opentelemetry.io/otel"
12
11
)
13
12
14
13
// rawCBOR wraps raw bytes to satisfy cbg.CBORMarshaler.
···
20
19
return err
21
20
}
22
21
23
-
// bulkRecord holds a single record to import.
24
-
type bulkRecord struct {
25
-
Collection string
26
-
Rkey string
27
-
Data rawCBOR
28
-
}
29
-
30
22
// ImportResult summarizes a CAR import operation.
31
23
type ImportResult struct {
32
24
Total int
···
52
44
}
53
45
54
46
// Collect all records
55
-
var records []bulkRecord
47
+
var records []BulkRecord
56
48
err = sourceRepo.ForEach(ctx, "", func(k string, v cid.Cid) error {
57
49
_, recBytes, err := sourceRepo.GetRecordBytes(ctx, k)
58
50
if err != nil {
···
64
56
return fmt.Errorf("unexpected record path format: %s", k)
65
57
}
66
58
67
-
records = append(records, bulkRecord{
59
+
records = append(records, BulkRecord{
68
60
Collection: parts[0],
69
61
Rkey: parts[1],
70
62
Data: rawCBOR(*recBytes),
···
79
71
return &ImportResult{PerCollection: map[string]int{}}, nil
80
72
}
81
73
82
-
// Bulk upsert all records in a single commit
83
-
if err := p.bulkImportRecords(ctx, records); err != nil {
74
+
// Bulk upsert all records in a single commit via the RepoOperator interface
75
+
if err := p.repomgr.BulkUpsert(ctx, p.uid, records); err != nil {
84
76
return nil, fmt.Errorf("failed to import records: %w", err)
85
77
}
86
78
···
94
86
}
95
87
return result, nil
96
88
}
97
-
98
-
// bulkImportRecords writes all records in a single delta session + commit.
99
-
// Each record is upserted: created if new, updated if exists.
100
-
func (p *HoldPDS) bulkImportRecords(ctx context.Context, records []bulkRecord) error {
101
-
ctx, span := otel.Tracer("repoman").Start(ctx, "BulkImportRecords")
102
-
defer span.End()
103
-
104
-
unlock := p.repomgr.lockUser(ctx, p.uid)
105
-
defer unlock()
106
-
107
-
rev, err := p.repomgr.cs.GetUserRepoRev(ctx, p.uid)
108
-
if err != nil {
109
-
return err
110
-
}
111
-
112
-
ds, err := p.repomgr.cs.NewDeltaSession(ctx, p.uid, &rev)
113
-
if err != nil {
114
-
return err
115
-
}
116
-
117
-
head := ds.BaseCid()
118
-
r, err := repo.OpenRepo(ctx, ds, head)
119
-
if err != nil {
120
-
return err
121
-
}
122
-
123
-
// Capture previous MST root before commit overwrites it
124
-
var prevData *cid.Cid
125
-
if head.Defined() {
126
-
pd := r.DataCid()
127
-
prevData = &pd
128
-
}
129
-
130
-
ops := make([]RepoOp, 0, len(records))
131
-
for _, rec := range records {
132
-
rpath := rec.Collection + "/" + rec.Rkey
133
-
134
-
// Check if record exists to determine create vs update
135
-
_, _, getErr := r.GetRecordBytes(ctx, rpath)
136
-
recordExists := getErr == nil
137
-
138
-
var cc cid.Cid
139
-
var evtKind EventKind
140
-
if recordExists {
141
-
cc, err = r.UpdateRecord(ctx, rpath, rec.Data)
142
-
evtKind = EvtKindUpdateRecord
143
-
} else {
144
-
cc, err = r.PutRecord(ctx, rpath, rec.Data)
145
-
evtKind = EvtKindCreateRecord
146
-
}
147
-
if err != nil {
148
-
return fmt.Errorf("failed to write %s: %w", rpath, err)
149
-
}
150
-
151
-
ops = append(ops, RepoOp{
152
-
Kind: evtKind,
153
-
Collection: rec.Collection,
154
-
Rkey: rec.Rkey,
155
-
RecCid: &cc,
156
-
})
157
-
}
158
-
159
-
nroot, nrev, err := r.Commit(ctx, p.repomgr.kmgr.SignForUser)
160
-
if err != nil {
161
-
return err
162
-
}
163
-
164
-
rslice, err := ds.CloseWithRoot(ctx, nroot, nrev)
165
-
if err != nil {
166
-
return fmt.Errorf("close with root: %w", err)
167
-
}
168
-
169
-
var oldroot *cid.Cid
170
-
if head.Defined() {
171
-
oldroot = &head
172
-
}
173
-
174
-
if p.repomgr.events != nil {
175
-
p.repomgr.events(ctx, &RepoEvent{
176
-
User: p.uid,
177
-
OldRoot: oldroot,
178
-
NewRoot: nroot,
179
-
PrevData: prevData,
180
-
Rev: nrev,
181
-
Since: &rev,
182
-
Ops: ops,
183
-
RepoSlice: rslice,
184
-
})
185
-
}
186
-
187
-
return nil
188
-
}
+548
pkg/hold/pds/repo.go
+548
pkg/hold/pds/repo.go
···
1
+
package pds
2
+
3
+
// repo.go — DirectRepoOperator manages ATProto repository operations using
4
+
// indigo/repo directly, replacing RepoManager with a simpler implementation.
5
+
//
6
+
// Key simplifications vs RepoManager:
7
+
// - Single sync.Mutex instead of per-user lock map (hold is always uid=1)
8
+
// - No OpenTelemetry tracing
9
+
// - No gorm dependency
10
+
//
11
+
// Implements the RepoOperator interface (see repo_operator.go).
12
+
// See docs/REPOMGR_MIGRATION.md for the migration plan.
13
+
14
+
import (
15
+
"context"
16
+
"fmt"
17
+
"io"
18
+
"log/slog"
19
+
"sync"
20
+
21
+
holddb "atcr.io/pkg/hold/db"
22
+
atproto "github.com/bluesky-social/indigo/api/atproto"
23
+
bsky "github.com/bluesky-social/indigo/api/bsky"
24
+
"github.com/bluesky-social/indigo/atproto/syntax"
25
+
"github.com/bluesky-social/indigo/models"
26
+
"github.com/bluesky-social/indigo/repo"
27
+
"github.com/bluesky-social/indigo/util"
28
+
29
+
blocks "github.com/ipfs/go-block-format"
30
+
"github.com/ipfs/go-cid"
31
+
cbg "github.com/whyrusleeping/cbor-gen"
32
+
)
33
+
34
+
// Compile-time check that DirectRepoOperator implements RepoOperator.
35
+
var _ RepoOperator = (*DirectRepoOperator)(nil)
36
+
37
+
// DirectRepoOperator implements RepoOperator using indigo/repo directly.
38
+
type DirectRepoOperator struct {
39
+
cs holddb.CarStore
40
+
kmgr KeyManager
41
+
mu sync.Mutex // single mutex (hold is always uid=1)
42
+
events func(context.Context, *RepoEvent)
43
+
hydrateRecords bool
44
+
log *slog.Logger
45
+
clk *syntax.TIDClock // for BatchWrite auto-rkey generation
46
+
}
47
+
48
+
// NewDirectRepoOperator creates a new DirectRepoOperator.
49
+
func NewDirectRepoOperator(cs holddb.CarStore, kmgr KeyManager) *DirectRepoOperator {
50
+
return &DirectRepoOperator{
51
+
cs: cs,
52
+
kmgr: kmgr,
53
+
log: slog.Default().With("system", "repo"),
54
+
clk: syntax.NewTIDClock(0),
55
+
}
56
+
}
57
+
58
+
// writeSession holds state for an in-progress write transaction.
59
+
type writeSession struct {
60
+
ds *holddb.DeltaSession
61
+
r *repo.Repo
62
+
head cid.Cid
63
+
prevData *cid.Cid
64
+
rev string
65
+
}
66
+
67
+
// openWriteSession opens a write session, acquiring the lock.
68
+
// On success, the caller holds d.mu and must defer d.mu.Unlock().
69
+
// On error, the lock is released before returning.
70
+
func (d *DirectRepoOperator) openWriteSession(ctx context.Context, user models.Uid) (*writeSession, error) {
71
+
d.mu.Lock()
72
+
73
+
rev, err := d.cs.GetUserRepoRev(ctx, user)
74
+
if err != nil {
75
+
d.mu.Unlock()
76
+
return nil, err
77
+
}
78
+
79
+
ds, err := d.cs.NewDeltaSession(ctx, user, &rev)
80
+
if err != nil {
81
+
d.mu.Unlock()
82
+
return nil, err
83
+
}
84
+
85
+
head := ds.BaseCid()
86
+
87
+
r, err := repo.OpenRepo(ctx, ds, head)
88
+
if err != nil {
89
+
d.mu.Unlock()
90
+
return nil, err
91
+
}
92
+
93
+
var prevData *cid.Cid
94
+
if head.Defined() {
95
+
pd := r.DataCid()
96
+
prevData = &pd
97
+
}
98
+
99
+
return &writeSession{
100
+
ds: ds,
101
+
r: r,
102
+
head: head,
103
+
prevData: prevData,
104
+
rev: rev,
105
+
}, nil
106
+
}
107
+
108
+
// commitWrite commits a write session and emits an event if configured.
109
+
func (d *DirectRepoOperator) commitWrite(ctx context.Context, ws *writeSession, user models.Uid, ops []RepoOp) (cid.Cid, string, error) {
110
+
nroot, nrev, err := ws.r.Commit(ctx, d.kmgr.SignForUser)
111
+
if err != nil {
112
+
return cid.Undef, "", err
113
+
}
114
+
115
+
rslice, err := ws.ds.CloseWithRoot(ctx, nroot, nrev)
116
+
if err != nil {
117
+
return cid.Undef, "", fmt.Errorf("close with root: %w", err)
118
+
}
119
+
120
+
if d.events != nil {
121
+
var oldroot *cid.Cid
122
+
if ws.head.Defined() {
123
+
oldroot = &ws.head
124
+
}
125
+
126
+
d.events(ctx, &RepoEvent{
127
+
User: user,
128
+
OldRoot: oldroot,
129
+
NewRoot: nroot,
130
+
PrevData: ws.prevData,
131
+
Rev: nrev,
132
+
Since: &ws.rev,
133
+
Ops: ops,
134
+
RepoSlice: rslice,
135
+
})
136
+
}
137
+
138
+
return nroot, nrev, nil
139
+
}
140
+
141
+
func (d *DirectRepoOperator) SetEventHandler(cb func(context.Context, *RepoEvent), hydrateRecords bool) {
142
+
d.events = cb
143
+
d.hydrateRecords = hydrateRecords
144
+
}
145
+
146
+
func (d *DirectRepoOperator) CreateRecord(ctx context.Context, user models.Uid, collection string, rec cbg.CBORMarshaler) (string, cid.Cid, error) {
147
+
ws, err := d.openWriteSession(ctx, user)
148
+
if err != nil {
149
+
return "", cid.Undef, err
150
+
}
151
+
defer d.mu.Unlock()
152
+
153
+
cc, tid, err := ws.r.CreateRecord(ctx, collection, rec)
154
+
if err != nil {
155
+
return "", cid.Undef, err
156
+
}
157
+
158
+
ops := []RepoOp{{
159
+
Kind: EvtKindCreateRecord,
160
+
Collection: collection,
161
+
Rkey: tid,
162
+
Record: rec, // CreateRecord always includes Record (no hydration check)
163
+
RecCid: &cc,
164
+
}}
165
+
166
+
if _, _, err := d.commitWrite(ctx, ws, user, ops); err != nil {
167
+
return "", cid.Undef, err
168
+
}
169
+
170
+
return collection + "/" + tid, cc, nil
171
+
}
172
+
173
+
func (d *DirectRepoOperator) UpdateRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (cid.Cid, error) {
174
+
ws, err := d.openWriteSession(ctx, user)
175
+
if err != nil {
176
+
return cid.Undef, err
177
+
}
178
+
defer d.mu.Unlock()
179
+
180
+
rpath := collection + "/" + rkey
181
+
cc, err := ws.r.UpdateRecord(ctx, rpath, rec)
182
+
if err != nil {
183
+
return cid.Undef, err
184
+
}
185
+
186
+
op := RepoOp{
187
+
Kind: EvtKindUpdateRecord,
188
+
Collection: collection,
189
+
Rkey: rkey,
190
+
RecCid: &cc,
191
+
}
192
+
if d.hydrateRecords {
193
+
op.Record = rec
194
+
}
195
+
196
+
if _, _, err := d.commitWrite(ctx, ws, user, []RepoOp{op}); err != nil {
197
+
return cid.Undef, err
198
+
}
199
+
200
+
return cc, nil
201
+
}
202
+
203
+
func (d *DirectRepoOperator) PutRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, error) {
204
+
ws, err := d.openWriteSession(ctx, user)
205
+
if err != nil {
206
+
return "", cid.Undef, err
207
+
}
208
+
defer d.mu.Unlock()
209
+
210
+
rpath := collection + "/" + rkey
211
+
cc, err := ws.r.PutRecord(ctx, rpath, rec)
212
+
if err != nil {
213
+
return "", cid.Undef, err
214
+
}
215
+
216
+
op := RepoOp{
217
+
Kind: EvtKindCreateRecord,
218
+
Collection: collection,
219
+
Rkey: rkey,
220
+
RecCid: &cc,
221
+
}
222
+
if d.hydrateRecords {
223
+
op.Record = rec
224
+
}
225
+
226
+
if _, _, err := d.commitWrite(ctx, ws, user, []RepoOp{op}); err != nil {
227
+
return "", cid.Undef, err
228
+
}
229
+
230
+
return rpath, cc, nil
231
+
}
232
+
233
+
func (d *DirectRepoOperator) UpsertRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, bool, error) {
234
+
ws, err := d.openWriteSession(ctx, user)
235
+
if err != nil {
236
+
return "", cid.Undef, false, err
237
+
}
238
+
defer d.mu.Unlock()
239
+
240
+
rpath := collection + "/" + rkey
241
+
242
+
// Check if record exists
243
+
_, _, err = ws.r.GetRecordBytes(ctx, rpath)
244
+
recordExists := err == nil
245
+
246
+
var cc cid.Cid
247
+
var evtKind EventKind
248
+
if recordExists {
249
+
cc, err = ws.r.UpdateRecord(ctx, rpath, rec)
250
+
evtKind = EvtKindUpdateRecord
251
+
} else {
252
+
cc, err = ws.r.PutRecord(ctx, rpath, rec)
253
+
evtKind = EvtKindCreateRecord
254
+
}
255
+
if err != nil {
256
+
return "", cid.Undef, false, err
257
+
}
258
+
259
+
op := RepoOp{
260
+
Kind: evtKind,
261
+
Collection: collection,
262
+
Rkey: rkey,
263
+
RecCid: &cc,
264
+
}
265
+
if d.hydrateRecords {
266
+
op.Record = rec
267
+
}
268
+
269
+
if _, _, err := d.commitWrite(ctx, ws, user, []RepoOp{op}); err != nil {
270
+
return "", cid.Undef, false, err
271
+
}
272
+
273
+
return rpath, cc, !recordExists, nil
274
+
}
275
+
276
+
func (d *DirectRepoOperator) DeleteRecord(ctx context.Context, user models.Uid, collection, rkey string) error {
277
+
ws, err := d.openWriteSession(ctx, user)
278
+
if err != nil {
279
+
return err
280
+
}
281
+
defer d.mu.Unlock()
282
+
283
+
rpath := collection + "/" + rkey
284
+
if err := ws.r.DeleteRecord(ctx, rpath); err != nil {
285
+
return err
286
+
}
287
+
288
+
ops := []RepoOp{{
289
+
Kind: EvtKindDeleteRecord,
290
+
Collection: collection,
291
+
Rkey: rkey,
292
+
}}
293
+
294
+
_, _, err = d.commitWrite(ctx, ws, user, ops)
295
+
return err
296
+
}
297
+
298
+
func (d *DirectRepoOperator) BatchWrite(ctx context.Context, user models.Uid, writes []*atproto.RepoApplyWrites_Input_Writes_Elem) error {
299
+
ws, err := d.openWriteSession(ctx, user)
300
+
if err != nil {
301
+
return err
302
+
}
303
+
defer d.mu.Unlock()
304
+
305
+
ops := make([]RepoOp, 0, len(writes))
306
+
for _, w := range writes {
307
+
switch {
308
+
case w.RepoApplyWrites_Create != nil:
309
+
c := w.RepoApplyWrites_Create
310
+
var rkey string
311
+
if c.Rkey != nil {
312
+
rkey = *c.Rkey
313
+
} else {
314
+
rkey = d.clk.Next().String()
315
+
}
316
+
317
+
nsid := c.Collection + "/" + rkey
318
+
cc, err := ws.r.PutRecord(ctx, nsid, c.Value.Val)
319
+
if err != nil {
320
+
return err
321
+
}
322
+
323
+
op := RepoOp{
324
+
Kind: EvtKindCreateRecord,
325
+
Collection: c.Collection,
326
+
Rkey: rkey,
327
+
RecCid: &cc,
328
+
}
329
+
if d.hydrateRecords {
330
+
op.Record = c.Value.Val
331
+
}
332
+
ops = append(ops, op)
333
+
334
+
case w.RepoApplyWrites_Update != nil:
335
+
u := w.RepoApplyWrites_Update
336
+
337
+
// Known quirk: uses PutRecord (mst.Add) not UpdateRecord
338
+
cc, err := ws.r.PutRecord(ctx, u.Collection+"/"+u.Rkey, u.Value.Val)
339
+
if err != nil {
340
+
return err
341
+
}
342
+
343
+
op := RepoOp{
344
+
Kind: EvtKindUpdateRecord,
345
+
Collection: u.Collection,
346
+
Rkey: u.Rkey,
347
+
RecCid: &cc,
348
+
}
349
+
if d.hydrateRecords {
350
+
op.Record = u.Value.Val
351
+
}
352
+
ops = append(ops, op)
353
+
354
+
case w.RepoApplyWrites_Delete != nil:
355
+
del := w.RepoApplyWrites_Delete
356
+
357
+
if err := ws.r.DeleteRecord(ctx, del.Collection+"/"+del.Rkey); err != nil {
358
+
return err
359
+
}
360
+
361
+
ops = append(ops, RepoOp{
362
+
Kind: EvtKindDeleteRecord,
363
+
Collection: del.Collection,
364
+
Rkey: del.Rkey,
365
+
})
366
+
367
+
default:
368
+
return fmt.Errorf("no operation set in write enum")
369
+
}
370
+
}
371
+
372
+
_, _, err = d.commitWrite(ctx, ws, user, ops)
373
+
return err
374
+
}
375
+
376
+
func (d *DirectRepoOperator) BulkUpsert(ctx context.Context, user models.Uid, records []BulkRecord) error {
377
+
ws, err := d.openWriteSession(ctx, user)
378
+
if err != nil {
379
+
return err
380
+
}
381
+
defer d.mu.Unlock()
382
+
383
+
ops := make([]RepoOp, 0, len(records))
384
+
for _, rec := range records {
385
+
rpath := rec.Collection + "/" + rec.Rkey
386
+
387
+
// Check if record exists to determine create vs update
388
+
_, _, getErr := ws.r.GetRecordBytes(ctx, rpath)
389
+
recordExists := getErr == nil
390
+
391
+
var cc cid.Cid
392
+
var evtKind EventKind
393
+
if recordExists {
394
+
cc, err = ws.r.UpdateRecord(ctx, rpath, rec.Data)
395
+
evtKind = EvtKindUpdateRecord
396
+
} else {
397
+
cc, err = ws.r.PutRecord(ctx, rpath, rec.Data)
398
+
evtKind = EvtKindCreateRecord
399
+
}
400
+
if err != nil {
401
+
return fmt.Errorf("failed to write %s: %w", rpath, err)
402
+
}
403
+
404
+
// No hydration for BulkUpsert (records never included in events)
405
+
ops = append(ops, RepoOp{
406
+
Kind: evtKind,
407
+
Collection: rec.Collection,
408
+
Rkey: rec.Rkey,
409
+
RecCid: &cc,
410
+
})
411
+
}
412
+
413
+
_, _, err = d.commitWrite(ctx, ws, user, ops)
414
+
return err
415
+
}
416
+
417
+
func (d *DirectRepoOperator) InitNewActor(ctx context.Context, user models.Uid, handle, did, displayname string, declcid, actortype string) error {
418
+
d.mu.Lock()
419
+
defer d.mu.Unlock()
420
+
421
+
if did == "" {
422
+
return fmt.Errorf("must specify DID for new actor")
423
+
}
424
+
425
+
if user == 0 {
426
+
return fmt.Errorf("must specify user for new actor")
427
+
}
428
+
429
+
ds, err := d.cs.NewDeltaSession(ctx, user, nil)
430
+
if err != nil {
431
+
return fmt.Errorf("creating new delta session: %w", err)
432
+
}
433
+
434
+
r := repo.NewRepo(ctx, did, ds)
435
+
436
+
profile := &bsky.ActorProfile{
437
+
DisplayName: &displayname,
438
+
}
439
+
440
+
_, err = r.PutRecord(ctx, "app.bsky.actor.profile/self", profile)
441
+
if err != nil {
442
+
return fmt.Errorf("setting initial actor profile: %w", err)
443
+
}
444
+
445
+
root, nrev, err := r.Commit(ctx, d.kmgr.SignForUser)
446
+
if err != nil {
447
+
return fmt.Errorf("committing repo for actor init: %w", err)
448
+
}
449
+
450
+
rslice, err := ds.CloseWithRoot(ctx, root, nrev)
451
+
if err != nil {
452
+
return fmt.Errorf("close with root: %w", err)
453
+
}
454
+
455
+
if d.events != nil {
456
+
op := RepoOp{
457
+
Kind: EvtKindCreateRecord,
458
+
Collection: "app.bsky.actor.profile",
459
+
Rkey: "self",
460
+
}
461
+
462
+
if d.hydrateRecords {
463
+
op.Record = profile
464
+
}
465
+
466
+
d.events(ctx, &RepoEvent{
467
+
User: user,
468
+
NewRoot: root,
469
+
Rev: nrev,
470
+
Ops: []RepoOp{op},
471
+
RepoSlice: rslice,
472
+
})
473
+
}
474
+
475
+
return nil
476
+
}
477
+
478
+
func (d *DirectRepoOperator) GetRecord(ctx context.Context, user models.Uid, collection string, rkey string, maybeCid cid.Cid) (cid.Cid, cbg.CBORMarshaler, error) {
479
+
bs, err := d.cs.ReadOnlySession(user)
480
+
if err != nil {
481
+
return cid.Undef, nil, err
482
+
}
483
+
484
+
head, err := d.cs.GetUserRepoHead(ctx, user)
485
+
if err != nil {
486
+
return cid.Undef, nil, err
487
+
}
488
+
489
+
r, err := repo.OpenRepo(ctx, bs, head)
490
+
if err != nil {
491
+
return cid.Undef, nil, err
492
+
}
493
+
494
+
ocid, val, err := r.GetRecord(ctx, collection+"/"+rkey)
495
+
if err != nil {
496
+
return cid.Undef, nil, err
497
+
}
498
+
499
+
if maybeCid.Defined() && ocid != maybeCid {
500
+
return cid.Undef, nil, fmt.Errorf("record at specified key had different CID than expected")
501
+
}
502
+
503
+
return ocid, val, nil
504
+
}
505
+
506
+
func (d *DirectRepoOperator) GetRecordProof(ctx context.Context, user models.Uid, collection string, rkey string) (cid.Cid, []blocks.Block, error) {
507
+
robs, err := d.cs.ReadOnlySession(user)
508
+
if err != nil {
509
+
return cid.Undef, nil, err
510
+
}
511
+
512
+
bs := util.NewLoggingBstore(robs)
513
+
514
+
head, err := d.cs.GetUserRepoHead(ctx, user)
515
+
if err != nil {
516
+
return cid.Undef, nil, err
517
+
}
518
+
519
+
r, err := repo.OpenRepo(ctx, bs, head)
520
+
if err != nil {
521
+
return cid.Undef, nil, err
522
+
}
523
+
524
+
_, _, err = r.GetRecordBytes(ctx, collection+"/"+rkey)
525
+
if err != nil {
526
+
return cid.Undef, nil, err
527
+
}
528
+
529
+
return head, bs.GetLoggedBlocks(), nil
530
+
}
531
+
532
+
func (d *DirectRepoOperator) GetRepoRoot(ctx context.Context, user models.Uid) (cid.Cid, error) {
533
+
d.mu.Lock()
534
+
defer d.mu.Unlock()
535
+
536
+
return d.cs.GetUserRepoHead(ctx, user)
537
+
}
538
+
539
+
func (d *DirectRepoOperator) GetRepoRev(ctx context.Context, user models.Uid) (string, error) {
540
+
d.mu.Lock()
541
+
defer d.mu.Unlock()
542
+
543
+
return d.cs.GetUserRepoRev(ctx, user)
544
+
}
545
+
546
+
func (d *DirectRepoOperator) ReadRepo(ctx context.Context, user models.Uid, since string, w io.Writer) error {
547
+
return d.cs.ReadUserCar(ctx, user, since, true, w)
548
+
}
+96
pkg/hold/pds/repo_operator.go
+96
pkg/hold/pds/repo_operator.go
···
1
+
// Package pds implements a minimal ATProto PDS for the hold service.
2
+
package pds
3
+
4
+
import (
5
+
"context"
6
+
"io"
7
+
8
+
atproto "github.com/bluesky-social/indigo/api/atproto"
9
+
"github.com/bluesky-social/indigo/models"
10
+
11
+
blocks "github.com/ipfs/go-block-format"
12
+
"github.com/ipfs/go-cid"
13
+
cbg "github.com/whyrusleeping/cbor-gen"
14
+
)
15
+
16
+
// RepoOperator defines the interface for ATProto repository operations.
17
+
// RepoManager implements this interface. Future implementations (e.g., using
18
+
// indigo/repo directly) can be swapped in behind this interface.
19
+
// See docs/REPOMGR_MIGRATION.md for the migration plan.
20
+
type RepoOperator interface {
21
+
// Record CRUD
22
+
CreateRecord(ctx context.Context, user models.Uid, collection string, rec cbg.CBORMarshaler) (string, cid.Cid, error)
23
+
UpdateRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (cid.Cid, error)
24
+
PutRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, error)
25
+
UpsertRecord(ctx context.Context, user models.Uid, collection, rkey string, rec cbg.CBORMarshaler) (string, cid.Cid, bool, error)
26
+
DeleteRecord(ctx context.Context, user models.Uid, collection, rkey string) error
27
+
BatchWrite(ctx context.Context, user models.Uid, writes []*atproto.RepoApplyWrites_Input_Writes_Elem) error
28
+
BulkUpsert(ctx context.Context, user models.Uid, records []BulkRecord) error
29
+
30
+
// Read
31
+
GetRecord(ctx context.Context, user models.Uid, collection string, rkey string, maybeCid cid.Cid) (cid.Cid, cbg.CBORMarshaler, error)
32
+
GetRecordProof(ctx context.Context, user models.Uid, collection string, rkey string) (cid.Cid, []blocks.Block, error)
33
+
GetRepoRoot(ctx context.Context, user models.Uid) (cid.Cid, error)
34
+
GetRepoRev(ctx context.Context, user models.Uid) (string, error)
35
+
ReadRepo(ctx context.Context, user models.Uid, since string, w io.Writer) error
36
+
37
+
// Lifecycle
38
+
InitNewActor(ctx context.Context, user models.Uid, handle, did, displayname string, declcid, actortype string) error
39
+
SetEventHandler(cb func(context.Context, *RepoEvent), hydrateRecords bool)
40
+
}
41
+
42
+
// KeyManager handles cryptographic signing for repository commits.
43
+
type KeyManager interface {
44
+
VerifyUserSignature(context.Context, string, []byte, []byte) error
45
+
SignForUser(context.Context, string, []byte) ([]byte, error)
46
+
}
47
+
48
+
// ActorInfo holds identity information for a repository actor.
49
+
type ActorInfo struct {
50
+
Did string
51
+
Handle string
52
+
DisplayName string
53
+
Type string
54
+
}
55
+
56
+
// Compile-time check that RepoManager implements RepoOperator.
57
+
var _ RepoOperator = (*RepoManager)(nil)
58
+
59
+
// RepoEvent represents a mutation event emitted by a RepoOperator.
60
+
type RepoEvent struct {
61
+
User models.Uid
62
+
OldRoot *cid.Cid
63
+
NewRoot cid.Cid
64
+
PrevData *cid.Cid // MST root CID of the previous commit (for firehose prevData field)
65
+
Since *string
66
+
Rev string
67
+
RepoSlice []byte
68
+
PDS uint
69
+
Ops []RepoOp
70
+
}
71
+
72
+
// RepoOp represents a single operation within a RepoEvent.
73
+
type RepoOp struct {
74
+
Kind EventKind
75
+
Collection string
76
+
Rkey string
77
+
RecCid *cid.Cid
78
+
Record any
79
+
ActorInfo *ActorInfo
80
+
}
81
+
82
+
// EventKind identifies the type of repository mutation.
83
+
type EventKind string
84
+
85
+
const (
86
+
EvtKindCreateRecord = EventKind("create")
87
+
EvtKindUpdateRecord = EventKind("update")
88
+
EvtKindDeleteRecord = EventKind("delete")
89
+
)
90
+
91
+
// BulkRecord holds a single record for bulk import/upsert operations.
92
+
type BulkRecord struct {
93
+
Collection string
94
+
Rkey string
95
+
Data cbg.CBORMarshaler
96
+
}
+418
pkg/hold/pds/repo_operator_benchmark_test.go
+418
pkg/hold/pds/repo_operator_benchmark_test.go
···
1
+
package pds
2
+
3
+
import (
4
+
"bytes"
5
+
"context"
6
+
"fmt"
7
+
"io"
8
+
"log/slog"
9
+
"os"
10
+
"path/filepath"
11
+
"testing"
12
+
13
+
"atcr.io/pkg/atproto"
14
+
"atcr.io/pkg/auth/oauth"
15
+
holddb "atcr.io/pkg/hold/db"
16
+
"github.com/bluesky-social/indigo/models"
17
+
"github.com/ipfs/go-cid"
18
+
19
+
indigoatproto "github.com/bluesky-social/indigo/api/atproto"
20
+
lexutil "github.com/bluesky-social/indigo/lex/util"
21
+
)
22
+
23
+
// benchSetup creates a RepoOperator and returns it with the user ID.
24
+
type benchSetup func(b *testing.B) (RepoOperator, models.Uid)
25
+
26
+
func suppressLogs(b *testing.B) {
27
+
b.Helper()
28
+
prev := slog.Default()
29
+
slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, nil)))
30
+
b.Cleanup(func() { slog.SetDefault(prev) })
31
+
}
32
+
33
+
func setupBenchRepoManager(b *testing.B) (RepoOperator, models.Uid) {
34
+
b.Helper()
35
+
suppressLogs(b)
36
+
ctx := context.Background()
37
+
tmpDir := b.TempDir()
38
+
keyPath := filepath.Join(tmpDir, "signing-key")
39
+
40
+
if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil {
41
+
b.Fatalf("write signing key: %v", err)
42
+
}
43
+
44
+
pds, err := NewHoldPDS(ctx, "did:web:hold.bench", "https://hold.bench", "https://atcr.io", ":memory:", keyPath, false)
45
+
if err != nil {
46
+
b.Fatalf("NewHoldPDS: %v", err)
47
+
}
48
+
49
+
if err := pds.repomgr.InitNewActor(ctx, pds.uid, "", pds.did, "", "", ""); err != nil {
50
+
b.Fatalf("InitNewActor: %v", err)
51
+
}
52
+
53
+
b.Cleanup(func() { pds.Close() })
54
+
return pds.repomgr, pds.uid
55
+
}
56
+
57
+
func setupBenchDirectRepoOperator(b *testing.B) (RepoOperator, models.Uid) {
58
+
b.Helper()
59
+
suppressLogs(b)
60
+
ctx := context.Background()
61
+
keyPath := filepath.Join(b.TempDir(), "signing-key")
62
+
63
+
if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil {
64
+
b.Fatalf("write signing key: %v", err)
65
+
}
66
+
signingKey, err := oauth.GenerateOrLoadPDSKey(keyPath)
67
+
if err != nil {
68
+
b.Fatalf("GenerateOrLoadPDSKey: %v", err)
69
+
}
70
+
71
+
sqlStore := new(holddb.SQLiteStore)
72
+
if err := sqlStore.Open(":memory:"); err != nil {
73
+
b.Fatalf("SQLiteStore.Open: %v", err)
74
+
}
75
+
b.Cleanup(func() { sqlStore.Close() })
76
+
77
+
kmgr := NewHoldKeyManager(signingKey)
78
+
op := NewDirectRepoOperator(sqlStore, kmgr)
79
+
uid := models.Uid(1)
80
+
81
+
if err := op.InitNewActor(ctx, uid, "", "did:web:hold.bench", "", "", ""); err != nil {
82
+
b.Fatalf("InitNewActor: %v", err)
83
+
}
84
+
85
+
return op, uid
86
+
}
87
+
88
+
// seedRecords writes n records and returns their rkeys.
89
+
func seedRecords(b *testing.B, op RepoOperator, uid models.Uid, n int) []string {
90
+
b.Helper()
91
+
ctx := context.Background()
92
+
rkeys := make([]string, n)
93
+
for i := 0; i < n; i++ {
94
+
rkey := fmt.Sprintf("seed%d", i)
95
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:seed%d", i)))
96
+
if err != nil {
97
+
b.Fatalf("seed PutRecord %d: %v", i, err)
98
+
}
99
+
rkeys[i] = rkey
100
+
}
101
+
return rkeys
102
+
}
103
+
104
+
func runRepoOperatorBenchmarks(b *testing.B, name string, setup benchSetup) {
105
+
b.Run(name, func(b *testing.B) {
106
+
b.Run("CreateRecord", func(b *testing.B) {
107
+
op, uid := setup(b)
108
+
ctx := context.Background()
109
+
110
+
b.ResetTimer()
111
+
for i := 0; i < b.N; i++ {
112
+
_, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, newCrewRecord(fmt.Sprintf("did:plc:bench%d", i)))
113
+
if err != nil {
114
+
b.Fatalf("CreateRecord: %v", err)
115
+
}
116
+
}
117
+
})
118
+
119
+
b.Run("PutRecord", func(b *testing.B) {
120
+
op, uid := setup(b)
121
+
ctx := context.Background()
122
+
123
+
b.ResetTimer()
124
+
for i := 0; i < b.N; i++ {
125
+
rkey := fmt.Sprintf("put%d", i)
126
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:put%d", i)))
127
+
if err != nil {
128
+
b.Fatalf("PutRecord: %v", err)
129
+
}
130
+
}
131
+
})
132
+
133
+
b.Run("UpdateRecord", func(b *testing.B) {
134
+
op, uid := setup(b)
135
+
ctx := context.Background()
136
+
137
+
// Seed one record to update repeatedly
138
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "updbench", newCrewRecord("did:plc:updbench"))
139
+
if err != nil {
140
+
b.Fatalf("seed PutRecord: %v", err)
141
+
}
142
+
143
+
b.ResetTimer()
144
+
for i := 0; i < b.N; i++ {
145
+
rec := newCrewRecord("did:plc:updbench")
146
+
rec.Role = fmt.Sprintf("role%d", i)
147
+
_, err := op.UpdateRecord(ctx, uid, atproto.CrewCollection, "updbench", rec)
148
+
if err != nil {
149
+
b.Fatalf("UpdateRecord: %v", err)
150
+
}
151
+
}
152
+
})
153
+
154
+
b.Run("DeleteRecord", func(b *testing.B) {
155
+
op, uid := setup(b)
156
+
ctx := context.Background()
157
+
158
+
// Pre-create all records to delete
159
+
rkeys := make([]string, b.N)
160
+
for i := 0; i < b.N; i++ {
161
+
rkey := fmt.Sprintf("del%d", i)
162
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:del%d", i)))
163
+
if err != nil {
164
+
b.Fatalf("seed PutRecord: %v", err)
165
+
}
166
+
rkeys[i] = rkey
167
+
}
168
+
169
+
b.ResetTimer()
170
+
for i := 0; i < b.N; i++ {
171
+
if err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, rkeys[i]); err != nil {
172
+
b.Fatalf("DeleteRecord: %v", err)
173
+
}
174
+
}
175
+
})
176
+
177
+
b.Run("GetRecord", func(b *testing.B) {
178
+
op, uid := setup(b)
179
+
ctx := context.Background()
180
+
181
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "getbench", newCrewRecord("did:plc:getbench"))
182
+
if err != nil {
183
+
b.Fatalf("seed PutRecord: %v", err)
184
+
}
185
+
186
+
b.ResetTimer()
187
+
for i := 0; i < b.N; i++ {
188
+
_, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "getbench", cid.Undef)
189
+
if err != nil {
190
+
b.Fatalf("GetRecord: %v", err)
191
+
}
192
+
}
193
+
})
194
+
195
+
b.Run("GetRepoRev", func(b *testing.B) {
196
+
op, uid := setup(b)
197
+
ctx := context.Background()
198
+
199
+
b.ResetTimer()
200
+
for i := 0; i < b.N; i++ {
201
+
_, err := op.GetRepoRev(ctx, uid)
202
+
if err != nil {
203
+
b.Fatalf("GetRepoRev: %v", err)
204
+
}
205
+
}
206
+
})
207
+
208
+
b.Run("GetRepoRoot", func(b *testing.B) {
209
+
op, uid := setup(b)
210
+
ctx := context.Background()
211
+
212
+
b.ResetTimer()
213
+
for i := 0; i < b.N; i++ {
214
+
_, err := op.GetRepoRoot(ctx, uid)
215
+
if err != nil {
216
+
b.Fatalf("GetRepoRoot: %v", err)
217
+
}
218
+
}
219
+
})
220
+
221
+
// BatchWrite at different sizes — shows commit overhead vs per-record cost
222
+
for _, size := range []int{1, 10, 100} {
223
+
b.Run(fmt.Sprintf("BatchWrite_%d", size), func(b *testing.B) {
224
+
op, uid := setup(b)
225
+
ctx := context.Background()
226
+
227
+
b.ResetTimer()
228
+
for i := 0; i < b.N; i++ {
229
+
writes := make([]*indigoatproto.RepoApplyWrites_Input_Writes_Elem, size)
230
+
for j := 0; j < size; j++ {
231
+
rkey := fmt.Sprintf("batch%d_%d", i, j)
232
+
writes[j] = &indigoatproto.RepoApplyWrites_Input_Writes_Elem{
233
+
RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{
234
+
Collection: atproto.CrewCollection,
235
+
Rkey: &rkey,
236
+
Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord(fmt.Sprintf("did:plc:batch%d_%d", i, j))},
237
+
},
238
+
}
239
+
}
240
+
if err := op.BatchWrite(ctx, uid, writes); err != nil {
241
+
b.Fatalf("BatchWrite: %v", err)
242
+
}
243
+
}
244
+
})
245
+
}
246
+
247
+
// ReadRepo at different repo sizes
248
+
for _, size := range []int{10, 100, 500} {
249
+
b.Run(fmt.Sprintf("ReadRepo_%drecords", size), func(b *testing.B) {
250
+
op, uid := setup(b)
251
+
ctx := context.Background()
252
+
seedRecords(b, op, uid, size)
253
+
254
+
var buf bytes.Buffer
255
+
b.ResetTimer()
256
+
for i := 0; i < b.N; i++ {
257
+
buf.Reset()
258
+
if err := op.ReadRepo(ctx, uid, "", &buf); err != nil {
259
+
b.Fatalf("ReadRepo: %v", err)
260
+
}
261
+
}
262
+
b.SetBytes(int64(buf.Len()))
263
+
})
264
+
}
265
+
266
+
// Event overhead: with vs without handler
267
+
b.Run("PutRecord_NoEvents", func(b *testing.B) {
268
+
op, uid := setup(b)
269
+
ctx := context.Background()
270
+
// No event handler set
271
+
272
+
b.ResetTimer()
273
+
for i := 0; i < b.N; i++ {
274
+
rkey := fmt.Sprintf("noevt%d", i)
275
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:noevt%d", i)))
276
+
if err != nil {
277
+
b.Fatalf("PutRecord: %v", err)
278
+
}
279
+
}
280
+
})
281
+
282
+
b.Run("PutRecord_WithEvents", func(b *testing.B) {
283
+
op, uid := setup(b)
284
+
ctx := context.Background()
285
+
op.SetEventHandler(func(_ context.Context, _ *RepoEvent) {}, false)
286
+
287
+
b.ResetTimer()
288
+
for i := 0; i < b.N; i++ {
289
+
rkey := fmt.Sprintf("evt%d", i)
290
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:evt%d", i)))
291
+
if err != nil {
292
+
b.Fatalf("PutRecord: %v", err)
293
+
}
294
+
}
295
+
})
296
+
297
+
b.Run("PutRecord_WithHydration", func(b *testing.B) {
298
+
op, uid := setup(b)
299
+
ctx := context.Background()
300
+
op.SetEventHandler(func(_ context.Context, _ *RepoEvent) {}, true)
301
+
302
+
b.ResetTimer()
303
+
for i := 0; i < b.N; i++ {
304
+
rkey := fmt.Sprintf("hyd%d", i)
305
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:hyd%d", i)))
306
+
if err != nil {
307
+
b.Fatalf("PutRecord: %v", err)
308
+
}
309
+
}
310
+
})
311
+
312
+
// Read from a repo with many records (MST depth)
313
+
b.Run("GetRecord_LargeRepo", func(b *testing.B) {
314
+
op, uid := setup(b)
315
+
ctx := context.Background()
316
+
rkeys := seedRecords(b, op, uid, 500)
317
+
target := rkeys[len(rkeys)/2] // pick a middle record
318
+
319
+
b.ResetTimer()
320
+
for i := 0; i < b.N; i++ {
321
+
_, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, target, cid.Undef)
322
+
if err != nil {
323
+
b.Fatalf("GetRecord: %v", err)
324
+
}
325
+
}
326
+
})
327
+
328
+
// ReadRepo incremental (since) vs full
329
+
b.Run("ReadRepo_Incremental", func(b *testing.B) {
330
+
op, uid := setup(b)
331
+
ctx := context.Background()
332
+
seedRecords(b, op, uid, 100)
333
+
334
+
// Get rev before the last write
335
+
rev, err := op.GetRepoRev(ctx, uid)
336
+
if err != nil {
337
+
b.Fatalf("GetRepoRev: %v", err)
338
+
}
339
+
340
+
// Write one more record
341
+
_, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "incremental", newCrewRecord("did:plc:incremental"))
342
+
if err != nil {
343
+
b.Fatalf("PutRecord: %v", err)
344
+
}
345
+
346
+
var buf bytes.Buffer
347
+
b.ResetTimer()
348
+
for i := 0; i < b.N; i++ {
349
+
buf.Reset()
350
+
if err := op.ReadRepo(ctx, uid, rev, &buf); err != nil {
351
+
b.Fatalf("ReadRepo: %v", err)
352
+
}
353
+
}
354
+
b.SetBytes(int64(buf.Len()))
355
+
})
356
+
})
357
+
}
358
+
359
+
func BenchmarkRepoManager(b *testing.B) {
360
+
runRepoOperatorBenchmarks(b, "RepoManager", setupBenchRepoManager)
361
+
}
362
+
363
+
func BenchmarkDirectRepoOperator(b *testing.B) {
364
+
runRepoOperatorBenchmarks(b, "DirectRepoOperator", setupBenchDirectRepoOperator)
365
+
}
366
+
367
+
// BenchmarkBatchVsSingle compares the cost of N individual PutRecord calls
368
+
// vs a single BatchWrite with N records.
369
+
func BenchmarkBatchVsSingle(b *testing.B) {
370
+
for _, impl := range []struct {
371
+
name string
372
+
setup benchSetup
373
+
}{
374
+
{"RepoManager", setupBenchRepoManager},
375
+
{"DirectRepoOperator", setupBenchDirectRepoOperator},
376
+
} {
377
+
for _, size := range []int{1, 10, 50} {
378
+
b.Run(fmt.Sprintf("%s/Single_%d", impl.name, size), func(b *testing.B) {
379
+
op, uid := impl.setup(b)
380
+
ctx := context.Background()
381
+
382
+
b.ResetTimer()
383
+
for i := 0; i < b.N; i++ {
384
+
for j := 0; j < size; j++ {
385
+
rkey := fmt.Sprintf("single%d_%d", i, j)
386
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, newCrewRecord(fmt.Sprintf("did:plc:s%d_%d", i, j)))
387
+
if err != nil {
388
+
b.Fatalf("PutRecord: %v", err)
389
+
}
390
+
}
391
+
}
392
+
})
393
+
394
+
b.Run(fmt.Sprintf("%s/Batch_%d", impl.name, size), func(b *testing.B) {
395
+
op, uid := impl.setup(b)
396
+
ctx := context.Background()
397
+
398
+
b.ResetTimer()
399
+
for i := 0; i < b.N; i++ {
400
+
writes := make([]*indigoatproto.RepoApplyWrites_Input_Writes_Elem, size)
401
+
for j := 0; j < size; j++ {
402
+
rkey := fmt.Sprintf("batch%d_%d", i, j)
403
+
writes[j] = &indigoatproto.RepoApplyWrites_Input_Writes_Elem{
404
+
RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{
405
+
Collection: atproto.CrewCollection,
406
+
Rkey: &rkey,
407
+
Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord(fmt.Sprintf("did:plc:b%d_%d", i, j))},
408
+
},
409
+
}
410
+
}
411
+
if err := op.BatchWrite(ctx, uid, writes); err != nil {
412
+
b.Fatalf("BatchWrite: %v", err)
413
+
}
414
+
}
415
+
})
416
+
}
417
+
}
418
+
}
+1413
pkg/hold/pds/repo_operator_test.go
+1413
pkg/hold/pds/repo_operator_test.go
···
1
+
package pds
2
+
3
+
import (
4
+
"bytes"
5
+
"context"
6
+
"fmt"
7
+
"os"
8
+
"path/filepath"
9
+
"strings"
10
+
"sync"
11
+
"testing"
12
+
13
+
"atcr.io/pkg/atproto"
14
+
"atcr.io/pkg/auth/oauth"
15
+
holddb "atcr.io/pkg/hold/db"
16
+
indigoatproto "github.com/bluesky-social/indigo/api/atproto"
17
+
lexutil "github.com/bluesky-social/indigo/lex/util"
18
+
"github.com/bluesky-social/indigo/models"
19
+
"github.com/bluesky-social/indigo/repo"
20
+
"github.com/ipfs/go-cid"
21
+
)
22
+
23
+
// setupTestRepoOperator creates a fresh RepoOperator (backed by RepoManager)
24
+
// and returns it along with the user ID. Each call gets an isolated instance.
25
+
func setupTestRepoOperator(t *testing.T) (RepoOperator, models.Uid) {
26
+
t.Helper()
27
+
ctx := context.Background()
28
+
tmpDir := t.TempDir()
29
+
30
+
dbPath := ":memory:"
31
+
keyPath := filepath.Join(tmpDir, "signing-key")
32
+
33
+
if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil {
34
+
t.Fatalf("Failed to write signing key: %v", err)
35
+
}
36
+
37
+
pds, err := NewHoldPDS(ctx, "did:web:hold.test", "https://hold.test", "https://atcr.io", dbPath, keyPath, false)
38
+
if err != nil {
39
+
t.Fatalf("NewHoldPDS: %v", err)
40
+
}
41
+
42
+
if err := pds.repomgr.InitNewActor(ctx, pds.uid, "", pds.did, "", "", ""); err != nil {
43
+
t.Fatalf("InitNewActor: %v", err)
44
+
}
45
+
46
+
t.Cleanup(func() { pds.Close() })
47
+
return pds.repomgr, pds.uid
48
+
}
49
+
50
+
// setupTestDirectRepoOperator creates a fresh DirectRepoOperator
51
+
// and returns it along with the user ID.
52
+
func setupTestDirectRepoOperator(t *testing.T) (RepoOperator, models.Uid) {
53
+
t.Helper()
54
+
ctx := context.Background()
55
+
56
+
keyPath := filepath.Join(t.TempDir(), "signing-key")
57
+
if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil {
58
+
t.Fatalf("Failed to write signing key: %v", err)
59
+
}
60
+
signingKey, err := oauth.GenerateOrLoadPDSKey(keyPath)
61
+
if err != nil {
62
+
t.Fatalf("GenerateOrLoadPDSKey: %v", err)
63
+
}
64
+
65
+
sqlStore := new(holddb.SQLiteStore)
66
+
if err := sqlStore.Open(":memory:"); err != nil {
67
+
t.Fatalf("SQLiteStore.Open: %v", err)
68
+
}
69
+
t.Cleanup(func() { sqlStore.Close() })
70
+
71
+
kmgr := NewHoldKeyManager(signingKey)
72
+
op := NewDirectRepoOperator(sqlStore, kmgr)
73
+
74
+
uid := models.Uid(1)
75
+
if err := op.InitNewActor(ctx, uid, "", "did:web:hold.test", "", "", ""); err != nil {
76
+
t.Fatalf("InitNewActor: %v", err)
77
+
}
78
+
79
+
return op, uid
80
+
}
81
+
82
+
// newCrewRecord creates a test crew record with the given member DID.
83
+
func newCrewRecord(member string) *atproto.CrewRecord {
84
+
return &atproto.CrewRecord{
85
+
Type: atproto.CrewCollection,
86
+
Member: member,
87
+
Role: "writer",
88
+
Permissions: []string{"blob:read", "blob:write"},
89
+
AddedAt: "2026-01-01T00:00:00Z",
90
+
}
91
+
}
92
+
93
+
// runRepoOperatorTests runs the full RepoOperator test suite against any implementation.
94
+
// An optional freshSetup function returns an operator WITHOUT InitNewActor pre-called,
95
+
// for testing InitNewActor event emission.
96
+
func runRepoOperatorTests(t *testing.T, setup func(t *testing.T) (RepoOperator, models.Uid), freshSetup ...func(t *testing.T) (RepoOperator, models.Uid, string)) {
97
+
t.Run("CreateRecord", func(t *testing.T) {
98
+
op, uid := setup(t)
99
+
ctx := context.Background()
100
+
101
+
rec := newCrewRecord("did:plc:alice")
102
+
path, cc, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec)
103
+
if err != nil {
104
+
t.Fatalf("CreateRecord: %v", err)
105
+
}
106
+
107
+
if !strings.HasPrefix(path, atproto.CrewCollection+"/") {
108
+
t.Errorf("path should start with collection, got %q", path)
109
+
}
110
+
if !cc.Defined() {
111
+
t.Error("expected defined CID")
112
+
}
113
+
114
+
// Extract rkey and verify it looks like a TID (13 chars, base32-sortable)
115
+
rkey := strings.TrimPrefix(path, atproto.CrewCollection+"/")
116
+
if len(rkey) != 13 {
117
+
t.Errorf("expected 13-char TID rkey, got %q (len=%d)", rkey, len(rkey))
118
+
}
119
+
120
+
// Round-trip via GetRecord
121
+
gotCid, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef)
122
+
if err != nil {
123
+
t.Fatalf("GetRecord: %v", err)
124
+
}
125
+
if !gotCid.Equals(cc) {
126
+
t.Errorf("CID mismatch: create=%s get=%s", cc, gotCid)
127
+
}
128
+
})
129
+
130
+
t.Run("UpdateRecord", func(t *testing.T) {
131
+
op, uid := setup(t)
132
+
ctx := context.Background()
133
+
134
+
rec := newCrewRecord("did:plc:bob")
135
+
path, createCid, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec)
136
+
if err != nil {
137
+
t.Fatalf("CreateRecord: %v", err)
138
+
}
139
+
rkey := strings.TrimPrefix(path, atproto.CrewCollection+"/")
140
+
141
+
// Update with different data
142
+
updated := newCrewRecord("did:plc:bob")
143
+
updated.Role = "admin"
144
+
updateCid, err := op.UpdateRecord(ctx, uid, atproto.CrewCollection, rkey, updated)
145
+
if err != nil {
146
+
t.Fatalf("UpdateRecord: %v", err)
147
+
}
148
+
149
+
if createCid.Equals(updateCid) {
150
+
t.Error("expected CID to change after update")
151
+
}
152
+
153
+
// Verify new data via GetRecord
154
+
_, val, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef)
155
+
if err != nil {
156
+
t.Fatalf("GetRecord: %v", err)
157
+
}
158
+
crew, ok := val.(*atproto.CrewRecord)
159
+
if !ok {
160
+
t.Fatalf("expected *CrewRecord, got %T", val)
161
+
}
162
+
if crew.Role != "admin" {
163
+
t.Errorf("expected role=admin, got %q", crew.Role)
164
+
}
165
+
})
166
+
167
+
t.Run("PutRecord", func(t *testing.T) {
168
+
op, uid := setup(t)
169
+
ctx := context.Background()
170
+
171
+
rec := newCrewRecord("did:plc:charlie")
172
+
path, cc, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "mykey", rec)
173
+
if err != nil {
174
+
t.Fatalf("PutRecord: %v", err)
175
+
}
176
+
177
+
if !strings.HasSuffix(path, "/mykey") {
178
+
t.Errorf("expected path ending in /mykey, got %q", path)
179
+
}
180
+
if !cc.Defined() {
181
+
t.Error("expected defined CID")
182
+
}
183
+
184
+
// Round-trip
185
+
gotCid, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "mykey", cid.Undef)
186
+
if err != nil {
187
+
t.Fatalf("GetRecord: %v", err)
188
+
}
189
+
if !gotCid.Equals(cc) {
190
+
t.Errorf("CID mismatch")
191
+
}
192
+
})
193
+
194
+
t.Run("PutRecord_DuplicateRkey", func(t *testing.T) {
195
+
op, uid := setup(t)
196
+
ctx := context.Background()
197
+
198
+
rec := newCrewRecord("did:plc:dave")
199
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "dupekey", rec)
200
+
if err != nil {
201
+
t.Fatalf("first PutRecord: %v", err)
202
+
}
203
+
204
+
rec2 := newCrewRecord("did:plc:dave2")
205
+
_, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "dupekey", rec2)
206
+
if err == nil {
207
+
t.Error("expected error on duplicate rkey PutRecord")
208
+
}
209
+
})
210
+
211
+
t.Run("UpsertRecord_Create", func(t *testing.T) {
212
+
op, uid := setup(t)
213
+
ctx := context.Background()
214
+
215
+
rec := newCrewRecord("did:plc:eve")
216
+
path, cc, created, err := op.UpsertRecord(ctx, uid, atproto.CrewCollection, "upsert1", rec)
217
+
if err != nil {
218
+
t.Fatalf("UpsertRecord: %v", err)
219
+
}
220
+
221
+
if !created {
222
+
t.Error("expected created=true for new record")
223
+
}
224
+
if !strings.HasSuffix(path, "/upsert1") {
225
+
t.Errorf("expected path ending in /upsert1, got %q", path)
226
+
}
227
+
if !cc.Defined() {
228
+
t.Error("expected defined CID")
229
+
}
230
+
231
+
// Verify retrievable
232
+
_, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "upsert1", cid.Undef)
233
+
if err != nil {
234
+
t.Fatalf("GetRecord after upsert-create: %v", err)
235
+
}
236
+
})
237
+
238
+
t.Run("UpsertRecord_Update", func(t *testing.T) {
239
+
op, uid := setup(t)
240
+
ctx := context.Background()
241
+
242
+
rec := newCrewRecord("did:plc:frank")
243
+
_, cid1, _, err := op.UpsertRecord(ctx, uid, atproto.CrewCollection, "upsert2", rec)
244
+
if err != nil {
245
+
t.Fatalf("first UpsertRecord: %v", err)
246
+
}
247
+
248
+
updated := newCrewRecord("did:plc:frank")
249
+
updated.Role = "admin"
250
+
_, cid2, created, err := op.UpsertRecord(ctx, uid, atproto.CrewCollection, "upsert2", updated)
251
+
if err != nil {
252
+
t.Fatalf("second UpsertRecord: %v", err)
253
+
}
254
+
255
+
if created {
256
+
t.Error("expected created=false for existing record")
257
+
}
258
+
if cid1.Equals(cid2) {
259
+
t.Error("expected CID to change on upsert-update")
260
+
}
261
+
})
262
+
263
+
t.Run("DeleteRecord", func(t *testing.T) {
264
+
op, uid := setup(t)
265
+
ctx := context.Background()
266
+
267
+
rec := newCrewRecord("did:plc:grace")
268
+
path, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec)
269
+
if err != nil {
270
+
t.Fatalf("CreateRecord: %v", err)
271
+
}
272
+
rkey := strings.TrimPrefix(path, atproto.CrewCollection+"/")
273
+
274
+
if err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, rkey); err != nil {
275
+
t.Fatalf("DeleteRecord: %v", err)
276
+
}
277
+
278
+
_, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef)
279
+
if err == nil {
280
+
t.Error("expected error getting deleted record")
281
+
}
282
+
})
283
+
284
+
t.Run("DeleteRecord_NotFound", func(t *testing.T) {
285
+
op, uid := setup(t)
286
+
ctx := context.Background()
287
+
288
+
err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, "nonexistent")
289
+
if err == nil {
290
+
t.Error("expected error deleting non-existent record")
291
+
}
292
+
})
293
+
294
+
t.Run("BatchWrite_CreateAndDelete", func(t *testing.T) {
295
+
op, uid := setup(t)
296
+
ctx := context.Background()
297
+
298
+
// First, create a record to delete in the batch
299
+
rec := newCrewRecord("did:plc:todelete")
300
+
path, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec)
301
+
if err != nil {
302
+
t.Fatalf("CreateRecord: %v", err)
303
+
}
304
+
deleteRkey := strings.TrimPrefix(path, atproto.CrewCollection+"/")
305
+
306
+
// Batch: create 2 + delete 1
307
+
rkey1 := "batchkey1"
308
+
rkey2 := "batchkey2"
309
+
writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{
310
+
{
311
+
RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{
312
+
Collection: atproto.CrewCollection,
313
+
Rkey: &rkey1,
314
+
Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:batch1")},
315
+
},
316
+
},
317
+
{
318
+
RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{
319
+
Collection: atproto.CrewCollection,
320
+
Rkey: &rkey2,
321
+
Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:batch2")},
322
+
},
323
+
},
324
+
{
325
+
RepoApplyWrites_Delete: &indigoatproto.RepoApplyWrites_Delete{
326
+
Collection: atproto.CrewCollection,
327
+
Rkey: deleteRkey,
328
+
},
329
+
},
330
+
}
331
+
332
+
if err := op.BatchWrite(ctx, uid, writes); err != nil {
333
+
t.Fatalf("BatchWrite: %v", err)
334
+
}
335
+
336
+
// Verify created records exist
337
+
_, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, rkey1, cid.Undef)
338
+
if err != nil {
339
+
t.Errorf("batch-created record 1 not found: %v", err)
340
+
}
341
+
_, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, rkey2, cid.Undef)
342
+
if err != nil {
343
+
t.Errorf("batch-created record 2 not found: %v", err)
344
+
}
345
+
346
+
// Verify deleted record is gone
347
+
_, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, deleteRkey, cid.Undef)
348
+
if err == nil {
349
+
t.Error("expected batch-deleted record to be gone")
350
+
}
351
+
})
352
+
353
+
t.Run("BulkUpsert", func(t *testing.T) {
354
+
op, uid := setup(t)
355
+
ctx := context.Background()
356
+
357
+
records := []BulkRecord{
358
+
{Collection: atproto.CrewCollection, Rkey: "bulk1", Data: newCrewRecord("did:plc:bulk1")},
359
+
{Collection: atproto.CrewCollection, Rkey: "bulk2", Data: newCrewRecord("did:plc:bulk2")},
360
+
}
361
+
362
+
if err := op.BulkUpsert(ctx, uid, records); err != nil {
363
+
t.Fatalf("BulkUpsert: %v", err)
364
+
}
365
+
366
+
// Verify both exist
367
+
_, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "bulk1", cid.Undef)
368
+
if err != nil {
369
+
t.Errorf("bulk record 1 not found: %v", err)
370
+
}
371
+
_, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "bulk2", cid.Undef)
372
+
if err != nil {
373
+
t.Errorf("bulk record 2 not found: %v", err)
374
+
}
375
+
376
+
// Re-upsert with changed data
377
+
updatedRec := newCrewRecord("did:plc:bulk1")
378
+
updatedRec.Role = "admin"
379
+
if err := op.BulkUpsert(ctx, uid, []BulkRecord{
380
+
{Collection: atproto.CrewCollection, Rkey: "bulk1", Data: updatedRec},
381
+
}); err != nil {
382
+
t.Fatalf("BulkUpsert update: %v", err)
383
+
}
384
+
385
+
// Verify updated data
386
+
_, val, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "bulk1", cid.Undef)
387
+
if err != nil {
388
+
t.Fatalf("GetRecord after re-upsert: %v", err)
389
+
}
390
+
crew, ok := val.(*atproto.CrewRecord)
391
+
if !ok {
392
+
t.Fatalf("expected *CrewRecord, got %T", val)
393
+
}
394
+
if crew.Role != "admin" {
395
+
t.Errorf("expected role=admin after re-upsert, got %q", crew.Role)
396
+
}
397
+
})
398
+
399
+
t.Run("GetRecord_CidMatch", func(t *testing.T) {
400
+
op, uid := setup(t)
401
+
ctx := context.Background()
402
+
403
+
rec := newCrewRecord("did:plc:cidmatch")
404
+
_, cc, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "cidtest", rec)
405
+
if err != nil {
406
+
t.Fatalf("PutRecord: %v", err)
407
+
}
408
+
409
+
// Exact CID match succeeds
410
+
_, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "cidtest", cc)
411
+
if err != nil {
412
+
t.Errorf("GetRecord with matching CID should succeed: %v", err)
413
+
}
414
+
415
+
// Wrong CID fails
416
+
wrongCid, _ := cid.Decode("bafyreigdvqptwntkto5jag4rr7oydencsj4m2t5pdgkhmdwyxlayuncm7e")
417
+
_, _, err = op.GetRecord(ctx, uid, atproto.CrewCollection, "cidtest", wrongCid)
418
+
if err == nil {
419
+
t.Error("expected error with mismatched CID")
420
+
}
421
+
})
422
+
423
+
t.Run("GetRecord_NotFound", func(t *testing.T) {
424
+
op, uid := setup(t)
425
+
ctx := context.Background()
426
+
427
+
_, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "doesnotexist", cid.Undef)
428
+
if err == nil {
429
+
t.Error("expected error for non-existent record")
430
+
}
431
+
})
432
+
433
+
t.Run("GetRecordProof", func(t *testing.T) {
434
+
op, uid := setup(t)
435
+
ctx := context.Background()
436
+
437
+
rec := newCrewRecord("did:plc:proof")
438
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "proofkey", rec)
439
+
if err != nil {
440
+
t.Fatalf("PutRecord: %v", err)
441
+
}
442
+
443
+
head, blocks, err := op.GetRecordProof(ctx, uid, atproto.CrewCollection, "proofkey")
444
+
if err != nil {
445
+
t.Fatalf("GetRecordProof: %v", err)
446
+
}
447
+
448
+
if !head.Defined() {
449
+
t.Error("expected defined head CID")
450
+
}
451
+
if len(blocks) == 0 {
452
+
t.Error("expected non-empty proof blocks")
453
+
}
454
+
})
455
+
456
+
t.Run("GetRepoRoot", func(t *testing.T) {
457
+
op, uid := setup(t)
458
+
ctx := context.Background()
459
+
460
+
root1, err := op.GetRepoRoot(ctx, uid)
461
+
if err != nil {
462
+
t.Fatalf("GetRepoRoot: %v", err)
463
+
}
464
+
if !root1.Defined() {
465
+
t.Error("expected defined root CID after InitNewActor")
466
+
}
467
+
468
+
// Write a record and verify root changes
469
+
_, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "roottest", newCrewRecord("did:plc:root"))
470
+
if err != nil {
471
+
t.Fatalf("PutRecord: %v", err)
472
+
}
473
+
474
+
root2, err := op.GetRepoRoot(ctx, uid)
475
+
if err != nil {
476
+
t.Fatalf("GetRepoRoot after write: %v", err)
477
+
}
478
+
if root1.Equals(root2) {
479
+
t.Error("expected root to change after write")
480
+
}
481
+
})
482
+
483
+
t.Run("GetRepoRev", func(t *testing.T) {
484
+
op, uid := setup(t)
485
+
ctx := context.Background()
486
+
487
+
rev1, err := op.GetRepoRev(ctx, uid)
488
+
if err != nil {
489
+
t.Fatalf("GetRepoRev: %v", err)
490
+
}
491
+
if rev1 == "" {
492
+
t.Error("expected non-empty rev")
493
+
}
494
+
495
+
// Write a record and verify rev changes
496
+
_, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "revtest", newCrewRecord("did:plc:rev"))
497
+
if err != nil {
498
+
t.Fatalf("PutRecord: %v", err)
499
+
}
500
+
501
+
rev2, err := op.GetRepoRev(ctx, uid)
502
+
if err != nil {
503
+
t.Fatalf("GetRepoRev after write: %v", err)
504
+
}
505
+
if rev1 == rev2 {
506
+
t.Error("expected rev to change after write")
507
+
}
508
+
})
509
+
510
+
t.Run("ReadRepo", func(t *testing.T) {
511
+
op, uid := setup(t)
512
+
ctx := context.Background()
513
+
514
+
// Write something first
515
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "readrepo", newCrewRecord("did:plc:readrepo"))
516
+
if err != nil {
517
+
t.Fatalf("PutRecord: %v", err)
518
+
}
519
+
520
+
var buf bytes.Buffer
521
+
if err := op.ReadRepo(ctx, uid, "", &buf); err != nil {
522
+
t.Fatalf("ReadRepo: %v", err)
523
+
}
524
+
525
+
if buf.Len() == 0 {
526
+
t.Error("expected non-empty CAR output")
527
+
}
528
+
})
529
+
530
+
t.Run("EventEmission_Create", func(t *testing.T) {
531
+
op, uid := setup(t)
532
+
ctx := context.Background()
533
+
534
+
var events []*RepoEvent
535
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
536
+
events = append(events, evt)
537
+
}, false)
538
+
539
+
rec := newCrewRecord("did:plc:evt-create")
540
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evtcreate", rec)
541
+
if err != nil {
542
+
t.Fatalf("PutRecord: %v", err)
543
+
}
544
+
545
+
if len(events) != 1 {
546
+
t.Fatalf("expected 1 event, got %d", len(events))
547
+
}
548
+
549
+
evt := events[0]
550
+
if evt.User != uid {
551
+
t.Errorf("expected user=%d, got %d", uid, evt.User)
552
+
}
553
+
if !evt.NewRoot.Defined() {
554
+
t.Error("expected defined NewRoot")
555
+
}
556
+
if evt.PrevData == nil {
557
+
t.Error("expected non-nil PrevData")
558
+
}
559
+
if evt.Rev == "" {
560
+
t.Error("expected non-empty Rev")
561
+
}
562
+
if len(evt.RepoSlice) == 0 {
563
+
t.Error("expected non-empty RepoSlice")
564
+
}
565
+
if len(evt.Ops) != 1 {
566
+
t.Fatalf("expected 1 op, got %d", len(evt.Ops))
567
+
}
568
+
569
+
op0 := evt.Ops[0]
570
+
if op0.Kind != EvtKindCreateRecord {
571
+
t.Errorf("expected kind=create, got %q", op0.Kind)
572
+
}
573
+
if op0.Collection != atproto.CrewCollection {
574
+
t.Errorf("expected collection=%s, got %q", atproto.CrewCollection, op0.Collection)
575
+
}
576
+
if op0.Rkey != "evtcreate" {
577
+
t.Errorf("expected rkey=evtcreate, got %q", op0.Rkey)
578
+
}
579
+
if op0.RecCid == nil {
580
+
t.Error("expected non-nil RecCid for create op")
581
+
}
582
+
})
583
+
584
+
t.Run("EventEmission_Update_PrevData", func(t *testing.T) {
585
+
op, uid := setup(t)
586
+
ctx := context.Background()
587
+
588
+
// Create first (no event handler yet)
589
+
rec := newCrewRecord("did:plc:evt-update")
590
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evtupdate", rec)
591
+
if err != nil {
592
+
t.Fatalf("PutRecord: %v", err)
593
+
}
594
+
595
+
// Now set handler and update
596
+
var events []*RepoEvent
597
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
598
+
events = append(events, evt)
599
+
}, false)
600
+
601
+
updated := newCrewRecord("did:plc:evt-update")
602
+
updated.Role = "admin"
603
+
_, err = op.UpdateRecord(ctx, uid, atproto.CrewCollection, "evtupdate", updated)
604
+
if err != nil {
605
+
t.Fatalf("UpdateRecord: %v", err)
606
+
}
607
+
608
+
if len(events) != 1 {
609
+
t.Fatalf("expected 1 event, got %d", len(events))
610
+
}
611
+
612
+
evt := events[0]
613
+
if evt.PrevData == nil {
614
+
t.Error("expected non-nil PrevData on update event")
615
+
}
616
+
if evt.OldRoot == nil {
617
+
t.Error("expected non-nil OldRoot on update event")
618
+
}
619
+
if evt.Since == nil {
620
+
t.Error("expected non-nil Since on update event")
621
+
}
622
+
if len(evt.Ops) != 1 {
623
+
t.Fatalf("expected 1 op, got %d", len(evt.Ops))
624
+
}
625
+
if evt.Ops[0].Kind != EvtKindUpdateRecord {
626
+
t.Errorf("expected kind=update, got %q", evt.Ops[0].Kind)
627
+
}
628
+
})
629
+
630
+
t.Run("EventEmission_Delete", func(t *testing.T) {
631
+
op, uid := setup(t)
632
+
ctx := context.Background()
633
+
634
+
rec := newCrewRecord("did:plc:evt-delete")
635
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evtdelete", rec)
636
+
if err != nil {
637
+
t.Fatalf("PutRecord: %v", err)
638
+
}
639
+
640
+
var events []*RepoEvent
641
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
642
+
events = append(events, evt)
643
+
}, false)
644
+
645
+
if err := op.DeleteRecord(ctx, uid, atproto.CrewCollection, "evtdelete"); err != nil {
646
+
t.Fatalf("DeleteRecord: %v", err)
647
+
}
648
+
649
+
if len(events) != 1 {
650
+
t.Fatalf("expected 1 event, got %d", len(events))
651
+
}
652
+
653
+
evt := events[0]
654
+
if len(evt.Ops) != 1 {
655
+
t.Fatalf("expected 1 op, got %d", len(evt.Ops))
656
+
}
657
+
if evt.Ops[0].Kind != EvtKindDeleteRecord {
658
+
t.Errorf("expected kind=delete, got %q", evt.Ops[0].Kind)
659
+
}
660
+
if evt.Ops[0].RecCid != nil {
661
+
t.Error("expected nil RecCid for delete op")
662
+
}
663
+
})
664
+
665
+
t.Run("EventEmission_Hydrate", func(t *testing.T) {
666
+
op, uid := setup(t)
667
+
ctx := context.Background()
668
+
669
+
var events []*RepoEvent
670
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
671
+
events = append(events, evt)
672
+
}, true) // hydrateRecords = true
673
+
674
+
rec := newCrewRecord("did:plc:evt-hydrate")
675
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "evthydrate", rec)
676
+
if err != nil {
677
+
t.Fatalf("PutRecord: %v", err)
678
+
}
679
+
680
+
if len(events) != 1 {
681
+
t.Fatalf("expected 1 event, got %d", len(events))
682
+
}
683
+
if events[0].Ops[0].Record == nil {
684
+
t.Error("expected non-nil Record when hydrateRecords=true")
685
+
}
686
+
})
687
+
688
+
// --- Error path and edge case tests ---
689
+
690
+
t.Run("UpdateRecord_NotFound", func(t *testing.T) {
691
+
op, uid := setup(t)
692
+
ctx := context.Background()
693
+
694
+
rec := newCrewRecord("did:plc:ghost")
695
+
_, err := op.UpdateRecord(ctx, uid, atproto.CrewCollection, "nonexistent", rec)
696
+
if err == nil {
697
+
t.Error("expected error updating non-existent record")
698
+
}
699
+
})
700
+
701
+
t.Run("UpdateRecord_Hydrate", func(t *testing.T) {
702
+
op, uid := setup(t)
703
+
ctx := context.Background()
704
+
705
+
rec := newCrewRecord("did:plc:hydrate-upd")
706
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "hydupd", rec)
707
+
if err != nil {
708
+
t.Fatalf("PutRecord: %v", err)
709
+
}
710
+
711
+
var events []*RepoEvent
712
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
713
+
events = append(events, evt)
714
+
}, true)
715
+
716
+
updated := newCrewRecord("did:plc:hydrate-upd")
717
+
updated.Role = "admin"
718
+
_, err = op.UpdateRecord(ctx, uid, atproto.CrewCollection, "hydupd", updated)
719
+
if err != nil {
720
+
t.Fatalf("UpdateRecord: %v", err)
721
+
}
722
+
723
+
if len(events) != 1 {
724
+
t.Fatalf("expected 1 event, got %d", len(events))
725
+
}
726
+
if events[0].Ops[0].Record == nil {
727
+
t.Error("expected non-nil Record on update with hydrateRecords=true")
728
+
}
729
+
})
730
+
731
+
t.Run("PutRecord_Hydrate", func(t *testing.T) {
732
+
op, uid := setup(t)
733
+
ctx := context.Background()
734
+
735
+
var events []*RepoEvent
736
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
737
+
events = append(events, evt)
738
+
}, true)
739
+
740
+
rec := newCrewRecord("did:plc:hydrate-put")
741
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "hydput", rec)
742
+
if err != nil {
743
+
t.Fatalf("PutRecord: %v", err)
744
+
}
745
+
746
+
if len(events) != 1 {
747
+
t.Fatalf("expected 1 event, got %d", len(events))
748
+
}
749
+
if events[0].Ops[0].Record == nil {
750
+
t.Error("expected non-nil Record on put with hydrateRecords=true")
751
+
}
752
+
})
753
+
754
+
t.Run("InitNewActor_EmptyDID", func(t *testing.T) {
755
+
op, uid := setup(t)
756
+
ctx := context.Background()
757
+
758
+
err := op.InitNewActor(ctx, uid, "", "", "", "", "")
759
+
if err == nil {
760
+
t.Error("expected error for empty DID")
761
+
}
762
+
})
763
+
764
+
t.Run("InitNewActor_ZeroUser", func(t *testing.T) {
765
+
op, _ := setup(t)
766
+
ctx := context.Background()
767
+
768
+
err := op.InitNewActor(ctx, 0, "", "did:web:test", "", "", "")
769
+
if err == nil {
770
+
t.Error("expected error for zero user")
771
+
}
772
+
})
773
+
774
+
t.Run("InitNewActor_EventEmission", func(t *testing.T) {
775
+
if len(freshSetup) == 0 || freshSetup[0] == nil {
776
+
t.Skip("no freshSetup provided")
777
+
}
778
+
779
+
op, uid, did := freshSetup[0](t)
780
+
ctx := context.Background()
781
+
782
+
var events []*RepoEvent
783
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
784
+
events = append(events, evt)
785
+
}, true)
786
+
787
+
err := op.InitNewActor(ctx, uid, "", did, "Test User", "", "")
788
+
if err != nil {
789
+
t.Fatalf("InitNewActor: %v", err)
790
+
}
791
+
792
+
if len(events) != 1 {
793
+
t.Fatalf("expected 1 event from InitNewActor, got %d", len(events))
794
+
}
795
+
evt := events[0]
796
+
if len(evt.Ops) != 1 {
797
+
t.Fatalf("expected 1 op, got %d", len(evt.Ops))
798
+
}
799
+
if evt.Ops[0].Kind != EvtKindCreateRecord {
800
+
t.Errorf("expected kind=create, got %q", evt.Ops[0].Kind)
801
+
}
802
+
if evt.Ops[0].Collection != "app.bsky.actor.profile" {
803
+
t.Errorf("expected collection=app.bsky.actor.profile, got %q", evt.Ops[0].Collection)
804
+
}
805
+
if evt.Ops[0].Record == nil {
806
+
t.Error("expected non-nil Record with hydrateRecords=true")
807
+
}
808
+
})
809
+
810
+
t.Run("GetRecordProof_NotFound", func(t *testing.T) {
811
+
op, uid := setup(t)
812
+
ctx := context.Background()
813
+
814
+
_, _, err := op.GetRecordProof(ctx, uid, atproto.CrewCollection, "nonexistent")
815
+
if err == nil {
816
+
t.Error("expected error for proof of non-existent record")
817
+
}
818
+
})
819
+
820
+
t.Run("GetRecordProof_NoRepo", func(t *testing.T) {
821
+
// Use a user ID that has no repo initialized — triggers OpenRepo error
822
+
op, _ := setup(t)
823
+
ctx := context.Background()
824
+
825
+
_, _, err := op.GetRecordProof(ctx, models.Uid(9999), atproto.CrewCollection, "anything")
826
+
if err == nil {
827
+
t.Error("expected error for user with no repo")
828
+
}
829
+
})
830
+
831
+
t.Run("BatchWrite_Update", func(t *testing.T) {
832
+
op, uid := setup(t)
833
+
ctx := context.Background()
834
+
835
+
// NOTE: BatchWrite internally uses r.PutRecord for updates (mst.Add),
836
+
// which means the update write type is effectively a create-or-fail.
837
+
// This test uses a fresh rkey to exercise the update code path.
838
+
updated := newCrewRecord("did:plc:batchupd")
839
+
updated.Role = "admin"
840
+
writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{
841
+
{
842
+
RepoApplyWrites_Update: &indigoatproto.RepoApplyWrites_Update{
843
+
Collection: atproto.CrewCollection,
844
+
Rkey: "batchupd",
845
+
Value: &lexutil.LexiconTypeDecoder{Val: updated},
846
+
},
847
+
},
848
+
}
849
+
850
+
if err := op.BatchWrite(ctx, uid, writes); err != nil {
851
+
t.Fatalf("BatchWrite update: %v", err)
852
+
}
853
+
854
+
// Verify data was written
855
+
_, val, err := op.GetRecord(ctx, uid, atproto.CrewCollection, "batchupd", cid.Undef)
856
+
if err != nil {
857
+
t.Fatalf("GetRecord after batch update: %v", err)
858
+
}
859
+
crew, ok := val.(*atproto.CrewRecord)
860
+
if !ok {
861
+
t.Fatalf("expected *CrewRecord, got %T", val)
862
+
}
863
+
if crew.Role != "admin" {
864
+
t.Errorf("expected role=admin, got %q", crew.Role)
865
+
}
866
+
})
867
+
868
+
t.Run("BatchWrite_AutoRkey", func(t *testing.T) {
869
+
op, uid := setup(t)
870
+
ctx := context.Background()
871
+
872
+
var events []*RepoEvent
873
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
874
+
events = append(events, evt)
875
+
}, false)
876
+
877
+
// Create with nil Rkey — should auto-generate TID
878
+
writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{
879
+
{
880
+
RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{
881
+
Collection: atproto.CrewCollection,
882
+
Rkey: nil, // auto-generate
883
+
Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:autorkey")},
884
+
},
885
+
},
886
+
}
887
+
888
+
if err := op.BatchWrite(ctx, uid, writes); err != nil {
889
+
t.Fatalf("BatchWrite: %v", err)
890
+
}
891
+
892
+
if len(events) != 1 {
893
+
t.Fatalf("expected 1 event, got %d", len(events))
894
+
}
895
+
rkey := events[0].Ops[0].Rkey
896
+
if len(rkey) != 13 {
897
+
t.Errorf("expected 13-char auto-generated TID rkey, got %q (len=%d)", rkey, len(rkey))
898
+
}
899
+
900
+
// Verify the record exists
901
+
_, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef)
902
+
if err != nil {
903
+
t.Errorf("auto-rkey record not found: %v", err)
904
+
}
905
+
})
906
+
907
+
t.Run("BatchWrite_DeleteNotFound", func(t *testing.T) {
908
+
op, uid := setup(t)
909
+
ctx := context.Background()
910
+
911
+
writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{
912
+
{
913
+
RepoApplyWrites_Delete: &indigoatproto.RepoApplyWrites_Delete{
914
+
Collection: atproto.CrewCollection,
915
+
Rkey: "nonexistent",
916
+
},
917
+
},
918
+
}
919
+
920
+
err := op.BatchWrite(ctx, uid, writes)
921
+
if err == nil {
922
+
t.Error("expected error deleting non-existent record in batch")
923
+
}
924
+
})
925
+
926
+
t.Run("BatchWrite_EmptyWriteElem", func(t *testing.T) {
927
+
op, uid := setup(t)
928
+
ctx := context.Background()
929
+
930
+
// Write elem with no operation set
931
+
writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{
932
+
{}, // all nil
933
+
}
934
+
935
+
err := op.BatchWrite(ctx, uid, writes)
936
+
if err == nil {
937
+
t.Error("expected error for empty write elem")
938
+
}
939
+
})
940
+
941
+
t.Run("BatchWrite_EventEmission", func(t *testing.T) {
942
+
op, uid := setup(t)
943
+
ctx := context.Background()
944
+
945
+
// Create a record to delete
946
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "batchevtdel", newCrewRecord("did:plc:batchevtdel"))
947
+
if err != nil {
948
+
t.Fatalf("PutRecord: %v", err)
949
+
}
950
+
951
+
var events []*RepoEvent
952
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
953
+
events = append(events, evt)
954
+
}, true)
955
+
956
+
rkey := "batchevt1"
957
+
writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{
958
+
{
959
+
RepoApplyWrites_Create: &indigoatproto.RepoApplyWrites_Create{
960
+
Collection: atproto.CrewCollection,
961
+
Rkey: &rkey,
962
+
Value: &lexutil.LexiconTypeDecoder{Val: newCrewRecord("did:plc:batchevt1")},
963
+
},
964
+
},
965
+
{
966
+
RepoApplyWrites_Delete: &indigoatproto.RepoApplyWrites_Delete{
967
+
Collection: atproto.CrewCollection,
968
+
Rkey: "batchevtdel",
969
+
},
970
+
},
971
+
}
972
+
973
+
if err := op.BatchWrite(ctx, uid, writes); err != nil {
974
+
t.Fatalf("BatchWrite: %v", err)
975
+
}
976
+
977
+
if len(events) != 1 {
978
+
t.Fatalf("expected 1 event, got %d", len(events))
979
+
}
980
+
981
+
evt := events[0]
982
+
if len(evt.Ops) != 2 {
983
+
t.Fatalf("expected 2 ops in batch event, got %d", len(evt.Ops))
984
+
}
985
+
if evt.PrevData == nil {
986
+
t.Error("expected non-nil PrevData")
987
+
}
988
+
if evt.OldRoot == nil {
989
+
t.Error("expected non-nil OldRoot")
990
+
}
991
+
992
+
// First op: create with hydrated record
993
+
if evt.Ops[0].Kind != EvtKindCreateRecord {
994
+
t.Errorf("op[0] expected kind=create, got %q", evt.Ops[0].Kind)
995
+
}
996
+
if evt.Ops[0].Record == nil {
997
+
t.Error("op[0] expected non-nil Record with hydrateRecords=true")
998
+
}
999
+
1000
+
// Second op: delete
1001
+
if evt.Ops[1].Kind != EvtKindDeleteRecord {
1002
+
t.Errorf("op[1] expected kind=delete, got %q", evt.Ops[1].Kind)
1003
+
}
1004
+
})
1005
+
1006
+
t.Run("BatchWrite_UpdateHydrate", func(t *testing.T) {
1007
+
op, uid := setup(t)
1008
+
ctx := context.Background()
1009
+
1010
+
// NOTE: BatchWrite uses r.PutRecord for updates (mst.Add), so use a fresh rkey.
1011
+
var events []*RepoEvent
1012
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
1013
+
events = append(events, evt)
1014
+
}, true)
1015
+
1016
+
updated := newCrewRecord("did:plc:batchhydupd")
1017
+
updated.Role = "admin"
1018
+
writes := []*indigoatproto.RepoApplyWrites_Input_Writes_Elem{
1019
+
{
1020
+
RepoApplyWrites_Update: &indigoatproto.RepoApplyWrites_Update{
1021
+
Collection: atproto.CrewCollection,
1022
+
Rkey: "batchhydupd",
1023
+
Value: &lexutil.LexiconTypeDecoder{Val: updated},
1024
+
},
1025
+
},
1026
+
}
1027
+
1028
+
if err := op.BatchWrite(ctx, uid, writes); err != nil {
1029
+
t.Fatalf("BatchWrite: %v", err)
1030
+
}
1031
+
1032
+
if len(events) != 1 {
1033
+
t.Fatalf("expected 1 event, got %d", len(events))
1034
+
}
1035
+
if events[0].Ops[0].Kind != EvtKindUpdateRecord {
1036
+
t.Errorf("expected kind=update, got %q", events[0].Ops[0].Kind)
1037
+
}
1038
+
if events[0].Ops[0].Record == nil {
1039
+
t.Error("expected non-nil Record on batch update with hydrateRecords=true")
1040
+
}
1041
+
})
1042
+
1043
+
t.Run("BulkUpsert_EventEmission", func(t *testing.T) {
1044
+
op, uid := setup(t)
1045
+
ctx := context.Background()
1046
+
1047
+
var events []*RepoEvent
1048
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
1049
+
events = append(events, evt)
1050
+
}, true)
1051
+
1052
+
records := []BulkRecord{
1053
+
{Collection: atproto.CrewCollection, Rkey: "bulkevt1", Data: newCrewRecord("did:plc:bulkevt1")},
1054
+
{Collection: atproto.CrewCollection, Rkey: "bulkevt2", Data: newCrewRecord("did:plc:bulkevt2")},
1055
+
}
1056
+
1057
+
if err := op.BulkUpsert(ctx, uid, records); err != nil {
1058
+
t.Fatalf("BulkUpsert: %v", err)
1059
+
}
1060
+
1061
+
if len(events) != 1 {
1062
+
t.Fatalf("expected 1 event, got %d", len(events))
1063
+
}
1064
+
if len(events[0].Ops) != 2 {
1065
+
t.Fatalf("expected 2 ops, got %d", len(events[0].Ops))
1066
+
}
1067
+
// Both should be creates since records are new
1068
+
for i, eop := range events[0].Ops {
1069
+
if eop.Kind != EvtKindCreateRecord {
1070
+
t.Errorf("op[%d] expected kind=create, got %q", i, eop.Kind)
1071
+
}
1072
+
}
1073
+
})
1074
+
1075
+
t.Run("ReadRepo_WithSince", func(t *testing.T) {
1076
+
op, uid := setup(t)
1077
+
ctx := context.Background()
1078
+
1079
+
// Get initial rev
1080
+
rev1, err := op.GetRepoRev(ctx, uid)
1081
+
if err != nil {
1082
+
t.Fatalf("GetRepoRev: %v", err)
1083
+
}
1084
+
1085
+
// Write a record
1086
+
_, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "since1", newCrewRecord("did:plc:since1"))
1087
+
if err != nil {
1088
+
t.Fatalf("PutRecord: %v", err)
1089
+
}
1090
+
1091
+
// ReadRepo with since should produce smaller output than full export
1092
+
var fullBuf bytes.Buffer
1093
+
if err := op.ReadRepo(ctx, uid, "", &fullBuf); err != nil {
1094
+
t.Fatalf("ReadRepo full: %v", err)
1095
+
}
1096
+
1097
+
var sinceBuf bytes.Buffer
1098
+
if err := op.ReadRepo(ctx, uid, rev1, &sinceBuf); err != nil {
1099
+
t.Fatalf("ReadRepo since: %v", err)
1100
+
}
1101
+
1102
+
if sinceBuf.Len() == 0 {
1103
+
t.Error("expected non-empty incremental CAR")
1104
+
}
1105
+
if sinceBuf.Len() >= fullBuf.Len() {
1106
+
t.Errorf("expected incremental CAR (%d) < full CAR (%d)", sinceBuf.Len(), fullBuf.Len())
1107
+
}
1108
+
})
1109
+
1110
+
t.Run("CreateRecord_NoEventWithoutHandler", func(t *testing.T) {
1111
+
op, uid := setup(t)
1112
+
ctx := context.Background()
1113
+
1114
+
// Don't set event handler — should not panic
1115
+
rec := newCrewRecord("did:plc:nohandler")
1116
+
_, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec)
1117
+
if err != nil {
1118
+
t.Fatalf("CreateRecord without handler: %v", err)
1119
+
}
1120
+
})
1121
+
1122
+
t.Run("CreateRecord_AlwaysHydrates", func(t *testing.T) {
1123
+
op, uid := setup(t)
1124
+
ctx := context.Background()
1125
+
1126
+
var events []*RepoEvent
1127
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
1128
+
events = append(events, evt)
1129
+
}, false) // hydration OFF
1130
+
1131
+
rec := newCrewRecord("did:plc:always-hydrate")
1132
+
_, _, err := op.CreateRecord(ctx, uid, atproto.CrewCollection, rec)
1133
+
if err != nil {
1134
+
t.Fatalf("CreateRecord: %v", err)
1135
+
}
1136
+
1137
+
if len(events) != 1 {
1138
+
t.Fatalf("expected 1 event, got %d", len(events))
1139
+
}
1140
+
// CreateRecord always includes Record regardless of hydrateRecords
1141
+
if events[0].Ops[0].Record == nil {
1142
+
t.Error("expected non-nil Record on CreateRecord even with hydrateRecords=false")
1143
+
}
1144
+
})
1145
+
1146
+
t.Run("PutRecord_NoHydrate", func(t *testing.T) {
1147
+
op, uid := setup(t)
1148
+
ctx := context.Background()
1149
+
1150
+
var events []*RepoEvent
1151
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
1152
+
events = append(events, evt)
1153
+
}, false) // hydration OFF
1154
+
1155
+
rec := newCrewRecord("did:plc:put-nohydrate")
1156
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "putnohydrate", rec)
1157
+
if err != nil {
1158
+
t.Fatalf("PutRecord: %v", err)
1159
+
}
1160
+
1161
+
if len(events) != 1 {
1162
+
t.Fatalf("expected 1 event, got %d", len(events))
1163
+
}
1164
+
if events[0].Ops[0].Record != nil {
1165
+
t.Error("expected nil Record on PutRecord with hydrateRecords=false")
1166
+
}
1167
+
})
1168
+
1169
+
t.Run("UpdateRecord_NoHydrate", func(t *testing.T) {
1170
+
op, uid := setup(t)
1171
+
ctx := context.Background()
1172
+
1173
+
// Create record first (no handler)
1174
+
rec := newCrewRecord("did:plc:upd-nohydrate")
1175
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "updnohydrate", rec)
1176
+
if err != nil {
1177
+
t.Fatalf("PutRecord: %v", err)
1178
+
}
1179
+
1180
+
var events []*RepoEvent
1181
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
1182
+
events = append(events, evt)
1183
+
}, false) // hydration OFF
1184
+
1185
+
updated := newCrewRecord("did:plc:upd-nohydrate")
1186
+
updated.Role = "admin"
1187
+
_, err = op.UpdateRecord(ctx, uid, atproto.CrewCollection, "updnohydrate", updated)
1188
+
if err != nil {
1189
+
t.Fatalf("UpdateRecord: %v", err)
1190
+
}
1191
+
1192
+
if len(events) != 1 {
1193
+
t.Fatalf("expected 1 event, got %d", len(events))
1194
+
}
1195
+
if events[0].Ops[0].Record != nil {
1196
+
t.Error("expected nil Record on UpdateRecord with hydrateRecords=false")
1197
+
}
1198
+
})
1199
+
1200
+
t.Run("BulkUpsert_NeverHydrates", func(t *testing.T) {
1201
+
op, uid := setup(t)
1202
+
ctx := context.Background()
1203
+
1204
+
var events []*RepoEvent
1205
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
1206
+
events = append(events, evt)
1207
+
}, true) // hydration ON — but BulkUpsert should still not hydrate
1208
+
1209
+
records := []BulkRecord{
1210
+
{Collection: atproto.CrewCollection, Rkey: "bulknohydrate", Data: newCrewRecord("did:plc:bulk-nohydrate")},
1211
+
}
1212
+
if err := op.BulkUpsert(ctx, uid, records); err != nil {
1213
+
t.Fatalf("BulkUpsert: %v", err)
1214
+
}
1215
+
1216
+
if len(events) != 1 {
1217
+
t.Fatalf("expected 1 event, got %d", len(events))
1218
+
}
1219
+
if events[0].Ops[0].Record != nil {
1220
+
t.Error("expected nil Record on BulkUpsert even with hydrateRecords=true")
1221
+
}
1222
+
})
1223
+
1224
+
t.Run("RevChain_Sequential", func(t *testing.T) {
1225
+
op, uid := setup(t)
1226
+
ctx := context.Background()
1227
+
1228
+
var events []*RepoEvent
1229
+
op.SetEventHandler(func(_ context.Context, evt *RepoEvent) {
1230
+
events = append(events, evt)
1231
+
}, false)
1232
+
1233
+
// Get initial rev
1234
+
rev0, err := op.GetRepoRev(ctx, uid)
1235
+
if err != nil {
1236
+
t.Fatalf("GetRepoRev: %v", err)
1237
+
}
1238
+
1239
+
// Write 3 records sequentially
1240
+
for i := 0; i < 3; i++ {
1241
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, fmt.Sprintf("chain%d", i), newCrewRecord(fmt.Sprintf("did:plc:chain%d", i)))
1242
+
if err != nil {
1243
+
t.Fatalf("PutRecord %d: %v", i, err)
1244
+
}
1245
+
}
1246
+
1247
+
if len(events) != 3 {
1248
+
t.Fatalf("expected 3 events, got %d", len(events))
1249
+
}
1250
+
1251
+
// Revs increase monotonically (TIDs are lexicographically sortable)
1252
+
revs := []string{rev0, events[0].Rev, events[1].Rev, events[2].Rev}
1253
+
for i := 1; i < len(revs); i++ {
1254
+
if revs[i] <= revs[i-1] {
1255
+
t.Errorf("rev[%d]=%q should be > rev[%d]=%q", i, revs[i], i-1, revs[i-1])
1256
+
}
1257
+
}
1258
+
1259
+
// Each event's Since equals the previous rev
1260
+
for i, evt := range events {
1261
+
expectedSince := revs[i] // rev before this write
1262
+
if evt.Since == nil {
1263
+
t.Errorf("event[%d] Since is nil", i)
1264
+
} else if *evt.Since != expectedSince {
1265
+
t.Errorf("event[%d] Since=%q, expected %q", i, *evt.Since, expectedSince)
1266
+
}
1267
+
}
1268
+
1269
+
// Each event's OldRoot equals the previous NewRoot
1270
+
// First event's OldRoot should be defined (repo initialized by setup)
1271
+
for i := 1; i < len(events); i++ {
1272
+
if events[i].OldRoot == nil {
1273
+
t.Errorf("event[%d] OldRoot is nil", i)
1274
+
} else if !events[i].OldRoot.Equals(events[i-1].NewRoot) {
1275
+
t.Errorf("event[%d] OldRoot=%s != event[%d] NewRoot=%s", i, *events[i].OldRoot, i-1, events[i-1].NewRoot)
1276
+
}
1277
+
}
1278
+
1279
+
// Roots change each time
1280
+
roots := []cid.Cid{events[0].NewRoot, events[1].NewRoot, events[2].NewRoot}
1281
+
for i := 1; i < len(roots); i++ {
1282
+
if roots[i].Equals(roots[i-1]) {
1283
+
t.Errorf("root[%d] should differ from root[%d]", i, i-1)
1284
+
}
1285
+
}
1286
+
})
1287
+
1288
+
t.Run("ConcurrentWrites", func(t *testing.T) {
1289
+
op, uid := setup(t)
1290
+
ctx := context.Background()
1291
+
1292
+
const n = 10
1293
+
var wg sync.WaitGroup
1294
+
errs := make([]error, n)
1295
+
1296
+
for i := 0; i < n; i++ {
1297
+
wg.Add(1)
1298
+
go func(i int) {
1299
+
defer wg.Done()
1300
+
rkey := fmt.Sprintf("concurrent%d", i)
1301
+
rec := newCrewRecord(fmt.Sprintf("did:plc:concurrent%d", i))
1302
+
_, _, errs[i] = op.PutRecord(ctx, uid, atproto.CrewCollection, rkey, rec)
1303
+
}(i)
1304
+
}
1305
+
wg.Wait()
1306
+
1307
+
for i, err := range errs {
1308
+
if err != nil {
1309
+
t.Errorf("goroutine %d: PutRecord failed: %v", i, err)
1310
+
}
1311
+
}
1312
+
1313
+
// Verify all records exist
1314
+
for i := 0; i < n; i++ {
1315
+
rkey := fmt.Sprintf("concurrent%d", i)
1316
+
_, _, err := op.GetRecord(ctx, uid, atproto.CrewCollection, rkey, cid.Undef)
1317
+
if err != nil {
1318
+
t.Errorf("record %q not found after concurrent writes: %v", rkey, err)
1319
+
}
1320
+
}
1321
+
})
1322
+
1323
+
t.Run("ReadRepo_ContainsRecords", func(t *testing.T) {
1324
+
op, uid := setup(t)
1325
+
ctx := context.Background()
1326
+
1327
+
// Write 2 records with known rkeys
1328
+
_, _, err := op.PutRecord(ctx, uid, atproto.CrewCollection, "cartest1", newCrewRecord("did:plc:cartest1"))
1329
+
if err != nil {
1330
+
t.Fatalf("PutRecord 1: %v", err)
1331
+
}
1332
+
_, _, err = op.PutRecord(ctx, uid, atproto.CrewCollection, "cartest2", newCrewRecord("did:plc:cartest2"))
1333
+
if err != nil {
1334
+
t.Fatalf("PutRecord 2: %v", err)
1335
+
}
1336
+
1337
+
// Export CAR
1338
+
var buf bytes.Buffer
1339
+
if err := op.ReadRepo(ctx, uid, "", &buf); err != nil {
1340
+
t.Fatalf("ReadRepo: %v", err)
1341
+
}
1342
+
1343
+
// Parse it back
1344
+
r, err := repo.ReadRepoFromCar(ctx, &buf)
1345
+
if err != nil {
1346
+
t.Fatalf("ReadRepoFromCar: %v", err)
1347
+
}
1348
+
1349
+
// Both records should be retrievable
1350
+
for _, rkey := range []string{"cartest1", "cartest2"} {
1351
+
rpath := atproto.CrewCollection + "/" + rkey
1352
+
_, _, err := r.GetRecord(ctx, rpath)
1353
+
if err != nil {
1354
+
t.Errorf("record %q not found in CAR: %v", rpath, err)
1355
+
}
1356
+
}
1357
+
1358
+
// Unknown record should fail
1359
+
_, _, err = r.GetRecord(ctx, atproto.CrewCollection+"/doesnotexist")
1360
+
if err == nil {
1361
+
t.Error("expected error for non-existent record in CAR")
1362
+
}
1363
+
})
1364
+
}
1365
+
1366
+
// setupFreshRepoManager returns a RepoManager WITHOUT InitNewActor called.
1367
+
func setupFreshRepoManager(t *testing.T) (RepoOperator, models.Uid, string) {
1368
+
t.Helper()
1369
+
ctx := context.Background()
1370
+
keyPath := filepath.Join(t.TempDir(), "signing-key")
1371
+
if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil {
1372
+
t.Fatalf("Failed to write signing key: %v", err)
1373
+
}
1374
+
pds, err := NewHoldPDS(ctx, "did:web:hold.init-test", "https://hold.init-test", "https://atcr.io", ":memory:", keyPath, false)
1375
+
if err != nil {
1376
+
t.Fatalf("NewHoldPDS: %v", err)
1377
+
}
1378
+
t.Cleanup(func() { pds.Close() })
1379
+
return pds.repomgr, pds.uid, pds.did
1380
+
}
1381
+
1382
+
// setupFreshDirectRepoOperator returns a DirectRepoOperator WITHOUT InitNewActor called.
1383
+
func setupFreshDirectRepoOperator(t *testing.T) (RepoOperator, models.Uid, string) {
1384
+
t.Helper()
1385
+
1386
+
keyPath := filepath.Join(t.TempDir(), "signing-key")
1387
+
if err := os.WriteFile(keyPath, sharedTestKey, 0600); err != nil {
1388
+
t.Fatalf("Failed to write signing key: %v", err)
1389
+
}
1390
+
signingKey, err := oauth.GenerateOrLoadPDSKey(keyPath)
1391
+
if err != nil {
1392
+
t.Fatalf("GenerateOrLoadPDSKey: %v", err)
1393
+
}
1394
+
1395
+
sqlStore := new(holddb.SQLiteStore)
1396
+
if err := sqlStore.Open(":memory:"); err != nil {
1397
+
t.Fatalf("SQLiteStore.Open: %v", err)
1398
+
}
1399
+
t.Cleanup(func() { sqlStore.Close() })
1400
+
1401
+
kmgr := NewHoldKeyManager(signingKey)
1402
+
op := NewDirectRepoOperator(sqlStore, kmgr)
1403
+
1404
+
return op, models.Uid(1), "did:web:hold.test"
1405
+
}
1406
+
1407
+
func TestRepoManager(t *testing.T) {
1408
+
runRepoOperatorTests(t, setupTestRepoOperator, setupFreshRepoManager)
1409
+
}
1410
+
1411
+
func TestDirectRepoOperator(t *testing.T) {
1412
+
runRepoOperatorTests(t, setupTestDirectRepoOperator, setupFreshDirectRepoOperator)
1413
+
}
+62
-662
pkg/hold/pds/repomgr.go
+62
-662
pkg/hold/pds/repomgr.go
···
1
-
// Package pds contains a vendored copy of RepoManager from github.com/bluesky-social/indigo
1
+
package pds
2
+
3
+
// repomgr.go — RepoManager manages ATProto repository operations for the hold PDS.
2
4
//
3
-
// Source: github.com/bluesky-social/indigo/repomgr (v0.0.0-20251014222321)
4
-
// Reference: github.com/streamplace/indigo (67ae5a5) for PutRecord implementation
5
-
// Reason: The indigo library is unmaintained and contains a critical bug in UpdateRecord
5
+
// Originally vendored from github.com/bluesky-social/indigo/repomgr with fixes:
6
+
// - Fixed UpdateRecord bug (was calling PutRecord internally)
7
+
// - Added PutRecord/UpsertRecord for explicit rkey operations
8
+
// - Added prevData support for Sync 1.1
6
9
//
7
-
// Modifications from original:
8
-
// - Changed package from 'repomgr' to 'pds' for integration with hold service
9
-
// - Fixed UpdateRecord bug (line 263): Changed r.PutRecord to r.UpdateRecord
10
-
// (UpdateRecord was incorrectly calling PutRecord, causing incorrect MST operations)
11
-
// - Removed 5 Prometheus metrics calls (openAndSigCheckDuration, calcDiffDuration,
12
-
// writeCarSliceDuration, repoOpsImported) as metrics are not used in this project
13
-
// - Added PutRecord method (lines 309-381) for creating records with explicit rkeys
14
-
// (like CreateRecord but with specified rkey instead of auto-generated TID)
15
-
// Based on streamplace/indigo implementation
16
-
// - Added prevData to support Sync 1.1
17
-
package pds
10
+
// Implements the RepoOperator interface (see repo_operator.go).
11
+
// See docs/REPOMGR_MIGRATION.md for planned migration to indigo/repo directly.
18
12
19
13
import (
20
-
"bytes"
21
14
"context"
22
-
"errors"
23
15
"fmt"
24
16
"io"
25
17
"log/slog"
26
-
"strings"
27
18
"sync"
28
19
29
20
holddb "atcr.io/pkg/hold/db"
30
21
atproto "github.com/bluesky-social/indigo/api/atproto"
31
22
bsky "github.com/bluesky-social/indigo/api/bsky"
32
23
"github.com/bluesky-social/indigo/atproto/syntax"
33
-
lexutil "github.com/bluesky-social/indigo/lex/util"
34
24
"github.com/bluesky-social/indigo/models"
35
-
"github.com/bluesky-social/indigo/mst"
36
25
"github.com/bluesky-social/indigo/repo"
37
26
"github.com/bluesky-social/indigo/util"
38
27
39
28
blocks "github.com/ipfs/go-block-format"
40
29
"github.com/ipfs/go-cid"
41
-
"github.com/ipfs/go-datastore"
42
-
blockstore "github.com/ipfs/go-ipfs-blockstore"
43
-
ipld "github.com/ipfs/go-ipld-format"
44
-
"github.com/ipld/go-car"
45
30
cbg "github.com/whyrusleeping/cbor-gen"
46
31
"go.opentelemetry.io/otel"
47
-
"go.opentelemetry.io/otel/attribute"
48
-
"gorm.io/gorm"
49
32
)
50
33
51
34
func NewRepoManager(cs holddb.CarStore, kmgr KeyManager) *RepoManager {
···
56
39
userLocks: make(map[models.Uid]*userLock),
57
40
kmgr: kmgr,
58
41
log: slog.Default().With("system", "repomgr"),
59
-
noArchive: false, // NonArchivalCarstore not used in hold service
60
42
clk: clk,
61
43
}
62
44
}
63
45
64
-
type KeyManager interface {
65
-
VerifyUserSignature(context.Context, string, []byte, []byte) error
66
-
SignForUser(context.Context, string, []byte) ([]byte, error)
67
-
}
68
-
69
46
func (rm *RepoManager) SetEventHandler(cb func(context.Context, *RepoEvent), hydrateRecords bool) {
70
47
rm.events = cb
71
48
rm.hydrateRecords = hydrateRecords
···
81
58
events func(context.Context, *RepoEvent)
82
59
hydrateRecords bool
83
60
84
-
log *slog.Logger
85
-
noArchive bool
86
-
61
+
log *slog.Logger
87
62
clk *syntax.TIDClock
88
63
}
89
64
90
-
// NextTID generates a new TID for use as a record key.
91
-
func (rm *RepoManager) NextTID() string {
92
-
return rm.clk.Next().String()
93
-
}
94
-
95
-
type ActorInfo struct {
96
-
Did string
97
-
Handle string
98
-
DisplayName string
99
-
Type string
100
-
}
101
-
102
-
type RepoEvent struct {
103
-
User models.Uid
104
-
OldRoot *cid.Cid
105
-
NewRoot cid.Cid
106
-
PrevData *cid.Cid // MST root CID of the previous commit (for firehose prevData field)
107
-
Since *string
108
-
Rev string
109
-
RepoSlice []byte
110
-
PDS uint
111
-
Ops []RepoOp
112
-
}
113
-
114
-
type RepoOp struct {
115
-
Kind EventKind
116
-
Collection string
117
-
Rkey string
118
-
RecCid *cid.Cid
119
-
Record any
120
-
ActorInfo *ActorInfo
121
-
}
122
-
123
-
type EventKind string
124
-
125
-
const (
126
-
EvtKindCreateRecord = EventKind("create")
127
-
EvtKindUpdateRecord = EventKind("update")
128
-
EvtKindDeleteRecord = EventKind("delete")
129
-
)
130
-
131
-
type RepoHead struct {
132
-
gorm.Model
133
-
Usr models.Uid `gorm:"uniqueIndex"`
134
-
Root string
135
-
}
136
-
137
65
type userLock struct {
138
66
lk sync.Mutex
139
67
count int
···
168
96
}
169
97
rm.lklk.Unlock()
170
98
}
171
-
}
172
-
173
-
func (rm *RepoManager) CarStore() holddb.CarStore {
174
-
return rm.cs
175
99
}
176
100
177
101
func (rm *RepoManager) CreateRecord(ctx context.Context, user models.Uid, collection string, rec cbg.CBORMarshaler) (string, cid.Cid, error) {
···
707
631
return head, bs.GetLoggedBlocks(), nil
708
632
}
709
633
710
-
func (rm *RepoManager) GetProfile(ctx context.Context, uid models.Uid) (*bsky.ActorProfile, error) {
711
-
bs, err := rm.cs.ReadOnlySession(uid)
712
-
if err != nil {
713
-
return nil, err
714
-
}
715
-
716
-
head, err := rm.cs.GetUserRepoHead(ctx, uid)
717
-
if err != nil {
718
-
return nil, err
719
-
}
720
-
721
-
r, err := repo.OpenRepo(ctx, bs, head)
722
-
if err != nil {
723
-
return nil, err
724
-
}
725
-
726
-
_, val, err := r.GetRecord(ctx, "app.bsky.actor.profile/self")
727
-
if err != nil {
728
-
return nil, err
729
-
}
730
-
731
-
ap, ok := val.(*bsky.ActorProfile)
732
-
if !ok {
733
-
return nil, fmt.Errorf("found wrong type in actor profile location in tree")
734
-
}
735
-
736
-
return ap, nil
737
-
}
738
-
739
-
func (rm *RepoManager) CheckRepoSig(ctx context.Context, r *repo.Repo, expdid string) error {
740
-
ctx, span := otel.Tracer("repoman").Start(ctx, "CheckRepoSig")
741
-
defer span.End()
742
-
743
-
repoDid := r.RepoDid()
744
-
if expdid != repoDid {
745
-
return fmt.Errorf("DID in repo did not match (%q != %q)", expdid, repoDid)
746
-
}
747
-
748
-
scom := r.SignedCommit()
749
-
750
-
usc := scom.Unsigned()
751
-
sb, err := usc.BytesForSigning()
752
-
if err != nil {
753
-
return fmt.Errorf("commit serialization failed: %w", err)
754
-
}
755
-
if err := rm.kmgr.VerifyUserSignature(ctx, repoDid, scom.Sig, sb); err != nil {
756
-
return fmt.Errorf("signature check failed (sig: %x) (sb: %x) : %w", scom.Sig, sb, err)
757
-
}
758
-
759
-
return nil
760
-
}
761
-
762
-
func (rm *RepoManager) HandleExternalUserEvent(ctx context.Context, pdsid uint, uid models.Uid, did string, since *string, nrev string, carslice []byte, ops []*atproto.SyncSubscribeRepos_RepoOp) error {
763
-
if rm.noArchive {
764
-
return rm.handleExternalUserEventNoArchive(ctx, pdsid, uid, did, since, nrev, carslice, ops)
765
-
} else {
766
-
return rm.handleExternalUserEventArchive(ctx, pdsid, uid, did, since, nrev, carslice, ops)
767
-
}
768
-
}
769
-
770
-
func (rm *RepoManager) handleExternalUserEventNoArchive(ctx context.Context, pdsid uint, uid models.Uid, did string, since *string, nrev string, carslice []byte, ops []*atproto.SyncSubscribeRepos_RepoOp) error {
771
-
ctx, span := otel.Tracer("repoman").Start(ctx, "HandleExternalUserEvent")
772
-
defer span.End()
773
-
774
-
span.SetAttributes(attribute.Int64("uid", int64(uid)))
775
-
776
-
rm.log.Debug("HandleExternalUserEvent", "pds", pdsid, "uid", uid, "since", since, "nrev", nrev)
777
-
778
-
unlock := rm.lockUser(ctx, uid)
779
-
defer unlock()
780
-
781
-
root, ds, err := rm.cs.ImportSlice(ctx, uid, since, carslice)
782
-
if err != nil {
783
-
return fmt.Errorf("importing external carslice: %w", err)
784
-
}
785
-
786
-
r, err := repo.OpenRepo(ctx, ds, root)
787
-
if err != nil {
788
-
return fmt.Errorf("opening external user repo (%d, root=%s): %w", uid, root, err)
789
-
}
790
-
791
-
if err := rm.CheckRepoSig(ctx, r, did); err != nil {
792
-
return fmt.Errorf("check repo sig: %w", err)
793
-
}
794
-
795
-
// Capture previous MST root from old repo state if it exists
796
-
var prevData *cid.Cid
797
-
if ds.BaseCid().Defined() {
798
-
oldrepo, err := repo.OpenRepo(ctx, ds, ds.BaseCid())
799
-
if err == nil {
800
-
pd := oldrepo.DataCid()
801
-
prevData = &pd
802
-
}
803
-
}
804
-
805
-
evtops := make([]RepoOp, 0, len(ops))
806
-
for _, op := range ops {
807
-
parts := strings.SplitN(op.Path, "/", 2)
808
-
if len(parts) != 2 {
809
-
return fmt.Errorf("invalid rpath in mst diff, must have collection and rkey")
810
-
}
811
-
812
-
switch EventKind(op.Action) {
813
-
case EvtKindCreateRecord:
814
-
rop := RepoOp{
815
-
Kind: EvtKindCreateRecord,
816
-
Collection: parts[0],
817
-
Rkey: parts[1],
818
-
RecCid: (*cid.Cid)(op.Cid),
819
-
}
820
-
821
-
if rm.hydrateRecords {
822
-
_, rec, err := r.GetRecord(ctx, op.Path)
823
-
if err != nil {
824
-
return fmt.Errorf("reading changed record from car slice: %w", err)
825
-
}
826
-
rop.Record = rec
827
-
}
828
-
829
-
evtops = append(evtops, rop)
830
-
case EvtKindUpdateRecord:
831
-
rop := RepoOp{
832
-
Kind: EvtKindUpdateRecord,
833
-
Collection: parts[0],
834
-
Rkey: parts[1],
835
-
RecCid: (*cid.Cid)(op.Cid),
836
-
}
837
-
838
-
if rm.hydrateRecords {
839
-
_, rec, err := r.GetRecord(ctx, op.Path)
840
-
if err != nil {
841
-
return fmt.Errorf("reading changed record from car slice: %w", err)
842
-
}
843
-
844
-
rop.Record = rec
845
-
}
846
-
847
-
evtops = append(evtops, rop)
848
-
case EvtKindDeleteRecord:
849
-
evtops = append(evtops, RepoOp{
850
-
Kind: EvtKindDeleteRecord,
851
-
Collection: parts[0],
852
-
Rkey: parts[1],
853
-
})
854
-
default:
855
-
return fmt.Errorf("unrecognized external user event kind: %q", op.Action)
856
-
}
857
-
}
858
-
859
-
if rm.events != nil {
860
-
rm.events(ctx, &RepoEvent{
861
-
User: uid,
862
-
//OldRoot: prev,
863
-
NewRoot: root,
864
-
PrevData: prevData,
865
-
Rev: nrev,
866
-
Since: since,
867
-
Ops: evtops,
868
-
RepoSlice: carslice,
869
-
PDS: pdsid,
870
-
})
871
-
}
872
-
873
-
return nil
874
-
}
875
-
876
-
func (rm *RepoManager) handleExternalUserEventArchive(ctx context.Context, pdsid uint, uid models.Uid, did string, since *string, nrev string, carslice []byte, ops []*atproto.SyncSubscribeRepos_RepoOp) error {
877
-
ctx, span := otel.Tracer("repoman").Start(ctx, "HandleExternalUserEvent")
878
-
defer span.End()
879
-
880
-
span.SetAttributes(attribute.Int64("uid", int64(uid)))
881
-
882
-
rm.log.Debug("HandleExternalUserEvent", "pds", pdsid, "uid", uid, "since", since, "nrev", nrev)
883
-
884
-
unlock := rm.lockUser(ctx, uid)
885
-
defer unlock()
886
-
887
-
root, ds, err := rm.cs.ImportSlice(ctx, uid, since, carslice)
888
-
if err != nil {
889
-
return fmt.Errorf("importing external carslice: %w", err)
890
-
}
891
-
892
-
r, err := repo.OpenRepo(ctx, ds, root)
893
-
if err != nil {
894
-
return fmt.Errorf("opening external user repo (%d, root=%s): %w", uid, root, err)
895
-
}
896
-
897
-
if err := rm.CheckRepoSig(ctx, r, did); err != nil {
898
-
return err
899
-
}
900
-
901
-
var skipcids map[cid.Cid]bool
902
-
var prevData *cid.Cid
903
-
if ds.BaseCid().Defined() {
904
-
oldrepo, err := repo.OpenRepo(ctx, ds, ds.BaseCid())
905
-
if err != nil {
906
-
return fmt.Errorf("failed to check data root in old repo: %w", err)
907
-
}
908
-
909
-
// Capture previous MST root for prevData
910
-
pd := oldrepo.DataCid()
911
-
prevData = &pd
912
-
913
-
// if the old commit has a 'prev', CalcDiff will error out while trying
914
-
// to walk it. This is an old repo thing that is being deprecated.
915
-
// This check is a temporary workaround until all repos get migrated
916
-
// and this becomes no longer an issue
917
-
prev, _ := oldrepo.PrevCommit(ctx)
918
-
if prev != nil {
919
-
skipcids = map[cid.Cid]bool{
920
-
*prev: true,
921
-
}
922
-
}
923
-
}
924
-
925
-
if err := ds.CalcDiff(ctx, skipcids); err != nil {
926
-
return fmt.Errorf("failed while calculating mst diff (since=%v): %w", since, err)
927
-
}
928
-
929
-
evtops := make([]RepoOp, 0, len(ops))
930
-
931
-
for _, op := range ops {
932
-
parts := strings.SplitN(op.Path, "/", 2)
933
-
if len(parts) != 2 {
934
-
return fmt.Errorf("invalid rpath in mst diff, must have collection and rkey")
935
-
}
936
-
937
-
switch EventKind(op.Action) {
938
-
case EvtKindCreateRecord:
939
-
rop := RepoOp{
940
-
Kind: EvtKindCreateRecord,
941
-
Collection: parts[0],
942
-
Rkey: parts[1],
943
-
RecCid: (*cid.Cid)(op.Cid),
944
-
}
945
-
946
-
if rm.hydrateRecords {
947
-
_, rec, err := r.GetRecord(ctx, op.Path)
948
-
if err != nil {
949
-
return fmt.Errorf("reading changed record from car slice: %w", err)
950
-
}
951
-
rop.Record = rec
952
-
}
953
-
954
-
evtops = append(evtops, rop)
955
-
case EvtKindUpdateRecord:
956
-
rop := RepoOp{
957
-
Kind: EvtKindUpdateRecord,
958
-
Collection: parts[0],
959
-
Rkey: parts[1],
960
-
RecCid: (*cid.Cid)(op.Cid),
961
-
}
962
-
963
-
if rm.hydrateRecords {
964
-
_, rec, err := r.GetRecord(ctx, op.Path)
965
-
if err != nil {
966
-
return fmt.Errorf("reading changed record from car slice: %w", err)
967
-
}
968
-
969
-
rop.Record = rec
970
-
}
971
-
972
-
evtops = append(evtops, rop)
973
-
case EvtKindDeleteRecord:
974
-
evtops = append(evtops, RepoOp{
975
-
Kind: EvtKindDeleteRecord,
976
-
Collection: parts[0],
977
-
Rkey: parts[1],
978
-
})
979
-
default:
980
-
return fmt.Errorf("unrecognized external user event kind: %q", op.Action)
981
-
}
982
-
}
983
-
984
-
rslice, err := ds.CloseWithRoot(ctx, root, nrev)
985
-
if err != nil {
986
-
return fmt.Errorf("close with root: %w", err)
987
-
}
988
-
989
-
if rm.events != nil {
990
-
rm.events(ctx, &RepoEvent{
991
-
User: uid,
992
-
//OldRoot: prev,
993
-
NewRoot: root,
994
-
PrevData: prevData,
995
-
Rev: nrev,
996
-
Since: since,
997
-
Ops: evtops,
998
-
RepoSlice: rslice,
999
-
PDS: pdsid,
1000
-
})
1001
-
}
1002
-
1003
-
return nil
1004
-
}
1005
-
1006
634
func (rm *RepoManager) BatchWrite(ctx context.Context, user models.Uid, writes []*atproto.RepoApplyWrites_Input_Writes_Elem) error {
1007
635
ctx, span := otel.Tracer("repoman").Start(ctx, "BatchWrite")
1008
636
defer span.End()
···
1131
759
return nil
1132
760
}
1133
761
1134
-
func (rm *RepoManager) ImportNewRepo(ctx context.Context, user models.Uid, repoDid string, r io.Reader, rev *string) error {
1135
-
ctx, span := otel.Tracer("repoman").Start(ctx, "ImportNewRepo")
762
+
// BulkUpsert writes multiple records in a single delta session and commit.
763
+
// Each record is upserted: created if new, updated if it already exists.
764
+
func (rm *RepoManager) BulkUpsert(ctx context.Context, user models.Uid, records []BulkRecord) error {
765
+
ctx, span := otel.Tracer("repoman").Start(ctx, "BulkUpsert")
1136
766
defer span.End()
1137
767
1138
768
unlock := rm.lockUser(ctx, user)
1139
769
defer unlock()
1140
770
1141
-
currev, err := rm.cs.GetUserRepoRev(ctx, user)
771
+
rev, err := rm.cs.GetUserRepoRev(ctx, user)
1142
772
if err != nil {
1143
773
return err
1144
774
}
1145
775
1146
-
curhead, err := rm.cs.GetUserRepoHead(ctx, user)
776
+
ds, err := rm.cs.NewDeltaSession(ctx, user, &rev)
1147
777
if err != nil {
1148
778
return err
1149
779
}
1150
780
1151
-
if rev != nil && *rev == "" {
1152
-
rev = nil
1153
-
}
1154
-
if rev == nil {
1155
-
// if 'rev' is nil, this implies a fresh sync.
1156
-
// in this case, ignore any existing blocks we have and treat this like a clean import.
1157
-
curhead = cid.Undef
1158
-
}
1159
-
1160
-
if rev != nil && *rev != currev {
1161
-
// TODO: we could probably just deal with this
1162
-
return fmt.Errorf("ImportNewRepo called with incorrect base")
1163
-
}
1164
-
1165
-
// Capture previous MST root before import overwrites it
1166
-
var prevData *cid.Cid
1167
-
if curhead.Defined() {
1168
-
robs, err := rm.cs.ReadOnlySession(user)
1169
-
if err == nil {
1170
-
oldrepo, err := repo.OpenRepo(ctx, robs, curhead)
1171
-
if err == nil {
1172
-
pd := oldrepo.DataCid()
1173
-
prevData = &pd
1174
-
}
1175
-
}
1176
-
}
1177
-
1178
-
err = rm.processNewRepo(ctx, user, r, rev, func(ctx context.Context, root cid.Cid, finish func(context.Context, string) ([]byte, error), bs blockstore.Blockstore) error {
1179
-
r, err := repo.OpenRepo(ctx, bs, root)
1180
-
if err != nil {
1181
-
return fmt.Errorf("opening new repo: %w", err)
1182
-
}
1183
-
1184
-
scom := r.SignedCommit()
1185
-
1186
-
usc := scom.Unsigned()
1187
-
sb, err := usc.BytesForSigning()
1188
-
if err != nil {
1189
-
return fmt.Errorf("commit serialization failed: %w", err)
1190
-
}
1191
-
if err := rm.kmgr.VerifyUserSignature(ctx, repoDid, scom.Sig, sb); err != nil {
1192
-
return fmt.Errorf("new user signature check failed: %w", err)
1193
-
}
1194
-
1195
-
diffops, err := r.DiffSince(ctx, curhead)
1196
-
if err != nil {
1197
-
return fmt.Errorf("diff trees (curhead: %s): %w", curhead, err)
1198
-
}
1199
-
1200
-
ops := make([]RepoOp, 0, len(diffops))
1201
-
for _, op := range diffops {
1202
-
out, err := rm.processOp(ctx, bs, op, rm.hydrateRecords)
1203
-
if err != nil {
1204
-
rm.log.Error("failed to process repo op", "err", err, "path", op.Rpath, "repo", repoDid)
1205
-
}
1206
-
1207
-
if out != nil {
1208
-
ops = append(ops, *out)
1209
-
}
1210
-
}
1211
-
1212
-
slice, err := finish(ctx, scom.Rev)
1213
-
if err != nil {
1214
-
return err
1215
-
}
1216
-
1217
-
if rm.events != nil {
1218
-
rm.events(ctx, &RepoEvent{
1219
-
User: user,
1220
-
//OldRoot: oldroot,
1221
-
NewRoot: root,
1222
-
PrevData: prevData,
1223
-
Rev: scom.Rev,
1224
-
Since: &currev,
1225
-
RepoSlice: slice,
1226
-
Ops: ops,
1227
-
})
1228
-
}
1229
-
1230
-
return nil
1231
-
})
1232
-
if err != nil {
1233
-
return fmt.Errorf("process new repo (current rev: %s): %w", currev, err)
1234
-
}
1235
-
1236
-
return nil
1237
-
}
1238
-
1239
-
func (rm *RepoManager) processOp(ctx context.Context, bs blockstore.Blockstore, op *mst.DiffOp, hydrateRecords bool) (*RepoOp, error) {
1240
-
parts := strings.SplitN(op.Rpath, "/", 2)
1241
-
if len(parts) != 2 {
1242
-
return nil, fmt.Errorf("repo mst had invalid rpath: %q", op.Rpath)
1243
-
}
1244
-
1245
-
switch op.Op {
1246
-
case "add", "mut":
1247
-
1248
-
kind := EvtKindCreateRecord
1249
-
if op.Op == "mut" {
1250
-
kind = EvtKindUpdateRecord
1251
-
}
1252
-
1253
-
outop := &RepoOp{
1254
-
Kind: kind,
1255
-
Collection: parts[0],
1256
-
Rkey: parts[1],
1257
-
RecCid: &op.NewCid,
1258
-
}
1259
-
1260
-
if hydrateRecords {
1261
-
blk, err := bs.Get(ctx, op.NewCid)
1262
-
if err != nil {
1263
-
return nil, err
1264
-
}
1265
-
1266
-
rec, err := lexutil.CborDecodeValue(blk.RawData())
1267
-
if err != nil {
1268
-
if !errors.Is(err, lexutil.ErrUnrecognizedType) {
1269
-
return nil, err
1270
-
}
1271
-
1272
-
rm.log.Warn("failed processing repo diff", "err", err)
1273
-
} else {
1274
-
outop.Record = rec
1275
-
}
1276
-
}
1277
-
1278
-
return outop, nil
1279
-
case "del":
1280
-
return &RepoOp{
1281
-
Kind: EvtKindDeleteRecord,
1282
-
Collection: parts[0],
1283
-
Rkey: parts[1],
1284
-
RecCid: nil,
1285
-
}, nil
1286
-
1287
-
default:
1288
-
return nil, fmt.Errorf("diff returned invalid op type: %q", op.Op)
1289
-
}
1290
-
}
1291
-
1292
-
func (rm *RepoManager) processNewRepo(ctx context.Context, user models.Uid, r io.Reader, rev *string, cb func(ctx context.Context, root cid.Cid, finish func(context.Context, string) ([]byte, error), bs blockstore.Blockstore) error) error {
1293
-
ctx, span := otel.Tracer("repoman").Start(ctx, "processNewRepo")
1294
-
defer span.End()
1295
-
1296
-
carr, err := car.NewCarReader(r)
781
+
head := ds.BaseCid()
782
+
r, err := repo.OpenRepo(ctx, ds, head)
1297
783
if err != nil {
1298
784
return err
1299
785
}
1300
786
1301
-
if len(carr.Header.Roots) != 1 {
1302
-
return fmt.Errorf("invalid car file, header must have a single root (has %d)", len(carr.Header.Roots))
787
+
// Capture previous MST root before commit overwrites it
788
+
var prevData *cid.Cid
789
+
if head.Defined() {
790
+
pd := r.DataCid()
791
+
prevData = &pd
1303
792
}
1304
793
1305
-
membs := blockstore.NewBlockstore(datastore.NewMapDatastore())
794
+
ops := make([]RepoOp, 0, len(records))
795
+
for _, rec := range records {
796
+
rpath := rec.Collection + "/" + rec.Rkey
1306
797
1307
-
for {
1308
-
blk, err := carr.Next()
1309
-
if err != nil {
1310
-
if err == io.EOF {
1311
-
break
1312
-
}
1313
-
return err
1314
-
}
798
+
// Check if record exists to determine create vs update
799
+
_, _, getErr := r.GetRecordBytes(ctx, rpath)
800
+
recordExists := getErr == nil
1315
801
1316
-
if err := membs.Put(ctx, blk); err != nil {
1317
-
return err
802
+
var cc cid.Cid
803
+
var evtKind EventKind
804
+
if recordExists {
805
+
cc, err = r.UpdateRecord(ctx, rpath, rec.Data)
806
+
evtKind = EvtKindUpdateRecord
807
+
} else {
808
+
cc, err = r.PutRecord(ctx, rpath, rec.Data)
809
+
evtKind = EvtKindCreateRecord
1318
810
}
1319
-
}
1320
-
1321
-
seen := make(map[cid.Cid]bool)
1322
-
1323
-
root := carr.Header.Roots[0]
1324
-
// TODO: if there are blocks that get convergently recreated throughout
1325
-
// the repos lifecycle, this will end up erroneously not including
1326
-
// them. We should compute the set of blocks needed to read any repo
1327
-
// ops that happened in the commit and use that for our 'output' blocks
1328
-
cids, err := rm.walkTree(ctx, seen, root, membs, true)
1329
-
if err != nil {
1330
-
return fmt.Errorf("walkTree: %w", err)
1331
-
}
1332
-
1333
-
ds, err := rm.cs.NewDeltaSession(ctx, user, rev)
1334
-
if err != nil {
1335
-
return fmt.Errorf("opening delta session: %w", err)
1336
-
}
1337
-
1338
-
for _, c := range cids {
1339
-
blk, err := membs.Get(ctx, c)
1340
811
if err != nil {
1341
-
return fmt.Errorf("copying walked cids to carstore: %w", err)
812
+
return fmt.Errorf("failed to write %s: %w", rpath, err)
1342
813
}
1343
814
1344
-
if err := ds.Put(ctx, blk); err != nil {
1345
-
return err
1346
-
}
815
+
ops = append(ops, RepoOp{
816
+
Kind: evtKind,
817
+
Collection: rec.Collection,
818
+
Rkey: rec.Rkey,
819
+
RecCid: &cc,
820
+
})
1347
821
}
1348
822
1349
-
finish := func(ctx context.Context, nrev string) ([]byte, error) {
1350
-
return ds.CloseWithRoot(ctx, root, nrev)
1351
-
}
1352
-
1353
-
if err := cb(ctx, root, finish, ds); err != nil {
1354
-
return fmt.Errorf("cb errored root: %s, rev: %s: %w", root, stringOrNil(rev), err)
1355
-
}
1356
-
1357
-
return nil
1358
-
}
1359
-
1360
-
func stringOrNil(s *string) string {
1361
-
if s == nil {
1362
-
return "nil"
1363
-
}
1364
-
return *s
1365
-
}
1366
-
1367
-
// walkTree returns all cids linked recursively by the root, skipping any cids
1368
-
// in the 'skip' map, and not erroring on 'not found' if prevMissing is set
1369
-
func (rm *RepoManager) walkTree(ctx context.Context, skip map[cid.Cid]bool, root cid.Cid, bs blockstore.Blockstore, prevMissing bool) ([]cid.Cid, error) {
1370
-
// TODO: what if someone puts non-cbor links in their repo?
1371
-
if root.Prefix().Codec != cid.DagCBOR {
1372
-
return nil, fmt.Errorf("can only handle dag-cbor objects in repos (%s is %d)", root, root.Prefix().Codec)
1373
-
}
1374
-
1375
-
blk, err := bs.Get(ctx, root)
1376
-
if err != nil {
1377
-
return nil, err
1378
-
}
1379
-
1380
-
var links []cid.Cid
1381
-
if err := cbg.ScanForLinks(bytes.NewReader(blk.RawData()), func(c cid.Cid) {
1382
-
if c.Prefix().Codec == cid.Raw {
1383
-
rm.log.Debug("skipping 'raw' CID in record", "recordCid", root, "rawCid", c)
1384
-
return
1385
-
}
1386
-
if skip[c] {
1387
-
return
1388
-
}
1389
-
1390
-
links = append(links, c)
1391
-
skip[c] = true
1392
-
}); err != nil {
1393
-
return nil, err
1394
-
}
1395
-
1396
-
out := []cid.Cid{root}
1397
-
skip[root] = true
1398
-
1399
-
// TODO: should do this non-recursive since i expect these may get deep
1400
-
for _, c := range links {
1401
-
sub, err := rm.walkTree(ctx, skip, c, bs, prevMissing)
1402
-
if err != nil {
1403
-
if prevMissing && !ipld.IsNotFound(err) {
1404
-
return nil, err
1405
-
}
1406
-
}
1407
-
1408
-
out = append(out, sub...)
1409
-
}
1410
-
1411
-
return out, nil
1412
-
}
1413
-
1414
-
func (rm *RepoManager) TakeDownRepo(ctx context.Context, uid models.Uid) error {
1415
-
unlock := rm.lockUser(ctx, uid)
1416
-
defer unlock()
1417
-
1418
-
return rm.cs.WipeUserData(ctx, uid)
1419
-
}
1420
-
1421
-
// ResetRepo is technically identical to TakeDownRepo, for now
1422
-
func (rm *RepoManager) ResetRepo(ctx context.Context, uid models.Uid) error {
1423
-
unlock := rm.lockUser(ctx, uid)
1424
-
defer unlock()
1425
-
1426
-
return rm.cs.WipeUserData(ctx, uid)
1427
-
}
1428
-
1429
-
func (rm *RepoManager) VerifyRepo(ctx context.Context, uid models.Uid) error {
1430
-
ses, err := rm.cs.ReadOnlySession(uid)
823
+
nroot, nrev, err := r.Commit(ctx, rm.kmgr.SignForUser)
1431
824
if err != nil {
1432
825
return err
1433
826
}
1434
827
1435
-
r, err := repo.OpenRepo(ctx, ses, ses.BaseCid())
828
+
rslice, err := ds.CloseWithRoot(ctx, nroot, nrev)
1436
829
if err != nil {
1437
-
return err
830
+
return fmt.Errorf("close with root: %w", err)
1438
831
}
1439
832
1440
-
if err := r.ForEach(ctx, "", func(k string, v cid.Cid) error {
1441
-
_, err := ses.Get(ctx, v)
1442
-
if err != nil {
1443
-
return fmt.Errorf("failed to get record %s (%s): %w", k, v, err)
1444
-
}
833
+
var oldroot *cid.Cid
834
+
if head.Defined() {
835
+
oldroot = &head
836
+
}
1445
837
1446
-
return nil
1447
-
}); err != nil {
1448
-
return err
838
+
if rm.events != nil {
839
+
rm.events(ctx, &RepoEvent{
840
+
User: user,
841
+
OldRoot: oldroot,
842
+
NewRoot: nroot,
843
+
PrevData: prevData,
844
+
Rev: nrev,
845
+
Since: &rev,
846
+
Ops: ops,
847
+
RepoSlice: rslice,
848
+
})
1449
849
}
1450
850
1451
851
return nil
+6
-6
pkg/hold/pds/server.go
+6
-6
pkg/hold/pds/server.go
···
41
41
appviewURL string
42
42
appviewMeta *atproto.AppviewMetadata
43
43
carstore holddb.CarStore
44
-
repomgr *RepoManager
44
+
repomgr RepoOperator
45
45
dbPath string
46
46
uid models.Uid
47
47
signingKey *atcrypto.PrivateKeyK256
···
106
106
// Create KeyManager wrapper for our signing key
107
107
kmgr := NewHoldKeyManager(signingKey)
108
108
109
-
// Create RepoManager - it will handle all session/repo lifecycle
110
-
rm := NewRepoManager(cs, kmgr)
109
+
// Create repo operator - handles all session/repo lifecycle
110
+
rm := NewDirectRepoOperator(cs, kmgr)
111
111
112
112
// Check if repo already exists, if not create initial commit
113
113
head, err := cs.GetUserRepoHead(ctx, uid)
···
162
162
cs := sqlStore
163
163
uid := models.Uid(1)
164
164
kmgr := NewHoldKeyManager(signingKey)
165
-
rm := NewRepoManager(cs, kmgr)
165
+
rm := NewDirectRepoOperator(cs, kmgr)
166
166
167
167
head, err := cs.GetUserRepoHead(ctx, uid)
168
168
hasValidRepo := (err == nil && head.Defined())
···
200
200
return p.signingKey
201
201
}
202
202
203
-
// RepomgrRef returns a reference to the RepoManager for event handler setup
204
-
func (p *HoldPDS) RepomgrRef() *RepoManager {
203
+
// RepomgrRef returns a reference to the RepoOperator for event handler setup
204
+
func (p *HoldPDS) RepomgrRef() RepoOperator {
205
205
return p.repomgr
206
206
}
207
207