this repo has no description
Update readme
+66
-6
+38
-6
cmd/butterfly/README.md
+38
-6
cmd/butterfly/README.md
···
2
2
3
3
A sync engine for atproto with an optional baked-in database.
4
4
5
+
```
6
+
# Sync from a CAR file
7
+
butterfly sync --input carfile --car file.car --did did:plc:example --store duckdb
8
+
9
+
# Sync from a PDS
10
+
butterfly sync --input pds --pds https://pds.example.com --did did:plc:example
11
+
12
+
# Subscribe to a relay
13
+
butterfly sync --input relay --relay wss://relay.example.com --store stdout
14
+
15
+
# Discover repositories on a PDS
16
+
butterfly discover --input pds --pds https://pds.example.com --limit 50 --format json
17
+
18
+
# Discover with identity resolution
19
+
butterfly discover --input relay --relay wss://relay.example.com --store stdout
20
+
```
21
+
5
22
## WIP notes
6
23
7
24
- Until the Store interface has stabilized, let's only work on stdout and duckdb to keep things simple.
8
25
26
+
```
27
+
go run ./cmd/butterfly sync -input pds -pds https://morel.us-east.host.bsky.network -did did:plc:ragtjsm2j2vknwkz3zp4oxrd -store duckdb -db ~/tmp/test-tarfiles/duckdb.sql
28
+
go run ./cmd/butterfly sync -input carfile -car /Users/paulfrazee/tmp/carfiles/pfrazee.car -did did:plc:ragtjsm2j2vknwkz3zp4oxrd -store duckdb -db ~/tmp/test-tarfiles/duckdb.sql
29
+
go run ./cmd/butterfly discover -input pds -pds https://morel.us-east.host.bsky.network -limit 10 -store stdout
30
+
```
31
+
9
32
## TODOs
10
33
11
34
v1
12
35
13
-
- Create first working implementations of all Remote interfaces so that we build familiarity with their semantics and idiosyncracies
14
-
- Implement repo discovery interfaces on Remote
15
-
- Implement bidi identity resolution and caching
16
-
- Implement a work-scheduler which abstracts Remote, Identity, and Store to backfill & sync using Selectors
17
-
- Implement Store querying interfaces; develop indexing strategies
18
-
- Create v1 CLI and APIs
36
+
- Backfill
37
+
- Implement all Remote FetchRepo
38
+
- Implement all Remote ListRepos
39
+
- Implement bidi identity resolution and caching
40
+
- Core
41
+
- Implement a work-scheduler which abstracts Remote, Identity, and Store to backfill using Selectors
42
+
- Track "known" Lexicons and maintain an active definitions cache
43
+
- Automatically generate secondary indexes which are generally useful
44
+
- Investigate indexes generated by Lexicon definitions
45
+
- Implement Store querying interfaces
46
+
- Active sync
47
+
- Implement all Remote SyncRecords
48
+
- Update sync-state-tracking (repo state)
49
+
- Interface
50
+
- Create v1 CLI and APIs
19
51
20
52
future
21
53
+28
cmd/butterfly/store/thing_test.go
+28
cmd/butterfly/store/thing_test.go
···
1
+
package store
2
+
3
+
import (
4
+
"context"
5
+
"fmt"
6
+
"testing"
7
+
"time"
8
+
)
9
+
10
+
func TestPebbleThing(t *testing.T) {
11
+
start := time.Now()
12
+
ctx := context.Background()
13
+
s := NewPebbleStore("/Users/paulfrazee/tmp/test-tarfiles/pebble.db/")
14
+
s.Setup(ctx)
15
+
records, _ := s.ListAllRecords(ctx, "did:plc:ragtjsm2j2vknwkz3zp4oxrd", 0)
16
+
collections := 0
17
+
recordCount := 0
18
+
for _, sub := range records {
19
+
collections++
20
+
for range sub {
21
+
recordCount++
22
+
}
23
+
}
24
+
fmt.Printf("%d collections, %d records\n", collections, recordCount)
25
+
s.Close()
26
+
elapsed := time.Since(start)
27
+
fmt.Printf("Completed in %s", elapsed)
28
+
}