lightweight
com.atproto.sync.listReposByCollection
readme tweaks
+15
-14
+15
-14
readme.md
+15
-14
readme.md
···
36
36
37
37
### API
38
38
39
-
#### `com.atproto.sync.listReposByCollection`
40
-
41
-
[Query docs](https://docs.bsky.app/docs/api/com-atproto-sync-list-repos-by-collection)
39
+
#### `com.atproto.sync.listReposByCollection`: [Query docs](https://docs.bsky.app/docs/api/com-atproto-sync-list-repos-by-collection)
42
40
43
41
Lightrail implements some [proposed changes](https://github.com/bluesky-social/atproto/pull/4733) to this query:
44
42
···
50
48
- `limit` can be up to 10,000 (lexicon specifies 2,000 max). This matches `collectiondir`'s limit.
51
49
52
50
53
-
#### `com.atproto.sync.listRepos`
51
+
#### `com.atproto.sync.listRepos`: [Query docs](https://docs.bsky.app/docs/api/com-atproto-sync-list-repos)
54
52
55
-
[Query docs](https://docs.bsky.app/docs/api/com-atproto-sync-list-repos)
56
53
57
-
58
-
#### `com.atproto.sync.getRepoStatus`
59
-
60
-
[Query docs](https://docs.bsky.app/docs/api/com-atproto-sync-get-repo-status)
54
+
#### `com.atproto.sync.getRepoStatus`: [Query docs](https://docs.bsky.app/docs/api/com-atproto-sync-get-repo-status)
61
55
62
56
63
57
## Lightrail server quick start
···
78
72
79
73
### Key configs
80
74
75
+
- **`--db-path`**, default `./lightrail.db`: where to write lightrail's [fjall](https://fjall-rs.github.io/) db
76
+
- **`--listen`**, default `0.0.0.0:2511`: host and port to bind
77
+
78
+
81
79
```bash
82
-
# you can list all config options with:
80
+
# you can list *all* available options with:
83
81
cargo run -- --help
84
82
```
85
83
86
-
- **`--db-path`**, default `./lightrail.db`: where to write lightrail's [fjall](https://fjall-rs.github.io/) db
87
-
- **`--listen`**, default `0.0.0.0:2511`: host and port to bind
84
+
Environment vars can be used for all configs, prefixed with `LIGHTRAIL_`, like `LIGHTRAIL_DB_PATH=/path/to/wherever.db`.
88
85
89
86
90
87
#### Atmosphere configs
···
115
112
- Lightrail's ordering of DIDs in the `listReposByCollection` response is different from `collectiondir`
116
113
117
114
- `collectiondir` always inserts new DIDs at the end of the paginated response
118
-
- Lightrail makes no guarantee except that the response will not contain duplicates
115
+
- Lightrail makes no ordering guarantee except that the paged response will not contain duplicates
116
+
117
+
You should start your firehose listener *before* crawling `listReposByCollection`, because DIDs newly indexed by lightrail after the first is requested are not guaranteed to be present in the total paged set.
119
118
120
119
121
120
- If you see a log line like
···
145
144
When we call `sync.getRecord`, we provide a made-up collection and rkey, which works for our purposes because the response will contain a _proof of absense_ if the key doesn't exist in the repo: a CAR slice (with rev + data from the commit object!) containing adjacent keys (that we'll use!). Unfortunately, not every PDS implements proof of absense responses, notably **bridgy** currently returns an error for non-existent keys.
146
145
147
146
148
-
#### `sync.getRepo` resync fallback
147
+
#### Resync fallback with `sync.getRepo`
149
148
150
149
If the `describeRepo` approach fails for any reason, lightstream attempt to resync from a full repo export.
150
+
151
+
Eg., Bridgy's PDS often fails the `sync.getRecord` probe (see above) but its repos are still indexed by lightrail via this full-repo fallback.
151
152
152
153
153
154
### Sync1.1