lightweight
com.atproto.sync.listReposByCollection
update readme with new configs
+4
-1
+1
hacking.md
+1
hacking.md
···
103
103
- [x] account status convergeance: if we receive commits from apparently-inactive accounts, should we check upstream status to make sure we're not stale?
104
104
- [ ] resync short-circuit: tiny repos may actually return their entire CAR for getRecord
105
105
- [ ] commit CAR handling: generate a list of keys with gaps noted, to reliably detect missing adjacent keys
106
+
- [ ] repo-stream: drop record block contents with processor fn
106
107
107
108
108
109
very much still todo but i'm getting tired
+3
-1
readme.md
+3
-1
readme.md
···
90
90
- **`--plc-url`**, default: `https://plc.directory`: where to resolve `did:plc` identities. To use microcosm's mirror: `--plc-url https://plc.wtf`.
91
91
- **`--slingshot-url`**, default: `https://slingshot.microcosm.blue`: enables slingshot for identity reoslution (PLC directory acts as fallback.
92
92
- **`--deep-crawl`**, default: `[unset]`. enumerate hosts from upstream with `com.atproto.sync.listHosts` and then crawl those hosts each directly with `com.atproto.sync.listRepos`.
93
+
- **`--heavy`**, default: `[unset]`. always resync with `com.atproto.sync.getRepo` instead of trying the lightweight `com.atproto.repo.describeRepo` query first. this pull a lot more data!!
93
94
94
95
95
96
#### Operational configs
···
100
101
101
102
more knobs you can twist:
102
103
103
-
- **`--ident-cache-size`**, default: `2_000_000`: identity resolution provides repo signing keys and PDS hostnames. a larger cache reduces outbound resolution requests at the cost of more memory used.
104
+
- **`--ident-cache-size`**, default: `2_000_000`: identity resolution provides repo signing keys and PDS hostnames. a larger cache reduces outbound resolution requests at the cost of more memory used. turn down on memory-constrained systems
105
+
- **`--fjall-cache-mb`**, default `256`: the database cache size. turn down on memory-constrained systems.
104
106
- **`--max-firehose-workers`**, default: `6`: max firehose event processing concurrency.
105
107
- **`--cursor-save-interval-secs`**, default `1`
106
108
- **`--describe-repo-fetch-timeout-secs`**, default `30`