this repo has no description
add some dev notes
+99
+99
HACKING.md
+99
HACKING.md
···
1
+
2
+
## git repo contents
3
+
4
+
Commands (run with, eg, `go run ./cmd/bigsky`):
5
+
6
+
- `cmd/bigsky`: BGS+indexer daemon
7
+
- `cmd/gosky`: client CLI for talking to a PDS
8
+
- `cmd/lexgen`: codegen tool for lexicons (Lexicon JSON to golang package)
9
+
- `cmd/pds`: PDS daemon
10
+
- `cmd/stress`: connects to local/default PDS and creates a ton of random posts
11
+
- `gen`: dev tool to run CBOR type codegen
12
+
13
+
Packages:
14
+
15
+
- `api`: mostly output of lexgen (codegen) for lexicons: structs, CBOR marshaling. some higher-level code, and a PLC client (may rename)
16
+
- `bgs`: server implementation for crawling, etc
17
+
- `carstore`: library for storing repo data in CAR files on disk, plus a metadata SQL db
18
+
- `events`: types, codegen CBOR helpers, and persistence for event feeds
19
+
- `indexer`: aggregator, handling like counts etc in SQL database
20
+
- `key`: did:key crypto helpers (serialization, signing) (may rename or spin out)
21
+
- `lex`: implements codegen for Lexicons (!)
22
+
- `mst`: merkle search tree implementation
23
+
- `notifs`: helpers for notification objects (hydration, etc)
24
+
- `plc`: implementation of a *fake* PLC server (not persisted), and a PLC client
25
+
- `repo`: implements atproto repo on top of a blockstore. CBOR types
26
+
- `repomgr`: wraps many repos with a single carstore backend. handles events, locking
27
+
- `pds`: PDS server implementation
28
+
- `testing`: integration tests; testing helpers
29
+
- `models`: database types/models/schemas; shared in several places
30
+
- `util`: a few common definitions (may rename)
31
+
- `xrpc`: XRPC client (not server) helpers
32
+
33
+
Other:
34
+
35
+
- `testscripts`: shell scripts that run gosky (CLI client) against local PDS
36
+
37
+
38
+
## jargon
39
+
40
+
- BGS: Big Graph Service (or Server), which centrals crawls/consumes content from "all" PDSs and re-broadcasts as a firehose
41
+
- PDS: Personal Data Server (or Service), which stores user atproto repositories and acts as a user agent in the network
42
+
- CLI: Command Line Tool
43
+
- CBOR: a binary serialization format, smilar to JSON
44
+
- PLC: "placeholder" DID provider
45
+
- DID: Decentralized IDentifier, a flexible W3C specification for persistent identifiers in URI form (eg, "did:plc:abcd1234")
46
+
- XRPC: atproto convention for HTTP GET and POST endpoints specified by namespaced Lexicon schemas
47
+
- CAR: simple file format for storing binary content-addressed blocks/blobs, sort of like .tar files
48
+
- CID: content identifier for binary blobs, basically a flexible encoding of hash values
49
+
- MST: Merkle Search Tree, a key/value map data structure using content addressed nodes
50
+
51
+
52
+
## lexicon and CBOR marshaling code generation
53
+
54
+
`gen/main.go` has a list of types internal to packages in this repo which need CBOR helper codegen. If you edit those types, or update the listed types/packages, re-run codegen like:
55
+
56
+
# make sure everything can build cleanly first
57
+
make build
58
+
59
+
# then generate
60
+
go run ./gen
61
+
62
+
To run codegen for new or updated Lexicons, using lexgen, first place (or git
63
+
checout) the JSON lexicon files `$SOMEWHERE`. Also, install the `goimports`
64
+
tool. Then, in *this* repository (indigo), run commands like:
65
+
66
+
go run ./cmd/lexgen/ --package bsky --prefix app.bsky --outdir api/bsky $SOMEWHERE/lexicons/app/bsky/
67
+
go run ./cmd/lexgen/ --package atproto --prefix com.atproto --outdir api/bsky $SOMEWHERE/lexicons/com/atproto/
68
+
69
+
You may want to delete all the codegen files before re-generating, to detect deleted files.
70
+
71
+
It can require some manual munging between the lexgen step and a later `go run ./gen` to make sure things compile at least temporarily; otherwise the `gen` will not run.
72
+
73
+
To generate server stubs and handlers, push them in a temporary directory
74
+
first, then merge changes in to the actual PDS code:
75
+
76
+
go run ./cmd/lexgen/ --package pds --gen-server --types-import com.atproto:github.com/bluesky-social/indigo/api/atproto --types-import app.bsky:github.com/bluesky-social/indigo/api/bsky --outdir pds $SOMEWHERE/lexicons
77
+
78
+
mkdir tmppds
79
+
go run ./cmd/lexgen/ --package pds --gen-server --types-import com.atproto:github.com/bluesky-social/indigo/api/atproto --types-import app.bsky:github.com/bluesky-social/indigo/api/bsky --outdir tmppds --gen-handlers $SOMEWHERE/lexicons
80
+
81
+
82
+
## tips and tricks
83
+
84
+
When debugging websocat streams, the `websocat` tool (rust) can be helpful. CBOR binary is sort of mangled in to text by default. Eg:
85
+
86
+
# consume repo events from PDS
87
+
websocat ws://localhost:4989/events
88
+
89
+
# consume repo events from BGS
90
+
websocat ws://localhost:2470/events
91
+
92
+
Send the BGS a ding-dong:
93
+
94
+
# tell BGS to consume from PDS
95
+
http --json post localhost:2470/add-target host="localhost:4989"
96
+
97
+
Set the log level to be more verbose, using an env variable:
98
+
99
+
GOLOG_LOG_LEVEL=info go run ./cmd/pds