this repo has no description
update READMEs and dev docs
+69
-63
+21
-46
HACKING.md
+21
-46
HACKING.md
···
1
1
2
2
## git repo contents
3
3
4
-
Commands (run with, eg, `go run ./cmd/bigsky`):
4
+
Run with, eg, `go run ./cmd/bigsky`):
5
5
6
6
- `cmd/bigsky`: BGS+indexer daemon
7
7
- `cmd/palomar`: search indexer and query servcie (OpenSearch)
8
8
- `cmd/gosky`: client CLI for talking to a PDS
9
9
- `cmd/lexgen`: codegen tool for lexicons (Lexicon JSON to Go package)
10
-
- `cmd/laputa`: PDS daemon
10
+
- `cmd/laputa`: partial PDS daemon (not usable or under development)
11
11
- `cmd/stress`: connects to local/default PDS and creates a ton of random posts
12
12
- `cmd/beemo`: slack bot for moderation reporting (Bluesky Moderation Observer)
13
13
- `cmd/fakermaker`: helper to generate fake accounts and content for testing
14
+
- `cmd/supercollider`: event stream load generation tool
15
+
- `cmd/sonar`: event stream monitoring tool
14
16
- `gen`: dev tool to run CBOR type codegen
15
17
16
18
Packages:
17
19
18
20
- `api`: mostly output of lexgen (codegen) for lexicons: structs, CBOR marshaling. some higher-level code, and a PLC client (may rename)
19
-
- `api/atprot`: generated types for `com.atproto` lexicon
21
+
- `api/atproto`: generated types for `com.atproto` lexicon
20
22
- `api/bsky`: generated types for `app.bsky` lexicon
23
+
- `atproto/crypto`: crytographic helpers (signing, key generation and serialization)
24
+
- `atproto/syntax`: string types and parsers for identifiers, datetimes, etc
25
+
- `atproto/identity`: DID and handle resolution
21
26
- `bgs`: server implementation for crawling, etc
22
27
- `carstore`: library for storing repo data in CAR files on disk, plus a metadata SQL db
23
28
- `events`: types, codegen CBOR helpers, and persistence for event feeds
···
35
40
- `util`: a few common definitions (may rename)
36
41
- `xrpc`: XRPC client (not server) helpers
37
42
38
-
Other:
39
-
40
-
- `testscripts`: shell scripts that run gosky (CLI client) against local PDS
41
43
42
-
43
-
## jargon
44
+
## Jargon
44
45
45
46
- BGS: Big Graph Service (or Server), which centrals crawls/consumes content from "all" PDSs and re-broadcasts as a firehose
46
47
- PDS: Personal Data Server (or Service), which stores user atproto repositories and acts as a user agent in the network
47
48
- CLI: Command Line Tool
48
49
- CBOR: a binary serialization format, smilar to JSON
49
-
- PLC: "placeholder" DID provider
50
-
- DID: Decentralized IDentifier, a flexible W3C specification for persistent identifiers in URI form (eg, "did:plc:abcd1234")
50
+
- PLC: "placeholder" DID provider, see <https://web.plc.directory>
51
+
- DID: Decentralized IDentifier, a flexible W3C specification for persistent identifiers in URI form (eg, `did:plc:abcd1234`)
51
52
- XRPC: atproto convention for HTTP GET and POST endpoints specified by namespaced Lexicon schemas
52
53
- CAR: simple file format for storing binary content-addressed blocks/blobs, sort of like .tar files
53
54
- CID: content identifier for binary blobs, basically a flexible encoding of hash values
54
55
- MST: Merkle Search Tree, a key/value map data structure using content addressed nodes
55
56
56
57
57
-
## lexicon and CBOR marshaling code generation
58
+
## Lexicon and CBOR code generation
58
59
59
60
`gen/main.go` has a list of types internal to packages in this repo which need CBOR helper codegen. If you edit those types, or update the listed types/packages, re-run codegen like:
60
61
···
64
65
# then generate
65
66
go run ./gen
66
67
67
-
To run codegen for new or updated Lexicons, using lexgen, first place (or git
68
-
checkout) the JSON lexicon files at `../atproto/`.
69
-
Then, in *this* repository (indigo), run commands like:
68
+
To run codegen for new or updated Lexicons, using lexgen, first place (or git checkout) the JSON lexicon files at `../atproto/`. Then, in *this* repository (indigo), run commands like:
70
69
71
70
go run ./cmd/lexgen/ --package bsky --prefix app.bsky --outdir api/bsky ../atproto/lexicons/app/bsky/
72
71
go run ./cmd/lexgen/ --package atproto --prefix com.atproto --outdir api/atproto ../atproto/lexicons/com/atproto/
73
72
74
73
You may want to delete all the codegen files before re-generating, to detect deleted files.
75
74
76
-
It can require some manual munging between the lexgen step and a later `go run ./gen` to make sure things compile at least temporarily; otherwise the `gen` will not run.
77
-
In some cases, you might also need to add new types to ./gen/main.go.
75
+
It can require some manual munging between the lexgen step and a later `go run ./gen` to make sure things compile at least temporarily; otherwise the `gen` will not run. In some cases, you might also need to add new types to `./gen/main.go`.
78
76
79
-
To generate server stubs and handlers, push them in a temporary directory
80
-
first, then merge changes in to the actual PDS code:
77
+
To generate server stubs and handlers, push them in a temporary directory first, then merge changes in to the actual PDS code:
81
78
82
79
mkdir tmppds
83
80
go run ./cmd/lexgen/ --package pds --gen-server --types-import com.atproto:github.com/bluesky-social/indigo/api/atproto --types-import app.bsky:github.com/bluesky-social/indigo/api/bsky --outdir tmppds --gen-handlers ../atproto/lexicons
84
81
85
82
86
-
## tips and tricks
83
+
## Tips and Tricks
87
84
88
85
When debugging websocket streams, the `websocat` tool (rust) can be helpful. CBOR binary is sort of mangled in to text by default. Eg:
89
86
···
103
100
GOLOG_LOG_LEVEL=info go run ./cmd/pds
104
101
105
102
106
-
## gosky basic usage
103
+
## `gosky` basic usage
107
104
108
105
Running against local typescript PDS in `dev-env` mode:
109
106
110
107
# as "alice" user
111
108
go run ./cmd/gosky/ --pds http://localhost:2583 createSession alice.test hunter2 > bsky.auth
112
109
113
-
The `bsky.auth` file is the default place that `gosky` and other client
114
-
commands will look for auth info.
110
+
The `bsky.auth` file is the default place that `gosky` and other client commands will look for auth info.
115
111
116
112
117
-
## slack report bot basic usage
113
+
## Integrated Development
118
114
119
-
You need an admin token, slack webhook URL, and auth file (see gosky above).
120
-
The auth file isn't actually used, only the admin token.
115
+
Sometimes it is helpful to run a PLC, PDS, BGS, labelmaker, and other components, all locally on your laptop, across languages. This section describes one setup for this.
121
116
122
-
# configure a slack webhook
123
-
export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T028K87/B04NBDB/oWbsHasdf23r2d
124
-
125
-
# example pulling admin token out of `pass` password manager
126
-
export ATP_AUTH_ADMIN_PASSWORD=`pass bsky/pds-admin-staging | head -n1`
127
-
128
-
# example just setting admin token directly
129
-
export ATP_AUTH_ADMIN_PASSWORD="someinsecurething123"
130
-
131
-
# run the bot
132
-
GOLOG_LOG_LEVEL=debug go run ./cmd/beemo/ --pds https://pds.staging.example.com --auth bsky.auth notify-reports
133
-
134
-
## integrated development
135
-
136
-
Sometimes it is helpful to run a PLC, PDS, BGS, labelmaker, and other
137
-
components, all locally on your laptop, across languages. This section
138
-
describes one setup for this.
139
-
140
-
First, you need PostgreSQL running locally. This could be via docker, or the
141
-
following commands assume some kind of debian/ubuntu setup with a postgres
142
-
server package installed and running.
117
+
First, you need PostgreSQL running locally. This could be via docker, or the following commands assume some kind of debian/ubuntu setup with a postgres server package installed and running.
143
118
144
119
Create a user and databases for PLC+PDS:
145
120
+31
-17
README.md
+31
-17
README.md
···
1
1
2
2
3
3
4
-
indigo: golang code for Bluesky's atproto services
5
-
==================================================
4
+
indigo: atproto libraries and services in golang
5
+
================================================
6
6
7
7
Some Bluesky software is developed in Typescript, and lives in the [bluesky-social/atproto](https://github.com/bluesky-social/atproto) repository. Some is developed in Go, and lives here.
8
8
9
-
<p align="center"><img src="https://static.bnewbold.net/tmp/under_construction_bar.gif" /></p>
9
+
Services implemented in this repository:
10
10
11
-
Everything in this repository is an work in progress. Features and "Lexicons" may be removed or updated, software interfaces broken, etc.
11
+
* **`bigsky`** ([README](./cmd/bigsky/README.md)): "Big Graph Service" (BGS) reference implementation, running at `bsky.network`
12
+
* **`palomar`** ([README](./cmd/palomar/README.md)): fulltext search service for <https://bsky.app>
12
13
13
-
We are developing in the open, but not ready to accept or review significant contributions. Keep checking back!
14
+
15
+
## Development Quickstart
16
+
17
+
All the packages in this repository are under active development. Features and software interfaces have not stabilized and may break or be removed.
14
18
15
19
<p align="center"><img src="https://static.bnewbold.net/tmp/under_construction_bar.gif" /></p>
16
20
21
+
First, you will need the Go toolchain installed. We develop using the latest stable version of the language.
22
+
23
+
The Makefile provides wrapper commands for basic development:
24
+
25
+
make build
26
+
make test
27
+
make fmt
28
+
make lint
29
+
30
+
Individual commands can be run like:
31
+
32
+
go run ./cmd/bigsky
33
+
34
+
The [HACKING](./HACKING.md) file has a list of commands and packages in this repository and some other development tips.
35
+
17
36
18
37
## What is atproto?
19
38
···
21
40
22
41
The Authenticated Transfer Protocol ("ATP" or "atproto") is a decentralized social media protocol, developed by [Bluesky PBC](https://blueskyweb.xyz). Learn more at:
23
42
24
-
- [Protocol Documentation](https://atproto.com/docs)
25
-
- [Overview Guide](https://atproto.com/guides/overview) 👈 Good place to start
43
+
- [Overview and Guides](https://atproto.com/guides/overview) 👈 Best starting point
44
+
- [Github Discussions](https://github.com/bluesky-social/atproto/discussions) 👈 Great place to ask questions
45
+
- [Protocol Specifications](https://atproto.com/specs/atp)
26
46
- [Blogpost on self-authenticating data structures](https://blueskyweb.xyz/blog/3-6-2022-a-self-authenticating-social-protocol)
27
47
28
-
29
-
## Development
48
+
The Bluesky Social application encompasses a set of schemas and APIs built in the overall AT Protocol framework. The namespace for these "Lexicons" is `app.bsky.*`.
30
49
31
-
First, you will need the Go toolchain installed. We develop using the latest stable version of the language.
32
50
33
-
The Makefile provides wrapper commands for basic development:
51
+
## Contributions
34
52
35
-
make build
36
-
make test
37
-
make fmt
38
-
make lint
53
+
We are working in the open, but not ready to actively collaborate on larger contributions to this codebase.
39
54
40
-
Individual commands can be run like:
55
+
Please at least open an issue ahead of time, *before* starting any non-trivial work that you hope to get reviewed or merged to this repo.
41
56
42
-
go run ./cmd/bigsky
43
57
44
58
## Are you a developer interested in building on atproto?
45
59
+17
cmd/beemo/README.md
+17
cmd/beemo/README.md
···
1
+
2
+
## beemo: Slack notification bot for moderation reports
3
+
4
+
You need an admin token, slack webhook URL, and auth file (see gosky docs).
5
+
The auth file isn't actually used, only the admin token.
6
+
7
+
# configure a slack webhook
8
+
export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T028K87/B04NBDB/oWbsHasdf23r2d
9
+
10
+
# example pulling admin token out of `pass` password manager
11
+
export ATP_AUTH_ADMIN_PASSWORD=`pass bsky/pds-admin-staging | head -n1`
12
+
13
+
# example just setting admin token directly
14
+
export ATP_AUTH_ADMIN_PASSWORD="someinsecurething123"
15
+
16
+
# run the bot
17
+
GOLOG_LOG_LEVEL=debug go run ./cmd/beemo/ --pds https://pds.staging.example.com --auth bsky.auth notify-reports