tranquil.farm / tranquil-pds
Our Personal Data Server from scratch! tranquil.farm
pds rust database fun oauth atproto
238
fork

Configure Feed

Select the types of activity you want to include in your feed.

adding k8s docs and a more generalized doc #12

open opened by
dsx.sh targeting main from dsx.sh/tranquil-pds: k8s-doc-updates

I recently went through some pain in getting Tranquil all set up on my k8s cluster. Luckily some of my 'biggest' pain was in the mystery of the email server, which has been resolved in the meantime.

Regardless, I added a bunch more detail on a somewhat generic k8s deployment, with some context for secrets, secret generation, etc.

I also set up a "general configuration" doc that took some of the details and context from all the install-x docs and compiled it together into a more generalized information source with all the env vars I could find centralized into one place.

Sorry if this kind of work is already being done elsewhere, I wanted to get this somewhere so I can point friends to so they do their own PDS.

Labels

None yet.

assignee

None yet.

Participants 1
AT URI
at://did:plc:6so2rykrnjmzebbr2zbwbqbx/sh.tangled.repo.pull/3mlcn4ezr6p22
+363 -14
Diff #0
+154
docs/general-configuration.md
··· 1 + # Tranquil PDS Configuration Reference 2 + 3 + Every option in `example.toml` is overridable by an environment variable. This document focuses on what each setting *does* and why some of them are critical to get right before you start. The install guides (`install-containers.md`, `install-kubernetes.md`, `install-nix.md`) cover the mechanics of standing up the service; this covers what to put in it. 4 + 5 + ## Configuration methods 6 + 7 + The server accepts config in two forms, applied in this order (later wins): 8 + 9 + 1. A TOML file, defaulting to `/etc/tranquil-pds/config.toml`. Override the path with `TRANQUIL_PDS_CONFIG`. 10 + 2. Environment variables. Every TOML key has a corresponding env var documented in `example.toml`. 11 + 12 + In practice: TOML for static, non-sensitive settings; env vars (injected from secrets) for anything that would be unfortunate in a file you may want to commit. 13 + 14 + --- 15 + 16 + ## Required settings 17 + 18 + These have no defaults and the Tranqil will not start without them. 19 + 20 + ### `PDS_HOSTNAME` 21 + 22 + The public hostname of your PDS. Example: `pds.example.com`. 23 + 24 + This is used in DID documents, OAuth metadata, and the `/.well-known/atproto-did` endpoint. Getting it wrong after you have accounts is painful: DIDs are registered against this hostname in the PLC directory, and changing it requires a PLC rotation operation. Set it correctly before creating any accounts. 25 + 26 + If your PDS hostname is also the apex domain (e.g. `example.com` rather than `pds.example.com`), users can have their handle *be* the apex domain (e.g. `@example.com`) since the PDS serves `/.well-known/atproto-did` directly without needing a DNS TXT record. 27 + 28 + ### `DATABASE_URL` 29 + 30 + PostgreSQL connection string. Example: `postgres://tranquil_pds:password@host:5432/pds` 31 + 32 + Tranquil PDS uses Postgres for all persistent state: accounts, repos, sessions, the comms queue, and (when using the default `postgres` repo backend) the full block store for every account's atproto repository. 33 + 34 + --- 35 + 36 + ## Required secret/sensitive env vars 37 + 38 + These three are the most sensitive values in the entire deployment. They should never appear in a committed file, in any circumstance. 39 + 40 + ### `JWT_SECRET` 41 + 42 + Used to sign and verify all session JWTs issued by the PDS. 43 + 44 + Generate: `openssl rand -base64 48` 45 + 46 + ### `DPOP_SECRET` 47 + 48 + Used to validate DPoP (Demonstrating Proof of Possession) proofs in OAuth flows. DPoP is the binding mechanism that ties an OAuth access token to the client's key pair, preventing token theft from replaying the token with a different client. 49 + 50 + Generate: `openssl rand -base64 48` 51 + 52 + ### `MASTER_KEY` 53 + 54 + Used as input to HKDF key derivation and for encrypting per-account signing keys at rest. Every account's atproto repo signing key is derived from or wrapped with this value. 55 + 56 + Generate: `openssl rand -base64 48` 57 + 58 + --- 59 + 60 + ## Handle domains 61 + 62 + ### `PDS_USER_HANDLE_DOMAINS` 63 + 64 + Comma-separated list of domains under which handles are issued. Defaults to `PDS_HOSTNAME` if unset. 65 + 66 + If your PDS hostname is `pds.example.com` but you want handles like `alice.example.com`, set this to `example.com`. You need a wildcard TLS cert and DNS wildcard record for the domain(s) you list here. 67 + 68 + ### `PDS_BANNED_WORDS` 69 + 70 + Comma-separated list of substring patterns blocked at registration time. Useful to prevent handles that would collide with your infrastructure (e.g. `grafana`, `vault`, `relay`). The server has a built-in exact-match reserved list (`admin`, `api`, `git`, `mail`, `ssh`, etc.); this adds substring matches for anything missing from that list. 71 + 72 + ### `INVITE_CODE_REQUIRED` 73 + 74 + Default `true`. Keep this enabled on any public-facing PDS unless you specifically want open registration. Invite codes are generated via the admin API/UI. 75 + 76 + --- 77 + 78 + ## Blob storage 79 + 80 + ### `BLOB_STORAGE_BACKEND` 81 + 82 + `filesystem` (default) or `s3`. Pretty straight forward, and dependent on your personal infrastructure requirements and availability. 83 + 84 + ### `BLOB_STORAGE_PATH` 85 + 86 + Path on disk for the filesystem backend. Default `/var/lib/tranquil-pds/blobs`. 87 + 88 + For S3: set `S3_BUCKET` and optionally `S3_ENDPOINT` for S3-compatible stores (Tigris, Cloudflare R2, MinIO, etc.). 89 + 90 + --- 91 + 92 + ## Federation 93 + 94 + ### `CRAWLERS` 95 + 96 + Comma-separated list of relay URLs to notify when new events are committed to an account's repo. Often `https://bsky.network` for production, this is what allows your PDS to be federated to on the wider network. 97 + 98 + --- 99 + 100 + ## Email 101 + 102 + Email is sorta optional but required for account verification, password resets, and 2FA backup codes. If `MAIL_FROM_ADDRESS` is not set, email sending is disabled entirely. 103 + 104 + Tranquil supports two delivery modes: 105 + 106 + ### Smarthost (SMTP relay) 107 + 108 + Set `MAIL_SMARTHOST_HOST`, `MAIL_SMARTHOST_PORT`, `MAIL_SMARTHOST_USERNAME`, and `MAIL_SMARTHOST_PASSWORD`. Use this if you're routing through a sending service (Postmark, SES, etc.) or a local MTA. 109 + 110 + TLS mode defaults to `starttls`. Setting `tls = "none"` alongside a password is rejected at startup. 111 + 112 + ### DKIM signing (`email.dkim`) 113 + 114 + Set `MAIL_DKIM_SELECTOR`, `MAIL_DKIM_DOMAIN`, and `MAIL_DKIM_KEY_PATH` to sign outgoing mail. The key file should be PEM-format RSA or Ed25519. The corresponding DNS TXT record at `<selector>._domainkey.<domain>` must be published before mail is sent. 115 + 116 + --- 117 + 118 + ## Notifications 119 + 120 + Beyond email, Tranquil supports Discord, Telegram, and Signal for user notifications. Each is opt-in: 121 + 122 + - **Discord**: Set `DISCORD_BOT_TOKEN`. 123 + - **Telegram**: Set `TELEGRAM_BOT_TOKEN` and `TELEGRAM_WEBHOOK_SECRET`. 124 + - **Signal**: Set `SIGNAL_ENABLED=true` after linking a device via the admin API. State is persisted in `signal_*` tables in Postgres. 125 + 126 + --- 127 + 128 + ## SSO 129 + 130 + OAuth SSO is supported for GitHub, Discord, Google, GitLab, generic OIDC, and Apple. Each provider requires `enabled = true`, a `client_id`, and a `client_secret`. GitLab and OIDC also need an `issuer` URL. 131 + 132 + `client_secret` values should be injected as env vars (`SSO_GITHUB_CLIENT_SECRET`, etc.) and not placed in config files. 133 + 134 + --- 135 + 136 + ## Cache backend 137 + 138 + ### `CACHE_BACKEND` 139 + 140 + `ripple` (default) or `valkey`. Ripple is the a built-in in-process gossip cache. Valkey is an external Redis fork with it's own complications. 141 + 142 + --- 143 + 144 + ## Optional keys worth knowing 145 + 146 + | Env var | Default | Notes | 147 + |---|---|---| 148 + | `PLC_ROTATION_KEY` | — | Operator-level PLC rotation key (DID key). If unset, per-user keys are used. | 149 + | `ENABLE_PDS_HOSTED_DID_WEB` | `false` | Opt-in did:web hosting. Long-term commitment — only enable if you intend to serve DID documents indefinitely. | 150 + | `MAX_BLOB_SIZE` | 10 GiB | Per-blob upload limit in bytes. | 151 + | `DISABLE_RATE_LIMITING` | `false` | Only for testing. Avoid in production. | 152 + | `DATABASE_MAX_CONNECTIONS` | 100 | Connection pool ceiling. Tune against your Postgres `max_connections`. | 153 + | `FIREHOSE_BACKFILL_HOURS` | 72 | How many hours of history relays can replay from cursor. | 154 + | `REPO_BACKEND` | `postgres` | `tranquil-store` is the experimental embedded backend. Careful if attempting to run in production. |
+209 -14
docs/install-kubernetes.md
··· 1 - # Tranquil PDS on kubernetes 1 + # Tranquil PDS on Kubernetes 2 2 3 - If you're reaching for kubernetes for this app, you're experienced enough to know how to spin up: 3 + If you're reaching for Kubernetes for this app, you're experienced enough to know how to spin up: 4 4 5 5 - cloudnativepg (or your preferred postgres operator) 6 6 - a PersistentVolume for blob storage 7 7 - the app itself (it's just a container with some env vars) 8 8 9 - You'll need a wildcard TLS certificate for `*.your-pds-hostname.example.com`. User handles are served as subdomains. 9 + See [configuration.md](configuration.md) for what each env var does and why the secret ones matter. This guide covers the Kubernetes-specific wiring. 10 10 11 - The container image expects: 12 - - A TOML config file mounted at `/etc/tranquil-pds/config.toml` (or passed via `--config`) 13 - - `DATABASE_URL` - postgres connection string 14 - - `BLOB_STORAGE_PATH` - path to blob storage (mount a PV here) 15 - - `PDS_HOSTNAME` - your PDS hostname (without protocol) 16 - - `JWT_SECRET`, `DPOP_SECRET`, `MASTER_KEY` - generate with `openssl rand -base64 48` 17 - - `CRAWLERS` - typically `https://bsky.network` 11 + --- 18 12 19 - and more, check the example.toml for all options. Environment variables can override any TOML value. 20 - You can also point to a config file via the `TRANQUIL_PDS_CONFIG` env var. 13 + ## TLS and DNS 21 14 22 - Health check: `GET /xrpc/_health` 15 + You need a wildcard TLS certificate covering `*.your-pds-hostname.example.com` — user handles resolve as subdomains, so every user's handle requires a matching cert SAN. 16 + 17 + An approach using Cert Manager would look something like this: 18 + 19 + ```yaml 20 + apiVersion: cert-manager.io/v1 21 + kind: Certificate 22 + metadata: 23 + name: pds-cert 24 + namespace: pds 25 + spec: 26 + secretName: pds-tls 27 + dnsNames: 28 + - pds.example.com 29 + - "*.pds.example.com" 30 + ``` 31 + 32 + If your PDS hostname is the apex domain (so handles are issued under it, not under a subdomain), include the apex in `dnsNames` alongside the wildcard. 33 + 34 + --- 35 + 36 + ## Secrets 37 + 38 + The three secrets primary key secrets (`JWT_SECRET`, `DPOP_SECRET`, `MASTER_KEY`) must never appear in a manifest or config file. Inject them as a Kubernetes Secret, sourced from wherever you manage secrets. 39 + 40 + To create the Secret directly and manage rotation manually: 41 + 42 + ```bash 43 + kubectl create secret generic pds-secrets -n pds \ 44 + --from-literal=DATABASE_URL='postgres://...' \ 45 + --from-literal=JWT_SECRET="$(openssl rand -base64 48)" \ 46 + --from-literal=DPOP_SECRET="$(openssl rand -base64 48)" \ 47 + --from-literal=MASTER_KEY="$(openssl rand -base64 48)" 48 + ``` 49 + 50 + Then reference it in your Deployment: 51 + 52 + ```yaml 53 + envFrom: 54 + - secretRef: 55 + name: pds-secrets 56 + ``` 57 + 58 + --- 59 + 60 + ## PostgreSQL 61 + 62 + CloudNativePG is an easy recommendation, an example for the purpose of Tranquil PDS: 63 + 64 + ```yaml 65 + apiVersion: postgresql.cnpg.io/v1 66 + kind: Cluster 67 + metadata: 68 + name: postgres 69 + namespace: pds 70 + spec: 71 + instances: 1 72 + bootstrap: 73 + initdb: 74 + database: pds 75 + owner: tranquil_pds 76 + secret: 77 + name: postgres-user-secret # k8s Secret with username/password 78 + storage: 79 + storageClass: your-storage-class 80 + size: 10Gi 81 + ``` 82 + 83 + The `postgres-user-secret` Secret needs `username` and `password` keys. The password you put here is what goes into `DATABASE_URL`. 84 + 85 + Any standard Postgres setup works in place of CNPG. Tranquil does not require anything special for a basic installation. 86 + 87 + --- 88 + 89 + ## Storage PVC (if using local-path, nfs, or similar) 90 + 91 + ```yaml 92 + apiVersion: v1 93 + kind: PersistentVolumeClaim 94 + metadata: 95 + name: pds-blobs 96 + namespace: pds 97 + spec: 98 + storageClassName: your-storage-class 99 + accessModes: 100 + - ReadWriteOnce 101 + resources: 102 + requests: 103 + storage: 50Gi 104 + ``` 105 + 106 + Full Deployment example: 107 + 108 + ```yaml 109 + apiVersion: apps/v1 110 + kind: Deployment 111 + metadata: 112 + name: pds 113 + namespace: pds 114 + spec: 115 + replicas: 1 116 + selector: 117 + matchLabels: 118 + app: pds 119 + strategy: 120 + type: Recreate 121 + template: 122 + metadata: 123 + labels: 124 + app: pds 125 + spec: 126 + containers: 127 + - name: pds 128 + image: atcr.io/tranquil.farm/tranquil-pds:latest 129 + ports: 130 + - name: http 131 + containerPort: 3000 132 + env: 133 + - name: SERVER_HOST 134 + value: "0.0.0.0" 135 + - name: SERVER_PORT 136 + value: "3000" 137 + - name: PDS_HOSTNAME 138 + value: "pds.example.com" 139 + - name: PDS_USER_HANDLE_DOMAINS 140 + value: "example.com" 141 + - name: BLOB_STORAGE_BACKEND 142 + value: filesystem 143 + - name: BLOB_STORAGE_PATH 144 + value: /var/lib/tranquil/blobs 145 + - name: CRAWLERS 146 + value: "https://bsky.network" 147 + - name: INVITE_CODE_REQUIRED 148 + value: "true" 149 + envFrom: 150 + - secretRef: 151 + name: pds-secrets 152 + volumeMounts: 153 + - mountPath: /var/lib/tranquil/blobs 154 + name: blobs 155 + readinessProbe: 156 + httpGet: 157 + path: /xrpc/_health 158 + port: http 159 + initialDelaySeconds: 15 160 + periodSeconds: 10 161 + failureThreshold: 6 162 + resources: 163 + requests: 164 + memory: 256Mi 165 + cpu: 100m 166 + limits: 167 + memory: 1Gi 168 + volumes: 169 + - name: blobs 170 + persistentVolumeClaim: 171 + claimName: pds-blobs 172 + ``` 173 + 174 + `SERVER_HOST: "0.0.0.0"` is required — the default `127.0.0.1` isn't reachable by either the Kubelet for health checks or your ingress controller. 175 + 176 + --- 177 + 178 + ## Ingress 179 + 180 + The ingress rule must match both the PDS hostname itself and the wildcard for user handles. 181 + 182 + ```yaml 183 + apiVersion: networking.k8s.io/v1 184 + kind: Ingress 185 + metadata: 186 + name: pds 187 + namespace: pds 188 + spec: 189 + tls: 190 + - hosts: 191 + - pds.example.com 192 + - "*.example.com" 193 + secretName: pds-tls 194 + rules: 195 + - host: pds.example.com 196 + http: 197 + paths: 198 + - path: / 199 + pathType: Prefix 200 + backend: 201 + service: 202 + name: pds 203 + port: 204 + number: 3000 205 + - host: "*.example.com" 206 + http: 207 + paths: 208 + - path: / 209 + pathType: Prefix 210 + backend: 211 + service: 212 + name: pds 213 + port: 214 + number: 3000 215 + ``` 216 + 217 + --- 23 218 24 219 ## Custom homepage 25 220 26 - Mount a ConfigMap with your `homepage.html` into the container's frontend directory and it becomes your landing page. Go nuts with it. Account dashboard is at `/app/` so you won't break anything. 221 + Mount a ConfigMap with your `homepage.html` into the container's frontend directory and it becomes your landing page. The account dashboard lives at `/app/` so you won't displace it. 27 222 28 223 ```yaml 29 224 apiVersion: v1

History

4 rounds 0 comments
sign up or login to add to the discussion
dsx.sh submitted #3
diff interdiff
6 commits
expand
863d8480
adding k8s docs and a more generalized doc
d9c99c04
oops didnt actually mean to take that out
44c7255f
removing gen config altogether, agree with nel
697cf8c2
expanding example.toml a tiny bit and reworking k8s a little
541abaf4
rewording, and some sentence simplification
30e09e90
more rewording
merge conflicts detected
expand
expand 0 comments
dsx.sh submitted #2
diff interdiff
5 commits
expand
863d8480
adding k8s docs and a more generalized doc
d9c99c04
oops didnt actually mean to take that out
44c7255f
removing gen config altogether, agree with nel
697cf8c2
expanding example.toml a tiny bit and reworking k8s a little
541abaf4
rewording, and some sentence simplification
expand 0 comments
dsx.sh submitted #1
diff interdiff
2 commits
expand
863d8480
adding k8s docs and a more generalized doc
d9c99c04
oops didnt actually mean to take that out
expand 0 comments
dsx.sh submitted #0
1 commit
expand
863d8480
adding k8s docs and a more generalized doc
expand 0 comments