tranquil.farm / tranquil-pds
Our Personal Data Server from scratch! tranquil.farm
pds rust database fun oauth atproto
238
fork

Configure Feed

Select the types of activity you want to include in your feed.

adding k8s docs and a more generalized doc #12

open opened by
dsx.sh targeting main from dsx.sh/tranquil-pds: k8s-doc-updates

I recently went through some pain in getting Tranquil all set up on my k8s cluster. Luckily some of my 'biggest' pain was in the mystery of the email server, which has been resolved in the meantime.

Regardless, I added a bunch more detail on a somewhat generic k8s deployment, with some context for secrets, secret generation, etc.

I also set up a "general configuration" doc that took some of the details and context from all the install-x docs and compiled it together into a more generalized information source with all the env vars I could find centralized into one place.

Sorry if this kind of work is already being done elsewhere, I wanted to get this somewhere so I can point friends to so they do their own PDS.

Labels

None yet.

assignee

None yet.

Participants 1
AT URI
at://did:plc:6so2rykrnjmzebbr2zbwbqbx/sh.tangled.repo.pull/3mlcn4ezr6p22
+365 -8
Diff #1
+154
docs/general-configuration.md
··· 1 + # Tranquil PDS Configuration Reference 2 + 3 + Every option in `example.toml` is overridable by an environment variable. This document focuses on what each setting *does* and why some of them are critical to get right before you start. The install guides (`install-containers.md`, `install-kubernetes.md`, `install-nix.md`) cover the mechanics of standing up the service; this covers what to put in it. 4 + 5 + ## Configuration methods 6 + 7 + The server accepts config in two forms, applied in this order (later wins): 8 + 9 + 1. A TOML file, defaulting to `/etc/tranquil-pds/config.toml`. Override the path with `TRANQUIL_PDS_CONFIG`. 10 + 2. Environment variables. Every TOML key has a corresponding env var documented in `example.toml`. 11 + 12 + In practice: TOML for static, non-sensitive settings; env vars (injected from secrets) for anything that would be unfortunate in a file you may want to commit. 13 + 14 + --- 15 + 16 + ## Required settings 17 + 18 + These have no defaults and the Tranqil will not start without them. 19 + 20 + ### `PDS_HOSTNAME` 21 + 22 + The public hostname of your PDS. Example: `pds.example.com`. 23 + 24 + This is used in DID documents, OAuth metadata, and the `/.well-known/atproto-did` endpoint. Getting it wrong after you have accounts is painful: DIDs are registered against this hostname in the PLC directory, and changing it requires a PLC rotation operation. Set it correctly before creating any accounts. 25 + 26 + If your PDS hostname is also the apex domain (e.g. `example.com` rather than `pds.example.com`), users can have their handle *be* the apex domain (e.g. `@example.com`) since the PDS serves `/.well-known/atproto-did` directly without needing a DNS TXT record. 27 + 28 + ### `DATABASE_URL` 29 + 30 + PostgreSQL connection string. Example: `postgres://tranquil_pds:password@host:5432/pds` 31 + 32 + Tranquil PDS uses Postgres for all persistent state: accounts, repos, sessions, the comms queue, and (when using the default `postgres` repo backend) the full block store for every account's atproto repository. 33 + 34 + --- 35 + 36 + ## Required secret/sensitive env vars 37 + 38 + These three are the most sensitive values in the entire deployment. They should never appear in a committed file, in any circumstance. 39 + 40 + ### `JWT_SECRET` 41 + 42 + Used to sign and verify all session JWTs issued by the PDS. 43 + 44 + Generate: `openssl rand -base64 48` 45 + 46 + ### `DPOP_SECRET` 47 + 48 + Used to validate DPoP (Demonstrating Proof of Possession) proofs in OAuth flows. DPoP is the binding mechanism that ties an OAuth access token to the client's key pair, preventing token theft from replaying the token with a different client. 49 + 50 + Generate: `openssl rand -base64 48` 51 + 52 + ### `MASTER_KEY` 53 + 54 + Used as input to HKDF key derivation and for encrypting per-account signing keys at rest. Every account's atproto repo signing key is derived from or wrapped with this value. 55 + 56 + Generate: `openssl rand -base64 48` 57 + 58 + --- 59 + 60 + ## Handle domains 61 + 62 + ### `PDS_USER_HANDLE_DOMAINS` 63 + 64 + Comma-separated list of domains under which handles are issued. Defaults to `PDS_HOSTNAME` if unset. 65 + 66 + If your PDS hostname is `pds.example.com` but you want handles like `alice.example.com`, set this to `example.com`. You need a wildcard TLS cert and DNS wildcard record for the domain(s) you list here. 67 + 68 + ### `PDS_BANNED_WORDS` 69 + 70 + Comma-separated list of substring patterns blocked at registration time. Useful to prevent handles that would collide with your infrastructure (e.g. `grafana`, `vault`, `relay`). The server has a built-in exact-match reserved list (`admin`, `api`, `git`, `mail`, `ssh`, etc.); this adds substring matches for anything missing from that list. 71 + 72 + ### `INVITE_CODE_REQUIRED` 73 + 74 + Default `true`. Keep this enabled on any public-facing PDS unless you specifically want open registration. Invite codes are generated via the admin API/UI. 75 + 76 + --- 77 + 78 + ## Blob storage 79 + 80 + ### `BLOB_STORAGE_BACKEND` 81 + 82 + `filesystem` (default) or `s3`. Pretty straight forward, and dependent on your personal infrastructure requirements and availability. 83 + 84 + ### `BLOB_STORAGE_PATH` 85 + 86 + Path on disk for the filesystem backend. Default `/var/lib/tranquil-pds/blobs`. 87 + 88 + For S3: set `S3_BUCKET` and optionally `S3_ENDPOINT` for S3-compatible stores (Tigris, Cloudflare R2, MinIO, etc.). 89 + 90 + --- 91 + 92 + ## Federation 93 + 94 + ### `CRAWLERS` 95 + 96 + Comma-separated list of relay URLs to notify when new events are committed to an account's repo. Often `https://bsky.network` for production, this is what allows your PDS to be federated to on the wider network. 97 + 98 + --- 99 + 100 + ## Email 101 + 102 + Email is sorta optional but required for account verification, password resets, and 2FA backup codes. If `MAIL_FROM_ADDRESS` is not set, email sending is disabled entirely. 103 + 104 + Tranquil supports two delivery modes: 105 + 106 + ### Smarthost (SMTP relay) 107 + 108 + Set `MAIL_SMARTHOST_HOST`, `MAIL_SMARTHOST_PORT`, `MAIL_SMARTHOST_USERNAME`, and `MAIL_SMARTHOST_PASSWORD`. Use this if you're routing through a sending service (Postmark, SES, etc.) or a local MTA. 109 + 110 + TLS mode defaults to `starttls`. Setting `tls = "none"` alongside a password is rejected at startup. 111 + 112 + ### DKIM signing (`email.dkim`) 113 + 114 + Set `MAIL_DKIM_SELECTOR`, `MAIL_DKIM_DOMAIN`, and `MAIL_DKIM_KEY_PATH` to sign outgoing mail. The key file should be PEM-format RSA or Ed25519. The corresponding DNS TXT record at `<selector>._domainkey.<domain>` must be published before mail is sent. 115 + 116 + --- 117 + 118 + ## Notifications 119 + 120 + Beyond email, Tranquil supports Discord, Telegram, and Signal for user notifications. Each is opt-in: 121 + 122 + - **Discord**: Set `DISCORD_BOT_TOKEN`. 123 + - **Telegram**: Set `TELEGRAM_BOT_TOKEN` and `TELEGRAM_WEBHOOK_SECRET`. 124 + - **Signal**: Set `SIGNAL_ENABLED=true` after linking a device via the admin API. State is persisted in `signal_*` tables in Postgres. 125 + 126 + --- 127 + 128 + ## SSO 129 + 130 + OAuth SSO is supported for GitHub, Discord, Google, GitLab, generic OIDC, and Apple. Each provider requires `enabled = true`, a `client_id`, and a `client_secret`. GitLab and OIDC also need an `issuer` URL. 131 + 132 + `client_secret` values should be injected as env vars (`SSO_GITHUB_CLIENT_SECRET`, etc.) and not placed in config files. 133 + 134 + --- 135 + 136 + ## Cache backend 137 + 138 + ### `CACHE_BACKEND` 139 + 140 + `ripple` (default) or `valkey`. Ripple is the a built-in in-process gossip cache. Valkey is an external Redis fork with it's own complications. 141 + 142 + --- 143 + 144 + ## Optional keys worth knowing 145 + 146 + | Env var | Default | Notes | 147 + |---|---|---| 148 + | `PLC_ROTATION_KEY` | — | Operator-level PLC rotation key (DID key). If unset, per-user keys are used. | 149 + | `ENABLE_PDS_HOSTED_DID_WEB` | `false` | Opt-in did:web hosting. Long-term commitment — only enable if you intend to serve DID documents indefinitely. | 150 + | `MAX_BLOB_SIZE` | 10 GiB | Per-blob upload limit in bytes. | 151 + | `DISABLE_RATE_LIMITING` | `false` | Only for testing. Avoid in production. | 152 + | `DATABASE_MAX_CONNECTIONS` | 100 | Connection pool ceiling. Tune against your Postgres `max_connections`. | 153 + | `FIREHOSE_BACKFILL_HOURS` | 72 | How many hours of history relays can replay from cursor. | 154 + | `REPO_BACKEND` | `postgres` | `tranquil-store` is the experimental embedded backend. Careful if attempting to run in production. |
+211 -8
docs/install-kubernetes.md
··· 1 - # Tranquil PDS on kubernetes 1 + # Tranquil PDS on Kubernetes 2 2 3 - If you're reaching for kubernetes for this app, you're experienced enough to know how to spin up: 3 + If you're reaching for Kubernetes for this app, you're experienced enough to know how to spin up: 4 4 5 5 - cloudnativepg (or your preferred postgres operator) 6 6 - a PersistentVolume for blob storage 7 7 - the app itself (it's just a container with some env vars) 8 8 9 - You'll need a wildcard TLS certificate for `*.your-pds-hostname.example.com`. User handles are served as subdomains. 9 + See [configuration.md](configuration.md) for what each env var does and why the secret ones matter. This guide covers the Kubernetes-specific wiring. 10 10 11 - The container image expects: 11 + Simply, the container image expects: 12 12 - A TOML config file mounted at `/etc/tranquil-pds/config.toml` (or passed via `--config`) 13 13 - `DATABASE_URL` - postgres connection string 14 14 - `BLOB_STORAGE_PATH` - path to blob storage (mount a PV here) ··· 16 16 - `JWT_SECRET`, `DPOP_SECRET`, `MASTER_KEY` - generate with `openssl rand -base64 48` 17 17 - `CRAWLERS` - typically `https://bsky.network` 18 18 19 - and more, check the example.toml for all options. Environment variables can override any TOML value. 20 - You can also point to a config file via the `TRANQUIL_PDS_CONFIG` env var. 19 + --- 21 20 22 - Health check: `GET /xrpc/_health` 21 + ## TLS and DNS 22 + 23 + You need a wildcard TLS certificate covering `*.your-pds-hostname.example.com` — user handles resolve as subdomains, so every user's handle requires a matching cert SAN. 24 + 25 + An approach using Cert Manager would look something like this: 26 + 27 + ```yaml 28 + apiVersion: cert-manager.io/v1 29 + kind: Certificate 30 + metadata: 31 + name: pds-cert 32 + namespace: pds 33 + spec: 34 + secretName: pds-tls 35 + dnsNames: 36 + - pds.example.com 37 + - "*.pds.example.com" 38 + ``` 39 + 40 + If your PDS hostname is the apex domain (so handles are issued under it, not under a subdomain), include the apex in `dnsNames` alongside the wildcard. 41 + 42 + --- 43 + 44 + ## Secrets 45 + 46 + The three secrets primary key secrets (`JWT_SECRET`, `DPOP_SECRET`, `MASTER_KEY`) must never appear in a manifest or config file. Inject them as a Kubernetes Secret, sourced from wherever you manage secrets. 47 + 48 + To create the Secret directly and manage rotation manually: 49 + 50 + ```bash 51 + kubectl create secret generic pds-secrets -n pds \ 52 + --from-literal=DATABASE_URL='postgres://...' \ 53 + --from-literal=JWT_SECRET="$(openssl rand -base64 48)" \ 54 + --from-literal=DPOP_SECRET="$(openssl rand -base64 48)" \ 55 + --from-literal=MASTER_KEY="$(openssl rand -base64 48)" 56 + ``` 57 + 58 + Then reference it in your Deployment: 59 + 60 + ```yaml 61 + envFrom: 62 + - secretRef: 63 + name: pds-secrets 64 + ``` 65 + 66 + --- 67 + 68 + ## PostgreSQL 69 + 70 + CloudNativePG is an easy recommendation, an example for the purpose of Tranquil PDS: 71 + 72 + ```yaml 73 + apiVersion: postgresql.cnpg.io/v1 74 + kind: Cluster 75 + metadata: 76 + name: postgres 77 + namespace: pds 78 + spec: 79 + instances: 1 80 + bootstrap: 81 + initdb: 82 + database: pds 83 + owner: tranquil_pds 84 + secret: 85 + name: postgres-user-secret # k8s Secret with username/password 86 + storage: 87 + storageClass: your-storage-class 88 + size: 10Gi 89 + ``` 90 + 91 + The `postgres-user-secret` Secret needs `username` and `password` keys. The password you put here is what goes into `DATABASE_URL`. 92 + 93 + Any standard Postgres setup works in place of CNPG. Tranquil does not require anything special for a basic installation. 94 + 95 + --- 96 + 97 + ## Storage PVC (if using local-path, nfs, or similar) 98 + 99 + ```yaml 100 + apiVersion: v1 101 + kind: PersistentVolumeClaim 102 + metadata: 103 + name: pds-blobs 104 + namespace: pds 105 + spec: 106 + storageClassName: your-storage-class 107 + accessModes: 108 + - ReadWriteOnce 109 + resources: 110 + requests: 111 + storage: 50Gi 112 + ``` 113 + 114 + Full Deployment example: 115 + 116 + ```yaml 117 + apiVersion: apps/v1 118 + kind: Deployment 119 + metadata: 120 + name: pds 121 + namespace: pds 122 + spec: 123 + replicas: 1 124 + selector: 125 + matchLabels: 126 + app: pds 127 + strategy: 128 + type: Recreate 129 + template: 130 + metadata: 131 + labels: 132 + app: pds 133 + spec: 134 + containers: 135 + - name: pds 136 + image: atcr.io/tranquil.farm/tranquil-pds:latest 137 + ports: 138 + - name: http 139 + containerPort: 3000 140 + env: 141 + - name: SERVER_HOST 142 + value: "0.0.0.0" 143 + - name: SERVER_PORT 144 + value: "3000" 145 + - name: PDS_HOSTNAME 146 + value: "pds.example.com" 147 + - name: PDS_USER_HANDLE_DOMAINS 148 + value: "example.com" 149 + - name: BLOB_STORAGE_BACKEND 150 + value: filesystem 151 + - name: BLOB_STORAGE_PATH 152 + value: /var/lib/tranquil/blobs 153 + - name: CRAWLERS 154 + value: "https://bsky.network" 155 + - name: INVITE_CODE_REQUIRED 156 + value: "true" 157 + envFrom: 158 + - secretRef: 159 + name: pds-secrets 160 + volumeMounts: 161 + - mountPath: /var/lib/tranquil/blobs 162 + name: blobs 163 + readinessProbe: 164 + httpGet: 165 + path: /xrpc/_health 166 + port: http 167 + initialDelaySeconds: 15 168 + periodSeconds: 10 169 + failureThreshold: 6 170 + resources: 171 + requests: 172 + memory: 256Mi 173 + cpu: 100m 174 + limits: 175 + memory: 1Gi 176 + volumes: 177 + - name: blobs 178 + persistentVolumeClaim: 179 + claimName: pds-blobs 180 + ``` 181 + 182 + `SERVER_HOST: "0.0.0.0"` is required — the default `127.0.0.1` isn't reachable by either the Kubelet for health checks or your ingress controller. 183 + 184 + --- 185 + 186 + ## Ingress 187 + 188 + The ingress rule must match both the PDS hostname itself and the wildcard for user handles. 189 + 190 + ```yaml 191 + apiVersion: networking.k8s.io/v1 192 + kind: Ingress 193 + metadata: 194 + name: pds 195 + namespace: pds 196 + spec: 197 + tls: 198 + - hosts: 199 + - pds.example.com 200 + - "*.example.com" 201 + secretName: pds-tls 202 + rules: 203 + - host: pds.example.com 204 + http: 205 + paths: 206 + - path: / 207 + pathType: Prefix 208 + backend: 209 + service: 210 + name: pds 211 + port: 212 + number: 3000 213 + - host: "*.example.com" 214 + http: 215 + paths: 216 + - path: / 217 + pathType: Prefix 218 + backend: 219 + service: 220 + name: pds 221 + port: 222 + number: 3000 223 + ``` 224 + 225 + --- 23 226 24 227 ## Custom homepage 25 228 26 - Mount a ConfigMap with your `homepage.html` into the container's frontend directory and it becomes your landing page. Go nuts with it. Account dashboard is at `/app/` so you won't break anything. 229 + Mount a ConfigMap with your `homepage.html` into the container's frontend directory and it becomes your landing page. The account dashboard lives at `/app/` so you won't displace it. 27 230 28 231 ```yaml 29 232 apiVersion: v1

History

4 rounds 0 comments
sign up or login to add to the discussion
dsx.sh submitted #3
diff interdiff
6 commits
expand
863d8480
adding k8s docs and a more generalized doc
d9c99c04
oops didnt actually mean to take that out
44c7255f
removing gen config altogether, agree with nel
697cf8c2
expanding example.toml a tiny bit and reworking k8s a little
541abaf4
rewording, and some sentence simplification
30e09e90
more rewording
merge conflicts detected
expand
expand 0 comments
dsx.sh submitted #2
diff interdiff
5 commits
expand
863d8480
adding k8s docs and a more generalized doc
d9c99c04
oops didnt actually mean to take that out
44c7255f
removing gen config altogether, agree with nel
697cf8c2
expanding example.toml a tiny bit and reworking k8s a little
541abaf4
rewording, and some sentence simplification
expand 0 comments
dsx.sh submitted #1
interdiff
2 commits
expand
863d8480
adding k8s docs and a more generalized doc
d9c99c04
oops didnt actually mean to take that out
expand 0 comments
dsx.sh submitted #0
diff
1 commit
expand
863d8480
adding k8s docs and a more generalized doc
expand 0 comments