Mirror from bluesky-social/pds
1# PDS
2
3Welcome to the repository for the official Bluesky PDS (Personal Data Server). This repository includes container images and documentation designed to assist technical people with hosting a Bluesky PDS.
4
5Head over to the [ATProto Touchers Discord](https://discord.atprotocol.dev/) to chat with other folks hosting instances and get important updates about the PDS distribution!
6
7## Table of Contents
8
9<!-- markdown-toc -i README.md -->
10
11<!-- toc -->
12
13- [PDS](#pds)
14 - [Table of Contents](#table-of-contents)
15 - [FAQ](#faq)
16 - [What is Bluesky?](#what-is-bluesky)
17 - [What is AT Protocol?](#what-is-at-protocol)
18 - [Where is the code?](#where-is-the-code)
19 - [What is the current status of federation?](#what-is-the-current-status-of-federation)
20 - [Self-hosting a PDS](#self-hosting-a-pds)
21 - [Deploying a PDS onto a VPS](#deploying-a-pds-onto-a-vps)
22 - [Open your cloud firewall for HTTP and HTTPS](#open-your-cloud-firewall-for-http-and-https)
23 - [Configure DNS for your domain](#configure-dns-for-your-domain)
24 - [Check that DNS is working as expected](#check-that-dns-is-working-as-expected)
25 - [Installing on Ubuntu 20.04/22.04/24.04 and Debian 11/12/13](#installing-on-ubuntu-200422042404-and-debian-111213)
26 - [Verifying that your PDS is online and accessible](#verifying-that-your-pds-is-online-and-accessible)
27 - [goat CLI](#goat-cli)
28 - [Creating an account](#creating-an-account)
29 - [Creating an account using an invite code](#creating-an-account-using-an-invite-code)
30 - [Using the Bluesky app with your PDS](#using-the-bluesky-app-with-your-pds)
31 - [Setting up SMTP](#setting-up-smtp)
32 - [Common SMTP issues](#common-smtp-issues)
33 - [Logging](#logging)
34 - [Updating your PDS](#updating-your-pds)
35 - [Environment Variables](#environment-variables)
36 - [Migrating your PDS](#migrating-your-pds)
37 - [Fixing a Relay desync](#fixing-a-relay-desync)
38 - [License](#license)
39
40<!-- tocstop -->
41
42## FAQ
43
44### What is Bluesky?
45
46Bluesky is a social media application built on AT Protocol.
47
48Please visit the [Bluesky website](https://bsky.social/) for more information.
49
50### What is AT Protocol?
51
52The Authenticated Transfer Protocol, aka atproto, is a protocol for large-scale distributed social applications.
53
54Please visit the [AT Protocol docs](https://atproto.com/guides/overview) for additional information.
55
56### Where is the code?
57
58* [TypeScript code](https://github.com/bluesky-social/atproto/tree/main/packages/pds)
59* [Go code](https://github.com/bluesky-social/indigo)
60
61### What is the current status of federation?
62
63The AT Protocol network is open to federation!
64
65✅ Federated domain handles (e.g. `@nytimes.com`)
66
67✅ Federated feed generators (custom algorithms)
68
69✅ Federated relays (event firehose)
70
71✅ Federated app views (API service)
72
73✅ Federated data (PDS hosting)
74
75✅ Federated moderation (labeling)
76
77## Self-hosting a PDS
78
79Self-hosting a Bluesky PDS means running your own Personal Data Server that is capable of federating with the wider Bluesky social network.
80
81### Deploying a PDS onto a VPS
82
83This README provides instructions for deploying a PDS using our install script onto a Virtual Private Server. [Digital Ocean](https://digitalocean.com/) and [Vultr](https://vultr.com/) are two popular choices for VPS hosting.
84
85Ensure that you can ssh to your server and have root access.
86
87**Server Requirements**
88* Public IPv4 address
89* Public DNS name
90* Public inbound internet access permitted on port 80/tcp and 443/tcp
91
92**Server Recommendations**
93| | |
94| ---------------- | ------------ |
95| Operating System | Ubuntu 24.04 |
96| Memory (RAM) | 1 GB |
97| CPU Cores | 1 |
98| Storage | 20 GB SSD |
99| Architectures | amd64, arm64 |
100| Number of users | 1-20 |
101
102**Note:** It is a good security practice to restrict inbound ssh access (port 22/tcp) to your own computer's public IP address. You can check your current public IP address using [ifconfig.me](https://ifconfig.me/).
103
104### Open your cloud firewall for HTTP and HTTPS
105
106One of the most common sources of misconfiguration is not opening firewall ports correctly. Please be sure to double check this step.
107
108In your cloud provider's console, the following ports should be open to inbound access from the public internet.
109
110* 80/tcp (Used only for TLS certification verification)
111* 443/tcp (Used for all application requests)
112
113**Note:** there is no need to set up TLS or redirect requests from port 80 to 443 because the Caddy web server, included in the Docker compose file, will handle this for you.
114
115### Configure DNS for your domain
116
117From your DNS provider's control panel, set up a domain with records pointing to your server.
118
119| Name | Type | Value | TTL |
120| --------------- | ---- | ------------- | --- |
121| `example.com` | `A` | `12.34.56.78` | 600 |
122| `*.example.com` | `A` | `12.34.56.78` | 600 |
123
124**Note:**
125* Replace `example.com` with your domain name.
126* Replace `12.34.56.78` with your server's IP address.
127* Some providers may use the `@` symbol to represent the root of your domain.
128* The wildcard record is required when allowing users to create new accounts on your PDS.
129* The TTL can be anything but 600 (10 minutes) is reasonable
130
131### Check that DNS is working as expected
132
133Use a service like [DNS Checker](https://dnschecker.org/) to verify that you can resolve domain names.
134
135Examples to check (record type `A`):
136* `example.com`
137* `random.example.com`
138* `test123.example.com`
139
140These should all return your server's public IP.
141
142### Installing on Ubuntu 20.04/22.04/24.04 and Debian 11/12/13
143
144Note that this script assumes a relatively "fresh" VPS that is not also concurrently hosting a web server or anything else on port 80/443. If you intend to run a PDS alongside an existing webserver on the same VPS, you will not want to use this install script.
145
146On your server, download the install script using `curl`:
147
148```bash
149curl https://raw.githubusercontent.com/bluesky-social/pds/main/installer.sh > installer.sh
150```
151
152And then run the installer using `bash`. You will need `sudo` permissions to continue:
153
154```bash
155sudo bash installer.sh
156```
157
158The install script is interactive and will prompt for input during the install process. You will need to provide your public DNS address, an admin email address (which does not need to be from the same domain), and be prompted to create a PDS user account with its own email address and handle. If you plan to reuse an existing AT handle, you can skip user account creation, though if it is your first time deploying a PDS you may want to create an account using your domain like `account.your-domain.net` for testing purposes.
159
160Upon completion of a successful installation, you'll receive output similar to the following:
161
162```
163========================================================================
164PDS installation successful!
165------------------------------------------------------------------------
166
167Check service status : sudo systemctl status pds
168Watch service logs : sudo docker logs -f pds
169Backup service data : /pds
170PDS Admin command : pdsadmin
171
172Required Firewall Ports
173------------------------------------------------------------------------
174Service Direction Port Protocol Source
175------- --------- ---- -------- ----------------------
176HTTP TLS verification Inbound 80 TCP Any
177HTTP Control Panel Inbound 443 TCP Any
178
179Required DNS entries
180------------------------------------------------------------------------
181Name Type Value
182------- --------- ---------------
183your-domain.net A your-ip-address
184*.your-domain.net A your-ip-address
185
186Detected public IP of this server: your-ip-address
187
188To see pdsadmin commands, run "pdsadmin help"
189
190========================================================================
191```
192
193And, following account creation:
194
195```
196Account created successfully!
197-----------------------------
198Handle : handle.your-domain.net
199DID : did:plc:your-did
200Password : your-password
201-----------------------------
202Save this password, it will not be displayed again.
203```
204
205### Verifying that your PDS is online and accessible
206
207> [!TIP]
208> The most common problems with getting PDS content consumed in the live network usually result from users trying to port the provided Caddy configuration to Nginx, Apache, or other reverse proxies. Getting TLS certificates, WebSockets, and virtual server names provisioned can be challenging. We are not currently providing tech support for other configurations.
209
210After installation, your PDS should be live and accessible on the web. You can check if your server is online and healthy by making a request to `https://your-domain.net/xrpc/_health` (the healthcheck endpoint). You should see a JSON response with a version, like:
211
212Visit `https://your-domain.net/xrpc/_health` in your browser. You should see a JSON response with a version, like:
213
214```
215{"version":"0.2.2-beta.2"}
216```
217
218You'll also need to check that WebSockets are working, for the rest of the network to pick up content from your PDS. You can test by installing a tool like `wsdump` and running a command like:
219
220```bash
221wsdump "wss://example.com/xrpc/com.atproto.sync.subscribeRepos?cursor=0"
222```
223
224Note that there will be no events output on the WebSocket until they are created in the PDS, so the above command may continue to run with no output immediately post-installation.
225
226### goat CLI
227
228The PDS image includes [goat](https://github.com/bluesky-social/goat), our command line tool for performing admin functions. `goat` can be used locally with a `--pds-host` parameter or within the container installed by this script.
229
230### Creating an account
231
232You can run the `goat` command included in this container with `docker exec pds goat`.
233
234Use `goat pds admin account create` to create an account if you haven't already:
235
236```bash
237docker exec pds goat pds admin account create --admin-password `PDS_ADMIN_PASSWORD` --handle newuser.pds.net --email new-user@email.com --password new-password
238```
239
240> [!NOTE]
241> You can find `PDS_ADMIN_PASSWORD` in `/pds/pds.env` following installation. You can typically skip this arg when running `goat` from the `docker` container as `PDS_ADMIN_PASSWORD` should already be set.
242
243### Creating an account using an invite code
244
245If needed, use `goat` to create an invite code:
246
247```bash
248docker exec pds goat pds admin --admin-password PDS_ADMIN_PASSWORD create-invites
249```
250
251When creating an account using the app, enter this invite code.
252
253### Using the Bluesky app with your PDS
254
255You can use the Bluesky app to connect to your PDS.
256
2571. Get the Bluesky app
258 * [Bluesky for Web](https://bsky.app/)
259 * [Bluesky for iPhone](https://apps.apple.com/us/app/bluesky-social/id6444370199)
260 * [Bluesky for Android](https://play.google.com/store/apps/details?id=xyz.blueskyweb.app)
2611. Enter the URL of your PDS (e.g. `https://example.com/`)
262
263_Note: because the subdomain TLS certificate is created on-demand, it may take 10-30s for your handle to be accessible. If you aren't seeing your first post/profile, wait 30s and try to make another post._
264
265### Setting up SMTP
266
267To be able to verify users' email addresses and send other emails, you need to set up an SMTP server.
268
269As an alternative to running your own SMTP server, you can use an email service. [Resend](https://resend.com/) and [SendGrid](https://sendgrid.com/) are two popular choices.
270
271Create an account and API key on an email service, ensure your server allows access on the required ports, and then you can add these configuration variables to `/pds/pds.env` on your server (example with Resend):
272
273```
274PDS_EMAIL_SMTP_URL=smtps://resend:<your api key here>@smtp.resend.com:465/
275PDS_EMAIL_FROM_ADDRESS=admin@your.domain
276```
277
278If you prefer to use a standard SMTP server (a local one or from your email provider), put your account's username and password in the URL:
279
280```
281PDS_EMAIL_SMTP_URL=smtps://username:password@smtp.example.com/
282```
283
284Alternatively, if you're running a local sendmail-compatible mail service like Postfix or Exim on the same host, you can configure the PDS to use the sendmail transport by using such URL:
285
286```
287PDS_EMAIL_SMTP_URL=smtp:///?sendmail=true
288```
289
290_Note: Your PDS will need to be restarted with those variables. This varies depending on your setup. If you followed this installation guide, run `systemctl restart pds`. You might need to restart the server or recreate the container, depending on what you are using._
291
292#### Common SMTP issues
293
294If you find that your test messages using cURL or other sources go out correctly, but you are not receiving emails from your PDS, you may need to URL encode your username and password on `/pds/pds.env` and restart the PDS service.
295
296If the username and/or password contain special characters, the special characters will need to be [percent encoded](https://en.wikipedia.org/wiki/Percent-encoding). For some email services, the username will contain an extra `@` symbol that will also need to be percent encoded. For example, the URL `user&name@oci:p@ssword@smtphost:465` after percent encoding for the username and password fields would become `user%26name%40oci:p%40ssword@smtphost:465`.
297
298If you are migrating an account, Bluesky's UI will ask you to confirm your email address. The confirmation code email is meant to come from your PDS. If you are encountering issues with SMTP and want to confirm the address before solving it, you can find the confirmation code on the `email_token` table on `accounts.sqlite`.
299
300### Logging
301
302By default, logs from the PDS are printed to `stdout` and end up in Docker's log. You can browse them by running:
303
304```
305[sudo] docker logs pds
306```
307
308Note: these logs are not persisted, so they will be lost after server reboot.
309
310Alternatively, you can configure the logs to be printed to a file by setting `LOG_DESTINATION`:
311
312```
313LOG_DESTINATION=/pds/pds.log
314```
315
316You can also change the minimum level of logs to be printed (default: `info`):
317
318```
319LOG_LEVEL=debug
320```
321
322### Updating your PDS
323
324It is recommended that you keep your PDS up to date with new versions. You can use the `pdsadmin` tool to update your PDS.
325
326```bash
327sudo pdsadmin update
328```
329
330### Environment Variables
331
332| Environment Variable | Default |
333| ------------------------------------------- | ---------------------------------- |
334| `PDS_HOSTNAME` | None |
335| `PDS_JWT_SECRET` | None |
336| `PDS_ADMIN_PASSWORD` | None |
337| `PDS_PLC_ROTATION_KEY_K256_PRIVATE_KEY_HEX` | None |
338| `PDS_DATA_DIRECTORY` | `/pds` |
339| `PDS_BLOBSTORE_DISK_LOCATION` | `/pds/blocks` |
340| `PDS_BLOB_UPLOAD_LIMIT` | `104857600` (100MB) |
341| `PDS_DID_PLC_URL` | `https://plc.directory` |
342| `PDS_BSKY_APP_VIEW_URL` | `https://api.bsky.app` |
343| `PDS_BSKY_APP_VIEW_DID` | `did:web:api.bsky.app` |
344| `PDS_REPORT_SERVICE_URL` | `https://mod.bsky.app` |
345| `PDS_REPORT_SERVICE_DID` | `did:plc:ar7c4by46qjdydhdevvrndac` |
346| `PDS_CRAWLERS` | `https://bsky.network` |
347| `LOG_ENABLED` | `true` |
348| `PDS_EMAIL_SMTP_URL` | None |
349| `PDS_EMAIL_FROM_ADDRESS` | None |
350| `PDS_CONTACT_EMAIL_ADDRESS` | None |
351| `PDS_PRIVACY_POLICY_URL` | None |
352| `PDS_TERMS_OF_SERVICE_URL` | None |
353| `PDS_RATE_LIMITS_ENABLED` | `true` |
354| `PDS_INVITE_REQUIRED` | `true` |
355
356### Migrating your PDS
357
358Once you've deployed a PDS, it will automatically begin broadcasting events to Relay servers.
359
360This means that if you wipe and reinstall your PDS on the same hostname, or move it to a new hostname without performing the appropriate cutover steps, the PDS and Relay can go out of sync.
361
362To avoid this, you should always [migrate accounts individually](https://atproto.com/guides/account-migration) from one PDS to another when updating your PDS host configuration.
363
364### Fixing a Relay desync
365
366If you become desynced from the Relay due to migration issues — i.e., new records created on your PDS aren't being picked up by other applications — you can fix it with these steps:
367
3681. Look up the last known sequence number for your PDS host. You can use `goat` to retrieve the last observed cursor from the `bsky.network` relay:
369
370```bash
371docker exec pds sh -c 'goat relay host status "$PDS_HOSTNAME" --json'
372```
373
374```
375{"hostname":"justdothings.net","seq":12,"status":"active"}
376```
377
3782. Stop the `pds` service:
379
380```bash
381systemctl stop pds
382```
383
3843. Set the cursor sequence on your PDS to the value of `seq` from step 1, incremented by 1:
385
386```bash
387sqlite3 /pds/sequencer.sqlite "UPDATE sqlite_sequence SET seq = {new_sequence_number} WHERE name = 'repo_seq';"
388```
389
3904. Restart your `pds` service:
391
392```bash
393systemctl start pds
394```
395
396New records should now be indexed as normal.
397
398## License
399
400This project is dual-licensed under MIT and Apache 2.0 terms:
401
402- MIT license ([LICENSE-MIT.txt](https://github.com/bluesky-social/pds/blob/main/LICENSE-MIT.txt) or http://opensource.org/licenses/MIT)
403- Apache License, Version 2.0, ([LICENSE-APACHE.txt](https://github.com/bluesky-social/pds/blob/main/LICENSE-APACHE.txt) or http://www.apache.org/licenses/LICENSE-2.0)
404
405Downstream projects and end users may choose either license individually, or both together, at their discretion. The motivation for this dual-licensing is the additional software patent assurance provided by Apache 2.0.