···11---
22-title: Architecture Guide
22+title: Architecture
33description: How the hosting service, firehose service, and tiered storage work together
44---
5566-Wisp.place's serving infrastructure is split into two microservices: the **firehose service** (write path) and the **hosting service** (read path). They communicate through S3-compatible storage and Redis pub/sub.
66+Wisp.place splits into two microservices: the **firehose service** (write path) and the **hosting service** (read path). They communicate through S3-compatible storage and Redis pub/sub.
7788-## Service Overview
99-1010-### Firehose Service
1111-1212-The firehose service watches the AT Protocol firehose (Jetstream WebSocket) for `place.wisp.fs` and `place.wisp.settings` record changes. When a site is created, updated, or deleted, it:
1313-1414-1. Downloads all blobs from the user's PDS
1515-2. Decompresses gzipped content
1616-3. Rewrites HTML for subdirectory serving (absolute paths become relative)
1717-4. Writes the processed files to S3 (or disk)
1818-5. Publishes a cache invalidation event to Redis
88+## Firehose Service
1992020-The firehose service is **write-only** — it never serves requests to end users.
1010+The firehose service watches the AT Protocol Jetstream for `place.wisp.fs` and `place.wisp.settings` record changes. When a site is created or updated, it downloads all blobs from the user's PDS, decompresses gzipped content, rewrites HTML for subdirectory serving, writes processed files to S3 (or disk), then publishes a cache invalidation event to Redis.
21112222-**Key configuration:**
1212+It's write-only — it never serves requests to end users.
23132414```bash
2525-# Firehose connection
2626-FIREHOSE_URL="wss://jetstream2.us-east.bsky.network/subscribe"
2727-2828-# S3 storage (recommended for production)
1515+FIREHOSE_SERVICE="wss://bsky.network"
1616+FIREHOSE_MAX_CONCURRENCY=5
2917S3_BUCKET="wisp-sites"
3030-S3_REGION="auto"
1818+S3_REGION="us-east-1"
3119S3_ENDPOINT="https://your-account.r2.cloudflarestorage.com"
3232-S3_ACCESS_KEY_ID="..."
3333-S3_SECRET_ACCESS_KEY="..."
3434-3535-# Redis for cache invalidation
2020+S3_FORCE_PATH_STYLE="false"
2121+S3_PREFIX="sites/"
2222+AWS_ACCESS_KEY_ID="..."
2323+AWS_SECRET_ACCESS_KEY="..."
3624REDIS_URL="redis://localhost:6379"
3737-3838-# Concurrency control
3939-FIREHOSE_CONCURRENCY=5 # Max parallel event processing
4025```
41264242-**Backfill mode:** Start with `--backfill` to do a one-time bulk sync of all existing sites from the database into the cache.
2727+Start with `--backfill` to do a one-time bulk sync of all existing sites into cache.
43284444-### Hosting Service
2929+## Hosting Service
45304646-The hosting service is a **read-only** CDN built with Node.js and Hono. It serves static files from a three-tier cache and handles routing for custom domains, wisp subdomains, and direct URLs.
3131+The hosting service is a read-only CDN built with Hono. It resolves sites from the request hostname or path, looks up files in tiered storage (hot → warm → cold), fetches directly from the user's PDS on a cache miss, applies HTML path rewriting and `_redirects` rules, and serves the file.
47324848-On each request, the hosting service:
4949-5050-1. Resolves the site from the request hostname/path
5151-2. Looks up the file in tiered storage (hot → warm → cold)
5252-3. On a cache miss, fetches from the PDS on-demand and populates the cache
5353-4. Applies HTML path rewriting if serving from a subdirectory
5454-5. Processes `_redirects` rules
5555-6. Serves the file with appropriate headers
5656-5757-The hosting service subscribes to Redis pub/sub for cache invalidation messages from the firehose service. When it receives an invalidation, it evicts the affected entries from its hot and warm tiers so the next request fetches fresh content.
3333+It subscribes to Redis pub/sub for invalidation events from the firehose service. On invalidation, it evicts affected entries from hot and warm tiers so the next request fetches fresh content.
58345935## Tiered Storage
60366161-The `@wispplace/tiered-storage` package implements a three-tier cascading cache. Data flows **down** on writes and is looked up **upward** on reads.
3737+`@wispplace/tiered-storage` implements a three-tier cascading cache:
62386339```
6464-Read path: Hot (memory) → Warm (disk) → Cold (S3/disk)
6565-Write path: Hot ← Warm ← Cold (writes cascade down through all tiers)
4040+Read: Hot (memory) → Warm (disk) → Cold (S3/disk)
4141+Write: Hot ← Warm ← Cold
6642```
67436868-### Hot Tier (Memory)
6969-7070-- **Implementation:** In-memory LRU cache
7171-- **Eviction:** Size-based (bytes) and count-based (max items)
7272-- **Use case:** Frequently accessed files (index.html, CSS, JS)
7373-- **Lost on restart** — repopulated from warm/cold tiers on access
4444+The **hot tier** is an in-memory LRU cache. Fast, small, and lost on restart — repopulated from warm/cold on access.
74457546```bash
7676-HOT_CACHE_SIZE=104857600 # 100 MB (default)
7777-HOT_CACHE_COUNT=500 # Max items
4747+HOT_CACHE_SIZE=104857600 # 100 MB
4848+HOT_CACHE_COUNT=500
7849```
79508080-### Warm Tier (Disk)
8181-8282-- **Implementation:** Filesystem with human-readable paths
8383-- **Eviction:** Configurable — `lru` (default), `fifo`, or `size`
8484-- **Structure:** `cache/sites/{did}/{sitename}/path/to/file`
8585-- **Survives restarts** — provides fast local reads without network calls
5151+The **warm tier** is a disk cache at `cache/sites/{did}/{sitename}/path`. It survives restarts and requires no network.
86528753```bash
8888-WARM_CACHE_SIZE=10737418240 # 10 GB (default)
8989-WARM_EVICTION_POLICY=lru # lru, fifo, or size
5454+WARM_CACHE_SIZE=10737418240 # 10 GB
5555+WARM_EVICTION_POLICY=lru # lru, fifo, or size
9056CACHE_DIR=./cache/sites
9157```
92589393-The warm tier is optional when S3 is configured. Without S3, disk acts as the cold (source of truth) tier.
9494-9595-### Cold Tier (S3 or Disk)
9696-9797-- **With S3:** The firehose service writes here; the hosting service reads (read-only wrapper)
9898-- **Without S3:** A disk-based tier serves as both warm and cold
9999-- **Compatible with:** Cloudflare R2, MinIO, AWS S3, or any S3-compatible endpoint
5959+The **cold tier** is S3 (or disk if S3 isn't configured). The firehose writes here; the hosting service reads. Without S3, disk serves as both warm and cold.
1006010161```bash
10262S3_BUCKET="wisp-sites"
103103-S3_REGION="auto"
6363+S3_REGION="us-east-1"
10464S3_ENDPOINT="https://your-account.r2.cloudflarestorage.com"
105105-S3_ACCESS_KEY_ID="..."
106106-S3_SECRET_ACCESS_KEY="..."
107107-S3_METADATA_BUCKET="wisp-metadata" # Optional, recommended for production
6565+S3_FORCE_PATH_STYLE="false"
6666+S3_PREFIX="sites/"
6767+AWS_ACCESS_KEY_ID="..."
6868+AWS_SECRET_ACCESS_KEY="..."
10869```
10970110110-### Tier Placement Rules
111111-112112-Not all files are placed on every tier. The hosting service uses placement rules to keep the hot tier efficient:
113113-114114-| File Pattern | Tiers | Rationale |
115115-|---|---|---|
116116-| `index.html`, `*.css`, `*.js` | Hot, Warm, Cold | Critical for page loads |
117117-| Rewritten HTML (`.rewritten/`) | Hot, Warm, Cold | Pre-processed for fast serving |
118118-| Images, fonts, media (`*.jpg`, `*.woff2`, etc.) | Warm, Cold | Already compressed, large — skip memory |
119119-| Everything else | Warm, Cold | Default placement |
120120-121121-### Promotion and Bootstrap
122122-123123-When a file is found in a lower tier but not a higher one, it's **eagerly promoted** upward. For example, a cache miss on hot that hits warm will copy the file into hot for future requests.
124124-125125-On startup, the hosting service can **bootstrap** tiers:
126126-- Hot bootstraps from warm by loading the most-accessed items
127127-- Warm bootstraps from cold by loading recently written items
7171+Not everything goes on every tier. HTML, CSS, and JS go hot/warm/cold since they're critical for page loads. Large files like images and fonts skip hot — they'd just eat memory. When a file is found in a lower tier but not a higher one, it's promoted upward so the next request is faster.
1287212973## Cache Invalidation
13074131131-The firehose service and hosting service communicate through Redis pub/sub:
132132-13375```
134134-Firehose Service Hosting Service
135135- │ │
136136- │ ── Redis pub/sub ──────────────→ │
137137- │ (wisp:revalidate) │
138138- │ │
139139- │ Site updated/deleted: │ Receives invalidation:
140140- │ 1. Write new files to S3 │ 1. Evict from hot tier
141141- │ 2. Publish invalidation │ 2. Evict from warm tier
142142- │ │ 3. Next request fetches fresh
7676+Firehose Hosting
7777+ │ │
7878+ │ ── Redis pub/sub ────────────→ │
7979+ │ (wisp:revalidate) │
8080+ │ │
8181+ │ Site updated: │ Receives invalidation:
8282+ │ 1. Write new files to S3 │ 1. Evict from hot tier
8383+ │ 2. Publish invalidation │ 2. Evict from warm tier
8484+ │ │ 3. Next request fetches fresh
14385```
14486145145-If Redis is not configured, the hosting service still works — it just won't receive real-time invalidation and will rely on TTL-based expiry (default 14 days) and on-demand fetching.
8787+Without Redis the hosting service still works — it falls back to TTL-based expiry (14 days default) and on-demand fetching.
14688147147-## On-Demand Cache Population
8989+## Cache Misses
14890149149-When the hosting service receives a request for a site that isn't in any cache tier, it fetches directly from the user's PDS:
9191+The hosting service handles cache misses in two ways depending on whether it knows about the site.
15092151151-1. Resolves the user's DID to their PDS endpoint
152152-2. Downloads the `place.wisp.fs` record
153153-3. Fetches the requested blob
154154-4. Decompresses and processes the file
155155-5. Stores it in the appropriate tiers based on placement rules
156156-6. Serves the response
9393+If a site **is in the database** but its files are missing from all storage tiers, the request returns 503 and a revalidation job is enqueued to Redis for the firehose service to re-sync from the PDS. No direct PDS fetch happens here.
15794158158-This means the hosting service works even without the firehose service running — it just won't have pre-populated caches.
9595+If a site **is not in the database at all**, the hosting service fetches it directly from the PDS: it resolves the DID, downloads the `place.wisp.fs` record, fetches all blobs, writes them to hot and warm tiers, and then enqueues a revalidation job so the firehose backfills S3.
1599616097## Deployment Scenarios
16198162162-### Minimal (Disk Only)
163163-164164-No S3 or Redis required. The hosting service uses disk as both warm and cold tier. Best for small deployments or development.
9999+**Disk only** — No S3 or Redis. The hosting service uses disk as both warm and cold. Good for small deployments and development.
165100166101```bash
167167-# Hosting service only
168102CACHE_DIR=./cache/sites
169103HOT_CACHE_SIZE=104857600
170104```
171105172172-### Production (S3 + Redis)
173173-174174-The firehose service pre-populates S3 and notifies the hosting service of changes via Redis. Multiple hosting service instances can share the same S3 backend.
106106+**S3 + Redis** — The firehose pre-populates S3 and notifies the hosting service of changes. Multiple hosting instances can share the same S3 backend.
175107176108```bash
177177-# Both services
178109S3_BUCKET=wisp-sites
179110S3_ENDPOINT=https://account.r2.cloudflarestorage.com
111111+AWS_ACCESS_KEY_ID=...
112112+AWS_SECRET_ACCESS_KEY=...
180113REDIS_URL=redis://localhost:6379
181181-182182-# Hosting service
183114HOT_CACHE_SIZE=104857600
184115WARM_CACHE_SIZE=10737418240
185116```
186117187187-### Scaled (Multiple Hosting Instances)
188188-189189-Run multiple hosting service instances behind a load balancer. Each has its own hot and warm tiers, but they share the S3 cold tier and receive the same Redis invalidation events.
118118+**Scaled** — Run multiple hosting instances behind a load balancer. Each has its own hot and warm tiers but shares S3 and Redis invalidation.
190119191120```
192192- Load Balancer
193193- / | \
194194- Hosting-1 Hosting-2 Hosting-3
195195- (hot+warm) (hot+warm) (hot+warm)
196196- \ | /
197197- S3 (cold tier)
198198- |
199199- Firehose Service
121121+ Load Balancer
122122+ / | \
123123+ Hosting-1 Hosting-2 Hosting-3
124124+ (hot+warm) (hot+warm) (hot+warm)
125125+ \ | /
126126+ S3 (cold tier)
127127+ |
128128+ Firehose Service
200129```
201130202131## Observability
203132204204-Both services expose internal observability endpoints:
133133+Both services expose internal endpoints:
205134206206-- `/__internal__/observability/logs` — Recent log entries
207207-- `/__internal__/observability/errors` — Error log entries
208208-- `/__internal__/observability/metrics` — Prometheus-format metrics
209209-- `/__internal__/observability/cache` — Cache tier statistics (hosting service only)
135135+- `/__internal__/observability/logs`
136136+- `/__internal__/observability/errors`
137137+- `/__internal__/observability/metrics`
138138+- `/__internal__/observability/cache` (hosting service only)
210139211211-See [Monitoring & Metrics](/monitoring) for Grafana integration details.
140140+See [Monitoring & Metrics](/monitoring).
+148-197
docs/src/content/docs/cli.md
···11---
22-title: Wisp CLI v1.0.0
22+title: Wisp CLI
33description: Command-line tool for deploying static sites to the AT Protocol
44---
5566-**Deploy static sites to the AT Protocol**
66+`wispctl` deploys static sites to your AT Protocol account from the terminal. Supports incremental updates, OAuth and app password auth, and a local dev server with live firehose updates.
7788-The Wisp CLI is a command-line tool for deploying static websites directly to your AT Protocol account. Host your sites on wisp.place with full ownership and control, backed by the decentralized AT Protocol.
88+## Installation
991010-## Features
1010+```bash
1111+npm install -g wispctl
1212+```
11131212-- **Deploy**: Push static sites directly from your terminal
1313-- **Pull**: Download sites from the PDS for development or backup
1414-- **Serve**: Run a local server with real-time firehose updates
1515-- **Authenticate** with app password or OAuth
1616-- **Incremental updates**: Only upload changed files
1414+Then use `wispctl` anywhere:
1515+1616+```bash
1717+wispctl deploy your-handle.bsky.social --path ./dist --site my-site
1818+```
17191818-## Downloads
2020+## Quick Deploy
2121+2222+No install needed — use `npm create wisp` to deploy directly:
2323+2424+```bash
2525+npm create wisp your-handle.bsky.social --path ./dist --site my-site
2626+```
2727+2828+Or with `npx`:
2929+3030+```bash
3131+npx wispctl deploy your-handle.bsky.social --path ./dist --site my-site
3232+```
3333+3434+## Deploying a Site
3535+3636+```bash
3737+wispctl deploy your-handle.bsky.social --path ./dist --site my-site
3838+```
3939+4040+Your site will be at `https://sites.wisp.place/your-handle/my-site`.
4141+4242+The CLI tracks files by content hash (CID), so subsequent deploys only upload what actually changed. First deploy uploads everything; after that, deploys complete in seconds when only a few files differ.
19432020-<div class="downloads">
4444+## Authentication
21452222-<h2>Download v1.0.0</h2>
4646+OAuth is the default — it opens your browser and saves a session to `/tmp/wisp-oauth-session.json`. For CI/CD or headless environments, use an app password instead:
23472424-<a href="https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-aarch64-darwin" class="download-link" download="">
4848+```bash
4949+wispctl deploy your-handle.bsky.social \
5050+ --path ./dist \
5151+ --site my-site \
5252+ --password YOUR_APP_PASSWORD
5353+```
25542626-<span class="platform">macOS (Apple Silicon):</span> wisp-cli-aarch64-darwin
5555+Generate app passwords from your AT Protocol account settings. Don't use your main password.
27562828-</a>
5757+## Domain Management
29583030-<a href="https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-x86_64-darwin" class="download-link" download="">
5959+```bash
6060+# Claim a wisp.place subdomain
6161+wispctl domain claim-subdomain your-handle.bsky.social --subdomain alice
31623232-<span class="platform">macOS (Intel):</span> wisp-cli-x86_64-darwin
6363+# Claim a custom domain
6464+wispctl domain claim your-handle.bsky.social --domain example.com
33653434-</a>
6666+# Check domain status
6767+wispctl domain status your-handle.bsky.social --domain example.com
35683636-<a href="https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-aarch64-linux" class="download-link" download="">
6969+# Attach a site to a domain
7070+wispctl domain add-site your-handle.bsky.social --domain example.com --site mysite
37713838-<span class="platform">Linux (ARM64):</span> wisp-cli-aarch64-linux
7272+# Delete a domain or site
7373+wispctl domain delete your-handle.bsky.social --domain example.com
7474+wispctl site delete your-handle.bsky.social --site mysite
7575+```
39764040-</a>
7777+```bash
7878+wispctl list domains your-handle.bsky.social
7979+wispctl list sites your-handle.bsky.social
8080+```
41814242-<a href="https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-x86_64-linux" class="download-link" download="">
8282+## Pulling a Site
43834444-<span class="platform">Linux (x86_64):</span> wisp-cli-x86_64-linux
8484+Download a site from the PDS to your local machine:
45854646-</a>
8686+```bash
8787+wispctl pull your-handle.bsky.social --site my-site --path ./my-site
8888+```
47894848-<h3 style="margin-top: 1.5rem; margin-bottom: 0.5rem;">SHA-256 Checksums</h3>
9090+## Local Dev Server
49915050-<pre style="font-size: 0.75rem; padding: 1rem;" class="language-bash" tabindex="0"><code class="language-bash">
5151-06544b3a3e27a4b8d7b3a46a39fb7205cf90b3061e19fe533b090facd604f375 wisp-cli-aarch64-darwin
5252-9ec523e3ceef927b37adc52d449dcd9e13ea84fa49b0b77f0d5932c94cfe262e wisp-cli-x86_64-darwin
5353-42a262668e13dce36173a4096cdc2b22358b805cf192335f84534c7f695d395b wisp-cli-aarch64-linux
5454-589ee59f3959ddfbc12fea38d2bcb91701f1362f560ae6fd506bebea3150e2cc wisp-cli-x86_64-linux
5555-</code></pre>
9292+Serve a site locally with real-time updates from the firehose:
56935757-</div>
9494+```bash
9595+wispctl serve your-handle.bsky.social --site my-site
9696+wispctl serve your-handle.bsky.social --site my-site --port 3000
9797+wispctl serve your-handle.bsky.social --site my-site --spa # serve index.html for all routes
9898+wispctl serve your-handle.bsky.social --site my-site --directory # directory listing
9999+```
581005959-## CI/CD Integration
101101+## CI/CD
6010261103Deploy automatically on every push using Tangled Spindle:
62104···7011271113dependencies:
72114 nixpkgs:
7373- - nodejs
74115 - coreutils
75116 - curl
117117+ - glibc
76118 github:NixOS/nixpkgs/nixpkgs-unstable:
77119 - bun
78120···85127 - name: build site
86128 command: |
87129 export PATH="$HOME/.nix-profile/bin:$PATH"
8888-8989- # you may need to regenerate the lockfile due to nixery being weird
9090- # rm package-lock.json bun.lock
91130 bun install
9292-93131 bun run build
9413295133 - name: deploy to wisp
96134 command: |
9797- # Download Wisp CLI
9898- curl https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-x86_64-linux -o wisp-cli
9999- chmod +x wisp-cli
100100-101101- # Deploy to Wisp
102102- ./wisp-cli \
135135+ curl -fsSL https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-x86_64-linux -o wispctl
136136+ chmod +x wispctl
137137+ ./wispctl deploy \
103138 "$WISP_HANDLE" \
104139 --path "$SITE_PATH" \
105140 --site "$SITE_NAME" \
106141 --password "$WISP_APP_PASSWORD"
107142```
108143109109-**Note:** Set `WISP_APP_PASSWORD` as a secret in your Tangled Spindle repository settings. Generate an app password from your AT Protocol account settings.
110110-111111-## Basic Usage
112112-113113-### Deploy a Site
114114-115115-```bash
116116-# Download and make executable
117117-curl -O https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-aarch64-darwin
118118-chmod +x wisp-cli-aarch64-darwin
144144+Set `WISP_APP_PASSWORD` as a secret in your Tangled Spindle repository settings.
119145120120-# Deploy your site
121121-./wisp-cli-aarch64-darwin deploy your-handle.bsky.social \
122122- --path ./dist \
123123- --site my-site
124124-```
146146+## File Processing
125147126126-Your site will be available at: `https://sites.wisp.place/your-handle/my-site`
148148+Files are gzip-compressed at level 9 and uploaded as `application/octet-stream` blobs with the original MIME type stored in the manifest. They may also be base64-encoded to bypass content sniffing on legacy reference PDS. The hosting service handles decompression transparently.
127149128128-### Domain Management
150150+Common build artifacts like `.git`, `node_modules`, and `.env` are excluded automatically. Customize this with a [`.wispignore` file](/file-filtering).
129151130130-```bash
131131-# Claim a custom domain
132132-./wisp-cli domain claim your-handle.bsky.social --domain example.com
152152+## Limits
133153134134-# Claim a subdomain
135135-./wisp-cli domain claim-subdomain your-handle.bsky.social --subdomain alice
154154+- Max file size: 100 MB (after compression)
155155+- Max total size: 300 MB per site
156156+- Max files: 1,000 per site
157157+- Site name: alphanumeric, hyphens, underscores (AT Protocol rkey format)
136158137137-# Check domain status
138138-./wisp-cli domain status your-handle.bsky.social --domain example.com
159159+## Command Reference
139160140140-# Attach a site to a domain
141141-./wisp-cli domain add-site your-handle.bsky.social --domain example.com --site mysite
161161+### deploy
142162143143-# Delete a domain or site
144144-./wisp-cli domain delete your-handle.bsky.social --domain example.com
145145-./wisp-cli site delete your-handle.bsky.social --site mysite
146163```
164164+wispctl deploy [OPTIONS] <INPUT>
147165148148-### List Domains & Sites
166166+Arguments:
167167+ <INPUT> Handle (e.g., alice.bsky.social), DID, or PDS URL
149168150150-```bash
151151-./wisp-cli list domains your-handle.bsky.social
152152-./wisp-cli list sites your-handle.bsky.social
169169+Options:
170170+ -p, --path <PATH> Path to site directory [default: .]
171171+ -s, --site <SITE> Site name (defaults to directory name)
172172+ --store <STORE> OAuth session file [default: /tmp/wisp-oauth-session.json]
173173+ --password <PASSWORD> App password
153174```
154175155155-### Options
156156-157157-Use an alternate proxy service DID:
176176+### pull
158177159159-```bash
160160-./wisp-cli list domains your-handle.bsky.social --service did:web:example.com
161178```
162162-163163-### Pull a Site from PDS
164164-165165-Download a site from the PDS to your local machine:
179179+wispctl pull [OPTIONS] --site <SITE> <INPUT>
166180167167-```bash
168168-# Pull a site to a specific directory
169169-wisp-cli pull your-handle.bsky.social \
170170- --site my-site \
171171- --path ./my-site
172172-173173-# Pull to current directory
174174-wisp-cli pull your-handle.bsky.social \
175175- --site my-site
176176-```
177177-178178-### Serve a Site Locally with Real-Time Updates
179179-180180-Run a local server that monitors the firehose for real-time updates:
181181-182182-```bash
183183-# Serve on http://localhost:8080 (default)
184184-wisp-cli serve your-handle.bsky.social \
185185- --site my-site
186186-187187-# Serve on a custom port
188188-wisp-cli serve your-handle.bsky.social \
189189- --site my-site \
190190- --port 3000
191191-192192-# Enable SPA mode (serve index.html for all routes)
193193-wisp-cli serve your-handle.bsky.social \
194194- --site my-site \
195195- --spa
181181+Arguments:
182182+ <INPUT> Handle or DID
196183197197-# Enable directory listing for paths without index files
198198-wisp-cli serve your-handle.bsky.social \
199199- --site my-site \
200200- --directory
184184+Options:
185185+ -s, --site <SITE> Site name to download
186186+ -p, --path <PATH> Output directory [default: .]
201187```
202188203203-Downloads site, serves it, and watches firehose for live updates!
204204-205205-## Authentication
206206-207207-### OAuth (Recommended)
208208-209209-The CLI uses OAuth by default, opening your browser for secure authentication:
189189+### serve
210190211211-```bash
212212-wisp-cli deploy your-handle.bsky.social --path ./dist --site my-site
213191```
192192+wispctl serve [OPTIONS] --site <SITE> <INPUT>
214193215215-This creates a session stored locally (default: `/tmp/wisp-oauth-session.json`).
194194+Arguments:
195195+ <INPUT> Handle or DID
216196217217-### App Password
218218-219219-For headless environments or CI/CD, use an app password:
220220-221221-```bash
222222-wisp-cli deploy your-handle.bsky.social \
223223- --path ./dist \
224224- --site my-site \
225225- --password YOUR_APP_PASSWORD
197197+Options:
198198+ -s, --site <SITE> Site name
199199+ -p, --path <PATH> Site files directory [default: .]
200200+ -P, --port <PORT> Port [default: 8080]
201201+ --spa Serve index.html for all routes
202202+ --directory Directory listing for paths without index files
226203```
227204228228-**Generate app passwords** from your AT Protocol account settings.
229229-230230-## File Processing
231231-232232-The CLI handles all file processing automatically to ensure reliable storage and delivery. Files are compressed with gzip at level 9 for optimal size reduction, then base64 encoded to bypass PDS content sniffing restrictions. Everything is uploaded as `application/octet-stream` blobs while preserving the original MIME type as metadata. When serving your site, the hosting service automatically decompresses non-HTML/CSS/JS files, ensuring your content is delivered correctly to visitors.
205205+## Binary Downloads
233206234234-**File Filtering**: The CLI automatically excludes common files like `.git`, `node_modules`, `.env`, and other development artifacts. Customize this with a [`.wispignore` file](/file-filtering).
207207+Pre-built binaries are available if you can't use npm.
235208236236-## Incremental Updates
209209+<div class="downloads">
237210238238-The CLI tracks file changes using CID-based content addressing to minimize upload times and bandwidth usage. On your first deploy, all files are uploaded to establish the initial site. For subsequent deploys, the CLI compares content-addressed CIDs to detect which files have actually changed, uploading only those that differ from the previous version. This makes fast iterations possible even for large sites, with deploys completing in seconds when only a few files have changed.
211211+<h2>Download v1.0.0</h2>
239212240240-## Limits
213213+<a href="https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-aarch64-darwin" class="download-link" download="">
241214242242-- **Max file size**: 100MB per file (after compression)
243243-- **Max total size**: 300MB per site
244244-- **Max files**: 1000 files per site
245245-- **Site name**: Must follow AT Protocol rkey format (alphanumeric, hyphens, underscores)
215215+<span class="platform">macOS (Apple Silicon):</span> wisp-cli-aarch64-darwin
246216247247-## Command Reference
217217+</a>
248218249249-### Deploy Command
219219+<a href="https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-x86_64-darwin" class="download-link" download="">
250220251251-```bash
252252-wisp-cli deploy [OPTIONS] <INPUT>
221221+<span class="platform">macOS (Intel):</span> wisp-cli-x86_64-darwin
253222254254-Arguments:
255255- <INPUT> Handle (e.g., alice.bsky.social), DID, or PDS URL
223223+</a>
256224257257-Options:
258258- -p, --path <PATH> Path to site directory [default: .]
259259- -s, --site <SITE> Site name (defaults to directory name)
260260- --store <STORE> OAuth session file path [default: /tmp/wisp-oauth-session.json]
261261- --password <PASSWORD> App password for authentication
262262- -h, --help Print help
263263-```
225225+<a href="https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-aarch64-linux" class="download-link" download="">
264226265265-### Pull Command
227227+<span class="platform">Linux (ARM64):</span> wisp-cli-aarch64-linux
266228267267-```bash
268268-wisp-cli pull [OPTIONS] --site <SITE> <INPUT>
229229+</a>
269230270270-Arguments:
271271- <INPUT> Handle or DID
231231+<a href="https://sites.wisp.place/nekomimi.pet/wisp-cli-binaries/wisp-cli-x86_64-linux" class="download-link" download="">
272232273273-Options:
274274- -s, --site <SITE> Site name to download
275275- -p, --path <PATH> Output directory [default: .]
276276- -h, --help Print help
277277-```
233233+<span class="platform">Linux (x86_64):</span> wisp-cli-x86_64-linux
278234279279-### Serve Command
235235+</a>
280236281281-```bash
282282-wisp-cli serve [OPTIONS] --site <SITE> <INPUT>
237237+<h3 style="margin-top: 1.5rem; margin-bottom: 0.5rem;">SHA-256 Checksums</h3>
283238284284-Arguments:
285285- <INPUT> Handle or DID
239239+<pre style="font-size: 0.75rem; padding: 1rem;" class="language-bash" tabindex="0"><code class="language-bash">
240240+06544b3a3e27a4b8d7b3a46a39fb7205cf90b3061e19fe533b090facd604f375 wisp-cli-aarch64-darwin
241241+9ec523e3ceef927b37adc52d449dcd9e13ea84fa49b0b77f0d5932c94cfe262e wisp-cli-x86_64-darwin
242242+42a262668e13dce36173a4096cdc2b22358b805cf192335f84534c7f695d395b wisp-cli-aarch64-linux
243243+589ee59f3959ddfbc12fea38d2bcb91701f1362f560ae6fd506bebea3150e2cc wisp-cli-x86_64-linux
244244+</code></pre>
286245287287-Options:
288288- -s, --site <SITE> Site name to serve
289289- -p, --path <PATH> Site files directory [default: .]
290290- -P, --port <PORT> Port to serve on [default: 8080]
291291- --spa Enable SPA mode (serve index.html for all routes)
292292- --directory Enable directory listing mode for paths without index files
293293- -h, --help Print help
294294-```
246246+</div>
295247296296-## Development
248248+## Building from Source
297249298298-The CLI is written in Rust using the Jacquard AT Protocol library. To build from source:
250250+The CLI is written in TypeScript and supports both Node.js and Bun runtimes. Run directly with Bun during development, or build a Node.js-compatible bundle for distribution.
299251300252```bash
301253git clone https://tangled.org/@nekomimi.pet/wisp.place-monorepo
302254cd cli
303303-cargo build --release
304304-```
255255+bun install
305256306306-Built binaries are available in `target/release/`.
257257+# Run directly with Bun
258258+bun run index.ts
307259308308-## Related
309309-310310-- [place.wisp.fs](/lexicons/place-wisp-fs) - Site manifest lexicon
311311-- [place.wisp.subfs](/lexicons/place-wisp-subfs) - Subtree records for large sites
312312-- [AT Protocol](https://atproto.com) - The decentralized protocol powering Wisp
260260+# Build a Node.js bundle (outputs to dist/)
261261+bun run build
262262+node dist/index.js
263263+```
+62-272
docs/src/content/docs/deployment.md
···11---
22-title: Self-Hosting Guide
22+title: Self-Hosting
33description: Deploy your own Wisp.place instance
44---
5566-This guide covers deploying your own Wisp.place instance. Wisp.place consists of three services: the main backend (handles OAuth, uploads, domains), the firehose service (watches the AT Protocol firehose and populates the cache), and the hosting service (serves cached sites). See the [Architecture Guide](/architecture) for a detailed breakdown of how these services work together.
77-88-## Prerequisites
99-1010-- **PostgreSQL** database (14 or newer)
1111-- **Bun** runtime for the main backend and firehose service
1212-- **Node.js** (18+) for the hosting service
1313-- **Caddy** (optional, for custom domain TLS)
1414-- **Domain name** for your instance
1515-- **S3-compatible storage** (optional, recommended for production — Cloudflare R2, MinIO, etc.)
1616-- **Redis** (optional, for real-time cache invalidation between services)
1717-1818-## Architecture Overview
66+Wisp.place consists of three services: the **main backend** handles OAuth, uploads, and domain management; the **firehose service** watches the AT Protocol firehose and populates the cache; the **hosting service** serves cached sites. See [Architecture](/architecture) for how they fit together.
197208```
219┌──────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────────────┐
···3018 └────────┬───────────────┘ └─────────────────────┘
3119 ▼
3220┌─────────────────────────────────────────┐
3333-│ PostgreSQL Database │
3434-│ - User sessions │
2121+│ PostgreSQL │
2222+│ - OAuth sessions + keys │
3523│ - Domain mappings │
3624│ - Site metadata │
3725└─────────────────────────────────────────┘
3826```
39274040-## Database Setup
2828+**You'll need:** PostgreSQL 14+, Bun (main backend + firehose), Node.js 18+ (hosting service), and a domain. S3-compatible storage (Cloudflare R2, MinIO, etc.) and Redis are optional but recommended for production.
41294242-Create a PostgreSQL database for Wisp.place:
3030+## Database
43314432```bash
4533createdb wisp
4634```
47354848-The schema is automatically created on first run. Tables include:
4949-- `oauth_states`, `oauth_sessions`, `oauth_keys` - OAuth flow
5050-- `domains` - Wisp subdomains (*.yourdomain.com)
5151-- `custom_domains` - User custom domains with DNS verification
5252-- `sites` - Site metadata cache
5353-- `cookie_secrets` - Session signing keys
3636+The schema is created automatically on first run.
54375555-## Main Backend Setup
5656-5757-### Environment Variables
5858-5959-Create a `.env` file or set these environment variables:
3838+## Main Backend
60396140```bash
6241# Required
6342DATABASE_URL="postgres://user:password@localhost:5432/wisp"
6464-BASE_DOMAIN="wisp.place" # Your domain (without protocol)
6565-DOMAIN="https://wisp.place" # Full domain with protocol
6666-CLIENT_NAME="Wisp.place" # OAuth client name
4343+BASE_DOMAIN="wisp.place"
4444+DOMAIN="https://wisp.place"
4545+CLIENT_NAME="Wisp.place"
67466847# Optional
6969-NODE_ENV="production" # production or development
7070-PORT="8000" # Default: 8000
4848+NODE_ENV="production"
4949+PORT="8000"
7150```
72517373-### Installation
7474-7552```bash
7676-# Install dependencies
7753bun install
7878-7979-# Development mode (with hot reload)
8080-bun run dev
8181-8282-# Production mode
8383-bun run start
8484-8585-# Or compile to binary
8686-bun run build
8787-./server
8888-```
8989-9090-The backend will:
9191-1. Initialize the database schema
9292-2. Generate OAuth keys (stored in DB)
9393-3. Start DNS verification worker (checks custom domains every 10 minutes)
9494-4. Listen on port 8000
9595-9696-### First-Time Admin Setup
9797-9898-On first run, you'll be prompted to create an admin account:
9999-100100-```
101101-No admin users found. Create one now? (y/n):
5454+bun run start # production
5555+bun run dev # dev with hot reload
5656+bun run build # compile to a binary
10257```
10358104104-Or create manually:
5959+On first run you'll be prompted to create an admin account. You can also run it manually:
1056010661```bash
10762bun run scripts/create-admin.ts
10863```
10964110110-Admin panel is available at `https://yourdomain.com/admin`
6565+Admin panel is at `https://yourdomain.com/admin`.
11166112112-## Firehose Service Setup
113113-114114-The firehose service watches the AT Protocol firehose for site changes and pre-populates the cache. It is **write-only** — it never serves requests to users.
115115-116116-### Environment Variables
6767+## Firehose Service
1176811869```bash
11970# Required
12071DATABASE_URL="postgres://user:password@localhost:5432/wisp"
12172122122-# S3 storage (recommended for production)
7373+# S3 storage (recommended)
12374S3_BUCKET="wisp-sites"
124124-S3_REGION="auto"
7575+S3_REGION="us-east-1"
12576S3_ENDPOINT="https://your-account.r2.cloudflarestorage.com"
126126-S3_ACCESS_KEY_ID="..."
127127-S3_SECRET_ACCESS_KEY="..."
128128-S3_METADATA_BUCKET="wisp-metadata" # Optional, recommended
7777+S3_FORCE_PATH_STYLE="false" # set true for MinIO and most non-AWS endpoints
7878+S3_PREFIX="sites/"
7979+AWS_ACCESS_KEY_ID="..."
8080+AWS_SECRET_ACCESS_KEY="..."
12981130130-# Redis (for notifying hosting service of changes)
8282+# Redis (for notifying the hosting service of changes)
13183REDIS_URL="redis://localhost:6379"
13284133133-# Firehose
134134-FIREHOSE_URL="wss://jetstream2.us-east.bsky.network/subscribe"
135135-FIREHOSE_CONCURRENCY=5 # Max parallel event processing
8585+FIREHOSE_SERVICE="wss://bsky.network"
8686+FIREHOSE_MAX_CONCURRENCY=5
13687137137-# Optional
138138-CACHE_DIR="./cache/sites" # Fallback if S3 not configured
8888+HEALTH_PORT=3001
8989+9090+# Fallback disk path if S3 is not configured
9191+CACHE_DIR="./cache/sites"
13992```
14093141141-### Installation
142142-14394```bash
14495cd firehose-service
145145-146146-# Install dependencies
14796bun install
148148-149149-# Production mode
15097bun run start
151151-152152-# With backfill (one-time bulk sync of all existing sites)
153153-bun run start -- --backfill
9898+bun run start -- --backfill # one-time bulk sync of all existing sites
15499```
155100156156-The firehose service will:
157157-1. Connect to the AT Protocol firehose (Jetstream)
158158-2. Filter for `place.wisp.fs` and `place.wisp.settings` events
159159-3. Download blobs, decompress, and rewrite HTML paths
160160-4. Write files to S3 (or disk)
161161-5. Publish cache invalidation events to Redis
162162-163163-## Hosting Service Setup
164164-165165-The hosting service is a **read-only** CDN that serves cached sites through a three-tier storage system (memory, disk, S3).
166166-167167-### Environment Variables
101101+## Hosting Service
168102169103```bash
170104# Required
171105DATABASE_URL="postgres://user:password@localhost:5432/wisp"
172172-BASE_HOST="wisp.place" # Same as main backend
106106+BASE_HOST="wisp.place"
107107+PORT=3001
173108174109# Tiered storage
175175-HOT_CACHE_SIZE=104857600 # Hot tier: 100 MB (memory, LRU)
176176-HOT_CACHE_COUNT=500 # Max items in hot tier
110110+CACHE_DIR="./cache/sites"
111111+HOT_CACHE_SIZE=104857600 # 100 MB, in-memory LRU
112112+HOT_CACHE_COUNT=500
113113+HOT_CACHE_TTL=60 # seconds
114114+WARM_CACHE_SIZE=10737418240 # 10 GB, disk
115115+WARM_EVICTION_POLICY="lru" # lru, fifo, or size
177116178178-WARM_CACHE_SIZE=10737418240 # Warm tier: 10 GB (disk, LRU)
179179-WARM_EVICTION_POLICY="lru" # lru, fifo, or size
180180-CACHE_DIR="./cache/sites" # Warm tier directory
117117+# Bootstrap hot tier from warm on startup
118118+BOOTSTRAP_HOT_ON_STARTUP=false
119119+BOOTSTRAP_HOT_LIMIT=100
181120182182-# S3 cold tier (same bucket as firehose service, read-only)
121121+# S3 cold tier (same bucket as firehose, read-only)
183122S3_BUCKET="wisp-sites"
184184-S3_REGION="auto"
123123+S3_REGION="us-east-1"
185124S3_ENDPOINT="https://your-account.r2.cloudflarestorage.com"
186186-S3_ACCESS_KEY_ID="..."
187187-S3_SECRET_ACCESS_KEY="..."
188188-S3_METADATA_BUCKET="wisp-metadata"
125125+S3_FORCE_PATH_STYLE="false"
126126+S3_PREFIX="sites/"
127127+AWS_ACCESS_KEY_ID="..."
128128+AWS_SECRET_ACCESS_KEY="..."
189129190190-# Redis (receive cache invalidation from firehose service)
191130REDIS_URL="redis://localhost:6379"
192131193132# Optional
194194-PORT="3001" # Default: 3001
133133+CACHE_ONLY=false # serve from cache only, no PDS fallback
134134+TRACE_REQUESTS=false
195135```
196136197197-### Installation
198198-199137```bash
200138cd hosting-service
201201-202202-# Install dependencies
203139npm install
204204-205205-# Development mode
206206-npm run dev
207207-208208-# Production mode
209140npm run start
210141```
211142212212-The hosting service will:
213213-1. Initialize tiered storage (hot → warm → cold)
214214-2. Subscribe to Redis for cache invalidation events
215215-3. Serve sites on port 3001
216216-217217-### Cache Behavior
218218-219219-Files are cached across three tiers with automatic promotion:
220220-221221-- **Hot (memory):** Fastest, limited by `HOT_CACHE_SIZE`. Evicted on restart.
222222-- **Warm (disk):** Fast local reads at `CACHE_DIR`. Survives restarts.
223223-- **Cold (S3):** Shared source of truth, populated by firehose service.
224224-225225-On a cache miss at all tiers, the hosting service fetches directly from the user's PDS and promotes the file into the appropriate tiers.
226226-227227-**Without S3:** Disk acts as both warm and cold tier. The hosting service still works — it just relies on on-demand fetching instead of pre-populated S3 cache.
143143+## Reverse Proxy
228144229229-## Reverse Proxy Setup
230230-231231-### Caddy Configuration
232232-233233-Caddy handles TLS, on-demand certificates for custom domains, and routing:
145145+Caddy is the recommended reverse proxy — it handles TLS and on-demand certificates for custom domains automatically.
234146235147```
236148{
···239151 }
240152}
241153242242-# Wisp subdomains and DNS hash routing
243154*.dns.wisp.place *.wisp.place {
244155 reverse_proxy localhost:3001
245156}
246157247247-# Main web interface and API
248158wisp.place {
249159 reverse_proxy localhost:8000
250160}
251161252252-# Custom domains (on-demand TLS)
253162https:// {
254163 tls {
255164 on_demand
···258167}
259168```
260169261261-### Nginx Alternative
170170+Nginx works too, but custom domain TLS requires dynamic certificate provisioning that you'll need to manage separately.
262171263172```nginx
264264-# Main backend
265173server {
266174 listen 443 ssl http2;
267175 server_name wisp.place;
268268-269176 ssl_certificate /path/to/cert.pem;
270177 ssl_certificate_key /path/to/key.pem;
271271-272178 location / {
273179 proxy_pass http://localhost:8000;
274180 proxy_set_header Host $host;
···276182 }
277183}
278184279279-# Hosting service
280185server {
281186 listen 443 ssl http2;
282187 server_name *.wisp.place sites.wisp.place;
283283-284188 ssl_certificate /path/to/wildcard-cert.pem;
285189 ssl_certificate_key /path/to/wildcard-key.pem;
286286-287190 location / {
288191 proxy_pass http://localhost:3001;
289192 proxy_set_header Host $host;
···292195}
293196```
294197295295-**Note:** Custom domain TLS requires dynamic certificate provisioning. Caddy's on-demand TLS is the easiest solution.
296296-297297-## OAuth Configuration
298298-299299-Wisp.place uses AT Protocol OAuth. Your instance needs to be publicly accessible for OAuth callbacks.
300300-301301-Required endpoints:
302302-- `/.well-known/atproto-did` - Returns your DID for lexicon resolution
303303-- `/oauth-client-metadata.json` - OAuth client metadata
304304-- `/jwks.json` - OAuth signing keys
305305-306306-These are automatically served by the backend.
307307-308308-## DNS Configuration
309309-310310-For your main domain:
198198+## DNS
311199312200```
313201wisp.place A YOUR_SERVER_IP
···316204sites.wisp.place A YOUR_SERVER_IP
317205```
318206319319-Or use CNAME records if you're behind a CDN:
207207+## OAuth
320208321321-```
322322-wisp.place CNAME your-server.example.com
323323-*.wisp.place CNAME your-server.example.com
324324-```
209209+Your instance needs to be publicly accessible for OAuth callbacks. The backend automatically serves `/.well-known/atproto-did`, `/oauth-client-metadata.json`, and `/jwks.json`.
325210326211## Custom Domain Verification
327212328328-Users can add custom domains via DNS TXT records:
213213+Users add custom domains by creating a DNS TXT record:
329214330215```
331216_wisp.example.com TXT did:plc:abc123xyz...
332217```
333218334334-The DNS verification worker checks these every 10 minutes. Trigger manually:
219219+The verification worker checks every 10 minutes. Trigger it manually:
335220336221```bash
337222curl -X POST https://yourdomain.com/api/admin/verify-dns
338338-```
339339-340340-## Production Checklist
341341-342342-Before going live:
343343-344344-- [ ] PostgreSQL database configured with backups
345345-- [ ] `DATABASE_URL` set with secure credentials
346346-- [ ] `BASE_DOMAIN` and `DOMAIN` configured correctly
347347-- [ ] Admin account created
348348-- [ ] Reverse proxy (Caddy/Nginx) configured
349349-- [ ] DNS records pointing to your server
350350-- [ ] TLS certificates configured
351351-- [ ] Hosting service cache directory has sufficient space
352352-- [ ] Firewall allows ports 80/443
353353-- [ ] Process manager (systemd, pm2) configured for auto-restart
354354-355355-## Monitoring
356356-357357-### Health Checks
358358-359359-Main backend:
360360-```bash
361361-curl https://yourdomain.com/api/health
362362-```
363363-364364-Hosting service:
365365-```bash
366366-curl http://localhost:3001/health
367367-```
368368-369369-### Logs
370370-371371-The services log to stdout. View with your process manager:
372372-373373-```bash
374374-# systemd
375375-journalctl -u wisp-backend -f
376376-journalctl -u wisp-hosting -f
377377-378378-# pm2
379379-pm2 logs wisp-backend
380380-pm2 logs wisp-hosting
381381-```
382382-383383-### Admin Panel
384384-385385-Access observability metrics at `https://yourdomain.com/admin`:
386386-- Recent logs
387387-- Error tracking
388388-- Performance metrics
389389-- Cache statistics
390390-391391-## Scaling Considerations
392392-393393-- **Multiple hosting instances**: Run multiple hosting services behind a load balancer — each has its own hot/warm tiers but shares the S3 cold tier and Redis invalidation
394394-- **Separate databases**: Split read/write with replicas
395395-- **CDN**: Put Cloudflare or Bunny in front for global caching
396396-- **S3 cold tier**: Shared storage across all hosting instances (Cloudflare R2, MinIO, AWS S3)
397397-- **Redis**: Required for real-time cache invalidation between firehose and hosting services at scale
398398-399399-## Security Notes
400400-401401-- Use strong cookie secrets (auto-generated and stored in DB)
402402-- Keep dependencies updated: `bun update`, `npm update`
403403-- Enable rate limiting in reverse proxy
404404-- Set up fail2ban for brute force protection
405405-- Regular database backups
406406-- Monitor logs for suspicious activity
407407-408408-## Updates
409409-410410-To update your instance:
411411-412412-```bash
413413-# Pull latest code
414414-git pull
415415-416416-# Update dependencies
417417-bun install
418418-cd hosting-service && npm install && cd ..
419419-420420-# Restart services
421421-# (The database schema updates automatically)
422422-```
423423-424424-## Support
425425-426426-For issues and questions:
427427-- Check the [documentation](https://docs.wisp.place)
428428-- Review [Tangled issues](https://tangled.org/nekomimi.pet/wisp.place-monorepo)
429429-- Join the [Bluesky community](https://bsky.app)
430430-431431-## License
432432-433433-Wisp.place is MIT licensed. You're free to host your own instance and modify it as needed.
223223+```
+6-41
docs/src/content/docs/file-filtering.md
···33description: Control which files are uploaded to your Wisp site
44---
5566-# File Filtering & .wispignore
77-88-Wisp automatically excludes common files that shouldn't be deployed (`.git`, `node_modules`, `.env`, etc.).
99-1010-## Default Exclusions
1111-1212-- Version control: `.git`, `.github`, `.gitlab`
1313-- Dependencies: `node_modules`, `__pycache__`, `*.pyc`
1414-- Secrets: `.env`, `.env.*`
1515-- OS files: `.DS_Store`, `Thumbs.db`, `._*`
1616-- Cache: `.cache`, `.temp`, `.tmp`
1717-- Dev tools: `.vscode`, `*.swp`, `*~`, `.tangled`
1818-- Virtual envs: `.venv`, `venv`, `env`
1919-2020-## Custom Patterns
66+Wisp automatically excludes common files that shouldn't be deployed — version control (`.git`, `.github`), dependencies (`node_modules`, `__pycache__`), secrets (`.env`, `.env.*`), OS files (`.DS_Store`, `Thumbs.db`), caches, and dev tooling.
2172222-Create a `.wispignore` file in your site root using gitignore syntax:
88+To exclude additional files, create a `.wispignore` in your site root using gitignore syntax:
2392410```
2525-# Build outputs
1111+# Build artifacts
2612dist/
2713*.map
28142929-# Logs and temp files
1515+# Logs
3016*.log
3117temp/
32183333-# Keep one exception
1919+# Keep a specific file
3420!important.log
3521```
36223737-### Pattern Syntax
3838-3939-- `file.txt` - exact match
4040-- `*.log` - wildcard
4141-- `logs/` - directory
4242-- `src/**/*.test.js` - glob pattern
4343-- `!keep.txt` - exception (don't ignore)
4444-4545-## Usage
4646-4747-**CLI**: Place `.wispignore` in your upload directory
4848-```bash
4949-wisp-cli handle.bsky.social --path ./my-site --site my-site
5050-```
5151-5252-**Web**: Include `.wispignore` when uploading files
5353-5454-## Notes
5555-5656-- Custom patterns add to (not replace) default patterns
5757-- Works in both CLI and web uploads
5858-- The CLI logs which files are skipped
2323+Custom patterns are added on top of the defaults, not in place of them. The CLI logs which files are skipped.
-11
docs/src/content/docs/guides/example.md
···11----
22-title: Example Guide
33-description: A guide in my new Starlight docs site.
44----
55-66-Guides lead a user through a specific task they want to accomplish, often with a sequence of steps.
77-Writing a good guide requires thinking about what your users are trying to do.
88-99-## Further reading
1010-1111-- Read [about how-to guides](https://diataxis.fr/how-to-guides/) in the Diátaxis framework
+1-143
docs/src/content/docs/index.mdx
···2626Your site will be available at:
2727```
2828https://sites.wisp.place/{your-did}/{site-name}
2929-```
3030-3131-## Key Features
3232-3333-### Decentralized Storage
3434-- Sites stored as `place.wisp.fs` records in your AT Protocol repo
3535-- Cryptographically verifiable ownership
3636-- Your PDS is the source of truth
3737-- Portable across hosting providers
3838-3939-### Custom Domains
4040-- Point your own domain to your Wisp site
4141-- DNS verification ensures secure ownership
4242-- Automatic SSL/TLS certificates
4343-4444-### URL Redirects & Rewrites
4545-- Netlify-style `_redirects` file support
4646-- Single-page app (SPA) routing
4747-- API proxying and conditional routing
4848-- Custom 404 pages
4949-5050-### Efficient Deployment
5151-- **Incremental updates**: Only upload changed files
5252-- **Smart compression**: Automatic gzip for text files
5353-- **Large site support**: Automatic splitting into subfs records for sites with 250+ files to get around 150KB record size limit
5454-- **Blob reuse**: Content-addressed storage prevents duplicate uploads
5555-5656-## How It Works
5757-5858-The deployment process starts when you upload your files. Each file is compressed with gzip, base64-encoded, and uploaded as a blob to your PDS. A `place.wisp.fs` record then stores the complete site structure with references to these blobs, creating a verifiable manifest of your site.
5959-6060-The **firehose service** continuously watches the AT Protocol firehose for new and updated sites. When a site is created or updated, it downloads the manifest and blobs from the PDS, writes them to S3 (or disk), and publishes a cache invalidation event via Redis. The **hosting service** is a read-only CDN that serves files from a three-tier cache (memory, disk, S3). When a file isn't in cache, the hosting service fetches it on-demand from the PDS and promotes it through the tiers.
6161-6262-Custom domains work through DNS verification, allowing your site to be served from your own domain while maintaining the cryptographic guarantees of the AT Protocol.
6363-6464-## Architecture Overview
6565-6666-```
6767- ┌──────────────┐ ┌──────────────┐
6868- │ wisp-cli │ │ wisp.place │
6969- │ (Rust Binary)│ │ Website │
7070- │ │ │ (React UI) │
7171- └──────────────┘ └──────────────┘
7272- │ │
7373- ▼ ▼
7474-┌─────────────────────────────────────────────────────────┐
7575-│ AT Protocol PDS │
7676-│ (Authoritative Source - Cryptographically Signed) │
7777-│ │
7878-│ ┌──────────────────────────────────────────────┐ │
7979-│ │ place.wisp.fs record │ │
8080-│ │ - Site manifest (directory tree) │ │
8181-│ │ - Blob references (CID-based) │ │
8282-│ │ - Metadata (file count, timestamps) │ │
8383-│ └──────────────────────────────────────────────┘ │
8484-│ │
8585-│ ┌──────────────────────────────────────────────┐ │
8686-│ │ Blobs (gzipped + base64 encoded) │ │
8787-│ │ - index.html, styles.css, assets/* │ │
8888-│ └──────────────────────────────────────────────┘ │
8989-└─────────────────────────────────────────────────────────┘
9090- │
9191- ▼
9292- ┌─────────────────────────────────┐
9393- │ AT Protocol Firehose │
9494- │ (Jetstream WebSocket Stream) │
9595- └─────────────────────────────────┘
9696- │
9797- ▼
9898-┌─────────────────────────────────────────────────────────┐
9999-│ Firehose Service (Write Path) │
100100-│ │
101101-│ - Watches firehose for place.wisp.fs changes │
102102-│ - Downloads blobs from PDS │
103103-│ - Writes cached files to S3 / disk │
104104-│ - Publishes cache invalidation via Redis │
105105-└─────────────────────────────────────────────────────────┘
106106- │ │
107107- │ (S3 / Disk) │ (Redis pub/sub)
108108- ▼ ▼
109109-┌─────────────────────────────────────────────────────────┐
110110-│ Hosting Service (Read Path) │
111111-│ │
112112-│ ┌──────────────────────────────────────────────┐ │
113113-│ │ Tiered Storage │ │
114114-│ │ ┌──────┐ ┌──────┐ ┌──────────────┐ │ │
115115-│ │ │ Hot │ → │ Warm │ → │ Cold │ │ │
116116-│ │ │(Mem) │ │(Disk)│ │(S3/Disk) │ │ │
117117-│ │ └──────┘ └──────┘ └──────────────┘ │ │
118118-│ │ On miss: fetch from PDS and promote up │ │
119119-│ └──────────────────────────────────────────────┘ │
120120-│ │
121121-│ ┌──────────────────────────────────────────────┐ │
122122-│ │ Routing & Serving │ │
123123-│ │ - Custom domains (example.com) │ │
124124-│ │ - Subdomains (alice.wisp.place) │ │
125125-│ │ - Direct URLs (sites.wisp.place/did/site) │ │
126126-│ └──────────────────────────────────────────────┘ │
127127-└─────────────────────────────────────────────────────────┘
128128- │
129129- ▼
130130- ┌─────────────┐
131131- │ Browser │
132132- └─────────────┘
133133-```
134134-135135-For a detailed breakdown of the services and storage system, see the [Architecture Guide](/architecture).
136136-137137-## Tech Stack
138138-139139-- **Backend**: Bun + Elysia + PostgreSQL
140140-- **Frontend**: React 19 + Tailwind 4 + Radix UI
141141-- **Hosting Service**: Node.js + Hono
142142-- **Firehose Service**: Bun
143143-- **CLI**: Rust + Jacquard (AT Protocol library)
144144-- **Protocol**: AT Protocol OAuth + custom lexicons
145145-- **Storage**: S3-compatible (Cloudflare R2, MinIO, etc.) + Redis for cache invalidation
146146-147147-## Limits
148148-149149-- **Max file size**: 100MB per file (PDS limit)
150150-- **Max files**: 1000 files per site
151151-- **Max total size**: 300MB per site (compressed)
152152-153153-Files are automatically compressed with gzip before upload, so actual limits may be higher depending on your content compressibility.
154154-155155-## Getting Started
156156-157157-- [CLI Documentation](/cli) - Deploy sites from the command line
158158-- [Architecture Guide](/architecture) - How hosting, firehose, and tiered storage work
159159-- [Self-Hosting Guide](/deployment) - Deploy your own instance
160160-- [Lexicons](/lexicons) - AT Protocol record schemas and data structures
161161-162162-## Links
163163-164164-- **Website**: [https://wisp.place](https://wisp.place)
165165-- **Repository**: [https://tangled.org/@nekomimi.pet/wisp.place-monorepo](https://tangled.org/@nekomimi.pet/wisp.place-monorepo)
166166-- **AT Protocol**: [https://atproto.com](https://atproto.com)
167167-- **Jacquard Library**: [https://tangled.org/@nonbinary.computer/jacquard](https://tangled.org/@nonbinary.computer/jacquard)
168168-169169-## License
170170-171171-MIT License - See the repository for details.
2929+```
+9-46
docs/src/content/docs/lexicons/index.md
···33description: AT Protocol lexicons used by Wisp.place
44---
5566-Wisp.place uses custom AT Protocol lexicons to store and manage static site data. These lexicons define the structure of records stored in your PDS.
66+Wisp.place uses three custom AT Protocol lexicons to store site data in your PDS.
7788-## Available Lexicons
99-1010-### [place.wisp.fs](/lexicons/place-wisp-fs)
1111-The main lexicon for storing static site manifests. Contains the directory tree structure with references to file blobs.
88+**[place.wisp.fs](/lexicons/place-wisp-fs)** — the main site manifest. Stores the full directory tree with references to file blobs.
1291313-### [place.wisp.subfs](/lexicons/place-wisp-subfs)
1414-Subtree lexicon for splitting large sites across multiple records. Entries from subfs records are merged (flattened) into the parent directory.
1010+**[place.wisp.subfs](/lexicons/place-wisp-subfs)** — subtree records for splitting large sites across multiple records. Entries from subfs records are merged into the parent directory.
15111616-### [place.wisp.domain](/lexicons/place-wisp-domain)
1717-Domain registration record for claiming wisp.place subdomains.
1212+**[place.wisp.domain](/lexicons/place-wisp-domain)** — metadata record for claiming a wisp.place subdomain.
18131919-## How Lexicons Work
1414+**[place.wisp.v2.wh](/lexicons/place-wisp-wh)** — webhook record for receiving HTTP callbacks when AT Protocol records change.
20152121-### Storage Model
1616+## Storage Model
22172318Sites are stored as `place.wisp.fs` records in your AT Protocol repository:
2419···2621at://did:plc:abc123/place.wisp.fs/my-site
2722```
28232929-Each record contains:
3030-- **Site metadata** (name, file count, timestamps)
3131-- **Directory tree** (hierarchical structure)
3232-- **Blob references** (content-addressed file storage)
2424+Files are gzipped for compression and uploaded as `application/octet-stream` blobs. They may also be base64-encoded to bypass content sniffing on legacy reference PDS. The original MIME type is preserved in the manifest.
33253434-### File Processing
2626+Sites with 250+ files are automatically split: large directories are extracted into `place.wisp.subfs` records, referenced by AT-URI from the main manifest, and merged back together by the hosting service at serve time. This keeps the main manifest under the 150 KB PDS record size limit.
35273636-1. Files are **gzipped** for compression
3737-2. Text files are **base64 encoded** to bypass PDS content sniffing
3838-3. Uploaded as blobs with `application/octet-stream` MIME type
3939-4. Original MIME type stored in manifest metadata
4040-4141-### Large Site Splitting
4242-4343-Sites with 250+ files are automatically split:
4444-4545-1. Large directories are extracted into `place.wisp.subfs` records
4646-2. Main manifest references subfs records via AT-URI
4747-3. Hosting services merge (flatten) subfs entries when serving
4848-4. Keeps manifest size under 150KB PDS limit
4949-5050-## Example Record Structure
2828+## Example Record
51295230```json
5331{
···7048 "mimeType": "text/html",
7149 "base64": true
7250 }
7373- },
7474- {
7575- "name": "assets",
7676- "node": {
7777- "type": "directory",
7878- "entries": [...]
7979- }
8051 }
8152 ]
8253 },
···8455 "createdAt": "2024-01-15T10:30:00Z"
8556}
8657```
8787-8888-## Learn More
8989-9090-- [place.wisp.fs Reference](/lexicons/place-wisp-fs)
9191-- [place.wisp.subfs Reference](/lexicons/place-wisp-subfs)
9292-- [place.wisp.domain Reference](/lexicons/place-wisp-domain)
9393-- [AT Protocol Lexicons](https://atproto.com/specs/lexicon)
9494-
···33description: Reference for the place.wisp.domain lexicon
44---
5566-**Lexicon Version:** 1
77-88-## Overview
66+Metadata record for a claimed wisp.place subdomain (e.g. `alice.wisp.place`). The record lives in the user's PDS as an audit trail — actual routing decisions use the PostgreSQL `domains` table, not this record.
971010-The `place.wisp.domain` lexicon defines **metadata records for wisp.place subdomains**,
1111-such as `alice.wisp.place` or `miku-fan.wisp.place`.
1212-1313-- **What lives in the PDS:** a small record that says “this DID claimed this domain at this time”.
1414-- **What is authoritative:** the PostgreSQL `domains` table on the wisp.place backend
1515- (routing and availability checks use the DB, not this record).
1616-1717-Use this page as a schema reference; routing and TLS details are covered elsewhere.
1818-1919----
2020-2121-## Record: `main`
2222-2323-<a name="main"></a>
2424-2525-### `main` (record)
88+**Lexicon version:** 1
2692727-**Type:** `record`
1010+## main (record)
28112929-**Description:** Metadata record for a claimed wisp.place subdomain.
1212+| Property | Type | Required | Constraints |
1313+| --- | --- | --- | --- |
1414+| `domain` | `string` | ✅ | Full domain, e.g. `alice.wisp.place` |
1515+| `createdAt` | `string` | ✅ | Format: `datetime` |
30163131-**Properties:**
1717+## Record Key
32183333-| Name | Type | Req'd | Description | Constraints |
3434-| ----------- | -------- | ----- | --------------------------------------------- | ------------------ |
3535-| `domain` | `string` | ✅ | Full domain name, e.g. `alice.wisp.place` | |
3636-| `createdAt` | `string` | ✅ | When the domain was claimed | Format: `datetime` |
1919+The record key is the subdomain label (the part before `.wisp.place`). A DID with multiple subdomains has multiple records:
37203838----
2121+```
2222+at://did:plc:abc123/place.wisp.domain/alice
2323+at://did:plc:abc123/place.wisp.domain/miku-fan
2424+```
39254040-## Claim Flow & `rkey`
4141-4242-### Subdomain claiming (high‑level)
2626+## Claim Flow
43274428When a user claims `handle.wisp.place`:
45294646-1. **User authenticates** via OAuth (proves DID control).
4747-2. **Handle is validated**:
4848- - 3–63 characters
4949- - `a-z`, `0-9`, `-` only
5050- - Does not start/end with `-`
5151- - Not in the reserved set (`www`, `api`, `admin`, `static`, `public`, `preview`, …)
5252-3. **Domain limit enforced:** max 3 wisp.place subdomains per DID.
5353-4. **Database row created** in `domains`:
3030+1. User authenticates via OAuth (proves DID control)
3131+2. Handle is validated (3–63 chars, `a-z0-9-`, no leading/trailing hyphen, not reserved)
3232+3. Domain limit checked: max 3 wisp.place subdomains per DID
3333+4. Database row inserted in `domains`
3434+5. `place.wisp.domain` record written to PDS
54355555-```sql
5656-INSERT INTO domains (domain, did, rkey)
5757-VALUES ('handle.wisp.place', did, NULL);
5858-```
3636+Reserved handles include `www`, `api`, `admin`, `static`, `public`, `preview`, and others.
59376060-5. **PDS record written** in `place.wisp.domain` as metadata.
3838+## Domain Rules
61396262-### Record key (`rkey`)
6363-6464-The **record key is the normalized handle** (the subdomain label):
6565-6666-```text
6767-at://did:plc:abc123/place.wisp.domain/wisp
6868-```
6969-7070-If a DID claims multiple subdomains, it will have multiple records:
7171-7272-- `at://did:plc:abc123/place.wisp.domain/wisp`
7373-- `at://did:plc:abc123/place.wisp.domain/miku-fan`
7474-7575----
4040+- Length: 3–63 characters
4141+- Characters: `a-z`, `0-9`, `-`
4242+- Must start and end with alphanumeric
4343+- Stored and compared in lowercase
4444+- Max 3 per DID
4545+- Each subdomain can only be owned by one DID
76467777-## Examples
4747+Valid: `alice`, `my-site`, `dev2024`
4848+Invalid: `ab` (too short), `-alice` (leading hyphen), `alice.bob` (dot), `alice_bob` (underscore)
78497979-### Basic domain record
5050+## Example
80518152```json
8253{
···8657}
8758```
88598989-### URI structure
9090-9191-Complete AT-URI for a domain record:
9292-9393-```text
9494-at://did:plc:7puq73yz2hkvbcpdhnsze2qw/place.wisp.domain/wisp
9595-```
9696-9797-Breakdown:
9898-9999-- **`did:plc:7puq73yz2hkvbcpdhnsze2qw`** – User DID
100100-- **`place.wisp.domain`** – Collection ID
101101-- **`wisp`** – Record key (subdomain handle)
102102-103103----
6060+## Database Schema
10461105105-## Domain rules (summary)
106106-107107-- **Length:** 3–64 characters
108108-- **Characters:** `a-z`, `0-9`, and `-`
109109-- **Shape:** must start and end with alphanumeric
110110-- **Case:** stored/compared in lowercase
111111-- **Limit:** up to **3** wisp.place subdomains per DID
112112-- **Uniqueness:** each `*.wisp.place` can only be owned by one DID at a time.
113113-114114-Valid examples:
115115-116116-- `alice` → `alice.wisp.place`
117117-- `my-site` → `my-site.wisp.place`
118118-- `dev2024` → `dev2024.wisp.place`
119119-120120-Invalid examples:
121121-122122-- `ab` (too short)
123123-- `-alice` / `alice-` (leading or trailing hyphen)
124124-- `alice.bob` (dot)
125125-- `alice_bob` (underscore)
126126-127127----
128128-129129-## Database & routing
130130-131131-The **lexicon record is not used for routing**. All real decisions use the DB:
6262+Routing and availability checks use the DB, not the PDS record:
1326313364```sql
13465CREATE TABLE domains (
13566 domain TEXT PRIMARY KEY, -- "alice.wisp.place"
136136- did TEXT NOT NULL, -- User DID
137137- rkey TEXT, -- Site rkey (place.wisp.fs)
6767+ did TEXT NOT NULL,
6868+ rkey TEXT, -- site rkey (place.wisp.fs)
13869 created_at BIGINT DEFAULT EXTRACT(EPOCH FROM NOW())
13970);
140140-141141-CREATE INDEX domains_did_rkey ON domains (did, rkey);
14271```
14372144144-The `domains` table powers:
145145-146146-- **Availability checks** (`/api/domain/check`)
147147-- **Mapping** from hostname → `(did, rkey)` for the hosting service
148148-- **Safety:** you must explicitly change/delete routing in the DB, avoiding accidental takeovers.
149149-150150-The `place.wisp.domain` PDS record is there for:
151151-152152-- **Audit trail** – who claimed what, when
153153-- **User-visible history** in their repo
154154-- **Optional cross‑checking** against the DB if you care to.
155155-156156----
157157-158158-## Related
159159-160160-- [place.wisp.fs](/lexicons/place-wisp-fs) – Site manifest lexicon
161161-- Custom domains / DNS verification – covered in separate routing/hosting docs
162162-- [AT Protocol Lexicons](https://atproto.com/specs/lexicon)
163163-164164-## Additional commentary
165165-166166-For a detailed write‑up of the full domain system (subdomains, custom domains, DNS TXT/CNAME flow, Caddy on‑demand TLS, and hosting routing), see
167167-**[How wisp.place maps domains to DIDs](https://nekomimi.leaflet.pub/3m5fy2jkurk2a)**.
168168-169169-7373+For a detailed write-up of the full domain system including custom domains, DNS verification, and Caddy on-demand TLS, see [How wisp.place maps domains to DIDs](https://nekomimi.leaflet.pub/3m5fy2jkurk2a).
+51-174
docs/src/content/docs/lexicons/place-wisp-fs.md
···33description: Reference for the place.wisp.fs lexicon
44---
5566-**Lexicon Version:** 1
66+The main lexicon for storing static site manifests. Each record represents a complete website with its directory tree and file blob references.
7788-## Overview
88+**Lexicon version:** 1
991010-The `place.wisp.fs` lexicon defines the structure for storing static site manifests in AT Protocol repositories. Each record represents a complete website with its directory structure and file references.
1010+## main (record)
11111212-## Record Structure
1212+Virtual filesystem manifest for a Wisp site.
13131414-<a name="main"></a>
1414+| Property | Type | Required | Constraints |
1515+| --- | --- | --- | --- |
1616+| `site` | `string` | ✅ | Site name (used as record key) |
1717+| `root` | [`#directory`](#directory) | ✅ | Root directory |
1818+| `fileCount` | `integer` | | Min: 0, Max: 1000 |
1919+| `createdAt` | `string` | ✅ | Format: `datetime` |
15201616-### `main` (Record)
2121+## entry
17221818-**Type:** `record`
2323+A named entry in a directory.
19242020-**Description:** Virtual filesystem manifest for a Wisp site
2525+| Property | Type | Required | Constraints |
2626+| --- | --- | --- | --- |
2727+| `name` | `string` | ✅ | Max: 255 chars |
2828+| `node` | Union of [`#file`](#file), [`#directory`](#directory), [`#subfs`](#subfs) | ✅ | |
21292222-**Properties:**
3030+## file
23312424-| Name | Type | Req'd | Description | Constraints |
2525-| ----------- | --------------------------- | ----- | ------------------------------------ | ------------------ |
2626-| `site` | `string` | ✅ | Site name (used as record key) | |
2727-| `root` | [`#directory`](#directory) | ✅ | Root directory of the site | |
2828-| `fileCount` | `integer` | | Total number of files in the site | Min: 0, Max: 1000 |
2929-| `createdAt` | `string` | ✅ | Timestamp of site creation/update | Format: `datetime` |
3232+| Property | Type | Required | Constraints |
3333+| --- | --- | --- | --- |
3434+| `type` | `string` | ✅ | Const: `"file"` |
3535+| `blob` | `blob` | ✅ | Max size: 1 GB |
3636+| `encoding` | `string` | | Enum: `["gzip"]` |
3737+| `mimeType` | `string` | | Original MIME type before compression |
3838+| `base64` | `boolean` | | True if blob is base64-encoded |
30393131----
4040+Files are gzip-compressed before upload and uploaded as `application/octet-stream`. They may also be base64-encoded to bypass content sniffing on legacy reference PDS. The original MIME type is stored in `mimeType`.
32413333-<a name="entry"></a>
4242+## directory
34433535-### `entry`
4444+| Property | Type | Required | Constraints |
4545+| --- | --- | --- | --- |
4646+| `type` | `string` | ✅ | Const: `"directory"` |
4747+| `entries` | Array of [`#entry`](#entry) | ✅ | Max: 500 entries |
36483737-**Type:** `object`
4949+## subfs
38503939-**Description:** Named entry in a directory (file, directory, or subfs)
5151+Reference to a `place.wisp.subfs` record for splitting large directories.
40524141-**Properties:**
4242-4343-| Name | Type | Req'd | Description | Constraints |
4444-| ------ | ------------------------------------------------------- | ----- | ----------------------------------- | ----------- |
4545-| `name` | `string` | ✅ | File or directory name | Max: 255 chars |
4646-| `node` | Union of [`#file`](#file), [`#directory`](#directory), [`#subfs`](#subfs) | ✅ | The node (file, directory, or subfs reference) | |
4747-4848----
4949-5050-## Type Definitions
5151-5252-<a name="file"></a>
5353-5454-### `file`
5555-5656-**Type:** `object`
5757-5858-**Description:** Represents a file node in the directory tree
5959-6060-**Properties:**
6161-6262-| Name | Type | Req'd | Description | Constraints |
6363-| ---------- | --------- | ----- | ------------------------------------------------------------ | ---------------------- |
6464-| `type` | `string` | ✅ | Node type identifier | Const: `"file"` |
6565-| `blob` | `blob` | ✅ | Content blob reference | Max size: 1000000000 (1GB) |
6666-| `encoding` | `string` | | Content encoding (e.g., gzip for compressed files) | Enum: `["gzip"]` |
6767-| `mimeType` | `string` | | Original MIME type before compression | |
6868-| `base64` | `boolean` | | True if blob content is base64-encoded (bypasses PDS sniffing) | |
5353+| Property | Type | Required | Constraints |
5454+| --- | --- | --- | --- |
5555+| `type` | `string` | ✅ | Const: `"subfs"` |
5656+| `subject` | `string` | ✅ | AT-URI to a `place.wisp.subfs` record |
5757+| `flat` | `boolean` | | Default: `true` |
69587070-**Notes:**
7171-- Files are typically gzip compressed before upload
7272-- Text files (HTML/CSS/JS) are also base64 encoded to prevent PDS content-type sniffing
7373-- The blob is uploaded with MIME type `application/octet-stream`
7474-- Original MIME type is preserved in the `mimeType` field
7575-7676----
7777-7878-<a name="directory"></a>
7979-8080-### `directory`
8181-8282-**Type:** `object`
8383-8484-**Description:** Represents a directory node in the file tree
8585-8686-**Properties:**
5959+When `flat` is true (default), the subfs record's entries are merged directly into the parent directory. When `flat` is false, entries are placed in a subdirectory named after the subfs entry. Used automatically when sites exceed 250 files or 140 KB.
87608888-| Name | Type | Req'd | Description | Constraints |
8989-| --------- | -------------------------- | ----- | ------------------------------ | ----------- |
9090-| `type` | `string` | ✅ | Node type identifier | Const: `"directory"` |
9191-| `entries` | Array of [`#entry`](#entry) | ✅ | Child entries in this directory | Max: 500 entries |
6161+## Examples
92629393-**Notes:**
9494-- Directories can contain files, subdirectories, or subfs references
9595-- Maximum 500 entries per directory to stay within record size limits
9696-9797-<a name="subfs"></a>
9898-9999-### `subfs`
100100-101101-**Type:** `object`
102102-103103-**Description:** Reference to a `place.wisp.subfs` record for splitting large directories
104104-105105-**Properties:**
106106-107107-| Name | Type | Req'd | Description | Constraints |
108108-| --------- | -------- | ----- | --------------------------------------------------------------------------- | ---------------- |
109109-| `type` | `string` | ✅ | Node type identifier | Const: `"subfs"` |
110110-| `subject` | `string` | ✅ | AT-URI pointing to a place.wisp.subfs record containing this subtree | Format: `at-uri` |
111111-| `flat` | `boolean` | | Controls merging behavior (default: true) | |
112112-113113-**Notes:**
114114-- When `flat` is true (default), the subfs record's root entries are **merged (flattened)** into the parent directory
115115-- When `flat` is false, the subfs entries are placed in a subdirectory with the subfs entry's name
116116-- The `flat` property controls whether the subfs acts as a content merge or directory replacement
117117-- Allows splitting large directories across multiple records while optionally maintaining flat or nested structure
118118-- Used automatically when sites exceed 250 files or 140KB manifest size
119119-120120----
121121-122122-## Usage Examples
123123-124124-### Simple Site
6363+### Simple site
1256412665```json
12766{
···13473 "name": "index.html",
13574 "node": {
13675 "type": "file",
137137- "blob": {
138138- "$type": "blob",
139139- "ref": { "$link": "bafyreiabc..." },
140140- "mimeType": "application/octet-stream",
141141- "size": 4521
142142- },
7676+ "blob": { "$type": "blob", "ref": { "$link": "bafyreiabc..." }, "mimeType": "application/octet-stream", "size": 4521 },
14377 "encoding": "gzip",
14478 "mimeType": "text/html",
14579 "base64": true
14680 }
147147- },
148148- {
149149- "name": "style.css",
150150- "node": {
151151- "type": "file",
152152- "blob": {
153153- "$type": "blob",
154154- "ref": { "$link": "bafyreidef..." },
155155- "mimeType": "application/octet-stream",
156156- "size": 2134
157157- },
158158- "encoding": "gzip",
159159- "mimeType": "text/css",
160160- "base64": true
161161- }
16281 }
16382 ]
16483 },
165165- "fileCount": 2,
166166- "createdAt": "2024-01-15T10:30:00.000Z"
167167-}
168168-```
169169-170170-### Site with Subdirectory
171171-172172-```json
173173-{
174174- "$type": "place.wisp.fs",
175175- "site": "portfolio",
176176- "root": {
177177- "type": "directory",
178178- "entries": [
179179- {
180180- "name": "index.html",
181181- "node": { "type": "file", "blob": {...}, "encoding": "gzip", "mimeType": "text/html", "base64": true }
182182- },
183183- {
184184- "name": "assets",
185185- "node": {
186186- "type": "directory",
187187- "entries": [
188188- {
189189- "name": "logo.png",
190190- "node": { "type": "file", "blob": {...}, "encoding": "gzip", "mimeType": "image/png", "base64": false }
191191- }
192192- ]
193193- }
194194- }
195195- ]
196196- },
197197- "fileCount": 2,
8484+ "fileCount": 1,
19885 "createdAt": "2024-01-15T10:30:00.000Z"
19986}
20087```
20188202202-### Large Site with Subfs
8989+### Large site with subfs
2039020491```json
20592{
···21097 "entries": [
21198 {
21299 "name": "index.html",
213213- "node": { "type": "file", "blob": {...}, "encoding": "gzip", "mimeType": "text/html", "base64": true }
100100+ "node": { "type": "file", "blob": { ... }, "encoding": "gzip", "mimeType": "text/html", "base64": true }
214101 },
215102 {
216103 "name": "docs",
···253140 "required": ["type", "blob"],
254141 "properties": {
255142 "type": { "type": "string", "const": "file" },
256256- "blob": { "type": "blob", "accept": ["*/*"], "maxSize": 1000000000, "description": "Content blob ref" },
257257- "encoding": { "type": "string", "enum": ["gzip"], "description": "Content encoding (e.g., gzip for compressed files)" },
258258- "mimeType": { "type": "string", "description": "Original MIME type before compression" },
259259- "base64": { "type": "boolean", "description": "True if blob content is base64-encoded (used to bypass PDS content sniffing)" }
143143+ "blob": { "type": "blob", "accept": ["*/*"], "maxSize": 1000000000 },
144144+ "encoding": { "type": "string", "enum": ["gzip"] },
145145+ "mimeType": { "type": "string" },
146146+ "base64": { "type": "boolean" }
260147 }
261148 },
262149 "directory": {
···264151 "required": ["type", "entries"],
265152 "properties": {
266153 "type": { "type": "string", "const": "directory" },
267267- "entries": {
268268- "type": "array",
269269- "maxLength": 500,
270270- "items": { "type": "ref", "ref": "#entry" }
271271- }
154154+ "entries": { "type": "array", "maxLength": 500, "items": { "type": "ref", "ref": "#entry" } }
272155 }
273156 },
274157 "entry": {
···284167 "required": ["type", "subject"],
285168 "properties": {
286169 "type": { "type": "string", "const": "subfs" },
287287- "subject": { "type": "string", "format": "at-uri", "description": "AT-URI pointing to a place.wisp.subfs record containing this subtree." },
288288- "flat": { "type": "boolean", "description": "If true (default), the subfs record's root entries are merged (flattened) into the parent directory, replacing the subfs entry. If false, the subfs entries are placed in a subdirectory with the subfs entry's name. Flat merging is useful for splitting large directories across multiple records while maintaining a flat structure." }
170170+ "subject": { "type": "string", "format": "at-uri" },
171171+ "flat": { "type": "boolean" }
289172 }
290173 }
291174 }
292175}
293176```
294294-295295-## Related
296296-297297-- [place.wisp.subfs](/lexicons/place-wisp-subfs) - Subtree records for large sites
298298-- [AT Protocol Lexicons](https://atproto.com/specs/lexicon)
299299-
···33description: Reference for the place.wisp.subfs lexicon
44---
5566-**Lexicon Version:** 1
66+Subtree records for splitting large sites across multiple AT Protocol records. When a site exceeds 250 files or 140 KB, large directories are extracted into `place.wisp.subfs` records and referenced from the main `place.wisp.fs` manifest.
7788-## Overview
88+**Lexicon version:** 1
991010-The `place.wisp.subfs` lexicon defines subtree records for splitting large sites across multiple AT Protocol records. When a site exceeds size limits (250+ files or 140KB manifest), large directories are extracted into separate `place.wisp.subfs` records.
1010+## main (record)
11111212-**Key Feature:** Subfs entries are referenced from `place.wisp.fs` records and can be either **merged (flattened)** into the parent directory or placed as a subdirectory, depending on the `flat` property in the parent `place.wisp.fs` record.
1212+| Property | Type | Required | Constraints |
1313+| --- | --- | --- | --- |
1414+| `root` | [`#directory`](#directory) | ✅ | Root directory of the subtree |
1515+| `fileCount` | `integer` | | Min: 0, Max: 1000 |
1616+| `createdAt` | `string` | ✅ | Format: `datetime` |
13171414-## Record Structure
1818+## file
15191616-<a name="main"></a>
2020+| Property | Type | Required | Constraints |
2121+| --- | --- | --- | --- |
2222+| `type` | `string` | ✅ | Const: `"file"` |
2323+| `blob` | `blob` | ✅ | Max size: 1 GB |
2424+| `encoding` | `string` | | Enum: `["gzip"]` |
2525+| `mimeType` | `string` | | Original MIME type |
2626+| `base64` | `boolean` | | True if base64-encoded to bypass content sniffing on legacy reference PDS |
17271818-### `main` (Record)
2828+## directory
19292020-**Type:** `record`
3030+| Property | Type | Required | Constraints |
3131+| --- | --- | --- | --- |
3232+| `type` | `string` | ✅ | Const: `"directory"` |
3333+| `entries` | Array of [`#entry`](#entry) | ✅ | Max: 500 entries |
21342222-**Description:** Virtual filesystem subtree referenced by place.wisp.fs records. How this subtree is integrated depends on the `flat` property in the referencing subfs entry.
3535+## entry
23362424-**Properties:**
3737+| Property | Type | Required | Constraints |
3838+| --- | --- | --- | --- |
3939+| `name` | `string` | ✅ | Max: 255 chars |
4040+| `node` | Union of [`#file`](#file), [`#directory`](#directory), [`#subfs`](#subfs) | ✅ | |
25412626-| Name | Type | Req'd | Description | Constraints |
2727-| ----------- | -------------------------- | ----- | ----------------------------------------- | ------------------ |
2828-| `root` | [`#directory`](#directory) | ✅ | Root directory containing subtree entries | |
2929-| `fileCount` | `integer` | | Number of files in this subtree | Min: 0, Max: 1000 |
3030-| `createdAt` | `string` | ✅ | Timestamp of subtree creation | Format: `datetime` |
4242+## subfs
31433232----
3333-3434-## Type Definitions
3535-3636-<a name="file"></a>
3737-3838-### `file`
3939-4040-**Type:** `object`
4141-4242-**Description:** Represents a file node in the directory tree
4343-4444-**Properties:**
4545-4646-| Name | Type | Req'd | Description | Constraints |
4747-| ---------- | --------- | ----- | ------------------------------------------------------------ | ------------------------- |
4848-| `type` | `string` | ✅ | Node type identifier | Const: `"file"` |
4949-| `blob` | `blob` | ✅ | Content blob reference | Max size: 1000000000 (1GB) |
5050-| `encoding` | `string` | | Content encoding (e.g., gzip for compressed files) | Enum: `["gzip"]` |
5151-| `mimeType` | `string` | | Original MIME type before compression | |
5252-| `base64` | `boolean` | | True if blob content is base64-encoded (bypasses PDS sniffing) | |
4444+Reference to another `place.wisp.subfs` record for nested subtrees. Subfs records can reference other subfs records recursively.
53455454----
4646+| Property | Type | Required | Constraints |
4747+| --- | --- | --- | --- |
4848+| `type` | `string` | ✅ | Const: `"subfs"` |
4949+| `subject` | `string` | ✅ | AT-URI to another `place.wisp.subfs` record |
55505656-<a name="directory"></a>
5151+## How Merging Works
57525858-### `directory`
5353+The `flat` property on the referencing entry in `place.wisp.fs` controls how subfs entries are integrated.
59546060-**Type:** `object`
5555+**With `flat: true` (default)** — subfs entries are merged directly into the parent directory:
61566262-**Description:** Represents a directory node in the file tree
6363-6464-**Properties:**
6565-6666-| Name | Type | Req'd | Description | Constraints |
6767-| --------- | -------------------------- | ----- | ------------------------------ | -------------------- |
6868-| `type` | `string` | ✅ | Node type identifier | Const: `"directory"` |
6969-| `entries` | Array of [`#entry`](#entry) | ✅ | Child entries in this directory | Max: 500 entries |
7070-7171----
7272-7373-<a name="entry"></a>
7474-7575-### `entry`
7676-7777-**Type:** `object`
7878-7979-**Description:** Named entry in a directory (file, directory, or nested subfs)
8080-8181-**Properties:**
8282-8383-| Name | Type | Req'd | Description | Constraints |
8484-| ------ | ------------------------------------------------------- | ----- | ----------------------------------- | ----------- |
8585-| `name` | `string` | ✅ | File or directory name | Max: 255 chars |
8686-| `node` | Union of [`#file`](#file), [`#directory`](#directory), [`#subfs`](#subfs) | ✅ | The node (file, directory, or subfs reference) | |
8787-8888----
8989-9090-<a name="subfs"></a>
9191-9292-### `subfs`
9393-9494-**Type:** `object`
9595-9696-**Description:** Reference to another `place.wisp.subfs` record for nested subtrees. When expanded, entries are merged (flattened) into the parent directory by default, unless the parent `place.wisp.fs` record specifies `flat: false`.
9797-9898-**Properties:**
9999-100100-| Name | Type | Req'd | Description | Constraints |
101101-| --------- | -------- | ----- | --------------------------------------------------------------------------- | ---------------- |
102102-| `type` | `string` | ✅ | Node type identifier | Const: `"subfs"` |
103103-| `subject` | `string` | ✅ | AT-URI pointing to another place.wisp.subfs record for nested subtrees | Format: `at-uri` |
104104-105105-**Notes:**
106106-- Subfs records can reference other subfs records recursively
107107-- When expanded, entries are merged (flattened) into the parent directory by default
108108-- The `flat` property in the parent `place.wisp.fs` record controls integration behavior
109109-- Allows splitting very large directory structures
110110-111111----
112112-113113-## How Subfs Merging Works
114114-115115-### Before Expansion
116116-117117-Main record (`place.wisp.fs`):
118118-```json
119119-{
120120- "root": {
121121- "type": "directory",
122122- "entries": [
123123- { "name": "index.html", "node": { "type": "file", ... } },
124124- { "name": "docs", "node": { "type": "subfs", "subject": "at://did:plc:abc/place.wisp.subfs/xyz" } }
125125- ]
126126- }
127127-}
12857```
5858+Main fs root/
5959+├── index.html
6060+└── [subfs ref] ──→ { guide.html, api.html }
12961130130-Referenced subfs record (`at://did:plc:abc/place.wisp.subfs/xyz`):
131131-```json
132132-{
133133- "root": {
134134- "type": "directory",
135135- "entries": [
136136- { "name": "guide.html", "node": { "type": "file", ... } },
137137- { "name": "api.html", "node": { "type": "file", ... } }
138138- ]
139139- }
140140-}
6262+After expansion:
6363+├── index.html
6464+├── guide.html
6565+└── api.html
14166```
14267143143-### After Expansion (What Hosting Service Sees)
6868+**With `flat: false`** — subfs entries become a subdirectory:
14469145145-**With `flat: true` (default):**
146146-147147-```json
148148-{
149149- "root": {
150150- "type": "directory",
151151- "entries": [
152152- { "name": "index.html", "node": { "type": "file", ... } },
153153- { "name": "guide.html", "node": { "type": "file", ... } },
154154- { "name": "api.html", "node": { "type": "file", ... } }
155155- ]
156156- }
157157-}
15870```
159159-160160-The subfs entries are merged directly into the parent directory.
161161-162162-**With `flat: false`:**
163163-164164-```json
165165-{
166166- "root": {
167167- "type": "directory",
168168- "entries": [
169169- { "name": "index.html", "node": { "type": "file", ... } },
170170- { "name": "docs", "node": {
171171- "type": "directory",
172172- "entries": [
173173- { "name": "guide.html", "node": { "type": "file", ... } },
174174- { "name": "api.html", "node": { "type": "file", ... } }
175175- ]
176176- }}
177177- ]
178178- }
179179-}
180180-```
181181-182182-The subfs entries are placed in a subdirectory named "docs".
183183-184184----
185185-186186-## Usage Examples
187187-188188-### Basic Subfs Record
189189-190190-```json
191191-{
192192- "$type": "place.wisp.subfs",
193193- "root": {
194194- "type": "directory",
195195- "entries": [
196196- {
197197- "name": "chapter1.html",
198198- "node": {
199199- "type": "file",
200200- "blob": {
201201- "$type": "blob",
202202- "ref": { "$link": "bafyreiabc..." },
203203- "mimeType": "application/octet-stream",
204204- "size": 8421
205205- },
206206- "encoding": "gzip",
207207- "mimeType": "text/html",
208208- "base64": true
209209- }
210210- },
211211- {
212212- "name": "chapter2.html",
213213- "node": {
214214- "type": "file",
215215- "blob": {
216216- "$type": "blob",
217217- "ref": { "$link": "bafyreidef..." },
218218- "mimeType": "application/octet-stream",
219219- "size": 9234
220220- },
221221- "encoding": "gzip",
222222- "mimeType": "text/html",
223223- "base64": true
224224- }
225225- }
226226- ]
227227- },
228228- "fileCount": 2,
229229- "createdAt": "2024-01-15T10:30:00.000Z"
230230-}
231231-```
232232-233233-### Nested Subfs (Recursive)
234234-235235-A subfs record can reference another subfs record:
236236-237237-```json
238238-{
239239- "$type": "place.wisp.subfs",
240240- "root": {
241241- "type": "directory",
242242- "entries": [
243243- {
244244- "name": "section-a.html",
245245- "node": { "type": "file", "blob": {...}, "encoding": "gzip", "mimeType": "text/html", "base64": true }
246246- },
247247- {
248248- "name": "subsection",
249249- "node": {
250250- "type": "subfs",
251251- "subject": "at://did:plc:abc123/place.wisp.subfs/nested123"
252252- }
253253- }
254254- ]
255255- },
256256- "fileCount": 50,
257257- "createdAt": "2024-01-15T10:30:00.000Z"
258258-}
7171+After expansion:
7272+├── index.html
7373+└── docs/
7474+ ├── guide.html
7575+ └── api.html
25976```
26077261261----
262262-263263-## When Are Subfs Records Created?
264264-265265-The Wisp CLI and web interface automatically create subfs records when:
266266-267267-1. **File count threshold**: Site has 250+ files (keeps main manifest under 200 files)
268268-2. **Size threshold**: Main manifest exceeds 140KB (PDS limit is 150KB)
269269-3. **Large directories**: Individual directories with many files
270270-271271-### Splitting Algorithm
272272-273273-The `flat` property in the parent `place.wisp.fs` record controls integration behavior:
274274-- `flat: true` (default): Merge subfs entries directly into parent directory
275275-- `flat: false`: Create subdirectory with the subfs entry's name
276276-277277----
278278-279279-## Best Practices
280280-281281-### For Hosting Services
282282-283283-- **Fetch recursively**: Load all subfs records referenced in the tree
284284-- **Merge entries**: Replace subfs nodes with directory nodes containing referenced entries
285285-- **Cache merged tree**: Store the fully expanded tree for serving
286286-- **Update on firehose**: Re-fetch and re-merge when subfs records change
287287-288288-### For Upload Tools
289289-290290-- **Reuse subfs records**: Check existing subfs URIs before creating new ones
291291-- **Clean up old records**: Delete unused subfs records after updates
292292-- **Maintain file paths**: Preserve original directory structure when extracting to subfs
293293-294294----
295295-29678## Lexicon Source
2977929880```json
···30284 "defs": {
30385 "main": {
30486 "type": "record",
305305- "description": "Virtual filesystem subtree referenced by place.wisp.fs records. When a subfs entry is expanded, its root entries are merged (flattened) into the parent directory, allowing large directories to be split across multiple records while maintaining a flat structure.",
30687 "record": {
30788 "type": "object",
30889 "required": ["root", "createdAt"],
···31899 "required": ["type", "blob"],
319100 "properties": {
320101 "type": { "type": "string", "const": "file" },
321321- "blob": { "type": "blob", "accept": ["*/*"], "maxSize": 1000000000, "description": "Content blob ref" },
322322- "encoding": { "type": "string", "enum": ["gzip"], "description": "Content encoding (e.g., gzip for compressed files)" },
323323- "mimeType": { "type": "string", "description": "Original MIME type before compression" },
324324- "base64": { "type": "boolean", "description": "True if blob content is base64-encoded (used to bypass PDS content sniffing)" }
102102+ "blob": { "type": "blob", "accept": ["*/*"], "maxSize": 1000000000 },
103103+ "encoding": { "type": "string", "enum": ["gzip"] },
104104+ "mimeType": { "type": "string" },
105105+ "base64": { "type": "boolean" }
325106 }
326107 },
327108 "directory": {
···329110 "required": ["type", "entries"],
330111 "properties": {
331112 "type": { "type": "string", "const": "directory" },
332332- "entries": {
333333- "type": "array",
334334- "maxLength": 500,
335335- "items": { "type": "ref", "ref": "#entry" }
336336- }
113113+ "entries": { "type": "array", "maxLength": 500, "items": { "type": "ref", "ref": "#entry" } }
337114 }
338115 },
339116 "entry": {
···349126 "required": ["type", "subject"],
350127 "properties": {
351128 "type": { "type": "string", "const": "subfs" },
352352- "subject": { "type": "string", "format": "at-uri", "description": "AT-URI pointing to another place.wisp.subfs record for nested subtrees. Integration behavior (flat vs nested) is controlled by the flat property in the parent place.wisp.fs record." }
129129+ "subject": { "type": "string", "format": "at-uri" }
353130 }
354131 }
355132 }
356133}
357134```
358358-359359-## Related
360360-361361-- [place.wisp.fs](/lexicons/place-wisp-fs) - Main site manifest lexicon
362362-- [AT Protocol Lexicons](https://atproto.com/specs/lexicon)
363363-
+142
docs/src/content/docs/lexicons/place-wisp-wh.md
···11+---
22+title: place.wisp.v2.wh
33+description: Webhook record lexicon for receiving HTTP callbacks on AT Protocol events
44+---
55+66+Webhooks let you receive HTTP POST notifications when AT Protocol records are created, updated, or deleted. They're scoped to an AT-URI — you can watch a specific record, an entire collection, or everything from a DID.
77+88+Webhooks are stored as `place.wisp.v2.wh` records in your AT Protocol repository. The webhook service watches the firehose and delivers payloads to your URL.
99+1010+## Creating a Webhook
1111+1212+Create and manage webhooks from the **Webhooks** tab in the editor, or via the [REST API](#rest-api).
1313+1414+**Scope** controls what you're watching:
1515+1616+| Scope AT-URI | Watches |
1717+|---|---|
1818+| `at://did:plc:abc` | All record changes from that DID |
1919+| `at://did:plc:abc/app.bsky.feed.post` | All posts from that DID |
2020+| `at://did:plc:abc/app.bsky.feed.post/rkey` | One specific record |
2121+2222+Enable **backlinks** to also fire when records in *any* repo reference your DID or collection — useful for watching Bluesky likes, Tangled pull requests, etc directed at you.
2323+2424+**Events** can be filtered to `create`, `update`, `delete`, or any combination. Omit the filter to receive all three.
2525+2626+**Secret** — if set, every delivery includes an `X-Webhook-Signature` header for verification.
2727+2828+## Payload
2929+3030+Each delivery is an HTTP POST with `Content-Type: application/json`:
3131+3232+```json
3333+{
3434+ "id": "550e8400-e29b-41d4-a716-446655440000",
3535+ "event": "create",
3636+ "did": "did:plc:abc123",
3737+ "collection": "app.bsky.feed.post",
3838+ "rkey": "3kl2jd9s8f7g",
3939+ "cid": "bafyreiabc...",
4040+ "record": { ... },
4141+ "timestamp": "2024-01-15T10:30:00.000Z"
4242+}
4343+```
4444+4545+`record` is the full record body and is absent on `delete` events.
4646+4747+**Headers:**
4848+4949+```
5050+User-Agent: wisp.place-webhook/1.0
5151+X-Webhook-Signature: sha256=<hex> (only if secret is set)
5252+```
5353+5454+## Verifying Signatures
5555+5656+If you set a secret, verify the `X-Webhook-Signature` header using HMAC-SHA256:
5757+5858+```typescript
5959+import { createHmac, timingSafeEqual } from 'crypto'
6060+6161+function verifySignature(body: string, secret: string, header: string): boolean {
6262+ const expected = 'sha256=' + createHmac('sha256', secret).update(body).digest('hex')
6363+ return timingSafeEqual(Buffer.from(header), Buffer.from(expected))
6464+}
6565+```
6666+6767+Always use a timing-safe comparison. Compute the HMAC over the raw request body before parsing.
6868+6969+## Delivery
7070+7171+The webhook service delivers with a 10 second timeout and retries up to 3 times with exponential backoff on failure. Your endpoint should return a 2xx response quickly — do any heavy processing asynchronously.
7272+7373+Delivery attempts are logged and visible in the editor under each webhook's event history (last 500 events per webhook).
7474+7575+## Record Schema
7676+7777+Webhooks are stored as `place.wisp.v2.wh` records in your PDS:
7878+7979+```json
8080+{
8181+ "$type": "place.wisp.v2.wh",
8282+ "scope": {
8383+ "aturi": "at://did:plc:abc123/app.bsky.feed.post",
8484+ "backlinks": false
8585+ },
8686+ "url": "https://example.com/webhook",
8787+ "events": ["create", "update"],
8888+ "secret": "your-hmac-secret",
8989+ "enabled": true,
9090+ "createdAt": "2024-01-15T10:30:00.000Z"
9191+}
9292+```
9393+9494+## REST API
9595+9696+Webhooks can also be managed via the main app API. All routes require the signed `did` cookie.
9797+9898+### `GET /api/webhook`
9999+100100+Lists all webhook records for the authenticated user.
101101+102102+### `POST /api/webhook`
103103+104104+Creates a new webhook. Body matches the `place.wisp.v2.wh` record shape.
105105+106106+### `DELETE /api/webhook/:rkey`
107107+108108+Deletes a webhook by its record key.
109109+110110+### `GET /api/webhook/events`
111111+112112+Returns the last 100 delivery events for the authenticated user.
113113+114114+```json
115115+[
116116+ {
117117+ "id": "...",
118118+ "rkey": "abc123",
119119+ "url": "https://example.com/webhook",
120120+ "event_kind": "create",
121121+ "event_did": "did:plc:...",
122122+ "event_collection": "app.bsky.feed.post",
123123+ "event_rkey": "3kl2jd9s8f7g",
124124+ "status": "ok",
125125+ "delivered_at": "2024-01-15T10:30:00.000Z"
126126+ }
127127+]
128128+```
129129+130130+## Self-Hosting
131131+132132+The webhook service is a separate Bun process in `apps/webhook-service`.
133133+134134+```bash
135135+DATABASE_URL="postgres://user:password@localhost:5432/wisp"
136136+JETSTREAM_URL="wss://jetstream2.us-east.bsky.network/subscribe"
137137+HEALTH_PORT=3003
138138+DELIVERY_TIMEOUT_MS=10000
139139+DELIVERY_MAX_RETRIES=3
140140+REDIS_URL="redis://localhost:6379"
141141+WEBHOOK_EVENTS_CHANNEL="webhook:events"
142142+```
+36-96
docs/src/content/docs/monitoring.md
···11---
22title: Monitoring & Metrics
33-description: Track performance and debug issues with Grafana integration
33+description: Grafana integration for logs and metrics
44---
5566-Wisp.place includes built-in observability with automatic Grafana integration for logs and metrics. Monitor request performance, track errors, and analyze usage patterns across both the main backend and hosting service.
77-88-## Quick Start
99-1010-Set environment variables to enable Grafana export:
66+Set these environment variables and restart your services. Metrics and logs will flow to Grafana automatically.
117128```bash
139# Grafana Cloud
···1713GRAFANA_PROMETHEUS_URL=https://prometheus-prod-xxx.grafana.net/api/prom
1814GRAFANA_PROMETHEUS_TOKEN=glc_xxx
19152020-# Self-hosted Grafana
1616+# Self-hosted (basic auth instead of bearer token)
2117GRAFANA_LOKI_USERNAME=your-username
2218GRAFANA_LOKI_PASSWORD=your-password
2319```
24202525-Restart services. Metrics and logs now flow to Grafana automatically.
2121+See [Grafana Setup](/guides/grafana-setup) for a step-by-step walkthrough.
26222727-## Metrics Collected
2323+## Metrics
28242929-### HTTP Requests
3030-- `http_requests_total` - Total request count by path, method, status
3131-- `http_request_duration_ms` - Request duration histogram
3232-- `errors_total` - Error count by service
2525+- `http_requests_total` — request count by path, method, and status
2626+- `http_request_duration_ms` — duration histogram (P50/P95/P99 available)
2727+- `errors_total` — error count by service
33283434-### Performance Stats
3535-- P50, P95, P99 response times
3636-- Requests per minute
3737-- Error rates
3838-- Average duration by endpoint
3939-4040-## Log Aggregation
2929+## Log Labels
41304242-Logs are sent to Loki with automatic categorization:
3131+Logs are tagged by service so you can filter them in Loki:
43324433```
4545-{job="main-app"} |= "error" # OAuth and upload errors
4646-{job="hosting-service"} |= "cache" # Cache operations
4747-{service="hosting-service", level="warn"} # Warnings only
3434+{job="main-app"} # OAuth, uploads, domain management
3535+{job="hosting-service"} # Firehose, caching, content serving
4836```
49375050-## Service Identification
5151-5252-Each service is tagged separately:
5353-- `main-app` - OAuth, uploads, domain management
5454-- `hosting-service` - Firehose, caching, content serving
5555-5656-## Configuration Options
5757-5858-### Environment Variables
3838+## All Options
59396040```bash
6161-# Required
6262-GRAFANA_LOKI_URL # Loki endpoint
6363-GRAFANA_PROMETHEUS_URL # Prometheus endpoint (add /api/prom for OTLP)
4141+GRAFANA_LOKI_URL # Loki push endpoint
4242+GRAFANA_PROMETHEUS_URL # Prometheus remote write endpoint
64436565-# Authentication (use one)
6666-GRAFANA_LOKI_TOKEN # Bearer token (Grafana Cloud)
6767-GRAFANA_LOKI_USERNAME # Basic auth (self-hosted)
4444+GRAFANA_LOKI_TOKEN # Bearer token (Grafana Cloud)
4545+GRAFANA_LOKI_USERNAME # Basic auth (self-hosted)
6846GRAFANA_LOKI_PASSWORD
69477070-# Optional
7171-GRAFANA_BATCH_SIZE=100 # Batch size before flush
7272-GRAFANA_FLUSH_INTERVAL=5000 # Flush interval in ms
7373-```
7474-7575-### Programmatic Setup
7676-7777-```typescript
7878-import { initializeGrafanaExporters } from '@wisp/observability'
7979-8080-initializeGrafanaExporters({
8181- lokiUrl: 'https://logs.grafana.net',
8282- lokiAuth: { bearerToken: 'token' },
8383- prometheusUrl: 'https://prometheus.grafana.net/api/prom',
8484- prometheusAuth: { bearerToken: 'token' },
8585- serviceName: 'my-service',
8686- batchSize: 100,
8787- flushIntervalMs: 5000
8888-})
4848+GRAFANA_BATCH_SIZE=100 # Entries per flush
4949+GRAFANA_FLUSH_INTERVAL=5000 # Flush interval in ms
8950```
90519191-## Grafana Dashboard Queries
5252+## Dashboard Queries
92539393-### Request Performance
9454```promql
9555# Average response time by endpoint
9656avg by (path) (
···9858 rate(http_request_duration_ms_count[5m])
9959)
10060101101-# Request rate
6161+# Request rate by service
10262sum(rate(http_requests_total[1m])) by (service)
1036310464# Error rate
···10666sum(rate(http_requests_total[5m])) by (service)
10767```
10868109109-### Log Analysis
11069```logql
111111-# Recent errors
11270{job="main-app"} |= "error" | json
113113-114114-# Slow requests (>1s)
11571{job="hosting-service"} |~ "duration.*[1-9][0-9]{3,}"
116116-117117-# Failed OAuth attempts
118118-{job="main-app"} |= "OAuth" |= "failed"
11972```
12073121121-## Troubleshooting
122122-123123-### Logs not appearing
124124-- Check `GRAFANA_LOKI_URL` is correct (no trailing `/loki/api/v1/push`)
125125-- Verify authentication token/credentials
126126-- Look for connection errors in service logs
127127-128128-### Metrics missing
129129-- Ensure `GRAFANA_PROMETHEUS_URL` includes `/api/prom` suffix
130130-- Check firewall rules allow outbound HTTPS
131131-- Verify OpenTelemetry export errors in logs
7474+## Without Grafana
13275133133-### High memory usage
134134-- Reduce `GRAFANA_BATCH_SIZE` (default: 100)
135135-- Lower `GRAFANA_FLUSH_INTERVAL` to flush more frequently
136136-137137-## Local Development
138138-139139-Metrics and logs are stored in-memory when Grafana isn't configured. Access them via:
7676+Metrics and logs are always stored in-memory. Access them directly:
1407714178- `http://localhost:8000/api/observability/logs`
14279- `http://localhost:8000/api/observability/metrics`
14380- `http://localhost:8000/api/observability/errors`
14481145145-## Testing Integration
8282+## Programmatic Setup
14683147147-Run integration tests to verify setup:
8484+```typescript
8585+import { initializeGrafanaExporters } from '@wisp/observability'
14886149149-```bash
150150-cd packages/@wisp/observability
151151-bun test src/integration-test.test.ts
152152-153153-# Test with live Grafana
154154-GRAFANA_LOKI_URL=... GRAFANA_LOKI_USERNAME=... GRAFANA_LOKI_PASSWORD=... \
155155-bun test src/integration-test.test.ts
156156-```8787+initializeGrafanaExporters({
8888+ lokiUrl: 'https://logs.grafana.net',
8989+ lokiAuth: { bearerToken: 'token' },
9090+ prometheusUrl: 'https://prometheus.grafana.net/api/prom',
9191+ prometheusAuth: { bearerToken: 'token' },
9292+ serviceName: 'my-service',
9393+ batchSize: 100,
9494+ flushIntervalMs: 5000
9595+})
9696+```
+31-174
docs/src/content/docs/redirects.md
···11---
22title: Redirects & Rewrites
33-description: Netlify-style _redirects file support for flexible URL routing
33+description: Netlify-style _redirects file support
44---
5566-# Redirects & Rewrites
77-88-Wisp.place supports Netlify-style `_redirects` files, giving you powerful control over URL routing and redirects. Whether you're migrating an old site, setting up a single-page app, or creating clean URLs, the `_redirects` file lets you handle complex routing scenarios without changing your actual file structure.
99-1010-## Getting Started
1111-1212-Drop a file named `_redirects` in your site's root directory. Each line defines a redirect rule with the format:
1313-1414-```
1515-/from/path /to/path [status] [conditions]
1616-```
66+Drop a `_redirects` file in your site's root directory. Each line is a rule — processed top to bottom, first match wins.
1771818-For example:
198```
2020-/old-page /new-page
2121-/blog/* /posts/:splat 301
2222-```
2323-2424-## Basic Redirects
2525-2626-The simplest redirects move traffic from one URL to another:
2727-2828-```
2929-/home /
3030-/about-us /about
3131-/old-blog /blog
99+/from/path /to/path [status]
3210```
3333-3434-These use a permanent redirect (301) by default, telling browsers and search engines the page has moved permanently.
35113612## Status Codes
37133838-You can specify different HTTP status codes to change how the redirect behaves:
1414+`301` is permanent (default), `302` is temporary, `200` serves new content while keeping the original URL, and `404` serves a custom error page. Append `!` to force a rule even when the source path exists as an actual file.
39154040-**301 - Permanent Redirect**
4116```
4242-/legacy-page /new-page 301
1717+/old-page /new-page 301
1818+/temp-sale /sale-page 302
1919+/api/* /functions/:splat 200
2020+/shop/* /shop-closed.html 404
2121+/file /other 200!
4322```
4444-Tells browsers and search engines the page has moved permanently. Good for SEO when content has truly moved.
45234646-**302 - Temporary Redirect**
4747-```
4848-/temp-sale /sale-page 302
4949-```
5050-Indicates a temporary move. Browsers won't cache this as strongly, and search engines won't transfer SEO value.
2424+## Wildcards & Placeholders
51255252-**200 - Rewrite**
5353-```
5454-/api/* /functions/:splat 200
5555-```
5656-Serves different content but keeps the original URL visible to users. Perfect for API routing or single-page apps.
2626+`:splat` captures the wildcard match:
57275858-**404 - Custom Error Page**
5959-```
6060-/shop/* /shop-closed.html 404
6161-```
6262-Shows a custom error page instead of the default 404. Useful for seasonal closures or section-specific error handling.
6363-6464-**Force with `!`**
6565-```
6666-/existing-file /other-file 200!
6767-```
6868-Normally, if the original path exists as a file, the redirect won't trigger. Add `!` to force it anyway.
6969-7070-## Wildcard Redirects
7171-7272-Splats (`*`) let you match entire path segments:
7373-7474-**Simple wildcards:**
7528```
7629/news/* /blog/:splat
7730/old-site/* /new-site/:splat
7831```
79328080-If someone visits `/news/tech-update`, they'll be redirected to `/blog/tech-update`.
8181-8282-**Multiple wildcards:**
8383-```
8484-/products/*/details/* /shop/:splat/info/:splat
8585-```
8686-8787-This captures multiple path segments and maps them to the new structure.
8888-8989-## Placeholders
9090-9191-Placeholders let you restructure URLs with named parameters:
3333+Named placeholders for structured URLs:
92349335```
9436/blog/:year/:month/:day/:slug /posts/:year-:month-:day/:slug
9537/products/:category/:id /shop/:category/item/:id
9638```
97399898-These are more precise than splats because you can reference the captured values by name. Visiting `/blog/2024/01/15/my-post` redirects to `/posts/2024-01-15/my-post`.
9999-100100-## Query Parameters
101101-102102-You can match and redirect based on URL parameters:
4040+Query parameters work too:
1034110442```
10543/store?id=:id /products/:id
10644/search?q=:query /find/:query
10745```
10846109109-The query parameter becomes part of the redirect path. `/store?id=123` becomes `/products/123`.
110110-11147## Conditional Redirects
11248113113-Make redirects happen only under certain conditions:
4949+Route based on country (ISO 3166-1 alpha-2), browser language, or cookie:
11450115115-**Country-based:**
11651```
117117-/ /us/ 302 Country=us
118118-/ /uk/ 302 Country=gb
119119-```
5252+/ /us/ 302 Country=us
5353+/ /uk/ 302 Country=gb
12054121121-Redirects users based on their country (using ISO 3166-1 alpha-2 codes).
5555+/products /en/products 301 Language=en
5656+/products /de/products 301 Language=de
12257123123-**Language-based:**
5858+/* /legacy/:splat 200 Cookie=is_legacy
12459```
125125-/products /en/products 301 Language=en
126126-/products /de/products 301 Language=de
127127-```
128128-129129-Routes based on browser language preferences.
130130-131131-**Cookie-based:**
132132-```
133133-/* /legacy/:splat 200 Cookie=is_legacy
134134-```
135135-136136-Only redirects if the user has a specific cookie set.
137137-138138-## Advanced Patterns
139139-140140-**Single-page app routing:**
141141-```
142142-/* /index.html 200
143143-```
144144-145145-Send all unmatched routes to your main app file. Perfect for React, Vue, or Angular apps.
146146-147147-**API proxying:**
148148-```
149149-/api/* https://api.example.com/:splat 200
150150-```
151151-152152-Proxy API calls to external services while keeping the URL clean.
153153-154154-**Domain redirects:**
155155-```
156156-http://blog.example.com/* https://example.com/blog/:splat 301!
157157-```
158158-159159-Redirect from subdomains or entirely different domains.
160160-161161-**Extension removal:**
162162-```
163163-/page.html /page
164164-```
165165-166166-Clean up old `.html` extensions for a modern look.
167167-168168-## How It Works
169169-170170-1. **Processing order:** Rules are checked from top to bottom - first match wins
171171-2. **Specificity:** More specific rules should come before general ones
172172-3. **Caching:** Redirects are cached for performance but respect the site's cache headers
173173-4. **Performance:** All processing happens at the edge, close to your users
174174-175175-## Examples
17660177177-Here's a complete `_redirects` file for a typical site migration:
6161+## Common Patterns
1786217963```
180180-# Old blog structure to new
181181-/blog/* /posts/:splat 301
6464+# SPA fallback
6565+/* /index.html 200
1826618367# API proxy
184184-/api/* https://api.example.com/:splat 200
6868+/api/* https://api.example.com/:splat 200
18569186186-# Country redirects for homepage
187187-/ /us/ 302 Country=us
188188-/ /uk/ 302 Country=gb
189189-190190-# Single-page app fallback
191191-/* /index.html 200
7070+# Remove .html extensions
7171+/page.html /page
19272193193-# Custom 404 for shop section
194194-/shop/* /shop/closed.html 404
7373+# Full example
7474+/blog/* /posts/:splat 301
7575+/api/* https://api.example.com/:splat 200
7676+/ /us/ 302 Country=us
7777+/ /uk/ 302 Country=gb
7878+/* /index.html 200
19579```
196196-197197-## Tips
198198-199199-- **Order matters:** Put specific rules before general ones
200200-- **Test thoroughly:** Use the preview feature to check your redirects
201201-- **Use 301 for SEO:** Permanent redirects pass SEO value to new pages
202202-- **Use 200 for SPAs:** Rewrites keep your app's routing intact
203203-- **Force when needed:** The `!` flag overrides existing files
204204-- **Keep it simple:** Most sites only need a few redirect rules
205205-206206-## Troubleshooting
207207-208208-**Redirect not working?**
209209-- Check the order - rules are processed top to bottom
210210-- Make sure the file is named exactly `_redirects` (no extension)
211211-- Verify the file is in your site's root directory
212212-213213-**Wildcard not matching?**
214214-- Wildcards only work at the end of paths
215215-- Use placeholders for more complex restructuring
216216-217217-**Conditional redirect not triggering?**
218218-- Country detection uses IP geolocation
219219-- Language uses Accept-Language headers
220220-- Cookies must match exactly
221221-222222-The `_redirects` system gives you the flexibility to handle complex routing scenarios while keeping your site structure clean and maintainable.
-11
docs/src/content/docs/reference/example.md
···11----
22-title: Example Reference
33-description: A reference page in my new Starlight docs site.
44----
55-66-Reference pages are ideal for outlining how things work in terse and clear terms.
77-Less concerned with telling a story or addressing a specific use case, they should give a comprehensive outline of what you're documenting.
88-99-## Further reading
1010-1111-- Read [about reference](https://diataxis.fr/reference/) in the Diátaxis framework