A fullstack app for indexing standard.site documents
1# AT Feeds
2
3A monorepo for indexing and displaying [Standard.site](https://standard.site) documents from the AT Protocol, powered by Cloudflare Workers, D1, and Queues.
4
5## Architecture
6
7```
8┌─────────────────────────────────────────────────────────────┐
9│ Cloudflare │
10├─────────────────────────────────────────────────────────────┤
11│ │
12│ ┌──────────────┐ ┌──────────────┐ ┌─────────────┐ │
13│ │ Pages │────▶│ Worker │────▶│ D1 │ │
14│ │ (Client) │ │ (API) │ │ (Database) │ │
15│ └──────────────┘ └──────────────┘ └─────────────┘ │
16│ ▲ ▲ │
17│ │ │ │
18│ ┌──────┴───────┐ ┌──────┴───────┐ │
19│ │ Queue │ │ Cron │ │
20│ │ (Resolver) │ │ (Refresh) │ │
21│ └──────┬───────┘ └──────────────┘ │
22│ │ │
23└──────────────────────────────┼──────────────────────────────┘
24 │ POST /webhook/tap
25 ┌──────────┴───────────┐
26 │ Tap Instance │
27 │ (External VPS) │
28 └──────────────────────┘
29```
30
31**Components:**
32
331. **Tap Indexer** (External) - Subscribes to the AT Protocol firehose and sends webhook events
342. **Server** (`packages/server`) - Cloudflare Worker with Hono API, D1 database, and Queue consumer
353. **Client** (`packages/client`) - Vite + React app deployed to Cloudflare Pages
36
37## Quick Start
38
39### Prerequisites
40
41- [Bun](https://bun.sh) installed
42- [Wrangler CLI](https://developers.cloudflare.com/workers/wrangler/) installed and authenticated
43- A tap instance running somewhere (VPS, Fly.io, etc.)
44
45### Setup
46
471. Install dependencies:
48
49```bash
50bun install
51```
52
532. Create the D1 database:
54
55```bash
56bun run db:create
57```
58
59Copy the database ID and update `packages/server/wrangler.toml`.
60
613. Create the queue:
62
63```bash
64wrangler queues create document-resolution
65```
66
674. Run database migrations:
68
69```bash
70# Local development
71bun run db:migrate
72
73# Production
74bun run db:migrate:prod
75```
76
775. (Optional) Set webhook secret:
78
79```bash
80bun run secret:set
81```
82
836. Deploy the worker:
84
85```bash
86bun run deploy
87```
88
897. Configure your tap instance:
90
91```bash
92TAP_WEBHOOK_URL=https://your-worker.workers.dev/webhook/tap
93TAP_SIGNAL_COLLECTION=site.standard.document
94TAP_COLLECTION_FILTERS=site.standard.document
95```
96
978. Trigger initial resolution of existing records:
98
99```bash
100curl -X POST https://your-worker.workers.dev/admin/resolve-all
101```
102
103## Local Development
104
1051. Start the worker locally:
106
107```bash
108bun run dev:server
109```
110
111The API will run on `http://localhost:8787`.
112
1132. Start the client (in a separate terminal):
114
115```bash
116bun run dev:client
117```
118
119The client will run on `http://localhost:5173`.
120
121## API Endpoints
122
123### Health & Stats
124
125| Endpoint | Method | Description |
126|----------|--------|-------------|
127| `/health` | GET | Health check |
128| `/stats` | GET | Database statistics |
129
130### Feed Endpoints
131
132| Endpoint | Method | Description |
133|----------|--------|-------------|
134| `/feed` | GET | Pre-resolved documents (fast) |
135| `/feed-raw` | GET | Raw record references (for client-side resolution) |
136| `/records/:did` | GET | Records by DID |
137
138### Webhook
139
140| Endpoint | Method | Description |
141|----------|--------|-------------|
142| `/webhook/tap` | POST | Receives events from tap |
143| `/webhook/tap/debug` | POST | Debug endpoint (echoes payload) |
144
145### Admin
146
147| Endpoint | Method | Description |
148|----------|--------|-------------|
149| `/admin/resolve-all` | POST | Queue unresolved records for processing |
150
151## How It Works
152
1531. **Tap** subscribes to the AT Protocol firehose and filters for `site.standard.document` records
1542. **Webhook** receives events and stores record references in D1, then pushes to the resolution queue
1553. **Queue consumer** resolves each document (PDS lookup → record fetch → publication URL) and stores in `resolved_documents`
1564. **Cron job** (every 15 min) refreshes stale documents and processes any missed records
1575. **`/feed` endpoint** reads directly from `resolved_documents` for instant responses
158
159## Project Structure
160
161```
162.
163├── package.json # Root workspace config
164└── packages/
165 ├── server/ # Cloudflare Worker
166 │ ├── wrangler.toml # Worker configuration
167 │ ├── schema.sql # D1 database schema
168 │ ├── package.json
169 │ └── src/
170 │ └── index.ts # API + Queue consumer + Cron handler
171 └── client/ # Vite + React app
172 ├── package.json
173 ├── vite.config.ts
174 └── src/
175 ├── main.tsx
176 └── App.tsx
177```
178
179## Scripts
180
181```bash
182# Development
183bun run dev # Run all packages in dev mode
184bun run dev:server # Run worker locally
185bun run dev:client # Run client locally
186
187# Deployment
188bun run deploy # Deploy worker to Cloudflare
189bun run deploy:client # Deploy client to Cloudflare Pages
190
191# Database
192bun run db:create # Create D1 database
193bun run db:migrate # Run migrations (local)
194bun run db:migrate:prod # Run migrations (production)
195
196# Secrets
197bun run secret:set # Set TAP_WEBHOOK_SECRET
198```
199
200## Resources
201
202- [tap Documentation](https://github.com/bluesky-social/indigo/tree/main/cmd/tap)
203- [AT Protocol Specs](https://atproto.com/)
204- [Cloudflare Workers](https://developers.cloudflare.com/workers/)
205- [Cloudflare D1](https://developers.cloudflare.com/d1/)
206- [Cloudflare Queues](https://developers.cloudflare.com/queues/)
207- [Hono Documentation](https://hono.dev/)
208
209## License
210
211MIT