stevedylan.dev / docs.surf
A fullstack app for indexing standard.site documents
8
fork

Configure Feed

Select the types of activity you want to include in your feed.

TypeScript 94.1%
CSS 4.8%
HTML 1.1%
4 1 0

Clone this repository

https://tangled.org/stevedylan.dev/docs.surf https://tangled.org/did:plc:ia2zdnhjaokf5lazhxrmj6eu/docs.surf
git@tangled.org:stevedylan.dev/docs.surf git@tangled.org:did:plc:ia2zdnhjaokf5lazhxrmj6eu/docs.surf

For self-hosted knots, clone URLs may differ based on your setup.

Download tar.gz
README.md

Document Feeds#

A monorepo for indexing and displaying Standard.site documents from the AT Protocol, powered by Cloudflare Workers, D1, and Queues.

Architecture#

┌─────────────────────────────────────────────────────────────┐
│                     Cloudflare                              │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────┐     ┌──────────────┐     ┌─────────────┐  │
│  │    Pages     │────▶│   Worker     │────▶│     D1      │  │
│  │   (Client)   │     │   (API)      │     │  (Database) │  │
│  └──────────────┘     └──────────────┘     └─────────────┘  │
│                              ▲                   ▲          │
│                              │                   │          │
│                       ┌──────┴───────┐    ┌──────┴───────┐  │
│                       │    Queue     │    │     Cron     │  │
│                       │  (Resolver)  │    │  (Refresh)   │  │
│                       └──────┬───────┘    └──────────────┘  │
│                              │                              │
└──────────────────────────────┼──────────────────────────────┘
                               │ POST /webhook/tap
                    ┌──────────┴───────────┐
                    │   Tap Instance       │
                    │   (External)         │
                    └──────────────────────┘

Components:

  1. Tap Indexer (External) - Subscribes to the AT Protocol firehose and sends webhook events
  2. Server (packages/server) - Cloudflare Worker with Hono API, D1 database, and Queue consumer
  3. Client (packages/client) - Vite + React app deployed to Cloudflare Pages

Local Development#

  1. Start the worker locally:
bun run dev:server

The API will run on http://localhost:8787.

  1. Start the client (in a separate terminal):
bun run dev:client

The client will run on http://localhost:5173.

API Endpoints#

Health & Stats#

Endpoint Method Description
/health GET Health check
/stats GET Database statistics

Feed Endpoints#

Endpoint Method Description
/feed GET Pre-resolved documents (fast)
/feed-raw GET Raw record references (for client-side resolution)
/records/:did GET Records by DID

Webhook#

Endpoint Method Description
/webhook/tap POST Receives events from tap
/webhook/tap/debug POST Debug endpoint (echoes payload)

Admin#

Endpoint Method Description
/admin/resolve-all POST Queue unresolved records for processing

How It Works#

  1. Tap subscribes to the AT Protocol firehose and filters for site.standard.document records
  2. Webhook receives events and stores record references in D1, then pushes to the resolution queue
  3. Queue consumer resolves each document (PDS lookup → record fetch → publication URL) and stores in resolved_documents
  4. Cron job (every 15 min) refreshes stale documents and processes any missed records
  5. /feed endpoint reads directly from resolved_documents for instant responses

Resources#

License#

MIT