···11+# Development Workflow
22+33+## Setting Up Firehose Feed with Known DIDs
44+55+For development and testing, you can populate your local feed with known Arabica users:
66+77+### 1. Create a Known DIDs File
88+99+Create `known-dids.txt` in the project root:
1010+1111+```bash
1212+cat > known-dids.txt << 'EOF'
1313+# Known Arabica users for development
1414+# Add one DID per line
1515+1616+# Example (replace with real DIDs):
1717+# did:plc:abc123xyz
1818+# did:plc:def456uvw
1919+2020+EOF
2121+```
2222+2323+### 2. Find DIDs to Add
2424+2525+You can find DIDs of Arabica users in several ways:
2626+2727+**From Bluesky profiles:**
2828+- Visit a user's profile on Bluesky
2929+- Check the URL or profile metadata for their DID
3030+3131+**From authenticated sessions:**
3232+- After logging into Arabica, check your browser cookies
3333+- The `did` cookie contains your DID
3434+3535+**From AT Protocol explorer tools:**
3636+- Use tools like `atproto.blue` to search for users
3737+3838+### 3. Run Server with Backfill
3939+4040+```bash
4141+# Start server with firehose and backfill
4242+go run cmd/server/main.go --firehose --known-dids known-dids.txt
4343+4444+# Or with nix (requires adding flags to flake.nix)
4545+nix run -- --firehose --known-dids known-dids.txt
4646+```
4747+4848+### 4. Monitor Backfill Progress
4949+5050+Watch the logs for backfill activity:
5151+5252+```
5353+{"level":"info","count":3,"file":"known-dids.txt","message":"Loaded known DIDs from file"}
5454+{"level":"info","did":"did:plc:abc123xyz","message":"backfilling user records"}
5555+{"level":"info","total":5,"success":5,"message":"Backfill complete"}
5656+```
5757+5858+### 5. Verify Feed Data
5959+6060+Once backfilled, check:
6161+- Home page feed should show brews from backfilled users
6262+- `/feed` endpoint should return feed items
6363+- Database should contain indexed records
6464+6565+## File Format Notes
6666+6767+The `known-dids.txt` file supports:
6868+6969+- **Comments**: Lines starting with `#`
7070+- **Empty lines**: Ignored
7171+- **Whitespace**: Automatically trimmed
7272+- **Validation**: Non-DID lines logged as warnings
7373+7474+Example valid file:
7575+7676+```
7777+# Coffee enthusiasts to follow
7878+did:plc:user1abc
7979+8080+# Another user
8181+did:plc:user2def
8282+8383+did:web:coffee.example.com # Web DID example
8484+```
8585+8686+## Security Note
8787+8888+⚠️ **Important**: The `known-dids.txt` file is gitignored by default. Do not commit DIDs unless you have permission from the users.
8989+9090+For production deployments, rely on organic discovery via firehose rather than manual DID lists.
+3
.gitignore
···45454646# Other
4747*.bak
4848+4949+# Development files
5050+known-dids.txt
+8-5
BACKLOG.md
···2424 - If adding mobile apps, third-party API consumers, or microservices architecture, revisit this
2525 - For now, monolithic approach is appropriate for HTMX-based web app with decentralized storage
26262727+- Backfill seems to be called when user hits homepage, probably only needs to be done on startup
2828+2729## Fixes
28302929-- [Future work]: adjust timing of caching in feed, maybe use firehose and a sqlite database since we are only storing a few anyway
3030- - Goal: reduce pings to server when idling
3131+- After adding a bean via add brew, that bean does not show up in the drop down until after a refresh
3232+ - Happens with grinders and likely brewers also
31333232-- Non-authed home page feed shows different from what the firehose version shows (should be cached and show same db contents I think)
3333-3434-- After adding a bean via add brew, that bean does not show up in the drop down until after a refresh
3434+- Adding a grinder via the new brew page does not populate fields correctly other than the name
3535+ - Also seems to happen to brewers
3636+ - To solve this issue and the above, we likely should consolidate creation to use the same popup as the manage page uses,
3737+ since that one works, and should already be a template partial.
+32-9
CLAUDE.md
···100100### Run Development Server
101101102102```bash
103103+# Basic mode (polling-based feed)
103104go run cmd/server/main.go
104104-# or
105105+106106+# With firehose (real-time AT Protocol feed)
107107+go run cmd/server/main.go --firehose
108108+109109+# With firehose + backfill known DIDs
110110+go run cmd/server/main.go --firehose --known-dids known-dids.txt
111111+112112+# Using nix
105113nix run
106114```
107115···117125go build -o arabica cmd/server/main.go
118126```
119127128128+## Command-Line Flags
129129+130130+| Flag | Type | Default | Description |
131131+| --------------- | ------ | ------- | ----------------------------------------------------- |
132132+| `--firehose` | bool | false | Enable real-time firehose feed via Jetstream |
133133+| `--known-dids` | string | "" | Path to file with DIDs to backfill (one per line) |
134134+135135+**Known DIDs File Format:**
136136+- One DID per line (e.g., `did:plc:abc123xyz`)
137137+- Lines starting with `#` are comments
138138+- Empty lines are ignored
139139+- See `known-dids.txt.example` for reference
140140+120141## Environment Variables
121142122122-| Variable | Default | Description |
123123-| ------------------- | --------------------------------- | ---------------------------- |
124124-| `PORT` | 18910 | HTTP server port |
125125-| `SERVER_PUBLIC_URL` | - | Public URL for reverse proxy |
126126-| `ARABICA_DB_PATH` | ~/.local/share/arabica/arabica.db | BoltDB path |
127127-| `SECURE_COOKIES` | false | Set true for HTTPS |
128128-| `LOG_LEVEL` | info | debug/info/warn/error |
129129-| `LOG_FORMAT` | console | console/json |
143143+| Variable | Default | Description |
144144+| --------------------------- | --------------------------------- | ---------------------------------- |
145145+| `PORT` | 18910 | HTTP server port |
146146+| `SERVER_PUBLIC_URL` | - | Public URL for reverse proxy |
147147+| `ARABICA_DB_PATH` | ~/.local/share/arabica/arabica.db | BoltDB path (sessions, registry) |
148148+| `ARABICA_FEED_INDEX_PATH` | ~/.local/share/arabica/feed-index.db | Firehose index BoltDB path |
149149+| `ARABICA_PROFILE_CACHE_TTL` | 1h | Profile cache duration |
150150+| `SECURE_COOKIES` | false | Set true for HTTPS |
151151+| `LOG_LEVEL` | info | debug/info/warn/error |
152152+| `LOG_FORMAT` | console | console/json |
130153131154## Code Patterns
132155
+30-2
README.md
···43434444## Configuration
45454646-Environment variables:
4646+### Command-Line Flags
4747+4848+- `--firehose` - Enable real-time feed via AT Protocol Jetstream (default: false)
4949+- `--known-dids <file>` - Path to file with DIDs to backfill on startup (one per line)
5050+5151+### Environment Variables
47524853- `PORT` - Server port (default: 18910)
4954- `SERVER_PUBLIC_URL` - Public URL for reverse proxy deployments (e.g., https://arabica.example.com)
5055- `ARABICA_DB_PATH` - BoltDB path (default: ~/.local/share/arabica/arabica.db)
5656+- `ARABICA_FEED_INDEX_PATH` - Firehose index BoltDB path (default: ~/.local/share/arabica/feed-index.db)
5757+- `ARABICA_PROFILE_CACHE_TTL` - Profile cache duration (default: 1h)
5158- `OAUTH_CLIENT_ID` - OAuth client ID (optional, uses localhost mode if not set)
5259- `OAUTH_REDIRECT_URI` - OAuth redirect URI (optional)
5360- `SECURE_COOKIES` - Set to true for HTTPS (default: false)
···58655966- Track coffee brews with detailed parameters
6067- Store data in your AT Protocol Personal Data Server
6161-- Community feed of recent brews from registered users
6868+- Community feed of recent brews from registered users (polling or real-time firehose)
6269- Manage beans, roasters, grinders, and brewers
6370- Export brew data as JSON
6471- Mobile-friendly PWA design
7272+7373+### Firehose Mode
7474+7575+Enable real-time feed updates via AT Protocol's Jetstream:
7676+7777+```bash
7878+# Basic firehose mode
7979+go run cmd/server/main.go --firehose
8080+8181+# With known DIDs for backfill
8282+go run cmd/server/main.go --firehose --known-dids known-dids.txt
8383+```
8484+8585+**Known DIDs file format:**
8686+```
8787+# Comments start with #
8888+did:plc:abc123xyz
8989+did:plc:def456uvw
9090+```
9191+9292+The firehose automatically indexes **all** Arabica records across the AT Protocol network. The `--known-dids` flag allows you to backfill historical records from specific users on startup (useful for development/testing).
65936694## Architecture
6795
+74-2
cmd/server/main.go
···11package main
2233import (
44+ "bufio"
45 "context"
56 "flag"
67 "fmt"
···89 "os"
910 "os/signal"
1011 "path/filepath"
1212+ "strings"
1113 "syscall"
1214 "time"
1315···2527func main() {
2628 // Parse command-line flags
2729 useFirehose := flag.Bool("firehose", false, "Enable firehose-based feed (Jetstream consumer)")
3030+ knownDIDsFile := flag.String("known-dids", "", "Path to file containing DIDs to backfill on startup (one per line)")
2831 flag.Parse()
29323033 // Configure zerolog
···186189187190 log.Info().Msg("Firehose consumer started")
188191189189- // Backfill registered users in background
192192+ // Backfill registered users and known DIDs in background
190193 go func() {
191194 time.Sleep(5 * time.Second) // Wait for initial connection
195195+196196+ // Collect all DIDs to backfill
197197+ didsToBackfill := make(map[string]struct{})
198198+199199+ // Add registered users
192200 for _, did := range feedRegistry.List() {
201201+ didsToBackfill[did] = struct{}{}
202202+ }
203203+204204+ // Add DIDs from known-dids file if provided
205205+ if *knownDIDsFile != "" {
206206+ knownDIDs, err := loadKnownDIDs(*knownDIDsFile)
207207+ if err != nil {
208208+ log.Warn().Err(err).Str("file", *knownDIDsFile).Msg("Failed to load known DIDs file")
209209+ } else {
210210+ for _, did := range knownDIDs {
211211+ didsToBackfill[did] = struct{}{}
212212+ }
213213+ log.Info().Int("count", len(knownDIDs)).Str("file", *knownDIDsFile).Msg("Loaded known DIDs from file")
214214+ }
215215+ }
216216+217217+ // Backfill all collected DIDs
218218+ successCount := 0
219219+ for did := range didsToBackfill {
193220 if err := firehoseConsumer.BackfillDID(ctx, did); err != nil {
194221 log.Warn().Err(err).Str("did", did).Msg("Failed to backfill user")
222222+ } else {
223223+ successCount++
195224 }
196225 }
197197- log.Info().Int("count", feedRegistry.Count()).Msg("Backfill of registered users complete")
226226+ log.Info().Int("total", len(didsToBackfill)).Int("success", successCount).Msg("Backfill complete")
198227 }()
199228 }
200229···299328300329 log.Info().Msg("Server stopped")
301330}
331331+332332+// loadKnownDIDs reads a file containing DIDs (one per line) and returns them as a slice.
333333+// Lines starting with # are treated as comments and ignored.
334334+// Empty lines and whitespace are trimmed.
335335+func loadKnownDIDs(filePath string) ([]string, error) {
336336+ file, err := os.Open(filePath)
337337+ if err != nil {
338338+ return nil, fmt.Errorf("failed to open file: %w", err)
339339+ }
340340+ defer file.Close()
341341+342342+ var dids []string
343343+ scanner := bufio.NewScanner(file)
344344+ lineNum := 0
345345+346346+ for scanner.Scan() {
347347+ lineNum++
348348+ line := strings.TrimSpace(scanner.Text())
349349+350350+ // Skip empty lines and comments
351351+ if line == "" || strings.HasPrefix(line, "#") {
352352+ continue
353353+ }
354354+355355+ // Basic DID validation (must start with "did:")
356356+ if !strings.HasPrefix(line, "did:") {
357357+ log.Warn().
358358+ Str("file", filePath).
359359+ Int("line", lineNum).
360360+ Str("content", line).
361361+ Msg("Skipping invalid DID (must start with 'did:')")
362362+ continue
363363+ }
364364+365365+ dids = append(dids, line)
366366+ }
367367+368368+ if err := scanner.Err(); err != nil {
369369+ return nil, fmt.Errorf("error reading file: %w", err)
370370+ }
371371+372372+ return dids, nil
373373+}
+110
cmd/server/main_test.go
···11+package main
22+33+import (
44+ "os"
55+ "path/filepath"
66+ "testing"
77+)
88+99+func TestLoadKnownDIDs(t *testing.T) {
1010+ // Create a temporary test file
1111+ tmpDir := t.TempDir()
1212+ testFile := filepath.Join(tmpDir, "test-dids.txt")
1313+1414+ content := `# This is a comment
1515+did:plc:test123abc
1616+did:web:example.com
1717+1818+# Another comment
1919+2020+did:plc:another456def
2121+2222+# Invalid lines below
2323+not-a-did
2424+just some text
2525+2626+# Valid DID after invalid ones
2727+did:plc:final789ghi
2828+`
2929+3030+ if err := os.WriteFile(testFile, []byte(content), 0644); err != nil {
3131+ t.Fatalf("Failed to create test file: %v", err)
3232+ }
3333+3434+ // Test loading DIDs
3535+ dids, err := loadKnownDIDs(testFile)
3636+ if err != nil {
3737+ t.Fatalf("loadKnownDIDs failed: %v", err)
3838+ }
3939+4040+ // Expected DIDs
4141+ expected := []string{
4242+ "did:plc:test123abc",
4343+ "did:web:example.com",
4444+ "did:plc:another456def",
4545+ "did:plc:final789ghi",
4646+ }
4747+4848+ if len(dids) != len(expected) {
4949+ t.Errorf("Expected %d DIDs, got %d", len(expected), len(dids))
5050+ }
5151+5252+ for i, expectedDID := range expected {
5353+ if i >= len(dids) {
5454+ t.Errorf("Missing DID at index %d: %s", i, expectedDID)
5555+ continue
5656+ }
5757+ if dids[i] != expectedDID {
5858+ t.Errorf("DID at index %d: expected %s, got %s", i, expectedDID, dids[i])
5959+ }
6060+ }
6161+}
6262+6363+func TestLoadKnownDIDs_EmptyFile(t *testing.T) {
6464+ tmpDir := t.TempDir()
6565+ testFile := filepath.Join(tmpDir, "empty.txt")
6666+6767+ if err := os.WriteFile(testFile, []byte(""), 0644); err != nil {
6868+ t.Fatalf("Failed to create test file: %v", err)
6969+ }
7070+7171+ dids, err := loadKnownDIDs(testFile)
7272+ if err != nil {
7373+ t.Fatalf("loadKnownDIDs failed: %v", err)
7474+ }
7575+7676+ if len(dids) != 0 {
7777+ t.Errorf("Expected 0 DIDs from empty file, got %d", len(dids))
7878+ }
7979+}
8080+8181+func TestLoadKnownDIDs_OnlyComments(t *testing.T) {
8282+ tmpDir := t.TempDir()
8383+ testFile := filepath.Join(tmpDir, "comments.txt")
8484+8585+ content := `# Comment 1
8686+# Comment 2
8787+8888+# Comment 3
8989+`
9090+9191+ if err := os.WriteFile(testFile, []byte(content), 0644); err != nil {
9292+ t.Fatalf("Failed to create test file: %v", err)
9393+ }
9494+9595+ dids, err := loadKnownDIDs(testFile)
9696+ if err != nil {
9797+ t.Fatalf("loadKnownDIDs failed: %v", err)
9898+ }
9999+100100+ if len(dids) != 0 {
101101+ t.Errorf("Expected 0 DIDs from comments-only file, got %d", len(dids))
102102+ }
103103+}
104104+105105+func TestLoadKnownDIDs_NonexistentFile(t *testing.T) {
106106+ _, err := loadKnownDIDs("/nonexistent/path/file.txt")
107107+ if err == nil {
108108+ t.Error("Expected error for nonexistent file, got nil")
109109+ }
110110+}
···11run:
22- @LOG_LEVEL=debug LOG_FORMAT=console go run cmd/server/main.go -firehose
22+ @LOG_LEVEL=debug LOG_FORMAT=console go run cmd/server/main.go -firehose -known-dids known-dids.txt
3344run-production:
55 @LOG_FORMAT=json SECURE_COOKIES=true go run cmd/server/main.go -firehose
+18
known-dids.txt.example
···11+# Known DIDs for Development Backfill
22+#
33+# This file contains DIDs that should be backfilled on startup when using
44+# the --known-dids flag. This is useful for development and testing to
55+# populate the feed with known coffee enthusiasts.
66+#
77+# Format: One DID per line
88+# Lines starting with # are comments
99+# Empty lines are ignored
1010+#
1111+# Example DIDs (replace with real DIDs):
1212+# did:plc:example1234567890abcdef
1313+# did:plc:another1234567890abcdef
1414+#
1515+# To use this file:
1616+# 1. Copy this file to known-dids.txt
1717+# 2. Add real DIDs (one per line)
1818+# 3. Run: ./arabica --firehose --known-dids known-dids.txt