Coffee journaling on ATProto (alpha)
alpha.arabica.social
coffee
1# Microcosm Tools Evaluation for Arabica
2
3## Executive Summary
4
5This document evaluates three community-built AT Protocol infrastructure tools from [microcosm.blue](https://microcosm.blue/) - **Constellation**, **Spacedust**, and **Slingshot** - for potential integration with Arabica's community feed feature.
6
7**Recommendation:** Adopt Constellation immediately for future social features (likes/comments). Consider Slingshot as an optional optimization for feed performance. Spacedust is ideal for real-time notifications when social features are implemented.
8
9---
10
11## Background: Current Arabica Architecture
12
13### The Problem
14
15Arabica's community feed (`internal/feed/service.go`) currently polls each registered user's PDS directly. For N registered users:
16
17| API Call Type | Count per Refresh |
18|---------------|-------------------|
19| Profile fetches | N |
20| Brew collections | N |
21| Bean collections | N |
22| Roaster collections | N |
23| Grinder collections | N |
24| Brewer collections | N |
25| Reference resolution | ~4N |
26| **Total** | **~10N API calls** |
27
28This approach has several issues:
29- **Latency**: Feed refresh is slow with many users
30- **Rate limits**: Risk of PDS rate limiting
31- **Reliability**: Feed fails if any PDS is slow/down
32- **Scalability**: Linear growth in API calls per user
33
34### Future Social Features
35
36Arabica plans to add likes, comments, and follows (see `AGENTS.md`). These interactions require **backlink queries** - given a brew, find all likes pointing at it. This is impossible with current polling approach.
37
38---
39
40## Tool 1: Constellation (Backlink Index)
41
42### What It Is
43
44Constellation is a **global backlink index** that crawls every record in the AT Protocol firehose and indexes all links (AT-URIs, DIDs, URLs). It answers "who/what points at this target?" queries.
45
46**Public Instance:** `https://constellation.microcosm.blue`
47
48### Key Capabilities
49
50| Feature | Description |
51|---------|-------------|
52| Backlink queries | Find all records linking to a target |
53| Like/follow counts | Get interaction counts instantly |
54| Any lexicon support | Works with `social.arabica.alpha.*` |
55| DID filtering | Filter links by specific users |
56| Distinct DID counts | Count unique users, not just records |
57
58### API Examples
59
60**Get like count for a brew:**
61```bash
62curl "https://constellation.microcosm.blue/links/count/distinct-dids" \
63 -G --data-urlencode "target=at://did:plc:xxx/social.arabica.alpha.brew/abc123" \
64 --data-urlencode "collection=social.arabica.alpha.like" \
65 --data-urlencode "path=.subject.uri"
66```
67
68**Get all users who liked a brew:**
69```bash
70curl "https://constellation.microcosm.blue/links/distinct-dids" \
71 -G --data-urlencode "target=at://did:plc:xxx/social.arabica.alpha.brew/abc123" \
72 --data-urlencode "collection=social.arabica.alpha.like" \
73 --data-urlencode "path=.subject.uri"
74```
75
76**Get all comments on a brew:**
77```bash
78curl "https://constellation.microcosm.blue/links" \
79 -G --data-urlencode "target=at://did:plc:xxx/social.arabica.alpha.brew/abc123" \
80 --data-urlencode "collection=social.arabica.alpha.comment" \
81 --data-urlencode "path=.subject.uri"
82```
83
84### How Arabica Could Use Constellation
85
86**Use Case 1: Social Interaction Counts**
87
88When displaying a brew in the feed, fetch interaction counts:
89
90```go
91// Get like count for a brew
92func (c *ConstellationClient) GetLikeCount(ctx context.Context, brewURI string) (int, error) {
93 url := fmt.Sprintf("%s/links/count/distinct-dids?target=%s&collection=%s&path=%s",
94 c.baseURL,
95 url.QueryEscape(brewURI),
96 "social.arabica.alpha.like",
97 url.QueryEscape(".subject.uri"))
98
99 // Returns {"total": 42}
100 var result struct { Total int `json:"total"` }
101 // ... fetch and decode
102 return result.Total, nil
103}
104```
105
106**Use Case 2: Comment Threads**
107
108Fetch all comments for a brew detail page:
109
110```go
111func (c *ConstellationClient) GetComments(ctx context.Context, brewURI string) ([]Comment, error) {
112 // Constellation returns the AT-URIs of comment records
113 // Then fetch each comment from Slingshot or user's PDS
114}
115```
116
117**Use Case 3: "Who liked this" List**
118
119```go
120func (c *ConstellationClient) GetLikers(ctx context.Context, brewURI string) ([]string, error) {
121 // Returns list of DIDs who liked this brew
122 // Can hydrate with profile info from Slingshot
123}
124```
125
126### Constellation Tradeoffs
127
128| Pros | Cons |
129|------|------|
130| Instant interaction counts (no polling) | Third-party dependency |
131| Works with any lexicon including Arabica's | Not self-hosted (yet) |
132| Handles likes from any PDS globally | Slight index delay (~seconds) |
133| 11B+ links indexed, production-ready | Trusts Constellation operator |
134| Free public instance | Query limits may apply |
135
136### Constellation Verdict
137
138**Essential for:** Social features (likes, comments, follows)
139
140**Not needed for:** Current feed polling (Constellation indexes interactions, not record listings)
141
142**Effort estimate:** Low (1 week)
143- Add HTTP client for Constellation API
144- Integrate counts into brew display
145- Cache counts locally (5-minute TTL)
146
147---
148
149## Tool 2: Spacedust (Interactions Firehose)
150
151### What It Is
152
153Spacedust extracts **links** from every record in the AT Protocol firehose and re-emits them over WebSocket. Unlike Jetstream (which emits full records), Spacedust emits just the link relationships.
154
155**Public Instance:** `wss://spacedust.microcosm.blue`
156
157### Key Capabilities
158
159| Feature | Description |
160|---------|-------------|
161| Real-time link events | Instantly know when someone likes/follows |
162| Filter by source/target | Subscribe to specific collections or targets |
163| Any lexicon support | Works with `social.arabica.alpha.*` |
164| Lightweight | Just links, not full records |
165
166### Example: Subscribe to Likes on Your Brews
167
168```javascript
169// WebSocket connection to Spacedust
170const ws = new WebSocket(
171 "wss://spacedust.microcosm.blue/subscribe" +
172 "?wantedSources=social.arabica.alpha.like:subject.uri" +
173 "&wantedSubjects=did:plc:your-did"
174);
175
176ws.onmessage = (event) => {
177 const link = JSON.parse(event.data);
178 // { source: "at://...", target: "at://...", ... }
179 console.log("Someone liked your brew!");
180};
181```
182
183### How Arabica Could Use Spacedust
184
185**Use Case: Real-time Notifications**
186
187When social features are added, Spacedust enables instant notifications:
188
189```go
190// Background goroutine subscribes to Spacedust
191func (s *NotificationService) subscribeToInteractions(userDID string) {
192 ws := dial("wss://spacedust.microcosm.blue/subscribe" +
193 "?wantedSources=social.arabica.alpha.like:subject.uri" +
194 "&wantedSubjects=" + userDID)
195
196 for {
197 link := readLink(ws)
198 // Someone liked a brew by userDID
199 s.notify(userDID, "Someone liked your brew!")
200 }
201}
202```
203
204**Use Case: Live Feed Updates**
205
206Push new brews to connected clients without polling:
207
208```go
209// Subscribe to all Arabica brew creations
210ws := dial("wss://spacedust.microcosm.blue/subscribe" +
211 "?wantedSources=social.arabica.alpha.brew:beanRef")
212
213// When a link event arrives, a new brew was created
214// Fetch full record from Slingshot and push to feed
215```
216
217### Spacedust Tradeoffs
218
219| Pros | Cons |
220|------|------|
221| Real-time, sub-second latency | Requires persistent WebSocket |
222| Lightweight link-only events | Still in v0 (missing some features) |
223| Filter by collection/target | No cursor replay yet |
224| Perfect for notifications | Need to hydrate records separately |
225
226### Spacedust Verdict
227
228**Ideal for:** Real-time notifications, live feed updates
229
230**Not suitable for:** Current feed needs (need full records, not just links)
231
232**Effort estimate:** Medium (2-3 weeks)
233- WebSocket client with reconnection
234- Notification service for social interactions
235- Integration with frontend for live updates
236- Depends on social features being implemented first
237
238---
239
240## Tool 3: Slingshot (Records & Identities Cache)
241
242### What It Is
243
244Slingshot is an **edge cache** for AT Protocol records and identities. It pre-caches records from the firehose and provides fast, authenticated access. Also resolves handles to DIDs with bi-directional verification.
245
246**Public Instance:** `https://slingshot.microcosm.blue`
247
248### Key Capabilities
249
250| Feature | Description |
251|---------|-------------|
252| Fast record fetching | Pre-cached from firehose |
253| Identity resolution | `resolveMiniDoc` for handle/DID |
254| Bi-directional verification | Only returns verified handles |
255| Works with slow PDS | Cache serves even if PDS is down |
256| Standard XRPC API | Drop-in replacement for PDS calls |
257
258### API Examples
259
260**Resolve identity:**
261```bash
262curl "https://slingshot.microcosm.blue/xrpc/com.bad-example.identity.resolveMiniDoc?identifier=bad-example.com"
263# Returns: { "did": "did:plc:...", "handle": "bad-example.com", "pds": "https://..." }
264```
265
266**Get record (standard XRPC):**
267```bash
268curl "https://slingshot.microcosm.blue/xrpc/com.atproto.repo.getRecord?repo=did:plc:xxx&collection=social.arabica.alpha.brew&rkey=abc123"
269```
270
271**List records:**
272```bash
273curl "https://slingshot.microcosm.blue/xrpc/com.atproto.repo.listRecords?repo=did:plc:xxx&collection=social.arabica.alpha.brew&limit=10"
274```
275
276### How Arabica Could Use Slingshot
277
278**Use Case 1: Faster Feed Fetching**
279
280Replace direct PDS calls with Slingshot for public data:
281
282```go
283// Before: Each user's PDS
284pdsEndpoint, _ := c.GetPDSEndpoint(ctx, did)
285url := fmt.Sprintf("%s/xrpc/com.atproto.repo.listRecords...", pdsEndpoint)
286
287// After: Single Slingshot endpoint
288url := fmt.Sprintf("https://slingshot.microcosm.blue/xrpc/com.atproto.repo.listRecords...")
289```
290
291**Benefits:**
292- Eliminates N DNS lookups for N user PDS endpoints
293- Single, fast endpoint for all public record fetches
294- Continues working even if individual PDS is slow/down
295- Pre-cached records = faster response times
296
297**Use Case 2: Identity Resolution**
298
299Replace multiple API calls with single `resolveMiniDoc`:
300
301```go
302// Before: Two calls
303handle := resolveHandle(did) // Call 1
304pds := resolvePDSEndpoint(did) // Call 2
305
306// After: One call
307mini := resolveMiniDoc(did)
308// { handle: "user.bsky.social", pds: "https://...", did: "did:plc:..." }
309```
310
311**Use Case 3: Hydrate Records from Constellation**
312
313When Constellation returns AT-URIs (e.g., comments on a brew), fetch the actual records from Slingshot:
314
315```go
316// Constellation returns: ["at://did:plc:a/social.arabica.alpha.comment/123", ...]
317commentURIs := constellation.GetComments(ctx, brewURI)
318
319// Fetch each comment record from Slingshot
320for _, uri := range commentURIs {
321 record := slingshot.GetRecord(ctx, uri)
322 // ...
323}
324```
325
326### Implementation: Slingshot-Backed PublicClient
327
328```go
329// internal/atproto/slingshot_client.go
330
331const SlingshotBaseURL = "https://slingshot.microcosm.blue"
332
333type SlingshotClient struct {
334 baseURL string
335 httpClient *http.Client
336}
337
338func NewSlingshotClient() *SlingshotClient {
339 return &SlingshotClient{
340 baseURL: SlingshotBaseURL,
341 httpClient: &http.Client{Timeout: 10 * time.Second},
342 }
343}
344
345// ListRecords uses Slingshot instead of user's PDS
346func (c *SlingshotClient) ListRecords(ctx context.Context, did, collection string, limit int) (*PublicListRecordsOutput, error) {
347 // Same XRPC API, different endpoint
348 url := fmt.Sprintf("%s/xrpc/com.atproto.repo.listRecords?repo=%s&collection=%s&limit=%d&reverse=true",
349 c.baseURL, url.QueryEscape(did), url.QueryEscape(collection), limit)
350 // ... standard HTTP request
351}
352
353// ResolveMiniDoc gets handle + PDS in one call
354func (c *SlingshotClient) ResolveMiniDoc(ctx context.Context, identifier string) (*MiniDoc, error) {
355 url := fmt.Sprintf("%s/xrpc/com.bad-example.identity.resolveMiniDoc?identifier=%s",
356 c.baseURL, url.QueryEscape(identifier))
357 // ... returns { did, handle, pds }
358}
359```
360
361### Slingshot Tradeoffs
362
363| Pros | Cons |
364|------|------|
365| Faster than direct PDS calls | Third-party dependency |
366| Single endpoint for all users | May not have custom lexicons cached |
367| Identity verification built-in | Not all XRPC APIs implemented |
368| Resilient to slow/down PDS | Trusts Slingshot operator |
369| Pre-cached from firehose | Still in v0, some features missing |
370
371### Slingshot Verdict
372
373**Recommended for:** Feed performance optimization, identity resolution
374
375**Not suitable for:** Authenticated user operations (still need direct PDS)
376
377**Effort estimate:** Low (3-5 days)
378- Add SlingshotClient as optional PublicClient backend
379- Feature flag to toggle between direct PDS and Slingshot
380- Test with Arabica collections to ensure they're indexed
381
382---
383
384## Comparison: Current vs. Microcosm Tools
385
386| Aspect | Current Polling | + Slingshot | + Constellation | + Spacedust |
387|--------|-----------------|-------------|-----------------|-------------|
388| Feed refresh latency | Slow (N PDS calls) | Fast (1 endpoint) | N/A | Real-time |
389| Like/comment counts | Impossible | Impossible | Instant | N/A |
390| Rate limit risk | High | Low | Low | None |
391| PDS failure resilience | Poor | Good | N/A | N/A |
392| Real-time updates | No (5min cache) | No | No | Yes |
393| Effort to integrate | N/A | Low | Low | Medium |
394
395---
396
397## Recommendation
398
399### Immediate (Social Features Prerequisite)
400
401**1. Integrate Constellation when adding likes/comments**
402
403Constellation is essential for social features. When a brew is displayed, use Constellation to:
404- Show like count
405- Show comment count
406- Power "who liked this" lists
407- Power comment threads
408
409**Implementation priority:** Do this alongside `social.arabica.alpha.like` and `social.arabica.alpha.comment` lexicon implementation.
410
411### Short Term (Performance Optimization)
412
413**2. Evaluate Slingshot for feed performance**
414
415If feed latency becomes an issue:
416- Add SlingshotClient as alternative to direct PDS calls
417- A/B test performance improvement
418- Use for public record fetches only (keep direct PDS for authenticated writes)
419
420**Trigger:** When registered users exceed ~20-30, or feed refresh exceeds 5 seconds
421
422### Medium Term (Real-time Features)
423
424**3. Add Spacedust for notifications**
425
426When social features are live and users want notifications:
427- Subscribe to Spacedust for likes/comments on user's content
428- Push notifications via WebSocket to connected clients
429- Optional: background job for email notifications
430
431**Trigger:** After social features launch, when users request notifications
432
433---
434
435## Comparison with Official Tools (Jetstream/Tap)
436
437See `jetstream-tap-evaluation.md` for official Bluesky tools. Key differences:
438
439| Aspect | Microcosm Tools | Official Tools |
440|--------|-----------------|----------------|
441| Focus | Links/interactions | Full records |
442| Backlink queries | Constellation (yes) | Not available |
443| Record caching | Slingshot | Not available |
444| Real-time | Spacedust (links) | Jetstream (records) |
445| Self-hosting | Not yet documented | Available |
446| Community | Community-supported | Bluesky-supported |
447
448**Recommendation:** Use Microcosm tools for social features (likes/comments/follows) where backlink queries are essential. Consider Jetstream for full feed real-time if needed later.
449
450---
451
452## Implementation Plan
453
454### Phase 1: Constellation Integration (with social features)
455
456```
4571. Create internal/atproto/constellation.go
458 - ConstellationClient with HTTP client
459 - GetBacklinks(), GetLinkCount(), GetDistinctDIDs()
460
4612. Create internal/social/interactions.go
462 - GetBrewLikeCount(brewURI)
463 - GetBrewComments(brewURI)
464 - GetBrewLikers(brewURI)
465
4663. Update templates to show interaction counts
467 - Modify feed item display
468 - Add like button (when like lexicon ready)
469```
470
471### Phase 2: Slingshot Optimization (optional)
472
473```
4741. Create internal/atproto/slingshot.go
475 - SlingshotClient implementing same interface as PublicClient
476
4772. Add feature flag: ARABICA_USE_SLINGSHOT=true
478
4793. Modify feed/service.go to use SlingshotClient
480 - Keep PublicClient as fallback
481```
482
483### Phase 3: Spacedust Notifications (future)
484
485```
4861. Create internal/notifications/spacedust.go
487 - WebSocket client with reconnection
488 - Subscribe to user's content interactions
489
4902. Create notification storage (BoltDB)
491
4923. Add /api/notifications endpoint for frontend polling
493
4944. Optional: WebSocket to frontend for real-time
495```
496
497---
498
499## Related Documentation
500
501- Microcosm Main: https://microcosm.blue/
502- Constellation API: https://constellation.microcosm.blue/
503- Source Code: https://github.com/at-microcosm/microcosm-rs
504- Discord: https://discord.gg/tcDfe4PGVB
505- See also: `jetstream-tap-evaluation.md` for official Bluesky tools