a love letter to tangled (android, iOS, and a search API)
1---
2title: Data Sources & Integration
3updated: 2026-03-24
4---
5
6Twisted pulls data from four external sources and authenticates users via Bluesky OAuth. Each source has a distinct role — no single source is authoritative for everything.
7
8## Source Overview
9
10| Source | What it provides | Access pattern |
11| ------------------------ | ------------------------------------------------------------------------------ | ---------------------------------------------------------- |
12| **Tangled XRPC (Knots)** | Git data — file trees, blobs, commits, branches, diffs, tags | Direct XRPC calls to the knot hosting each repo |
13| **AT Protocol (PDS)** | User records — profiles, repos, issues, PRs, comments, stars, follows | `com.atproto.repo.getRecord` / `listRecords` on user's PDS |
14| **Constellation** | Social signals — star counts, follower counts, reaction counts, backlink lists | Public JSON API at `constellation.microcosm.blue` |
15| **Tap** | Real-time firehose of AT Protocol record events for indexing | WebSocket consumer, feeds our search index |
16
17## Constellation
18
19[Constellation](https://constellation.microcosm.blue) is a public, self-hosted index of AT Protocol backlinks. It answers "who linked to this?" across the entire network — making it the right source for aggregated social signals instead of maintaining our own counters.
20
21### Key Endpoints
22
23**`GET /xrpc/blue.microcosm.links.getBacklinks`** — Get records linking to a target.
24
25- `subject` (required) — The target (AT-URI, DID, or URL)
26- `source` (required) — Collection and path, e.g. `sh.tangled.feed.star:subject.uri`
27- `did` — Filter to specific users (repeatable)
28- `limit` — Default 16, max 100
29- `reverse` — Reverse ordering
30
31**`GET /xrpc/blue.microcosm.links.getBacklinksCount`** — Count of links to a target.
32
33- `subject`, `source` — Same as above
34
35**`GET /xrpc/blue.microcosm.links.getManyToManyCounts`** — Secondary link counts in many-to-many relationships.
36
37- `subject`, `source`, `pathToOther` (required)
38- `did`, `otherSubject`, `limit` (optional)
39
40### Usage in Twisted
41
42| Need | Constellation call |
43| ------------------------- | ---------------------------------------------------------------------------------------- |
44| Star count for a repo | `getBacklinksCount(subject=repo_at_uri, source=sh.tangled.feed.star:subject.uri)` |
45| Who starred a repo | `getBacklinks(subject=repo_at_uri, source=sh.tangled.feed.star:subject.uri)` |
46| Follower count for a user | `getBacklinksCount(subject=user_did, source=sh.tangled.graph.follow:subject)` |
47| Who follows a user | `getBacklinks(subject=user_did, source=sh.tangled.graph.follow:subject)` |
48| Reaction count on content | `getBacklinksCount(subject=content_at_uri, source=sh.tangled.feed.reaction:subject.uri)` |
49
50This replaces the need to index and count interaction records ourselves. Our Tap pipeline still indexes interaction records for search and graph discovery, but Constellation is the source of truth for counts and lists.
51
52### Integration Notes
53
54- No authentication required. Constellation asks for a user-agent header with project name and contact.
55- Responses are paginated via cursor. Plan for multiple pages when listing (e.g., all followers).
56- The API is read-only — social actions (star, follow, react) are still AT Protocol record writes to the user's PDS.
57
58## Tangled XRPC (Knots)
59
60Knots are Tangled's git hosting servers. Each repo lives on a specific knot, identified by the knot DID in the repo's AT Protocol record.
61
62### Endpoints Used
63
64- `sh.tangled.repo.tree` — File tree for a ref
65- `sh.tangled.repo.blob` — File content
66- `sh.tangled.repo.log` — Commit history
67- `sh.tangled.repo.branches` / `sh.tangled.repo.tags` — Refs
68- `sh.tangled.repo.getDefaultBranch` — Default branch name
69- `sh.tangled.repo.diff` / `sh.tangled.repo.compare` — Diffs
70- `sh.tangled.repo.languages` — Language breakdown
71- `sh.tangled.knot.version` — Knot software version
72
73### Routing
74
75The app resolves which knot hosts a repo by reading the repo's AT Protocol record (which contains the knot DID), then resolving the knot DID to its service endpoint. XRPC calls go directly to that knot.
76
77The Tangled appview at `tangled.org` serves HTML only — there is no JSON API at the appview level.
78
79## AT Protocol (PDS)
80
81Standard AT Protocol record access for reading and writing user data.
82
83### Read Operations
84
85- `com.atproto.repo.getRecord` — Fetch a single record by collection + rkey
86- `com.atproto.repo.listRecords` — List records in a collection with pagination
87
88Used for: profiles, repo metadata, issues, PRs, comments, stars, follows, reactions.
89
90### Write Operations (Authenticated)
91
92- `com.atproto.repo.createRecord` — Create a new record (star, follow, react, issue, comment)
93- `com.atproto.repo.deleteRecord` — Delete a record (unstar, unfollow)
94
95All writes go to the authenticated user's PDS using their OAuth session.
96
97### Identity Resolution
98
99- Handle → DID via `com.atproto.identity.resolveHandle`
100- DID → DID document via PLC Directory (`plc.directory`) or `.well-known/did.json`
101- DID document → PDS endpoint (from `#atprotoPersonalDataServer` service)
102
103## Tap (Firehose)
104
105Tap provides a filtered firehose of AT Protocol events. Our indexer consumes Tap via WebSocket, indexing records into the search database.
106
107### What We Index via Tap
108
109- Repos, issues, PRs, comments, strings, profiles — for full-text search
110- Follows — for graph discovery during backfill
111- Issue state and PR status changes — for state filtering in search
112
113### What We Don't Need to Count via Tap
114
115Stars, followers, reactions — Constellation handles counts and lists. We still process these events for graph discovery but don't need to maintain our own counters.
116
117### Tap Protocol
118
119- WebSocket connection with cursor-based resume
120- Events contain: operation (create/update/delete), DID, collection, rkey, CID, record payload
121- Acks required after processing each event
122- Backfill via `/repos/add` endpoint to request historical data for specific users
123
124## Bluesky OAuth
125
126Authentication uses AT Protocol OAuth via `@atcute/oauth-browser-client`.
127
128### Flow
129
1301. User enters their handle
1312. App resolves handle → DID → PDS → authorization server metadata
1323. App initiates OAuth with requested scopes
1334. User authorizes in browser, redirected back to app
1345. App exchanges code for tokens
1356. Session provides `dpopFetch` for authenticated XRPC calls
136
137### Scopes
138
139The app requests scopes for:
140
141- `sh.tangled.feed.star` — Star/unstar repos
142- `sh.tangled.graph.follow` — Follow/unfollow users
143- `sh.tangled.feed.reaction` — Add reactions
144- `sh.tangled.actor.profile` — Edit profile
145- `sh.tangled.repo.issue` / `sh.tangled.repo.issue.comment` — Create issues and comments
146- `sh.tangled.repo.pull.comment` — Comment on PRs
147
148### Capacitor Integration
149
150On native platforms, OAuth callback uses a deep link URL scheme registered with Capacitor. The app listens via `App.addListener('appUrlOpen', ...)` to catch the redirect.
151
152### Session Management
153
154Tokens are stored in secure storage (encrypted localStorage on web, Capacitor Secure Storage on native). Sessions auto-refresh. The app supports multiple accounts with an account switcher.