learn and share notes on atproto (wip) 🦉
malfestio.stormlightlabs.org/
readability
solid
axum
atproto
srs
docs: simplify README
* add personas and architecture
* expand AT Protocol concepts
* update project roadmap
+343
-96
+7
-52
README.md
+7
-52
README.md
···
1
1
# Malfestio
2
2
3
-
Malfestio is a learning OS: flashcards + notes + lectures + articles, designed for daily study.
3
+
Malfestio is a social learning platform: flashcards + notes + lectures + articles, designed for daily study.
4
4
5
5
Social layer: publish/share/remix learning artifacts; follow curators; discuss.
6
6
7
-
## Personas
8
-
9
-
- **Learner**: studies daily; imports content; wants fast "review queue".
10
-
- **Creator**: makes decks/notes; publishes updates; wants feedback + forks.
11
-
- **Curator/Teacher**: bundles content into learning paths; annotates lectures/articles.
12
-
- **Moderator/Community admin**: handles reports, takedowns, spam.
13
-
14
-
## Principles
15
-
16
-
- Local-first study experience; offline study must not feel "second-class".
17
-
- Shareable artifacts are portable: Lexicon-defined schemas + stable IDs.
18
-
- Privacy by design: progress + recall history are private unless explicitly
19
-
shared.
20
-
21
-
### Data Model
7
+
## Documentation
22
8
23
-
- Note: markdown + structure + citations + links to sources.
24
-
- Card: front/back (+ optional cloze, audio, image, code block).
25
-
- Deck: ordered/clustered cards (+ metadata, tags).
26
-
- Lecture: external URL + outline + timestamps + linked notes/cards.
27
-
- Article: URL + extracted text (readability style heuristics) + highlights +
28
-
linked notes/cards.
29
-
- Collection/Path: curated bundle of decks + notes + sources.
30
-
31
-
## System Architecture
32
-
33
-
### Frontend (SolidJS)
34
-
35
-
- App shell + router-driven workspaces (Library / Study / Create / Social).
36
-
- Signals as primary state primitive; keep study session state in signals/store.
37
-
38
-
### Backend (Rust)
39
-
40
-
- Axum API gateway: REST/XRPC-ish endpoints, tower middleware, typed extractors.
41
-
- Services (logical, not necessarily microservices):
42
-
- Identity/Auth service (local + optional ATProto OAuth integration)
43
-
- Content service (notes/cards/decks/sources)
44
-
- Study service (queue generation + grading + scheduling)
45
-
- Social service (follows, feeds, comments, notifications)
46
-
- Search service (indexing + query)
47
-
- Moderation service (reports, takedowns, rules)
48
-
49
-
### Storage
50
-
51
-
- Postgres: canonical app DB (users, private study state, cache of published records).
52
-
- Object storage: images/audio, extracted article snapshots (if you store them).
53
-
- Search index: separate system (Meilisearch/Typesense/ZincSearch-pick one later).
54
-
55
-
### Eventing
56
-
57
-
- Internal outbox pattern (DB table) for:
58
-
- reindex jobs, notification fanout, federation publish steps
9
+
- [Personas & Principles](./docs/personas.md) – Target users and design philosophy
10
+
- [Architecture](./docs/architecture.md) – System components and data model
11
+
- [Information Architecture](./docs/information-architecture.md) – Navigation and URL structure
12
+
- [Data Model Mapping](./docs/data-model-mapping.md) – Lexicon to database mapping
13
+
- [Roadmap](./docs/todo.md) – Development milestones
+43
docs/architecture.md
+43
docs/architecture.md
···
1
+
# System Architecture
2
+
3
+
This document describes the technical architecture of Malfestio.
4
+
5
+
## Frontend (SolidJS)
6
+
7
+
- App shell + router-driven workspaces (Library / Study / Create / Social).
8
+
- Signals as primary state primitive; keep study session state in signals/store.
9
+
10
+
## Backend (Rust)
11
+
12
+
- Axum API gateway: REST/XRPC-ish endpoints, tower middleware, typed extractors.
13
+
- Services (logical, not necessarily microservices):
14
+
- Identity/Auth service (local + optional ATProto OAuth integration)
15
+
- Content service (notes/cards/decks/sources)
16
+
- Study service (queue generation + grading + scheduling)
17
+
- Social service (follows, feeds, comments, notifications)
18
+
- Search service (indexing + query)
19
+
- Moderation service (reports, takedowns, rules)
20
+
21
+
## Storage
22
+
23
+
- Postgres: canonical app DB (users, private study state, cache of published records).
24
+
- Object storage: images/audio, extracted article snapshots (if you store them).
25
+
- Search index: separate system (Meilisearch/Typesense/ZincSearch-pick one later).
26
+
27
+
## Eventing
28
+
29
+
- Internal outbox pattern (DB table) for:
30
+
- reindex jobs, notification fanout, federation publish steps
31
+
32
+
## Data Model
33
+
34
+
See [Data Model Mapping](./data-model-mapping.md) for the mapping between public Lexicon records and internal database tables.
35
+
36
+
### Core Entities
37
+
38
+
- **Note**: markdown + structure + citations + links to sources.
39
+
- **Card**: front/back (+ optional cloze, audio, image, code block).
40
+
- **Deck**: ordered/clustered cards (+ metadata, tags).
41
+
- **Lecture**: external URL + outline + timestamps + linked notes/cards.
42
+
- **Article**: URL + extracted text (readability style heuristics) + highlights + linked notes/cards.
43
+
- **Collection/Path**: curated bundle of decks + notes + sources.
+55
-11
docs/at-notes.md
+55
-11
docs/at-notes.md
···
1
1
# AT Protocol Research Notes
2
2
3
+
Reference material for AT Protocol integration. For implementation details, see [todo.md](todo.md).
4
+
3
5
## OAuth 2.1 Specification
4
6
5
7
AT Protocol uses a specific profile of OAuth 2.1 for client↔PDS authorization.
···
44
46
45
47
Example: `at://did:plc:abc123/app.malfestio.deck/3k5abc123`
46
48
47
-
## Firehose Consumption
49
+
## Firehose / Jetstream
48
50
49
-
For social features (trending, discovery, feeds):
51
+
### Raw Firehose
50
52
51
-
- **WebSocket Connection**: Subscribe to `com.atproto.sync.subscribeRepos` from a Relay
52
-
- **CBOR Decoding**: Parse incoming events (or use Jetstream for JSON)
53
+
- **WebSocket**: Subscribe to `com.atproto.sync.subscribeRepos` from a Relay
54
+
- **CBOR Decoding**: Parse incoming events
53
55
- **Cursor Management**: Track position for reconnection
54
56
55
-
## AppView Pattern
57
+
### Jetstream (Recommended)
56
58
57
-
Index network-wide records to power discovery features:
59
+
Bluesky's simplified JSON firehose:
58
60
59
-
- Index `app.malfestio.*` records from firehose
60
-
- Implement `getFeedSkeleton` for custom algorithmic feeds
61
-
- Hydration service combines skeletons with full content from PDSes
61
+
- JSON format (no CBOR decoding)
62
+
- Reduced bandwidth (zstd compression)
63
+
- Collection/repo filtering at source
64
+
- Simpler reconnection with cursors
62
65
63
66
## Well-Known Endpoints
64
67
···
66
69
- `/.well-known/oauth-protected-resource` — PDS OAuth metadata
67
70
- `/.well-known/oauth-authorization-server` — Auth server metadata
68
71
72
+
## Labelers
73
+
74
+
**Architecture:**
75
+
76
+
1. Labels = metadata (source DID + subject AT-URI + value string)
77
+
2. User Subscription = users subscribe to labelers; clients include in API requests
78
+
3. Label Interpretation = per-user config to hide, warn, or ignore content
79
+
80
+
**Structure:**
81
+
82
+
```json
83
+
{
84
+
"src": "did:plc:labeler",
85
+
"uri": "at://did:user/app.bsky.feed.post/123",
86
+
"val": "spam",
87
+
"cts": "2026-01-01T00:00:00Z"
88
+
}
89
+
```
90
+
91
+
## Feeds
92
+
93
+
**Core Flow**:
94
+
95
+
1. User requests feed via at-uri of declared feed
96
+
2. PDS resolves at-uri → Feed Generator's DID doc
97
+
3. PDS sends `getFeedSkeleton` to service endpoint (authenticated by user's JWT)
98
+
4. Feed Generator returns skeleton (list of post URIs + cursor)
99
+
5. PDS hydrates skeleton with full content (via AppView)
100
+
6. Hydrated feed returned to user
101
+
102
+
## AppView
103
+
104
+
**Responsibilities**:
105
+
106
+
1. Record Processing & Indexing - consume firehose, build indices for likes, threads, follows
107
+
2. Moderation Enforcement - apply labels from subscribed labelers
108
+
3. Query Interface - expose XRPC API (proxied through PDS)
109
+
4. Media CDN - fetch/cache blobs from upstream PDSes, generate thumbnails
110
+
5. Search & Discovery - full-text search, type-ahead, content ranking
111
+
69
112
## Patterns from Real AT Protocol Apps
70
113
71
114
### plyr.fm (Music)
···
73
116
- OAuth 2.1 via `@atproto/oauth-client` library
74
117
- Records synced to PDS: tracks, likes, playlists
75
118
- Separate moderation service (Rust labeler)
76
-
- Data ownership: "tracks, likes, playlists synced to your PDS as ATProto records"
77
119
78
120
### leaflet.pub (Writing)
79
121
80
122
- React/Next.js frontend with Supabase + Replicache for sync
81
123
- Bluesky integration via dedicated `lexicons/` and `appview/` directories
82
-
- Publications posted to Bluesky
83
124
84
125
### wisp.place (Static Sites)
85
126
···
101
142
- [Repository & XRPC](https://atproto.com/specs/xrpc)
102
143
- [Feed Generator Starter Kit](https://github.com/bluesky-social/feed-generator)
103
144
- [atproto TypeScript SDK](https://github.com/bluesky-social/atproto)
145
+
- [Ozone Moderation Service](https://github.com/bluesky-social/ozone)
146
+
- [Jetstream Firehose](https://docs.bsky.app/blog/jetstream)
147
+
- [Labels and Moderation Guide](https://docs.bsky.app/docs/advanced-guides/moderation)
+14
docs/personas.md
+14
docs/personas.md
···
1
+
# Personas & Principles
2
+
3
+
## Personas
4
+
5
+
- **Learner**: studies daily; imports content; wants fast "review queue".
6
+
- **Creator**: makes decks/notes; publishes updates; wants feedback + forks.
7
+
- **Curator/Teacher**: bundles content into learning paths; annotates lectures/articles.
8
+
- **Moderator/Community admin**: handles reports, takedowns, spam.
9
+
10
+
## Design Principles
11
+
12
+
- Local-first study experience; offline study must not feel "second-class".
13
+
- Shareable artifacts are portable: Lexicon-defined schemas + stable IDs.
14
+
- Privacy by design: progress + recall history are private unless explicitly shared.
+224
-33
docs/todo.md
+224
-33
docs/todo.md
···
21
21
22
22
- Lexicon defines record types + XRPC endpoints; JSON-schema-like constraints.
23
23
- Use "optional fields" heavily; avoid enums that will calcify the product too early.
24
-
- Versioning: add fields, don’t rename; never rely on being able to rewrite history.
24
+
- Versioning: add fields, don't rename; never rely on being able to rewrite history.
25
25
26
26
### Schema boundaries (important)
27
27
···
45
45
- **(Done) Milestone D**: Identity + Permissions + Publishing Model.
46
46
- Auth MVP, Permission model (Private/Public/SharedWith), and basic Publishing flow implemented.
47
47
- Backend API and Frontend Editor updated with tests covering permissions and publishing.
48
+
- **(Done) Milestone E**: Internal component library/UI Foundation + Animations.
48
49
- **(Done) Milestone F**: OAuth + PDS Record Publishing.
49
50
- OAuth 2.1 client flow (PKCE, DPoP, handle/DID resolution, token refresh).
50
51
- PDS client for `putRecord`, `deleteRecord`, `uploadBlob`.
51
52
- TID generation and AT-URI builder in core crate.
52
-
- **(Done) Milestone E**: Internal component library/UI Foundation + Animations.
53
-
- **(Done) Milestone F**: Content Authoring (Notes + Cards + Deck Builder).
54
-
- **(Done) Milestone G**: Study Engine (SRS) + Daily Review UX.
53
+
- **(Done) Milestone G**: Content Authoring (Notes + Cards + Deck Builder).
54
+
- **(Done) Milestone H**: Study Engine (SRS) + Daily Review UX.
55
55
- SM-2 spaced repetition scheduler.
56
-
- **(Done) Milestone H**: Social Layer v1: Follow graph, Feeds (Follows/Trending), Forking workflow, and Threaded comments.
57
-
- **(Done) Milestone I**: Search + Discovery + Taxonomy.
56
+
- **(Done) Milestone I**: Social Layer v1: Follow graph, Feeds (Follows/Trending), Forking workflow, and Threaded comments.
58
57
- Full-text search with pg_trgm/unaccent, visibility filtering, and unified search index.
59
58
- Tag taxonomy and Discovery page with top tags.
60
59
61
-
### Milestone J - Moderation + Abuse Resistance
60
+
### Milestone J - Static Content & Landing Page
61
+
62
+
#### Deliverables
63
+
64
+
**Marketing/Static Site:**
65
+
66
+
- [ ] Landing page with hero section, feature highlights, social proof
67
+
- Hero should have a graph paper/grid background
68
+
- Floating flash cards, notes
69
+
- [ ] "How it works" section with study flow visualization
70
+
- [ ] About page with team/mission
71
+
72
+
**App Vision Content:**
73
+
74
+
- [ ] Onboarding flow with persona selection (Learner/Creator/Curator)
75
+
- [ ] Empty states with helpful prompts for new users
76
+
- [ ] Tutorial/walkthrough for first deck creation
77
+
- [ ] Help center or FAQ section -> Should mention that the app is still in development and subject to change.
78
+
79
+
**SEO & Meta:**
80
+
81
+
- [ ] Open Graph / Twitter Card meta tags
82
+
- [ ] Scripted with HTML2Canvas/PNG generation
83
+
- Graph paper background
84
+
- Floating flash cards, notes
85
+
- [ ] Sitemap.xml generation
86
+
- [ ] robots.txt configuration
87
+
88
+
#### Acceptance
89
+
90
+
- New visitors understand the value proposition within 10 seconds.
91
+
- Onboarding flow guides users to create their first deck.
92
+
93
+
### Milestone K - AppView Indexing
94
+
95
+
#### Deliverables
96
+
97
+
**Firehose Enhancement:**
98
+
99
+
- [ ] Upgrade firehose consumer to store full record content (not just metadata)
100
+
- [ ] Add `indexed_decks`, `indexed_cards`, `indexed_notes` tables for remote content
101
+
- [ ] Track latest processed revision per repo; handle deletions
102
+
103
+
**Search & Discovery:**
104
+
105
+
- [ ] Implement search over indexed remote records (extend `search_items` view)
106
+
- [ ] User profile aggregation: follower counts, deck counts from federated sources
107
+
108
+
**Export & Interop:**
109
+
110
+
- [ ] Export local records as valid Lexicon JSON (`/api/export/:collection`)
111
+
- [ ] Read-only "federated library" view showing remote decks
112
+
113
+
#### Acceptance
114
+
115
+
- A deck published from Malfestio can be discovered via another AT Protocol client.
116
+
- Remote decks from followed users appear in search results.
117
+
118
+
### Milestone L - ATProto Integration Pass
62
119
63
120
#### Deliverables
64
121
65
-
- Look into [Ozone](https://github.com/bluesky-social/ozone)
66
-
- Reporting pipeline + review queue
67
-
- Rate limits + spam heuristics
68
-
- Takedown/visibility states (shadowed, removed, quarantined)
69
-
- Audit logging for moderation actions
122
+
**Identity & Auth:**
123
+
124
+
- [ ] OAuth login directly to user's PDS (vs. local-only auth)
125
+
- [ ] Handle resolution via DNS TXT or `/.well-known/atproto-did`
126
+
- [ ] DPoP token binding for secure API calls
127
+
128
+
**Sync & Conflict Resolution:**
129
+
130
+
- [ ] Bi-directional sync: local drafts → PDS records, PDS records → local cache
131
+
- [ ] Conflict resolution strategy for concurrent edits (last-write-wins or merge UI)
132
+
- [ ] Offline queue for pending publishes
133
+
134
+
**Deep Linking:**
135
+
136
+
- [ ] AT-URI deep linking from external clients
137
+
- [ ] Handle `at://` URL scheme in app
70
138
71
139
#### Acceptance
72
140
73
-
- You can safely operate an open publishing surface.
141
+
- User can log in with their existing Bluesky/PDS identity.
142
+
- Local drafts sync correctly after reconnecting.
74
143
75
-
### Milestone K - Federation / ATProto Integration Pass
144
+
#### Implementation Details
145
+
146
+
**Considerations:**
147
+
148
+
- Scalability: substantial compute; caching, DB optimization, distributed processing
149
+
- Lexicon Validation: validate schemas, ignore invalid records gracefully
150
+
- Account State: track latest processed revision per repo; handle deletions
151
+
- Bluesky's AppView uses PostgreSQL or ScyllaDB + image proxy + AppView core
152
+
153
+
**Identity:**
154
+
155
+
- Use `did:web` for simplicity, `did:plc` for long-term stability
156
+
- ATProto OAuth is the forward path
157
+
158
+
### Milestone M - Custom Feed Generator
76
159
77
160
#### Deliverables
78
161
79
-
- Phase 1 (minimum):
80
-
- export Lexicon records
81
-
- ingest remote records into a read-only "federated library"
82
-
- Phase 2:
83
-
- OAuth login to PDS + publish records directly (client or server mediated)
84
-
- reconcile local drafts with remote published state
162
+
**Infrastructure:**
163
+
164
+
- [ ] Feed Generator service with `did:web` identity
165
+
- [ ] Publish `app.bsky.feed.generator` declaration record to creator's repo
166
+
- [ ] DID document with service endpoint for feed requests
167
+
168
+
**Endpoints:**
169
+
170
+
- [ ] `app.bsky.feed.getFeedSkeleton` - Return post URIs + cursor for pagination
171
+
- [ ] `app.bsky.feed.describeFeedGenerator` - Feed metadata (DID, name, description)
172
+
- [ ] JWT authentication for user-personalized feeds
173
+
174
+
**Algorithms:**
175
+
176
+
- [ ] "Trending Decks" - Top decks by fork/like count in last 7 days
177
+
- [ ] "New from Following" - Latest decks from followed creators
178
+
- [ ] "Study Streak Leaders" - Decks with highest completion rates (anonymized)
179
+
180
+
**Indexing:**
181
+
182
+
- [ ] Subscribe to `com.atproto.sync.subscribeRepos` (or Jetstream) for `app.malfestio.*` records
183
+
- [ ] Index posts with compound cursor (timestamp::CID) for deterministic pagination
184
+
- [ ] Garbage collect indexed data older than 48 hours (except pinned content)
85
185
86
186
#### Acceptance
87
187
88
-
- A published artifact is portable beyond your app.
188
+
- Custom feed appears in Bluesky and other AT Protocol clients.
189
+
- Feed surfaces relevant learning content based on engagement signals.
190
+
- Pagination works correctly across feed refreshes.
89
191
90
-
#### Notes
192
+
#### Implementation Details
91
193
92
-
- ATProto OAuth is the forward path; plan on it.
93
-
- XRPC endpoint patterns and legacy session behavior exist, but treat them as transitional.
194
+
**Core Flow:** see [AT Notes](./at-notes.md#feeds)
94
195
95
-
### Milestone L - Reliability, Observability, Launch
196
+
**Skeleton Response Format:**
197
+
198
+
```json
199
+
{
200
+
"feed": [
201
+
{"post": "at://did:example/app.bsky.feed.post/1"},
202
+
{"post": "at://did:example/app.bsky.feed.post/2"}
203
+
],
204
+
"cursor": "1683654690921::bafyrei..."
205
+
}
206
+
```
207
+
208
+
**Skeleton Metadata Types:**
209
+
210
+
```typescript
211
+
type SkeletonItem = {
212
+
post: string // post URI
213
+
reason?: Reason // optional context (e.g., repost)
214
+
}
215
+
type ReasonRepost = {
216
+
$type: 'app.bsky.feed.defs#skeletonReasonRepost'
217
+
repost: string // repost URI
218
+
}
219
+
```
220
+
221
+
**Considerations:**
222
+
223
+
- Validate user JWTs if feed depends on user state (follows, likes)
224
+
- Use compound cursor (timestamp::CID) for deterministic pagination
225
+
- Most feeds can garbage collect data older than 48 hours
226
+
- Reference: [Feed Generator Starter Kit](https://github.com/bluesky-social/feed-generator)
227
+
228
+
### Milestone N - Reliability, Observability, Launch
96
229
97
230
#### Deliverables
98
231
99
-
- Metrics + tracing + structured logs
100
-
- Backups + restore drills
101
-
- Load test targets (study session + feed + search)
102
-
- Beta program + feedback loop + roadmap iteration
232
+
**Observability:**
103
233
104
-
## Open Questions (Parked Decisions)
234
+
- [ ] Structured logging with correlation IDs
235
+
- [ ] Metrics collection (Prometheus/OpenTelemetry)
236
+
- [ ] Distributed tracing for request flows
237
+
- [ ] Error tracking (Sentry or similar)
105
238
106
-
- Local-first mechanics: full offline authoring + later publish, or online-only creation?
107
-
- Federation depth: read-only ingest first, or publish-to-PDS in the first public beta?
108
-
- Content extraction: store extracted article snapshots (legal/ops implications), or store only metadata + highlights?
239
+
**Reliability:**
240
+
241
+
- [ ] Database backups + restore drills
242
+
- [ ] Health check endpoints (`/health`, `/ready`)
243
+
- [ ] Graceful shutdown handling
244
+
- [ ] Circuit breakers for external dependencies
245
+
246
+
**Load Testing:**
247
+
248
+
- [ ] Study session throughput targets
249
+
- [ ] Feed generation latency benchmarks
250
+
- [ ] Search query performance under load
251
+
252
+
**Launch Prep:**
253
+
254
+
- [ ] Beta program signup flow
255
+
- [ ] Feedback collection mechanism
256
+
- [ ] Feature flags for gradual rollout
257
+
258
+
#### Acceptance
259
+
260
+
- System handles 10x expected load without degradation.
261
+
- Mean time to recovery < 5 minutes for common failures.
262
+
263
+
### Milestone O - Moderation + Abuse Resistance
264
+
265
+
#### Deliverables
266
+
267
+
**Labeler Infrastructure:**
268
+
269
+
- [ ] Dedicated Bluesky account for labeler service
270
+
- [ ] Publish `app.bsky.labeler.service` record to make discoverable
271
+
- [ ] Self-host Ozone backend + UI (Docker setup in `HOSTING.md`)
272
+
- [ ] Configure report types via `goat` CLI
273
+
274
+
**Endpoints:**
275
+
276
+
- [ ] `com.atproto.label.subscribeLabels` - real-time label stream
277
+
- [ ] `com.atproto.label.queryLabels` - query published labels
278
+
- [ ] `com.atproto.report.createReport` - accept user reports
279
+
280
+
**Moderation Features:**
281
+
282
+
- [ ] Reporting pipeline + review queue UI
283
+
- [ ] Rate limits + spam heuristics
284
+
- [ ] Takedown/visibility states (shadowed, removed, quarantined)
285
+
- [ ] Audit logging for moderation actions
286
+
287
+
#### Acceptance
288
+
289
+
- You can safely operate an open publishing surface.
290
+
- Users can subscribe to your labeler and see moderation applied.
291
+
292
+
**Reference:** [Ozone Moderation Service](https://github.com/bluesky-social/ozone)
293
+
294
+
## Open Question/Parked Decisions
295
+
296
+
- Full offline authoring + later publish
297
+
- Federation depth: publish-to-PDS in the first public beta
298
+
- Content extraction: store extracted article snapshots locally (browser)
299
+
- Persist only metadata + highlights