A social RSS reader built on the AT Protocol. glean.at
glean atproto atmosphere rss feed social app
14
fork

Configure Feed

Select the types of activity you want to include in your feed.

Improve docs

+68 -71
+3 -3
.env.example
··· 3 3 GLEAN_SESSION_KEY=change-me-to-a-random-string 4 4 GLEAN_JETSTREAM=wss://jetstream.glean.at 5 5 GLEAN_PLC_URL=https://didplc.glean.at 6 - GLEAN_SYNC_INTERVAL=10m 7 - GLEAN_CLUSTER_INTERVAL=15m 8 - GLEAN_FETCH_INTERVAL=5m 6 + GLEAN_SYNC_INTERVAL=30m 7 + GLEAN_CLUSTER_INTERVAL=60m 8 + GLEAN_FETCH_INTERVAL=15m 9 9 GLEAN_COLLECTION_DIR_URL=https://lightrail.microcosm.blue/xrpc/com.atproto.sync.listReposByCollection?collection=at.glean.subscription 10 10 GLEAN_BACKFILL_CONCURRENCY=5 11 11 # Leave empty for localhost OAuth (development)
+1 -2
docs/design.md
··· 167 167 | Article Detail | `max-w-3xl` centered | Content, like/share/read buttons, annotations | 168 168 | Feeds | 2/3 + 1/3 grid | Feed list with categories + add/import sidebar, refresh button | 169 169 | Trending | Full-width list | Like/annotation counts on each article | 170 - | Discover | Mixed grid | Recommendations + people + browse all | 171 - | Annotations | Full-width list | Filter by article URL, load more | 170 + | Library | Full-width list | Liked articles and annotations | 172 171 | Profile | `max-w-2xl` centered | Avatar, stats, feeds, annotations |
+47 -49
docs/specs.md
··· 298 298 299 299 The scheduler uses a configurable tick interval with in-flight deduplication: 300 300 301 - - **Tick interval**: The scheduler checks for stale feeds every `GLEAN_FETCH_INTERVAL` (default 5 minutes) 301 + - **Tick interval**: The scheduler checks for stale feeds every `GLEAN_FETCH_INTERVAL` (default 15 minutes) 302 302 - **Staleness threshold**: Feeds not fetched in the last 30 minutes are eligible 303 303 - **Subscriber filter**: Only feeds with `subscriber_count > 0` are fetched 304 304 - **In-flight dedup**: If a feed is already being fetched (e.g., manual refresh and background scheduler overlap), the second caller waits for the first to complete rather than fetching again ··· 395 395 396 396 - **Auto-discovery**: When fetching a feed, parse `<link rel="alternate" type="application/rss+xml">` from the feed's site URL to discover related feeds 397 397 - **Feedfavicon**: Fetch `favicon.ico` or `/apple-touch-icon.png` from the feed's site URL for display 398 - - **Dead feed detection**: If a feed fails for 7 consecutive fetches (14 days at base interval), mark it as dead. Notify the user and offer to remove it. 398 + - **Dead feed detection**: If a feed fails for 7 consecutive fetches, mark it as dead. Notify the user and offer to remove it. 399 399 400 400 ## 5. System Architecture 401 401 ··· 668 668 For any two users, compute Jaccard over their subscription sets, plus like co-occurrence (time-decayed) and tag overlap: 669 669 670 670 ``` 671 - J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + follow_boost 671 + J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + 0.5 * follow_boost 672 672 ``` 673 673 674 674 Like overlap uses exponential time decay: `EXP(-0.023 * age_days)` (30-day half-life). ··· 818 818 819 819 ### 8.1 Pages 820 820 821 - | Route | Method | Description | 822 - | ------------------------------ | ------ | -------------------------------------------------------- | 823 - | `/` | GET | Landing page / auth redirect | 824 - | `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar | 825 - | `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) | 826 - | `/feeds/list` | GET | Feed list fragment (htmx partial) | 827 - | `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions | 828 - | `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) | 829 - | `/feeds/add` | POST | Add a single feed URL | 830 - | `/feeds/remove` | DELETE | Remove a feed | 831 - | `/feeds/refresh` | POST | Refresh all subscribed feeds | 832 - | `/feeds/retry` | POST | Retry a failed feed | 833 - | `/feeds/clear` | POST | Clear all subscriptions | 834 - | `/feeds/dismiss` | POST | Dismiss a feed recommendation | 835 - | `/articles` | GET | Read articles (paginated, filterable by feed) | 836 - | `/articles/new-count` | GET | Get count of new articles (for badge updates) | 837 - | `/articles/{id}` | GET | Article detail view | 838 - | `/articles/{id}/read` | POST | Mark article as read | 839 - | `/articles/{id}/unread` | POST | Mark article as unread | 840 - | `/articles/{id}/like` | POST | Like an article | 841 - | `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL | 842 - | `/articles/mark-all-read` | POST | Mark all articles as read | 843 - | `/articles/dismiss` | POST | Dismiss an article recommendation | 844 - | `/trending` | GET | Community feed: articles ranked by likes | 845 - | `/library` | GET | Liked articles and annotations | 846 - | `/library/create` | POST | Create annotation on an article | 847 - | `/library/{id}/delete` | POST | Delete an annotation | 848 - | `/profile/{did}` | GET | Public profile: their feeds, likes, annotations | 821 + | Route | Method | Description | 822 + | ------------------------------ | ------ | ------------------------------------------------------------------- | 823 + | `/` | GET | Landing page / auth redirect | 824 + | `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar | 825 + | `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) | 826 + | `/feeds/list` | GET | Feed list fragment (htmx partial) | 827 + | `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions (redirects to /feeds) | 828 + | `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) | 829 + | `/feeds/add` | POST | Add a single feed URL | 830 + | `/feeds/remove` | DELETE | Remove a feed | 831 + | `/feeds/refresh` | POST | Refresh all subscribed feeds | 832 + | `/feeds/retry` | POST | Retry a failed feed | 833 + | `/feeds/clear` | POST | Clear all subscriptions | 834 + | `/feeds/dismiss` | POST | Dismiss a feed recommendation | 835 + | `/articles` | GET | Read articles (paginated, filterable by feed) | 836 + | `/articles/new-count` | GET | Get count of new articles (for badge updates) | 837 + | `/articles/{id}` | GET | Article detail view | 838 + | `/articles/{id}/read` | POST | Mark article as read | 839 + | `/articles/{id}/unread` | POST | Mark article as unread | 840 + | `/articles/{id}/like` | POST | Like an article | 841 + | `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL | 842 + | `/articles/mark-all-read` | POST | Mark all articles as read | 843 + | `/articles/dismiss` | POST | Dismiss an article recommendation | 844 + | `/trending` | GET | Community feed: articles ranked by likes | 845 + | `/library` | GET | Liked articles and annotations | 846 + | `/library/create` | POST | Create annotation on an article | 847 + | `/library/{id}/delete` | POST | Delete an annotation | 848 + | `/profile/{did}` | GET | Public profile: their feeds, likes, annotations | 849 849 850 850 ### 8.2 htmx Patterns 851 851 ··· 881 881 │ │ ├── sync.go # PDS record reconciliation 882 882 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints) 883 883 │ ├── db/ 884 - │ │ ├── db.go # SQLite connection, single-DB schema 885 - │ │ ├── multi.go # Multi-DB setup with ATTACH for cross-database queries 884 + │ │ ├── db.go # SQLite connection with ATTACH for cross-database queries 886 885 │ │ ├── user.go # User queries 887 886 │ │ ├── feed.go # Feed + subscription queries 888 887 │ │ ├── article.go # Article queries 889 888 │ │ ├── social.go # Like, annotation queries 890 889 │ │ ├── follow.go # Follow queries 891 - │ │ ├── cluster.go # Similarity + recommendation queries 892 890 │ │ ├── oauth_store.go # OAuth session storage 893 891 │ │ └── store.go # FeedStore adapter for scheduler 894 892 │ ├── feed/ ··· 904 902 │ │ └── metrics.go # Prometheus metrics definitions 905 903 │ ├── cluster/ 906 904 │ │ ├── jaccard.go # Jaccard similarity computation 907 - │ │ ├── recommender.go # Feed + people recommendation queries (on-demand) 908 - │ │ ├── scoring.go # Multi-signal composite scoring queries 905 + │ │ ├── scoring.go # Feed + people + article recommendation queries (on-demand) 909 906 │ │ ├── social.go # Follow-distance computation (1-2 hop) 910 907 │ │ ├── dismiss.go # Dismiss + impression tracking 911 908 │ │ ├── weights.go # Bandit-style signal weight auto-tuning ··· 976 973 ├─► Fetch each feed, validate + store in `feeds` table 977 974 ├─► For each feed, create an `at.glean.subscription` record 978 975 │ via XRPC write to user's PDS 979 - ├─► Insert subscriptions in local `subscriptions` table 980 - └─◄ Return updated feed list fragment (htmx) 976 + ├─► Insert subscriptions in local `subscriptions` table 977 + └─◄ Redirect to `/feeds` 981 978 ``` 982 979 983 980 ### 11.2 Reading the Feed ··· 993 990 ### 11.3 Recommendations 994 991 995 992 ``` 996 - Cron (every 6h) ──► Cluster Engine 997 - 998 - ├─► SELECT user similarity pairs 999 - ├─► Compute recommendation scores 1000 - └─► INSERT into user_feed_recommendations 993 + Cron (every 10m) ──► Cluster Engine 994 + 995 + ├─► Compute feed similarity 996 + ├─► Compute user similarity 997 + ├─► Compute follow distances 998 + ├─► Compute signal profiles 999 + └─► Auto-dismiss stale recommendations 1001 1000 1002 - Browser ──GET /discover/feeds──► Server 1003 - 1004 - ├─► SELECT from user_feed_recommendations 1005 - ├─► Fetch feed metadata 1006 - └─◄ Render recommendation cards (htmx) 1001 + Browser ──GET /dashboard──► Server 1002 + 1003 + ├─► Compute recommendations on-demand 1004 + ├─► Fetch feed metadata 1005 + └─◄ Render recommendation cards (htmx) 1007 1006 ``` 1008 1007 1009 1008 ## 12. Key Design Decisions ··· 1035 1034 - **`glean_jetstream_reconnects_total`** — Jetstream reconnection count 1036 1035 - **`glean_http_requests_total`** — HTTP request counts labeled by method, path, and status 1037 1036 - **`glean_http_request_duration_seconds`** — HTTP request duration labeled by method and path 1038 - - **`glean_users_active_total`** — Number of users with active sessions 1039 1037 - **`glean_pds_sync_runs_total`** / **`glean_pds_sync_errors_total`** — PDS sync runs and errors 1040 1038 - **`glean_cluster_runs_total`** / **`glean_cluster_duration_seconds`** — Recommendation engine runs and timing 1041 1039
+3 -3
main.go
··· 23 23 addr := flag.String("addr", envOr("GLEAN_ADDR", ":8080"), "listen address") 24 24 dbPath := flag.String("db", envOr("GLEAN_DB", "glean.db"), "database path") 25 25 jetstreamURL := flag.String("jetstream", envOr("GLEAN_JETSTREAM", "wss://jetstream.glean.at"), "Jetstream URL") 26 - syncInterval := flag.Duration("sync-interval", envDuration("GLEAN_SYNC_INTERVAL", 1*time.Hour), "PDS sync interval") 27 - clusterInterval := flag.Duration("cluster-interval", envDuration("GLEAN_CLUSTER_INTERVAL", 10*time.Minute), "cluster recomputation interval") 28 - fetchInterval := flag.Duration("fetch-interval", envDuration("GLEAN_FETCH_INTERVAL", 5*time.Minute), "feed fetch tick interval") 26 + syncInterval := flag.Duration("sync-interval", envDuration("GLEAN_SYNC_INTERVAL", 30*time.Minute), "PDS sync interval") 27 + clusterInterval := flag.Duration("cluster-interval", envDuration("GLEAN_CLUSTER_INTERVAL", 1*time.Hour), "cluster recomputation interval") 28 + fetchInterval := flag.Duration("fetch-interval", envDuration("GLEAN_FETCH_INTERVAL", 15*time.Minute), "feed fetch tick interval") 29 29 collectionDirURL := flag.String("collection-dir", envOr("GLEAN_COLLECTION_DIR_URL", ""), "collection directory URL for startup backfill") 30 30 backfillConcurrency := flag.Int("backfill-concurrency", envInt("GLEAN_BACKFILL_CONCURRENCY", 5), "max concurrent backfill workers") 31 31 sessionKey := envOr("GLEAN_SESSION_KEY", "")
+14 -14
readme.md
··· 37 37 38 38 ## Configuration 39 39 40 - | Variable | Default | What it does | 41 - | -------------------------- | -------------------------- | --------------------------------------------------------- | 42 - | `GLEAN_SESSION_KEY` | _(required)_ | Secret key for signing session cookies (any random string) | 43 - | `GLEAN_ADDR` | `:8080` | Listen address | 44 - | `GLEAN_DB` | `glean.db` | SQLite base path (`_users`, `_articles`, `_recs` suffixes) | 45 - | `GLEAN_JETSTREAM` | `wss://jetstream.glean.at` | Jetstream WebSocket URL | 46 - | `GLEAN_SYNC_INTERVAL` | `1h` | PDS sync interval (Go duration: `30m`, `2h30m`, etc.) | 47 - | `GLEAN_CLUSTER_INTERVAL` | `10m` | Cluster recomputation interval (Go duration) | 48 - | `GLEAN_FETCH_INTERVAL` | `5m` | Feed fetch scheduler tick interval (Go duration) | 49 - | `GLEAN_COLLECTION_DIR_URL` | _(empty)_ | Collection directory URL for startup backfill | 50 - | `GLEAN_BACKFILL_CONCURRENCY` | `5` | Max concurrent backfill workers | 51 - | `GLEAN_PLC_URL` | `https://didplc.glean.at` | PLC directory URL for DID resolution | 52 - | `GLEAN_OAUTH_CLIENT_ID` | _(empty)_ | OAuth client metadata URL (leave empty for localhost dev) | 53 - | `GLEAN_OAUTH_REDIRECT_URL` | _(empty)_ | OAuth redirect URL (leave empty for localhost dev) | 40 + | Variable | Default | What it does | 41 + | ---------------------------- | -------------------------- | ---------------------------------------------------------- | 42 + | `GLEAN_SESSION_KEY` | _(required)_ | Secret key for signing session cookies (any random string) | 43 + | `GLEAN_ADDR` | `:8080` | Listen address | 44 + | `GLEAN_DB` | `glean.db` | SQLite base path (`_users`, `_articles`, `_recs` suffixes) | 45 + | `GLEAN_JETSTREAM` | `wss://jetstream.glean.at` | Jetstream WebSocket URL | 46 + | `GLEAN_SYNC_INTERVAL` | `30m` | PDS sync interval (Go duration: `30m`, `2h30m`, etc.) | 47 + | `GLEAN_CLUSTER_INTERVAL` | `1h` | Cluster recomputation interval (Go duration) | 48 + | `GLEAN_FETCH_INTERVAL` | `15m` | Feed fetch scheduler tick interval (Go duration) | 49 + | `GLEAN_COLLECTION_DIR_URL` | _(empty)_ | Collection directory URL for startup backfill | 50 + | `GLEAN_BACKFILL_CONCURRENCY` | `5` | Max concurrent backfill workers | 51 + | `GLEAN_PLC_URL` | `https://didplc.glean.at` | PLC directory URL for DID resolution | 52 + | `GLEAN_OAUTH_CLIENT_ID` | _(empty)_ | OAuth client metadata URL (leave empty for localhost dev) | 53 + | `GLEAN_OAUTH_REDIRECT_URL` | _(empty)_ | OAuth redirect URL (leave empty for localhost dev) | 54 54 55 55 For production: 56 56