Improve docs · julien.rbrt.fr/glean@af6a931

+3 -3

.env.example

··· 3 3 GLEAN_SESSION_KEY=change-me-to-a-random-string 4 4 GLEAN_JETSTREAM=wss://jetstream.glean.at 5 5 GLEAN_PLC_URL=https://didplc.glean.at 6 - GLEAN_SYNC_INTERVAL=10m 7 - GLEAN_CLUSTER_INTERVAL=15m 8 - GLEAN_FETCH_INTERVAL=5m 6 + GLEAN_SYNC_INTERVAL=30m 7 + GLEAN_CLUSTER_INTERVAL=60m 8 + GLEAN_FETCH_INTERVAL=15m 9 9 GLEAN_COLLECTION_DIR_URL=https://lightrail.microcosm.blue/xrpc/com.atproto.sync.listReposByCollection?collection=at.glean.subscription 10 10 GLEAN_BACKFILL_CONCURRENCY=5 11 11 # Leave empty for localhost OAuth (development)

+1 -2

docs/design.md

+47 -49

docs/specs.md

··· 298 298 299 299 The scheduler uses a configurable tick interval with in-flight deduplication: 300 300 301 - - **Tick interval**: The scheduler checks for stale feeds every `GLEAN_FETCH_INTERVAL` (default 5 minutes) 301 + - **Tick interval**: The scheduler checks for stale feeds every `GLEAN_FETCH_INTERVAL` (default 15 minutes) 302 302 - **Staleness threshold**: Feeds not fetched in the last 30 minutes are eligible 303 303 - **Subscriber filter**: Only feeds with `subscriber_count > 0` are fetched 304 304 - **In-flight dedup**: If a feed is already being fetched (e.g., manual refresh and background scheduler overlap), the second caller waits for the first to complete rather than fetching again ··· 395 395 396 396 - **Auto-discovery**: When fetching a feed, parse `<link rel="alternate" type="application/rss+xml">` from the feed's site URL to discover related feeds 397 397 - **Feedfavicon**: Fetch `favicon.ico` or `/apple-touch-icon.png` from the feed's site URL for display 398 - - **Dead feed detection**: If a feed fails for 7 consecutive fetches (14 days at base interval), mark it as dead. Notify the user and offer to remove it. 398 + - **Dead feed detection**: If a feed fails for 7 consecutive fetches, mark it as dead. Notify the user and offer to remove it. 399 399 400 400 ## 5. System Architecture 401 401 ··· 668 668 For any two users, compute Jaccard over their subscription sets, plus like co-occurrence (time-decayed) and tag overlap: 669 669 670 670 ``` 671 - J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + follow_boost 671 + J(U1, U2) = jaccard_subscriptions + 0.3 * jaccard_likes + 0.2 * jaccard_tags + 0.5 * follow_boost 672 672 ``` 673 673 674 674 Like overlap uses exponential time decay: `EXP(-0.023 * age_days)` (30-day half-life). ··· 818 818 819 819 ### 8.1 Pages 820 820 821 - | Route | Method | Description | 822 - | ------------------------------ | ------ | -------------------------------------------------------- | 823 - | `/` | GET | Landing page / auth redirect | 824 - | `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar | 825 - | `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) | 826 - | `/feeds/list` | GET | Feed list fragment (htmx partial) | 827 - | `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions | 828 - | `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) | 829 - | `/feeds/add` | POST | Add a single feed URL | 830 - | `/feeds/remove` | DELETE | Remove a feed | 831 - | `/feeds/refresh` | POST | Refresh all subscribed feeds | 832 - | `/feeds/retry` | POST | Retry a failed feed | 833 - | `/feeds/clear` | POST | Clear all subscriptions | 834 - | `/feeds/dismiss` | POST | Dismiss a feed recommendation | 835 - | `/articles` | GET | Read articles (paginated, filterable by feed) | 836 - | `/articles/new-count` | GET | Get count of new articles (for badge updates) | 837 - | `/articles/{id}` | GET | Article detail view | 838 - | `/articles/{id}/read` | POST | Mark article as read | 839 - | `/articles/{id}/unread` | POST | Mark article as unread | 840 - | `/articles/{id}/like` | POST | Like an article | 841 - | `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL | 842 - | `/articles/mark-all-read` | POST | Mark all articles as read | 843 - | `/articles/dismiss` | POST | Dismiss an article recommendation | 844 - | `/trending` | GET | Community feed: articles ranked by likes | 845 - | `/library` | GET | Liked articles and annotations | 846 - | `/library/create` | POST | Create annotation on an article | 847 - | `/library/{id}/delete` | POST | Delete an annotation | 848 - | `/profile/{did}` | GET | Public profile: their feeds, likes, annotations | 821 + | Route | Method | Description | 822 + | ------------------------------ | ------ | ------------------------------------------------------------------- | 823 + | `/` | GET | Landing page / auth redirect | 824 + | `/dashboard` | GET | Main dashboard: unread articles, recommendations sidebar | 825 + | `/feeds` | GET | Manage RSS subscriptions (OPML import for onboarding) | 826 + | `/feeds/list` | GET | Feed list fragment (htmx partial) | 827 + | `/feeds/opml/upload` | POST | Upload OPML file to bulk-import subscriptions (redirects to /feeds) | 828 + | `/feeds/opml/download` | GET | Export subscriptions as OPML (offboarding) | 829 + | `/feeds/add` | POST | Add a single feed URL | 830 + | `/feeds/remove` | DELETE | Remove a feed | 831 + | `/feeds/refresh` | POST | Refresh all subscribed feeds | 832 + | `/feeds/retry` | POST | Retry a failed feed | 833 + | `/feeds/clear` | POST | Clear all subscriptions | 834 + | `/feeds/dismiss` | POST | Dismiss a feed recommendation | 835 + | `/articles` | GET | Read articles (paginated, filterable by feed) | 836 + | `/articles/new-count` | GET | Get count of new articles (for badge updates) | 837 + | `/articles/{id}` | GET | Article detail view | 838 + | `/articles/{id}/read` | POST | Mark article as read | 839 + | `/articles/{id}/unread` | POST | Mark article as unread | 840 + | `/articles/{id}/like` | POST | Like an article | 841 + | `/articles/{id}/fetch-content` | POST | Fetch full article content from original URL | 842 + | `/articles/mark-all-read` | POST | Mark all articles as read | 843 + | `/articles/dismiss` | POST | Dismiss an article recommendation | 844 + | `/trending` | GET | Community feed: articles ranked by likes | 845 + | `/library` | GET | Liked articles and annotations | 846 + | `/library/create` | POST | Create annotation on an article | 847 + | `/library/{id}/delete` | POST | Delete an annotation | 848 + | `/profile/{did}` | GET | Public profile: their feeds, likes, annotations | 849 849 850 850 ### 8.2 htmx Patterns 851 851 ··· 881 881 │ │ ├── sync.go # PDS record reconciliation 882 882 │ │ └── xrpc.go # XRPC query handlers (AppView endpoints) 883 883 │ ├── db/ 884 - │ │ ├── db.go # SQLite connection, single-DB schema 885 - │ │ ├── multi.go # Multi-DB setup with ATTACH for cross-database queries 884 + │ │ ├── db.go # SQLite connection with ATTACH for cross-database queries 886 885 │ │ ├── user.go # User queries 887 886 │ │ ├── feed.go # Feed + subscription queries 888 887 │ │ ├── article.go # Article queries 889 888 │ │ ├── social.go # Like, annotation queries 890 889 │ │ ├── follow.go # Follow queries 891 - │ │ ├── cluster.go # Similarity + recommendation queries 892 890 │ │ ├── oauth_store.go # OAuth session storage 893 891 │ │ └── store.go # FeedStore adapter for scheduler 894 892 │ ├── feed/ ··· 904 902 │ │ └── metrics.go # Prometheus metrics definitions 905 903 │ ├── cluster/ 906 904 │ │ ├── jaccard.go # Jaccard similarity computation 907 - │ │ ├── recommender.go # Feed + people recommendation queries (on-demand) 908 - │ │ ├── scoring.go # Multi-signal composite scoring queries 905 + │ │ ├── scoring.go # Feed + people + article recommendation queries (on-demand) 909 906 │ │ ├── social.go # Follow-distance computation (1-2 hop) 910 907 │ │ ├── dismiss.go # Dismiss + impression tracking 911 908 │ │ ├── weights.go # Bandit-style signal weight auto-tuning ··· 976 973 ├─► Fetch each feed, validate + store in `feeds` table 977 974 ├─► For each feed, create an `at.glean.subscription` record 978 975 │ via XRPC write to user's PDS 979 - ├─► Insert subscriptions in local `subscriptions` table 980 - └─◄ Return updated feed list fragment (htmx) 976 + ├─► Insert subscriptions in local `subscriptions` table 977 + └─◄ Redirect to `/feeds` 981 978 ``` 982 979 983 980 ### 11.2 Reading the Feed ··· 993 990 ### 11.3 Recommendations 994 991 995 992 ``` 996 - Cron (every 6h) ──► Cluster Engine 997 - │ 998 - ├─► SELECT user similarity pairs 999 - ├─► Compute recommendation scores 1000 - └─► INSERT into user_feed_recommendations 993 + Cron (every 10m) ──► Cluster Engine 994 + │ 995 + ├─► Compute feed similarity 996 + ├─► Compute user similarity 997 + ├─► Compute follow distances 998 + ├─► Compute signal profiles 999 + └─► Auto-dismiss stale recommendations 1001 1000 1002 - Browser ──GET /discover/feeds──► Server 1003 - │ 1004 - ├─► SELECT from user_feed_recommendations 1005 - ├─► Fetch feed metadata 1006 - └─◄ Render recommendation cards (htmx) 1001 + Browser ──GET /dashboard──► Server 1002 + │ 1003 + ├─► Compute recommendations on-demand 1004 + ├─► Fetch feed metadata 1005 + └─◄ Render recommendation cards (htmx) 1007 1006 ``` 1008 1007 1009 1008 ## 12. Key Design Decisions ··· 1035 1034 - **`glean_jetstream_reconnects_total`** — Jetstream reconnection count 1036 1035 - **`glean_http_requests_total`** — HTTP request counts labeled by method, path, and status 1037 1036 - **`glean_http_request_duration_seconds`** — HTTP request duration labeled by method and path 1038 - - **`glean_users_active_total`** — Number of users with active sessions 1039 1037 - **`glean_pds_sync_runs_total`** / **`glean_pds_sync_errors_total`** — PDS sync runs and errors 1040 1038 - **`glean_cluster_runs_total`** / **`glean_cluster_duration_seconds`** — Recommendation engine runs and timing 1041 1039

+3 -3

main.go

··· 23 23 addr := flag.String("addr", envOr("GLEAN_ADDR", ":8080"), "listen address") 24 24 dbPath := flag.String("db", envOr("GLEAN_DB", "glean.db"), "database path") 25 25 jetstreamURL := flag.String("jetstream", envOr("GLEAN_JETSTREAM", "wss://jetstream.glean.at"), "Jetstream URL") 26 - syncInterval := flag.Duration("sync-interval", envDuration("GLEAN_SYNC_INTERVAL", 1*time.Hour), "PDS sync interval") 27 - clusterInterval := flag.Duration("cluster-interval", envDuration("GLEAN_CLUSTER_INTERVAL", 10*time.Minute), "cluster recomputation interval") 28 - fetchInterval := flag.Duration("fetch-interval", envDuration("GLEAN_FETCH_INTERVAL", 5*time.Minute), "feed fetch tick interval") 26 + syncInterval := flag.Duration("sync-interval", envDuration("GLEAN_SYNC_INTERVAL", 30*time.Minute), "PDS sync interval") 27 + clusterInterval := flag.Duration("cluster-interval", envDuration("GLEAN_CLUSTER_INTERVAL", 1*time.Hour), "cluster recomputation interval") 28 + fetchInterval := flag.Duration("fetch-interval", envDuration("GLEAN_FETCH_INTERVAL", 15*time.Minute), "feed fetch tick interval") 29 29 collectionDirURL := flag.String("collection-dir", envOr("GLEAN_COLLECTION_DIR_URL", ""), "collection directory URL for startup backfill") 30 30 backfillConcurrency := flag.Int("backfill-concurrency", envInt("GLEAN_BACKFILL_CONCURRENCY", 5), "max concurrent backfill workers") 31 31 sessionKey := envOr("GLEAN_SESSION_KEY", "")

+14 -14

readme.md

··· 37 37 38 38 ## Configuration 39 39 40 - | Variable | Default | What it does | 41 - | -------------------------- | -------------------------- | --------------------------------------------------------- | 42 - | `GLEAN_SESSION_KEY` | _(required)_ | Secret key for signing session cookies (any random string) | 43 - | `GLEAN_ADDR` | `:8080` | Listen address | 44 - | `GLEAN_DB` | `glean.db` | SQLite base path (`_users`, `_articles`, `_recs` suffixes) | 45 - | `GLEAN_JETSTREAM` | `wss://jetstream.glean.at` | Jetstream WebSocket URL | 46 - | `GLEAN_SYNC_INTERVAL` | `1h` | PDS sync interval (Go duration: `30m`, `2h30m`, etc.) | 47 - | `GLEAN_CLUSTER_INTERVAL` | `10m` | Cluster recomputation interval (Go duration) | 48 - | `GLEAN_FETCH_INTERVAL` | `5m` | Feed fetch scheduler tick interval (Go duration) | 49 - | `GLEAN_COLLECTION_DIR_URL` | _(empty)_ | Collection directory URL for startup backfill | 50 - | `GLEAN_BACKFILL_CONCURRENCY` | `5` | Max concurrent backfill workers | 51 - | `GLEAN_PLC_URL` | `https://didplc.glean.at` | PLC directory URL for DID resolution | 52 - | `GLEAN_OAUTH_CLIENT_ID` | _(empty)_ | OAuth client metadata URL (leave empty for localhost dev) | 53 - | `GLEAN_OAUTH_REDIRECT_URL` | _(empty)_ | OAuth redirect URL (leave empty for localhost dev) | 40 + | Variable | Default | What it does | 41 + | ---------------------------- | -------------------------- | ---------------------------------------------------------- | 42 + | `GLEAN_SESSION_KEY` | _(required)_ | Secret key for signing session cookies (any random string) | 43 + | `GLEAN_ADDR` | `:8080` | Listen address | 44 + | `GLEAN_DB` | `glean.db` | SQLite base path (`_users`, `_articles`, `_recs` suffixes) | 45 + | `GLEAN_JETSTREAM` | `wss://jetstream.glean.at` | Jetstream WebSocket URL | 46 + | `GLEAN_SYNC_INTERVAL` | `30m` | PDS sync interval (Go duration: `30m`, `2h30m`, etc.) | 47 + | `GLEAN_CLUSTER_INTERVAL` | `1h` | Cluster recomputation interval (Go duration) | 48 + | `GLEAN_FETCH_INTERVAL` | `15m` | Feed fetch scheduler tick interval (Go duration) | 49 + | `GLEAN_COLLECTION_DIR_URL` | _(empty)_ | Collection directory URL for startup backfill | 50 + | `GLEAN_BACKFILL_CONCURRENCY` | `5` | Max concurrent backfill workers | 51 + | `GLEAN_PLC_URL` | `https://didplc.glean.at` | PLC directory URL for DID resolution | 52 + | `GLEAN_OAUTH_CLIENT_ID` | _(empty)_ | OAuth client metadata URL (leave empty for localhost dev) | 53 + | `GLEAN_OAUTH_REDIRECT_URL` | _(empty)_ | OAuth redirect URL (leave empty for localhost dev) | 54 54 55 55 For production: 56 56

Configure Feed

Configure Feed