Your calm window into the Atmosphere. Mirror of https://github.com/hfrdmnk/morgenblau morgen.blue

feat(feeds): end-to-end RSS pipeline: fetch, process, schedule #20

Merged dominikhofer.me
targeting main
Merged opened by and assigned to dominikhofer.me
targeting main

Summary #

Ships the full RSS pipeline (subscribe → fetch → process → render) and hardens it to the shape of FreshRSS / Miniflux. The digest read path no longer triggers fetches — SPEC's three refresh triggers (30-min cron, manual, on-add) are now the only paths into the queue.

What's new #

  • Outbound HTTP — OutboundFeedClient blocks SSRF (loopback, RFC1918, link-local, IPv4-mapped IPv6); ConditionalFeedClient plumbs If-None-Match / If-Modified-Since and reads ETag / Last-Modified / Retry-After.
  • Fetch result variants — FeedFetcher returns Modified | NotModified | Gone | RateLimited | Failed; the job branches on case kind, not exception messages.
  • Processor seam — ContentTypeClassifier (blogpost / microblog / video / podcast) + HtmlSanitizer (HTMLPurifier via stevebauman/purify) run between fetch and upsert. Sanitized HTML is persisted; the reader renders verbatim.
  • Per-feed scheduling — next_check_at filter, per-feed last_dispatched_at stamped inside the dispatch loop (no bulk pre-stamp race), 30-min cron.
  • Failure handling — exponential backoff (5m → 15m → 1h → 6h → 24h), silent auto-mute at 20 consecutive failures, daily retry command silently re-enables on success.
  • Batch upsert — FeedEntry::upsert() instead of per-entry SELECT+INSERT/UPDATE; first_seen_at preserved.
  • Digest read path — ConsumeController reduced to FeedEntryRepository::digestEntries(...); stale-on-visit refresh removed.
  • PDS reconcile — SubscriptionService::reconcileLocalMirror wrapped in DB::transaction(); new feeds since last login are sync-fetched at login.
  • Model hygiene — $fillable allow-lists replace $guarded = []; state transitions move to methods (markFetched, markFailed, markGone, mute, …).

SPEC #

<feed-sources> and <content-types> updated inline during planning: three-trigger refresh model, fetch-time classification, and the "30-min default is a product choice, not an architectural ceiling" posture.

Tests #

  • Per-variant job tests (Modified / NotModified / Gone / RateLimited / Failed).
  • Backoff ladder + auto-mute + daily retry.
  • Classifier + sanitizer table tests.
  • Scheduler eligibility (next_check_at honored, per-feed stamp, dispatch failure leaves row unstamped).
  • SSRF dataset over loopback / RFC1918 / link-local / IPv4-mapped IPv6.
  • ReconcileLocalMirrorTest covers transactional update + cross-user isolation.

bun run lint, bun run types, pint --dirty, php artisan test --compact — all green.

Out of scope (P2) #

Dedup hash fallback for guid-less + link-less entries, body-size streaming cap, outbound port allow-list, identifying User-Agent, redirect TOCTOU pinning via CURLOPT_RESOLVE, reading-time / YouTube-ID / podcast-enclosure processors (no-op stubs in the seam today), and full-article fetching ("Readability") — tracked as a separate PRD.

This pull request has no versions submitted yet.