atauth
AtAuth
Star 7
Fork 0
Watch
atauth docs RCA-2026-03-01-dual-atauth-instances.md
7.7 kB · 121 lines
Markdown
at commit 957730e5

Root Cause Analysis: Dual ATAuth Instances & PDS Server Version Drift (2026-03-01) #

Incident #

Blog authentication flow (blog.arcnode.xyz) failed with "OAuth state not found or expired" after PDS OAuth completed successfully. Investigation revealed a second undocumented atauth instance on the PDS Hetzner server running v1.0.0 (3+ months behind production v2.2.0). Attempted fix temporarily broke authentication for all apps using the production K8s atauth.

Duration: ~3 hours active debugging Impact:

  • Blog login broken (original issue)
  • ~10 minute outage for all apps using K8s atauth (mspdesk, search, etc.) due to incorrect tunnel reroute
  • PDS server running atauth v1.0.0 without passkey, proxy auth, OIDC, or admin dashboard features Severity: High (P2) — partial auth outage, undocumented infrastructure drift

Timeline #

  1. Blog login flow completed PDS OAuth but atauth returned "OAuth state not found"
  2. Traced root cause: blog server-side called local atauth (172.17.0.1:3100 on PDS server) for /auth/init, creating state locally. But PDS redirected browser to apricot.workingtitle.zip/auth/callback (K8s atauth) which had no knowledge of that state.
  3. Mistake: Rerouted apricot.workingtitle.zip tunnel from K8s atauth to PDS server atauth — immediately broke all apps (mspdesk, search, etc.) that use K8s atauth with different HMAC secrets and app registrations.
  4. Reverted tunnel within ~10 minutes.
  5. Fix: Created atauth.arcnode.xyz as a separate public URL for the PDS server's atauth instance, keeping apricot.workingtitle.zip pointed at K8s.
  6. Discovered PDS server was running atauth v1.0.0 (Dec 2025 image) while K8s was running v2.2.0 (Feb 2026). Built v2.2.0 from source and deployed.

Root Causes #

1. Undocumented second atauth instance #

The PDS Hetzner server (46.224.33.19) had atauth deployed at /opt/atauth/ using Docker image ghcr.io/cache8063/atauth-gateway:latest (v1.0.0, built Dec 11 2025). This instance was:

  • Not documented in any infrastructure docs
  • Not tracked in the Cloudflare tunnel config
  • Running a version 3+ months behind production
  • Had its own separate SQLite database with its own app registrations and HMAC secrets
  • Accessible only via Docker bridge IP (172.17.0.1:3100) — no public URL

2. Mixed internal/external URL configuration #

The atauth instance on the PDS server had:

  • OAUTH_CLIENT_ID=https://apricot.workingtitle.zip/client-metadata.json
  • OAUTH_REDIRECT_URI=https://apricot.workingtitle.zip/auth/callback

This meant it told the PDS "I am apricot.workingtitle.zip" but it was NOT apricot.workingtitle.zip. The PDS would redirect the browser to the real apricot.workingtitle.zip (K8s atauth) which had no knowledge of the state created by the PDS server's atauth.

3. Stale container image — no CI/CD pipeline to PDS server #

The Gitea CI workflow builds images and pushes to gitea.cloudforest-basilisk.ts.net/arcnode.xyz/atauth-gateway. But:

  • The PDS server was pulling from ghcr.io/cache8063/atauth-gateway:latest (different registry)
  • No Watchtower or auto-update mechanism for atauth on the PDS server
  • The Gitea CI has been failing since Feb 25 (runner issues), so even the Gitea registry image was stale (Dec 2025)
  • Had to build v2.2.0 from source directly on the server

4. Shared tunnel namespace without isolation #

Both atauth instances shared the same apricot.workingtitle.zip identity. Changing the tunnel route for one immediately broke the other because:

  • Different app registrations (different HMAC secrets)
  • Different session stores (SQLite on each)
  • Different feature sets (v1.0.0 vs v2.2.0)

What Data Was At Risk #

Checked and confirmed all data survived:

  • All 5 registered apps intact (mspdesk, apricot, md, mdeditor, arcnode-blog)
  • All HMAC secrets preserved (64 bytes each)
  • All sessions preserved (4 active)
  • K8s atauth completely untouched
  • Docker volume atauth_atauth-data used by both old and new containers (same project prefix)

Fix Applied #

  1. Created atauth.arcnode.xyz DNS + tunnel route → PDS server atauth (46.224.33.19:3100)
  2. Updated PDS server atauth env:
    • OAUTH_CLIENT_ID=https://atauth.arcnode.xyz/client-metadata.json
    • OAUTH_REDIRECT_URI=https://atauth.arcnode.xyz/auth/callback
  3. Built and deployed atauth v2.2.0 from source on PDS server
  4. Enabled passkey (RP ID: atauth.arcnode.xyz), forward-auth proxy
  5. Updated blog to use atauth proxy login flow (no more client-side callback delay)
  6. K8s atauth (apricot.workingtitle.zip) left completely untouched
  7. Fixed PDS nginx Sec-Fetch-Mode header — PDS started enforcing this header on /oauth/authorize, and Cloudflare Tunnel strips it. Added Sec-Fetch-Mode: navigate and Sec-Fetch-Dest: document to the nginx proxy config alongside the existing Sec-Fetch-Site: cross-site. This was breaking both search.arcnode.xyz (K8s atauth) and blog.arcnode.xyz (PDS atauth) since both OAuth flows go through the same PDS.
  8. Cleaned up stale PDS atauth app registrations — removed duplicate mdeditor (same product as md), stale mspdesk and apricot entries that only belong on K8s instance

Current State (Post-Fix) #

Instance URL Version Apps
K8s (DO) apricot.workingtitle.zip v2.2.0 search, mspdesk, others
PDS (Hetzner) atauth.arcnode.xyz v2.2.0 arcnode-blog, md

Note: Consolidation target is the K8s instance (apricot.workingtitle.zip). Once kubeconfig access is available, migrate PDS instance apps to K8s and decommission the PDS sidecar.

Prevention #

Immediate #

  1. Document all atauth instances — Update infrastructure overview with both instances, their URLs, versions, and which apps use which
  2. Each instance MUST have its own unique public URL — Never configure OAUTH_CLIENT_ID/OAUTH_REDIRECT_URI pointing to a different instance
  3. Add version endpoint — GET /version returning build commit and version to quickly identify running versions

Short-term #

  1. Fix Gitea CI pipeline — Runner has been broken since Feb 25. Fix it so images auto-build on push
  2. Add Watchtower or auto-pull for atauth on PDS server to track Gitea registry
  3. Consolidate to single atauth instance — Both instances have diverged data. Pick one as source of truth and migrate (see cleanup task below)

Long-term #

  1. Migrate atauth to dedicated instance — Currently running as a sidecar on the PDS server. Should be its own container/VM with proper CI/CD, monitoring, and backup
  2. Infrastructure-as-code — Tunnel routes, DNS, and service configs should be in version control, not manually configured
  3. Add health monitoring — Alert if any atauth instance returns non-200 or version mismatch

Cleanup Task #

Migrate atauth to a dedicated instance off the PDS server. This was identified as necessary during this incident. Track in project TODO.

Files Changed #

  • /opt/atauth/.env (PDS server) — Updated OAUTH_CLIENT_ID/REDIRECT_URI to atauth.arcnode.xyz
  • /opt/atauth/docker-compose.yml (PDS server) — Switched from ghcr.io to locally built v2.2.0 image
  • /pds/nginx/nginx.conf (PDS server) — Added Sec-Fetch-Mode and Sec-Fetch-Dest headers to /oauth/authorize location
  • Cloudflare tunnel — Added atauth.arcnode.xyz ingress rule
  • arcnode-blog/src/lib/auth.ts — Rewrote to use proxy auth flow
  • arcnode-blog/src/pages/write.astro — Redirect to atauth proxy login instead of inline form
  • arcnode-blog/src/pages/api/publish.ts — Use session cookies instead of HMAC token
  • Deleted: arcnode-blog/src/pages/api/login.ts, api/session.ts, auth/callback.astro