From a6676ff36bf00416783b65b606d7ee9b521b3219 Mon Sep 17 00:00:00 2001 From: Bryan Brooks Date: Sun, 22 Feb 2026 19:27:07 -0600 Subject: [PATCH] =?UTF-8?q?docs:=20project=20checkpoint=20=E2=80=94=20upda?= =?UTF-8?q?te=20CLAUDE.md,=20CHANGELOG,=20deploy=20runbook,=20gitignore?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - CLAUDE.md: Update test counts (262), add OIDC key files, dashboard routes, ABS integration details, fixed-bug troubleshooting entries, accurate pending work list - CHANGELOG.md: Add v2.0.0 entry covering OIDC provider, forward-auth proxy, admin dashboard, setup wizard, CI/CD migration, and bug fixes - SAFE-DEPLOY.md: Rewrite for DigitalOcean (was stale k3s references) - .gitignore: Add *.db-shm, *.db-wal, *.db-journal patterns --- .gitignore | 3 + CHANGELOG.md | 46 +++++++ CLAUDE.md | 67 +++++++--- gateway/SAFE-DEPLOY.md | 271 ++++++++++++----------------------------- 4 files changed, 181 insertions(+), 206 deletions(-) diff --git a/.gitignore b/.gitignore index c907be5..e4859ad 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,9 @@ Thumbs.db # Test artifacts *.db +*.db-shm +*.db-wal +*.db-journal *.sqlite # Environment files (keep examples) diff --git a/CHANGELOG.md b/CHANGELOG.md index 84e608a..ae3bc7b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,51 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [2.0.0] - 2026-02-22 + +### Added +- **OIDC Provider**: Full OpenID Connect provider using AT Protocol OAuth as identity source + - Authorization endpoint with PKCE support + - Token endpoint (authorization_code + refresh_token grants) + - UserInfo endpoint with DID-to-handle resolution via AT Protocol API + - Token revocation endpoint + - JWKS endpoint with ES256 key rotation + - Discovery document (`/.well-known/openid-configuration`) +- **Forward-Auth SSO Proxy**: nginx `auth_request` based single sign-on for arbitrary services + - Per-user DID and handle pattern access control rules + - Deny-overrides evaluation model + - Per-origin and global rule scoping + - HMAC-SHA256 signed session cookies with `typ` discriminator + - Ticket-based exchange for X-Auth-DID/X-Auth-Handle headers +- **Admin Dashboard**: Server-rendered HTML admin UI replacing static React SPA + - OIDC client management (list, create, edit, delete, secret rotation) + - Setup wizard with presets for 10 common self-hosted apps (Audiobookshelf, Jellyfin, Nextcloud, Gitea, Immich, Grafana, WikiJS, Portainer, Outline, Mealie) + - Forward-auth proxy quick setup wizard with nginx/k8s config snippet generation + - Proxy origin management, access rule management, session management + - Access check dry-run tool + - HMAC-signed CSRF tokens on all forms + - Cookie-based auth (24h TTL) alongside Bearer token auth +- **CI/CD**: Gitea Actions pipeline (test -> build -> push -> deploy to DigitalOcean k8s) + +### Changed +- **Infrastructure**: Migrated from local k3s (Proxmox) to DigitalOcean Managed Kubernetes +- **Registry**: Moved from Gitea container registry to DigitalOcean Container Registry +- **Deployment strategy**: Recreate (RWO PVC for SQLite) +- **Test suite**: Expanded from ~50 to 262 tests across 11 test files + +### Fixed +- OIDC token endpoint: hash incoming client secret before comparing against stored SHA-256 hash +- OIDC authorize: pass explicit redirect_uri to AT Protocol OAuth callback (prevents `@atproto/oauth-client` fallback mismatch) +- OIDC userinfo: resolve DID to handle via `app.bsky.actor.getProfile` when no cached mapping exists +- Legacy `/auth/init`: separate OAuth redirect_uri from downstream app callback URL +- Register OIDC and forward-auth callback URIs in NodeOAuthClient metadata + +### Security +- Client secrets stored as SHA-256 hashes (not plaintext) +- Constant-time comparison for all secret verification +- PKCE required for OIDC clients (configurable per-client) +- Access control deny-overrides: deny rules always win regardless of order + ## [1.3.0] - 2025-12-15 ### Changed @@ -69,6 +114,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Input validation and sanitization - Rate limiting on all endpoints +[2.0.0]: https://gitea.cloudforest-basilisk.ts.net/Arcnode.xyz/atauth/compare/v1.3.0...main [1.3.0]: https://github.com/Cache8063/atauth/compare/v1.2.0...v1.3.0 [1.2.0]: https://github.com/Cache8063/atauth/compare/v1.0.0...v1.2.0 [1.0.0]: https://github.com/Cache8063/atauth/releases/tag/v1.0.0 diff --git a/CLAUDE.md b/CLAUDE.md index 913ca9a..33f3f75 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -69,16 +69,19 @@ Protected service receives X-Auth-DID, X-Auth-Handle headers ### Deployment Process (Manual Fallback) ```bash -# Build and push (from gateway/ directory) +# Build and push (from repo root, use unique tag to avoid k8s image caching) +TAG=$(git rev-parse --short HEAD) docker build --platform linux/amd64 \ - -t registry.digitalocean.com/ghostmesh-registry/atauth:$(git rev-parse --short HEAD) \ + -t registry.digitalocean.com/ghostmesh-registry/atauth:$TAG \ -t registry.digitalocean.com/ghostmesh-registry/atauth:latest \ gateway/ -docker push registry.digitalocean.com/ghostmesh-registry/atauth:$(git rev-parse --short HEAD) +docker push registry.digitalocean.com/ghostmesh-registry/atauth:$TAG docker push registry.digitalocean.com/ghostmesh-registry/atauth:latest -# Deploy -kubectl rollout restart deployment/atauth -n atauth +# Deploy with specific tag (avoids IfNotPresent caching of :latest) +kubectl set image deployment/atauth \ + atauth=registry.digitalocean.com/ghostmesh-registry/atauth:$TAG \ + -n atauth kubectl rollout status deployment/atauth -n atauth --timeout=120s ``` @@ -91,7 +94,7 @@ Config preserved in `gateway/k8s/overlays/staging/` and `gateway/k8s/overlays/pr ### Admin Dashboard (Web UI) - **URL**: `https://apricot.workingtitle.zip/admin/login` - **Auth**: Admin token from Vaultwarden (sets `_atauth_admin` cookie, 24h TTL) -- **Pages**: Overview, Origins, Access Rules, Sessions, Access Check Tool +- **Pages**: Overview, OIDC Clients, Setup Wizard, Origins, Access Rules, Sessions, Proxy Setup, Access Check Tool - **CSRF**: All forms include HMAC-signed hidden tokens (1h validity) ### Admin Token @@ -187,11 +190,17 @@ All secrets in **Vaultwarden** (LXC 120 @ vaultwarden.cloudforest-basilisk.ts.ne | File | Purpose | |------|---------| | `src/routes/proxy-auth.ts` | Forward-auth routes (`/auth/verify`, `/auth/login`, `/auth/callback`) + `enforceAccess` | +| `src/routes/oidc/authorize.ts` | OIDC authorization endpoint + AT Protocol OAuth callback | +| `src/routes/oidc/token.ts` | OIDC token exchange (auth_code + refresh_token grants) | +| `src/routes/oidc/userinfo.ts` | OIDC userinfo endpoint (DID-to-handle resolution) | +| `src/routes/oidc/revoke.ts` | OIDC token revocation | | `src/utils/proxy-auth.ts` | Cookie create/verify, ticket functions, parsing utilities | | `src/utils/access-check.ts` | Pure `matchHandlePattern` + `checkAccess` functions | | `src/routes/admin.ts` | Admin API (Bearer + cookie auth), login/logout routes | | `src/routes/admin-dashboard.ts` | Server-rendered HTML dashboard (all pages + CSRF) | +| `src/data/oidc-presets.ts` | OIDC setup wizard app presets (ABS, Jellyfin, Gitea, etc.) | | `src/services/database.ts` | SQLite schema, migrations, all DB methods | +| `src/services/oauth.ts` | AT Protocol OAuth client (`generateAuthUrl`, `handleCallback`) | | `src/types/proxy.ts` | `ProxyAccessRule`, `AccessCheckResult`, proxy config types | ### Important Notes @@ -248,8 +257,8 @@ kubectl exec -n atauth deploy/atauth -- sqlite3 /app/data/gateway.db ".tables" ### Audiobookshelf - **Location**: LXC 107 on `pv4` - **Tailscale IP**: 100.115.188.60 -- **URL**: `https://audiobookshelf.cloudforest-basilisk.ts.net` (via Tailscale serve) -- **Internal**: nginx (8080) → ABS (13378) +- **URL**: `https://audiobookshelf.cloudforest-basilisk.ts.net` (via Tailscale serve → ABS:13378 directly) +- **OIDC**: Configured with ATAuth (`client_id: audiobookshelf`, `authOpenIDAutoRegister: true`) - **SSH via pv4**: `ssh root@pv4.cloudforest-basilisk.ts.net "pct exec 107 -- "` ### Vaultwarden @@ -286,6 +295,12 @@ kubectl exec -n atauth deploy/atauth -- sqlite3 /app/data/gateway.db ".tables" | `GET /admin/dashboard/access` | Manage access rules (HTML) | | `GET /admin/dashboard/sessions` | Manage proxy sessions (HTML) | | `GET /admin/dashboard/check` | Access check tool (HTML) | +| `GET /admin/dashboard/clients` | OIDC client list (HTML) | +| `GET /admin/dashboard/clients/new` | Create OIDC client form (HTML) | +| `GET /admin/dashboard/clients/:id/edit` | Edit OIDC client form (HTML) | +| `GET /admin/dashboard/clients/wizard` | Setup wizard app selection (HTML) | +| `GET /admin/dashboard/clients/wizard/:preset` | Setup wizard configure form (HTML) | +| `GET /admin/dashboard/proxy-wizard` | Forward-auth proxy setup wizard (HTML) | | `GET /admin/oidc/clients` | List OIDC clients | | `POST /admin/oidc/clients` | Create OIDC client | | `GET /admin/oidc/clients/:id` | Get client details | @@ -305,7 +320,7 @@ kubectl exec -n atauth deploy/atauth -- sqlite3 /app/data/gateway.db ".tables" ## Testing -### Test Suite (233 tests) +### Test Suite (262 tests) ```bash cd gateway npm run test:run # All tests once @@ -320,12 +335,16 @@ npm run lint # ESLint | File | Tests | Coverage | |------|-------|----------| | `src/utils/access-check.test.ts` | 16 | matchHandlePattern, checkAccess evaluation | -| `src/utils/proxy-auth.test.ts` | 16 | Cookie create/verify, tickets, parsing, admin cookie | -| `src/routes/admin-dashboard.test.ts` | 16 | Dashboard pages, CRUD forms, CSRF, cookie auth | -| `src/routes/admin.proxy.test.ts` | 19 | Access rules API, cookie auth login/logout | -| `src/routes/proxy-auth.test.ts` | ~50 | Forward-auth flows, access control integration | -| `src/services/database.proxy.test.ts` | 14 | Access rules CRUD, cascade delete, partitioning | -| `tests/oidc-flow.test.ts` | ~80 | Full OIDC provider E2E | +| `src/utils/proxy-auth.test.ts` | 31 | Cookie create/verify, tickets, parsing, admin cookie | +| `src/routes/admin-dashboard.test.ts` | 37 | Dashboard pages, OIDC client CRUD, wizard, proxy wizard, CSRF | +| `src/routes/admin.proxy.test.ts` | 32 | Access rules API, cookie auth login/logout | +| `src/routes/auth.test.ts` | 8 | Auth redirect_uri handling, regression tests | +| `src/routes/proxy-auth.test.ts` | 28 | Forward-auth flows, access control integration | +| `src/services/database.proxy.test.ts` | 23 | Access rules CRUD, cascade delete, partitioning | +| `src/services/oidc/tokens.test.ts` | 20 | OIDC token generation and verification | +| `src/services/oidc/claims.test.ts` | 28 | UserInfo claims building, scope filtering | +| `src/services/oidc/pkce.test.ts` | 17 | PKCE challenge/verifier generation and validation | +| `tests/oidc-flow.test.ts` | 22 | Full OIDC provider E2E flow | ## Troubleshooting @@ -356,9 +375,27 @@ ssh root@pv4.cloudforest-basilisk.ts.net "pct exec 111 -- bash -c 'systemctl res ``` If still stuck, deploy manually (see Deployment Process above). +### "invalid_client" on OIDC token exchange + +**Cause**: Client secret stored as SHA-256 hash but token endpoint compared raw secret. +**Fixed**: `token.ts` and `revoke.ts` now hash incoming secret before comparison (`3fc67a7`). + +### "invalid_grant" redirect_uri mismatch on AT Protocol token exchange + +**Cause**: `@atproto/oauth-client` falls back to `clientMetadata.redirect_uris[0]` during token exchange. Authorization used `/oauth/callback` but exchange used `/auth/callback`. +**Fixed**: `authorize.ts` now passes explicit redirect_uri to `handleCallback()` (`3fc67a7`). + +### Userinfo returns empty `preferred_username` + +**Cause**: `db.getUserMapping()` returns nothing for new OIDC sessions; handle defaulted to empty string. +**Fixed**: `userinfo.ts` resolves DID to handle via `app.bsky.actor.getProfile` API (`3fc67a7`). + ### Dockerfile still builds admin-ui stage The `admin-ui-builder` stage in the Dockerfile builds the old static admin UI. It's now unused (replaced by server-rendered dashboard) but harmless. Can be removed in a future cleanup. ## Pending Work - **Dependency upgrades**: nodemailer 6->8 (CVE-2025-14874), @simplewebauthn/server 10->13, uuid 9->13, better-sqlite3 11->12 - **Dockerfile cleanup**: Remove `admin-ui-builder` stage and `admin-ui/` static files +- **Stale deployment configs**: `gateway/tekton/` (old Tekton CI), `gateway/helm/` (unused Helm chart), `.github/` (old GitHub Actions) -- can be removed +- **Rotate admin token**: Current placeholder `atauth-admin-token-2026-change-in-production` should be replaced with a proper random token +- **Setup wizard bug**: Double-protocol in redirect_uri when domain input includes `https://` prefix diff --git a/gateway/SAFE-DEPLOY.md b/gateway/SAFE-DEPLOY.md index 3658518..35147a3 100644 --- a/gateway/SAFE-DEPLOY.md +++ b/gateway/SAFE-DEPLOY.md @@ -1,218 +1,132 @@ # Safe Deployment Runbook for ATAuth Gateway -This document outlines the safe deployment procedure to avoid breaking production. - -## Deployment Strategy - -``` -┌─────────────┐ ┌─────────────┐ ┌─────────────┐ -│ Develop │────▶│ Staging │────▶│ Production │ -│ (local) │ │ (k3s) │ │ (k3s) │ -└─────────────┘ └─────────────┘ └─────────────┘ - │ │ - auth-staging.* auth.* (existing) -``` +## Current Environment + +| Setting | Value | +|---------|-------| +| Cluster | DigitalOcean Managed Kubernetes (`storm-dr-cluster`, nyc1) | +| Namespace | `atauth` | +| Registry | `registry.digitalocean.com/ghostmesh-registry` | +| Image | `registry.digitalocean.com/ghostmesh-registry/atauth` | +| URL | `https://auth-staging.workingtitle.zip` | +| Apricot | `https://apricot.workingtitle.zip` | +| Storage | SQLite on `do-block-storage` PVC | +| Strategy | Recreate (RWO PVC -- cannot use RollingUpdate) | ## Pre-Deployment Checklist -### Before ANY deployment: - -- [ ] All tests pass locally -- [ ] Code reviewed and merged to appropriate branch +- [ ] All tests pass locally (`cd gateway && npx vitest run`) +- [ ] TypeScript compiles clean (`npx tsc --noEmit`) +- [ ] ESLint passes (`npx eslint src/`) - [ ] Database schema changes are backward compatible - [ ] New environment variables documented -### Before Production deployment: - -- [ ] Successfully deployed and tested in staging -- [ ] Backup job completed successfully -- [ ] Rollback procedure reviewed -- [ ] Maintenance window communicated (if needed) - --- -## Step-by-Step Deployment +## Automated Deployment (Gitea Actions) -### 1. Deploy to Staging First +Push to `main` triggers the CI pipeline: -```bash -# Build and push staging image -git checkout develop -docker build -t gitea.cloudforest-basilisk.ts.net/arcnode.xyz/atauth-gateway:staging ./gateway -docker push gitea.cloudforest-basilisk.ts.net/arcnode.xyz/atauth-gateway:staging +1. **Test** -- typecheck + lint + vitest +2. **Build** -- Docker image with `--platform linux/amd64` +3. **Push** -- to DO registry with commit SHA + `latest` tags +4. **Deploy** -- `kubectl set image` + `kubectl rollout status` -# Deploy to staging namespace -kubectl apply -k gateway/k8s/overlays/staging - -# Verify deployment -kubectl -n atauth-staging rollout status deployment/atauth-gateway-staging -kubectl -n atauth-staging logs -f deployment/atauth-gateway-staging -``` - -### 2. Test Staging Environment +Workflow file: `.gitea/workflows/deploy.yml` +**Known issue**: `needs` job scheduling can stall after test jobs complete. +Fix: restart Gitea then act_runner on LXC 111: ```bash -# Health check -curl https://auth-staging.cloudforest-basilisk.ts.net/health - -# Test OIDC discovery -curl https://auth-staging.cloudforest-basilisk.ts.net/.well-known/openid-configuration - -# Test admin UI -open https://auth-staging.cloudforest-basilisk.ts.net/admin - -# Run integration tests against staging -# ... your test commands ... +ssh root@pv4.cloudforest-basilisk.ts.net \ + "pct exec 111 -- bash -c 'systemctl restart gitea && sleep 5 && systemctl restart act_runner'" ``` -### 3. Create Pre-Deployment Backup - -```bash -# Trigger backup job -kubectl -n atauth create job --from=cronjob/atauth-gateway-backup pre-deploy-$(date +%Y%m%d%H%M) +--- -# Wait for completion -kubectl -n atauth wait --for=condition=complete job/pre-deploy-$(date +%Y%m%d%H%M) --timeout=120s +## Manual Deployment (Fallback) -# Verify backup exists -kubectl -n atauth exec -it deploy/atauth-gateway -- ls -la /backups/ -``` - -### 4. Deploy to Production +### 1. Pre-deploy backup ```bash -# Tag the release -git checkout main -git tag v1.x.x -git push origin v1.x.x - -# Build production image with version tag -docker build -t gitea.cloudforest-basilisk.ts.net/arcnode.xyz/atauth-gateway:v1.x.x ./gateway -docker push gitea.cloudforest-basilisk.ts.net/arcnode.xyz/atauth-gateway:v1.x.x - -# Update production kustomization with new tag -cd gateway/k8s/overlays/production -# Edit kustomization.yaml to use new tag - -# Apply with dry-run first -kubectl apply -k . --dry-run=server - -# Apply for real -kubectl apply -k . - -# Watch rollout -kubectl -n atauth rollout status deployment/atauth-gateway +kubectl exec -n atauth deploy/atauth -- cp /app/data/gateway.db /app/data/gateway.db.backup ``` -### 5. Verify Production +### 2. Build and push ```bash -# Health check -curl https://auth.cloudforest-basilisk.ts.net/health - -# Check logs for errors -kubectl -n atauth logs -f deployment/atauth-gateway --since=5m - -# Verify existing sessions still work -# ... test with existing client apps ... -``` +cd /path/to/atauth ---- - -## Rollback Procedure +# Use unique tag (commit SHA or descriptive name) +TAG=$(cd gateway && git rev-parse --short HEAD) -### Quick Rollback (< 5 minutes) - -```bash -# Rollback to previous deployment -kubectl -n atauth rollout undo deployment/atauth-gateway +docker build --platform linux/amd64 \ + -t registry.digitalocean.com/ghostmesh-registry/atauth:$TAG \ + -t registry.digitalocean.com/ghostmesh-registry/atauth:latest \ + gateway/ -# Verify rollback -kubectl -n atauth rollout status deployment/atauth-gateway +docker push registry.digitalocean.com/ghostmesh-registry/atauth:$TAG +docker push registry.digitalocean.com/ghostmesh-registry/atauth:latest ``` -### Rollback with Database Restore - -If the new version corrupted data: +### 3. Deploy ```bash -# 1. Scale down the deployment -kubectl -n atauth scale deployment/atauth-gateway --replicas=0 - -# 2. Find the backup to restore -kubectl -n atauth exec -it -- ls -la /backups/ - -# 3. Restore the database -kubectl -n atauth exec -it -- sh -c ' - cp /backups/pre-deploy-YYYYMMDD-HHMMSS.db /data/gateway.db -' - -# 4. Rollback the deployment -kubectl -n atauth rollout undo deployment/atauth-gateway +# Set the specific image tag (avoids k8s caching issues with :latest) +kubectl set image deployment/atauth \ + atauth=registry.digitalocean.com/ghostmesh-registry/atauth:$TAG \ + -n atauth -# 5. Scale back up -kubectl -n atauth scale deployment/atauth-gateway --replicas=1 +kubectl rollout status deployment/atauth -n atauth --timeout=120s ``` -### Full Disaster Recovery +### 4. Verify ```bash -# If everything is broken, restore from last known good state: +# Health check +curl -s https://auth-staging.workingtitle.zip/health | jq . -# 1. Delete the deployment -kubectl -n atauth delete deployment atauth-gateway +# OIDC discovery +curl -s https://auth-staging.workingtitle.zip/.well-known/openid-configuration | jq . -# 2. Restore database from backup -# (access the PVC directly or use a debug pod) +# Admin dashboard (requires auth) +curl -s -o /dev/null -w "%{http_code}" https://apricot.workingtitle.zip/admin/login -# 3. Deploy previous known-good version -kubectl set image deployment/atauth-gateway \ - gateway=gitea.cloudforest-basilisk.ts.net/arcnode.xyz/atauth-gateway:v1.2.0 \ - -n atauth +# Check logs +kubectl -n atauth logs deploy/atauth --tail=20 ``` --- -## Database Migration Safety +## Rollback -The new version adds these tables (backward compatible): +### Quick Rollback -```sql --- New tables (won't affect existing functionality) -- oidc_keys -- authorization_codes -- refresh_tokens -- passkey_credentials -- mfa_totp -- mfa_backup_codes -- user_emails -- email_verification_codes - --- Modified tables (new columns with defaults) -- apps: client_type, client_secret, redirect_uris, etc. +```bash +kubectl -n atauth rollout undo deployment/atauth +kubectl -n atauth rollout status deployment/atauth ``` -### Migration is SAFE because: - -1. **New tables only** - existing tables unchanged -2. **New columns have defaults** - existing rows work -3. **Legacy mode supported** - old apps use `client_type='legacy'` -4. **No breaking changes to existing API** - `/auth/*` routes unchanged +10 revision history is kept by default. -### To verify schema compatibility: +### Rollback with Database Restore ```bash -# Export current schema -kubectl -n atauth exec deploy/atauth-gateway -- \ - sqlite3 /app/data/gateway.db ".schema" > schema-before.sql +# Scale down +kubectl -n atauth scale deployment/atauth --replicas=0 -# After deployment, compare -kubectl -n atauth exec deploy/atauth-gateway -- \ - sqlite3 /app/data/gateway.db ".schema" > schema-after.sql +# Restore from pre-deploy backup +kubectl -n atauth exec -it -- \ + cp /app/data/gateway.db.backup /app/data/gateway.db -diff schema-before.sql schema-after.sql +# Rollback image and scale up +kubectl -n atauth rollout undo deployment/atauth +kubectl -n atauth scale deployment/atauth --replicas=1 ``` +### Restore from B2 Backup + +Automated backups run every 2 hours (CronJob in `backups` namespace, age-encrypted to Backblaze B2). + --- ## Monitoring During Deployment @@ -222,42 +136,17 @@ diff schema-before.sql schema-after.sql watch kubectl -n atauth get pods # Stream logs -kubectl -n atauth logs -f deployment/atauth-gateway +kubectl -n atauth logs -f deployment/atauth # Check events kubectl -n atauth get events --sort-by='.lastTimestamp' # Monitor health endpoint -watch -n 5 'curl -s https://auth.cloudforest-basilisk.ts.net/health | jq' +watch -n 5 'curl -s https://auth-staging.workingtitle.zip/health | jq' ``` ---- - -## Environment URLs - -| Environment | URL | Namespace | -|-------------|-----|-----------| -| Staging | https://auth-staging.cloudforest-basilisk.ts.net | atauth-staging | -| Production | https://auth.cloudforest-basilisk.ts.net | atauth | - -## Quick Commands Reference +## Important Notes -```bash -# Deploy staging -kubectl apply -k gateway/k8s/overlays/staging - -# Deploy production -kubectl apply -k gateway/k8s/overlays/production - -# Rollback -kubectl -n atauth rollout undo deployment/atauth-gateway - -# Check status -kubectl -n atauth get all - -# Logs -kubectl -n atauth logs -f deploy/atauth-gateway - -# Backup now -kubectl -n atauth create job --from=cronjob/atauth-gateway-backup manual-backup-$(date +%s) -``` +- **Image tag caching**: k8s nodes cache `:latest` with `imagePullPolicy: IfNotPresent`. Always use a unique tag (commit SHA or descriptive name) to ensure the new image is pulled. +- **RWO PVC**: Strategy must be `Recreate` since the SQLite PVC is ReadWriteOnce. RollingUpdate will deadlock when the new pod lands on a different node. +- **Domain**: Must use `workingtitle.zip`, NOT `arcnode.xyz` (same-site header issue with PDS). -- knotmirror 0.0.1