spindle
Self-hosted Tangled spindle
Star 0
Fork 0
Watch

S3: Document the spindle operations runbook (ENG-471) #5

Merged demi.dev
targeting main from stoopkidddd/eng-471-s3-runbook-upgrades-token-rotation-backups-and-outages
Merged opened by and assigned to demi.dev
targeting main from stoopkidddd/eng-471-s3-runbook-upgrades-token-rotation-backups-and-outages

Linear: ENG-471

Summary #

docs/runbook.md: one page for operating the spindle. The README now links to it.

  • Architecture and state: where each piece of state lives (mill /data, the VM's /var/lib/secrets, the bucket, the signing key).
  • Health and connection checks: /_health, did.json, executor list, and the mill and VM connection log lines.
  • Upgrading the pinned Tangled version:
    • one PR that changes both pins;
    • railway up;
    • the proven detached build and switch-to-configuration on the VM.
  • Executor tokens: the commands proven in ENG-468.
    • Rotation uses a new registration name, because add rejects an existing name with a different token. It also needs executor reset-stream on the idle VM before the reconnect, then revoke of the old name.
    • The lead will run this drill exactly as written.
  • Bucket credentials, signing key (restore and lost-key), mill backups and /data restore, outages, cost.

Steps that haven't been exercised yet are marked "(not yet drilled)".

Verify #

$ mise run check   → exit 0 (… nixos-system-spindle-exec-1 … bgyd1x38 drv)
all 19 documented shell blocks pass `bash -n` (syntax only, not executed)
secret-pattern grep over docs/runbook.md: 0 hits

🤖 Generated with Claude Code

Commits 0