S0: Executor NixOS configuration (nixery engine, LAN firewall) (ENG-464) #1

Merged

demi.dev 1w ago

targeting main from stoopkidddd/eng-464-s0-executor-nixos-configuration-nixery-engine-lan-firewall

Merged opened by and assigned to

demi.dev 1w ago targeting main from stoopkidddd/eng-464-s0-executor-nixos-configuration-nixery-engine-lan-firewall

Linear: ENG-464

Summary #

A NixOS config that turns the VM spindle-exec-1 into a spindle executor that dials out to the mill.

executor/flake.nix: nixpkgs nixos-26.05 (fa13731, the same as the bootstrap) and Tangled v1.20.0 (locked at 2096d06d). Output nixosConfigurations.executor.
services.tangled.spindle:
- role executor, connecting to wss://spindle.demi.dev/mill;
- the token is read from /var/lib/secrets/spindle-executor.token (LoadCredential), and the S3 credentials from /var/lib/secrets/spindle-executor.env;
- nixery engine: workflow timeout 15m, 4 workflows at once, 6144 MiB per job;
- seats 4, labels linux and x86_64;
- artifact store s3. The bucket name is a placeholder until ENG-465.
Docker logs rotate at 10 MiB × 3. Each week a timer prunes unused Docker data older than 7 days and collects Nix garbage older than 14 days.
nftables blocks CI jobs on docker0 and br-* from reaching RFC 1918, link-local and ULA addresses. DNS to 1.1.1.1 and the internet stay reachable.
SSH is key-only, uses the lead key, and is only reachable from RFC 1918 sources.
hardware-configuration.nix keeps the bootstrap's boot setup: GRUB on /dev/vda, the ext4 root labelled nixos with growPartition, the qemu guest agent and the ttyS0 console.

Verify #

$ mise run check
hadolint: no files yet
all checks passed!
/nix/store/cw7cjgqzmbgja8lndhsbvnza1mwzm3cq-nixos-system-spindle-exec-1-26.05.20261031.fa13731.drv
check_exit=0

$ nix eval (selected fields)
{"docker":true,"mill":"wss://spindle.demi.dev/mill","nft":true,"role":"executor","ssh22open":false,"timeout":"15m"}

The secret-pattern grep over the diff has 0 hits. The lane is local-only until the S3 smoke issue, and there is no CI yet.

Deviations #

DNS is pinned to 1.1.1.1 instead of the LAN resolver, so the firewall needs no hole into the LAN. Follow-up: ENG-474.
SSH is allowed from all RFC 1918 ranges, because the exact LAN subnet wasn't specified.
ICMPv6 neighbor discovery is allowed on Docker bridges; without it, public IPv6 egress breaks.

Review #

Two rounds of pneumo-reviewer, both done on the local branch before this PR could open:

round 1: REQUEST_CHANGES, for an out-of-scope README section;
round 2: APPROVE.

Both are posted as comments. The deploy checks are tracked in ENG-468.

🤖 Generated with Claude Code

Commits 0

Could not load commits.

Comments

v0 1w ago

demi.dev 1w ago

Review round 1 (pneumo-reviewer): REQUEST_CHANGES #

Reviewed on the local branch at 55075ed, before this PR existed.

The executor config is correct, and there is no FIX-level bug. It was checked against Tangled v1.20.0's nix/modules/spindle.nix, spindle/config/config.go and the nixery engine. The nftables ruleset was built under checkRuleset, and its netlink bytecode was dumped.

Only blocker (scope): the commit adds an "Executor configuration" section to README.md. That file is hot and isn't in the issue's Touches list. Round 2 removes it, and the deploy checklist moves to ENG-468.

Findings #

FILE (often 1 × bad 4 = 4). executor/configuration.nix:4,9-10,42: the VM and Docker use the single public resolver 1.1.1.1 with DHCP DNS disabled. If it is unreachable, the executor can't redial the mill and image pulls fail. Filed as ENG-474.
FILE (often 2 × bad 2 = 4). executor/configuration.nix:20: the executor requests the labels linux and x86_64. The mill must authorize both, or it drops the hello (spindle/mill/handler.go:132-136). Filed as ENG-475, to be fixed in ENG-468.

Spec and DoD checks #

ADR 0001 is met:
- executor role, nixery on Docker, dials wss://spindle.demi.dev/mill;
- no inbound internet ports: the only inbound rule is SSH from RFC 1918;
- tangled is locked at 2096d06d.
Module wiring:
- the token goes through LoadCredential, and the env file is a plain path, so neither enters the Nix store;
- the generated environment has workflow timeout 15m, 4 concurrent workflows, 6144 MiB, seats 4, labels linux,x86_64 and artifact store s3.
Firewall:
- the br-* prefix and docker0 are matched;
- the drop sets are 10/8, 172.16/12, 192.168/16, 169.254/16, fc00::/7 and fe80::/10;
- the forward and input hooks run at priority -10, before Docker's accepts;
- flushRuleset = false.
Boot matches executor/bootstrap/base.nix: GRUB on /dev/vda, the ext4 root by label nixos, growPartition, ttyS0, qemuGuest.
mise run check exits 0, and the toplevel drvPath is cw7cjgqz…-nixos-system-spindle-exec-1-26.05.20261031.fa13731.drv.
No secret values: the diff has only paths, a placeholder bucket and an SSH public key.
Not verified by eval: the toplevel build, since spindle compiles on the first switch. The deploy health check in ENG-468 covers this, and the same check confirms at runtime that NixOS nftables works alongside Docker 29's iptables-nft.

Review round 2 (pneumo-reviewer): APPROVE #

Reviewed on the local branch at c017df5, before this PR existed.

c017df5 only deletes the out-of-scope README section. git diff main..HEAD -- README.md is empty, and the diff now touches only the four files in Touches: executor/configuration.nix, flake.lock, flake.nix and hardware-configuration.nix. git diff 55075ed..HEAD -- executor is empty, so every round-1 check still holds. No scores changed.

The FILE findings are filed as ENG-474 and ENG-475. ENG-468 must verify two things at deploy that eval can't show:

the toplevel builds, with spindle compiled on the first nixos-rebuild;
one nixery job has public egress, and a LAN IP is unreachable from it.

mise run check: exit 0, toplevel cw7cjgqz…-nixos-system-spindle-exec-1-26.05.20261031.fa13731.drv.