S3: Add a secondary public DNS resolver (ENG-474) #7

Merged

demi.dev 1w ago

targeting main from stoopkidddd/eng-474-review-follow-up-executor-dns-depends-on-the-single-public

Merged opened by and assigned to

demi.dev 1w ago targeting main from stoopkidddd/eng-474-review-follow-up-executor-dns-depends-on-the-single-public

Linear: ENG-474. It blocks the S3 gate, ENG-472.

Why #

Pneumo's push pipeline 3mzbfhbmlxh22 went red in android.yml with java.net.UnknownHostException: github.com. In about 45 minutes of heavy CI, Docker's embedded resolver logged two i/o timeouts against its only upstream, 1.1.1.1. With a single upstream, one dropped UDP query fails the lookup.

Change #

executor/configuration.nix: dnsResolver becomes dnsResolvers = [ "1.1.1.1" "9.9.9.9" ]. The list is used for:

networking.nameservers;
Docker daemon.settings.dns;
the nftables docker-egress DNS exception, TCP and UDP 53.

Every other drop rule is unchanged.

Verify #

$ mise run check   → exit 0 (toplevel 82irpkh6…-nixos-system-spindle-exec-1…drv)
$ nix eval (selected)
{"docker":["1.1.1.1","9.9.9.9"],"ns":["1.1.1.1","9.9.9.9"],"nft":"… ip daddr { 1.1.1.1, 9.9.9.9 } meta l4proto { tcp, udp } th dport 53 accept …"}

After the merge, the lead deploys the change to spindle-exec-1 and re-runs the container DNS and LAN-isolation probes.

🤖 Generated with Claude Code

Commits 0

Could not load commits.

Comments

v0 1w ago

demi.dev 1w ago

Review round 1 (pneumo-reviewer): APPROVE #

Reviewed at 00b7438 (v0). The change is executor/configuration.nix only, +4/-4. It adds 9.9.9.9 after 1.1.1.1 in three places:

networking.nameservers;
Docker daemon.settings.dns;
the nftables docker-egress DNS accept rule.

The drop sets are byte-for-byte unchanged.

Checks:

Fallback works. In Moby 29.9.2's daemon/libnetwork/resolver.go, an upstream timeout makes exchange() return nil, and forwardExtDNS() then continues to the next upstream (extIOTimeout is 4 s). With a single upstream, the client got SERVFAIL, which Java reports as UnknownHostException. That matches pipeline 3mzbfhbmlxh22. Nixery seats are user-defined networks, so jobs use 127.0.0.11 with the daemon's list.
The ruleset is valid. The real nftables-rules derivation (nft --check) built. It renders as ip daddr { 1.1.1.1, 9.9.9.9 } meta l4proto { tcp, udp } th dport 53 accept. A negative control with a malformed set failed, which shows the check catches errors.
LAN isolation is not weakened.

Findings: one FILE (3 × 1 = 3), docs/runbook.md:401 "Container DNS uses only 1.1.1.1", which goes stale after the deploy. Added to ENG-496.

Deploy notes:

The switch restarts dockerd (live-restore is false), so switch only while the executor is idle.
The new DNS list applies to containers created after the restart.

DoD: hadolint and shellcheck are clean; mise run check exits 0 (drv 82irpkh6…); the PR pipeline passed 1/1.

Why #

Change #

Verify #

Versions

Review round 1 (pneumo-reviewer): APPROVE #