Skip to content

hack: fix DNS on IPv6-only kind clusters - #958

Open
Yuan Gao (ygao-g) wants to merge 3 commits into
agent-substrate:mainfrom
ygao-g:kind-ipv6-coredns
Open

hack: fix DNS on IPv6-only kind clusters#958
Yuan Gao (ygao-g) wants to merge 3 commits into
agent-substrate:mainfrom
ygao-g:kind-ipv6-coredns

Conversation

@ygao-g

@ygao-g Yuan Gao (ygao-g) commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Part of #246.

  • Tests pass
  • Appropriate changes to documentation are included in the PR (none needed — the new IPV6_DNS_UPSTREAM knob is documented in the scripts' --help)

On a fresh IP_FAMILY=ipv6 cluster nothing resolves from inside a pod and no actor boots: CoreDNS runs dnsPolicy: Default and inherits the node's IPv4 resolver, which a v6-only pod cannot reach. Two Corefile clauses fix it — a forward to an IPv6 upstream, overridable with IPV6_DNS_UPSTREAM, and a hosts entry for kind-registry, which step 3 wires into containerd on the node but atelet pulls from its own pod netns. IPv4 and dual-stack pods get both names from Docker's embedded resolver, so this is gated on ipv6 and those families are untouched.

hack/verify-ipv6-dns.sh proves it and runs last in create-kind-cluster.sh, but stays separate so a live cluster can be re-checked: the hosts entry records an address the registry can move off, and the script names that case instead of blaming a registry that is down (#1049). CI has no IPv6 job (#939 is signal-only), so it runs by hand; results in a comment below. atenet-egress still crashloops on v6-only, for an unrelated Envoy bind bug.

🤖 Generated with Claude Code

@ygao-g
Yuan Gao (ygao-g) force-pushed the kind-ipv6-coredns branch 2 times, most recently from 45bac3d to ecf1fc6 Compare August 18, 2026 16:00
@ygao-g Yuan Gao (ygao-g) changed the title hack/create-kind-cluster.sh: fix DNS on IPv6-only kind clusters hack: fix DNS on IPv6-only kind clusters Aug 18, 2026
@ygao-g
Yuan Gao (ygao-g) force-pushed the kind-ipv6-coredns branch 3 times, most recently from e812e99 to 3350f55 Compare August 18, 2026 22:07
@ygao-g Yuan Gao (ygao-g) added kind/bug Something isn't working / bugfixes area/network labels Aug 18, 2026
@ygao-g
Yuan Gao (ygao-g) marked this pull request as ready for review August 19, 2026 00:32
@ygao-g

Yuan Gao (ygao-g) commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

Testing

Each check is paired with a control that must fail — the failure mode this guards against is a probe that passes without proving anything.

  • IPv6-only. IP_FAMILY=ipv6 hack/create-kind-cluster.sh--deploy-ate-system--deploy-demo-counter: atelet pulls the gVisor tarball from storage.googleapis.com over IPv6 from its own pod netns, the actor image pulls through kind-registry, and a golden actor runs and checkpoints. 11 of 11 fresh clusters exit 0 on a single probe pod. hack/run-e2e-kind.sh: demo 19/0/5 with a skip set identical to the IPv4 control, networking 7/2 — both failures are atenet-egress crashlooping on v6-only for an unrelated Envoy bind.
  • Controls. IPV6_DNS_UPSTREAM=2001:db8::1 exits 1 in ~28s, naming each replica that did not answer. Pointing the hosts entry at an address the registry is not on exits 1 with the Corefile says [...] but the registry is at [...] — the hack: IPv6-only kind clusters resolve a stale registry address #1049 case, told apart from a registry that is down. The stock Corefile fails the probe; dropping only the hosts block leaves external names working and the registry fetch failing.
  • IPv4 regression. Step 4.5 and the verifier do not run, the Corefile keeps forward . /etc/resolv.conf, and the demo stays green. dual takes the same path.

CI has no IPv6 job and this PR does not add one — #939 is a signal-only draft that produces that coverage on demand.

On a fresh IP_FAMILY=ipv6 cluster nothing resolves from inside a pod and
no actor boots: CoreDNS runs dnsPolicy: Default and inherits the node's
IPv4 resolver, which a v6-only pod cannot reach. Two Corefile clauses
fix it -- a forward to an IPv6 upstream, overridable with
IPV6_DNS_UPSTREAM, and a hosts entry for kind-registry. Step 3 already
wires the registry into containerd on the node, but atelet pulls from
its own pod netns, where the name does not resolve. IPv4 and dual-stack
pods get both names from Docker's embedded resolver, so this is gated on
ipv6 and those families are untouched.

hack/verify-ipv6-dns.sh proves it and runs last in
create-kind-cluster.sh, but stays separate so a live cluster can be
re-checked: the hosts entry records an address the registry can move
off, and the script names that case instead of blaming a registry that
is down (agent-substrate#1049).
The check queries each CoreDNS replica directly so that one bad replica
is named rather than showing up as a coin flip, but a failed leg only
printed a line into the probe log: the Service-level lookup still
answered, so the marker check passed and the message was never surfaced.
One bad replica out of two is half the pods in the cluster unable to
resolve, and the check reported success.

The legs now emit a marker the caller matches on, and a replica that
fails while the Service answers fails the check and is listed by
address.
The repoint was unconditional, but localhost reaches the apiserver only
where it resolves to ::1, and Debian-family hosts give ::1 the names
ip6-localhost and ip6-loopback instead. On a native Linux Docker host
kind publishes the apiserver on [::1] and that address already works,
so the rewrite replaced a working address with a refused one and every
later kubectl in the script died at connect -- including the DNS check
this series adds, which could not have run there at all.

It now repoints only when [::1] does not answer, which is the
VM-hosted-daemon case it was written for. On Ubuntu 25.10 with a native
daemon, [::1] serves /healthz, localhost is refused, and a full
IPv6-only bring-up including the DNS check completes.
@ygao-g

Copy link
Copy Markdown
Collaborator Author

Ping — this one is a two-file hack/ change, and it unblocks the IPv6-only CI job in #939, which carries it verbatim and can't merge first.

Worth knowing that the branch has moved since it was last looked at. Two commits went on, both fixes for defects I hit while trying to run it rather than responses to review: the DNS gate was reporting green with one of kind's two CoreDNS replicas unable to resolve, and the IPv6-only kubeconfig repoint was replacing a working apiserver address with a refused one on a native-Linux Docker host, which killed every kubectl after it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/network kind/bug Something isn't working / bugfixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant