atenet: bind the Envoy sockets and Services dual-stack - #911
Open
Yuan Gao (ygao-g) wants to merge 2 commits into
Open
atenet: bind the Envoy sockets and Services dual-stack#911Yuan Gao (ygao-g) wants to merge 2 commits into
Yuan Gao (ygao-g) wants to merge 2 commits into
Conversation
Yuan Gao (ygao-g)
marked this pull request as ready for review
August 13, 2026 15:19
Collaborator
Author
|
Bowei Du (@bowei) mind taking a look? |
This was referenced Aug 13, 2026
Yuan Gao (ygao-g)
force-pushed
the
atenet-envoy-dualstack
branch
from
August 14, 2026 18:10
5308c09 to
3162d63
Compare
Yuan Gao (ygao-g)
force-pushed
the
atenet-envoy-dualstack
branch
2 times, most recently
from
August 18, 2026 14:59
5257d6b to
580afef
Compare
Yuan Gao (ygao-g)
changed the title
This was referenced Aug 18, 2026
Yuan Gao (ygao-g)
added
kind/bug
Yuan Gao (ygao-g)
force-pushed
the
atenet-envoy-dualstack
branch
3 times, most recently
from
August 19, 2026 03:40
378c535 to
5c35bdc
Compare
Yuan Gao (ygao-g)
force-pushed
the
atenet-envoy-dualstack
branch
3 times, most recently
from
August 19, 2026 22:58
24055c6 to
6c1910d
Compare
Yuan Gao (ygao-g)
force-pushed
the
atenet-envoy-dualstack
branch
from
August 19, 2026 23:48
6c1910d to
cff4f76
Compare
Yuan Gao (ygao-g)
force-pushed
the
atenet-envoy-dualstack
branch
from
August 20, 2026 00:34
be492e5 to
e48bdf3
Compare
The router and egress manifests bind the IPv4 wildcard on every Envoy socket, and neither gateway's Service asks for a second IP family. On a dual-stack cluster the router answers on its IPv4 ClusterIP and on nothing at all for IPv6; on an IPv6-primary cluster the kubelet cannot probe the pod on its only address, so atenet-egress crashloops while Envoy itself starts fine. Both gateways now bind `::` as well and ask for PreferDualStack. That makes them accept IPv6, not reach it: the egress dns_lookup_family, the DNS AAAA path, and atunnel's original-destination lookup stay IPv4. The experimental sdsmint egress variant is untouched.
Yuan Gao (ygao-g)
force-pushed
the
atenet-envoy-dualstack
branch
from
August 20, 2026 05:27
e48bdf3 to
cf406b1
Compare
Yuan Gao (ygao-g)
added a commit
to ygao-g/substrate
that referenced
this pull request
Aug 20, 2026
Nothing checked that the router's dataplane listeners bind more than an IPv4 socket, and nothing reached an actor over the router's IPv6 ClusterIP. Every other path a test has into the router -- a port-forward, the pods/proxy and services/proxy subresources -- is mediated by the API server, which picks the family, so no existing test could have caught a listener that lost its IPv6 socket. Reads the bound addresses from Envoy's own admin /listeners, and drives an in-cluster probe pod at the router over each ClusterIP in turn. Red until agent-substrate#911 binds those sockets, so this stays a draft until then. The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
Yuan Gao (ygao-g)
added a commit
to ygao-g/substrate
that referenced
this pull request
Aug 20, 2026
Nothing checked that the router's dataplane listeners bind more than an IPv4 socket, and nothing reached an actor over the router's IPv6 ClusterIP. Every other path a test has into the router -- a port-forward, the pods/proxy and services/proxy subresources -- is mediated by the API server, which picks the family, so no existing test could have caught a listener that lost its IPv6 socket. Reads the bound addresses from Envoy's own admin /listeners, and drives an in-cluster probe pod at the router over each ClusterIP in turn. Red until agent-substrate#911 binds those sockets, so this stays a draft until then. The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
Yuan Gao (ygao-g)
added a commit
to ygao-g/substrate
that referenced
this pull request
Aug 21, 2026
Nothing checked that the router's dataplane listeners bind more than an IPv4 socket, and nothing reached an actor over the router's IPv6 ClusterIP. Every other path a test has into the router -- a port-forward, the pods/proxy and services/proxy subresources -- is mediated by the API server, which picks the family, so no existing test could have caught a listener that lost its IPv6 socket. Reads the bound addresses from Envoy's own admin /listeners, and drives an in-cluster probe pod at the router over each ClusterIP in turn. Red until agent-substrate#911 binds those sockets, so this stays a draft until then. The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
Yuan Gao (ygao-g)
added a commit
to ygao-g/substrate
that referenced
this pull request
Aug 21, 2026
The rcode assertions in the previous commit prove the zone stops failing an AAAA query, not that it ever answers one. Nothing checks that the record the zone does publish is the router's IPv6 ClusterIP, so agent-substrate#938 could regress to an empty answer and every existing test would still be green. Kept separate from TestActorDNSZone because it is the only assertion here whose expected result changes with the cluster: it skips wherever atenet-router has a single ClusterIP, which is every cluster until agent-substrate#911 gives the Service a dual-stack policy. Part of agent-substrate#246.
Yuan Gao (ygao-g)
added a commit
to ygao-g/substrate
that referenced
this pull request
Aug 21, 2026
Nothing checked that the router's dataplane listeners bind more than an IPv4 socket, and nothing reached an actor over the router's IPv6 ClusterIP. Every other path a test has into the router -- a port-forward, the pods/proxy and services/proxy subresources -- is mediated by the API server, which picks the family, so no existing test could have caught a listener that lost its IPv6 socket. Reads the bound addresses from Envoy's own admin /listeners, and drives an in-cluster probe pod at the router over each ClusterIP in turn. Red until agent-substrate#911 binds those sockets, so this stays a draft until then. The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
Yuan Gao (ygao-g)
added a commit
to ygao-g/substrate
that referenced
this pull request
Aug 21, 2026
Nothing checked that the router's dataplane listeners bind more than an IPv4 socket, and nothing reached an actor over the router's IPv6 ClusterIP. Every other path a test has into the router -- a port-forward, the pods/proxy and services/proxy subresources -- is mediated by the API server, which picks the family, so no existing test could have caught a listener that lost its IPv6 socket. Reads the bound addresses from Envoy's own admin /listeners, and drives an in-cluster probe pod at the router over each ClusterIP in turn. Red until agent-substrate#911 binds those sockets, so this stays a draft until then. The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
| @@ -0,0 +1,86 @@ | |||
| #!/usr/bin/env bash | |||
Collaborator
There was a problem hiding this comment.
what is this script used for
I think we should just rely on e2e tests to tell us that IPv6 is working.
Collaborator
Author
There was a problem hiding this comment.
Removed. CI does catch it on the egress side — the kubelet probes that admin port, so dropping ipv4_compat means the pod never goes Ready and every run turns red. The router's admin port has no such probe, and the drain reads a refused dial as "Envoy already exited", so I dropped the dependency instead of guarding it: the in-pod callers dial localhost now, with a unit test against an IPv6-only loopback listener.
Collaborator
|
Changes look ok, but can we remove the extra script? |
The router's dataplane health check and the drain sequence dial the Envoy admin interface on the IPv4 loopback, which now depends on the admin socket keeping ipv4_compat set alongside its `::` bind. Losing that fails silently: the drain reads the refused dial as "Envoy already exited" and reports a drain it never performed. Both callers now dial localhost, which resolves to either loopback, so the drain no longer depends on how the socket is spelled. ipv4_compat stays for the egress kubelet probe, where a regression turns any IPv4 run red on the spot.
Yuan Gao (ygao-g)
force-pushed
the
atenet-envoy-dualstack
branch
from
August 21, 2026 19:18
cf406b1 to
20951c0
Compare
Collaborator
Author
|
Done. The in-pod callers now dial |
Bowei Du (bowei)
approved these changes
Aug 21, 2026
Yuan Gao (ygao-g)
added a commit
to ygao-g/substrate
that referenced
this pull request
Aug 21, 2026
The rcode assertions in the previous commit prove the zone stops failing an AAAA query, not that it ever answers one. Nothing checks that the record the zone does publish is the router's IPv6 ClusterIP, so agent-substrate#938 could regress to an empty answer and every existing test would still be green. Kept separate from TestActorDNSZone because it is the only assertion here whose expected result changes with the cluster: it skips wherever atenet-router has a single ClusterIP, which is every cluster until agent-substrate#911 gives the Service a dual-stack policy. Part of agent-substrate#246.
Yuan Gao (ygao-g)
added a commit
to ygao-g/substrate
that referenced
this pull request
Aug 21, 2026
Nothing checked that the router's dataplane listeners bind more than an IPv4 socket, and nothing reached an actor over the router's IPv6 ClusterIP. Every other path a test has into the router -- a port-forward, the pods/proxy and services/proxy subresources -- is mediated by the API server, which picks the family, so no existing test could have caught a listener that lost its IPv6 socket. Reads the bound addresses from Envoy's own admin /listeners, and drives an in-cluster probe pod at the router over each ClusterIP in turn. Red until agent-substrate#911 binds those sockets, so this stays a draft until then. The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
Yuan Gao (ygao-g)
added a commit
to ygao-g/substrate
that referenced
this pull request
Aug 21, 2026
Nothing checked that the router's dataplane listeners bind more than an IPv4 socket, and nothing reached an actor over the router's IPv6 ClusterIP. Every other path a test has into the router -- a port-forward, the pods/proxy and services/proxy subresources -- is mediated by the API server, which picks the family, so no existing test could have caught a listener that lost its IPv6 socket. Reads the bound addresses from Envoy's own admin /listeners, and drives an in-cluster probe pod at the router over each ClusterIP in turn. Red until agent-substrate#911 binds those sockets, so this stays a draft until then. The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #910
Part of #246
The router and egress manifests bind the IPv4 wildcard on every Envoy socket, and neither gateway's Service asks for a second IP family. On a dual-stack cluster the router answers on its IPv4 ClusterIP and on nothing at all for IPv6; on an IPv6-primary cluster the kubelet cannot probe the pod on its only address, so
atenet-egresscrashloops while Envoy itself starts fine.Both gateways now bind
::as well, and both Services ask forPreferDualStack. A newhack/verifycheck guards theipv4_compatflag that the in-pod loopback callers depend on, since nothing else in the tree reads these manifests. The experimentalatenet-egress-with-sdsmint.yamlvariant is deliberately out of scope.This makes both gateways accept IPv6; it does not make them reach IPv6 destinations. The egress
dns_lookup_family, the DNS AAAA path, and atunnel's original-destination lookup stay IPv4, and are tracked in #246 and #686.Testing
CI has no IP-family matrix, so the green e2e run here is the IPv4 regression check and nothing more. The other two families were verified by hand on kind, using
IP_FAMILYfrom #877.[::]:443socket. Reverting only the two bind lines turns those assertions red, so they are not passing vacuously.ipv4_compatwas read from/config_dump—/listenersnever emits it.atenet-egressgoes from1/2 CrashLoopBackOff(64 restarts, startup probe refused) to2/2 Running, andTestActorDirectAccesspasses, including through the router's ingress.TestActorEgressstill fails inside the actor's netns, which is ateomnet: enable IPv6 forwarding in worker pod netns #979's scope.🤖 Generated with Claude Code