atenet: bind the Envoy listeners and admin sockets dual-stack by ygao-g · Pull Request #2 · ygao-g/substrate

ygao-g · 2026-08-12T17:57:00Z

On an IPv6-primary cluster the kubelet probes a pod on its only address, which is
IPv6, and every Envoy socket in atenet was bound to the IPv4 wildcard.
atenet-egress crashlooped for 14h on a v6-only kind cluster from exactly that —
Envoy started fine, logged admin address: 0.0.0.0:15000, and then:

Startup probe failed: Get "http://[fd00:10:244::18]:15000/ready": connect: connection refused

The ingress listeners had the same gap on the data path: on a dual-stack cluster
Envoy answered on the router Service's IPv4 ClusterIP and nothing at all on its
IPv6 one.

Two socket shapes, two fixes. Ingress listeners pair each 0.0.0.0 primary
with an additional :: address on the same port, ipv4_compat: false — setting
it would clear IPV6_V6ONLY and collide with the primary already bound to that
port, and Envoy rejects the whole listener when an additional address fails to
bind, taking down all ingress rather than just the IPv6 half. The two admin
sockets and the egress :443 listener are single sockets, so they bind ::
with ipv4_compat: true. That flag is load-bearing, not incidental: dataplane.go
health-checks the router admin over http://127.0.0.1:9901/ready, and the egress
ext-proc sidecar's drainer dials 127.0.0.1:15000, where envoydrain.go reads a
refusal as "Envoy already exited" and skips the drain silently.

The atenet-router Service carried no ipFamilyPolicy, which defaults to
SingleStack — an IPv4 ClusterIP and nothing else, leaving the listeners above with
no IPv6 address to answer on. It becomes PreferDualStack (not Require, which
fails Service creation on a single-stack cluster), on both the shipped Service and
the one the router creates in managed mode.

No behaviour change on IPv4-only: the ingress primary is untouched, and a host
without IPv6 simply has no second socket to bind.

Not in scope: dns_lookup_family: V4_ONLY on the egress dynamic forward proxy, so
the gateway now accepts IPv6 downstream but still forwards only to IPv4
destinations.

Testing

go test ./cmd/atenet/internal/router/... passes.
TestReconcileEnvoyConfigMap_AdminAddress pins the generated bootstrap on both
the create and overwrite-a-stale-ConfigMap paths,
TestReconcileEnvoyService_IPFamilyPolicy covers create / SingleStack upgrade /
already-dual-stack, and TestXdsServer_UpdateSnapshot{,_WithHttps} assert the
:: additional address with ipv4_compat false.
On an IPv6-only kind cluster, atenet-egress went from 1/2 CrashLoopBackOff
(64 restarts, startup probe refused) to 2/2 Running with 0 restarts, Envoy
logging admin address: [::]:15000.
hack/install-ate-kind.sh --deploy-ate-system completes on that cluster.
Envoy's /listeners then reports all four sockets —
ingress_http_listener::0.0.0.0:8080, ingress_http_listener::[::]:8080,
ingress_https_listener::0.0.0.0:8443, ingress_https_listener::[::]:8443 —
and the router Service comes up PreferDualStack.
From inside the egress pod's netns, http://127.0.0.1:15000/ready and
http://[::1]:15000/ready both return 200, so the single ipv4_compat socket
still serves the drainer's IPv4 loopback dial.
Not run: the networking e2e suites. They need actors, and actors cannot boot on
a v6-only kind cluster until the CoreDNS resolver fix lands.

ygao-g · 2026-08-13T04:04:46Z

    admin:
      address:
-        socket_address: { address: 0.0.0.0, port_value: 15000 }
+        # ipv4_compat because the ext-proc sidecar's drainer dials this over IPv4


evaluate whether we need this comment.

ygao-g · 2026-08-13T04:12:33Z

+// existing Service rather than replacing the spec, so a Service created by an
+// earlier router keeps whatever policy it was created with unless this field is
+// copied too, and never gets the IPv6 ClusterIP the "::" listeners need.
+func TestReconcileEnvoyService_IPFamilyPolicy(t *testing.T) {


Seems like a lot of redundancy in test setup, can we extend TestReconcileEnvoyConfigMap_AdminAddress instead to check additional fields

ygao-g · 2026-08-13T04:18:36Z

+
+		addrs := l.GetAdditionalAddresses()
+		if len(addrs) == 0 {
+			t.Errorf("Expected additional addresses, got none")


Nit. return early here to avoid else branch

The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a dual-stack cluster Envoy answered on the router Service's IPv4 ClusterIP and nothing at all on its IPv6 one. Pair each primary socket with an additional "::" address on the same port. Ipv4Compat is false on the additional address. Setting it would clear IPV6_V6ONLY and collide with the primary IPv4 wildcard already bound to that port, and Envoy rejects the whole listener when an additional address fails to bind -- that would take down all ingress, not just the IPv6 half. Hoisting the literal into a helper keeps the two listeners from drifting. No behaviour change on an IPv4-only cluster: the primary address is untouched, and a host without IPv6 simply has no second socket to bind.

The Envoy admin socket bound 0.0.0.0, leaving it reachable over IPv4 only. It binds "::" with ipv4_compat now -- one socket for both families. ipv4_compat is required rather than incidental here: Envoy sets IPV6_V6ONLY without it, and dataplane.go health-checks the listener over http://127.0.0.1:9901/ready, so dropping it would take the dataplane component of /statusz unhealthy. The atenet-router Service carried no ipFamilyPolicy, which the API server defaults to SingleStack -- an IPv4 ClusterIP and nothing else, which leaves the listeners above with no IPv6 address to answer on. Prefer, not Require, so this stays valid on a single-stack cluster, where it is a no-op. spec.ipFamilies is deliberately left alone: the primary family is immutable and the API server appends the secondary one itself.

…families The gateway's two Envoy sockets bound 0.0.0.0, so on an IPv6-primary cluster the kubelet's startup probe against the admin port was refused and atenet-egress crashlooped while Envoy itself started fine and logged "admin address: 0.0.0.0:15000". The :443 listener had the same gap, leaving no v6 path for an actor's CONNECT. Both are single sockets, so they bind "::" with ipv4_compat rather than taking the additional-address pairing the ingress listeners use. On the admin socket ipv4_compat is load-bearing: the ext-proc sidecar's drainer reaches it at 127.0.0.1:15000 and envoydrain.go reads a refusal as "Envoy already exited", so a bare "::" would silently skip the drain. The Service gets PreferDualStack for the same reason the router's does -- without it a dual-stack cluster hands out one ClusterIP and the new v6 bind is unreachable.

ygao-g · 2026-08-13T15:06:54Z

Superseded by agent-substrate#911, which carries the same work rebased onto current upstream main.

The rebase dropped the envoyrunner.go changes and envoyrunner_test.go from this PR: upstream agent-substrate#635 (atenet: drop redundant deployment mode) deleted that file, so the shipped manifest is now the only copy of the router bootstrap. What remains is the xDS listener change plus the two manifests — 3 commits, 4 files, +74/-3. The egress Service also picked up the ipFamilyPolicy it was missing.

Tracked upstream as agent-substrate#910.

…-substrate#1002) Fixes agent-substrate#1001 `TestK8sResolverEndpointSliceUpdates` intermittently fails at `resolver_test.go:215` — 8 of the 30 most recent failed `pr-workflow` runs (~27%): ``` updated state.Addresses = [{Addr: "10.0.0.1:443", ServerName: "", }], want [{Addr: "10.0.0.1:443", ServerName: "", } {Addr: "10.0.0.2:443", ServerName: "", }] ``` `Build` starts a goroutine that calls `updateState` once `WaitForCacheSync` returns. That report is deliberate: a service with no EndpointSlices never fires `AddFunc`, so without it the resolver would stay silent instead of telling gRPC the answer is an empty set. But `WaitForCacheSync` polls at `syncedPollPeriod = 100ms`, so it fires roughly 100ms after Build — and if the test has not yet created the second slice by then, that update still carries only `10.0.0.1` and sits in the channel ahead of the real one. The second `select` took whatever came next, so it asserted against the stale update. Locally Build-to-Create is 0.6ms, well ahead of the timer, which is why this only shows up on loaded runners. The resolver is not at fault — it promises eventual convergence, not that the first update after a change is final, and a duplicate update costs gRPC nothing. So both waits now go through one `waitForAddrs` helper that consumes updates until the set matches, with a timeout so a genuinely broken resolver still fails and reports the last set it saw. Note this changes the first wait as well: it asserted the *first* update equals `[10.0.0.1]`, and now waits for that set instead. The same argument applies there — nothing promises the first update is final. ## Verification `-count=N` proves nothing here: the unfixed test passes locally at any count because the window is never hit. A/B with the Build→Create delay as the only variable: | delay | old assertion | new assertion | |---|---|---| | 0ms | 5/5 pass | 5/5 pass | | 150ms | **0/5 pass** | **5/5 pass** | Measured timeline with the 150ms stall in place: ``` [ 0.5ms] update #1: [10.0.0.1] <- AddFunc for slice1 first select takes it ... 150ms stall ... [101.1ms] update #2: [10.0.0.1] <- the WaitForCacheSync goroutine test creates slice2 [151.4ms] update #3: [10.0.0.1, 10.0.0.2] ``` The probe tests used for this are not included. Rebased over agent-substrate#1013. That fixes a different bug — concurrent `updateState` calls letting an older address set win — and does not close this one: the two updates here are ~100ms apart, so the queue has nothing to coalesce. Re-measured on top of it, unchanged: at a 150ms delay the old assertion is 0/5 and the new one 5/5.

ygao-g commented Aug 13, 2026

View reviewed changes

ygao-g force-pushed the atenet-envoy-dualstack branch from a0a0067 to 7b109d0 Compare August 13, 2026 04:20

ygao-g added 3 commits August 13, 2026 07:19

ygao-g force-pushed the atenet-envoy-dualstack branch from 697cd22 to 5308c09 Compare August 13, 2026 14:25

ygao-g closed this Aug 13, 2026

Navigation Menu

atenet: bind the Envoy listeners and admin sockets dual-stack - #2

atenet: bind the Envoy listeners and admin sockets dual-stack#2
ygao-g wants to merge 3 commits into
mainfrom
atenet-envoy-dualstack

ygao-g commented Aug 12, 2026

Uh oh!

ygao-g Aug 13, 2026

Uh oh!

ygao-g Aug 13, 2026

Uh oh!

ygao-g Aug 13, 2026

Uh oh!

ygao-g commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ygao-g commented Aug 12, 2026

Testing

Uh oh!

ygao-g Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

ygao-g Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

ygao-g Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

ygao-g commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant