egress: add egress gateway support - atenet-egress by LiorLieberman · Pull Request #693 · agent-substrate/substrate

Lior Lieberman (LiorLieberman) · 2026-08-02T03:48:28Z

Added pluggable egress PEP support. (Feedback on atenet-egress name is welcomed - will open a separate PR to rename atenet-router to atenet-ingress )

#559 shipped the actor egress data path without any egress gateway. This adds an Envoy deployment that terminates actor CONNECTs. It requires downstream mTLS, so only a worker's atunnel can reach it.

The gateway consists of an Envoy and an atenet router --standalone ext_proc sidecar.

The same ext_proc binary now can serve both directions. It has a mode=ingress|egress|all.

--egress-gateway-address flag, This turns on pluggable actor egress cluster-wide. ateapi stamps this address onto every atelet Run/Restore, ateom hands it to atunnel, and actor TCP egress is transparently redirected into atunnel, which wraps it in mTLS + HTTP CONNECT to the egress gateway.

We should remove this when egress API pr is merged

One more FYI

The egress listener's is pinned to the actor-identity CA root alone, so the trust anchor is a property of the filter chain rather than the request, any client certificate not signed by the actor-identity CA fails the TLS handshake, and only traffic with actor-identities can open a tunnel through the egress gateway.

Tests pass
Appropriate changes to documentation are included in the PR

Lior Lieberman (LiorLieberman) · 2026-08-02T03:58:34Z

FYI Dmitry Berkovich (@dberkov)

haiyanmeng · 2026-08-04T01:11:51Z

+      # Envoy calls it over localhost to authenticate actor identity against the
+      # ate API on every CONNECT. This mirrors the ingress gateway topology
+      # (Envoy + ext_proc in one pod); a shared/standalone ext_proc is a future step.
+      - name: ext-proc


Let us make ext-proc a native sidecar (initContainers + restartPolicy: Always).
Regular containers get SIGTERM together, ext_proc has no drain and will exit first,
and with failure_mode_allow: false every CONNECT arriving during Envoy's drain
window fails closed with a 503.

good point. created an issue for that - #760

I've actually opened a PR to handle the graceful termination of the atenet-router (#774, which should help mitigate the Envoy drain window issue mentioned here.

You can also just have ext-proc register the SIGTERM but not quit (alternatively)

haiyanmeng · 2026-08-04T01:14:00Z

+      # Co-located ext_proc server (the atenet router, ext_proc-only). The egress
+      # Envoy calls it over localhost to authenticate actor identity against the
+      # ate API on every CONNECT. This mirrors the ingress gateway topology
+      # (Envoy + ext_proc in one pod); a shared/standalone ext_proc is a future step.


a shared/standalone ext_proc is a future step: can we first move the exe_proc logic into a separate binary?

ext_proc is already separate

I don't think so. See Line 268 in this file:

image: ko://github.com/agent-substrate/substrate/cmd/atenet

with what we have now - atenet == extproc server
atenet-egress deployment == envoy + atenet (handling egress releated extproc).

atenet-ingress deployment == envoy + atenet(handling ingress releated extproc).

Bowei Du (bowei) · 2026-08-04T21:07:17Z

implementation itself looks ok, but would like to fix a few organization things:

Make it so that egress, ingress for ext_proc can be enabled separately (add flags and gated deployment). We currently deploy both in the same deployment, but at scale, you would probably want to have separate ingress and egress deployments as they will be allocated resources very differently.
Put ingress,egress logic in .../atenet/router/{ingress,egress} packages.
router/ext_proc just has mux logic

probably something like:

switch {
case EgressEnabled && IsEgressRequest(...):
case IngressEnabled:
default:
   // send 4xx nothing is enabled.
}

Bowei Du (bowei)

You need to scrub through all of the comments -- there seems to some internal notes to yourself and your friend Claude.

Bowei Du (bowei) · 2026-08-05T06:06:31Z

Needs rebase

Eitan Yarmush (EItanya)

Offline we discussed cleaning up the locations in which the egressgateway address needs to be set, right now it's set on quite a few pods, but we can definitely simplify.

Eitan Yarmush (EItanya) · 2026-08-06T16:08:47Z

+	// assigned, and atunnel documents it as a lower bound on trustworthy actor
+	// metadata. If our authoritative view is older than what the worker asserts,
+	// we cannot yet vouch for the identity, so reject rather than allow blindly.
+	if assertedVersion != "" {


Will not be necessary once my PR goes in, this version is removed

Eitan Yarmush (EItanya) · 2026-08-06T17:09:42Z

Here is the list of specific PEP todos as a result of #708

  - Require mTLS client certificates signed by the actor-identity CA.
  - Verify validity period, ClientAuth EKU, and IsCA == false.
  - Require exactly one valid ActorIdentity extension.
  - Require non-empty atespace, actor name, and actor UID.
  - Require Purpose == "atunnel"; reject generic, missing, or unknown purposes.
  - Authorize using the verified actor UID—not CONNECT headers or actor-provided metadata.

Lior Lieberman (LiorLieberman) · 2026-08-07T22:28:34Z

+              # SEE(lior): this pair is how the actor's certificate reaches the
+              # ext_proc handler. ext_proc can request Envoy attributes, but none
+              # of them carry a custom X.509 extension, so the only way to check
+              # ActorIdentity in Go is to have Envoy hand over the raw chain.
+              # SANITIZE_SET drops whatever x-forwarded-client-cert the client
+              # sent and writes Envoy's own view of the verified peer, and
+              # chain: true puts the full URL-encoded PEM chain in it.


verify this is what we want

botengyao Taahir Ahmed (@ahmedtd)

It would be ideal to have a cleaner solution ... and ideally not in the config itself, but we should not block on getting it working.

verified with Boteng. This is ~fine. Added a TODO. Dont think an issue is needed since "cleaner" is undefined yet.

Bowei Du (bowei) · 2026-08-10T05:55:43Z

Probably closes #338

Bowei Du (bowei)

Mostly looks ok.

Bowei Du (bowei) · 2026-08-10T05:59:51Z

+              # SEE(lior): this pair is how the actor's certificate reaches the
+              # ext_proc handler. ext_proc can request Envoy attributes, but none
+              # of them carry a custom X.509 extension, so the only way to check
+              # ActorIdentity in Go is to have Envoy hand over the raw chain.
+              # SANITIZE_SET drops whatever x-forwarded-client-cert the client
+              # sent and writes Envoy's own view of the verified peer, and
+              # chain: true puts the full URL-encoded PEM chain in it.


It would be ideal to have a cleaner solution ... and ideally not in the config itself, but we should not block on getting it working.

Bowei Du (bowei) · 2026-08-10T06:02:55Z

+#   hack/install-ate-kind.sh --deploy-demo-egress
+#
+# The egress demo Actor accepts {"url":"..."} and performs an HTTP GET. With
+# egress turned on (ate-api-server --egress-gateway-address, which ateapi stamps


probably detail is no needed and could change in the future, can be confusing for humans and agents.

I believe you meant for the thing in the "()". Removed.

Bowei Du (bowei) · 2026-08-10T06:30:59Z

+
+	// Confirm the certified actor still exists. The name is only a lookup key
+	// here; the UID below is what actually authorizes.
+	actor, err := h.apiClient.GetActor(ctx, &ateapipb.GetActorRequest{


we should be careful here as this results in load to the ATEAPI server

file a TODO as a sub issue of the actor state caching issue

#592

added a ref to this issue here

Bowei Du (bowei) · 2026-08-10T06:33:35Z

+// is ever loosened, and costs one signature check per CONNECT rather than per
+// request. The IsCA, ClientAuth-EKU, and purpose checks below have no Envoy-side
+// equivalent at all.
+func (h *Handler) verifyActorCertificate(chain []*x509.Certificate) (*substratex509.ActorIdentity, error) {


put this and the functions it calls in its own package

egress/cert

cert.VerifyActor(...)

If a func doesn't really need to be a method, you should make it a free func for easier testing.

this PR has grew very big (partly because I addressed a similar comment from you to split ingress + egress under router). Lets take this as a follow up

The gateway terminates actor's CONNECT request. It requires downstream mTLS, so only a worker's atunnel can reach it. The gateway consists of an Envoy and an `atenet router --standalone` ext_proc sidecar.

demos/egress is a small Actor that fetches a URL it is given and echoes the upstream status and body back, which makes the egress path observable from outside the sandbox. hack/install-demo-egress.sh registers it as a --deploy-demo-egress fixture and hack/verify-egress-demo.sh drives it and checks the atenet-egress logs for the corresponding authorized CONNECT. TestActorEgress in the networking suite covers the same path automatically: it creates an Actor from the demo template, POSTs a fetch request through atenet-router, and asserts 200. The suite's actor helper is parameterised by template so the ingress test keeps using the counter fixture.

The gateway access-log assertion read the log once, immediately after curl returned, and failed if it saw nothing new. That is racy twice over: * Envoy emits the CONNECT access-log entry asynchronously, so for an external destination it can land seconds after the actor's response. * The Actor's HTTP client keeps the tunnel alive. A repeat fetch to a host it already reached rides the open tunnel and produces no new entry at all, so a run against a warm actor failed even though egress was working. Poll for the new entry, and fall back to any tunnel already open for this actor's SAN before declaring failure. Also read -c envoy explicitly (the pod also runs the ext-proc sidecar) and mark the script executable, as every other directly-invoked script in hack/ is.

Bowei Du (bowei) · 2026-08-11T06:20:58Z

+	// egress CONNECTs, and so the one that selects the egress handler. It must
+	// stay in sync with the filter chain name in
+	// manifests/ate-install/atenet-egress.yaml.
+	EgressFilterChainName = "egress"


Take this as a follow-up: it should be named something more unique to give a clue that there is something special about the name. Ideally, git grep <name> should just yield this and the configuration location.

Bowei Du (bowei) · 2026-08-11T06:28:02Z

There are a number of follow ups. Let's make sure we get them in as well.

Benjamin Elder (BenTheElder) · 2026-08-11T06:38:52Z

Bowei Du (@bowei) if we don't have clear logical commits, let's squash merge?

Lior Lieberman (LiorLieberman) requested review from Bowei Du (bowei) and John Howard (howardjohn) August 2, 2026 03:48

Lior Lieberman (LiorLieberman) force-pushed the pr2-atenet-egress branch from 305b232 to 598a548 Compare August 3, 2026 19:02

haiyanmeng reviewed Aug 4, 2026

View reviewed changes

Bowei Du (bowei) reviewed Aug 4, 2026

View reviewed changes

Comment thread cmd/atelet/main.go Outdated

Comment thread cmd/atelet/main.go Outdated

Comment thread cmd/atelet/main.go Outdated

Comment thread cmd/atenet/internal/router/extproc.go Outdated

Comment thread cmd/atenet/internal/router/extproc_egress.go Outdated

Lior Lieberman (LiorLieberman) mentioned this pull request Aug 5, 2026

Move atenet to a native sidecar container #760

Open

Eitan Yarmush (EItanya) reviewed Aug 6, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) force-pushed the pr2-atenet-egress branch from c133167 to b4bc40a Compare August 7, 2026 22:11

Lior Lieberman (LiorLieberman) requested review from Eitan Yarmush (EItanya) and Bowei Du (bowei) August 7, 2026 22:13

Lior Lieberman (LiorLieberman) force-pushed the pr2-atenet-egress branch from b4bc40a to acc96e7 Compare August 7, 2026 22:27

Lior Lieberman (LiorLieberman) commented Aug 7, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) force-pushed the pr2-atenet-egress branch from 9bdfc39 to ae9b6e9 Compare August 10, 2026 05:24

Bowei Du (bowei) reviewed Aug 10, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) force-pushed the pr2-atenet-egress branch 2 times, most recently from fbddff3 to 6ec3e03 Compare August 10, 2026 23:04

Lior Lieberman (LiorLieberman) requested a review from Bowei Du (bowei) August 10, 2026 23:07

Lior Lieberman (LiorLieberman) added 5 commits August 10, 2026 17:03

egress: add an Envoy egress gateway.

9220b82

The gateway terminates actor's CONNECT request. It requires downstream mTLS, so only a worker's atunnel can reach it. The gateway consists of an Envoy and an `atenet router --standalone` ext_proc sidecar.

fixes

17572a6

apply feedback 08/06

86d8bcf

Lior Lieberman (LiorLieberman) mentioned this pull request Aug 11, 2026

Future of sh scripts #842

Open

fixes

da323ec

Lior Lieberman (LiorLieberman) force-pushed the pr2-atenet-egress branch from 6ec3e03 to da323ec Compare August 11, 2026 00:10

Bowei Du (bowei) reviewed Aug 11, 2026

View reviewed changes

Bowei Du (bowei) approved these changes Aug 11, 2026

View reviewed changes

Bowei Du (bowei) merged commit 24cc538 into agent-substrate:main Aug 11, 2026
11 checks passed

Uh oh!

Conversation

Lior Lieberman (LiorLieberman) commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Lior Lieberman (LiorLieberman) commented Aug 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shrutiyam-glitch Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bowei Du (bowei) commented Aug 4, 2026

Uh oh!

Bowei Du (bowei) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Bowei Du (bowei) commented Aug 5, 2026

Uh oh!

Eitan Yarmush (EItanya) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Eitan Yarmush (EItanya) commented Aug 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bowei Du (bowei) commented Aug 10, 2026

Uh oh!

Bowei Du (bowei) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Bowei Du (bowei) commented Aug 11, 2026

Uh oh!

Lior Lieberman (LiorLieberman) commented Aug 2, 2026 •

edited

Loading

shrutiyam-glitch Aug 6, 2026 •

edited

Loading