feat(atenet): graceful termination of atenet-router

[shrutiyam-glitch]

Which issue(s) this PR is related to:

Fixes #721
Required for System Upgrade flow (#473)

What this PR does / why we need it:

This PR implements graceful termination and zero-outage rolling updates for atenet-router , building on the readiness probe and graceful drain patterns established in #719 (atelet).

Draining a two-container networking pod (atenet-router Go control-plane + envoy C++ proxy dataplane) introduces complex lifecycle interdependencies. This PR resolves the dual-container SIGTERM race, respects Envoy's failClosed ext_proc filter dependency, preserves parked requests riding out worker pool saturation, and accelerates idle deployments via an event-driven file handshake.

1. Event-Driven Dataplane Synchronization (emptyDir Marker File)

Envoy fast-exits by default on SIGTERM. To keep Envoy alive while the Go control-plane orchestrates the drain, we configured an IPC handshake between containers via a pod-shared emptyDir volume mounted at /var/run/atenet:

• Go Router Container: Removes any stale marker at startup, then writes /var/run/atenet/drain-complete when its shutdown sequence finishes.
• Envoy Container preStop Hook: Runs while [ ! -f /var/run/atenet/drain-complete ]; do sleep 0.5; done.

Outcome: Envoy exits as soon as — the drain is done, rather than wasting time in a fixed worst-case sleep. If the router crashes, Kubelet terminates the preStop hook at terminationGracePeriodSeconds: 60—slower cleanup, never a wedge.

2. The Multi-Container Shutdown Sequence (drain.go)

Because Kubernetes issues SIGTERM to both containers at once, a coordination state machine ensures Envoy never drops a connection and ext_proc is never stopped prematurely:

Phase / (best-case ex.) Timeline	atenet-router	envoy	K8s / Service Status
SIGTERM Sent ($t=0\text{s}$)	Catches SIGTERM • Flips /readyz $\rightarrow$ 503. • Starts 13s drain-delay.	Enters lifecycle.preStop hook: while [ ! -f .../drain-complete ]; do sleep 0.5; done • Kubelet holds SIGTERM back.	Deployment will recreate the pod. EndpointSlice controller begins dropping Old Pod IP.
Propagation ($t=0\text{s}$ – $t=13\text{s}$)	Keeps serving normally. • ext_proc continues unparking/routing requests.	Runs normally inside preStop loop. • Serves active TCP connections.	Service endpoint removal completes. No new connections arrive at Old Pod.
Envoy drain ($t=13\text{s}$)	Issues Envoy admin API calls: • /healthcheck/fail • /drain_listeners?graceful&skip_exit • Polls /stats for active downstream connections.	Begins graceful listener drain: • GOAWAY / Connection: close on established connections. • In-flight requests keep running.	In-flight HTTP requests finish executing through Envoy.
ext_proc drain ($t\approx18\text{s}$)	Active Envoy connections hit 0 (the poll exits early). • Calls extproc.GracefulStop(). • Parked request streams finish.	All downstream connections closed. • Still waiting in the preStop loop.	All client HTTP responses delivered.
Handshake & Exit ($t=18.5\text{s}$)	ext_proc drain finishes. • Writes drain-complete marker. • Hard-stops xDS (Stop()) & exits.	preStop loop detects marker file! • preStop exits 0. • Kubelet sends SIGTERM to Envoy. • Envoy exits immediately.	Old Pod deleted cleanly at ~18.5s (well under the 60s budget).

Envoy ref: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/operations/draining

Testing & Verification

1. Unit Tests (drain_test.go): Added comprehensive unit tests covering:

• Graceful completion of in-flight ext_proc streams within deadline.
• Force-stopping straggler streams past --drain-timeout.
• Envoy admin API interaction and connection polling.
• Stale marker cleanup and marker file creation.

2. Local testing on kind cluster (Observations):
   Test script: hack/verify-atenet-drain.sh

• Steady state: /readyz=200, /healthz=200; the instant the pod turned Terminating: /readyz=503 while /healthz=200 — NotReady but alive, for the whole drain.
• The parked request survived the shutdown: fired at a busy 1-worker pool → parked → pod deleted while parked → worker freed → HTTP=200, total=1.89s, body hello from: 169.254.17.2 | preserved memory count: 1 |
  preserved file counter: 1 — park → resume → route, served by the Terminating pod during its drain-delay window.
• Pod terminated 14s after deletion — the idle floor, confirmed across three runs: >13s drain-delay (sequence ran), ≪60s grace (marker handshake released Envoy's preStop; no SIGKILL).
• Log sequence captured verbatim, drain-delay honored to the millisecond (17:34:41.706 → 17:34:54.707):

Shutdown signal received; draining
(+13.000s) Draining Envoy {window: 15s}
Envoy drained
Starting ext_proc drain
ext_proc drain completed within deadline
Drain-complete marker written {path: /var/run/atenet/drain-complete}
Shutdown complete

• Tests pass
• Appropriate changes to documentation are included in the PR

[shrutiyam-glitch]

/retest

[Lior Lieberman (LiorLieberman)]

Thanks shrutiyam-glitch !

[Lior Lieberman (LiorLieberman)]

LGTM