Temporary: stage local-sandbox execution fallback (Agent Sandbox CRD unavailable)

[aredenba-rh]

Summary

The stage cluster does not yet have the Agent Sandbox CRD/controller installed (blocker tracked informally as "Blocker A" in scripts/verify-fleet-apps-kartograph.sh), so GMA sticky sessions and batch extraction jobs cannot run their OpenShell sandboxes in-cluster on stage today — only in dev. This is expected to be resolved eventually by the platform team (see #774), but until then, this issue tracks a temporary design so both GMA and extraction jobs can still run on stage by delegating Sandbox execution to each user's own machine, reusing dev's existing local-sandbox code paths as-is.

This is explicitly meant to be backed out once the Agent Sandbox CRD lands on stage (#774) — no permanent architectural fork.

Design (from discussion, see chat transcript)

• Stage's API cannot dial into a user's laptop (NAT/firewalls). Direction is inverted: a small local agent (same shape as the ArgoCD PKCE helper in .cursor/skills/kartograph/scripts/argocd_pkce_login.py) runs on each user's machine, logs in once against stage via OAuth2 PKCE (own OAuth client id, revocable independently of the user's normal session), and long-polls stage for pending work — no inbound connectivity required.
• Reuse existing gateway-agnostic OpenShell code (extraction/infrastructure/openshell/sandbox.py, policy.py, gateway.py) unchanged — the local agent points the same functions at a locally-registered gateway/provider instead of the in-cluster one. Only the target changes, not sandbox lifecycle logic.
• New pieces needed are narrow:
  • A LocalRuntimeWorkRequest-style dispatch (pending/claimed/completed/failed) so stage can hand off "a Sandbox is needed for user X" without calling into the laptop directly.
  • Identity-to-gateway routing: each user's local agent is 1:1 with their own Kartograph identity (mirrors dev, where there's implicitly one gateway per developer).
  • Fail-closed Landlock enforcement on the local agent — Landlock is Linux-only; refuse to run rather than silently downgrading hard_requirement to soft (see extraction/infrastructure/openshell/policy.py).
• Credential model (decided): do not ship the long-lived Vault-backed Vertex ADC/service-account key to laptops. Mint a short-lived, single-job-scoped downscoped Google Cloud access token server-side and inject it directly via OpenShell's existing google-vertex-ai provider service_account_token credential (VERTEX_AI_SERVICE_ACCOUNT_TOKEN env var, --credential flag) — confirmed this credential type accepts a raw bearer token with no refresh capability, so it self-expires with no private key ever leaving the cluster. This directly informs the still-open design in Design prod Vertex/GCP credential injection for extraction agent containers #773.

Workstreams

• Spec: new specs/extraction/*.spec.md (or amendment to specs/nfr/workload-execution.spec.md's "Container-Only Agent Runtime" requirement) describing the temporary local-dispatch mode, per this repo's spec-first workflow.
• Local agent login/registration (~/.config/kartograph/local-agent.json), OAuth2 PKCE against Keycloak, own client id.
• Work dispatch: hosted-side IStickySessionRuntimeManager / IExtractionJobRunner adapter that writes work requests instead of calling openshell sandbox create in-cluster.
• Long-poll endpoint for the local agent to pull pending work (reuse existing Postgres LISTEN/NOTIFY outbox infra rather than a new transport).
• Downscoped Vertex token minting service (short TTL matched to job/session duration) + injection via OpenShell's service_account_token credential.
• Fail-closed Landlock check in the local agent before accepting work.
• Session/job UX for waiting/offline/claim-timeout states (new waiting_for_local_runner phase alongside existing starting/ready/unhealthy/failed).

Acceptance criteria

1. A GMA sticky session and an extraction job can both complete end-to-end on stage with the Sandbox physically running on a user's own machine.
2. No long-lived Vault-backed Vertex credential ever leaves the cluster; only short-lived, single-job-scoped tokens do.
3. Local agent refuses to execute if Landlock enforcement can't be verified on the host.
4. Removing this fallback (once Production: OpenShell-backed GMA and extraction workloads #774's CRD work lands) requires no changes to sandbox lifecycle code — only the adapter/routing layer.

References

• Production: OpenShell-backed GMA and extraction workloads #774 — Production: OpenShell-backed GMA and extraction workloads (tracks the real fix: Agent Sandbox CRD on stage)
• Design prod Vertex/GCP credential injection for extraction agent containers #773 — Design prod Vertex/GCP credential injection for extraction agent containers (this issue's credential decision directly informs it)
• specs/nfr/workload-execution.spec.md, specs/extraction/sticky-session-runtime.spec.md
• src/api/extraction/infrastructure/openshell/ (sandbox.py, policy.py, gateway.py, vertex_provider.py)
• scripts/verify-fleet-apps-kartograph.sh ("Blocker A")

Priority

Temporary workaround — priority depends on how urgently GMA/extraction is needed on stage before #774 lands.