Run NemoClaw E2E against an exact staging Brev Launchable

[cjagwani]

Summary

Add one Brev Launchable lane to the existing NemoClaw E2E workflow. The lane tests the image built for the exact candidate SHA as it is installed in the Launchable; it does not overlay a checkout, reinstall NemoClaw, or create a separate image-qualification system.

Target release: v0.0.91.

Required workflow

1. Resolve the exact candidate SHA selected by the trusted E2E controller.
2. Dispatch the existing nemoclaw-image staging build for that SHA and wait for that exact producer run to succeed.
3. Deploy the configured staging Brev Launchable.
4. Wait for the workspace, then require both the baked provision metadata and clean baked NemoClaw checkout to report the exact candidate SHA.
5. Run the existing test/e2e/live/full-e2e.test.ts suite in preinstalled mode.
6. Attach redacted logs and the candidate/workspace identities, then delete the workspace on every terminal path.

Candidate code remains data. Image-builder and Brev credentials stay in trusted workflow code.

Acceptance criteria

• .github/workflows/e2e.yaml has one Brev Launchable lane that can run for an authorized PR candidate and for the intended trusted manual or scheduled runs.
• The lane uses the exact candidate SHA selected by the E2E workflow and prepares the image once for that run.
• The lane waits for the exact producer run returned by the existing staging image workflow.
• The deployed workspace reports the exact candidate SHA in both its baked provision metadata and clean NemoClaw checkout before tests start.
• The lane runs the existing full E2E suite in preinstalled mode without rsync, install.sh, dependency installation, rebuild, or relink.
• The workflow uploads redacted failure or success logs with the candidate and workspace identities.
• Workspace deletion runs after success, failure, timeout, or cancellation and verifies the workspace is absent.
• Existing non-Launchable E2E jobs remain unchanged.
• Focused no-cloud tests cover the happy path, SHA mismatch, E2E failure with cleanup, and deletion failure.
• At least one authorized PR run completes successfully on a real Brev Launchable.

Non-goals

• A standalone image-qualification workflow, controller, state machine, evidence ledger, generalized manifest protocol, or in-guest cloud image-ID lookup.
• Release gating, production-image promotion, or moving latest or lkg.
• Creating, editing, repointing, or pinning a Launchable per candidate.
• Expanding the Launchable lane beyond the existing full E2E behavior needed for this issue.

Dependencies

• The existing exact-SHA staging image producer in brevdev/nemoclaw-image#80.
• A staging Launchable that actually boots the staging image family, supported noninteractive Brev authentication, and CI credentials.

[cjagwani]

Phase 1: current coverage audit — 2026-07-15

Parent scope: #6409

Audited against:

• NVIDIA/NemoClaw@88f2dd8b5d53cb6b499b6dd43dd66d7c3a704924
• brevdev/nemoclaw-image@1538dcbc15cb666a45f31f49aa70965aa5e6d474

Finding: the repositories have substantial component/canonical E2E coverage, a real source-based Brev nightly, and a real GCP image publisher, but no active lane joins them. Current release-boundary coverage does not execute exact candidate SHA → immutable staging image → real Brev Launchable → public onboarding → E2E verdict → pre-tag gate.

What the current lanes actually prove

Lane	Actual execution	Boundary it does not prove
launchable-smoke	Runs on ubuntu-latest, locally clones the checkout, executes brev-launchable-ci-cpu.sh, then uses non-interactive CLI onboarding and validates OpenClaw inference (workflow, test).	It does not boot a GCP image, create a Brev Launchable, use browser onboarding, or record image/Launchable provenance.
Typed brev-launchable-cloud-openclaw target	The target is declared with smoke and inference suites (definition).	Launchable platform/install support is explicitly unwired, so the target is unsupported (runtime support).
Brev nightly / branch validation	Creates a generic Brev VM with brev create --startup-script, rsyncs the current checkout, builds it, onboards, tests, and deletes it (workflow contract, runner).	It tests a source overlay, not the published staging image or a versioned Launchable. The workflow passes USE_LAUNCHABLE, but the runner does not consume it.
nemoclaw-image validation	Runs workflow lint, shell/static contract checks, dependency installation, and the Next.js build (workflow).	The repository's Vitest files are not invoked by the active validation workflow; there is no image boot or Launchable smoke.
nemoclaw-image build	Builds/captures an immutable GCP image, verifies GCP READY, then assigns the target family (workflow).	It does not cold-boot or behaviorally qualify the captured image before family publication; this is documented explicitly (README).

#6409 behavior coverage matrix

Adjacent coverage means the behavior is tested somewhere in NemoClaw or checked at source level. It is not evidence that the candidate Launchable passed.

Requirement	Current adjacent coverage	Exact staging-image / real-Launchable verdict
OpenClaw with default Brev hosted provider	Partial. The local smoke proves CLI onboarding, routing, and an agent turn, but forces NEMOCLAW_PROVIDER=custom.	Not covered. No default Brev provider through the public Launchable journey.
Hermes with a supported provider	Strong generic coverage. Live install, provider/policy state, health, and runtime checks exist (test).	Not covered.
Deep Code terminal flow	Strong generic coverage. Onboarding, inference.local, and a real dcode response are exercised (test).	Not covered. Historical manual Launchable evidence is not a current exact-image pass.
Compatible prebuilt artifact reuse	Source/build-time only. Provisioning prefers a baked local Docker image when present (implementation).	Not covered. No run proves which path was selected.
Deliberate artifact mismatch and fallback/failure	Source-only. A Dockerfile fallback exists (implementation).	Not covered. No injected mismatch, fallback/fail-closed assertion, or duration.
Browser/UI reconnect during onboarding	Unit-level only. SSE keepalive/abort behavior is tested.	Not covered. No browser reload/reconnect test against a Launchable.
Service or VM restart during/recovering from onboarding	Partial generic coverage. Container/supervisor recovery exists (test).	Not covered. No actual Launchable VM restart or public-flow interruption.
Re-onboard without data loss	Strong generic coverage. Same-name re-onboarding, sibling survival, and resume behavior exist (test). The image quickstart currently deletes the same-name sandbox before creation.	Not covered.
Policy tier/preset verification	Strong generic coverage. Live allow/deny, preset, hot-reload, and zero-preset behavior exist (test).	Not covered on the candidate image.
Provider secret-boundary verification	Strong generic/static coverage. Sandbox env/process/filesystem boundaries and image-source redaction/unset checks exist (test).	Not covered by a current passing candidate-image E2E.
Management-plane authorization and destructive operations	Partial. Approval/scoping and authorized sandbox destruction are tested generically.	Not covered. No Launchable authorization and negative destructive-operation matrix.
Update/rebuild/stale-image detection	Strong generic sandbox coverage. Stale agent versions and rebuild/state preservation are tested (test).	Not covered for GCP/Launchable image provenance.
Multi-sandbox coexistence or accepted limitation	Strong generic coverage. Concurrent gateways/dashboards and survivor behavior exist (test).	Not covered on the candidate Launchable; no accepted image-UI limitation is recorded.
Gateway ownership/failure-injection extension	Partial generic coverage. Gateway reuse, managed restart recovery, honest startup failure, and bounded soak tests exist.	Major gap. No exact-one-listener/declared-owner assertion after image boot; no competing listener, stale PID/state, mismatched identity, fail-before-mutation proof, service restart-count artifact, or image-level soak.

Performance and evidence gaps

Generic onboarding spans and a cold-path budget exist, but the requested candidate-image measurements are not independently captured for VM ready, gateway ready, UI ready, onboarding-to-Ready, agent/dashboard ready, prebuilt hit/miss, or fallback duration. Current successful image builds also emit no durable manifest tying image ID, nemoclaw-image SHA, NemoClaw SHA, Launchable revision, workspace ID, timings, and verdict together.

Parent acceptance criteria verdict

• CI boots the exact staging image that may be promoted.
• It completes onboarding through the same public interface used by the Brev UI.
• It verifies registry, policy, provider, readiness, lifecycle, and secret-boundary invariants together on that image.
• It proves the prebuilt fast path was used.
• Results and diagnostics are attached to image provenance and originating LKG build.
• Production promotion is blocked on this gate.

At the release boundary, all six remain open. The current lkg workflow starts only after lkg is pushed and dispatches the production image workflow without waiting for image build completion or Launchable qualification.

Historical signal

• brevdev/nemoclaw-image#44 previously added an exact-image cold-boot gate. Its runtime run caught an ambient gateway credential and blocked family promotion, demonstrating that this boundary catches defects missed by source/static validation.
• brevdev/nemoclaw-image#59 intentionally removed that runtime gate and assigned behavioral validation to NemoClaw. It is therefore historical evidence, not current coverage.

Scope resulting from this audit

This issue owns the first sequencing slice: exact candidate SHA → staging image → real Launchable → smoke E2E → exact-SHA verdict early enough to inform tagging. It should establish the orchestration and evidence contract, then reuse existing NemoClaw assertions rather than duplicating the entire behavioral suite in nemoclaw-image.

The image-side handoff is tracked in brevdev/nemoclaw-image#76.

[cjagwani]

Phase 2: exact-image handoff contract — 2026-07-15

Audited against:

• 88f2dd8
• brevdev/nemoclaw-image@1538dcbc15cb666a45f31f49aa70965aa5e6d474

Decision: the handoff is a run-bound, versioned artifact describing one immutable GCP Compute Engine image. The numeric GCP image ID is the acceptance authority. An image family is recorded only as observed mutable metadata and cannot satisfy, select, or qualify the downstream Launchable by itself.

This contract applies to pre-tag staging qualification. It does not change the existing post-tag lkg production-image dispatch.

Implementation alignment update — 2026-07-16: the review-draft producer and consumer now use the dedicated qualification workflow, persist an intent before the sole dispatch POST, reconcile an ambiguous response without redispatching, and retain cancellation evidence. They also reject noncanonical GCP identities, validate persisted state against durable intent before API use, bind downloaded hashes to artifact metadata, and persist a valid state transition before accepted evidence. The exact-image handoff stops before any Brev Launchable mutation; the supported Brev interface and live product gates below remain open.

Normative v1 success manifest

The artifact contains this strict JSON Schema Draft 2020-12 document shape. Unknown or additional properties fail closed.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "urn:brevdev:nemoclaw-image:exact-image-manifest:v1",
  "title": "NemoClaw exact staging image manifest",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "schemaVersion",
    "kind",
    "correlationId",
    "requesterRepository",
    "requesterWorkflowRunId",
    "requesterWorkflowRunAttempt",
    "nemoclawSha",
    "imageRepository",
    "imageRepositorySha",
    "producerWorkflow",
    "workflowRunId",
    "workflowRunAttempt",
    "imageOriginWorkflowRunId",
    "imageOriginWorkflowRunAttempt",
    "imageKind",
    "project",
    "imageName",
    "imageId",
    "imageSelfLink",
    "status",
    "imageCreationTimestamp",
    "manifestCreatedAt",
    "channel",
    "variant",
    "observedFamily",
    "result"
  ],
  "properties": {
    "schemaVersion": { "const": 1 },
    "kind": { "const": "nemoclaw-exact-image-manifest" },
    "correlationId": {
      "type": "string",
      "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$"
    },
    "requesterRepository": { "const": "NVIDIA/NemoClaw" },
    "requesterWorkflowRunId": { "type": "string", "pattern": "^[1-9][0-9]*$" },
    "requesterWorkflowRunAttempt": { "type": "integer", "minimum": 1 },
    "nemoclawSha": { "type": "string", "pattern": "^[0-9a-f]{40}$" },
    "imageRepository": { "const": "brevdev/nemoclaw-image" },
    "imageRepositorySha": { "type": "string", "pattern": "^[0-9a-f]{40}$" },
    "producerWorkflow": { "const": ".github/workflows/build-qualification-image.yml" },
    "workflowRunId": { "type": "string", "pattern": "^[1-9][0-9]*$" },
    "workflowRunAttempt": { "type": "integer", "minimum": 1 },
    "imageOriginWorkflowRunId": { "type": "string", "pattern": "^[1-9][0-9]*$" },
    "imageOriginWorkflowRunAttempt": { "type": "integer", "minimum": 1 },
    "imageKind": { "const": "compute#image" },
    "project": { "type": "string", "pattern": "^[a-z][a-z0-9-]{4,28}[a-z0-9]$" },
    "imageName": {
      "type": "string",
      "pattern": "^[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?$"
    },
    "imageId": { "type": "string", "pattern": "^[1-9][0-9]*$" },
    "imageSelfLink": {
      "type": "string",
      "format": "uri",
      "pattern": "^https://www[.]googleapis[.]com/compute/v1/projects/[a-z][a-z0-9-]{4,28}[a-z0-9]/global/images/[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?$"
    },
    "status": { "const": "READY" },
    "imageCreationTimestamp": { "type": "string", "format": "date-time" },
    "manifestCreatedAt": { "type": "string", "format": "date-time" },
    "channel": { "const": "staging" },
    "variant": { "enum": ["cpu", "gpu"] },
    "observedFamily": { "type": "string", "minLength": 1 },
    "result": { "enum": ["built", "reused"] }
  },
  "allOf": [
    {
      "if": {
        "properties": { "variant": { "const": "cpu" } },
        "required": ["variant"]
      },
      "then": {
        "properties": {
          "observedFamily": { "const": "nemoclaw-brev-staging-cpu" }
        }
      }
    },
    {
      "if": {
        "properties": { "variant": { "const": "gpu" } },
        "required": ["variant"]
      },
      "then": {
        "properties": {
          "observedFamily": { "const": "nemoclaw-brev-staging-gpu" }
        }
      }
    }
  ]
}

Run IDs and the GCP image ID are decimal strings so no JavaScript consumer can lose integer precision. The initial Phase 0 consumer accepts only cpu; gpu is rejected before dispatch with UNSUPPORTED_VARIANT even though the producer schema can describe the repository's existing gpu build variant.

The proposed builtAt field is split deliberately:

• imageCreationTimestamp is the authoritative GCP creationTimestamp of the immutable image.
• manifestCreatedAt is when the current handoff run created this evidence.
• For result=built, the image-origin run and attempt equal the current producer run and attempt.
• For result=reused, the origin run may differ and must identify a successful trusted build of the same full source tuple. A reused image without full origin evidence is rebuilt.

Semantic invariants

JSON Schema is necessary but not sufficient:

1. nemoclawSha and imageRepositorySha are the full lowercase 40-character SHAs actually used to build the image. Human-readable image names may remain abbreviated; GCP labels and reuse filters may not.
2. The selected image has full-SHA, channel, variant, origin-run, and origin-attempt labels matching the manifest. Legacy images without complete v1 provenance are not reusable.
3. Both built and reused paths converge on one final authoritative gcloud compute images describe, and the manifest is generated only from that response.
4. project is a canonical 6–30 character GCP project ID. project and imageName reconstruct imageSelfLink exactly; the link must parse as the canonical HTTPS Google Compute image URL with no userinfo, port, query, fragment, control character, encoding trick, or path normalization. It names an image, never an images/family path.
5. status is READY, imageId is nonzero, and imageCreationTimestamp is not later than manifestCreatedAt beyond bounded clock skew.
6. A reused staging image is at most 24 hours old at dispatch. This prevents exact source SHAs from hiding stale mutable base-package or dependency state in the daily cycle.
7. imageRepositorySha equals the verified current producer run head. A reused image's origin run exists, succeeded under trusted workflow code, and has the same head SHA.
8. The initial consumer requires channel=staging and variant=cpu.
9. observedFamily is evidence of publication only. No consumer uses it as immutable identity or silently falls back to it.
10. The image remains under a qualification hold through manifest validation and first verified workspace boot. Reverification immediately before provisioning must still return the same numeric ID; same-name delete/recreate is a mismatch.

Correlated dispatch and transport

NemoClaw should reuse the trusted-controller pattern in the PR E2E controller, not the fire-and-forget lkg dispatch:

1. The trusted controller freezes a full origin/main candidate SHA, records its requester run/attempt, snapshots the expected nemoclaw-image main SHA, and generates one lowercase UUIDv4 correlation ID.
2. Protected-environment approval occurs immediately before the first billable action.
3. It dispatches trusted nemoclaw-image main workflow code with the full candidate SHA, requester identity, correlation ID, staging channel, and cpu variant. The candidate SHA is data, not executable workflow selection.
4. Dispatch requests returned run details. The controller validates the exact repository, workflow path, workflow_dispatch event, producer head SHA, run ID, attempt, URL, correlation-bearing run name, and conclusion.
   • Before the sole dispatch POST, the controller atomically records a private intent containing the requester tuple, candidate SHA, correlation ID, producer workflow ID/path, and the producer main SHA observed before dispatch.
   • If the POST response is lost, malformed, or unusable, the controller never posts again. It searches only the exact workflow ID/path, main branch, workflow_dispatch event, attempt 1, correlation-bearing display title, canonical repository/URLs, and a bounded server-side creation window. Zero matches fail closed; one match is retained only for verified cancellation and is never accepted; multiple matches retain all exact run IDs and attempt cancellation before failing closed. Corrupt controller state is preserved as evidence and cleanup falls back to the durable intent.
5. It downloads only the deterministic artifact from that exact run and attempt; it never searches globally for the newest artifact.
6. Artifact name is nemoclaw-image-handoff-v1--.
7. A successful run contains exactly one regular UTF-8 file: nemoclaw-image-manifest.v1.json.
8. A failed run contains, when finalization can execute, exactly one regular UTF-8 file: nemoclaw-image-failure.v1.json.
9. Success plus a failure file, failure plus a success file, duplicate artifacts, extra files, symlinks, traversal paths, or a file over 64 KiB are contract failures.
10. Cancelled, timed-out, or runner-lost executions may have no artifact; the controller classifies them from trusted GitHub metadata.
11. The producer exposes artifact name, ID, URL, and upload digest as same-repository workflow outputs. The controller fetches exact artifact metadata, verifies the digest or attestation when available, hashes the accepted JSON bytes, and carries that hash into Launchable evidence.
12. Step summaries are diagnostic only. The cross-repository token remains in trusted controller steps and never reaches candidate or workspace code.

Retention:

• Producer artifact: at least 30 days.
• NemoClaw manifest, digest, run identity, validation, and verdict: at least 90 days or the release audit lifetime.
• Exact staging image: through terminal verdict plus at least 72 hours.
• Cleanup must be qualification-aware; a newest-N policy alone is not a sufficient hold.

Failure and retry contract

The producer failure artifact is intentionally a broad, best-effort receipt rather than a detailed diagnosis. Its strict v1 schema binds schemaVersion=1, kind=nemoclaw-image-failure, correlation and requester identity, the requested NemoClaw SHA, full image-repository SHA, dedicated producer workflow, producer run/attempt, startedAt, completedAt, phase, code, and retryable=false. It contains no arbitrary exception text, environment dump, or command output, and upload failure cannot turn the producer run green.

The allowed producer phase/code pairs are request/REQUEST_INVALID, validation/PRODUCER_VALIDATION_FAILED, source-verification/SOURCE_RESOLUTION_FAILED, image-build/BUILD_FAILED, cancelled/CANCELLED, and unknown/UNKNOWN. The bound run logs remain the source for the exact failed step. The current consumer fails on a non-success producer conclusion before downloading this diagnostic receipt; receipt consumption is follow-up work, not a prerequisite for safely rejecting the run.

The controller and later Launchable lane own the more specific orchestration classifications below:

Code	Meaning	Automatic action
REQUEST_INVALID / UNSUPPORTED_VARIANT	Invalid request tuple	None
DISPATCH_FORBIDDEN	Authentication or authorization failed	None
DISPATCH_AMBIGUOUS	Dispatch response lost or incomplete	Discover by correlation only; never blindly POST again
RUN_QUEUE_TIMEOUT	Run did not start within 10 minutes	Fail closed
SOURCE_RESOLUTION_FAILED	Exact SHA did not resolve to itself	None
PRODUCER_CONFIG_ERROR	Required GCP auth, project, or trusted config is missing	None
BUILD_FAILED	Provisioning, validation, or capture failed	None
IMAGE_NOT_READY	Image is absent, failed, or not READY	None
ARTIFACT_PENDING / ARTIFACT_DOWNLOAD_TRANSIENT	Propagation or 408/429/5xx read failure	Retry the same run within deadline
ARTIFACT_MISSING_OR_INVALID	Missing, duplicate, unsafe, or schema-invalid handoff	None
PROVENANCE_MISMATCH	Run, source, label, correlation, or origin disagrees	None; security-significant
IMAGE_IDENTITY_MISMATCH	GCP name, self-link, ID, status, or creation data disagrees	None; security-significant
BREV_IMAGE_RESOLUTION_MISMATCH	Brev resolved or booted a different image	None
QUALIFICATION_TIMEOUT	No accepted handoff by hard deadline	Fail closed
CANCELLED / SUPERSEDED / UNKNOWN	Terminal non-green state	None

Only read-only polling, run lookup, artifact download, and metadata reads receive bounded automatic retries. A full image build is never retried automatically.

Each protected approval authorizes one new dispatch and correlation ID. Phase 0 accepts only workflowRunAttempt=1. A GitHub UI rerun is UNAUTHORIZED_ATTEMPT; a new build attempt requires fresh approval, a new UUID, and a new dispatch. Late output cannot replace a timeout, superseded state, or other terminal verdict.

Timing contract

The latest 50 successful manually dispatched staging builds had a median of 18.9 minutes, p75 of 25.3 minutes, p90 of 31.1 minutes, and a maximum of 39.4 minutes. Representative recent builds were approximately 17–26 minutes.

Use one unambiguous clock:

• 10 minutes from dispatch: queue/start sub-deadline.
• 25 minutes from dispatch: operational warning.
• 45 minutes from dispatch: hard handoff deadline, including queue and artifact propagation.
• The 10-minute queue consumes the 45-minute total; using the full queue leaves at most 35 minutes.
• Poll at 15 seconds, increasing to 60 seconds with jitter, and honor Retry-After.
• A successful run gets at most two minutes of artifact-propagation retries, still inside the 45-minute total.

The tagger does not pay this latency. Qualification starts earlier in the daily cycle after provisional candidate selection and completes before cutoff. The tagger consumes an existing exact-SHA verdict or records the existing itemized maintainer exception.

Brev-supported interface status

Partially confirmed, not yet accepted. The family-at-instance-create proposal establishes a plausible mechanism: the image workflow advances staging, and Brev resolves the family to an exact image before instance creation. The mutable family is still insufficient evidence.

Preferred: Brev accepts imageSelfLink and pins that exact resource directly.

A family-only input is acceptable only if one supported operation atomically:

1. resolves the family once before any Launchable mutation or workspace creation;
2. pins the resulting immutable self-link for all later provisioning;
3. returns structured Launchable revision and workspace identities; and
4. returns imageName, numeric imageId, and imageSelfLink so the controller compares all three with the manifest before E2E.

Later re-resolution, missing numeric ID, mismatch after mutation, or family-only success fails BREV_IMAGE_RESOLUTION_MISMATCH.

The Brev owner still needs to accept:

• the merged/released CLI or API version and supported reference syntax;
• resolve-and-pin timing and failure semantics;
• structured create/revise output and immutable Launchable revision identity;
• immutable name, ID, and self-link provenance readback;
• non-interactive CI authentication; and
• deletion by workspace identity plus TTL/reconciliation.

Until then, the Brev interface checkbox remains open. UI analytics are a separate product question and are outside this handoff.

Release semantics

• The manifest is qualified input, not a green release verdict.
• The later Launchable job belongs in the candidate SHA's canonical .github/workflows/e2e.yaml evidence list; do not create a separate gating list.
• A verdict is eligible only when nemoclawSha exactly equals the release plan's frozen origin/main SHA. A changed SHA requires newly approved qualification.
• The tagger consumes completed evidence and never initiates image construction.
• observedFamily movement is not production promotion.
• Existing lkg-to-production automation remains separate.
• Because production builds separately, staging success does not claim bit-for-bit production identity.

Acceptance tests

Producer:

• Built and reused fixtures emit the identical success schema.
• Full source SHAs and origin run/attempt are written to and selected from labels; short-SHA lookup is rejected.
• One final authoritative describe populates every image field on both paths.
• Missing auth, unresolved SHA, missing metadata, non-READY state, label mismatch, or manifest upload failure cannot yield success.
• Legacy and over-24-hour-old images rebuild instead of reuse.
• Success and failure artifacts obey exclusive-file rules and contain no credentials.
• No-cloud tests cover missing/additional fields, invalid UUID, short/uppercase SHA, numeric JSON image ID, family self-link, non-READY status, built/reused origin rules, and producer failure phase/code pairs.

Consumer:

• Reject wrong repository, workflow, ref/head SHA, event, conclusion, run, attempt, requester, correlation, channel, or variant.
• Reject unsafe/duplicate/oversized artifacts, unsupported schema, digest mismatch, and semantic provenance mismatch.
• Fake-clock tests prove 10/25/45-minute boundaries and bounded same-run retries.
• Ambiguous dispatch discovery never starts a second build.
• UI reruns and late artifacts cannot supersede terminal state.
• Same-name delete/recreate with a different image ID fails before provisioning.
• Family movement or re-resolution to a different ID fails before Launchable mutation or workspace creation.
• The manifest and exact ID flow into later Launchable evidence and the exact-SHA release verdict.

This completes the Phase 2 contract definition. Review-draft implementation is now available in brevdev/nemoclaw-image#80 and NVIDIA/NemoClaw#7043. Together they cover exact-SHA dispatch, immutable manifest production/validation, bounded polling, ambiguous-dispatch reconciliation, and cleanup of an incomplete producer run. They intentionally stop before Brev Launchable mutation. Live dispatch remains gated on maintainer acceptance, protected environments, ref-bound WIF, a dedicated cross-repository token, and Brev-owner acceptance of the supported resolve-and-pin/readback interface.

[cjagwani]

Part 3: Brev-supported interface audit — 2026-07-16

Status: transport capabilities confirmed; unattended qualification contract not yet accepted. No live workspace was created during this audit.

Confirmed from current code

• Current dev-plane main passes a supplied GCP source-image reference through to Compute Engine. A family path therefore receives GCP native newest-non-deprecated resolution at VM creation, while an exact immutable image self-link can also be supplied unchanged (create path, source-image assignment).
• brev create --launchable <id> deploys an existing Launchable (CLI).
• Hidden API-key/org flags exist for noninteractive login, and brev delete accepts a workspace name or ID and sends an asynchronous deletion request (login, delete). These are implementation facts, not yet an accepted CI support policy.

What BREV-10816/Allow-latest-fetching changes

The unmerged branch is backend hardening/observability, not UI treatment. Its single commit detects family paths, calls GCP Images.GetFromFamily, replaces the family reference with the resolved immutable self-link before VM creation, fails early on resolution errors, and logs the resolution (commit, resolver). It is currently one commit ahead and 19 commits behind main, with no associated PR. It is not required for GCP native family boot, but it could make pre-create resolution explicit.

Brev-owner acceptance still required

1. Confirm that the staging Launchable may contain the family or exact self-link and that the server-side brev create --launchable path preserves and uses that image reference. The CLI request application does not visibly forward an image override (request application).
2. Provide machine-readable create/revise output containing workspace ID, Launchable ID, immutable Launchable revision/version, and correlation/idempotency identity. The current model has a Launchable ID but no revision/digest, and create has no JSON result (model, output).
3. Provide readback of the exact source image used to boot. Current GCP readback maps ImageID to the created disk self-link, not the source image (mapping).
4. Explicitly support the hidden API-key flow for CI, including role/org scope, expiry, rotation, revocation, and GitHub-secret handling. Its introducing work describes it as closed alpha; a human token is not suitable for unattended refresh (issue, PR).
5. Define structured provisioning/readiness/failure terminal states and polling semantics.
6. Define cleanup by workspace ID, idempotent already-absent behavior, terminal deletion verification, maximum lifetime, and janitor/TTL fallback. retire_timeout exists in the model but retirement is currently disabled (model, disabled behavior).

A first manually approved shadow proof can use the staging family only after those semantics and the #6943 product/environment gates are accepted. It must record the family input, expected immutable image manifest, resolved/booted source image, Launchable revision, workspace ID, readiness result, and verified cleanup. Family-only success without immutable readback remains non-green.

[cjagwani]

Phase 4: activation checklist and first shadow-run runbook — 2026-07-16

Status: no live shadow run is authorized yet. This is the operator handoff for the first run after the Phase 0 product decision and external interfaces are accepted. It is not canonical product documentation.

Current implementation:

• Producer image/receipt draft: brevdev/nemoclaw-image#80.
• Trusted consumer/controller draft: NVIDIA/NemoClaw#7043.
• Brev-owner interface request: brevdev/brev-cli#391.

Both draft workflows intentionally stop after an exact image manifest is accepted. They do not yet create a Launchable or workspace.

Go/no-go gates

Do not dispatch the image qualification lane until all of the following are true:

• Maintainers accept every Phase 0 decision in this issue and remove needs: design.
• Producer PR fix: detect sandbox context in status command #80 is merged first and its workflow exists on brevdev/nemoclaw-image@main.
• The producer staging environment is restricted to trusted main workflow code and has required reviewers.
• Qualification uses ref/workflow-bound GCP Workload Identity Federation through GCP_WORKLOAD_IDENTITY_PROVIDER and GCP_SERVICE_ACCOUNT; legacy GCP_CREDENTIALS_JSON and GCP_SA_KEY are removed and their key is revoked.
• The staging environment has the intended GCP_PROJECT, network, subnet, and network-tag variables, and the image-builder service account has only the documented compute/firewall/service-usage permissions.
• In NVIDIA/NemoClaw, approve-brev-launchable-qualification is created before the workflow is merged, requires maintain/admin reviewers, prevents self-review, preferably disables admin bypass, and permits trusted main only.
• NEMOCLAW_IMAGE_QUALIFICATION_TOKEN exists only in that protected environment and is scoped only to brevdev/nemoclaw-image with Contents: read and Actions: read/write.
• Core PR ci(e2e): add exact-SHA Brev Launchable E2E lane #7043 is merged after those controls exist.
• A separate read-only GCP identity is approved for immediate pre-provision image revalidation; the producer's build credential is never exported to candidate or workspace code.
• A Brev owner accepts exact-image preservation or atomic family resolve-and-pin, versioned structured receipts, source-image readback, noninteractive CI authentication, terminal readiness states, verified deletion, and TTL/janitor behavior.
• A dedicated staging Launchable, Brev organization, CI credential, and test inference credential are approved. No production Launchable or production family is mutated.
• Maintainers approve the first provider/model pair. The UI currently defaults to OpenRouter, so the test must select explicitly. Proposed first slice: OpenClaw + NVIDIA Cloud + nvidia/nemotron-3-super-120b-a12b.

PR #7043's currently pending ordinary E2E gate is separate from this activation. A NemoClaw maintainer/admin can run E2E / PR Gate Controller with operation=run-control-plane for the exact PR head/base. That runs local core regressions only; it does not build a GCP image or create a Brev resource.

First manually approved shadow attempt

1. Freeze one candidate early

Start when the provisional release candidate is selected and the soft merge freeze begins, not when the tagger is ready to tag.

Record:

• the full current origin/main SHA and matching release-plan SHA;
• operator, approver, reason, and expected daily tag window;
• approved provider/model and credential owner;
• the maximum of one image dispatch and one workspace;
• the previous dedicated staging Launchable revision for rollback;
• whether an active attempt or completed verdict already exists for the SHA.

The issue's 60-minute lead is only an advisory minimum. For the first shadow samples, start earlier—preferably at least 90 minutes before the tag decision—because the image handoff alone has a 45-minute hard deadline and Launchable/onboarding latency is not yet measured.

2. Dispatch the non-billable preflight

After both PRs and controls are merged:

CANDIDATE_SHA="$(gh api repos/NVIDIA/NemoClaw/git/ref/heads/main --jq .object.sha)"

gh workflow run "Draft / Brev Launchable Qualification" \
  --repo NVIDIA/NemoClaw \
  --ref main \
  -f candidate_sha="$CANDIDATE_SHA" \
  -f reason="First manually approved #6943 shadow qualification"

Record the controller run ID and URL. Do not use GitHub's rerun button; attempt 2 is unauthorized. Preflight must prove workflow dispatch, trusted main, exact candidate/workflow/current-main equality, and a triggering actor with maintain/admin permission.

3. Approve immediately before billing

The protected-environment reviewer rechecks that:

• Phase 0 and every environment gate above are accepted;
• main and the release-plan candidate still equal the requested SHA;
• there is no duplicate attempt;
• cost and cleanup owners are available;
• the Brev contract and credentials are ready for the later workspace step.

One approval authorizes one correlation ID, one producer dispatch, and at most one workspace. If main moves while approval is pending, reject or cancel the run.

4. Produce and accept the immutable image handoff

The implemented controller then:

1. snapshots producer main;
2. persists durable dispatch intent;
3. dispatches Build Qualification Staging Image exactly once;
4. enforces a ten-minute queue limit, 25-minute warning, 45-minute total deadline, and attempt 1 only;
5. never blindly redispatches after an ambiguous response;
6. downloads only the deterministic artifact from the bound run;
7. validates archive safety, digest, schema, exact provenance, timestamps, and controller state; and
8. retains the controller evidence for 90 days.

The accepted manifest must bind the exact candidate and producer SHAs, requester and producer run/attempt, correlation ID, canonical GCP project, immutable image name/numeric ID/self-link, READY, staging, cpu, observed family, build/reuse origin, and creation timestamps.

This proves image packaging only. It is not a Launchable or release verdict.

5. Revalidate before Brev mutation

With approved read-only GCP access:

• describe the exact image and compare name, numeric ID, self-link, status, full source labels, and active qualification hold with the manifest;
• resolve nemoclaw-brev-staging-cpu and require that its name and numeric ID still equal the manifest; and
• retain both structured responses.

A changed family head, same-name/different-ID resource, expired or missing hold, label mismatch, or non-READY image aborts before Launchable mutation.

6. Pin a dedicated staging Launchable

Use the manifest's immutable image self-link when supported. A family input is allowed only through a Brev operation that resolves once before mutation, pins that exact result, and returns:

• Launchable ID and immutable revision/version;
• correlation/idempotency identity;
• the supplied reference; and
• resolved image name, numeric ID, and immutable self-link.

All three image fields must match the manifest before workspace creation. Console text, family-only success, or a created-disk self-link is non-green. Do not substitute a private API, source rsync, startup script, hot-deployed UI, or the production Launchable.

This step is currently blocked on Brev-owner acceptance.

7. Create and verify exactly one workspace

Through the accepted Brev interface:

1. create one workspace with the same correlation/idempotency identity;
2. save the structured receipt and immutable workspace ID;
3. reconcile an ambiguous response by correlation rather than posting again;
4. poll the documented structured state machine to terminal readiness;
5. read back the booted source-image name, numeric ID, and self-link; and
6. compare all three with the manifest before E2E.

This step is also currently blocked. brev ls --json supplies workspace ID and several status fields, but the current CLI does not supply immutable Launchable revision or boot-source provenance.

8. Drive the public browser boundary

Open the workspace's public /onboard route in a clean browser profile:

• explicitly select OpenClaw, the approved provider, and the exact model;
• enter the dedicated credential only in the secret field and never capture it;
• require every visible stage to complete without retry, error, skip, or indefinite pending;
• require YOUR AGENT IS LIVE, Agent created successfully, and the expected summary;
• open CHAT WITH AGENT and require the dashboard to load.

Treat the browser journey as a black box. Do not bind the release gate directly to private /api/onboard or SSE schemas.

9. Reuse the OpenClaw runtime assertions

For the browser-created my-assistant sandbox, require:

• registry and status success;
• the approved inference route;
• exactly one unique listener on port 8080 owned by the baked openshell-gateway.service;
• a sandbox call through inference.local returning PONG;
• an OpenClaw agent turn returning 42;
• absence of the raw provider credential from the sandbox environment, process list, and a canary-validated sampled filesystem scan; and
• no raw credential, dashboard token, browser cookie, or authorization header in artifacts.

In release mode, skipped or pending assertions are failures.

Evidence and green verdict

Retain one redacted bundle keyed by candidate SHA and correlation ID:

• producer dispatch, run, artifact, manifest, digest, and validation evidence;
• exact-image and family readbacks;
• Launchable before/after revision and resolve-and-pin receipt;
• workspace create, readiness, boot-source, deletion, and terminal-absence records;
• browser completion proof;
• runtime, gateway-owner, inference, agent, and credential-isolation results;
• cleanup and rollback evidence; and
• a final immutable verdict.

Capture timestamps for candidate freeze, approval, producer queue/start/completion, manifest acceptance, Launchable mutation, workspace create/ready, UI ready, onboarding start/ready, inference/agent pass, cleanup start/absence, and total duration. Also record cost, built/reused image, failure class, and cleanup outcome for the shadow report.

A shadow verdict is green only if its SHA exactly equals the frozen release-plan SHA and every stage, including cleanup, passes. Shadow runs remain non-blocking until maintainers accept the latency/cost/reliability report and separately enable enforcement.

Cleanup, rollback, and retry

Always clean up after success, failure, cancellation, or supersession:

1. destroy my-assistant and verify registry absence;
2. delete the workspace by immutable ID and poll to terminal absence;
3. restore or deactivate only the dedicated staging Launchable revision and verify its pointer;
4. verify the producer's exact temporary firewall and current-run builder VM are absent; and
5. retain the exact image under its producer-owned 72-hour hold—do not delete it or manually move the family during controller cleanup.

Before resource creation, abort on missing approval, moved candidate, duplicate attempt, credentials, or provenance mismatch. After resources exist, mark the attempt non-green and perform bounded cleanup.

Do not automatically retry a full image build, Launchable mutation, or workspace create. A new attempt requires a new dispatch, correlation ID, and protected approval. An itemized maintainer exception may let release policy proceed, but it never converts missing, failed, superseded, or cleanup-failed evidence into a green verdict.

After Phase 0 acceptance, move this operator procedure to an internal path such as .github/runbooks/brev-launchable-qualification-shadow.md and link it from test/e2e/README.md. Do not publish it as supported user documentation until the product surface and Brev interface are accepted.