simulator: run container societies on local Kubernetes and GKE

[chughtapan]

Outcome

Target main and the original packages/simulator. Make container-society
execution part of the core simulator, not an example.

There is one Kubernetes execution path with two Effect Layers:

• a repository-owned local Kubernetes cluster;
• regional GKE Standard.

Both run the same code-first RunSpec through Run.execute. The selected
stack is Kubernetes, Kueue, Agent Sandbox, and Temporal. Docker may build images
or support the local cluster; it is not a second simulator backend.

Requirements boundary

The checked-in source-event conversations are the requirements boundary:

• docs/decision-evidence/20260727-code-first-simulator-trajectory.md;
• docs/decision-evidence/20260729-principal-runtime-gateway-trajectory.md;
• docs/decision-evidence/20260729-effect-native-evaluation-results-trajectory.md;
• docs/decision-evidence/20260801-main-kubernetes-society-execution-trajectory.md,
  which also retains the earlier Kubernetes/Kueue/Temporal/Agent Sandbox
  selections from the distributed-execution conversation.

Anything not discussed there is a non-goal. This issue is an implementation
plan, not normative authority; the main-track ADR owns the accepted contract.

Public shape

The public model is deliberately small:

export default RunSpec.define({
  id: "acme.echo/v1",
  events: [echoEvents],
  agents: {
    alice: openClawRuntime({ /* portable runtime configuration */ }),
    bob: scriptedRuntime({ /* portable runtime configuration */ }),
  },
  infrastructure: Kubernetes.local(),
  execute: ({ agents, events, network, ledger }) =>
    Effect.gen(function* () {
      // Instruct agents, observe the society, and determine completion.
    }),
});

Run.execute(spec) is the execution entry point. The infrastructure field
contains the selected local or GKE Effect Layer, while Kubernetes objects stay
out of the roster and customer Effect.

This reuses the existing EventCatalog, RunLedger, keyed StartedAgent gateway
types, network services, program Exit, infrastructure outcome, and
customer-owned completion policy. It does not create a new data model,
serialization framework, gateway union, or result system.

After the evaluations and local/GKE proofs migrate, remove the transitional
simulator.define(...).run(...) host path and standalone Docker example. Do
not keep a compatibility backend.

One experiment lifecycle

Each execution creates one society for one customer Effect and tears it down:

1. Temporal starts one coarse workflow for the run.
2. Kueue admits capacity for the complete roster.
3. The controller creates one Agent Sandbox with one application container per
   roster entry.
4. The controller waits for the exact full roster to be ready together.
5. The in-cluster controller invokes the customer execute Effect once.
6. The existing simulator outcome and ledger retain experiment and
   infrastructure evidence.
7. Temporal drives cleanup of run-owned Kubernetes resources.

A pre-gate backing-Pod restart simply delays readiness and exposes no public
generation model. An unrecoverable or never-ready agent fails acquisition.
After dispatch, runtime termination remains typed ledger evidence and customer
Effect policy decides what to do. Controller or infrastructure loss fails the run and
starts cleanup. The controller does not replay customer code, and the simulator
makes no exactly-once claim for external side effects.

Kueue owns capacity, not readiness. Kubernetes and Agent Sandbox own placement
and container lifecycle, not customer policy. Temporal owns coarse
orchestration and cleanup, not agent logic or simulator evidence.

One roster entry is one agent in one Sandbox application container. Real and
code/scripted agents can share a society, but every social action uses the
production MoltZap router. Principal instructions use each runtime's native
gateway.

A stable controller loads the experiment module late. The stock digest-pinned
OpenClaw image is the compatibility path; an image with MoltZap preinstalled is
only an optimization. Exact bundle transport and caching are private profile
details.

Profiles

The local profile installs the components needed for development and CI,
including a development Temporal deployment.

The GKE reference is regional GKE Standard with Agent Sandbox. Terraform and
Helm own reproducible cluster and add-on setup. Production Temporal hosting and
high availability remain unselected; GKE qualification may use a test
deployment or configured endpoint.

The Effect Layer boundary leaves room for a future scheduler without adding a
new provider framework now. Nomad, Slurm, and managed batch are not implemented.

Implementation slices

• 0 — Admit the small main decision. Reconcile the source trajectory,
  ADR, package law, prior-decision lineage, and affected docs. Freeze the exact
  candidate, pass a fresh blind teammate review, and obtain maintainer
  acceptance before production code.
• 1 — Core contract and fake. Add RunSpec and Run.execute as a
  small facade over the existing simulator concepts. Add the narrow private
  platform seam and fake tests for full-roster gating, one invocation,
  post-dispatch termination policy, outcomes, and cleanup.
• 2 — Local Kubernetes vertical slice. Bring up local Kubernetes with
  Kueue, Agent Sandbox, and Temporal. Run a two-agent stock-OpenClaw/core-router
  smoke through packages/simulator, retain ledger evidence, and verify zero
  run-owned residue.
• 3 — Four agents and evaluations. Prove the same path with four agents.
  Move all 32 OpenClaw/NanoClaw evaluation cells to Run.execute and record
  their real outcomes, including honest operational or behavioral failures.
  Remove the host executor and standalone example after the replacement proof.
• 4 — GKE qualification. Provision regional GKE Standard with
  Terraform/Helm, run the same small smoke and at least one OpenClaw evaluation
  through the unchanged authoring contract, retain evidence, and clean up.

Acceptance criteria

• packages/simulator owns the only real society-execution path.
• The same agents, events, and execute program run through local
  Kubernetes or GKE by changing only RunSpec.infrastructure.
• One Kueue cohort is admitted before one Sandbox/application container per
  roster entry is created.
• The customer Effect cannot start before the exact full roster is ready.
• The controller invokes it once and never replays it; no exactly-once
  external-side-effect claim is made.
• Principal control uses native gateways and all agent social traffic uses
  the production router.
• Existing typed ledger/outcome and evaluation grading/report contracts
  remain readable and current.
• Local two-agent, local four-agent, all 32 evaluation cells, and GKE smoke
  evidence use Run.execute and leave zero run-owned residue.
• No production code lands under v2/* and no new package is introduced.

Explicit non-goals

• generation identifiers or streams, customer-visible restart/recovery APIs,
  or post-dispatch replacement, rebinding, rejoin, persistent-state recovery,
  and recovery of in-flight work;
• customer Effect replay, exactly-once external effects, or distributed
  side-effect transactions;
• start-or-attach identity, artifact authorities, global execution-id
  namespaces, synthetic UUIDs, or normative resource-name hashing;
• a new immutable-data/JCS grammar or universal input/result/failure schema;
• arbitrary Pod templates or public Kubernetes/Kueue/Sandbox/Temporal APIs;
• per-agent Temporal workflows, warm societies, multi-run fairness,
  preemption, autoscaling, router HA, or production Temporal HA;
• scale qualification above four agents in this issue;
• a Nomad, Slurm, managed-batch, or Autopilot backend;
• a general multi-tenant Secret, NetworkPolicy, or security platform; and
• any implementation or contract change under v2/*.

Existing work to reuse

Reuse the current simulator kernel, EventCatalog, RunLedger, scoped cleanup,
runtime-native gateways, router, OpenClaw adapter, and evaluation report
pipeline. The Docker society example remains temporary compatibility evidence
until the core Kubernetes path replaces it.

[chughtapan]

Implementation checkpoint 0: active

The implementation target is main. Core code, orchestration, CLI, platform assets, and runtime integration live under packages/simulator; v2/* remains untouched. The issue body is the approved implementation contract and acceptance plan for this run.

Current execution state:

• Worktree: /home/tapanc/moltzap-pr-917-main
• Branch: impl/917-main-local-society
• Candidate: 515aa590
• Baseline: clean, 12 commits behind and 1 commit ahead of origin/main
• Existing ahead commit: the three-container example from PR feat(simulator): run container societies on Kubernetes, validated at a hundred agents #932; it is compatibility input, not the final product surface

The active sequence is:

1. Reconcile the branch with current main and freeze Slice 0 traceability.
2. Land the main-track ADR/public contract for RunSpec, Run.execute, and the declarative Kubernetes profile.
3. Implement the core generation lifecycle and local execution seam in @moltzap/simulator.
4. Add the kind/Kueue/Agent Sandbox/Temporal path and prove two/ten-agent gates.
5. Move every eval cell onto the single core executor and retain report/resume compatibility.
6. Qualify the same path on GKE and complete the acceptance audit.

Every checkpoint records the exact candidate SHA, commands, pass/fail evidence, remaining gaps, and next action. No completed claim is made from unit tests alone; acceptance requires actual eval execution through the new path.

Confidence: HIGH. The worktree, issue, PR, current package law, and branch divergence are directly inspected.

Process issues: issue #936 is the parent implementation contract and has no safer workflow labels or sub-issue; execution proceeds directly against it because the maintainer explicitly assigned this issue and worktree.

[chughtapan]

Implementation checkpoint 1: baseline and contract correction

Candidate 2d3fc412 merges current origin/main into impl/917-main-local-society. The worktree is clean after the merge.

Baseline verification is green:

pnpm nx run-many -t build typecheck:tests lint test \
  -p @moltzap/simulator @moltzap/evals --outputStyle=stream

@moltzap/simulator: 33 test files, 215 tests passed
@moltzap/evals:      9 test files, 75 tests passed
Nx: build, test typecheck, lint, and test succeeded for both projects and dependencies

The issue contract now includes the input/reconciliation boundary required by the actual eval application:

• one source module exports the code-defined RunSpec plus its Effect Schema input;
• each execution supplies schema-decoded, deeply snapshotted input;
• executionId is a start-or-attach identity bound to infrastructure authority, definition id, source/input digests, resolved roster, and profile;
• an identical retry attaches to or returns the same Temporal Workflow, controller, RunLedger, and receipt;
• an identity reused with different content fails before resource creation;
• eval attempt IDs are the execution identities, so process recovery cannot duplicate a live cell;
• OpenClaw and NanoClaw image/bootstrap conformance is in scope because end-to-end acceptance requires all 32 cells on the new executor.

Local platform evidence:

• Docker 29.3.0 is rootful on cgroup v2; the host has 8 CPUs, 33.6 GB RAM, and 164 GB free.
• kind, kubectl, Helm, Cilium CLI, gVisor/runsc, OpenTofu/Terraform, and the GKE auth plugin are not installed yet.
• packages/simulator currently contains no Kubernetes, Kueue, Temporal, Helm, Terraform, or live-platform targets.
• the stock digest-pinned OpenClaw image and current MoltZap/NanoClaw images are cached.

GKE qualification is externally blocked at this checkpoint: the configured account cannot access project moltzap-499707, and the Kubernetes Engine API is disabled. Local Slice 0 and Slice 1 work continues; GKE execution requires a billing-enabled accessible project and IAM for APIs, networking, GKE, Artifact Registry, storage, and workload identities.

Next action: freeze the main-track ADR and traceability, then implement the schema-bound definition/invocation/idempotency contracts before platform code.

Confidence: HIGH. Git, Nx, Docker, local toolchain, and GCP preflight outputs are directly inspected.

Process issues: the original static-RunSpec plan could not reconstruct dynamic eval cells or reconcile resumed work; the issue body now carries the corrected contract before code exists.

[chughtapan]

Implementation checkpoint 2: first ADR candidate failed cold review

The first frozen Slice 0 candidate was commit 0f152696e588538ffcbfac0162bcd1cf17bbaab3 (tree 0374e97d). Its full workspace precommit gate, decision-gate tests, drift check, link check, formatting, and Mermaid validation passed.

The mandatory isolated review nevertheless returned FAIL, so this candidate will not land and implementation remains gated. The unedited review is retained at docs/decision-evidence/20260801-main-kubernetes-society-execution-cold-review.md. Questions 1–3 and 5 passed; questions 4 and 6 failed because the candidate lacked final human acceptance for the expanded contract, contradicted itself about infrastructure authority, and left the public API, CLI, and durable identifier encodings underspecified.

The working candidate now corrects those findings:

• infrastructure authority scopes (authority, definitionId, executionId); the same definition/execution id under another authority is an independent run, while source/input/roster/profile conflicts apply only inside one authority;
• decoded and encoded input/result/failure values use one closed immutable-data grammar and reject cycles, sparse arrays, accessors, symbols, mutable built-ins, custom prototypes, non-finite numbers, and lone surrogates;
• the public namespaces, RunSpec, bridge/gateway, Run.execute, Run.open, receipt/outcome, pre-ledger errors, and cold generation stream have closed semantic shapes;
• pre-binding availability is split into RunPreparationUnavailable (compare-create not attempted, no binding/resources) and RunStartUnavailable (ambiguous or known binding, retry attaches);
• workflow ids, stable AgentIds, Kubernetes names, and resolved-roster digests use exact domain-separated RFC 8785 JCS + SHA-256 derivations;
• moltzap-sim run is the sole CLI operation; repeating it is start-or-attach, with a single versioned JSON stdout record and fixed exit mappings;
• the dispatch contract is at-most-once invocation attempt and makes no exactly-once external-side-effect claim;
• the pre-cutover standalone example is explicitly marked as transitional evidence rather than a backend.

Current checks on the corrected working tree are green: pnpm format:check, pnpm docs:check, pnpm docs:check:mermaid, and git diff --check.

Next gate: record explicit maintainer acceptance of the corrected outcome in the source trajectory, freeze a new candidate, and send it to a different fresh blind reviewer. No production code is being added before that review passes.

Confidence: HIGH for the mechanical results and review findings; acceptance of the corrected semantic contract is intentionally still pending.

[chughtapan]

Checkpoint 3 — requirements-bounded scope reset

Maintainer feedback rejected the previous candidate as overcomplicated for
experiment infrastructure and set the checked-in conversations as the
requirements boundary.

What changed:

• rewrote the main ADR from 1,518 lines to 246 lines;
• reduced the package law from 160+ lines to 108 lines;
• retained only RunSpec / Run.execute, one Sandbox container per agent,
  the exact full-roster gate, Kubernetes local/GKE Layers, aggregate Kueue, and
  one coarse Temporal/in-cluster-controller lifecycle;
• restored the existing EventCatalog, RunLedger, runtime-native gateway,
  termination-policy, and evaluation-result decisions instead of replacing
  them;
• removed generation streams, restart/rejoin semantics, artifact authority,
  start-or-attach identity, UUID/name derivations, custom immutable/JCS data
  rules, per-agent orchestration, production-HA claims, and the scale ladder
  from this issue;
• incorporated the earlier checked-in Kubernetes/Kueue/Temporal/Agent Sandbox
  source events into the main trajectory, including the explicit “get to 10
  agents first” deferral.

No production implementation has started because the repository ADR gate still
applies. Next: validate this small candidate, obtain maintainer acceptance of
the one-invocation/no-replay wording, freeze it, and run a different fresh blind
reviewer before touching the core implementation.

Worktree: /home/tapanc/moltzap-pr-917-main
Branch: impl/917-main-local-society
Current HEAD: 0f152696 (the first failed candidate; the simplified rewrite
is still uncommitted).

Audit follow-up: the accepted example keeps infrastructure inside
RunSpec; the Layer changes without changing agents/events/execute.
The eval acceptance remains the complete 32-cell OpenClaw/NanoClaw matrix.
OpenClaw is first in the local vertical slice, but NanoClaw is not removed from
the issue. A backing-Pod restart may delay the pre-dispatch gate without
creating a public generation API.

[chughtapan]

Checkpoint 4 — maintainer accepted the simplified ADR

The maintainer explicitly replied accept this ADR to the final contract summarized in the checked-in trajectory.

Accepted shape:

• RunSpec { id, events, agents, infrastructure, execute } and Run.execute(spec)
• local Kubernetes or GKE through the infrastructure Layer
• Kueue admission → one Sandbox/container per agent → exact full-roster gate → one execute invocation → existing ledger/outcome → Temporal cleanup
• two-agent, ten-agent, and all 32 OpenClaw/NanoClaw evaluation runs
• checked-in requirement conversations are the scope boundary; unrecorded mechanisms are non-goals
• no public generations, artifact authority, start-or-attach system, custom serialization grammar, or recovery platform

This also resolves the pending failure semantic: controller loss fails the run and triggers cleanup; execute is not replayed; the simulator makes no exactly-once guarantee for external effects.

Next gate: freeze the exact candidate, run the full repository checks, then submit that candidate to a different fresh blind reviewer. Production implementation has not started.

[chughtapan]

Checkpoint 5 — simplified candidate failed the runtime-boundary review

Frozen candidate 1939ee8b92e95151473c323de8dd702e880dbde5 (tree b2a487141545e4cced7bc7ab0e0d08f344cebea3) passed the repository gates but received an overall FAIL from the isolated blind review.

The review passed questions 1, 2, and 4. It found one architectural blocker across questions 3, 5, and 6:

• the new ADR requires every code/Effect peer to run in its own Agent Sandbox application container;
• the retained principal-gateway ADR still binds those peers to host-local effectRuntime({ build }) closures and shared in-process Effect state;
• the controller therefore cannot obtain that same gateway value without an admitted cross-process runtime boundary.

The unedited review is retained at docs/decision-evidence/20260804-main-kubernetes-society-execution-cold-review.md.

The correction is deliberately narrow and is already supported by the retained requirements that deterministic peer behavior belongs inside agent containers and that every agent gets one container:

1. retain exact runtime-native gateway types and the ban on a universal gateway union or command protocol;
2. replace only the host-bound effectRuntime({ build }) realization on the Kubernetes path;
3. make each container runtime own a private runtime-specific controller bridge that yields its exact gateway and termination observation after readiness;
4. package evaluation peer policy as its own agent-container entrypoint with its exact observation bridge;
5. remove the undefined scriptedRuntime spelling from the ADR example;
6. keep transport details, recovery, generations, artifact authority, and start-or-attach machinery out of scope.

Next: reconcile the ADR lineage and transition docs, freeze a new candidate, run the complete gates, then use a different fresh blind reviewer. Production implementation remains gated.

[chughtapan]

Checkpoint 6 — corrected ADR candidate passed blind review

The corrected semantic candidate is commit 2749adbd99eaffd16f063a45de7be01c253f7ef1 (tree ce6655004c93d03e6276a07756a5086ce68aa662). It passed the workspace precommit, docs link, Mermaid, decision-gate, drift, and diff-integrity checks.

A different fresh reviewer, /root/candidate_blind_review_3, reviewed only the frozen repository root and the six mandated questions from 2026-08-04T01:03:05Z through 2026-08-04T01:10:58Z (473 seconds). There were no author interventions or repository modifications.

Result: PASS.

• questions 1–6 all passed;
• no blockers or accidental implementation gaps were found;
• the earlier host-local effectRuntime({ build }) conflict is resolved through explicit supersession by per-runtime container entrypoints and runtime-specific bridges;
• exact native gateway semantics remain retained, and no universal command/gateway protocol was introduced;
• main/v1 ownership under packages/simulator remains distinct from the untouched v2 testbed contract.

The unedited review is retained at docs/decision-evidence/20260804-main-kubernetes-society-execution-second-cold-review.md with SHA-256 3aeb7f05a5c218c80e9768a343fd837d0b1ae359a6d0535ea5c5cb9e9e6562da. The trace-only evidence commit is 4bce9aef09a274d3853153de25dca1bde1acd91f.

The reviewer correctly notes that its PASS is not self-certifying. The remaining ADR gate is explicit maintainer acceptance of this review result; production implementation stays gated until then.

[chughtapan]

Checkpoint 7 — maintainer accepts the blind-review result

The maintainer replies accept to the explicit request to accept the now-visible blind-review PASS.

The ADR admission gate is complete:

• semantic candidate: 2749adbd99eaffd16f063a45de7be01c253f7ef1;
• audit evidence commit: 4bce9aef09a274d3853153de25dca1bde1acd91f;
• isolated result: all six questions PASS, no blockers;
• implementation authority: main/v1, under packages/simulator; v2/* remains untouched.

Implementation is now active. The first slice is the core public seam and private fake: RunSpec { id, events, agents, infrastructure, execute }, Run.execute(spec), exact keyed runtime gateways, one full-roster readiness barrier, one customer-Effect invocation without automatic replay, and existing ledger/outcome semantics. Kubernetes/Kueue/Agent Sandbox/Temporal remain behind the private platform boundary and enter in the next local-cluster slice.

[chughtapan]

Slice 1 checkpoint: the core simulator execution seam is frozen locally in commit 77178b2 (feat(simulator): add RunSpec execution seam).

Completed on main-track packages/simulator:

• Added the five-field RunSpec definition and Run.execute entry point while retaining the legacy path for migration.
• Routed every run through one private, mechanism-neutral society platform seam.
• Enforced one exact cohort dispatch fence: incomplete or pre-dispatch-lost rosters never reach customer code.
• Kept post-dispatch runtime termination as ledger evidence governed by customer policy.
• Exposed only SimulatorInfrastructureFailure at the customer boundary; Kubernetes, Kueue, Agent Sandbox, Temporal, controller, and fake-platform objects remain private.
• Added type canaries for exact heterogeneous runtime gateways and Effect/Layer requirements.
• Added deterministic lifecycle coverage for incomplete acquisition, loss while a peer is still acquiring, blocked-cohort loss, post-dispatch platform loss, no replay, cleanup, and ledger ordering.
• No v2 files changed.

Fresh engineering review: PASS after closing the two-agent acquisition-race regression gap; no production lifecycle defect remains.

Green gates:

• pnpm nx run @moltzap/simulator:build --skipNxCache
• pnpm nx run @moltzap/simulator:typecheck:tests --skipNxCache
• pnpm nx run @moltzap/simulator:lint --skipNxCache
• pnpm nx run @moltzap/simulator:arch:check --skipNxCache (0 findings)
• pnpm nx run @moltzap/simulator:test --skipNxCache (34 files, 224 tests)
• pnpm nx run workspace:format:check --skipNxCache
• pnpm nx run workspace:arch:config:check --skipNxCache
• documentation import and constant checks
• full workspace precommit hook

Next slice: implement the same private platform port for local Kubernetes using Kueue, Agent Sandbox, and one coarse Temporal lifecycle workflow. No generation stream, artifact authority, synthetic identity system, customer replay, or second backend.

[chughtapan]

Slice 5 checkpoint — static cutover complete

The uncommitted candidate in /home/tapanc/moltzap-pr-917-main on impl/917-main-local-society, atop 77178b2adbdfdfb6053db87ac78e973b7ce626f0, has reached the requested static cutover boundary.

Completed:

• packages/simulator now owns the Kubernetes/Kueue/Agent Sandbox/Temporal execution path for both local Kubernetes and GKE profiles behind the same RunSpec / Run.execute authoring contract.
• Evaluations submit each cell through that core path and retain exact artifact/result readback.
• Transitional host execution APIs and implementations were removed, including defineSimulator, simulatorLayer, effectRuntime, host-local OpenClaw/NanoClaw process/install/cache/workspace paths, and the Docker router/server-image backend.
• The standalone examples/simulator workspace was removed.
• Docker remains only where it constructs the controller image or supports local Kubernetes; it is not a simulator execution backend.
• v2/* is untouched.

Static evidence is green:

• @moltzap/simulator: build, test typecheck, lint, architecture check, local-profile check, GKE-profile check, and 38 test files / 176 tests.
• @moltzap/evals: build, test typecheck, lint, architecture check, and 12 test files / 85 tests.
• simulator packed-package consumer guard.
• full workspace build, typecheck, lint/dead-code/architecture checks, formatting, generated-doc drift, Mermaid, links, docs gate tests, and diff-integrity checks.
• CI-equivalent affected sweep: 33 tasks across 9 projects and their dependencies, including unit and integration targets.
• stale-surface search finds only intentional negative guards that assert the retired APIs, Docker backend files, and standalone example remain absent.

This is deliberately not a live-qualification claim. Not run yet:

• controller image construction;
• local Kubernetes two-agent or ten-agent execution and residue checks;
• the live 32-cell OpenClaw/NanoClaw evaluation matrix;
• GKE provisioning or smoke/evaluation qualification.

No implementation commit, push, or publication was made. The pre-existing staged blind-review evidence file remains staged and was not modified by this slice.

[chughtapan]

Live qualification checkpoint — controller image packaging fixed

The first live local gate found and fixed a real packaging defect in the uncommitted candidate.

Observed failure:

• pnpm nx run @moltzap/simulator:local-controller-image reached the controller image install, then failed because node_modules/@moltzap/evals/dist/peer-application.js was absent.
• The compiled file existed in the worktree, but an independently packed eval tarball contained only src/peer-application.ts, not dist/peer-application.js.

Root cause:

• packages/evals/package.json had no files allowlist.
• the repository ignores dist;
• pnpm pack therefore omitted the compiled eval application from this previously private, non-packed package.

Correction and evidence:

• added the eval package's dist directory to its package contents;
• added a local-profile regression assertion for the controller image's compiled eval entrypoint;
• proved the assertion fails before the manifest correction and passes afterward;
• independently repacked the package and verified package/dist/peer-application.js and .d.ts;
• reran the original image target successfully.

Fresh candidate image:

{
  "image": "moltzap-simulator-controller:acf6d2a418e234b9",
  "pinnedImage": "moltzap-simulator-controller@sha256:f02153a43cdfee9b9011b002468439da887e64d914802142a6c4df4c41b11dda",
  "controllerEntrypoint": "/opt/moltzap/dist/platform/controller/main.js",
  "supportBootstrap": "/opt/moltzap/dist/platform/kubernetes/bootstrap.js",
  "applicationOverlay": "/opt/moltzap/application-overlay"
}

The existing moltzap-simulator kind cluster is healthy and has zero run-owned namespaces, but it contains an older controller image. The next action is to recreate that exact repository-owned profile with the new digest and retain the bind-mounted artifact directory, then run the two-agent and ten-agent live gates.

The 32-cell eval is not claimed yet. Its live audit has also identified missing reproducible NanoClaw image construction and target-container model credential propagation; those remain implementation work after the smoke path is proven.

[chughtapan]

Live qualification checkpoint — local two-agent core path passes

The repository-owned kind profile was recreated with the fresh controller/support image:

moltzap-simulator-controller@sha256:f02153a43cdfee9b9011b002468439da887e64d914802142a6c4df4c41b11dda

The core simulator then completed local/two-agent-smoke.mjs through the same RunSpec / Run.execute path that evaluations submit:

MOLTZAP_CONTROLLER_IMAGE=…@sha256:f02153… \
MOLTZAP_TEMPORAL_ADDRESS=127.0.0.1:7233 \
pnpm nx run @moltzap/simulator:local-run -- local/two-agent-smoke.mjs

The final controller result had exitCode: 0 and a CompletedLedgerReceipt for run 564b38ea-e3e1-49c0-b4db-3895cd945baf, ledger ac47ab30-374f-4866-bdae-e37e66282b1c.

Completion-gated artifacts were retained under the local profile's bind-mounted artifact root and independently checked:

• completion.json records recordCount: 8;
• SHA-256(manifest.json) = 387a26210e867fa8789bfab6aeef821890b2eb703e21da9ff701351e114075be;
• SHA-256(records.ndjson) = 2edb030d29b8eea29c257f314266c8509190af59dfba80f88a514ec8ae7bf931;
• both exactly match the completion marker;
• the ordered ledger contains two moltzap.agent-runtime-ready/v1 records and one moltzap.program-succeeded/v1 record.

Post-run inspection of kind-moltzap-simulator is clean: zero simulator-managed namespaces, zero Kueue Workloads, and zero Agent Sandboxes. The profile controllers and Temporal service remain as intended.

This proves the two-agent local acceptance slice. Next work is to make the eval-native credential and NanoClaw application-image contracts executable, then run the ten-agent smoke and all 32 cells; no GKE qualification claim is made here.

[chughtapan]

2026-08-04 live-validation checkpoint

Verified before the current VM failure:

• Local Kind profile: three nodes (one control plane, two workers), Kueue, Agent Sandbox, and Temporal.
• Two-agent Run.execute smoke completed through the core controller:
  • namespace mz-2e623c96a82349dfb6e2f90a30c90acf
  • run 564b38ea-e3e1-49c0-b4db-3895cd945baf
  • ledger ac47ab30-374f-4866-bdae-e37e66282b1c
  • controller exit 0, CompletedLedgerReceipt, eight records
  • completion manifest and records digests matched the completion marker; the ledger contained both runtime-ready events and moltzap.program-succeeded/v1.
  • no simulator-owned namespace, Workload, or Sandbox residue remained.

The first ten-agent attempt found a real local-profile capacity mismatch, not a passing proof. Kueue admitted one cohort, then created exactly ten Sandboxes, but the then-current per-agent reservation was 2 CPU / 2 GiB. The two workers could schedule six application Pods; four remained unschedulable with Insufficient cpu. The exact test namespace was deleted and verified absent.

The candidate now reserves 1 CPU / 1 GiB per distributed application container and has static tests for that exact reservation. It also binds non-secret profile/image/Temporal/artifact-location facts into the immutable evaluation report plan, so a resume cannot silently move an unfinished sweep.

The corrected ten-agent rerun and 32-cell sweep are not complete. The rebuilt controller image is available as moltzap-simulator-controller@sha256:f33384f26543fe76760683a9a77e0837a3629ae71f5c3f8f2ac83b25734ee3b1, but importing its 292 MB payload into the three new Kind nodes stalled for more than seven minutes and the local API began resetting connections. Subsequent evaluation test workers entered uninterruptible I/O. I terminated only the owned Kind-import and test processes; artifacts and repository content were left intact. This VM/Docker fault prevents a trustworthy corrected ten-agent or full-evaluation result here.

No GKE mutation was made. The previously observed cloud prerequisites remain unavailable in this environment (project/billing/service access and the required GKE tooling).

[chughtapan]

Follow-up: the isolated Kind cluster did not recover. Its API continued resetting connections, and an exact kind delete cluster --name moltzap-simulator failed because Docker could not kill any of its three node containers: could not kill container ... did not receive an exit event. I did not restart Docker because the VM has unrelated long-running containers. Recovering the local proof now requires an owner-approved Docker restart (or a fresh VM), then recreation of only the named Kind cluster and a rerun with the pinned corrected controller image.