Add Vally-first Copilot on Rails evaluation system - #1669
Draft
Alex Weininger (alexweininger) wants to merge 24 commits into
Draft
Add Vally-first Copilot on Rails evaluation system#1669Alex Weininger (alexweininger) wants to merge 24 commits into
Alex Weininger (alexweininger) wants to merge 24 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
…evaluation-system
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
…opt-out - Split the single shared repair budget into per-stage budgets (build / integration / local) so exhausting build repairs no longer starves local-runtime of every retry. - Raise the maxAgentRetries cap from 2 to 8 and set 6 on the React CRUD scenario. - Add progressMetrics(): report how far a run got (gates passed, pipeline depth, furthest stage, per-group ratios) instead of a single binary pass/fail, so two runs that both score 0% are distinguishable. - Make maxSeriousAccessibilityViolations: null mean "scan and record, but never enforce". An omitted threshold still enforces zero so opting out must be deliberate. - Regenerate the native eval specs from the generator. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
…rney Across all runs whose primary failure was localBrowserFailed, 32 of the 34 with usable evidence had the app rendering real content: the apps were running and we failed to drive them, then recorded that as a product failure. 26 of 42 failures share one signature, "form never filled". Split the probe accordingly: - The load contract (page served, body rendered, expected text present, interactive elements exposed) now runs immediately after navigation, before any scripted interaction. It is entirely app-controlled, so a failure is always a real defect, and a mis-resolved selector can no longer hide the fact that the app rendered correctly. - The journey contract (clicks, form fills, assertions) runs in its own scope. A journeySeverity of advisory records the outcome and lets the run continue to persistence and the debugger; required, the default, keeps failing local runtime. - The browser gate now reads the load contract and a new browser-journey gate reads the journey, so a probe interaction failure is no longer reported as "the app does not run". Make the journey evidence truthful, since an advisory metric that lies is worse than no metric: - A form fill that filled no field is no longer counted as a completed action. This is what let a probe report 3 of 3 actions with an empty formFieldsFilled list. - Every action records whether it observably changed the page, so a click on a mis-resolved target is visible instead of silent. - The diagnosis follows the ledger rather than the action count, so a run that never reached the form is no longer described as having filled and submitted it. Fix a false negative found while verifying the split: the intent-equivalent target search only considered the requested ARIA role, so a create affordance rendered as a router link was invisible and the journey died at the first action. The search now widens across the activatable roles, exact role first, and records the substitution. Persistence re-asserts the record the journey created, so it reports not-attempted rather than failed when the journey did not complete; otherwise the journey's false negative simply moves one gate downstream. Verified live against two reference apps: a working app passes the journey 3 of 3 with the assertion satisfied, and an app whose create affordance is ambiguous still fails with a precise reason. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
…figurations resolveDebuggerPrerequisite only emitted a command for coreclr attach configurations, so a Node or browser launch configuration produced no kind=debugger evidence and the gate failed as missing rather than as a measured outcome. That made the gate impossible to pass for every non-.NET scenario while the generator still marked it required whenever a scenario declares debugParity. Node attach configurations are now verified through the inspector's own /json/list endpoint, which is what VS Code reads when it attaches, so a listening socket that never publishes a debug target is still reported as a failure. Browser launch configurations verify that the url serves and that webRoot exists, because an absent webRoot means source maps never resolve and no breakpoint can bind. Checks retry, since language workers often start lazily. Configuration shapes we cannot observe return no checks instead of a synthetic pass, so the gate reports missing evidence rather than claiming a success it did not measure. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
…-checked Deployment was the third gate declared in GATE_GROUPS with no implementation, after debugger and security. Because an empty case and the default branch returned the same shape, an unimplemented gate was indistinguishable at runtime from missing evidence and was reported as "missing applicable evidence". GATE_GROUPS lacked "as const", so GateName widened to string and the switch was never checked for exhaustiveness. Adding it, exporting GateName, typing both gate functions and the applicability record, and deleting the default branch turns the whole class into a compile error: adding a gate now fails the build until it is either implemented or explicitly declared unimplemented. Tier 1 (deploymentReadiness) grades what the product actually promises. The azure-deploy agent stops at "azd package", so a run is graded on whether Copilot produced artifacts azd accepts, with no Azure resources and no extra cost. Tier 2 (deployedAcceptance) captures the deployed endpoint and probes it. The existing harness recorded a resource inventory, which proves Azure created things rather than that the application serves traffic. Both tiers separate product defects from host conditions. A missing azd, a stopped container runtime, an egress-blocked download, and Azure capacity failures are excluded rather than counted against the success rate. Verified against real infrastructure rather than mocks: - azd package passes in an ACA sandbox and fails correctly without a container runtime. - azd up provisioned Container Apps in westus3, the probe returned 200 from the generated application, and teardown left zero resource groups. Three defects were found by running it rather than by review: - validateDeploymentArtifacts threw ENOENT instead of reporting a missing azure.yaml, so any incomplete workspace crashed the validator. - azd package aborts with "loading environment: prompt required" unless AZURE_ENV_NAME is set, which would have failed every scenario. - azd reports the deployed URL as ingressUrl, not endpoints, so parsing the documented shape found no endpoint for a healthy deployment. The eastus2 capacity outage hit during bring-up is kept as a regression test, and reference-deployable is a known-good workspace so a tier-2 failure indicts the harness rather than Copilot. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
Three consecutive matrix runs reported 0/13 with every scenario failing at the earliest agent stage. The captured trajectories showed `assistant.turn_start` followed by no events at all until the 300s idle timeout, while the same scenario and model passed in 22s once the window cleared. The upstream turn starts and never returns, and nothing surfaces the fault. Measure silence directly so a stall is caught in seconds instead of consuming the whole timeout budget, and classify it as `agentRunStalled` so it is never counted against the product. Make the SDK log level configurable so the transport logs are reachable when this recurs. Also derive gate applicability from real scenario contracts: 13 of 20 scenarios had no probes at all, so gates were waived for missing contracts rather than because they did not apply. Adding generic load and process probes takes local-runtime from 4 to 13 scenarios and browser from 1 to 5. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
Adds a hard-blocking security gate that verifies generated apps actually refuse unauthenticated requests, and injects the product's external azure-prepare skill dependency into the eval workspace at a pinned commit. The security gate runs executable checks per scenario: an unauthenticated call and a malformed-bearer call to each protected endpoint must return 401/403, and a declared public path must still return 200. The public-path control guards the gate itself; without it a crashed or blanket-deny app would pass trivially. Both a public control and a protected refusal are required for a conclusive verdict, so missing evidence fails rather than passing on a technicality. The deploy skill is fetched into the ephemeral workspace rather than vendored, so the eval keeps tracking the real dependency and every run carries a provenance record. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
All seven auth scenarios had acceptance: null, which is why the security gate had never actually run against anything. Each now declares a public health probe, a protected collection probe, and a security contract. The protected route is now stated in the scenario prompt rather than guessed by the harness. Guessing a resource path would have graded the harness's naming luck instead of the product's auth wiring; stating it is also what a real user would do. These contracts stay deliberately minimal -- health plus authorization, no browser journey. These scenarios exist to prove auth is wired correctly, and adding UI journeys here would reintroduce flake without adding auth evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
runDeploymentStages swallowed infrastructure failures entirely, so a run whose deploy gate never rendered a verdict still reported autonomous_success. That is the same absent-evidence-as-success problem the executable gates exist to prevent. Infrastructure failures now surface as failures and are classified as infrastructure by the report, which excludes the attempt from product-quality rates instead of inflating them. Absent evidence and passing evidence stay distinguishable. Found by the first real --through deploy run, which reported success while azd package had actually failed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
The compound launch path already distinguished a registry failure from a project failure, but the single path hardcoded localTaskFailed. Every non-compound scenario whose emulators come from a container image was therefore charged to the generated project when the sandbox could not pull the image. This produced a concrete false signal: api-node-express-redis failed 3/3 in the current matrix with what looked like a reproducible product defect. The real cause was a denied redis:7.4-alpine pull, an environment constraint the generated project has no control over. Both the pre-launch task path and the launch configuration path now classify pull denials as localContainerRegistryUnavailable, which the report already treats as infrastructure and excludes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
matrix-13-run3 recorded candidateCommit 65ffe24 while the working tree moved through four commits during the run. tsx recompiles on import, so the matrix silently executed a mixture of revisions. One scenario failed 3/3 with "contract is not defined" -- a half-finished edit picked up mid-flight, scored as a product failure. The provenance record was therefore false: it named a commit that was never executed. Numbers that cannot be reproduced are worse than no numbers, because they look like evidence. Runs now compare the tree against HEAD for the paths that determine what is measured (evals/, resources/, src/) and refuse to start on drift. COR_EVAL_ALLOW_DIRTY_SOURCE=true still allows local iteration and records dirty: true so the result is never mistaken for reproducible. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
The 90s silence budget was tuned against a single worker. Concurrency slows individual turns, so the same budget at eight workers would start reporting healthy-but-slow turns as agentRunStalled -- converting a throughput change into fabricated infrastructure failures. COR_EVAL_STALL_TIMEOUT_MS now sets it. Malformed, zero, and negative values fall back to the 90s default rather than silently disabling stall detection, since a disabled watchdog looks identical to a healthy one until a matrix is lost. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
The grader certification proved the golden fixture "passed" without proving each declared gate actually ran. Three gaps followed from that. The ACA golden case checked only `outcome === 'passed'`, which cannot distinguish a gate that ran and succeeded from one that never ran. It now asserts per-validator evidence drawn from the manifest's acaValidators list, so a silently skipped gate fails certification. That assertion immediately caught a real defect: resolveDebuggerPrerequisite handled attach+coreclr, attach+node, and launch+browser, but a Node `launch` configuration fell through to zero checks. Zero checks returned "no failure", which read as a verified debugger. Node launch configs declare their debug surface via --inspect in runtimeArgs rather than a port property, so the gate did nothing for the most common Node shape. resolveNodeLaunchPrerequisite now verifies the declared inspector port using the same attachable-target evidence as the attach path. Persistence and the debug surface had no negative control, so neither was proven able to fail. Two mutations close that: persistence-not-durable moves storage to a PID-keyed path so the journey still passes but the record does not survive restart, and debug-port-mismatch moves the inspector off the declared port while the app keeps serving. debug-port-mismatch expects localProbeFailed rather than localDebuggerUnavailable because probes carry debugPort and run first. Breaking an upstream prerequisite cannot isolate a downstream gate. Certification: 18/18 offline, 8/8 ACA. Tests 262/262. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
The security and deployment gates were the last two with no certification evidence at all. Both were reporting outcomes that nothing had checked. Security gate: - Adds a protected /api/admin/stats route and bearer-token authorization to the reference project, plus the matching acceptance contract. - Uses an admin route rather than protecting /api/items, because Playwright cannot authenticate and protecting the journey's own endpoint would break the browser and persistence gates. A downstream gate cannot be isolated by breaking an upstream prerequisite. - Declares protectedPaths explicitly. The default treats every backend probe that is not public as protected, which would wrongly include the UI route. - Defaults the expected token to randomBytes(32) rather than the empty string, so the comparison branch executes on the live path instead of short-circuiting. No credential is committed. - Adds the security-auth-bypass mutation as the negative control. It reports localSecurityFailed with real evidence: "Expected 401 or 403 but got 200". Deployment gate (tier 1, azd package): - Wires up reference-deployable, which was orphaned: nothing in the manifest or the TypeScript referenced it despite its header declaring its purpose. - Adds the missing .azure/deployment-plan.md, without which the golden case fails as deploymentPlanMissing. - Adds a deploy tier to the certifier with three mutations covering all three reachable failure codes, mapped empirically rather than by guesswork. Known blind spot, found by probing rather than assumed: dockerfile-broken reports passed. With remoteBuild: true, azd package never builds the image, so tier 1 cannot see Dockerfile defects. Those would surface only at azd up. Dockerfile defects are a plausible generation failure, so this is a real gap in the gate's reach and is deliberately recorded rather than papered over. Also fixes the .azure/.gitignore in the deploy fixture. It ignored everything, so the required deployment-plan.md would have been absent in a fresh checkout and CI would have failed the golden case for the wrong reason. azd environment state stays ignored. missingGateEvidence, the assertion that stops a golden case from passing when a validator never ran, had no test of its own. It now has eight, and graderCertification.ts gets the require.main entrypoint guard every other entrypoint already had. Offline 18/18, deploy 4/4, ACA 9/9, suite 270/270, tsc and lint clean. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
The gates were certified but the corpus never asked most of them anything. Across the last 20-scenario matrix, browser-journey, accessibility, persistence and deployment reported not-applicable on all 20 runs, so the measured pass rate was an optimistic upper bound: the four passing scenarios cleared a materially narrower bar than the gate list suggested. crud-react-functions-postgres is a React and Postgres CRUD app that passed with its journey, accessibility and persistence gates all switched off. Gate coverage across the 20-scenario corpus, before to after: browser 5 -> 12 accessibility 0 -> 12 browser-journey 0 -> 4 persistence 0 -> 4 What changed, and why each is fair rather than a guess: - crud-react-functions-postgres already had a complete journey and persistence contract that was simply disabled via journeySeverity advisory and a null accessibility budget. Only those two fields changed. - Journeys authored for the three scenarios whose browser probe checked page load only. Each prompt promises the behaviour being exercised, so the contract asserts the product's own stated job. - static-html-functions-blob gets accessibility but no create journey: its prompt promises browsing photo metadata, not creating it. - Seven CRUD scenarios had frontends that were never exercised at all, having no frontend probe of any kind. They now get a load and accessibility probe on the port the skill pins (Vite 5173, Angular 4200). Their journeys stay advisory because the UI sits behind a real auth provider the evaluator cannot log into, which is a genuine not-applicable rather than a missing gate. Journeys lean on machinery the harness already had: fillForm matches labels by case-insensitive substring and auto-fills whatever else the app decided to require, findIntentEquivalent strips stop words and widens across activatable roles, and unmatched optional actions are skipped. Contracts declare several candidate label keys with one distinctive value, so the assertion holds whichever field name the generated app chose. This measures whether the app works, not whether it guessed our labels. Persistence restarts the frontend rather than the backend for the three new contracts, because only crud-react-functions-postgres declares a backend readiness probe and multi-target restart needs local.compound. The scenario schema caught both, which is the schema doing its job. Also fixes a real bug in the ambiguous-click scorer: formFieldsFilled is an array, so `formFieldsFilled > 0` was always false and the preference for the filled form's own submit control never applied. A test asserted the presence of that expression, pinning the defect in place; it now asserts the fix. This path is exactly what the newly required journeys depend on. Certification unchanged: offline 18/18, ACA 9/9, suite 270/270, tsc and lint clean. Generated Vally specs regenerated for the new applicability tags. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
…e time The storage-event probe signed every Azurite request with a 25-character account key (17 bytes decoded) instead of Azurite's published 88-character, 64-byte development key. Verified against a live emulator: the corrupted key returns 403 AuthenticationFailed on the first request, the canonical key returns 201 Created. No generated app could ever have passed. The worker gate recorded 16 failures and zero passes across every run in the history of the eval system. Two scenarios -- 10% of the corpus -- were being charged for a harness defect. Adds gateHealth.ts to make this class of bug self-reporting. It flags gates that never passed (suspect probe), never failed (suspect vacuous), or were never attempted (starved by upstream cascade), and exits nonzero when a gate ran and never once succeeded. Applied to the current corpus it also shows that cascade failures are being recorded as status=failed, which inflated persistence from 0 real verdicts to an apparent 78 failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
The ACA certification suite scored 9/9, then 1/9, then 5/9 across three runs with no change to any grader. The failures were `aca sandbox apply` and `aca sandbox delete` returning non-zero -- control-plane flakiness, not graders missing a defect. In one case the grader had already produced exactly the expected signal (`persistence: locator.waitFor: Timeout 15000ms exceeded`) and was still recorded as failed because cleanup errored afterwards. Cases whose failure codes are sandbox lifecycle errors are now classified `inconclusive` and reported separately from grader mismatches. `sandboxCommandFailed` is deliberately excluded from that set: it is a real product signal that several mutations legitimately expect. Inconclusive still exits non-zero -- unknown is not the same as pass -- but it no longer reads as evidence that a gate is broken. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
… defects A degraded ACA run reported two "grader mismatches" that were both `localToolchainUnavailable: node local-runtime toolchain is unavailable` -- the sandbox had no node, so no grader could have rendered a verdict. Adds the host-environment codes to the infrastructure set. Deliberately keeps `localContainerRegistryUnavailable` out: an image pull can fail because the generated compose file names an image that does not exist, and classifying that as infrastructure would hide real product defects. The same suite now reports 5 passed / 4 inconclusive / 0 grader mismatches instead of 5/9 failed, which is the difference between "four gates are broken" and "ACA could not provision four sandboxes". Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
Three related honesty defects in the measurement instrument, all the same disease: the eval charged the product for failures it did not cause. 1. Cascade blocks were persisted as gate failures. The diagnostic layer already distinguished "ran and failed" from "never ran", but the executor copied only the explanation and discarded the status, leaving prose for downstream tools to pattern-match. Gates now carry an explicit notAttempted marker. The status deliberately stays failed -- a required gate that never ran must not score as a pass -- but per-gate statistics can now exclude blocks the product never earned. persistence had accumulated 78 lifetime "failures", zero of which were real verdicts. 2. A GitHub auth outage scored the product at 0%. The native runner correctly classified all 20 trials as harness_failure, and vally already knows to exclude those from product quality, but the model-observation assertion threw first and converted the classified failure into an unclassified executor error. The exemption is narrow: an autonomous success, or a failure blamed on the product, must still prove which model produced it. 3. Three gates had never failed and had no negative control, so there was no evidence they could detect anything. model, provenance and cleanup are the gates it is most dangerous to be wrong about -- a silently passing cleanup gate means leaked Azure resources billing real money, and silently passing identity gates make every reported number unverifiable. All four new controls pass, so their clean sheet is now credible rather than merely untested. Also wires three test files into the suite that were never being run, including the Azurite key regression test added in 6b1c0a3. A pinning test that no runner executes is decoration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cb4fb14d-a092-4456-b611-b7aa68be64e3
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Vally-first evaluation and release-evidence system for Copilot on Rails.
railsandbaseline-controlledarms with the same model, scenario, endpoint, and attempt.This is a stacked draft targeting
feat/CoRand does not modify the existing PR #1615.What the full suite runs
The standard suite has three layers:
The paired Vally experiment already runs both Rails and baseline. Do not run the standalone baseline command in addition unless debugging baseline behavior.
One-time teammate setup
Prerequisites
Install and authenticate the ACA CLI
Linux/macOS:
curl -fsSL https://aka.ms/aca-cli-install | shWindows PowerShell:
This same install path is also used inside sandboxes and containers for agent-driven self-installs.
Only open interactive sign-in when cached authentication is absent:
aca doctormust be green before running ACA tiers.An administrator can grant a teammate access to the shared group:
To create a separate evaluation group:
The group creator already receives Data Owner.
--set-configis required so subsequent evaluator commands resolve the group.The evaluator uses checked-in declarative manifests. For new manifests, the reproducible CI/CD flow is:
The manifest pattern is the recommended CI/CD and reproducibility path. Use
aca sandbox schemafor editor autocomplete.Local environment
COR_EVAL_OWNER_IDmust be lowercase alphanumeric/hyphen text and at most 63 characters. Never commit tokens or generated result workspaces.Run the complete local suite
1. Deterministic contracts
2. Real ACA grader certification
Stop if grader certification fails; model/product results should not be interpreted while an oracle is uncertified.
To diagnose one case:
3. Paired Rails and baseline E2E
Inspect the exact plan before making paid calls:
Run the primary-model compatibility pilot:
This command runs two scenarios x two arms x one attempt = four paid trials. Equivalent aliases exist for
claude-sonnet-5andgpt-5.4-mini.Aggregate durable evidence:
Review:
report.mdvally-native-report.jsonexperiment-input-manifest.jsonartifacts/native-summary.jsonartifacts/cor-validation.jsonartifacts/validation-manifest.jsoncandidatemeans all configured gates passed.holdmeans complete evidence contains a gate failure.insufficient_evidencemeans required coverage or proof is missing and is not a passing result.Cleanup verification
aca sandbox list -l "owner-id=$COR_EVAL_OWNER_ID" -o jsonThe result must be empty. If diagnostic state must be preserved before destructive cleanup:
Never use a broad selector or delete another run's sandboxes.
CI setup
Workflow:
.github/workflows/copilot-on-rails-evals.ymlRepository variables:
COR_EVAL_DAILY_ENABLED=trueCOR_EVAL_WEEKLY_ENABLED=trueCOR_EVAL_RESOURCE_GROUPCOR_EVAL_SANDBOX_GROUPCOR_EVAL_REGIONRepository secrets:
COR_EVAL_AZURE_CLIENT_IDCOR_EVAL_AZURE_TENANT_IDCOR_EVAL_AZURE_SUBSCRIPTION_IDCOR_EVAL_COPILOT_GITHUB_TOKENThe Azure identity needs
Container Apps SandboxGroup Data Owneron the configured group.Daily is four paid trials. Weekly is 48 paid trials. A local release experiment is 120 paid trials. CI caps concurrency at two, retains evidence for 30 days, and runs exact-owner cleanup even after failures.
The release dispatch currently fails closed before paid calls because provenance-bound real VS Code breakpoint evidence and explicitly authorized live-deployment evidence are not yet wired into that workflow.
Extending and acting on the evaluations
See
evals/CONTRIBUTING.mdfor:Validation completed
Initial evidence and limitations
The corrected three-model compatibility pilot produced six matched pairs:
The system therefore currently recommends
insufficient_evidence, not release. The strongest recurring product gap is the React + Functions + PostgreSQL journey. Current real breakpoint evidence predates the latest provenance contract, and live deployment remains explicitly authorization-gated.