Add run/artifact metadata to persisted evals records

[Copilot]

Persisted evals.jsonl records on the evals/* state branches did not carry enough provenance to tie a result back to the workflow run and uploaded artifact that produced it. This change threads both identifiers through the evals pipeline so downstream consumers can correlate stored results with the originating run and artifact.

• Propagate artifact metadata through the evals job

  • Give the evals artifact upload step a stable step ID.
  • Expose the uploaded artifact ID as a job output so downstream jobs can consume it.

• Enrich persisted eval records

  • Pass github.run_id and the evals artifact ID into push_evals_state.
  • Rewrite evals.jsonl during state publication so each JSONL record gains:
    • runid
    • artifactid
  • Leave malformed/non-JSON lines unchanged rather than failing the publish step.

• Keep the state-push path generic

  • The metadata enrichment is scoped to evals.jsonl; other state files continue to use the existing copy/push behavior unchanged.

• Add coverage for the new contract

  • Assert the generated workflow wires the artifact ID from the evals upload step into the push job.
  • Add JS coverage for JSONL enrichment behavior, including mixed valid/invalid lines.

Example persisted record shape:

{
  "id": "labels-applied",
  "question": "Did the agent apply at least one label?",
  "answer": "YES",
  "model": "small",
  "timestamp": "2026-07-15T13:00:59.932Z",
  "runid": "123456789",
  "artifactid": "987654321"
}

[pelikhan]

@copilot store runid when creating raw entry to avoid reparsing. Ignore artifact id

[Copilot]

@copilot store runid when creating raw entry to avoid reparsing. Ignore artifact id

Addressed in ab4e30f.

[github-actions]

✅ Test Quality Sentinel completed test quality analysis.

[Copilot]

Pull request overview

Adds workflow-run provenance to eval records, but artifact metadata propagation and state-publication enrichment remain unimplemented.

Changes:

• Passes github.run_id into eval parsing.
• Adds runid to generated eval records and tests.
• Regenerates affected workflow lock files.

Show a summary per file

File	Description
pkg/workflow/evals_steps.go	Passes the run ID to the parser.
pkg/workflow/evals_steps_test.go	Tests run-ID wiring.
actions/setup/js/run_evals.cjs	Adds runid to records.
actions/setup/js/run_evals.test.cjs	Tests persisted run IDs.
.github/workflows/smoke-copilot.lock.yml	Regenerates run-ID environment wiring.
.github/workflows/contribution-check.lock.yml	Regenerates run-ID environment wiring.
.github/workflows/breaking-change-checker.lock.yml	Regenerates run-ID environment wiring.
.github/workflows/auto-triage-issues.lock.yml	Regenerates run-ID environment wiring.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 8/8 changed files
• Comments generated: 1
• Review effort level: Medium

[github-actions]

🧪 Test Quality Sentinel Report

⚠️ Test Quality Score: 60/100 — Acceptable

Analyzed 2 test(s): 2 design, 0 implementation, 0 violation(s).

📊 Metrics (2 tests)

Metric	Value
Analyzed	2 (Go: 1, JS: 1)
✅ Design	2 (100%)
⚠️ Implementation	0 (0%)
Edge/error coverage	0 (0%)
Duplicate clusters	0
Inflation	Yes (JS: +67 lines vs +2 prod lines; Go: +4 vs +1 prod)
🚨 Violations	0

Test	File	Classification	Issues
stores the workflow run id when writing eval records	run_evals.test.cjs:48	design_test	happy-path only, no error assertions
TestBuildParseEvalsResultsStepUsesResolvedExecutionModel (+4 lines)	evals_steps_test.go:173	design_test	none

⚠️ Flagged Tests (1)

stores the workflow run id when writing eval records (run_evals.test.cjs:48) — design_test, high_value. Tests the happy path for runid written to JSONL records. No error-path assertions: missing GITHUB_RUN_ID env var (falls back to "unknown") and invalid log-file content are not exercised. Adding at least one edge assertion would raise the score.

Inflation note: run_evals.test.cjs is a new file (+67 lines) testing only 2 net lines added to run_evals.cjs (ratio ~33:1). This is expected when bootstrapping a test file from scratch, not a quality concern per se.

Verdict

✅ Passed. 0% implementation tests (threshold: 30%). No guideline violations. Edge/error coverage is weak — consider adding an assertion for the GITHUB_RUN_ID=undefined fallback path ("unknown").

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

🧪 Test quality analysis by Test Quality Sentinel · 33.5 AIC · ⌖ 18 AIC · ⊞ 6.8K · ◷
_{Comment /review to run again}

[github-actions]

✅ Test Quality Sentinel: 60/100. 0% implementation tests (threshold: 30%). No guideline violations.

[github-actions]

Review: Add run/artifact metadata to persisted evals records

The runid wiring is correct and the Go/lock-file changes look good. Three issues need resolution before merging:

Blocking

1. artifactid spec/implementation mismatch — The PR description and example JSON promise an artifactid field, but the diff never adds it. Either implement it or update the description. (comment on run_evals.cjs line 151)
2. Module re-import in beforeEach is a no-op — await import('./run_evals.cjs') returns the cached module on every call; shared mutable state (the results array) leaks across tests. Use createRequire at the top level, matching every other test file here. (comment on run_evals.test.cjs line 37)

Non-blocking (but worth fixing)
3. Raw process.env mutation — Use vi.stubEnv() / vi.unstubAllEnvs() for env isolation, consistent with all peer test files and safe against mid-test failures. (comment on run_evals.test.cjs line 32)

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 38.8 AIC · ⌖ 4.68 AIC · ⊞ 5K

Done

[pelikhan]

@copilot run pr-finisher skill

[Copilot]

@copilot run pr-finisher skill

Done in ef9b0d7. I ran a PR-finisher pass, fixed the evals JS test isolation issues, and verified the focused run_evals tests locally; the remaining make test/make agent-report-progress failure is the existing DNS-proxy bootstrap failure in pkg/cli.

[github-actions]

🎉 This pull request is included in a new release.

Release: v0.82.10