Skip to content

fix(benchmark): distinguish credential probes from task code - #3305

Closed
huangruiteng wants to merge 1 commit into
mainfrom
codex/benchmark-integrity-probe-intent
Closed

fix(benchmark): distinguish credential probes from task code#3305
huangruiteng wants to merge 1 commit into
mainfrom
codex/benchmark-integrity-probe-intent

Conversation

@huangruiteng

Copy link
Copy Markdown
Owner

Summary

  • distinguish credential observation attempts from source or fixture text that merely mentions environment APIs
  • retain fail-closed detection for shell environment dumps and explicit Python environment output
  • add semantic regressions for source patches, generated fixtures, and subprocess environment injection

Motivation

The integrity reducer previously treated any environment-related syntax in tool arguments as a credential probe. Benchmark tasks can legitimately edit or analyze code containing that syntax, so syntax alone is not evidence of runtime credential observation. This change narrows the signal to observation intent while preserving the existing isolation attestation boundary.

Validation

  • pytest -q tests/capabilities/test_benchmark_toolkit.py (18 passed)
  • python -m py_compile loopx/capabilities/benchmark_toolkit/integrity.py tests/capabilities/test_benchmark_toolkit.py
  • scripts/loopx canary premerge --from-git-diff (5 selected canaries passed; benchmark-sensitive manual review hold remains)

No runner, scoring, task, permission, or submission behavior changes. No raw benchmark evidence is included.

@huangruiteng

Copy link
Copy Markdown
Owner Author

Closing as a duplicate after PR-program reconciliation. #3298 already carries the more complete credential-probe intent classifier, includes the public contract documentation, and passes the new representative private trajectory replay. Keeping one review line avoids divergent implementations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant