feat: expect value pins on probes (v0.1.1) - #9
Conversation
Layer 3 gains deterministic value pins: expect = { field = value } on a
[[probes]] entry makes exact output values part of the contract. A pinned
field that comes back missing or unequal at check time is BREAKING --
schema and shape identical, meaning changed -- closing the gap between
the shape fingerprint and the advisory judge. Opt-in, exact equality,
no tolerance, no auto-pinning.
Also: README installs from PyPI (git+ snippet retired), monthly scheduled
CI run as a freshness alarm, checkout/setup-python bumped off Node 20,
CI asserts COVENANT_SEMANTIC_DRIFT=1 exits 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 55 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis PR adds exact-value pinning for probe results, wires it through config loading, diffing, and CLI checks, and updates docs, tests, and CI to cover the new drift behavior. ChangesValue Pin Drift Detection
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant covenant_cli as covenant/cli.py
participant covenant_config as covenant/config.py
participant covenant_diff as covenant/diff.py
participant live_probe as probe response
covenant_cli->>covenant_config: load probes with expect pins
covenant_cli->>live_probe: re-run configured probes
covenant_cli->>covenant_diff: diff_expect(tool, expect, response)
covenant_diff-->>covenant_cli: value_pin_missing / value_pin_mismatch changes
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
.github/workflows/release.yml (1)
11-12: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winSame credential-persistence hardening applies here.
Static analysis flags this
actions/checkoutstep for missingpersist-credentials: false, same as inci.yml.🔒️ Proposed fix
- uses: actions/checkout@v5 + with: + persist-credentials: false - uses: actions/setup-python@v6🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/release.yml around lines 11 - 12, The release workflow’s actions/checkout step has the same credential persistence issue as ci.yml. Update the checkout invocation in the release workflow to disable credential persistence by setting persist-credentials to false on the actions/checkout step, keeping the rest of the job unchanged.Source: Linters/SAST tools
.github/workflows/ci.yml (1)
30-31: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winConsider disabling credential persistence on checkout.
Static analysis flags both
actions/checkoutsteps for not settingpersist-credentials: false. The default persists the GitHub token in the local git config for the remainder of the job, which is exploitable if later steps execute untrusted code (e.g., third-party actions, dependency installs).🔒️ Proposed fix
- uses: actions/checkout@v5 + with: + persist-credentials: false - uses: actions/setup-python@v6Also applies to: 46-47
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/ci.yml around lines 30 - 31, The workflow’s actions/checkout steps are persisting the GitHub token in git config by default, which should be disabled. Update each checkout invocation in the CI workflow to set persist-credentials to false on the actions/checkout usages so later steps cannot reuse the token. Use the checkout step entries in the CI workflow as the target locations.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_cli.py`:
- Line 89: The snapshot step in this test is not being checked, so a silent
failure can surface later as an unrelated check failure; update the test around
the `runner.invoke(app, ["snapshot"])` call to capture the result and assert it
succeeded before continuing with the `check` assertions, matching the pattern
used in `test_probes_catch_behavioral_drift`.
---
Nitpick comments:
In @.github/workflows/ci.yml:
- Around line 30-31: The workflow’s actions/checkout steps are persisting the
GitHub token in git config by default, which should be disabled. Update each
checkout invocation in the CI workflow to set persist-credentials to false on
the actions/checkout usages so later steps cannot reuse the token. Use the
checkout step entries in the CI workflow as the target locations.
In @.github/workflows/release.yml:
- Around line 11-12: The release workflow’s actions/checkout step has the same
credential persistence issue as ci.yml. Update the checkout invocation in the
release workflow to disable credential persistence by setting
persist-credentials to false on the actions/checkout step, keeping the rest of
the job unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d3fc4a60-d55b-4f90-b62a-7584cd43c19a
📒 Files selected for processing (11)
.github/workflows/ci.yml.github/workflows/release.ymlCLAUDE.mdREADME.mdcovenant.tomlcovenant/cli.pycovenant/config.pycovenant/diff.pypyproject.tomltests/test_cli.pytests/test_probes.py
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Tip For best results, initiate chat on the files or code changes.
Just let me know what you'd like, or use (๑•̀ᴗ•́)و ✧ |
|
@CodeRabbit Review the pr please |
|
@coderabbitai review |
|
✅ Action performedReview finished.
|
✅ Action performedReview finished.
|
…ever, scope fence) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
What
Layer 3 gains value pins — the missing rung between the shape fingerprint and the advisory judge:
A pinned field that comes back missing or unequal at check time is BREAKING — schema and shape identical, meaning changed (dollars→cents, USD→ILS). Deterministic and exact by design: no tolerance, no regex, no auto-pinning. Opt-in — nothing is pinned unless you type it. Analogue:
pip --require-hashes.Detection ladder now: schema diff → shape fingerprint → value pins → LLM judge (advisory).
Also in this PR
pip install covenant-mcpeverywhere (git+ CI snippet retired — package is live on PyPI), pins documented, severity table updatedCOVENANT_SEMANTIC_DRIFT=1 covenant checkexits 1,checkout/setup-pythonbumped off Node 20Verification
ruff+mypy --strictclean; 136 passed, 3 skipped (Postgres skips expected)COVENANT_DRIFT1 /COVENANT_BEHAVIOR_DRIFT1 /COVENANT_SEMANTIC_DRIFT1 (new)covenant snapshot --forcere-run: lock byte-identical (pins live in config, never in the lock)After merge (release recipe)
tag
v0.1.1→ release workflow builds + publishes to PyPI via trusted publishing.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
checknow detects missing or changed pinned values even when the response structure matches.Documentation
Tests
Chores