Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,21 @@ adheres to [Semantic Versioning](https://semver.org/).

## [Unreleased]

## [0.2.1] - 2026-07-14

### Fixed

- Modernized packaging license metadata to the PEP 639/SPDX form
(`license = "MIT"`, `license-files = ["LICENSE"]`) and removed the
deprecated license classifier; builds no longer emit setuptools license
deprecation warnings.
- Converted README documentation links to absolute GitHub URLs so they work
from the PyPI long-description renderer.
- Corrected stale compatibility language that incorrectly claimed every
v0.1 score remained identical despite the documented nested-safety-scan
fix (70.0 -> 70.83 for one sample; verdict unchanged).


## [0.2.0] - 2026-07-14

### Added
Expand Down Expand Up @@ -90,8 +105,10 @@ adheres to [Semantic Versioning](https://semver.org/).

### Compatibility

- v0.1.x trace and policy files produce identical scores, verdicts, and
finding messages (pinned by golden tests). New report fields are additive.
- v0.1.x trace and policy files remain accepted; verdicts and finding
messages are preserved. The documented nested-safety-scan fix changes one
sample score from 70.0 to 70.83 (verdict unchanged). New report fields are
additive.
`arh compare` accepts v0.1.x baselines via a rule-id-agnostic fallback
match. Exit codes and console markers are unchanged. See COMPATIBILITY.md.

Expand Down
12 changes: 7 additions & 5 deletions COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@ forward.
- **Trace and policy files work unchanged.** The v0.1.x formats are strict
subsets of schema v1; a missing `schema_version` means `"1"`. No migration
step exists because none is needed.
- **Identical scores and verdicts.** New check categories (sequence, flow,
completion) are *not applicable* for policies that don't configure them, and
the score renormalizes over applicable categories only — so a v0.1.x policy
yields byte-identical scores. Pinned by `tests/test_compat_golden.py`
(sample scores 100.0 / 80.0 / 70.83, verdicts, and finding messages) and
- **Preserved verdicts; scores preserved except one documented safety fix.**
New check categories (sequence, flow, completion) are *not applicable* for
policies that do not configure them, and the score renormalizes over applicable
categories. Golden tests pin scores 100.0 / 80.0 unchanged; the unsafe sample
changes from 70.0 to 70.83 because v0.2 recursively scans nested content that
v0.1 silently skipped. All three verdicts and finding messages are preserved.
See `tests/test_compat_golden.py` and
`tests/test_trajectory_rules.py::TestLegacyScoreCompatibility`.
- **Finding messages** for all v0.1.x checks are unchanged (golden-tested).
- **Console markers** (`[PASS]`, `[FAIL]`, `[safety ]` category blocks,
Expand Down
48 changes: 25 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ GATE: FAIL
The gate fails on new error findings, pass→fail transitions, and added failing
traces — resolved findings and unchanged failures don't re-alarm. Gate modes:
`--fail-on regressions` (default), `failures`, `never`; add `--max-score-drop`
for score-based gating. See [docs/regression-testing.md](docs/regression-testing.md).
for score-based gating. See [docs/regression-testing.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/regression-testing.md).

## What it checks

Expand All @@ -96,23 +96,23 @@ Seven deterministic categories; every finding carries a stable rule ID
| flow | ARH-FLW-001..003 | ignored tool errors, retry storms, duplicate side effects |
| completion | ARH-CMP-001..002 | missing final response, failure to terminate |

Full reference: [docs/rules.md](docs/rules.md). Ordering rules are a
Full reference: [docs/rules.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/rules.md). Ordering rules are a
deliberate **partial order** (constraints), not an exact golden-trajectory
match — agents can legitimately reach a goal via different paths.

## Trace formats

`arh validate` accepts (auto-detected, or forced with `--format`):

- **arh** — the canonical JSON trace format ([TRACE-SPEC.md](TRACE-SPEC.md));
- **arh** — the canonical JSON trace format ([TRACE-SPEC.md](https://github.com/felmonon/agent-reliability-harness/blob/main/TRACE-SPEC.md));
- **openai-chat** — OpenAI Chat Completions message lists with `tool_calls`;
- **anthropic-messages** — Anthropic Messages conversations with
`tool_use`/`tool_result` blocks.

Adapters never guess: fields a transcript format cannot carry (latency, cost,
tokens) are left unset, which marks the dependent checks *not applicable*
instead of silently passing them. Unparseable input is recorded in the trace's
metadata, not dropped. See [docs/adapters.md](docs/adapters.md).
metadata, not dropped. See [docs/adapters.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/adapters.md).

## Reports

Expand All @@ -134,46 +134,48 @@ no timestamps, no randomness, no clock reads.
The action writes JSON/Markdown/JUnit/SARIF reports, appends the Markdown
summary to the workflow step summary, and fails according to the gate. It uses
no secrets and is fork-safe. Full reference and SARIF/JUnit upload examples:
[docs/ci.md](docs/ci.md).
[docs/ci.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/ci.md).

## Design principles

- **Zero runtime dependencies.** `pip install` adds nothing to your
application's dependency graph.
- **Deterministic core.** No model calls, no network, no telemetry, no clock
reads. Semantic (model-graded) evaluation is a planned, clearly separated
optional extra — never hidden inside deterministic scores ([ROADMAP.md](ROADMAP.md)).
optional extra — never hidden inside deterministic scores ([ROADMAP.md](https://github.com/felmonon/agent-reliability-harness/blob/main/ROADMAP.md)).
- **Additive schema evolution.** v0.1.x traces, policies, and baselines work
unchanged; unknown major schema versions are rejected loudly
([COMPATIBILITY.md](COMPATIBILITY.md)).
([COMPATIBILITY.md](https://github.com/felmonon/agent-reliability-harness/blob/main/COMPATIBILITY.md)).
- **Evidence over claims.** The benchmark suite contains 34 seeded cases
(28 expected-fail, 6 expected-pass controls) and measures detection:
currently 34/34 correct, precision 1.0, recall 1.0, 0 false
positives/negatives, byte-identical repeat runs, ~0.03 ms per trace. Those
numbers cover *seeded, deterministically detectable* failures only — scope
and limits are documented in [BENCHMARK-METHODOLOGY.md](BENCHMARK-METHODOLOGY.md),
results in [BENCHMARK-RESULTS.md](BENCHMARK-RESULTS.md).
and limits are documented in [BENCHMARK-METHODOLOGY.md](https://github.com/felmonon/agent-reliability-harness/blob/main/BENCHMARK-METHODOLOGY.md),
results in [BENCHMARK-RESULTS.md](https://github.com/felmonon/agent-reliability-harness/blob/main/BENCHMARK-RESULTS.md).

## Documentation

| Doc | What it covers |
|---|---|
| [docs/quickstart.md](docs/quickstart.md) | Five minutes from install to a regression gate |
| [docs/concepts.md](docs/concepts.md) | Traces, policies, findings, scores, baselines |
| [docs/policy-cookbook.md](docs/policy-cookbook.md) | Copy-paste recipes for every rule type |
| [docs/adapters.md](docs/adapters.md) | OpenAI/Anthropic transcript ingestion, field mapping |
| [docs/regression-testing.md](docs/regression-testing.md) | Baselines, fingerprints, gates |
| [docs/ci.md](docs/ci.md) | GitHub Action reference and workflows |
| [docs/rules.md](docs/rules.md) | Every rule ID with remediation |
| [docs/troubleshooting.md](docs/troubleshooting.md) | Error messages and fixes |
| [TRACE-SPEC.md](TRACE-SPEC.md) / [POLICY-SPEC.md](POLICY-SPEC.md) | Normative formats |
| [ARCHITECTURE.md](ARCHITECTURE.md) / [DECISIONS.md](DECISIONS.md) | Design and ADRs |
| [docs/quickstart.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/quickstart.md) | Five minutes from install to a regression gate |
| [docs/concepts.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/concepts.md) | Traces, policies, findings, scores, baselines |
| [docs/policy-cookbook.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/policy-cookbook.md) | Copy-paste recipes for every rule type |
| [docs/adapters.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/adapters.md) | OpenAI/Anthropic transcript ingestion, field mapping |
| [docs/regression-testing.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/regression-testing.md) | Baselines, fingerprints, gates |
| [docs/ci.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/ci.md) | GitHub Action reference and workflows |
| [docs/rules.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/rules.md) | Every rule ID with remediation |
| [docs/troubleshooting.md](https://github.com/felmonon/agent-reliability-harness/blob/main/docs/troubleshooting.md) | Error messages and fixes |
| [TRACE-SPEC.md](https://github.com/felmonon/agent-reliability-harness/blob/main/TRACE-SPEC.md) / [POLICY-SPEC.md](https://github.com/felmonon/agent-reliability-harness/blob/main/POLICY-SPEC.md) | Normative formats |
| [ARCHITECTURE.md](https://github.com/felmonon/agent-reliability-harness/blob/main/ARCHITECTURE.md) / [DECISIONS.md](https://github.com/felmonon/agent-reliability-harness/blob/main/DECISIONS.md) | Design and ADRs |

## Compatibility

v0.1.x trace and policy files produce identical scores, verdicts, and finding
messages under this version (pinned by golden tests). New fields are additive.
Details: [COMPATIBILITY.md](COMPATIBILITY.md).
v0.1.x trace and policy files remain accepted and preserve verdicts and finding
messages. Two review-driven detection fixes are documented exceptions: nested safety
scanning changes one sample score from 70.0 to 70.83, and strict telemetry validation
rejects malformed values that v0.1 accepted. New report fields are additive.
Details: [COMPATIBILITY.md](https://github.com/felmonon/agent-reliability-harness/blob/main/COMPATIBILITY.md).

## Development

Expand All @@ -187,7 +189,7 @@ python benchmarks/run.py # thresholds enforced
```

CI runs the suite on Linux/macOS/Windows across Python 3.11-3.13, plus
packaging checks. Contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md).
packaging checks. Contributions welcome — see [CONTRIBUTING.md](https://github.com/felmonon/agent-reliability-harness/blob/main/CONTRIBUTING.md).

## Author

Expand Down
8 changes: 4 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
[build-system]
requires = ["setuptools>=68.0"]
requires = ["setuptools>=77.0"]
build-backend = "setuptools.build_meta"

[project]
name = "agent-reliability-harness"
version = "0.2.0"
version = "0.2.1"
description = "Local-first policy and trajectory-regression harness for tool-using AI agents: deterministic trace validation, policy-as-code trajectory rules, baseline-vs-candidate regression gates, and CI-ready JSON/Markdown/JUnit/SARIF reports."
readme = "README.md"
requires-python = ">=3.11"
license = { file = "LICENSE" }
license = "MIT"
license-files = ["LICENSE"]
authors = [{ name = "Felmon Fekadu" }]
keywords = [
"agents",
Expand All @@ -25,7 +26,6 @@ keywords = [
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
Expand Down
2 changes: 1 addition & 1 deletion src/agent_reliability_harness/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
from agent_reliability_harness.rules import RULES, Rule
from agent_reliability_harness.validator import validate_trace

__version__ = "0.2.0"
__version__ = "0.2.1"

__all__ = [
"ArgSpec",
Expand Down
Loading