Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,20 @@ adheres to [Semantic Versioning](https://semver.org/).

## [Unreleased]

## [0.2.2] - 2026-07-14

### Added

- New `cohere-chat` adapter: imports Cohere Chat API v2 message lists
(`tool_plan` / `tool_calls` assistant messages, `document`-block tool
results, message-level citations) into the canonical trace format, with
auto-detection. Cohere shares the `tool_calls` wire shape with OpenAI, so
detection scans for Cohere-only markers (`tool_plan`, assistant
message-level `citations`, `document` tool-result blocks) before the
OpenAI rules. An assistant `tool_plan` maps to a `model_response` step
marked with `metadata.source_field: "tool_plan"` so plan text stays
visible to safety checks without being mistaken for the final answer.

## [0.2.1] - 2026-07-14

### Fixed
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,9 @@ match - agents can legitimately reach a goal via different paths.
- **arh** - the canonical JSON trace format ([TRACE-SPEC.md](https://github.com/felmonon/agent-reliability-harness/blob/main/TRACE-SPEC.md));
- **openai-chat** - OpenAI Chat Completions message lists with `tool_calls`;
- **anthropic-messages** - Anthropic Messages conversations with
`tool_use`/`tool_result` blocks.
`tool_use`/`tool_result` blocks;
- **cohere-chat** - Cohere Chat API v2 message lists with
`tool_plan`/`tool_calls` and `document`-block tool results.

Adapters never guess: fields a transcript format cannot carry (latency, cost,
tokens) are left unset, which marks the dependent checks *not applicable*
Expand Down
41 changes: 36 additions & 5 deletions docs/adapters.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,23 @@ recorded under `trace.metadata.adapter.notes`.
Applied in order:

1. JSON object with a `steps` list → `arh` (canonical, no conversion).
2. Object with a `messages` list (or a bare message list):
2. Object with a `messages` list (or a bare message list) is first scanned in
full for Cohere-only markers → `cohere-chat`. Cohere shares the
`tool_calls` / `role: "tool"` wire shape with OpenAI, so this scan must
run before the per-message rules below. Markers: `tool_plan` on an
assistant message, an assistant message-level `citations` list, or
`document` typed tool-result content blocks.
3. Otherwise, message by message:
- any assistant message with `tool_calls` or `function_call`, or any
`role: "tool"` message → `openai-chat`;
- any message whose `content` is a block list containing `tool_use` /
`tool_result` → `anthropic-messages`.
3. Text-only message lists default to `openai-chat` (both vendors' text-only
transcripts normalize identically).
4. Anything else: `error: ... cannot detect trace format`.
4. Text-only message lists default to `openai-chat` (all three vendors'
text-only transcripts normalize identically).
5. Anything else: `error: ... cannot detect trace format`.

Force a format with `--format openai-chat` or `--format anthropic-messages`.
Force a format with `--format openai-chat`, `--format anthropic-messages`,
or `--format cohere-chat`.

## openai-chat

Expand Down Expand Up @@ -60,6 +67,30 @@ strings or typed block lists.
embed `usage`). It *does* carry tool errors via `is_error`, so
`error_handling` rules work on this format.

## cohere-chat

Input: a Cohere Chat API v2 message list (`{"messages": [...]}` or a bare
list), per [Cohere's tool use docs](https://docs.cohere.com/docs/tool-use-overview).

| Transcript element | Canonical mapping |
|---|---|
| assistant `tool_plan` | `model_response` step (`step_id` = `m<i>-plan`, `metadata.source_field` = `"tool_plan"`) — model-generated text stays visible to safety checks |
| assistant `tool_calls[i]` | `tool_call` step; `step_id` = call id; `function.arguments` JSON-decoded into `arguments` (same wire shape as OpenAI) |
| `role: "tool"` message | `output` of the matching step via `tool_call_id`; `document` content blocks are flattened to their `document.data` payloads joined in order |
| assistant text `content` (string or `text` blocks) | `model_response` step |
| assistant message-level `citations` | citations on that message's `model_response` step (passed through as-is) |
| `system` / `user` messages | skipped (agent inputs, not agent behavior) |

**Cannot carry** (left unset → dependent checks become *not applicable*):
`latency_ms`, `cost_usd`, token counts, and step `status` — the transcript
has no error channel for tool results, so status is always `"ok"`.

Edge handling: unparseable `function.arguments` → empty `arguments` plus an
`argument_parse_error` note; orphan tool results, non-object messages,
non-`document` tool content blocks, and `document` blocks without string
`data` → recorded as notes. Valid `document.data` payloads are still
recovered from a partially-malformed block list.

## Trace identity

Wrap transcripts to control identity — top-level keys pass through:
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "agent-reliability-harness"
version = "0.2.1"
version = "0.2.2"
description = "Local-first policy and trajectory-regression harness for tool-using AI agents: deterministic trace validation, policy-as-code trajectory rules, baseline-vs-candidate regression gates, and CI-ready JSON/Markdown/JUnit/SARIF reports."
readme = "README.md"
requires-python = ">=3.11"
Expand Down
44 changes: 39 additions & 5 deletions src/agent_reliability_harness/adapters/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@
``tool_calls`` / ``role: "tool"`` messages.
- ``anthropic-messages``: an Anthropic Messages API conversation with
``tool_use`` / ``tool_result`` content blocks.
- ``cohere-chat``: a Cohere Chat API v2 message list with ``tool_plan`` /
``tool_calls`` assistant messages and ``document``-block tool results.

Fields the source format does not carry (see docs/adapters.md for the full
support matrix) are left unset, which automatically marks the dependent
Expand All @@ -25,34 +27,63 @@
from collections.abc import Callable
from typing import Any

from agent_reliability_harness.adapters import anthropic_messages, openai_chat
from agent_reliability_harness.adapters import anthropic_messages, cohere_chat, openai_chat

FORMAT_ARH = "arh"
FORMAT_OPENAI_CHAT = "openai-chat"
FORMAT_ANTHROPIC_MESSAGES = "anthropic-messages"
FORMAT_COHERE_CHAT = "cohere-chat"
FORMAT_AUTO = "auto"

FORMATS = (FORMAT_ARH, FORMAT_OPENAI_CHAT, FORMAT_ANTHROPIC_MESSAGES)
FORMATS = (FORMAT_ARH, FORMAT_OPENAI_CHAT, FORMAT_ANTHROPIC_MESSAGES, FORMAT_COHERE_CHAT)

_ADAPTERS: dict[str, Callable[[Any, str], dict[str, Any]]] = {
FORMAT_OPENAI_CHAT: openai_chat.to_trace_dict,
FORMAT_ANTHROPIC_MESSAGES: anthropic_messages.to_trace_dict,
FORMAT_COHERE_CHAT: cohere_chat.to_trace_dict,
}


def _has_cohere_markers(message: dict[str, Any]) -> bool:
"""True when a message carries a field unique to Cohere Chat v2.

Cohere transcripts also use ``tool_calls`` and ``role: "tool"`` (the
OpenAI wire shape), so detection relies on the fields only Cohere
emits: ``tool_plan`` on assistant messages, message-level
``citations``, and ``document`` typed tool-result content blocks.
"""
if "tool_plan" in message:
return True
if message.get("role") == "assistant" and "citations" in message:
return True
if message.get("role") == "tool":
content = message.get("content")
if isinstance(content, list) and any(
isinstance(block, dict) and block.get("type") == "document"
for block in content
):
return True
return False


def detect_format(raw: Any) -> str:
"""Deterministically detect the trace format of a parsed JSON document.

Detection rules, applied in order:

1. An object with a ``steps`` list is the canonical ``arh`` format.
2. An object with a ``messages`` list (or a bare message list) is
inspected message by message:
2. An object with a ``messages`` list (or a bare message list) is first
scanned in full for Cohere-only markers (``tool_plan``, assistant
message-level ``citations``, ``document`` typed tool-result blocks)
-> ``cohere-chat``. Cohere shares the ``tool_calls`` / ``role:
"tool"`` wire shape with OpenAI, so this scan must run before the
per-message rules below.
3. Otherwise the messages are inspected message by message:
a. any ``role: "assistant"`` message with ``tool_calls`` or
``function_call``, or any ``role: "tool"`` message -> ``openai-chat``;
b. any message whose ``content`` is a list containing ``tool_use`` /
``tool_result`` typed blocks -> ``anthropic-messages``.
3. A plain-text-only message list defaults to ``openai-chat`` (both
4. A plain-text-only message list defaults to ``openai-chat`` (all three
vendors' text-only transcripts are structurally identical; the
resulting canonical trace is the same either way).

Expand All @@ -70,6 +101,9 @@ def detect_format(raw: Any) -> str:
"cannot detect trace format: expected an object with 'steps' (arh), "
"an object with 'messages', or a bare message list"
)
for message in messages:
if isinstance(message, dict) and _has_cohere_markers(message):
return FORMAT_COHERE_CHAT
for message in messages:
if not isinstance(message, dict):
continue
Expand Down
Loading
Loading