diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d0b179..6535472 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,20 @@ adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] +## [0.2.2] - 2026-07-14 + +### Added + +- New `cohere-chat` adapter: imports Cohere Chat API v2 message lists + (`tool_plan` / `tool_calls` assistant messages, `document`-block tool + results, message-level citations) into the canonical trace format, with + auto-detection. Cohere shares the `tool_calls` wire shape with OpenAI, so + detection scans for Cohere-only markers (`tool_plan`, assistant + message-level `citations`, `document` tool-result blocks) before the + OpenAI rules. An assistant `tool_plan` maps to a `model_response` step + marked with `metadata.source_field: "tool_plan"` so plan text stays + visible to safety checks without being mistaken for the final answer. + ## [0.2.1] - 2026-07-14 ### Fixed diff --git a/README.md b/README.md index b888065..5391ff8 100644 --- a/README.md +++ b/README.md @@ -122,7 +122,9 @@ match - agents can legitimately reach a goal via different paths. - **arh** - the canonical JSON trace format ([TRACE-SPEC.md](https://github.com/felmonon/agent-reliability-harness/blob/main/TRACE-SPEC.md)); - **openai-chat** - OpenAI Chat Completions message lists with `tool_calls`; - **anthropic-messages** - Anthropic Messages conversations with - `tool_use`/`tool_result` blocks. + `tool_use`/`tool_result` blocks; +- **cohere-chat** - Cohere Chat API v2 message lists with + `tool_plan`/`tool_calls` and `document`-block tool results. Adapters never guess: fields a transcript format cannot carry (latency, cost, tokens) are left unset, which marks the dependent checks *not applicable* diff --git a/docs/adapters.md b/docs/adapters.md index da83ba1..b6252bf 100644 --- a/docs/adapters.md +++ b/docs/adapters.md @@ -10,16 +10,23 @@ recorded under `trace.metadata.adapter.notes`. Applied in order: 1. JSON object with a `steps` list → `arh` (canonical, no conversion). -2. Object with a `messages` list (or a bare message list): +2. Object with a `messages` list (or a bare message list) is first scanned in + full for Cohere-only markers → `cohere-chat`. Cohere shares the + `tool_calls` / `role: "tool"` wire shape with OpenAI, so this scan must + run before the per-message rules below. Markers: `tool_plan` on an + assistant message, an assistant message-level `citations` list, or + `document` typed tool-result content blocks. +3. Otherwise, message by message: - any assistant message with `tool_calls` or `function_call`, or any `role: "tool"` message → `openai-chat`; - any message whose `content` is a block list containing `tool_use` / `tool_result` → `anthropic-messages`. -3. Text-only message lists default to `openai-chat` (both vendors' text-only - transcripts normalize identically). -4. Anything else: `error: ... cannot detect trace format`. +4. Text-only message lists default to `openai-chat` (all three vendors' + text-only transcripts normalize identically). +5. Anything else: `error: ... cannot detect trace format`. -Force a format with `--format openai-chat` or `--format anthropic-messages`. +Force a format with `--format openai-chat`, `--format anthropic-messages`, +or `--format cohere-chat`. ## openai-chat @@ -60,6 +67,30 @@ strings or typed block lists. embed `usage`). It *does* carry tool errors via `is_error`, so `error_handling` rules work on this format. +## cohere-chat + +Input: a Cohere Chat API v2 message list (`{"messages": [...]}` or a bare +list), per [Cohere's tool use docs](https://docs.cohere.com/docs/tool-use-overview). + +| Transcript element | Canonical mapping | +|---|---| +| assistant `tool_plan` | `model_response` step (`step_id` = `m-plan`, `metadata.source_field` = `"tool_plan"`) — model-generated text stays visible to safety checks | +| assistant `tool_calls[i]` | `tool_call` step; `step_id` = call id; `function.arguments` JSON-decoded into `arguments` (same wire shape as OpenAI) | +| `role: "tool"` message | `output` of the matching step via `tool_call_id`; `document` content blocks are flattened to their `document.data` payloads joined in order | +| assistant text `content` (string or `text` blocks) | `model_response` step | +| assistant message-level `citations` | citations on that message's `model_response` step (passed through as-is) | +| `system` / `user` messages | skipped (agent inputs, not agent behavior) | + +**Cannot carry** (left unset → dependent checks become *not applicable*): +`latency_ms`, `cost_usd`, token counts, and step `status` — the transcript +has no error channel for tool results, so status is always `"ok"`. + +Edge handling: unparseable `function.arguments` → empty `arguments` plus an +`argument_parse_error` note; orphan tool results, non-object messages, +non-`document` tool content blocks, and `document` blocks without string +`data` → recorded as notes. Valid `document.data` payloads are still +recovered from a partially-malformed block list. + ## Trace identity Wrap transcripts to control identity — top-level keys pass through: diff --git a/pyproject.toml b/pyproject.toml index 829a04f..259ed3d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "agent-reliability-harness" -version = "0.2.1" +version = "0.2.2" description = "Local-first policy and trajectory-regression harness for tool-using AI agents: deterministic trace validation, policy-as-code trajectory rules, baseline-vs-candidate regression gates, and CI-ready JSON/Markdown/JUnit/SARIF reports." readme = "README.md" requires-python = ">=3.11" diff --git a/src/agent_reliability_harness/adapters/__init__.py b/src/agent_reliability_harness/adapters/__init__.py index 3b87c91..73716b9 100644 --- a/src/agent_reliability_harness/adapters/__init__.py +++ b/src/agent_reliability_harness/adapters/__init__.py @@ -13,6 +13,8 @@ ``tool_calls`` / ``role: "tool"`` messages. - ``anthropic-messages``: an Anthropic Messages API conversation with ``tool_use`` / ``tool_result`` content blocks. +- ``cohere-chat``: a Cohere Chat API v2 message list with ``tool_plan`` / + ``tool_calls`` assistant messages and ``document``-block tool results. Fields the source format does not carry (see docs/adapters.md for the full support matrix) are left unset, which automatically marks the dependent @@ -25,34 +27,63 @@ from collections.abc import Callable from typing import Any -from agent_reliability_harness.adapters import anthropic_messages, openai_chat +from agent_reliability_harness.adapters import anthropic_messages, cohere_chat, openai_chat FORMAT_ARH = "arh" FORMAT_OPENAI_CHAT = "openai-chat" FORMAT_ANTHROPIC_MESSAGES = "anthropic-messages" +FORMAT_COHERE_CHAT = "cohere-chat" FORMAT_AUTO = "auto" -FORMATS = (FORMAT_ARH, FORMAT_OPENAI_CHAT, FORMAT_ANTHROPIC_MESSAGES) +FORMATS = (FORMAT_ARH, FORMAT_OPENAI_CHAT, FORMAT_ANTHROPIC_MESSAGES, FORMAT_COHERE_CHAT) _ADAPTERS: dict[str, Callable[[Any, str], dict[str, Any]]] = { FORMAT_OPENAI_CHAT: openai_chat.to_trace_dict, FORMAT_ANTHROPIC_MESSAGES: anthropic_messages.to_trace_dict, + FORMAT_COHERE_CHAT: cohere_chat.to_trace_dict, } +def _has_cohere_markers(message: dict[str, Any]) -> bool: + """True when a message carries a field unique to Cohere Chat v2. + + Cohere transcripts also use ``tool_calls`` and ``role: "tool"`` (the + OpenAI wire shape), so detection relies on the fields only Cohere + emits: ``tool_plan`` on assistant messages, message-level + ``citations``, and ``document`` typed tool-result content blocks. + """ + if "tool_plan" in message: + return True + if message.get("role") == "assistant" and "citations" in message: + return True + if message.get("role") == "tool": + content = message.get("content") + if isinstance(content, list) and any( + isinstance(block, dict) and block.get("type") == "document" + for block in content + ): + return True + return False + + def detect_format(raw: Any) -> str: """Deterministically detect the trace format of a parsed JSON document. Detection rules, applied in order: 1. An object with a ``steps`` list is the canonical ``arh`` format. - 2. An object with a ``messages`` list (or a bare message list) is - inspected message by message: + 2. An object with a ``messages`` list (or a bare message list) is first + scanned in full for Cohere-only markers (``tool_plan``, assistant + message-level ``citations``, ``document`` typed tool-result blocks) + -> ``cohere-chat``. Cohere shares the ``tool_calls`` / ``role: + "tool"`` wire shape with OpenAI, so this scan must run before the + per-message rules below. + 3. Otherwise the messages are inspected message by message: a. any ``role: "assistant"`` message with ``tool_calls`` or ``function_call``, or any ``role: "tool"`` message -> ``openai-chat``; b. any message whose ``content`` is a list containing ``tool_use`` / ``tool_result`` typed blocks -> ``anthropic-messages``. - 3. A plain-text-only message list defaults to ``openai-chat`` (both + 4. A plain-text-only message list defaults to ``openai-chat`` (all three vendors' text-only transcripts are structurally identical; the resulting canonical trace is the same either way). @@ -70,6 +101,9 @@ def detect_format(raw: Any) -> str: "cannot detect trace format: expected an object with 'steps' (arh), " "an object with 'messages', or a bare message list" ) + for message in messages: + if isinstance(message, dict) and _has_cohere_markers(message): + return FORMAT_COHERE_CHAT for message in messages: if not isinstance(message, dict): continue diff --git a/src/agent_reliability_harness/adapters/cohere_chat.py b/src/agent_reliability_harness/adapters/cohere_chat.py new file mode 100644 index 0000000..1b2a296 --- /dev/null +++ b/src/agent_reliability_harness/adapters/cohere_chat.py @@ -0,0 +1,220 @@ +"""Adapter for Cohere Chat API v2 message lists. + +Maps a Cohere v2 conversation transcript (https://docs.cohere.com/reference/chat) +into a canonical ARH trace: + +- an assistant message's ``tool_plan`` becomes a ``model_response`` step + (it is model-generated text and must be visible to safety checks; the + step's ``metadata.source_field`` marks it as a plan, not a final answer); +- each entry of an assistant message's ``tool_calls`` becomes a + ``tool_call`` step (arguments JSON-decoded from ``function.arguments``, + the same wire shape as OpenAI tool calls); +- ``role: "tool"`` messages attach their content as the ``output`` of the + matching tool_call step (via ``tool_call_id``); Cohere ``document`` + content blocks are flattened to their ``document.data`` payloads; +- assistant text ``content`` (a string or ``text`` typed blocks) becomes a + ``model_response`` step; a message-level ``citations`` list (Cohere's + built-in citation feature) maps to that step's citations; +- ``system`` and ``user`` messages are skipped: they are inputs to the + agent, not agent behavior. + +Not carried by this format (documented, not guessed): latency, cost, token +usage per message, and step status (the transcript has no error channel for +tool results, so ``status`` is always ``"ok"``). Unparseable +``function.arguments``, orphan tool results, and malformed content blocks +are recorded under ``metadata.adapter`` instead of being dropped. +""" + +from __future__ import annotations + +import json +from typing import Any + +SOURCE = "cohere-chat" + + +def _tool_call_step( + call: dict[str, Any], step_id: str, adapter_notes: list[dict[str, Any]] +) -> dict[str, Any]: + function = call.get("function") or {} + name = function.get("name") or call.get("name") + raw_arguments = function.get("arguments", call.get("arguments")) + arguments: dict[str, Any] = {} + if isinstance(raw_arguments, dict): + arguments = raw_arguments + elif isinstance(raw_arguments, str) and raw_arguments.strip(): + try: + parsed = json.loads(raw_arguments) + if isinstance(parsed, dict): + arguments = parsed + else: + adapter_notes.append( + { + "step_id": step_id, + "issue": "arguments_not_object", + "raw_arguments": raw_arguments, + } + ) + except json.JSONDecodeError as exc: + adapter_notes.append( + { + "step_id": step_id, + "issue": "argument_parse_error", + "error": str(exc), + "raw_arguments": raw_arguments, + } + ) + return { + "step_id": step_id, + "type": "tool_call", + "tool_name": name, + "arguments": arguments, + } + + +def _tool_result_value( + content: Any, message_index: int, adapter_notes: list[dict[str, Any]] +) -> Any: + """Flatten a Cohere tool message content payload to a comparable value. + + Cohere v2 tool results are either a plain string or a list of + ``{"type": "document", "document": {"data": ...}}`` blocks. Document + ``data`` payloads are joined in order; malformed blocks are recorded. + """ + if not isinstance(content, list): + return content + parts: list[str] = [] + for block_index, block in enumerate(content): + if not isinstance(block, dict) or block.get("type") != "document": + adapter_notes.append( + { + "issue": "unrecognized_tool_content_block", + "message_index": message_index, + "block_index": block_index, + } + ) + continue + document = block.get("document") + data = document.get("data") if isinstance(document, dict) else None + if isinstance(data, str): + parts.append(data) + else: + adapter_notes.append( + { + "issue": "document_block_missing_data", + "message_index": message_index, + "block_index": block_index, + } + ) + if parts: + return "\n".join(parts) + return content + + +def _assistant_text(content: Any) -> str | None: + """Extract assistant text from a string or ``text`` typed block list.""" + if isinstance(content, str): + return content if content.strip() else None + if isinstance(content, list): + texts = [ + block.get("text", "") + for block in content + if isinstance(block, dict) and block.get("type") == "text" + ] + joined = "\n".join(text for text in texts if isinstance(text, str) and text.strip()) + return joined if joined else None + return None + + +def to_trace_dict(raw: Any, fallback_trace_id: str) -> dict[str, Any]: + if isinstance(raw, list): + wrapper: dict[str, Any] = {"messages": raw} + elif isinstance(raw, dict): + wrapper = raw + else: + raise ValueError( + f"cohere-chat input must be an object with 'messages' or a message list, " + f"got {type(raw).__name__}" + ) + messages = wrapper.get("messages") + if not isinstance(messages, list): + raise ValueError("cohere-chat input has no 'messages' list") + + steps: list[dict[str, Any]] = [] + step_by_call_id: dict[str, dict[str, Any]] = {} + adapter_notes: list[dict[str, Any]] = [] + + for index, message in enumerate(messages): + if not isinstance(message, dict): + adapter_notes.append({"issue": "non_object_message", "message_index": index}) + continue + role = message.get("role") + if role == "assistant": + tool_plan = message.get("tool_plan") + if isinstance(tool_plan, str) and tool_plan.strip(): + steps.append( + { + "step_id": f"m{index}-plan", + "type": "model_response", + "text": tool_plan, + "metadata": {"source_field": "tool_plan"}, + } + ) + tool_calls = message.get("tool_calls") + if isinstance(tool_calls, list): + for call_index, call in enumerate(tool_calls): + if not isinstance(call, dict): + adapter_notes.append( + { + "issue": "non_object_tool_call", + "message_index": index, + "tool_call_index": call_index, + } + ) + continue + step_id = str(call.get("id") or f"m{index}-tool{call_index}") + step = _tool_call_step(call, step_id, adapter_notes) + steps.append(step) + step_by_call_id[step_id] = step + text = _assistant_text(message.get("content")) + if text is not None: + step = { + "step_id": f"m{index}-response", + "type": "model_response", + "text": text, + } + citations = message.get("citations") + if isinstance(citations, list) and citations: + step["citations"] = [c for c in citations if isinstance(c, dict)] + steps.append(step) + elif role == "tool": + call_id = message.get("tool_call_id") + target = step_by_call_id.get(str(call_id)) if call_id is not None else None + if target is None: + adapter_notes.append( + { + "issue": "unmatched_tool_result", + "message_index": index, + "tool_call_id": call_id, + } + ) + else: + target["output"] = _tool_result_value( + message.get("content"), index, adapter_notes + ) + # system/user messages: agent inputs, intentionally skipped. + + trace: dict[str, Any] = { + "schema_version": "1", + "trace_id": str(wrapper.get("trace_id") or fallback_trace_id), + "agent_name": str(wrapper.get("agent_name") or "cohere-chat-agent"), + "workflow": str(wrapper.get("workflow") or "cohere-chat-import"), + "source": SOURCE, + "steps": steps, + } + metadata = dict(wrapper.get("metadata") or {}) + if adapter_notes: + metadata.setdefault("adapter", {})["notes"] = adapter_notes + if metadata: + trace["metadata"] = metadata + return trace diff --git a/src/agent_reliability_harness/cli.py b/src/agent_reliability_harness/cli.py index fad4926..e4d9195 100644 --- a/src/agent_reliability_harness/cli.py +++ b/src/agent_reliability_harness/cli.py @@ -118,8 +118,8 @@ def _build_parser() -> argparse.ArgumentParser: default=FORMAT_AUTO, help=( "Trace input format. 'auto' (default) detects between the canonical " - "'arh' format, 'openai-chat' message lists, and 'anthropic-messages' " - "conversations." + "'arh' format, 'openai-chat' message lists, 'anthropic-messages' " + "conversations, and 'cohere-chat' message lists." ), ) validate_parser.add_argument( diff --git a/tests/fixtures/cohere_chat_edge_cases.json b/tests/fixtures/cohere_chat_edge_cases.json new file mode 100644 index 0000000..6abc4f1 --- /dev/null +++ b/tests/fixtures/cohere_chat_edge_cases.json @@ -0,0 +1,29 @@ +{ + "trace_id": "cohere-edge-001", + "messages": [ + "not a message object", + { + "role": "assistant", + "tool_plan": "I will call a tool with broken arguments.", + "tool_calls": [ + { + "id": "call_bad", + "type": "function", + "function": {"name": "broken_tool", "arguments": "{not valid json"} + }, + "not a tool call object" + ] + }, + {"role": "tool", "tool_call_id": "call_missing", "content": "orphan result"}, + { + "role": "tool", + "tool_call_id": "call_bad", + "content": [ + {"type": "document", "document": {"data": "first part"}}, + {"type": "image", "url": "https://example.com/x.png"}, + {"type": "document", "document": {}} + ] + }, + {"role": "assistant", "content": "Done, I think."} + ] +} diff --git a/tests/fixtures/cohere_chat_refund.json b/tests/fixtures/cohere_chat_refund.json new file mode 100644 index 0000000..95a932e --- /dev/null +++ b/tests/fixtures/cohere_chat_refund.json @@ -0,0 +1,58 @@ +{ + "trace_id": "cohere-refund-001", + "agent_name": "refund-bot", + "workflow": "refund", + "messages": [ + {"role": "system", "content": "You are a refund agent."}, + {"role": "user", "content": "Please refund order ORD-9."}, + { + "role": "assistant", + "tool_plan": "I will look up order ORD-9 before issuing a refund.", + "tool_calls": [ + { + "id": "lookup_order_h8f2k1", + "type": "function", + "function": {"name": "lookup_order", "arguments": "{\"order_id\": \"ORD-9\"}"} + } + ] + }, + { + "role": "tool", + "tool_call_id": "lookup_order_h8f2k1", + "content": [ + {"type": "document", "document": {"data": "{\"status\": \"delivered\", \"amount\": 42.5}"}} + ] + }, + { + "role": "assistant", + "tool_plan": "The order is delivered; I will issue the refund of 42.5.", + "tool_calls": [ + { + "id": "issue_refund_p3m9x7", + "type": "function", + "function": {"name": "issue_refund", "arguments": "{\"order_id\": \"ORD-9\", \"amount\": 42.5}"} + } + ] + }, + {"role": "tool", "tool_call_id": "issue_refund_p3m9x7", "content": "refund issued"}, + { + "role": "assistant", + "content": [{"type": "text", "text": "Your refund of $42.50 for ORD-9 has been issued."}], + "citations": [ + { + "start": 5, + "end": 21, + "text": "refund of $42.50", + "type": "TEXT_CONTENT", + "sources": [ + { + "type": "tool", + "id": "issue_refund_p3m9x7:0", + "tool_output": {"data": "refund issued"} + } + ] + } + ] + } + ] +} diff --git a/tests/test_adapters.py b/tests/test_adapters.py index 0db038f..a25d04e 100644 --- a/tests/test_adapters.py +++ b/tests/test_adapters.py @@ -7,6 +7,7 @@ from agent_reliability_harness.adapters import ( FORMAT_ANTHROPIC_MESSAGES, FORMAT_ARH, + FORMAT_COHERE_CHAT, FORMAT_OPENAI_CHAT, detect_format, normalize, @@ -39,6 +40,36 @@ def test_detects_bare_list_openai(self): raw = [{"role": "assistant", "tool_calls": []}] self.assertEqual(detect_format(raw), FORMAT_OPENAI_CHAT) + def test_detects_cohere_chat(self): + self.assertEqual(detect_format(load("cohere_chat_refund.json")), FORMAT_COHERE_CHAT) + + def test_tool_plan_beats_shared_openai_wire_shape(self): + """Cohere shares tool_calls/role:'tool' with OpenAI; tool_plan must win.""" + raw = { + "messages": [ + {"role": "assistant", "tool_plan": "plan", "tool_calls": []}, + ] + } + self.assertEqual(detect_format(raw), FORMAT_COHERE_CHAT) + + def test_document_blocks_detected_as_cohere_without_tool_plan(self): + raw = { + "messages": [ + { + "role": "assistant", + "tool_calls": [ + {"id": "c1", "type": "function", "function": {"name": "t"}} + ], + }, + { + "role": "tool", + "tool_call_id": "c1", + "content": [{"type": "document", "document": {"data": "x"}}], + }, + ] + } + self.assertEqual(detect_format(raw), FORMAT_COHERE_CHAT) + def test_text_only_defaults_to_openai(self): raw = {"messages": [{"role": "user", "content": "hi"}]} self.assertEqual(detect_format(raw), FORMAT_OPENAI_CHAT) @@ -203,6 +234,133 @@ def test_rejects_garbage(self): normalize("nope", "anthropic-messages", "fb") +class TestCohereChatAdapter(unittest.TestCase): + def test_refund_transcript_maps_fully(self): + trace_dict = normalize(load("cohere_chat_refund.json"), "cohere-chat", "fb") + trace = Trace.from_dict(trace_dict) + self.assertEqual(trace.trace_id, "cohere-refund-001") + self.assertEqual(trace.source, "cohere-chat") + types = [(s.type, s.tool_name) for s in trace.steps] + self.assertEqual( + types, + [ + ("model_response", None), + ("tool_call", "lookup_order"), + ("model_response", None), + ("tool_call", "issue_refund"), + ("model_response", None), + ], + ) + lookup = [s for s in trace.steps if s.tool_name == "lookup_order"][0] + self.assertEqual(lookup.step_id, "lookup_order_h8f2k1") + self.assertEqual(lookup.arguments, {"order_id": "ORD-9"}) + # document block payload is flattened to its data string + self.assertEqual(lookup.output, '{"status": "delivered", "amount": 42.5}') + refund = [s for s in trace.steps if s.tool_name == "issue_refund"][0] + self.assertEqual(refund.arguments, {"order_id": "ORD-9", "amount": 42.5}) + self.assertEqual(refund.output, "refund issued") + + def test_tool_plan_becomes_marked_model_response(self): + trace_dict = normalize(load("cohere_chat_refund.json"), "cohere-chat", "fb") + trace = Trace.from_dict(trace_dict) + plans = [s for s in trace.steps if s.step_id.endswith("-plan")] + self.assertEqual(len(plans), 2) + for plan in plans: + self.assertEqual(plan.type, "model_response") + self.assertEqual(plan.metadata.get("source_field"), "tool_plan") + self.assertIn("look up order ORD-9", plans[0].text) + + def test_message_level_citations_map_to_final_response(self): + trace_dict = normalize(load("cohere_chat_refund.json"), "cohere-chat", "fb") + trace = Trace.from_dict(trace_dict) + final = trace.steps[-1] + self.assertEqual(final.type, "model_response") + self.assertIn("refund of $42.50", final.text) + self.assertEqual(len(final.citations), 1) + self.assertEqual(final.citations[0]["text"], "refund of $42.50") + + def test_edge_cases_recorded_not_dropped(self): + trace_dict = normalize(load("cohere_chat_edge_cases.json"), "cohere-chat", "fb") + trace = Trace.from_dict(trace_dict) + notes = trace.metadata["adapter"]["notes"] + issues = {n["issue"] for n in notes} + self.assertIn("argument_parse_error", issues) + self.assertIn("unmatched_tool_result", issues) + self.assertIn("non_object_message", issues) + self.assertIn("non_object_tool_call", issues) + self.assertIn("unrecognized_tool_content_block", issues) + self.assertIn("document_block_missing_data", issues) + # unparseable arguments became empty dict (missing-arg findings will fire) + bad = [s for s in trace.steps if s.step_id == "call_bad"][0] + self.assertEqual(bad.arguments, {}) + # valid document data is still recovered from a partially-bad block list + self.assertEqual(bad.output, "first part") + # plain-string final assistant content still maps + self.assertEqual(trace.steps[-1].type, "model_response") + self.assertIn("Done", trace.steps[-1].text) + + def test_round_trip_validation(self): + """An adapter-produced trace must be fully validatable.""" + trace_dict = normalize(load("cohere_chat_refund.json"), "cohere-chat", "fb") + policy = Policy.from_dict( + { + "policy_id": "refund-policy", + "allowed_tools": { + "lookup_order": {"required_arguments": {"order_id": "str"}}, + "issue_refund": { + "required_arguments": {"order_id": "str", "amount": "float"}, + "side_effect": True, + }, + }, + "sequence": {"call_order": ["lookup_order", "issue_refund"]}, + "completion": {"require_final_response": True}, + } + ) + report = validate_trace(Trace.from_dict(trace_dict), policy) + self.assertEqual([f for f in report.findings if f.severity == "error"], []) + self.assertTrue(report.passed) + + def test_equivalent_transcripts_produce_equivalent_verdicts(self): + """The same refund conversation via Cohere matches the OpenAI verdict.""" + policy = Policy.from_dict( + { + "policy_id": "refund-policy", + "allowed_tools": { + "lookup_order": {"required_arguments": {"order_id": "str"}}, + "issue_refund": { + "required_arguments": {"order_id": "str", "amount": "float"}, + "side_effect": True, + }, + }, + "sequence": {"call_order": ["lookup_order", "issue_refund"]}, + "completion": {"require_final_response": True}, + } + ) + openai_report = validate_trace( + Trace.from_dict(normalize(load("openai_chat_refund.json"), "openai-chat", "fb")), + policy, + ) + cohere_report = validate_trace( + Trace.from_dict(normalize(load("cohere_chat_refund.json"), "cohere-chat", "fb")), + policy, + ) + self.assertEqual(openai_report.passed, cohere_report.passed) + self.assertEqual( + sorted(f.rule_id for f in openai_report.findings), + sorted(f.rule_id for f in cohere_report.findings), + ) + + def test_fallback_trace_id_used(self): + trace_dict = normalize({"messages": []}, "cohere-chat", "my_file") + self.assertEqual(trace_dict["trace_id"], "my_file") + + def test_rejects_garbage(self): + with self.assertRaises(ValueError): + normalize(42, "cohere-chat", "fb") + with self.assertRaises(ValueError): + normalize({"nope": True}, "cohere-chat", "fb") + + class TestNormalizeDispatch(unittest.TestCase): def test_auto_dispatches(self): trace_dict = normalize(load("openai_chat_refund.json"), "auto", "fb")