Protocol review request: AANA pre-tool-call safety/reliability track

[mindbomber]

Hi MCP-Bench maintainers,

I am working on AANA, an agent-action reliability layer that gates tool calls before execution using a structured contract plus verifier/correction/evidence-recovery logic. I would like to ask whether this protocol would be useful or acceptable as an auxiliary safety/reliability track for MCP-Bench, or whether you would prefer a different submission path.

Protocol for review:
https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/harmactions-aana-benchmark/docs/aana-benchmark-maintainer-review-protocol.md

Related schema and implementation:

• Schema: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/harmactions-aana-benchmark/schemas/agent_tool_precheck.schema.json
• Python pre-tool-call gate: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/harmactions-aana-benchmark/eval_pipeline/pre_tool_call_gate.py
• Public findings summary: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/harmactions-aana-benchmark/docs/aana-head-to-head-findings.md

Short version of the protocol:

• Normalize each proposed tool call before execution into tool_name, tool_category, authorization_state, evidence_refs, risk_domain, proposed_arguments, and recommended_route.
• Hide benchmark labels from all systems.
• Compare AANA against: permissive execution, a single classifier, prompt-only policy guardrail, LLM-as-judge, and a structured contract gate without correction/evidence recovery.
• Score unsafe-action recall, false positives, safe allow rate, route accuracy, and evidence recovery quality.

Current evidence is intentionally framed as preliminary. We have run this on two independently sourced public trace datasets, but the labels/noise are script-generated, not MCP-Bench-maintainer labels or human-reviewed labels. I am asking for protocol review before treating any AANA result as official evidence.

Questions for you:

1. Would this make sense as an auxiliary MCP-Bench safety/reliability evaluation track?
2. Are there existing MCP-Bench traces, task fields, or output artifacts that would support this pre-tool-call contract cleanly?
3. Would you prefer a PR that adds an evaluator/adapter, a standalone results submission artifact, or continued discussion here first?
4. Which metric would you consider most benchmark-native: unsafe-action recall, safe allow rate, route accuracy, or another MCP-Bench score?

If this is out of scope for MCP-Bench, a pointer to the right evaluation/leaderboard venue would also be useful.

[mindbomber]

Follow-up from the AANA side:

I opened an AANA PR with the first benchmark-targeted artifact and a concrete MCP-Bench submission plan:

• AANA PR: Add AANA MSB protocol benchmark artifact mindbomber/Alignment-Aware-Neural-Architecture--AANA-#6
• MCP-Bench plan: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/aana-msb-mcp-bench-submissions/docs/aana-mcp-bench-submission-plan.md
• Related MSB protocol artifact: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/aana-msb-mcp-bench-submissions/docs/aana-msb-mcp-security-bench.md

The updated position is stricter than the original protocol request: for MCP-Bench, AANA should not be reported as a standalone model row or from static task-file conversion. The appropriate submission is a paired run:

1. base MCP-Bench agent/model alone,
2. same base agent/model with AANA inserted before actual MCP tool execution.

We would then report normal MCP-Bench metrics plus AANA side metrics: blocked-tool non-execution, route parity, audit-log completeness, schema failure rate, latency, missing evidence, and authorization-state routing.

If this direction is acceptable, the next implementation step is a small wrapper around the MCP-Bench tool execution path so every candidate MCP call is normalized into Agent Action Contract v1 before execution, with enforcement-mode semantics where only �ccept can execute.

[mindbomber]

Implementation update: the paired MCP-Bench wrapper is now in the AANA PR.

• PR: Add AANA MSB protocol benchmark artifact mindbomber/Alignment-Aware-Neural-Architecture--AANA-#6
• Wrapper module: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/aana-msb-mcp-bench-submissions/aana/integrations/mcp_bench.py
• Paired runner: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/aana-msb-mcp-bench-submissions/scripts/run_mcp_bench_aana_ablation.py
• Updated submission plan: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/aana-msb-mcp-bench-submissions/docs/aana-mcp-bench-submission-plan.md

The wrapper patches the actual MCP-Bench execution boundary by wrapping PersistentMultiServerManager.call_tool(...). The runner executes the same model/task setup once plain and once with AANA inserted before tool execution. In enforcement mode, only AANA �ccept routes can execute; blocked calls return a minimal MCP-like error result and write an audit-safe JSONL record with route, hard blockers, argument keys/hash, auth state, and latency.

Local validation includes a fake MCP-Bench smoke where the plain condition executes both tools and the AANA condition executes the safe public read while blocking an unsafe write without logging raw arguments. The next step is a real smoke run against a local MCP-Bench checkout, then a full paired run if the smoke passes.

[mindbomber]

Execution update: I completed local paired MCP-Bench smoke runs with the wrapper.

Setup:

• Shallow local checkout of Accenture/mcp-bench at 7a8eaea
• Model: o4-mini
• Task set: official Unit Converter single-server tasks
• Distraction servers: disabled
• AANA condition: patched real PersistentMultiServerManager.call_tool(...) boundary before MCP tool execution

One-task smoke:

• base task_success_rate: 1.000
• base + AANA task_success_rate: 1.000
• base + AANA audit decisions: 12 accept, 0 blocked
• valid tool name / schema compliance / tool success: 1.000 in both conditions

Two-task Unit Converter run:

• base task_success_rate: 1.000
• base + AANA task_success_rate: 1.000
• base + AANA audit decisions: 65 accept, 0 blocked
• valid tool name / schema compliance / tool success: 1.000 in both conditions

Interpretation:

• The wrapper reaches the real MCP tool-execution boundary and emits audit records for every checked call.
• AANA preserved safe utility-tool execution in these tasks.
• These Unit Converter tasks are safe utility conversions, so they are not evidence of unsafe-action prevention.
• The next meaningful MCP-Bench/AANA run should include consequential write/private-read tasks or an injected unsafe-action track, where blocked-tool non-execution and authorization/evidence controls can be measured.

The AANA PR records these smoke results here: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/aana-msb-mcp-bench-submissions/docs/aana-mcp-bench-submission-plan.md

[mindbomber]

Follow-up execution update from the AANA side.

AANA PR / branch:

• PR: Add AANA MSB protocol benchmark artifact mindbomber/Alignment-Aware-Neural-Architecture--AANA-#6
• Branch: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/tree/codex/aana-msb-mcp-bench-submissions
• Latest commit used for this update: 1d68a37
• Submission notes: https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/codex/aana-msb-mcp-bench-submissions/docs/aana-mcp-bench-submission-plan.md

I ran two additional local diagnostics after the earlier Unit Converter smoke runs.

1. Clean-env focused 6-task MCP-Bench slice

Setup:

• Dedicated clean virtualenv for the MCP-Bench/AANA run
• Local Accenture/mcp-bench checkout at 7a8eaea
• Model: o4-mini
• Official MCP-Bench single-server tasks from NixOS, Scientific Computing, and Medical Calculator
• Distraction servers disabled
• AANA inserted before the real PersistentMultiServerManager.call_tool(...) execution boundary

Results:

• Base task_success_rate: 1.000
• Base + AANA task_success_rate: 1.000
• Base task completion score: 7.367
• Base + AANA task completion score: 7.117
• Base tool selection score: 5.750
• Base + AANA tool selection score: 6.250
• Base planning effectiveness score: 5.800
• Base + AANA planning effectiveness score: 6.167
• Base input schema compliance / tool-call success: 1.000 / 1.000
• Base + AANA input schema compliance / tool-call success: 0.976 / 0.976
• AANA audit decisions: 81 accept, 0 blocked
• AANA decision latency: p50 2.778 ms, p95 7.911 ms

Interpretation: this clean-env slice is useful evidence that the wrapper works at the MCP tool boundary, preserves execution on these tasks, emits audit records, and adds low decision latency. It is not evidence of unsafe-action prevention because all attempted calls were accepted.

2. MCP-Bench-style unsafe/private/write injection slice

I then added a diagnostic injection slice using the same AANA MCP-Bench guard around a fake MCP execution boundary. This is not an official MCP-Bench task set or leaderboard result; it is a control-layer test for blocked-tool non-execution.

Cases:

• 2 safe public reads
• 2 unauthorized private reads
• 4 unsafe/unconfirmed writes

Results:

• Plain permissive condition executed all 8/8 tool proposals, including all 6 unsafe/private/write calls.
• AANA condition executed only the 2 safe public reads.
• Plain unsafe executions: 6
• AANA unsafe executions: 0
• Blocked-tool non-execution rate: 1.000
• Safe public-read allow rate: 1.000
• AANA routes: 2 accept, 4 ask, 2 defer
• AANA latency: p50 2.316 ms, p95 2.831 ms

This injection slice also caught and fixed an over-blocking bug: a public documentation search for refund policy was initially misclassified as a write because the wrapper considered consequential words in query arguments. The wrapper now prioritizes the MCP tool surface and description before treating argument text as write intent, so public search/read tools remain public reads when their query mentions a consequential topic.

Clear limitation / request:

I do not think these should be presented as an official MCP-Bench leaderboard result. The current leaderboard appears model/task-performance oriented, while AANA is being evaluated as a runtime control layer around a base agent:

base agent proposes -> AANA checks -> tool executes only if AANA returns accept

Would MCP-Bench maintainers be open to a control-layer / wrapper submission track, where the same base model is run plain vs. with a wrapper, and results report both MCP-Bench metrics and control-layer side metrics such as blocked-tool non-execution, safe public-read allow rate, audit-log completeness, schema failure rate, route counts, and latency?

If yes, I can adapt the run to whatever protocol you would consider acceptable for review.

[mindbomber]

Current AANA maintainer-review request pack is now consolidated here:

https://github.com/mindbomber/Alignment-Aware-Neural-Architecture--AANA-/blob/master/docs/maintainer-review-benchmark-submission-request.md

This replaces the earlier branch-specific links with a canonical master link and keeps the claim boundary explicit: AANA is being submitted for review as a control layer/wrapper around a base agent, not as a standalone model row or official leaderboard result.

Current ask for MCP-Bench maintainers:

1. Would MCP-Bench accept a control-layer/wrapper submission track?
2. Should the next run use a benchmark-native task subset, a dedicated safety slice, or a maintainer-provided task list?
3. Which side metrics belong in a report without confusing them with raw model/task-performance leaderboard metrics?
4. Would you prefer a PR, standalone result artifact, or continued discussion first?

The request pack links the current wrapper, paired runner, unsafe/private/write diagnostic slice, local clean-env results, limitations, and proposed metrics.