feat: add e2e test script for tool_calling agent across all benchmarks

[yoavkatz]

Summary

Add an e2e-test.sh script that deploys the tool_calling agent against each benchmark (gsm8k, tau2, appworld) and evaluates it, then prints a consolidated results table. This gives the project a single command that exercises the full deploy-and-evaluate pipeline end to end.

Motivation

Currently there is no automated way to validate the harness across all three benchmarks in one run. Each benchmark must be invoked separately, and results are not aggregated. The new script provides a fast smoke-test path (1 task per benchmark by default) and a more thorough multi-task path for pre-release validation.

Proposed CLI

# Default: run benchmarks sequentially, 1 task each, stop on first failure
./e2e-test.sh --agent tool_calling

# Run 3 tasks per benchmark
./e2e-test.sh --agent tool_calling --tasks 3

# Run all 3 benchmarks in parallel (collect all results, don't stop on failure)
./e2e-test.sh --agent tool_calling --parallel

# Dry run (forward --dry to deploy-and-evaluate.sh)
./e2e-test.sh --agent tool_calling --dry

Behaviour

Sequential mode (default)

Runs deploy-and-evaluate.sh for each benchmark in order: gsm8k → tau2 → appworld.

• If a benchmark fails, the script stops immediately and does not run subsequent benchmarks.
• Exit code is non-zero if any benchmark failed or was skipped.

Parallel mode (--parallel)

Runs all three benchmarks concurrently (background processes). Waits for all to finish before printing the results table. Does not stop on failure — all results are collected.

--tasks N (default: 1)

Sets MAX_TASKS=N in the environment before calling deploy-and-evaluate.sh. This maps to the existing ExgenticConfig.max_tasks field in exgentic_a2a_runner/config.py, which limits how many MCP sessions the runner creates.

Results table

After all benchmarks complete, print an ASCII/Markdown table to stdout and write it to e2e-results.md (for CI artifact upload):

| Benchmark | Status | Eval Success Rate | Avg Latency (s) | Failures |
|-----------|--------|-------------------|-----------------|----------|
| gsm8k     | PASS   | 100%              | 4.2             | 0        |
| tau2      | PASS   | 80%               | 12.7            | 1        |
| appworld  | FAIL   | 0%                | —               | 1        |

Stats (eval success rate, avg latency, failures) are parsed from the runner's print_summary() stdout output (already printed at end of each evaluate-benchmark.sh run).

Acceptance Criteria

• Sequential run: benchmarks execute in order gsm8k → tau2 → appworld
• Sequential run: if gsm8k fails, script exits without running tau2 or appworld
• Parallel run (--parallel): all three benchmarks start simultaneously; results collected for all regardless of individual pass/fail
• --tasks N controls the number of tasks per benchmark (default 1)
• Results table printed to stdout after all benchmarks complete
• Results table written to e2e-results.md
• Table columns: Benchmark | Status | Eval Success Rate | Avg Latency (s) | Failures
• Exit code 0 only when all benchmarks pass; non-zero otherwise
• --dry flag forwarded to deploy-and-evaluate.sh for cluster-free testing
• --help prints usage

Implementation Notes

• Build on top of deploy-and-evaluate.sh — do not duplicate deploy/eval logic.
• Each benchmark run's stdout should be tee'd to a temp file so stats can be parsed after the run completes (important for parallel mode where output is interleaved).
• Parse the runner's print_summary() output (already written to stdout by evaluate-benchmark.sh) to extract stats for the table. Key lines to grep: sessions_attempted, evaluation_success_rate, average_latency_seconds, sessions_with_error.
• For parallel mode use & + wait $PID with per-benchmark log files; collect exit codes via wait return values.
• MAX_TASKS env var is consumed by config.py:ExgenticConfig.max_tasks (read at runner startup).

Related

• deploy-and-evaluate.sh — the per-benchmark wrapper this script orchestrates
• evaluate-benchmark.sh — invokes uv run exgentic-a2a-runner and prints RunSummary
• exgentic_a2a_runner/config.py — ExgenticConfig.max_tasks / MAX_TASKS
• exgentic_a2a_runner/runner.py — RunSummary.print_summary() (the output to parse)

[yoavkatz]

There should be another flag to run the same as k8s jobs and not locally.