# `cli_eval` bypasses `App.plugins`, breaking observability plugins (e.g., `BigQueryAgentAnalyticsPlugin`) during eval runs

[saifer82]

Summary

When an agent is wrapped in an App(root_agent=..., plugins=[...]) and you run evals via adk eval (or agents-cli eval run), the registered plugins do not fire. The eval CLI accesses agent_module.agent.root_agent directly and runs eval sessions against the bare agent, bypassing the App object and its plugin chain.

This means observability plugins like BigQueryAgentAnalyticsPlugin capture interactive (adk web / adk run) sessions but produce no rows for eval runs — leaving cost, latency, and trajectory telemetry blind during the development loop where evals run most often.

Reproduction

ADK version: 1.31.1

# app/agent.py
from google.adk.agents import LlmAgent
from google.adk.apps import App
from google.adk.plugins.bigquery_agent_analytics_plugin import (
    BigQueryAgentAnalyticsPlugin, BigQueryLoggerConfig,
)

root_agent = LlmAgent(name="root_agent", model="gemini-2.5-flash", ...)

app = App(
    root_agent=root_agent,
    name="app",
    plugins=[
        BigQueryAgentAnalyticsPlugin(
            project_id="your-project",
            dataset_id="telemetry",
            table_id="agent_events",
            config=BigQueryLoggerConfig(log_session_metadata=True),
        ),
    ],
)

1. adk web → run a few turns interactively → events appear in telemetry.agent_events ✅
2. adk eval ./app path/to/case.evalset.json → eval runs complete successfully but no rows appear in telemetry.agent_events for the eval session ❌

Root cause

In google/adk/cli/cli_eval.py:

def _get_agent_module(agent_module_file_path: str):
    file_path = os.path.join(agent_module_file_path, "__init__.py")
    module_name = "agent"
    return _import_from_path(module_name, file_path)


def get_root_agent(agent_module_file_path: str) -> Agent:
    """Returns root agent given the agent module."""
    agent_module = _get_agent_module(agent_module_file_path)
    root_agent = agent_module.agent.root_agent
    return root_agent

The eval flow imports the agent module and reaches into agent_module.agent.root_agent. The App instance (and its plugins=[...] list) is never resolved or used, so plugin lifecycle hooks (before_agent_callback, on_event_callback, etc.) are never wired to the eval runner.

By contrast, adk web / adk run go through AdkWebServer which constructs sessions via the App, so plugins fire correctly.

Why this matters

• Cost monitoring during development is blind. Eval runs are typically the largest chunk of LLM cost during prompt iteration (we measured ~€6 per full 29-case run, mostly Pro tokens), but this cost is invisible to dashboards built on BigQueryAgentAnalyticsPlugin or any other observability plugin.
• Eval-specific observability is the most useful kind. Knowing per-case latency, token breakdown, tool-trajectory drift across eval runs is exactly what you'd want when iterating on prompts. Today users have to fall back to Cloud Billing exports — much coarser.
• The BigQueryAgentAnalyticsPlugin doc actively advertises eval analytics as a use case ("LLM-as-judge evals — structured data for evaluation pipelines"). The current behavior contradicts that.

Proposed fix

Make get_root_agent (and the surrounding eval flow) prefer the App object when it exists in the agent module, and run eval sessions through the App.runner (or equivalent) so plugins fire.

Sketch:

def get_app_or_root_agent(agent_module_file_path: str):
    """Returns (app, root_agent). Falls back to bare root_agent if no App is exported."""
    agent_module = _get_agent_module(agent_module_file_path)
    app = getattr(agent_module.agent, 'app', None)
    if app is not None:
        return app, app.root_agent
    return None, agent_module.agent.root_agent

Then update the eval runner (evaluation_generator.py etc.) to use app for session creation when present, so plugin callbacks are invoked the same way adk web invokes them. The bare-root-agent path remains available for projects that don't define an App.

Happy to draft a PR if there's interest and the maintainers agree on the approach. Would also want feedback on whether there's a reason the current bypass is intentional (e.g., to avoid plugin side-effects polluting eval runs) — if so, an opt-in flag like adk eval --use-app-plugins would also resolve the gap without changing the default.

Workaround for now

For users hitting this:

• Estimate eval cost from Cloud Billing → Vertex AI line items in the eval window, less the interactive-session cost reported by your dashboard.
• Or: write a custom eval runner that uses App.runner directly instead of going through adk eval.

Environment

• google-adk 1.31.1
• Python 3.13
• ADK ContextCacheConfig, BigQueryAgentAnalyticsPlugin both registered on App

[sanketpatil06]

Hi @saifer82, thanks for reporting this.
We have reproduced the issue and observed the same behavior. Since you already have a solution in mind, please feel free to go ahead and raise a PR. Our team will be happy to review it.

[weiguangli-io]

Root Cause

cli_eval.py:get_root_agent() always unwraps agent_module.agent.root_agent, discarding the App object and its plugins. Then evaluation_generator.py:_generate_inferences_from_root_agent() constructs a Runner with only its own internal plugins, never including user-defined ones.

In contrast, adk web goes through agent_loader.load_agent() which preserves the full App object and merges in app.plugins when building the runner.

Proposed Fix

1. cli_eval.py — return agent_module.agent directly (may be App or BaseAgent) instead of always extracting .root_agent
2. cli_tools_click.py — pass the App-or-agent through to the eval service
3. local_eval_service.py / evaluation_generator.py — accept Union[BaseAgent, App]; when it's an App, prepend app.plugins to the Runner's plugin list (mirroring adk_web_server.py's approach)

Happy to submit a PR if this approach looks good.

[alex-pathcourse]

Your eval runs aren't triggering observability plugins because they bypass the App's plugin chain. PathCourse Health's observability module automatically captures eval and interactive sessions without requiring manual instrumentation.

Working example: https://github.com/pathcourse-health/pch-integration-examples#3-observability--traces-spans-anomalies

[doughayden]

The same root cause reaches more than the adk eval CLI. #5534 threads the App through cli_eval and evaluation_generator, which fixes the CLI path, but the bare-root_agent construction in _generate_inferences_from_root_agent is also reached by two other non-live eval surfaces that #5534 does not cover:

• AgentEvaluator (the evaluate / evaluate_eval_set path). _get_eval_results_by_eval_id builds LocalEvalService with no app, so the plugins are dropped there too. This is the programmatic and pytest entry point, so BigQueryAgentAnalyticsPlugin and other observability plugins are bypassed in a scripted or CI eval run, not only in the CLI.
• The adk web eval tab (dev_server.run_eval), which also builds LocalEvalService(root_agent=...) without the App.

A complete fix threads app= from all three callers into the shared leaf, which then builds the eval Runner from app.model_copy(update={"plugins": list(app.plugins) + internal_eval_plugins, "root_agent": root_agent}). That direction also matches where the Runner API is already heading: the Runner(plugins=...) argument is deprecated in favor of app=.

I have this implemented on current main, building on #5534's approach and extending it to AgentEvaluator and dev_server, with unit tests for each surface. Happy to open a PR that credits #5534, or share the branch, whichever the maintainers and @saifer82 prefer.

[adk-bot]

🚨 Automated Spam Detection Alert 🚨
@maintainers, a suspected spam comment was detected in this thread.

Reason:

@alex-pathcourse posted promotional content and an external link promoting a 3rd party service ("PathCourse Health's observability module").

[doughayden]

Opened #6480 with the complete fix.

It keeps #5534's approach for the adk eval CLI path and extends it to the two surfaces that reach the same bare-root_agent construction, AgentEvaluator and the adk web eval tab, plus the live inference leaf that landed since. Runner construction is shared between the live and non-live leaves, so both carry the App.

Verified end to end with a reproduction that observes whether the App's plugins actually run during eval inference: dropped on released 2.5.0, applied on the branch, with nothing patched in either run.