Add budget governance attributes to invoke_agent span - #439
Conversation
Pull request dashboard statusWaiting on reviewers · refreshed 2026-08-24 06:37 UTC Review the latest changes. Status above doesn't look right?
|
There was a problem hiding this comment.
Pull request overview
This PR extends the OpenTelemetry GenAI semantic conventions for the invoke_agent operation by introducing agent budget governance attributes (token and iteration budgets) and a derived utilization metric, and updates the reference outputs to reflect the new conventions.
Changes:
- Added four new
gen_ai.agent.*_budget.{limit,consumed}attributes to the internalinvoke_agentattribute set. - Added a new histogram metric
gen_ai.invoke_agent.token_budget.utilization(consumed/limit). - Updated the Google ADK reference scenario and regenerated reference reports/docs to include the new budget attributes it can emit.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| reference/scenarios/google-adk/scenario.py | Emits iteration budget limit/consumed and token budget consumed on the invoke_agent span based on Google ADK runtime data. |
| reference/scenarios/google-adk/data.json | Updates expected span attributes for the Google ADK scenario to include the new budget attributes it emits. |
| reference/reports/invoke-agent-internal-span.md | Regenerated report reflecting new budget attributes’ supporting libraries for the internal span. |
| reference/reports/invoke-agent-client-span.md | Regenerated report showing no client-span support for the new budget attributes. |
| model/gen-ai/spans.yaml | Adds a new internal attribute group for agent budgets and includes it in internal invoke_agent attributes. |
| model/gen-ai/registry.yaml | Defines the four new budget attributes with guidance/constraints. |
| model/gen-ai/metrics.yaml | Adds the new utilization histogram metric and its semantic definition. |
| docs/registry/attributes/gen-ai.md | Regenerated registry page listing the new attributes and their notes. |
| docs/gen-ai/gen-ai-agent-spans.md | Regenerated span documentation tables/footnotes reflecting new attributes on invoke_agent. |
| changelog.d/426.enhancement.md | Adds a Towncrier fragment describing the new attributes and metric. |
Suppressed comments (1)
model/gen-ai/metrics.yaml:246
- The note for
gen_ai.invoke_agent.token_budget.utilizationdoesn't state which span it should be emitted alongside. Othergen_ai.invoke_agent.*metrics in this file explicitly say they should be emitted together with thegen_ai.invoke_agent.internalspan; adding the same guidance here would keep the conventions consistent for implementers.
This metric SHOULD only be emitted when a token budget is configured
(`gen_ai.agent.token_budget.limit` is set) and the invocation span
has ended.
| brief: Cumulative token budget configured for this agent invocation. | ||
| note: | | ||
| The cumulative token budget (input + output, across all inference | ||
| calls) configured for this agent invocation. When the budget is | ||
| exhausted the agent loop terminates. | ||
|
|
||
| Instrumentations SHOULD populate this attribute only when the | ||
| configured limit acts as a cumulative cap across the full | ||
| invocation, not when it constrains a single inference call | ||
| (use `gen_ai.request.max_tokens` for per-call limits). |
There was a problem hiding this comment.
The note scopes it to "configured for this agent invocation." No referenced framework supports mid-flight budget changes, so a last-applied-value sentence would be speculative. Happy to add one if a reviewer feels it's ambiguous.
| - id: attributes.gen_ai.agent.budget | ||
| visibility: internal | ||
| attributes: | ||
| - ref: gen_ai.agent.token_budget.limit | ||
| requirement_level: |
There was a problem hiding this comment.
AutoGen scenario emits iteration_budget.limit, .consumed, and token_budget.consumed (see autogen/scenario.py in this diff). token_budget.limit is intentionally unset because neither framework exposes a cumulative token cap. There's an explicit capture-gap comment in the ADK scenario explaining this.
There was a problem hiding this comment.
Correction: AutoGen does have a token cap (TokenUsageTermination), it's just team level. Updated in a53f0eb.
| - name: gen_ai.invoke_agent.token_budget.utilization | ||
| requirement_level: recommended | ||
| annotations: | ||
| code_generation: | ||
| metric_value_type: double |
There was a problem hiding this comment.
Covered in google-adk/data.json and scenario.py in this PR.
| @@ -0,0 +1 @@ | |||
| Add `gen_ai.agent.token_budget.limit`, `gen_ai.agent.token_budget.consumed`, `gen_ai.agent.iteration_budget.limit`, and `gen_ai.agent.iteration_budget.consumed` attributes and `gen_ai.invoke_agent.token_budget.utilization` metric for agent budget governance telemetry. | |||
There was a problem hiding this comment.
File is named 439.enhancement.md. The #426 in the description refers to the predecessor PR, not the fragment.
6b905b0 to
18fe015
Compare
| json.dumps([{"role": "user", "parts": [{"type": "text", "content": input_text}]}]), | ||
| ) | ||
| agent_span.set_attribute("gen_ai.tool.definitions", json.dumps(tool_defs)) | ||
| agent_span.set_attribute("gen_ai.agent.iteration_budget.limit", 2) |
There was a problem hiding this comment.
this seems to be a magic number. Please demonstrate where this actually can come from in this and other instrumentations.
There was a problem hiding this comment.
Fixed in a53f0eb. The attribute pulls straight from what you pass to the constructor (AssistantAgent(max_tool_iterations=N) for AutoGen, RunConfig(max_llm_calls=N) for ADK). No magic numbers.
| if total_output_tokens: | ||
| agent_span.set_attribute("gen_ai.usage.output_tokens", total_output_tokens) | ||
| agent_span.set_attribute("gen_ai.agent.iteration_budget.consumed", len(captured_results)) | ||
| agent_span.set_attribute("gen_ai.agent.token_budget.consumed", total_input_tokens + total_output_tokens) |
There was a problem hiding this comment.
is there a point in capturing a derived valus since it's already exposed via individual attributes?
There was a problem hiding this comment.
Agreed, gone. You can already sum it from the same span.
| agent_span.set_attribute("gen_ai.usage.input_tokens", total_input_tokens) | ||
| if total_output_tokens: | ||
| agent_span.set_attribute("gen_ai.usage.output_tokens", total_output_tokens) | ||
| agent_span.set_attribute("gen_ai.agent.iteration_budget.consumed", len(captured_results)) |
There was a problem hiding this comment.
this looks confusing - why number of return results has anything to do with budget?
There was a problem hiding this comment.
You're right. Bad proxy on my part. Line's gone entirely now that iteration_budget.consumed is dropped.
| agent_span.set_attribute("gen_ai.usage.input_tokens", total_input_tokens) | ||
| if total_output_tokens > 0: | ||
| agent_span.set_attribute("gen_ai.usage.output_tokens", total_output_tokens) | ||
| agent_span.set_attribute("gen_ai.agent.iteration_budget.consumed", call_counts["inference"]) |
There was a problem hiding this comment.
there is a metric gen_ai.invoke_agent.inference_calls - would it make sense to just define an attribute with the same name? Or, even better, consumers can count number of inference spans under this agent and then no attribute on a span is needed.
There was a problem hiding this comment.
Agreed. inference_calls covers consumption and survives sampling, so both .consumed attributes are gone.
The configured cap is a different story though. That only exists in framework config (max_tool_iterations, max_llm_calls), nowhere in spans or metrics. So the .limit pair stays.
On the token side, AutoGen does have a cap via TokenUsageTermination, but it's team level so the single agent scenario can't emit token_budget.limit. Note added. If you'd rather see it demonstrated before it lands, happy to split the token budget stuff into a follow up. The iteration budget half stands on its own.
|
Pushed a53f0eb. Both Motivation table, changelog, and generated docs updated. |
| note: | | ||
| The workflow name is usually a static, application-unique identifier defined | ||
| in a framework-specific way. | ||
| This attribute can be populated in different frameworks; for example, as the name of the first chain in LangChain or the name of the crew in CrewAI. |
There was a problem hiding this comment.
this does not seem to be related
| stability: development | ||
| - key: gen_ai.agent.token_budget.limit | ||
| type: int | ||
| brief: Cumulative token budget configured for this agent invocation. |
There was a problem hiding this comment.
there is no framework that has such a configuration option, at least none were demonstrated in this PR
| what `gen_ai.workflow.name` means in the context of that framework. | ||
| examples: ["multi_agent_rag", "customer_support_pipeline"] | ||
| Examples of framework parameters that map to this attribute: | ||
| AutoGen `TokenUsageTermination`. |
There was a problem hiding this comment.
is this framework deprecated / replaced?
| Examples of framework parameters that map to this attribute: | ||
| LangChain `max_iterations`, CrewAI `max_iter`, OpenAI Agents SDK | ||
| `max_turns`, LangGraph `recursion_limit`, Google ADK | ||
| `max_llm_calls`, AutoGen `max_tool_iterations`. |
There was a problem hiding this comment.
these things seem to be way too different - number of llm calls != recursion limit != tool iterations.
An attribute that attempts to represent them all is not usable in practice without specific library context. The iteration needs to be defined as something common, or, if there is no such generic concept, the limits could be recorded with framework specific attributes
Introduces a single attribute for recording the configured maximum iterations on an agent's invoke_agent span. Tightened scope compared to original submission: - Removed token_budget.limit and utilization metric (deferred to follow-up with Pydantic AI UsageLimits evidence) - Excluded graph traversal depth (LangGraph recursion_limit) and runner-wide inference counts (ADK max_llm_calls) via explicit MUST NOT clause - Demonstrated in autogen (max_tool_iterations) and crewai (max_iter) reference scenarios with sentinel guards - Added capture-gap comment to google-adk explaining scope mismatch
Aligns with the pattern established by gen_ai.workflow.name, which uses SHOULD for framework-specific documentation requirements.
a53f0eb to
923d25e
Compare
|
The workflow.name diff is rebase noise from a stale base. Gone now. I pulled AutoGen entered maintenance mode late 2025, Microsoft Agent Framework is its successor. MAF's Heterogeneity concern is fair. What's left is CrewAI Why unified rather than framework-specific: an ops team running CrewAI in one service and Agent Framework in another can't write one alert for "agent approaching ceiling" without enumerating every possible attribute name. Diff reflects all of this. If you'd still rather see framework-specific attributes I can go that route. |
|
Empirical update on the counting-method question. I modeled eleven frameworks from source code, then executed eight of them against a deterministic mock LLM with a request ledger. The mock gives ground truth (identical work across all frameworks, no network variance, fully reproducible). The source-code model was right for exactly half. The executed table, for identical ground truth of 4 LLM calls and 3 tool calls:
Every number is machine-derived from results/S2-executed.json in the linked repo. The I expected a majority cluster in the unit taxonomy. There isn't one. Across enforcing frameworks the observed units break down as composite_messages (1), llm_invocations (2), tool_cycles (2), graph_nodes (1), auto_invoke_rounds (1). Five distinct counting methods, no winner. Without a unit attribute on the telemetry, none of these numbers are comparable to each other. A consumed>3 alert fires for 4 of the 7 enforcing frameworks but not the other 3, for the exact same work. This supports "record verbatim, without normalization" in the current PR. If a consumed attribute comes back later it needs a mandatory counting_method enum. The measured chaos is the argument for the attribute. Side finding on Agno 1.2.5: tool_call_limit is a real propagated dataclass field and the enforcement code runs, but it doesn't actually stop the agent. Nine tool calls past a limit of three. Known bug (agno-agi/agno#8304), independently reproduced here blind before we found the existing reports. Deterministic mock removes model behavior as a variable. Harness is public at https://github.com/elang2/agent-budget-semantics (v0.3.0, pip-installable, runs without API keys). Happy to add any framework or scenario that would help here. |
Description
Add
gen_ai.agent.iteration_budget.limitattribute for theinvoke_agentspan.Resolves #425. Continuation of #426 (accidentally closed — I made my forked repo private which caused GitHub to close the PR and sever the fork relationship, making it non-reopenable).
New attribute on
invoke_agentspans:gen_ai.agent.iteration_budget.limit(int) — configured iteration limit for a single agent's control loop.Changes since #426
brief(not just the note) per reviewer feedback on summability.consumedattributes (derivable from same span tokens and theinference_callsmetric)token_budget.limitand utilization metric (deferred to follow-up with per-invocation evidence)Motivation
An SRE operating a multi-agent pipeline needs to know which agents are consistently running near their configured budget and alert before a runaway loop exhausts it. Today the
invoke_agentspan shows duration but not budget consumption. Operators only discover overruns when the monthly bill arrives or by manually summing child spans (which breaks under sampling since theinference_callsmetric survives but child spans may not).Agent frameworks that expose per-agent iteration caps:
max_itermax_iterationsmax_tool_iterationsExplicitly excluded (different semantic scope):
recursion_limitmax_turnsmax_llm_callsThere is no vendor-neutral telemetry standard for recording per-agent iteration caps on traces. This attribute captures the invocation-level envelope that frameworks enforce.
Prototype
Reference scenarios updated for AutoGen and CrewAI (
reference/scenarios/). ADK scenario includes a capture-gap comment explaining whymax_llm_callsis out of scope.Checklist
changelog.d/cc @meshailabs — this replaces #426 which was accidentally closed (I made my fork private which severed the fork relationship).
Changes since last review
token_budget.limitandtoken_budget.utilization(deferred to follow-up with Pydantic AI demonstration)iteration_budget.limitscope: excludedrecursion_limit,max_turns,max_llm_callsvia MUST NOT clausemax_iterationsto examplesgen_ai.workflow.namepattern)