Measure Search and inference usage in the Groq agent - #62
Draft
georgeatparallel wants to merge 2 commits into
Draft
Measure Search and inference usage in the Groq agent#62georgeatparallel wants to merge 2 commits into
georgeatparallel wants to merge 2 commits into
Conversation
Collaborator
Author
|
Tracking this in DEV-261, assigned to me. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Our Groq agent shows searches and answers, but it doesn't show how much retrieval and model work a run uses. We add that information to the existing stream and UI so developers can inspect their own runs without a separate benchmark harness.
We move the tool from beta Search to GA Search, keep
basicas the default, and make the Search mode and price assumptions configurable. A small tracker measures successful Search calls, latency, sources, excerpt characters and result bytes, then adds provider-reported model tokens and elapsed time to the final event. If a model step omits a token count, we leave that count unavailable for the whole run instead of showing a partial total.We label dollar amounts as estimates, not measured spend. Characters and bytes are not tokens, and model and total cost stay unavailable until both model rates and complete input/output usage are present. The estimates exclude failed requests, cache or reasoning-specific rates, taxes, credits and discounts. They are not billing receipts or benchmark results.
Testing: Previously recorded validation includes five passing tests on Node 20, 22 and 24, strict economics-module and full Worker typechecks, and a Wrangler dry-run bundle without deployment. The current diff also adds tests for missing usage across steps and Worker streaming with mocked network responses. Recorded live validation covers one bounded Turbo Search request returning one result and one Search SKU, not a full live Groq run or deployment.
Tracking: DEV-261.