Summary
Add token usage analytics per selected model, including a “tokens burned” overview and estimated cost savings compared to alternative models.
Problem / Motivation
PiPiMink helps route prompts to suitable models, but there is currently no clear analytics view showing how many tokens each selected model consumes over time. This makes it difficult to understand model usage patterns, compare efficiency between models, and evaluate the real impact of routing decisions.
Users who test multiple models, optimize prompts, or route requests across local and paid providers need visibility into token consumption to understand:
Which models consume the most input and output tokens
How token usage changes per prompt, model, provider, or time period
Whether a cheaper or more efficient model could have handled the request
How much money may have been saved by choosing one model over another
This is especially useful for users running mixed setups with OpenAI-compatible APIs, local models, and commercial providers.
Proposed Solution
Add a token usage analytics section that tracks token consumption for every selected model and displays it in the analytics dashboard.
The feature should include:
Token usage per request
Input tokens
Output tokens
Total tokens
Selected model
Provider
Timestamp
Prompt or request ID
Aggregated token analytics
Total tokens burned per model
Total tokens burned per provider
Token usage over time
Average tokens per request
Most expensive / most used models
Comparison between routed models
Optional cost estimation
Configurable price per 1M input/output tokens per model
Estimated cost per request
Estimated total cost per model/provider
Estimated cost savings compared to a configurable baseline model
Example:
{
"model": "gpt-4.1",
"provider": "openai",
"input_tokens": 1250,
"output_tokens": 430,
"total_tokens": 1680,
"estimated_cost": 0.0124,
"baseline_model": "gpt-4.1",
"alternative_model": "gpt-4.1-mini",
"estimated_savings": 0.0081
}
A possible configuration field could allow users to define model pricing manually:
{
"pricing": {
"openai/gpt-4.1": {
"input_per_1m_tokens": 2.00,
"output_per_1m_tokens": 8.00
},
"openai/gpt-4.1-mini": {
"input_per_1m_tokens": 0.40,
"output_per_1m_tokens": 1.60
},
"ollama/llama3.1": {
"input_per_1m_tokens": 0,
"output_per_1m_tokens": 0
}
}
}
The analytics UI could show a “Token Burned” dashboard with charts and tables for:
Tokens burned by model
Tokens burned by provider
Cost by model
Estimated savings from routing decisions
Comparison against a default or user-selected baseline model
Alternatives Considered
A simple workaround would be to rely on provider dashboards, such as OpenAI, Anthropic, or other API billing pages. However, those dashboards are provider-specific and do not give a unified view across all models and providers used by PiPiMink.
Another workaround would be to log raw API responses and manually calculate token usage later. This is inconvenient, error-prone, and does not provide immediate visibility inside PiPiMink.
A minimal alternative would be to only display total token usage per request without cost estimation. This would already be useful, but it would not fully show the financial impact or savings potential of routing decisions.
Additional Context
This would make PiPiMink analytics more useful for evaluating model routing quality and cost efficiency. Since PiPiMink is focused on selecting the best model for a given prompt, token and cost analytics would help users understand whether the chosen model was not only technically suitable, but also economically efficient.
Prior art exists in many LLM gateways and observability tools, where token usage, request cost, and model comparison are key analytics metrics. A similar feature in PiPiMink would make it easier to track real-world usage, optimize routing strategies, and demonstrate the value of model selection over always using a single large default model.
Summary
Add token usage analytics per selected model, including a “tokens burned” overview and estimated cost savings compared to alternative models.
Problem / Motivation
PiPiMink helps route prompts to suitable models, but there is currently no clear analytics view showing how many tokens each selected model consumes over time. This makes it difficult to understand model usage patterns, compare efficiency between models, and evaluate the real impact of routing decisions.
Users who test multiple models, optimize prompts, or route requests across local and paid providers need visibility into token consumption to understand:
Which models consume the most input and output tokens
How token usage changes per prompt, model, provider, or time period
Whether a cheaper or more efficient model could have handled the request
How much money may have been saved by choosing one model over another
This is especially useful for users running mixed setups with OpenAI-compatible APIs, local models, and commercial providers.
Proposed Solution
Add a token usage analytics section that tracks token consumption for every selected model and displays it in the analytics dashboard.
The feature should include:
Token usage per request
Input tokens
Output tokens
Total tokens
Selected model
Provider
Timestamp
Prompt or request ID
Aggregated token analytics
Total tokens burned per model
Total tokens burned per provider
Token usage over time
Average tokens per request
Most expensive / most used models
Comparison between routed models
Optional cost estimation
Configurable price per 1M input/output tokens per model
Estimated cost per request
Estimated total cost per model/provider
Estimated cost savings compared to a configurable baseline model
Example:
{
"model": "gpt-4.1",
"provider": "openai",
"input_tokens": 1250,
"output_tokens": 430,
"total_tokens": 1680,
"estimated_cost": 0.0124,
"baseline_model": "gpt-4.1",
"alternative_model": "gpt-4.1-mini",
"estimated_savings": 0.0081
}
A possible configuration field could allow users to define model pricing manually:
{
"pricing": {
"openai/gpt-4.1": {
"input_per_1m_tokens": 2.00,
"output_per_1m_tokens": 8.00
},
"openai/gpt-4.1-mini": {
"input_per_1m_tokens": 0.40,
"output_per_1m_tokens": 1.60
},
"ollama/llama3.1": {
"input_per_1m_tokens": 0,
"output_per_1m_tokens": 0
}
}
}
The analytics UI could show a “Token Burned” dashboard with charts and tables for:
Tokens burned by model
Tokens burned by provider
Cost by model
Estimated savings from routing decisions
Comparison against a default or user-selected baseline model
Alternatives Considered
A simple workaround would be to rely on provider dashboards, such as OpenAI, Anthropic, or other API billing pages. However, those dashboards are provider-specific and do not give a unified view across all models and providers used by PiPiMink.
Another workaround would be to log raw API responses and manually calculate token usage later. This is inconvenient, error-prone, and does not provide immediate visibility inside PiPiMink.
A minimal alternative would be to only display total token usage per request without cost estimation. This would already be useful, but it would not fully show the financial impact or savings potential of routing decisions.
Additional Context
This would make PiPiMink analytics more useful for evaluating model routing quality and cost efficiency. Since PiPiMink is focused on selecting the best model for a given prompt, token and cost analytics would help users understand whether the chosen model was not only technically suitable, but also economically efficient.
Prior art exists in many LLM gateways and observability tools, where token usage, request cost, and model comparison are key analytics metrics. A similar feature in PiPiMink would make it easier to track real-world usage, optimize routing strategies, and demonstrate the value of model selection over always using a single large default model.