feat(dashboard): /skills/ema endpoint reading SessionDB.query_skill_ema#9
Open
Brecht-H wants to merge 1 commit intostainlu:mainfrom
Open
feat(dashboard): /skills/ema endpoint reading SessionDB.query_skill_ema#9Brecht-H wants to merge 1 commit intostainlu:mainfrom
Brecht-H wants to merge 1 commit intostainlu:mainfrom
Conversation
Adds an HTTP wrapper around hermes-agent's
SessionDB.query_skill_ema() so the labyrinth dashboard can render
per-skill exponentially-weighted moving averages of success rate,
cost-per-call, and duration.
Endpoint:
GET /api/plugins/hermes-labyrinth/skills/ema
?window_days=14 (calendar-day window)
&alpha=0.3 (0 < alpha < 1; α=0.3 ≈ 1.94d half-life)
Returns a JSON list of dicts with the same shape as
query_skill_ema():
skill_name, model, provider,
sample_count, success_count, failure_count,
ema_success_rate, ema_duration_s, ema_cost_per_call,
days_with_data, last_invoked_at
Defensive on older hermes-agent installs:
- Returns [] if state.db is missing
- Returns [] if SessionDB has no query_skill_ema attribute
(hermes-agent < schema v12 / NousResearch/hermes-agent#19508)
- 500 only on unexpected exceptions
This is the F1 follow-up flagged in NousResearch/hermes-agent#19508
PR review (R1: Mac 3-lens consensus, 2026-05-04).
Smoke test (live):
systemctl restart hermes-dashboard.service
curl http://localhost:9119/api/plugins/hermes-labyrinth/skills/ema
-> 200 [] # empty until cron-fired skills populate skill_invocations
curl http://localhost:9119/api/plugins/hermes-labyrinth/journeys?...
-> 200 (no regression)
Frontend panel deliberately deferred to a follow-up PR — wiring
shape (tabular vs sparkline-per-skill) is a UX decision worth its
own review cycle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new dashboard API endpoint that exposes skill-level EMA metrics from hermes_state.SessionDB.query_skill_ema(), so the Hermes Labyrinth dashboard can surface recent per-model/per-skill performance data when the upstream agent schema supports it.
Changes:
- Adds
GET /api/plugins/hermes-labyrinth/skills/emato proxySessionDB.query_skill_ema(window_days, alpha). - Returns
[]whenstate.dbis missing or the installedSessionDBdoes not providequery_skill_ema, preserving compatibility with older hermes-agent installs. - Converts database initialization/query failures into HTTP 500 responses for the new endpoint.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+785
to
+792
| query = getattr(db, "query_skill_ema", None) | ||
| if not callable(query): | ||
| # hermes-agent older than PR #19508 — graceful empty response. | ||
| return [] | ||
| try: | ||
| return query(window_days=window_days, alpha=alpha) | ||
| except Exception as e: | ||
| raise HTTPException(status_code=500, detail=f"query_skill_ema failed: {e}") |
Comment on lines
+760
to
+790
| async def skills_ema(window_days: int = 14, alpha: float = 0.3): | ||
| """Per-(skill_name, model) exponentially-weighted moving averages. | ||
|
|
||
| Thin wrapper over ``hermes_state.SessionDB.query_skill_ema(...)``. | ||
| Requires hermes-agent ≥ schema v12 (PR | ||
| `NousResearch/hermes-agent#19508`). On older installs (no | ||
| ``skill_invocations`` table or no ``query_skill_ema`` method) we | ||
| return ``[]`` so the dashboard renders an empty panel rather than | ||
| 500ing. | ||
|
|
||
| Query params: | ||
| - ``window_days`` (default 14) — calendar-day window | ||
| - ``alpha`` (default 0.3) — exponential smoothing factor. | ||
| ``α=0.3`` ≈ 1.94-day half-life. Use ``α≈0.129`` for a true | ||
| 5-day half-life. ``0 < alpha < 1``. | ||
|
|
||
| Returns a JSON list of dicts. See ``query_skill_ema`` docstring | ||
| for field semantics. | ||
| """ | ||
| if not _state_db_exists(): | ||
| return [] | ||
| try: | ||
| db = _db() | ||
| except Exception as e: | ||
| raise HTTPException(status_code=500, detail=f"state.db unavailable: {e}") | ||
| query = getattr(db, "query_skill_ema", None) | ||
| if not callable(query): | ||
| # hermes-agent older than PR #19508 — graceful empty response. | ||
| return [] | ||
| try: | ||
| return query(window_days=window_days, alpha=alpha) |
Comment on lines
+783
to
+792
| except Exception as e: | ||
| raise HTTPException(status_code=500, detail=f"state.db unavailable: {e}") | ||
| query = getattr(db, "query_skill_ema", None) | ||
| if not callable(query): | ||
| # hermes-agent older than PR #19508 — graceful empty response. | ||
| return [] | ||
| try: | ||
| return query(window_days=window_days, alpha=alpha) | ||
| except Exception as e: | ||
| raise HTTPException(status_code=500, detail=f"query_skill_ema failed: {e}") |
Comment on lines
+759
to
+792
| @router.get("/skills/ema") | ||
| async def skills_ema(window_days: int = 14, alpha: float = 0.3): | ||
| """Per-(skill_name, model) exponentially-weighted moving averages. | ||
|
|
||
| Thin wrapper over ``hermes_state.SessionDB.query_skill_ema(...)``. | ||
| Requires hermes-agent ≥ schema v12 (PR | ||
| `NousResearch/hermes-agent#19508`). On older installs (no | ||
| ``skill_invocations`` table or no ``query_skill_ema`` method) we | ||
| return ``[]`` so the dashboard renders an empty panel rather than | ||
| 500ing. | ||
|
|
||
| Query params: | ||
| - ``window_days`` (default 14) — calendar-day window | ||
| - ``alpha`` (default 0.3) — exponential smoothing factor. | ||
| ``α=0.3`` ≈ 1.94-day half-life. Use ``α≈0.129`` for a true | ||
| 5-day half-life. ``0 < alpha < 1``. | ||
|
|
||
| Returns a JSON list of dicts. See ``query_skill_ema`` docstring | ||
| for field semantics. | ||
| """ | ||
| if not _state_db_exists(): | ||
| return [] | ||
| try: | ||
| db = _db() | ||
| except Exception as e: | ||
| raise HTTPException(status_code=500, detail=f"state.db unavailable: {e}") | ||
| query = getattr(db, "query_skill_ema", None) | ||
| if not callable(query): | ||
| # hermes-agent older than PR #19508 — graceful empty response. | ||
| return [] | ||
| try: | ||
| return query(window_days=window_days, alpha=alpha) | ||
| except Exception as e: | ||
| raise HTTPException(status_code=500, detail=f"query_skill_ema failed: {e}") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a single HTTP endpoint
GET /api/plugins/hermes-labyrinth/skills/emathat wrapshermes_state.SessionDB.query_skill_ema()from upstreamNousResearch/hermes-agentPR #19508.That PR adds a
skill_invocationstable +skill_stats_dailyview + a calendar-aware exponentially-weighted moving-average query for per-(skill_name, model) success rate / cost / duration. The endpoint surfaces those EMAs to the dashboard so operators can A/B local Qwen against external models (DeepSeek, Kimi-via-OpenRouter, etc.) without flying blind.Endpoint contract
Returns a JSON list of dicts (one per
(skill_name, model)bucket) with:skill_name,model,providersample_count,success_count,failure_countema_success_rate,ema_duration_s,ema_cost_per_calldays_with_data,last_invoked_atSorted by
last_invoked_atdescending.Defensive on older hermes-agent installs
state.dbmissing →[]SessionDBhas noquery_skill_emaattribute (hermes-agent < schema v12) →[]This means installations on hermes-agent versions predating
#19508continue to render the dashboard normally (just with an empty/skills/emapanel).Smoke test (live, this machine)
Frontend follow-up
Deliberately a separate PR — the wiring shape (tabular vs sparkline-per-skill vs scatter-plot-cost-vs-quality) is a UX decision worth its own review cycle. Happy to ship a frontend PR once we converge on the panel design.
Test plan
[]on emptyskill_invocationstable/skills,/journeys,/cron,/health) still respondskill_invocations(pending Mac/Allaert flipping the analyzer crons toenabled: true)🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmithwith what you need.