RFC: Align efficiency metrics with Anthropic and OpenAI's official best-practice guidance

[hackertaco]

Summary

The efficiency score in /admin/analytics mixes "cost optimization" signals
with what's labeled as "efficiency". Cross-referenced against both official
agentic-CLI best-practice guides that Zeude's telemetry covers:

• Anthropic Best Practices for Claude Code
• OpenAI Codex best practices

The two guides show significant overlap on recommended behaviors, yet most
of those behaviors are not measured at all in the current dashboard — and
a few are unintentionally penalized. Opening this for discussion before
proposing code, since a rewrite touches shim instrumentation, ClickHouse
schema, and dashboard UI.

(Observations are based on zep-us/zeude@main at the time of writing.)

Common ground across providers

Despite different vocabulary, both guides point to the same behaviors:

Behavior	Claude Code	Codex
Repo agent guide	CLAUDE.md	AGENTS.md
Context compaction	/compact	/compact (+ auto)
Plan before code	Plan Mode	Ask Mode → Code Mode
Verification loops	tests / lints / builds	tests / lints + /review
Subagents for investigation	Task tool	Subagents
Skills for repeatable workflows	.claude/skills/	SKILL.md
Explicit file / context referencing in prompts	@file references	Structured "Goal / Context / Constraints / Done"
Session reset between unrelated tasks	/clear	/fork, /resume

Note: /clear discards context; /fork preserves the original as a
sibling thread. Similar intent, different mechanics.

Since Zeude tracks both providers, metrics labeled "efficiency" should
ideally reflect behaviors that both providers officially endorse.
Today's efficiencyScore measures none of them directly.

Gaps between current metrics (main) and provider guidance

Behavior (both providers endorse)	Measured on main?
Slash-command context resets (/clear, /compact, /fork)	❌ Not tracked
Subagent / Task invocation	❌ Not tracked
Plan / Ask mode usage	❌ Not tracked
Verification runs (test / build / lint invocations)	❌ Not tracked
@file or scoped file references in prompts	🟡 Parseable, not aggregated
Repo agent guide (CLAUDE.md / AGENTS.md) presence & hygiene	❌ Not tracked
Short, focused sessions	🟡 Proxy via avgPromptsPerSession
Specific prompts	🟡 Proxy via avgPromptChars
Minimize retries	🟡 retry_density (timing heuristic)

Specific issues in current implementation

1. cacheHitRate rewards long sessions over skilled use.
Both guides explicitly recommend frequent context resets (/clear on
Claude Code, /fork / /resume on Codex). Doing that lowers
cacheHitRate. The metric rewards behavior the guides advise against.

1b. cacheEfficiency doc and code disagree.
In lib/efficiency.ts, the header comment documents the formula as
cache_read_tokens / (cache_read_tokens + input_tokens), but the
implementation uses cache_read_tokens + output_tokens as the denominator.
output_tokens is model-generated, so including it in a cache-utilization
denominator is semantically off. Which of the two was intended?

1c. cacheHitRate in /api/admin/analytics/efficiency/route.ts can
exceed 1.0.
Formula is cacheReadTokens / inputTokens. When cached tokens greatly
exceed new tokens (typical for OAuth users with short sessions), rates
like 36,000%+ appear. A proper hit rate should be
cache_read / (cache_read + input), bounded in [0, 1].

2. context_growth_rate ∈ [0.5, 2] "ideal" range is arbitrary.
Deep refactors legitimately produce 10x+ growth, and both providers
recommend Plan-then-Code for exactly that kind of work. The metric
penalizes the behavior.

3. cost_efficiency gives free users a flat 0.5 neutral.
OAuth users automatically pass while pay-per-token users carry a
structural handicap unrelated to skill.

4. retry_density uses a timing heuristic that misclassifies iteration.
The view flags "fast follow-up after short response" as a retry, which
catches natural iteration loops (common in both Claude Code and Codex) as
"failed prompts". A prototype using prompt-text similarity moved false
positives from ~80% → ~2% on our team data (small sample, ~4 users,
single workload profile).

5. Codex conversation.id doesn't rotate reliably.
In Codex telemetry, many sessions share one conversation.id, collapsing
distinct session_id to 1 and degenerating avgPromptsPerSession. A
timestamp-gap-based session derivation (> 30 min gap = new session)
produces meaningful values without waiting for an OpenAI-side fix.

Proposed direction

Rather than polishing the current proxies, introduce signals that directly
reflect the recommended behaviors.

New telemetry (to add to claude_code_logs or a new dedicated table):

• Slash-command events per session (/clear, /compact, /fork, /resume, /plan, etc.)
• Tool invocations per session (Read / Grep / Bash / Task / Edit counts + distinct tool diversity)
• Plan Mode / Ask Mode entry–exit events
• @file reference count per prompt (parse prompt_text)

New metrics (each with a clear ↑ or ↓ intent):

• reset_rate_per_session — how often users actively reset / fork context
• tool_diversity_per_session — distinct tools exercised
• verification_density — % of sessions that invoke test / build / lint via Bash
• plan_mode_adoption — % of multi-file changes that used Plan or Ask mode
• file_reference_rate — @file mentions per prompt

Future signals (privacy-sensitive, needs careful design):

• Presence + size + change frequency of CLAUDE.md / AGENTS.md in the
  working repo (hashed paths only; opt-in)

Presentation change:

• Drop the single composite "efficiency score" → multi-dimensional profile.
• Per-user style tags (e.g., "deep-work / clean / exploratory / …") instead
  of a ranking.
• Personal week-over-week trend instead of team leaderboard for judgmental
  metrics.

Questions for maintainers

1. Does this direction match where the project is heading? Issues Feature: Personal Skill Usage Analytics in Daily Dashboard #7–Feature: Team-level Skill Usage Analytics #8
   touch skill analytics — how should this intersect?
2. Slash-command and tool-invocation events — does the shim already
   capture any of these internally without exporting, or would this be
   new instrumentation?
3. Any constraints on ClickHouse schema changes we should be aware of
   (backfill strategy, MV / TTL, etc.)?

Happy to contribute if the direction lands. We have a local
disjoint-history copy with ~16 commits exploring parts of this (semantic
retry, timestamp-based session grouping, cache-rate bug fixes, user_id
dedup, style tags). No plan to push that as-is — will rebuild cleanly on
top of main once the direction is agreed.

Thanks for building Zeude. Happy to continue this discussion in Korean if
that's more natural for the maintainers.