feat(eval-eval): Add judges to eval PR to include in bench

[tmickleydoyle]

Example output:

2026-01-12T23:21:05.314Z [pr-eval] ## Benchmark Candidate Evaluation

PR: anomalyco/opencode-bench#16
Status: ❌ Rejected
Final Score: 41.1/100

---

Criterion Scores

Criterion	Score	Consensus
Scope & Clarity	54/100	🟢 High
Technical Quality	55/100	🟢 High
Evaluation Feasibility	46/100	🔴 Low
Reproducibility	39/100	🟡 Medium

---

Detailed Analysis

Scope & Clarity (54/100)

Judge Scores

• claude sonnet 4 5: 42/100
• gpt 5 codex: 55/100
• kimi k2: 65/100

Rationale

claude sonnet 4 5: SCOPE ISSUES (Major):

1. Poor self-containment: The PR is merging into tmd/add-time-rank-score branch, NOT main. This suggests it depends on uncommitted changes from another branch, making it impossible to reproduce in isolation.

2. Appropriate size: 9 files with 107 additions and 39 deletions is reasonable, but the changes span multiple agent implementations (claude-code, codex, opencode) plus CLI and type definitions.

3. Focused intent: The changes appear to add usage metrics (tokens per second, cost tracking) across multiple agents, which is reasonably focused, but the scope creep is evident with formatting changes and the cryptic "update" commits.

CLARITY ISSUES (Critical):

1. No PR description: This is a major red flag. There's literally "(No description provided)" - making it impossible to understand the motivation, acceptance criteria, or context without reading the entire diff.

2. Poor commit messages: Half the commits are just "update" with no context. Only the first commit "feat: add tokens per second metric" is meaningful, though the second "fix: keep cli aligned with main and add timing" provides some context.

3. Code readability concerns:

   • The diff shows incomplete code (truncated with "co..." at the end)
   • There's a fetch to "models.dev/api.json" added at the bottom of codex.ts with no context
   • Type changes from string[] to const arrays and generic type parameters suggest refactoring mixed with feature work

FATAL FLAW: The base branch being tmd/add-time-rank-score instead of main means this PR cannot be evaluated as a standalone task. It's building on top of unreleased work.

VERDICT: This PR is a poor benchmark candidate due to lack of description, dependency on another branch, mixed concerns (metrics + typing refactors), and poor commit hygiene. An AI agent would struggle to understand what success looks like without the PR description.

gpt 5 codex: Scope is moderate (9 files, cohesive focus on enriching usage metrics) and self-contained. However, the PR lacks any description, two commits are uninformative (“update”), and the intent must be inferred solely from the diff. This unclear documentation would make it hard to frame as a precise benchmark task, so clarity is insufficient despite manageable scope.

kimi k2: This PR has a moderate scope and clarity level, making it a marginal candidate for a benchmark task.

Scope Assessment:

• Self-containment: The changes appear to be self-contained within the agents and related infrastructure files. No external dependencies are evident.
• Appropriate size: 9 files with 107 additions and 39 deletions is within the acceptable range, though slightly on the larger side for a focused task.
• Focused intent: The PR appears to be adding cost calculation and timing metrics to agent usage tracking, which is a single cohesive feature. However, the lack of description makes this inference from code alone.

Clarity Assessment:

• PR description quality: No description is provided, which is a major flaw. The title "feat: add other information in the usage property" is vague and doesn't clearly explain what "other information" means.
• Commit message quality: Mixed quality - the first two commits are meaningful ("add tokens per second metric", "keep cli aligned with main and add timing"), but the last two are just "update" which provides no information.
• Code readability: The code changes are reasonably understandable - adding cost calculations, timing metrics, and type safety improvements. An AI agent could likely understand the goal from the diff, though it would require inference.

The main issues are the complete lack of PR description and partially uninformative commit messages. While the code changes themselves are focused and implement a clear feature (enhanced usage tracking with cost and timing), the documentation is insufficient for a benchmark task where clear requirements are essential.

Technical Quality (55/100)

Judge Scores

• claude sonnet 4 5: 42/100
• gpt 5 codex: 58/100
• kimi k2: 65/100

Rationale

claude sonnet 4 5: This PR adds cost tracking to the usage property across multiple agent implementations, which involves some meaningful logic changes, but has significant limitations for a coding benchmark:

Positive aspects:

1. Real logic changes: Adds cost calculation logic in codex.ts with pricing API integration, including billable vs cached token differentiation
2. Multiple file coordination: Changes span across 9 files with consistent pattern application
3. Type safety improvements: Adds const assertions for models array and generic type parameters
4. Production-relevant: Cost tracking is a real-world feature for AI agent systems

Significant weaknesses:

1. No test coverage: Despite modifying core functionality across multiple agents, zero tests are added or modified
2. Cannot verify correctness: Without tests, there's no way to programmatically verify that:
   • Cost calculations are accurate
   • Pricing API integration works correctly
   • Usage aggregation is correct
   • The changes don't break existing functionality
3. Limited technical depth: Most changes are repetitive additions of cost: 0 and usage.cost += ... patterns
4. Incomplete implementation: The codex.ts file has a dangling fetch call at the bottom that appears to be module-level code, which is questionable design
5. Mixed concerns: Includes unrelated changes like bun.lock formatting and type refactoring alongside the main feature

Verification challenges:

• No way to test if pricing calculations match expected values
• No way to verify API integration works
• No way to ensure backward compatibility
• Success/failure criteria is entirely subjective without tests

This represents a moderate feature addition with real logic, but the complete absence of tests and verification mechanisms makes it unsuitable as a high-quality benchmark task. An AI agent attempting this would have no automated way to confirm correctness.

gpt 5 codex: The PR introduces meaningful logic changes by threading a new cost field through agent usage reporting, adding type refinements, and computing cost for the Codex agent using pricing data. These are real functional modifications with some technical depth. However, there are no accompanying tests, so the new behavior (especially the external pricing fetch and cost aggregation) isn’t verifiable automatically. Without tests or clear verification paths, the changes fall short of benchmark-grade quality despite being substantive.

kimi k2: This PR demonstrates moderate technical quality with meaningful logic changes but lacks test coverage. The changes involve adding cost tracking functionality to AI agent usage metrics across multiple agent implementations (Claude, Codex, OpenCode). Key technical aspects include:

Positive aspects:

• Meaningful logic changes: Adds cost calculation logic using pricing models fetched from external API, modifies usage tracking to include cost field across all agents
• Type safety improvements: Introduces generic types for AgentRegistration and model constraints (TModel extends string)
• Real-world relevance: Cost tracking is essential for production AI systems, making this a practical feature
• Cross-cutting implementation: Changes span multiple files showing understanding of codebase architecture

Negative aspects:

• No test coverage: Zero test files modified or added, making verification difficult
• Incomplete implementation: The diff shows orphaned fetch code at the end of codex.ts that appears disconnected from the main logic
• Limited verification path: While the cost calculations could be tested, no tests exist to validate the new functionality
• Mixed complexity: Some changes are simple field additions while others involve external API calls and pricing calculations

The PR shows good technical understanding of the codebase and implements a valuable feature, but the lack of testing significantly reduces its benchmark suitability. The cost calculation logic has technical depth, but without tests, it's hard to verify correctness programmatically.

Evaluation Feasibility (46/100)

Judge Scores

• claude sonnet 4 5: 35/100
• gpt 5 codex: 25/100
• kimi k2: 78/100

Rationale

claude sonnet 4 5: This PR has several significant challenges for evaluation feasibility:

Size appropriateness (Moderate): 146 lines is within the acceptable range, though the changes are spread across 9 files which adds coordination complexity.

Critical Issues:

1. No description provided: The PR has no description explaining what "other information in the usage property" means. This creates ambiguity about the actual requirements and expected behavior.

2. No test files: The statistics explicitly state "Contains test files: No". Without tests, there's no automated way to verify correctness. This is a major red flag for objective evaluation.

3. Unclear verification method: Without tests or a clear description, evaluating success would require:

   • Manual inspection of the diff to identify patterns
   • Subjective assessment of whether "usage property" changes are correct
   • No clear pass/fail criteria

4. Context-dependent changes: The files modified (agents/claude-code.ts, agents/codex.ts, agents/opencode.ts, cli.ts) suggest this is modifying API usage tracking or reporting. Without seeing the actual diff content or having tests, it's unclear what the correct implementation should look like.

5. Potential external dependencies: The file names suggest this involves AI agent integrations (Claude, Codex) which might require API keys or external services for proper testing, though this cannot be confirmed without the actual diff.

Positive aspects:

• Reasonable diff size
• Appears to be a feature addition rather than a complex refactor
• Multiple related files suggest a cohesive change

Conclusion: The lack of tests and PR description makes objective evaluation very difficult. Success criteria are unclear, and verification would be largely subjective.

gpt 5 codex: The diff size is reasonable, but verification is problematic. The changes touch multiple agent integrations and CLI output, and validating the new “usage” information would require making real API calls to external LLM providers. The repository has no tests covering this behavior, so success can’t be checked deterministically without paid API access. Therefore it’s not feasible to use this PR as an objectively evaluable benchmark task.

kimi k2: This PR is FEASIBLE with minor challenges. At 146 lines across 9 files, it's within the ideal size range (50-500 lines) and represents a meaningful but manageable change. The modifications appear to be adding 'other information' to usage properties across multiple agent files and CLI components, suggesting a coherent feature addition rather than scattered fixes. However, the lack of test files is a significant drawback - we cannot verify correctness through automated tests without knowing the expected behavior. Verification would rely on: 1) Successful build/compilation (likely TypeScript compilation), 2) Diff pattern matching to ensure the 'other information' was added consistently, 3) Manual inspection of the usage property changes. The repository appears to be a standard TypeScript/Node.js project (bun.lock suggests Bun runtime), so environment setup should be straightforward with no special hardware or paid APIs required. The 30-minute completion window is reasonable for 146 lines of mostly additive changes. The main risk is the subjective nature of verifying 'other information' content without explicit test cases, but the structural changes should be objectively verifiable.

Reproducibility (39/100)

Judge Scores

• claude sonnet 4 5: 35/100
• gpt 5 codex: 58/100
• kimi k2: 25/100

Rationale

claude sonnet 4 5: This PR has significant reproducibility challenges:

Major Issues:

1. No PR Description (Critical): The PR has "(No description provided)", making it impossible to understand the intent without examining the code changes. A benchmark task requires deriving a natural prompt from the PR context.

2. Vague Commit Messages: Commits like "update" and "update" provide no information. "add tokens per second metric" and "keep cli aligned with main and add timing" give some hints but are insufficient to understand the full scope.

3. Unclear Task Intent: The title mentions "add other information in the usage property" but doesn't specify what information or why. Without the description, it's unclear what problem this solves or what the acceptance criteria are.

4. Multiple Concerns: The commits suggest at least two different concerns: (1) adding tokens per second metric, (2) keeping CLI aligned with main and adding timing. This suggests the PR may be mixing multiple changes, making it harder to frame as a single coherent task.

Positive Aspects:

1. Clear Base Branch: The base branch (main) is identifiable.

2. Standard Environment: TypeScript/Bun project appears to use standard tooling (bun.lock, .ts files).

3. Self-contained Changes: The modified files are all within the repository, suggesting no external dependencies for the changes themselves.

4. Reasonable Scope: 9 files with ~150 net lines changed is a manageable scope.

Why Low Score:

Without a PR description and with vague commits, it's nearly impossible to derive a natural task prompt that would guide an AI agent to implement similar functionality. A developer would need to reverse-engineer the intent from the code diff itself, which defeats the purpose of a benchmark task. The task cannot be framed as "a real developer request from a PM or lead" because there's no documented request - only implementation artifacts.

To be reproducible as a benchmark, this would need substantial additional context about what "other information" means, why it's needed, and what the expected behavior should be.

gpt 5 codex: The repository and base branch are clear and the environment appears standard, so a starting point is identifiable. However, the PR lacks any description and provides minimal commit messaging, making it difficult to derive an unambiguous task prompt or acceptance criteria without closely reverse-engineering the diff. While likely feasible with effort, the intent and expected outcome are not clearly articulated, which would hinder converting this PR into a clean, reproducible benchmark task.

kimi k2: This PR is poorly reproducible as a benchmark task for several critical reasons:

1. No PR description: The complete absence of a description makes it impossible to understand the intent or requirements. The title 'add other information in the usage property' is cryptic and provides no context about what 'other information' means or what 'usage property' refers to.

2. Unclear task intent: While commit messages mention 'tokens per second metric' and 'timing', there's no clear specification of what should be implemented. The phrase 'add other information' is too vague to derive a concrete task.

3. Repository context issues: The repository 'anomalyco/opencode-bench' appears to be a benchmarking framework itself, suggesting this PR might be modifying internal metrics collection rather than a standalone feature. This creates circular dependency issues for benchmarking.

4. Multiple agent files: Changes span across multiple agent implementations (claude-code.ts, codex.ts, opencode.ts), suggesting this is modifying internal framework behavior rather than adding a user-facing feature.

5. No clear success criteria: Without a description or tests, there's no way to determine what constitutes successful implementation. The 'usage property' could refer to anything from CLI output to internal data structures.

The only positive aspects are that it's a public repository and uses standard TypeScript/Bun environment, but the complete lack of documentation and unclear scope make this unsuitable for conversion to a reproducible benchmark task.

---

Scoring Details

• Base Score: 48.6/100
• Disagreement Penalty: -7.5
• Final Score: 41.1/100

❌ This PR does not appear suitable for the benchmark. See the detailed analysis above for specific issues. You may submit a different PR or provide additional context.

---

Evaluated by opencode-bench on 2026-01-12
Judges: claude sonnet 4 5, gpt 5 codex, kimi k2

[tmickleydoyle]

☝️ Command: orvl evaluate-pr #16