Skip to content

docs(benchmark): SWE-bench results at matched-50 + home-page rework#15

Merged
OsherElhadad merged 1 commit into
mainfrom
release/cache-aware-v2
Jul 25, 2026
Merged

docs(benchmark): SWE-bench results at matched-50 + home-page rework#15
OsherElhadad merged 1 commit into
mainfrom
release/cache-aware-v2

Conversation

@OsherElhadad

Copy link
Copy Markdown
Collaborator

Summary

Updates the live SWE-bench Verified benchmark story to the matched-50 set and reworks the docs home page so the numbers are legible and stated as percentages.

The two tasks that previously hit infrastructure exceptions in the context-guru arm (sympy__sympy-23413, django__django-14792) were rerun — both now score cleanly and both solve. All three arms (baseline / context-guru / headroom) now cover all 50 tasks with zero exceptions, so the comparison set grows 48 → 50 and every number is apples-to-apples on the same tasks.

Headline (matched-50, live · Claude Code · aws/claude-sonnet-5)

dimension baseline context-guru headroom
tasks solved 86% 88% 80%
billed cost vs baseline −13.2% −5.3%
cache-read tokens vs baseline −17.8% −6.3%
cache-write tokens vs baseline −0.4% −0.9%
mean steps / task vs baseline −13.9% −2.8%
  • context-guru is the cheapest arm and now solves the most tasks of the three.
  • Cache-write is a three-way wash (all within ~1% of baseline) — none of the arms busts the cache. The earlier "context-guru wins cache-write" claim is corrected to a tie.
  • headroom keeps its one edge: lower added latency (63 vs 117 ms/req), since it runs no model on the hot path.

Changes

  • Rows / analysis: merged the clean reruns into the canonical context-guru rows; regenerated the three-way deep analysis and the 6 benchmark plots at matched-50.
  • Docs: context-guru.md (50 tasks / 44 solved), comparison.md (headline + verdict + honest caveat), RESULTS.md, components.md head-to-head, README.md headline, and charts.js inline chart data — all now percentage-first and consistent.
  • Home page: reworked the "The numbers" section into a clear bottom-line conclusion + percentage stat cards with comparison context; bumped several too-small font sizes in the stylesheet.

mkdocs build --clean is warning-free; the full three-way comparison, per-task tables, and reproduce guide are all in the Benchmarks nav.

… page

Reran the 2 tasks that hit infrastructure exceptions in the context-guru arm
(sympy__sympy-23413, django__django-14792). Both now score cleanly and both
solve, so all three arms cover all 50 tasks with zero exceptions and the matched
comparison set grows 48 -> 50.

Updated results (matched-50), stated as percentages:
- context-guru solves the most tasks: 88% (baseline 86%, headroom 80%).
- Cheapest arm: -13.2% billed cost vs baseline (headroom -5.3%), driven by a
  -17.8% cache-read reduction; cache-write stays within 1% of baseline across
  all three arms, so none of them busts the cache.
- Corrected the earlier "context-guru wins cache-write" claim to a three-way
  tie (all within 0.9%); headroom keeps its latency edge (63 vs 117 ms/req).

Regenerated context-guru.md, comparison.md, RESULTS.md, components.md head-to-
head, the 6 benchmark plots, and charts.js inline data. Reworked the docs home
page "The numbers" section into a clear bottom-line + percentage stat cards, and
bumped several too-small font sizes in the stylesheet.

Assisted-By: Claude (Opus 4.8)
Signed-off-by: Osher Elhadad <osher.elhadad@live.biu.ac.il>
@OsherElhadad
OsherElhadad merged commit 07d324d into main Jul 25, 2026
6 checks passed
@github-project-automation github-project-automation Bot moved this from New/ToDo to Done in Rossoctl Issue Prioritization Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants