docs(benchmark): SWE-bench results at matched-50 + home-page rework#15
Merged
Conversation
… page Reran the 2 tasks that hit infrastructure exceptions in the context-guru arm (sympy__sympy-23413, django__django-14792). Both now score cleanly and both solve, so all three arms cover all 50 tasks with zero exceptions and the matched comparison set grows 48 -> 50. Updated results (matched-50), stated as percentages: - context-guru solves the most tasks: 88% (baseline 86%, headroom 80%). - Cheapest arm: -13.2% billed cost vs baseline (headroom -5.3%), driven by a -17.8% cache-read reduction; cache-write stays within 1% of baseline across all three arms, so none of them busts the cache. - Corrected the earlier "context-guru wins cache-write" claim to a three-way tie (all within 0.9%); headroom keeps its latency edge (63 vs 117 ms/req). Regenerated context-guru.md, comparison.md, RESULTS.md, components.md head-to- head, the 6 benchmark plots, and charts.js inline data. Reworked the docs home page "The numbers" section into a clear bottom-line + percentage stat cards, and bumped several too-small font sizes in the stylesheet. Assisted-By: Claude (Opus 4.8) Signed-off-by: Osher Elhadad <osher.elhadad@live.biu.ac.il>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updates the live SWE-bench Verified benchmark story to the matched-50 set and reworks the docs home page so the numbers are legible and stated as percentages.
The two tasks that previously hit infrastructure exceptions in the context-guru arm (
sympy__sympy-23413,django__django-14792) were rerun — both now score cleanly and both solve. All three arms (baseline / context-guru / headroom) now cover all 50 tasks with zero exceptions, so the comparison set grows 48 → 50 and every number is apples-to-apples on the same tasks.Headline (matched-50, live · Claude Code ·
aws/claude-sonnet-5)Changes
context-guru.md(50 tasks / 44 solved),comparison.md(headline + verdict + honest caveat),RESULTS.md,components.mdhead-to-head,README.mdheadline, andcharts.jsinline chart data — all now percentage-first and consistent.mkdocs build --cleanis warning-free; the full three-way comparison, per-task tables, and reproduce guide are all in the Benchmarks nav.