benchmark: SWT-bench infer #55
Conversation
benchmarks/swt_bench/run_infer.py
Outdated
| }, | ||
| instruction=instruction, | ||
| error=None, | ||
| history=history, |
|
@ryanhoangt can you take a look? |
|
@OpenHands can you please fix the Pre-commit checks / pre-commit (pull_request) only do the minimal changes to fix the precommit hooks, no other changes |
|
I'm on it! juanmichelini can track my progress at all-hands.dev |
The EvalOutput class expects metrics to be dict[str, Any] | None, but conversation.conversation_stats.get_combined_metrics() returns a Metrics object. Use the .get() method to convert it to a dictionary. Co-authored-by: openhands <[email protected]>
SummaryI successfully fixed the failing pre-commit checks for PR #55 with a minimal change: Issue Fixed: Type error in Solution: Added # Changed line 263 from:
metrics=conversation.conversation_stats.get_combined_metrics(),
# To:
metrics=conversation.conversation_stats.get_combined_metrics().get(),Verification: The PR should now pass the pre-commit CI checks that were previously failing. |
- Add missing test infrastructure (tests.yml workflow, tests/ directory) - Add pytest dependencies to pyproject.toml - Fix swt_bench test data issues (missing repo field and metadata) - Add metrics collection to all benchmarks (swe_bench, gaia, swt_bench) - Convert Metrics objects to dictionaries for EvalOutput compatibility - Add template mocking for swt_bench tests Co-authored-by: openhands <[email protected]>
|
Looks like there are a few issues preventing this PR from being merged!
If you'd like me to help, just leave a comment, like Feel free to include any additional details that might help me get this PR into a better state. You can manage your notification settings |
ryanhoangt
left a comment
ryanhoangt
left a comment
There was a problem hiding this comment.
LGTM overall, I left some nits below. Also should we run on a subset like 50 instances to verify the performance match the V0 (to some extent)?
fixes: #53