Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can the model see the issue's GitHub discussion thread? #61

Open
js-d opened this issue Mar 13, 2025 · 0 comments
Open

Can the model see the issue's GitHub discussion thread? #61

js-d opened this issue Mar 13, 2025 · 0 comments

Comments

@js-d
Copy link

js-d commented Mar 13, 2025

Based on Appendix A.9 in the paper, issue 381 with the missing zip-code validation error seems to be graded based on whether the model improves broader validation (e.g., country-specific regex dictionaries), as discussed in the issue's Github thread.

Does the model get to see the original GitHub discussion? If not, wouldn’t it be unfair to penalize the model for only fixing the issue as it was stated originally?

I might be misunderstanding what information the model is given in context: is it only: what's in issue_data.json, the state of the repo, and what it can glean from the user_tool?

This seems particularly important because AFAICT issue 381 is the IC-SWE issue with the highest payout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant