Data:HAL

[borgr]

https://hal.cs.princeton.edu/
They have a lot of agentic data

[borgr]

@Asaf-Yehudai , you and Jatin would have this data right? and a few more, think you can convert it to the schema? Or do we need to contact them?

[Asaf-Yehudai]

@borgr, if we only want to add the model and the agents' results, this will be relatively easy, given the proper schema adaptation. But they also have traces and instance-level results, which will be much more complicated to add. So, let's start by discussing what the schema should look like for agents, and then we can add HAL and contact them if needed.

[borgr]

Ok (and we do want to convert everything, we have the info...)
FYI @nelaturuharsha another reason to think about it (can't tag Elron)

[jatinganhotra]

@Asaf-Yehudai , you and Jatin would have this data right? and a few more, think you can convert it to the schema? Or do we need to contact them?

@borgr There are several new agentic SWE benchmarks which host their own leaderboards and have data in their own Github repos + S3 buckets.

• SWE-Bench Verified
• Multi-SWE-Bench
• SWE-PolyBench

In addition to SWE tasks, there are other agentic tasks e.g. code optimization at a repo level, which have their own leaderboards similar to SWE leaderboards:

• GSO-bench https://gso-bench.github.io/ --> experiments data in https://github.com/gso-bench/gso-experiments
• SWEfficiency https://swefficiency.com/ --> outputs in https://github.com/swefficiency/swefficiency/blob/swefficiency_base/predictions/converted/oh_claude46opus.jsonl and https://github.com/swefficiency/swefficiency/blob/swefficiency_base/eval_reports/eval_report_claude-3-7-sonnet-20250219_maxiter_100_N_v0.51.1-no-hint-run_1.csv
  This would require update to the schema if we want to ingest everything, e.g. the additional data (other than correctness) used for eval is:

raw_pred_speedup_ratio	pred_speedup_ratio	gold_speedup_ratio	human_speedup_ratio

[borgr]

Do we have all this data processed? Is the processing through code (no manual download or something). If so, maybe aim to contribute it after neurips (to still qualify for the shared task and be in time for the initial paper, agentic stuff would be a great addition, and of course contributors will be a part which is another motivation in terms of times).

[borgr]

@jatinganhotra reviving this. We will start writing the Neurips paper, and agentic data will be a nice addition