Skip to content

fix: make 'best' parent-selection mode reachable from CLI and sort by highest score#32

Open
mvanhorn wants to merge 1 commit into
jennyzzt:mainfrom
mvanhorn:fix/31-best-parent-selection
Open

fix: make 'best' parent-selection mode reachable from CLI and sort by highest score#32
mvanhorn wants to merge 1 commit into
jennyzzt:mainfrom
mvanhorn:fix/31-best-parent-selection

Conversation

@mvanhorn

Copy link
Copy Markdown

Summary

Fix the argparse choices list in DGM_outer.py by adding the missing comma so it reads ['random', 'score_prop', 'score_child_prop', 'best']. In choose_selfimproves(), add reverse=True to the sorted(...) call in the elif method == 'best': branch so the highest accuracy_score candidates are chosen first. Add a new root-level test_dgm_outer.py (mirroring the existing root-level test_swebench.py convention; the repo has a pytest.ini) with a unit test that calls choose_selfimproves(..., method='best') against synthetic candidate metadata (three fake commits with distinct scores, using tmp_path-backed metadata or monkeypatched loading as the function's data access requires) and asserts the top-scoring parent is selected first, plus a test that the argparse parser accepts both best and score_child_prop.

Why this matters

DGM_outer.py has two related bugs in the best parent-selection path. First, the argparse choices list for --choose_selfimproves_method is written as ['random', 'score_prop', 'score_child_prop' 'best'] - the missing comma makes Python concatenate the adjacent string literals into 'score_child_propbest', so best (and an explicitly passed score_child_prop) are rejected by the CLI. Second, the method == 'best' branch in choose_selfimproves() sorts candidates by accuracy_score in ascending order and takes the first N, so it selects the lowest-scoring parents instead of the highest-scoring ones. Because parent selection is the search pressure in DGM, an inverted best strategy silently corrupts any experiment comparing selection strategies.

See #31.

Testing

  • Happy path: choose_selfimproves(..., method='best') with three synthetic candidates scoring 0.2/0.5/0.9 and selfimprove_size=1 returns the 0.9-score commit. - selfimprove_size=2 returns the top two commits in descending score order. - selfimprove_size larger than the candidate count returns all candidates without error. - CLI parser accepts --choose_selfimproves_method best and --choose_selfimproves_method score_child_prop; an invalid value (e.g.

Fixes #31

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

best parent-selection mode is unreachable from CLI and selects lowest scores when called

1 participant