fix(skills): pass evolved_full to skill_structure validator#71
Open
jlvitorasso wants to merge 1 commit into
Open
fix(skills): pass evolved_full to skill_structure validator#71jlvitorasso wants to merge 1 commit into
jlvitorasso wants to merge 1 commit into
Conversation
The `skill_structure` constraint checks for YAML frontmatter (`---`), `name` field, and `description` field. It was being called with `evolved_body` (the body only), which meant that constraint *always* failed — every evolution was routed to `evolved_FAILED.md` even when the reassembled skill (`evolved_full`) was well-formed. Repro: 1. Run `python -m evolution.skills.evolve_skill --skill <any> --iterations 5 --eval-source synthetic` — the optimizer completes 10 trials, finds a best-scoring program, then validation fails with: `✗ skill_structure: Skill missing: YAML frontmatter (---), name field, description field`. 2. The output dir contains `evolved_FAILED.md` (saved on line ~206 from `evolved_full`, which IS well-formed) — confirming the only thing wrong was the validator input. Fix: pass `evolved_full` (built one line earlier on L185 via `reassemble_skill(skill["frontmatter"], evolved_body)`) instead of `evolved_body`. After the fix, the same run lands in `output/<skill>/<timestamp>/evolved_skill.md` with all 4 constraints passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evolve_skill.evolve()calls the constraints validator withevolved_body(the body only), but theskill_structureconstraint checks for YAML frontmatter (---),name, anddescription. That constraint always failed because those live in the frontmatter, not the body.Result: every evolution — including high-scoring ones — was routed to
output/<skill>/evolved_FAILED.mdeven when the reassembledevolved_fullskill was completely well-formed (and got saved asevolved_FAILED.mdimmediately after, with all the metadata intact).Fix
Pass
evolved_full(assembled one line earlier viareassemble_skill(skill[\"frontmatter\"], evolved_body)) to the validator instead ofevolved_body. One-line change.Repro before fix
Output (10 trials complete, optimizer succeeds):
evolved_FAILED.mdon disk has the full frontmatter + body — confirming the only thing wrong was the validator input.Repro after fix (same command)
Test plan
evolve_skillwith synthetic eval source on a real skill (github-code-review) — verified output now lands inoutput/<skill>/<timestamp>/evolved_skill.mdwith all 4 constraints passing.evolved_FAILED.mdsave) still usesevolved_full(unchanged at L206), so behavior on truly malformed evolutions is preserved.