Skip to content

fix(skills): pass evolved_full to skill_structure validator#71

Open
jlvitorasso wants to merge 1 commit into
NousResearch:mainfrom
jlvitorasso:fix/validator-pass-evolved-full-skill-structure
Open

fix(skills): pass evolved_full to skill_structure validator#71
jlvitorasso wants to merge 1 commit into
NousResearch:mainfrom
jlvitorasso:fix/validator-pass-evolved-full-skill-structure

Conversation

@jlvitorasso
Copy link
Copy Markdown

Summary

evolve_skill.evolve() calls the constraints validator with evolved_body (the body only), but the skill_structure constraint checks for YAML frontmatter (---), name, and description. That constraint always failed because those live in the frontmatter, not the body.

Result: every evolution — including high-scoring ones — was routed to output/<skill>/evolved_FAILED.md even when the reassembled evolved_full skill was completely well-formed (and got saved as evolved_FAILED.md immediately after, with all the metadata intact).

Fix

Pass evolved_full (assembled one line earlier via reassemble_skill(skill[\"frontmatter\"], evolved_body)) to the validator instead of evolved_body. One-line change.

Repro before fix

python -m evolution.skills.evolve_skill \
  --skill github-code-review \
  --iterations 5 \
  --eval-source synthetic \
  --optimizer-model openai/MiniMax-M2.7 \
  --eval-model openai/MiniMax-M2.7

Output (10 trials complete, optimizer succeeds):

Returning best identified program with score 60.87!
  Optimization completed in 364.3s

Validating evolved skill
  ✓ size_limit: Size OK: 13161/15000 chars
  ✓ non_empty: Artifact is non-empty
  ✗ skill_structure: Skill missing: YAML frontmatter (---), name field, description field
✗ Evolved skill FAILED constraints — not deploying
  Saved failed variant to output/github-code-review/evolved_FAILED.md

evolved_FAILED.md on disk has the full frontmatter + body — confirming the only thing wrong was the validator input.

Repro after fix (same command)

Validating evolved skill
  ✓ size_limit: Size OK: 13481/15000 chars
  ✓ growth_limit: Growth OK: +2.4% (max +20.0%)
  ✓ non_empty: Artifact is non-empty
  ✓ skill_structure: Skill has valid frontmatter (name + description)
  Output saved to output/github-code-review/20260509_224844/
✓ Evolution improved skill by +0.185 (+42.5%)

Test plan

  • Run evolve_skill with synthetic eval source on a real skill (github-code-review) — verified output now lands in output/<skill>/<timestamp>/evolved_skill.md with all 4 constraints passing.
  • Failed-constraint path (the evolved_FAILED.md save) still uses evolved_full (unchanged at L206), so behavior on truly malformed evolutions is preserved.

The `skill_structure` constraint checks for YAML frontmatter (`---`),
`name` field, and `description` field. It was being called with
`evolved_body` (the body only), which meant that constraint *always*
failed — every evolution was routed to `evolved_FAILED.md` even when
the reassembled skill (`evolved_full`) was well-formed.

Repro:
1. Run `python -m evolution.skills.evolve_skill --skill <any> --iterations 5
   --eval-source synthetic` — the optimizer completes 10 trials, finds a
   best-scoring program, then validation fails with:
   `✗ skill_structure: Skill missing: YAML frontmatter (---), name field,
   description field`.
2. The output dir contains `evolved_FAILED.md` (saved on line ~206 from
   `evolved_full`, which IS well-formed) — confirming the only thing
   wrong was the validator input.

Fix: pass `evolved_full` (built one line earlier on L185 via
`reassemble_skill(skill["frontmatter"], evolved_body)`) instead of
`evolved_body`. After the fix, the same run lands in
`output/<skill>/<timestamp>/evolved_skill.md` with all 4 constraints
passing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant