fix: load_skill handles dir paths; validate evolved_full not body#51
Open
shilei103-afk wants to merge 1 commit into
Open
fix: load_skill handles dir paths; validate evolved_full not body#51shilei103-afk wants to merge 1 commit into
shilei103-afk wants to merge 1 commit into
Conversation
Two bugs prevented the self-evolution pipeline from completing: 1. skill_module.py load_skill(): when find_skill() returns a directory path (e.g. /skills/github/github-code-review/), load_skill() tried to read it as a file, got empty/None content, and all constraint checks failed. Fixed by appending /SKILL.md when path is a directory. 2. evolve_skill.py: validate_all() was called on evolved_body (markdown only, no frontmatter) instead of evolved_full (frontmatter + body), so skill_structure constraint always failed. Also, holdout evaluation crashes (DSPy JSONAdapter on bash output) now caught so constraints- passed skill is always saved.
|
I did a quick pass over this because it overlaps with the local/fork fixes from #38/#49. Useful parts:
One thing to double-check before merge:
Potential concern:
|
This was referenced May 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two bugs prevented the self-evolution pipeline from completing:
Bug 1:
load_skill()receives directory path, reads empty contentIn
evolution/skills/skill_module.py,find_skill()returns a directory path (e.g./skills/github/github-code-review/).load_skill()tried to read this path as a file directly, got empty content, and all constraint checks failed silently.Fix: detect when path is a directory and append
/SKILL.md.Bug 2:
validate_all()called onevolved_bodyinstead ofevolved_fullIn
evolution/skills/evolve_skill.py, constraint validation was called onevolved_body(markdown body only, no frontmatter), soskill_structureconstraint always failed because it checks for YAML frontmatter---andname:field.Fix: validate
evolved_full(frontmatter + body) instead.Bonus: holdout eval crash now caught
DSPy JSONAdapter crashes when the skill outputs bash commands. This is now wrapped in try/except so constraints-passed skills are always saved.
Testing
✓ skill_structure: Skill has valid frontmatter (name + description)output/github-code-review/<timestamp>/