Skip to content

Bug: _check_skill_structure validates body (no frontmatter) instead of full skill text, causing false failures #74

@FONMaster

Description

@FONMaster

Summary

ConstraintValidator._check_skill_structure() always receives skill text without YAML frontmatter, so it perpetually fails with "Skill missing: YAML frontmatter (---)" even when the skill file is perfectly valid.

Root Cause

In evolution/skills/evolve_skill.py, two call sites pass the wrong slice of the parsed skill dict to validate_all():

Baseline check (line ~122):

# BUG: skill["body"] is the markdown body only — no frontmatter
baseline_constraints = validator.validate_all(skill["body"], "skill")

Evolved skill check (line ~189):

# BUG: evolved_body is also frontmatter-stripped; evolved_full has it
evolved_constraints = validator.validate_all(evolved_body, "skill", baseline_text=skill["body"])

skill_module.py documents the dict clearly:

  • "raw" — full file content (frontmatter + body)
  • "frontmatter" — YAML between --- markers
  • "body" — markdown after frontmatter

_check_skill_structure looks for text.strip().startswith("---"), which can never match when called with "body".

Impact

  • Every evolved skill is marked FAILED regardless of actual content quality
  • Evolved output is saved as evolved_FAILED.md and never deployed
  • The constraint gate is entirely non-functional for the skill_structure check
  • baseline_text passed to _check_growth is also wrong ("body" vs "raw"), making the growth-limit comparison slightly incorrect

Fix

# Line ~122 — baseline check
baseline_constraints = validator.validate_all(skill["raw"], "skill")

# Line ~189 — evolved check  
evolved_constraints = validator.validate_all(evolved_full, "skill", baseline_text=skill["raw"])

evolved_full is already computed just above via reassemble_skill(skill["frontmatter"], evolved_body).

Steps to Reproduce

  1. Run any skill evolution with --eval-source sessiondb or synthetic
  2. Observe that evolved output always lands in output/<skill>/evolved_FAILED.md
  3. Check the constraint log — skill_structure will always show ✗

Environment

  • hermes-agent-self-evolution cloned from main (tested 2026-05-10)
  • DSPy fallback to MIPROv2 (GEPA unavailable in installed version)
  • Both video-transcoding and audio-transcription skills affected

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions