feat(ess-maker-skills): add /harden instruction review skill - #247
Draft
amilandi wants to merge 10 commits into
Draft
feat(ess-maker-skills): add /harden instruction review skill#247amilandi wants to merge 10 commits into
amilandi wants to merge 10 commits into
Conversation
Adds a `/harden` capability that reviews an agent's system instructions for internal contradictions and for the gaps that let it answer from something other than its knowledge sources, or offer actions nothing authorized. - `.github/prompts/harden.prompt.md` - setup gate + delegation - `src/skills/instructions/harden/SKILL.md` - the ten-step flow - `src/reference/ess-docs/hardening/instruction-rules.md` - generic rule pack covering contradictions, grounding gaps, over-commitment, and the symmetric over-restriction risk - `scripts/check_instruction_budget.py` - deterministic character-budget probe whose verdict the skill treats as authoritative - registers `harden` in ADK_CAPABILITIES, the menu, and the README The skill asks what the maker has actually seen before analyzing, and does not tighten instructions when nothing specific is wrong: extra prohibitions cause the agent to decline questions its sources answer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
Two simulated end-to-end runs of /harden against a real extracted agent surfaced defects that reading the files could not. - list_agent_capabilities.py: a new probe that reports the agent's topics, workflows and acting steps. Capability checks previously had no resolvable source and were guesswork. Uses a line-based key extractor because Copilot Studio topic YAML contains unquoted '@type' keys and is not valid YAML -- yaml.safe_load fails on exactly the integration topics that matter most. - Keep descriptions out of the probe's JSON. Duplicating them into both the table and the JSON pushed a real agent's output past the caller's limit. - check_instruction_budget.py: accept a slug, a relative path or an absolute path for --agent. config.json stores agent.folder as a path, so the skill's documented invocation produced a doubled path. - SKILL.md: anchor findings at sentence level, state that a missing safeguard is a valid finding without a quoted line, require a concrete failing request before reporting a contradiction, and bound the capability review. - instruction-rules.md: correct the shipped-template claims against the instructions the template actually ships, and stop INSTR-003 from contradicting the contradiction threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
…ion check Three defects from running the skill against a live agent. - The intake question was answered with a menu of categories, and a picked category cannot be anchored to a change. Step 2 now forbids offering options when eliciting what the maker has seen, and requires a prose follow-up when the answer names a category rather than a behavior. - Runs ended on the diff without mentioning validation. Step 10 now runs on every path -- applied, declined, and nothing-proposed -- because reading instructions cannot show that the agent's answers improved. Step 8 no longer tells the model to stop. - Contradictions were only checked against the maker's existing text. Hardening adds prohibitions to a document that already has rules, so Step 6 now re-runs the contradiction pass against the whole candidate, and requires amending a colliding surviving rule rather than layering a stricter rule on top of it. Also: a branch-B maker who asks for a reported risk to be addressed now has a path back into the proposal steps, the Step 1 hard gates are carved out of the always-hand-off rule, and anchoring covers bullets and fragments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
amilandi
force-pushed
the
amilandin/harden-instructions
branch
from
August 25, 2026 02:21
0025160 to
5bbcf70
Compare
- Applying edited agent.mcs.yml and reported it as done, which reads as live. Step 9 now states that Copilot Studio still has the previous instructions until the maker runs /push. - Step 10 routed makers to /test to check an instruction change. /test drives a topic or workflow and does not exercise system instructions, and /evaluate authors cases for the Copilot Studio Evaluation portal, which runs against the agent as deployed. Neither reads the local agent.mcs.yml, so the push has to come first or the evaluation measures the old text. The handoff now says so and orders the steps accordingly. - The progress list was created and never updated. Adopt the fuller house wording used by /cleanup: mark each step in-progress and complete as it happens, which matters here because several steps wait on the maker. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
The handoff told makers that /test is not for system instructions. That is this skill's routing logic, not a next step -- the maker asked what to do and got a disqualification. Mark the explanation internal, recommend only the command that fits, and add a rule covering the general case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
- The proposal summary reported length but said nothing about the contradiction re-check, so a silent pass looked the same as a skipped one. Step 8 now states the result next to the length, including when it is 'none', and names any surviving rule that had to be amended. - A run dropped /push entirely. 'Recommend, don't disqualify' was read as a limit on how many commands to name; it is not, and the applied path must name /push as its own instruction before anything about evaluation. Step 9 now states where the file landed and defers the instruction to Step 10, so the maker is told once rather than twice. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
The harden handoff told makers that /evaluate writes a CSV they upload to the
Copilot Studio Evaluation portal. It does not: evaluations/create writes
.mcs.yml files to {agent.folder}/evaluations/ and pushes them to Copilot
Studio as botcomponent records, which the maker runs from the Evaluation tab.
I took the CSV wording from evaluate.prompt.md, which contradicts the skill it
routes to. Its 'workspace/tests/{date}/' path is referenced nowhere else in
the repo and no such directory exists, so that line is corrected here too
rather than left to mislead the next reader.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
_drive_turn located the message box via a placeholder-based selector and
then called box.press("Enter"). Playwright re-resolves a locator on every
call, and the test pane drops the placeholder attribute once the box has
text -- so the press could not re-resolve the box it had just filled and
timed out after 30s. The prompt was left typed but unsent, surfacing as a
hang rather than a send failure.
Press Enter via the page keyboard so the just-filled, still-focused
element receives it, and prefer the stable data-testid selector for the
input over the placeholder-based one.
Verified end-to-end against a live Copilot Studio test pane: drive_topic.py
now completes a turn and captures the reply instead of timing out.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
Step 10 described /push, /test, and /evaluate incorrectly. /push writes agent.mcs.yml to Copilot Studio through Dataverse and does not publish, so the change reaches the draft the test pane answers from while published channels keep serving the previous instructions. Step 10 now says that, so a maker knows the push alone does not reach their users. /test drives the test pane and captures the reply, which does exercise system instructions -- verified against a live agent, where an instruction pushed without a publish changed the test-pane answers. Step 10 previously claimed the opposite and sent the maker to /evaluate instead. /evaluate authors evaluation cases and is not the shortest path from an instruction change to an observed reply, so it is no longer named here. Also records that a pushed change can be served from a cached definition for several minutes, so stale behaviour is not read as a failed fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
Step 2 told the run it did not have an answer yet whenever the maker named a category rather than a behavior, and not to proceed until it got one. A maker reporting a general concern about responses was asked for a verbatim question/answer pair repeatedly, which reads as a hard requirement for a capability they had just asked for. It also contradicted Step 6, which already has a branch for a theme named without an example and a branch for no reported problem at all. The run had somewhere to go the whole time. Step 2 now follows up once for specificity, then proceeds on whatever the second answer gives it and says what it is working from. The prohibition on offering menus is unchanged -- a menu invites a category, whereas an open question sometimes gets a behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96774230-f664-4887-baf1-a5bbfbd6cbb2
Collaborator
|
We should have evals to show:
|
Collaborator
|
Are there guardrails in place for any parts of our instruction that we consider immutable? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADO: https://o365exchange.visualstudio.com/O365%20Core/_workitems/edit/7774542
Adds a
/hardencapability to the ESS Maker Kit: a review of an agent's system instructions (theinstructions:block inagent.mcs.yml) targeting the failures where an agent answers confidently from something other than its knowledge sources, or offers actions nothing authorized.Draft for review. The flow has been executed end to end twice against a real extracted agent with a scripted maker on the other side — once with a specific complaint, once without. See the validation comment below.
Why this, and why it looks the way it does
Hardening can regress an agent. An agent that declines questions its sources fully answer is also a failure, and it is the one a hardening pass is most likely to introduce. So the skill asks what the maker has actually observed before analyzing, and when nothing specific is wrong it reports findings and proposes only contradiction fixes rather than tightening pre-emptively. Part 4 of the rule pack is dedicated to catching over-restriction in the skill's own proposals.
The contradiction pass runs regardless of what the maker reports. A rule contradicted elsewhere is not in force however firmly it is written, and the author usually has no idea.
What is here
.github/prompts/harden.prompt.mdreview.prompt.mdsrc/skills/instructions/harden/SKILL.mdsrc/reference/ess-docs/hardening/instruction-rules.mdscripts/check_instruction_budget.pyscripts/list_agent_capabilities.pytests/scripts/Plus registration:
ADK_CAPABILITIES, thewiredset intest_adk_telemetry.py,/menu, and the README.The budget probe exists because hardening usually lengthens instructions, and instructions that exceed the ceiling can be silently truncated — which can drop the guardrail the pass just added. Asking the model to count characters was not reliable, so the skill is told the probe's verdict outranks its own estimate. Its error paths report
unknownrather than a falsely clean measurement.No changes to the push path were needed.
agent.mcs.ymlis already collected bycollect_files(), classified asbotcomponentbyclassify_path(), mapped to componenttype 15 in.component-map.json, and covered by the existing checkpoint/rollback.Review notes
agent.mcs.ymlis pushable by inspection of the code path, but whether a type-15 update takes effect against Copilot Studio has not been tested against a real agent. Worth validating before this is announced.--limitso a maker who knows their real ceiling is not blocked by ours.hardenneeds adding to the Aria cube dimension value list per the note inadk_telemetry.py(ADO #7532631). I cannot do that from here.main(installer and foundation-setup tests), unrelated to this change. Everything else passes.