Created by GitHub Ace · View Session
Draft for github/gh-aw — the body below is written from gh-aw's viewpoint and is ready to copy/paste into that repo. Delete this blockquote when filing. Tracked here only because it cannot be filed cross-repo from this environment.
Blocked on: #916, #954, #955 merging and a promoted detector release.
Context
gh-aw-threat-detection is landing a change that makes threat verdicts subject to structural eligibility: a category can only be raised if the artifacts actually contain a channel it could have come from. malicious_patch with zero patch files is now rejected as an unusable report rather than recorded as a finding. This was motivated by four false positives in our own runs — 32201512271, 32203115530, 32239956539, 32305540872 — where the detector scored instruction non-compliance and framework-rejected validation errors as security threats.
Eligibility is computed from declared channels, each carrying two independent signals: present (content was found) and uninspectable (the channel may hold content the detector could not read). Either one grants eligibility, so the check fails open — a staging fault can never suppress a real threat.
That design has a direct consequence for us: the detector can only see what gh-aw stages. Everything below is host-side work the detector cannot do for itself.
1. Stage evidence for channels the detector currently cannot see
This is the substantive item.
The detector's eligibility is scoped to the artifact bundle. MCP tool results, fetched web content, and the agent's own engine transcript are never staged as artifacts, so an injection delivered only through those channels leaves no trace in the analyzed inputs. The detector now documents this limitation explicitly rather than implying coverage it does not have.
The detector side of closing the gap is cheap — adding a channel is a few lines:
{
name: "an MCP tool result recorded during the run",
present: len(arts.MCPToolResults) > 0,
uninspectable: arts.HasWarningForField("mcp_tool_results"),
}But it stays inert until we write that data into the artifacts directory. The ask is to decide what is feasible to stage, in rough order of value:
- MCP tool results — the highest-value gap. A tool result is untrusted content that reaches the instruction channel by design, which is exactly the shape
prompt_injection is meant to catch.
- Fetched web content — same argument, if we can capture it at the fetch boundary.
- Outbound request metadata — even destinations without bodies would let the detector reason about exfiltration attempts it currently cannot observe at all.
Two constraints are worth settling up front, because they shape the format more than the content does:
- Size. These can be large and are wholly attacker-influenced. They need bounding at the staging boundary, the way
aw_info.json is already allowlisted and size-capped.
- Provenance. The detector distinguishes trusted template text from untrusted input. Staged tool results must be unambiguously marked untrusted, or they will degrade the trusted/untrusted analysis rather than improve it.
Scope note. Detection is a gate on downstream actions, not a post-hoc session audit. A secret exfiltrated mid-run has already left before detection runs — network boundaries and MCP constraints remain the defense for that, and this work does not change it. The value of staging tool results is catching injection that shaped the staged output, not retroactively catching exfiltration.
2. Plumb GH_AW_DETECTION_CONTINUE_ON_WARNING
Depends on #955.
The detector is gaining a second gate, separate from CONTINUE_ON_ERROR: when set to false, it refuses to return a clean verdict from a bundle it could not fully inspect, exiting 2 with reason=config_error. The default is true, so nothing changes for existing workflows.
The compiler already emits the sibling knob as a literal in the detection job — e.g. in smoke-copilot-standalone.lock.yml:
env:
GH_AW_DETECTION_CONTINUE_ON_ERROR: "false"
The new variable needs the same treatment: a frontmatter option, emitted into both the detection and conclude steps.
Keep the two flags separate. They express different host policies. CONTINUE_ON_ERROR=false means "the primary inputs must be staged correctly" — it is about our staging contract. CONTINUE_ON_WARNING=false means "I will not accept a clean verdict from a partially-read bundle" — it is about assurance strength. A host may reasonably want one without the other, and folding them together would silently tighten every workflow already running strict mode.
Recommend landing #954 first and reading the resulting warnings: [] data to see how often this actually fires before enabling it anywhere.
3. Audit comment-memory and patch staging
The detector previously described an unreadable channel to the model as empty — identical wording to a channel that genuinely holds nothing. The model then reported clean about content nobody looked at, and the run exited 0. That is now fixed: such channels are described as unexamined, and two previously-silent refusal paths in comment-memory loading emit warnings.
The practical consequence for us is that staging faults that were previously invisible will start producing ERR_VALIDATION warnings. Two are worth checking proactively:
- A
comment-memory path that is not a real directory. The detector refuses to follow a symlink here deliberately — following it would let the run under analysis point the detector at markdown outside the artifacts tree. If we ever stage this as a link, it will now warn.
HAS_PATCH=true with no readable, non-empty aw-*.patch / aw-*.bundle reaching the artifacts directory.
Neither is a new failure; both are newly visible. Expect some noise on first rollout, and treat it as pre-existing staging bugs surfacing rather than as a regression in the detector.
Explicitly not needed
No change is required for warnings: [] rendering. actions/setup/sh/conclude_threat_detection.sh on main already delegates to threat-detect conclude for everything except reporting its own absence from PATH:
exec threat-detect conclude \
--result-file "${RESULT_FILE}" \
--detection-log "${DETECTION_LOG_FILE}"So the ⚠️ warnings block from #954 will appear in compiled workflows' job logs automatically once a detector release carrying it is promoted. (This supersedes older notes in the detector repo claiming the script reads detection_result.json directly — that was true before the delegation landed.)
No recompile is needed to pick up the detector change. The locks emit the literal latest, which install_threat_detect_binary.sh resolves at run time to the newest non-prerelease detector release. Promoting the release is sufficient.
Checklist
Blocked on: #916, #954, #955 merging and a promoted detector release.
Context
gh-aw-threat-detectionis landing a change that makes threat verdicts subject to structural eligibility: a category can only be raised if the artifacts actually contain a channel it could have come from.malicious_patchwith zero patch files is now rejected as an unusable report rather than recorded as a finding. This was motivated by four false positives in our own runs — 32201512271, 32203115530, 32239956539, 32305540872 — where the detector scored instruction non-compliance and framework-rejected validation errors as security threats.Eligibility is computed from declared channels, each carrying two independent signals:
present(content was found) anduninspectable(the channel may hold content the detector could not read). Either one grants eligibility, so the check fails open — a staging fault can never suppress a real threat.That design has a direct consequence for us: the detector can only see what gh-aw stages. Everything below is host-side work the detector cannot do for itself.
1. Stage evidence for channels the detector currently cannot see
This is the substantive item.
The detector's eligibility is scoped to the artifact bundle. MCP tool results, fetched web content, and the agent's own engine transcript are never staged as artifacts, so an injection delivered only through those channels leaves no trace in the analyzed inputs. The detector now documents this limitation explicitly rather than implying coverage it does not have.
The detector side of closing the gap is cheap — adding a channel is a few lines:
{ name: "an MCP tool result recorded during the run", present: len(arts.MCPToolResults) > 0, uninspectable: arts.HasWarningForField("mcp_tool_results"), }But it stays inert until we write that data into the artifacts directory. The ask is to decide what is feasible to stage, in rough order of value:
prompt_injectionis meant to catch.Two constraints are worth settling up front, because they shape the format more than the content does:
aw_info.jsonis already allowlisted and size-capped.Scope note. Detection is a gate on downstream actions, not a post-hoc session audit. A secret exfiltrated mid-run has already left before detection runs — network boundaries and MCP constraints remain the defense for that, and this work does not change it. The value of staging tool results is catching injection that shaped the staged output, not retroactively catching exfiltration.
2. Plumb
GH_AW_DETECTION_CONTINUE_ON_WARNINGDepends on #955.
The detector is gaining a second gate, separate from
CONTINUE_ON_ERROR: when set tofalse, it refuses to return a clean verdict from a bundle it could not fully inspect, exiting2withreason=config_error. The default istrue, so nothing changes for existing workflows.The compiler already emits the sibling knob as a literal in the detection job — e.g. in
smoke-copilot-standalone.lock.yml:The new variable needs the same treatment: a frontmatter option, emitted into both the detection and conclude steps.
Keep the two flags separate. They express different host policies.
CONTINUE_ON_ERROR=falsemeans "the primary inputs must be staged correctly" — it is about our staging contract.CONTINUE_ON_WARNING=falsemeans "I will not accept a clean verdict from a partially-read bundle" — it is about assurance strength. A host may reasonably want one without the other, and folding them together would silently tighten every workflow already running strict mode.Recommend landing #954 first and reading the resulting
warnings: []data to see how often this actually fires before enabling it anywhere.3. Audit comment-memory and patch staging
The detector previously described an unreadable channel to the model as empty — identical wording to a channel that genuinely holds nothing. The model then reported clean about content nobody looked at, and the run exited
0. That is now fixed: such channels are described as unexamined, and two previously-silent refusal paths in comment-memory loading emit warnings.The practical consequence for us is that staging faults that were previously invisible will start producing
ERR_VALIDATIONwarnings. Two are worth checking proactively:comment-memorypath that is not a real directory. The detector refuses to follow a symlink here deliberately — following it would let the run under analysis point the detector at markdown outside the artifacts tree. If we ever stage this as a link, it will now warn.HAS_PATCH=truewith no readable, non-emptyaw-*.patch/aw-*.bundlereaching the artifacts directory.Neither is a new failure; both are newly visible. Expect some noise on first rollout, and treat it as pre-existing staging bugs surfacing rather than as a regression in the detector.
Explicitly not needed
No change is required for
warnings: []rendering.actions/setup/sh/conclude_threat_detection.shonmainalready delegates tothreat-detect concludefor everything except reporting its own absence fromPATH:So the⚠️ warnings block from #954 will appear in compiled workflows' job logs automatically once a detector release carrying it is promoted. (This supersedes older notes in the detector repo claiming the script reads
detection_result.jsondirectly — that was true before the delegation landed.)No recompile is needed to pick up the detector change. The locks emit the literal
latest, whichinstall_threat_detect_binary.shresolves at run time to the newest non-prerelease detector release. Promoting the release is sufficient.Checklist
GH_AW_DETECTION_CONTINUE_ON_WARNINGfrontmatter option; emit into detection + conclude stepscomment-memoryand patch staging against the newly-visible warnings