Skip to content

Prevent native tools from falling through to shell - #2050

Merged
Aaronontheweb merged 13 commits into
fix/repair-tool-rollout-contractsfrom
feat/native-tool-affordance
Aug 26, 2026
Merged

Prevent native tools from falling through to shell#2050
Aaronontheweb merged 13 commits into
fix/repair-tool-rollout-contractsfrom
feat/native-tool-affordance

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

Stacked on #2046.

What changed

  • Makes attach_file a policy-filtered parent-session Core tool and directs agents to attach the authorized source path without a preparatory copy.
  • Keeps attach_file unavailable to subagents until an internal child-to-parent attachment handoff exists.
  • Detects exact, policy-visible first-party tool names in complete ShellSyntaxTree command facts before shell grants, approval, or execution.
  • Returns one typed correction, exposes a deferred target schema actor-locally, and sends the eventual native call through normal authorization.
  • Preserves hard-deny precedence and excludes dynamic, incomplete, path-qualified, fuzzy, hidden, denied, MCP, and shell targets.

Verification

  • Release build: 0 warnings, 0 errors.
  • Full solution tests: 7,799 passed; 17 expected environment-gated skips; 0 failures.
  • Strict OpenSpec, headers, changed-file formatting, shell harness checks, diff check, and Slopwatch: pass.
  • Frozen-SHA adversarial review: pass.
  • PII-free behavioral aggregates are posted separately.

This PR does not change public or durable contracts.

@Aaronontheweb Aaronontheweb added enhancement New feature or request security Security-related changes sessions LLM session actor, turn lifecycle, pipelines subagents spawn_agent, SubAgentActor, definition loader, discovery context layer, and related features shell Issues related to the shell tool, since it has the largest security perimeter. tests All issues related to testing, quality assurance, and smoke testing. UX/DX UI / UX / DX friction issue or user-facing annoyances. tools Issues related to agent tools: file_read, web_search, shell_execute, image processing, etc. evals LLM efficacy evals for Netclaw labels Aug 21, 2026
@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

PII-free behavioral evidence from fixed synthetic prompts, five fresh sessions per variant:

Scenario Baseline Treatment
Direct attachment exact completion 0/5 5/5
Direct attachment tool calls 11 5
Direct attachment model requests 15 10
Native recovery exact completion 0/5 4/5
Native recovery tool calls 16 11
Native recovery model requests 19 16

Direct attachment treatment used exactly one attach_file call in every run, with no shell or file-mutation prelude. Baseline required load_tool in every run, and one run also read the source before attaching.

For native recovery, all five treatment runs authored the requested shell mistake, all five were intercepted before process start, all five emitted the typed correction, all five activated the deferred native schema, and all five retried natively. Treatment produced zero shell process starts and zero approval requests. One treatment run called the native tool twice, so the strict exact-sequence score is 4/5 rather than 5/5. Baseline emitted no typed correction and hit the approval gate in all five runs.

Only aggregate counts are published here; raw artifacts remain local.

Comment thread src/Netclaw.Actors.Tests/Tools/ToolFrictionReplayTests.cs Fixed
@Aaronontheweb

Aaronontheweb commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

Specification-quality update is now on the PR at d399e009 and the branch is restacked onto #2046.

What changed:

  • Linked the shared engineering glossary from every affected capability spec.
  • Added concrete positive and negative examples for exact native-tool shell mistakes, attachment behavior, child exclusions, authority, and persistence.
  • Added diagrams showing the actual ShellSyntaxTree to terminal preflight to correction to exposure to later authorization flow.
  • Distinguished the closed UseNativeTool remediation code from the NativeToolSuggested fact that carries the exact registered name.
  • Marked the superseded json_read, file_read_many, and earlier attach_file decisions in the original pit-of-success change.
  • Added a real-child regression proving that shell calls naming attach_file and spawn_agent do not reveal or expose child-denied tools.

Validation:

  • Focused child regression: 1/1 passed.
  • Strict OpenSpec validation: both affected changes passed.
  • Header verification and diff check passed.
  • Range-diff after restacking: all prior Prevent native tools from falling through to shell #2050 commits are patch-identical; only this new quality commit was added.

This update is documentation and regression coverage only; it does not change production behavior, so the previously recorded behavioral eval evidence remains the relevant evidence for the implementation. No hosted eval was rerun for this cleanup.

@Aaronontheweb
Aaronontheweb force-pushed the feat/native-tool-affordance branch from d399e00 to fd81c56 Compare August 25, 2026 20:34
@Aaronontheweb
Aaronontheweb force-pushed the feat/native-tool-affordance branch from fd81c56 to a17197c Compare August 25, 2026 20:52
@Aaronontheweb
Aaronontheweb force-pushed the feat/native-tool-affordance branch from a17197c to 2b3e7ea Compare August 25, 2026 20:58
@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Specification and verification follow-up is available at 4455fbba.

Changes:

  • Defined policy-visible tools, native Netclaw tools, and skill resources in the shared glossary.
  • Clarified that skill_read_resource reads additional skill files. skill_load reads SKILL.md.
  • Added exact correction, source-order, authority, and schema-exposure examples.
  • Added actor tests for policy changes and missing registrations before activation.
  • Added an executor test that proves a compound correction prevents earlier shell side effects.

Validation:

  • OpenSpec strict validation passed with 31 of 31 tasks complete.
  • OpenSpec Verify found no remaining warnings.
  • The focused specification suite passed 89 of 89 tests.
  • The full local suite passed 7,808 tests. It skipped 17 environment-specific tests.
  • Header verification, diff checks, and Slopwatch passed with zero findings.

This follow-up changes specifications and regression proof only. It does not change production behavior. The existing PII-free behavioral eval results remain applicable. No raw transcript data appears in this update.

Comment thread src/Netclaw.Actors.Tests/Tools/DispatchingToolExecutorTests.cs Fixed
@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Windows follow-up is fixed at 57d853fb.

The failing test was introduced by the previous specification-proof commit. It depended on a host-specific PowerShell command and filesystem side effect. Windows took a different authorization path, so the test expected the wrong terminal result.

The replacement is deterministic:

  • It runs the same authorization proof against explicit Bash and PowerShell grammars.
  • It replaces the host shell with a recording tool.
  • It proves the correction selects file_read.
  • It proves Netclaw never dispatches the shell tool.
  • It starts no process and writes no file.

Validation:

  • Focused cross-shell theory: 2 of 2 passed.
  • Native-correction suite: 28 of 28 passed.
  • Full actor suite: 3,611 passed; 3 platform-specific tests skipped.
  • Slopwatch and header checks passed.
  • The new GitHub Windows job passed.

@Aaronontheweb
Aaronontheweb merged commit 94d2bfc into dev Aug 26, 2026
23 checks passed
@Aaronontheweb
Aaronontheweb deleted the feat/native-tool-affordance branch August 26, 2026 03:17
@Aaronontheweb Aaronontheweb mentioned this pull request Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request evals LLM efficacy evals for Netclaw security Security-related changes sessions LLM session actor, turn lifecycle, pipelines shell Issues related to the shell tool, since it has the largest security perimeter. subagents spawn_agent, SubAgentActor, definition loader, discovery context layer, and related features tests All issues related to testing, quality assurance, and smoke testing. tools Issues related to agent tools: file_read, web_search, shell_execute, image processing, etc. UX/DX UI / UX / DX friction issue or user-facing annoyances.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant