Skip to content

test(approvals): batch persistent seed grants into one Ask per case - #1989

Merged
Aaronontheweb merged 4 commits into
devfrom
fix/batch-shell-approval-seeds
Aug 20, 2026
Merged

test(approvals): batch persistent seed grants into one Ask per case#1989
Aaronontheweb merged 4 commits into
devfrom
fix/batch-shell-approval-seeds

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

What

Fixes a flaky AskTimeoutException in ShellPolicyEvidenceFixtureTests.Policy_fixtures_execute_through_the_coordinator on Windows CI.

Refs: run 32149410006 / job 95751209585 (commit 00d93eb1).

Root cause

Each persistent grant was seeded with a separate Ask, and each Ask's handler (ToolApprovalActorToolApprovalStore.AddApprovals) does synchronous, cache-bypassing disk I/O on the shared default dispatcher: WriteThrough + Flush(flushToDisk: true) + File.Move under an exclusive lock.

Under full-suite parallel load on Test-windows-latest (Defender real-time-scanning each fresh tool-approvals.json in a new %TEMP% tree), the heaviest fixture case — D10, with 5 persistent seeds — occasionally pushed one seed Ask past the hard 5-second wall clock. The log confirms D02–D09 completed correctly before the timeout hit D10's seeding phase.

Ruled out (with code evidence):

  • Lock hang — stores use lockTimeout: TimeSpan.Zero, so AcquireLock is single-attempt fail-fast; contention throws, never blocks 5s.
  • Message loss — every seed Ask is awaited before GracefulStop; no Ask is in flight during actor recreation.
  • Logic regression — all completed cases print correct outcomes/traces; evaluation is pure in-memory.

Fix

Group persistent seeds by audience into one Ask per case instead of one per grant. The actor's RecordStructuredToolApproval handler already persists a whole grant list in a single locked atomic write (ToolApprovalStore.AddApprovals → one SaveLocked), so the resulting store state is equivalent to N sequential seed messages — it still exercises the real production path.

Cuts seed-phase file rewrites from 24 → 10 (D10: 5 → 1), removing the per-case worst-case latency from the 5s budget.

Test-only change. No production behavior touched; the hardcoded 5s Ask timeout in AkkaToolApprovalService is intentionally left alone.

Equivalence note (from adversarial review)

The only state divergence vs N sequential asks is per-entry CreatedAt stamping under a real clock (a batch stamps all grants at one instant). This is unobservable here: fixture tests run on a frozen FakeTimeProvider, and no harness test asserts seed timestamps. The partial-batch-failure case (one malformed grant failing an audience group) is real but not a regression — RecordStructuredToolApproval validates all grants and returns InvalidData, which AkkaToolApprovalService throws on in both old and new flows, so no caller can observe a silent partial-seed difference.

Verification

  • Build clean (0 warnings, 0 errors).
  • All 330 tests in ShellPolicyEvidenceFixtureTests + ShellApprovalDispositionMatrixTests pass in ~2s (the flaky path was ~23s+ before timing out).

The flake only reproduces under parallel Windows load, so CI on Windows is the real proof. Suggested stress loop if it recurs:

1..5 | ForEach-Object {
  dotnet test src/Netclaw.Actors.Tests/Netclaw.Actors.Tests.csproj -c Release `
    --filter "FullyQualifiedName~ShellPolicyEvidenceFixtureTests" --no-build
  if ($LASTEXITCODE -ne 0) { Write-Host "FAILED on iteration $_"; break }
}

Note (not addressed here)

ToolApprovalActor doing synchronous disk I/O inside message handlers is a latent production risk this test exposed. A follow-up could assign it a dedicated dispatcher or move the store calls off the actor thread — out of scope for this test-only fix.

ShellPolicyEvidenceFixtureTests.Policy_fixtures_execute_through_the_coordinator
flakes on Windows CI with an AskTimeoutException (5s) during harness seeding.

Each persistent grant was seeded with a separate Ask, and each Ask's handler
does synchronous cache-bypassing disk I/O (WriteThrough + Flush(flushToDisk:
true) + File.Move) on the shared default dispatcher. Under full-suite parallel
load (Defender scanning fresh %TEMP% store files), the heaviest case (D10, 5
seeds) occasionally pushed one Ask past the hard 5s wall clock.

The actor's RecordStructuredToolApproval handler already persists a whole grant
list in a single locked atomic write (ToolApprovalStore.AddApprovals -> one
SaveLocked), so grouping seeds by audience into one Ask per case is
byte-for-byte equivalent while cutting seed-phase file rewrites from 24 to 10.

Test-only change; no production behavior touched. Verified: all 330 tests in
ShellPolicyEvidenceFixtureTests + ShellApprovalDispositionMatrixTests pass in ~2s.

Refs run 32149410006 / job 95751209585 (commit 00d93eb).
Adversarial review nit: ToolApprovalStore.AddApprovals stamps CreatedAt per
entry, so under a real clock the batched flow stamps all grants at one instant
vs distinct instants. Unobservable in this suite (fixture tests use a frozen
FakeTimeProvider; no harness test asserts seed timestamps), but the comment
should not overclaim byte-for-byte equality.

Also note: the partial-batch-failure case (one malformed grant failing an
audience group) is real but not a regression -- RecordStructuredToolApproval
validates all grants and returns InvalidData, which AkkaToolApprovalService
throws on in both old and new flows, so no caller can observe a silent
partial-seed difference.

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - perf optimization

@Aaronontheweb
Aaronontheweb disabled auto-merge August 20, 2026 15:50
@Aaronontheweb
Aaronontheweb merged commit 0bd1a23 into dev Aug 20, 2026
23 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/batch-shell-approval-seeds branch August 20, 2026 15:50
@Aaronontheweb Aaronontheweb mentioned this pull request Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant