Skip to content

Commit fcb50dc

Browse files
JohnMcLearclaude
andcommitted
test(ci): probe G — install ProcDump as JIT debugger for silent-kill capture
Probe A (PR #7855) ruled out Defender as the killer: with DisableRealtimeMonitoring + DisableBehaviorMonitoring + DisableIOAVProtection all = True, the silent ELIFECYCLE still fired (run 26470378618, Win without plugins, `pad.ts > Tests > creates a new Pad with empty text`, kill +470 ms post-test-start, exit 255). The captured event logs showed: - Application log: empty (zero entries during test phase) - System log: only pre-test service stops; no SCM TerminateProcess - Defender Operational: only stale 2026-05-18 runner provisioning - Application Error / Hang / WER: zero entries The fixed tasklist sidecar showed the dying Node process (PID 7036) was completely healthy 1 second before death: HandleCount=323 stable, ThreadCount=17 stable, WorkingSetSize ~321 MB stable, KernelModeTime and UserModeTime growing linearly. No anomaly in OS-side process state. Then dead 1 second later with zero entry in any Windows event log. That fingerprint — silent external termination with no event-log trace and no anomaly in OS-side state — matches `__fastfail` (the `int 29h` fast-fail intrinsic). libuv on Windows uses `__fastfail` for certain internal assertion failures in its TCP and pipe paths (uv_win.c, tcp-win.c, pipe-win.c). When triggered, it immediately terminates the process bypassing all user-mode notification including WER. The only standard tool that catches state across __fastfail is a JIT-installed debugger. Install Sysinternals ProcDump as the system JIT debugger: - downloads procdump.zip from sysinternals.com - extracts to C:\procdump - `-i -ma` registers as the AeDebug handler, configured for full memory dumps - dumps land in node-report/dumps/ which the existing failure artifact picks up On the next silent ELIFECYCLE this captures a .dmp file with full call stack across the kill — loadable in WinDbg with "!analyze -v" to see the libuv assertion (or whatever else) that fired the fast-fail. That should be the final word on what's killing the process. Built on top of probe-flake-defender-eventlog-sidecar (#7855) because the event-log capture + sidecar fix are useful baselines even after Defender's ruled out. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 2415b5c commit fcb50dc

1 file changed

Lines changed: 54 additions & 0 deletions

File tree

.github/workflows/backend-tests.yml

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,33 @@ jobs:
214214
run: |
215215
powershell -Command "(gc settings.json.template) -replace '\"max\": 10', '\"max\": 10000' | Out-File -encoding ASCII settings.json.holder"
216216
powershell -Command "(gc settings.json.holder) -replace '\"points\": 10', '\"points\": 1000' | Out-File -encoding ASCII settings.json"
217+
-
218+
# PROBE G — install Sysinternals ProcDump as the system Just-In-Time
219+
# debugger. When ANY process on the runner crashes, fast-fails (via
220+
# int 29h), or hits an unhandled SEH exception, the OS hands it to
221+
# the JIT debugger, which writes a full memory dump to the configured
222+
# directory. This catches the silent-ELIFECYCLE kill class even when
223+
# it bypasses every event-log path: probe A+H (PR #7855) ruled out
224+
# Defender AND showed the kill produces NO entry in Application,
225+
# System, or Defender Operational logs. That fingerprint matches
226+
# __fastfail (int 29h) — used internally by libuv on Windows for
227+
# certain TCP/pipe state-corruption assertions. ProcDump as JIT
228+
# debugger is the only standard tool that captures user + kernel
229+
# state across __fastfail.
230+
name: Install ProcDump as JIT debugger (probe G)
231+
shell: powershell
232+
run: |
233+
$ErrorActionPreference = 'Continue'
234+
Invoke-WebRequest -Uri "https://download.sysinternals.com/files/Procdump.zip" -OutFile "$env:TEMP\Procdump.zip"
235+
Expand-Archive -Path "$env:TEMP\Procdump.zip" -DestinationPath "C:\procdump" -Force
236+
New-Item -ItemType Directory -Force -Path "${{ github.workspace }}\node-report\dumps" | Out-Null
237+
# -i installs procdump as the system JIT debugger.
238+
# -ma writes a full memory dump.
239+
# The dump path argument tells procdump where to write the dumps.
240+
& "C:\procdump\procdump64.exe" -accepteula -i -ma "${{ github.workspace }}\node-report\dumps"
241+
# Sanity: confirm the AeDebug registry key now points at procdump.
242+
Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug" -ErrorAction SilentlyContinue | Format-List
243+
Get-ItemProperty "HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug" -ErrorAction SilentlyContinue | Format-List
217244
-
218245
name: Run the backend tests
219246
shell: bash
@@ -397,6 +424,33 @@ jobs:
397424
run: |
398425
powershell -Command "(gc settings.json.template) -replace '\"max\": 10', '\"max\": 10000' | Out-File -encoding ASCII settings.json.holder"
399426
powershell -Command "(gc settings.json.holder) -replace '\"points\": 10', '\"points\": 1000' | Out-File -encoding ASCII settings.json"
427+
-
428+
# PROBE G — install Sysinternals ProcDump as the system Just-In-Time
429+
# debugger. When ANY process on the runner crashes, fast-fails (via
430+
# int 29h), or hits an unhandled SEH exception, the OS hands it to
431+
# the JIT debugger, which writes a full memory dump to the configured
432+
# directory. This catches the silent-ELIFECYCLE kill class even when
433+
# it bypasses every event-log path: probe A+H (PR #7855) ruled out
434+
# Defender AND showed the kill produces NO entry in Application,
435+
# System, or Defender Operational logs. That fingerprint matches
436+
# __fastfail (int 29h) — used internally by libuv on Windows for
437+
# certain TCP/pipe state-corruption assertions. ProcDump as JIT
438+
# debugger is the only standard tool that captures user + kernel
439+
# state across __fastfail.
440+
name: Install ProcDump as JIT debugger (probe G)
441+
shell: powershell
442+
run: |
443+
$ErrorActionPreference = 'Continue'
444+
Invoke-WebRequest -Uri "https://download.sysinternals.com/files/Procdump.zip" -OutFile "$env:TEMP\Procdump.zip"
445+
Expand-Archive -Path "$env:TEMP\Procdump.zip" -DestinationPath "C:\procdump" -Force
446+
New-Item -ItemType Directory -Force -Path "${{ github.workspace }}\node-report\dumps" | Out-Null
447+
# -i installs procdump as the system JIT debugger.
448+
# -ma writes a full memory dump.
449+
# The dump path argument tells procdump where to write the dumps.
450+
& "C:\procdump\procdump64.exe" -accepteula -i -ma "${{ github.workspace }}\node-report\dumps"
451+
# Sanity: confirm the AeDebug registry key now points at procdump.
452+
Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug" -ErrorAction SilentlyContinue | Format-List
453+
Get-ItemProperty "HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug" -ErrorAction SilentlyContinue | Format-List
400454
-
401455
name: Run the backend tests
402456
shell: bash

0 commit comments

Comments
 (0)