Mitigation for a GC Stress race after an inline pinvoke

[AndyAyersMS]

In the post-call part of a pinvoke inline call frame, it's not safe
to start a stress mode GC in the window between checking
g_TrapReturningThreads and the call to CORINFO_HELP_STOP_FOR_GC.

The call instruction is already getting special treatement, but there may
be other instructions between the check and call. Instead of trying
to pattern match them all, suppress GC stress if g_TrapReturningThreads
is true, the thread is in cooperative mode, and there's an active inline
call frame.

Closes #37236.

[AndyAyersMS]

@jkotas PTAL

With this change, the test case from #37236 has now run ~5000 times without a failure; without the fix I was able to repro after less than 1000 tries. Will keep it going a bit further.

[AndyAyersMS]

GC count looks comparable before/after fix:

;; Before
GC Counts: Gen0=16546, Gen1=16557, Gen2=1367
GC Counts: Gen0=16389, Gen1=16400, Gen2=1459
GC Counts: Gen0=16477, Gen1=16488, Gen2=1764

;; After
GC Counts: Gen0=16564, Gen1=16575, Gen2=1329
GC Counts: Gen0=16505, Gen1=16516, Gen2=1238
GC Counts: Gen0=16455, Gen1=16466, Gen2=1678

[jkotas]

You can try to delete all special-casing of JIT_RareDisableHelper from gccover.cpp. This check should be able to handle all situations that the special-casing was for. If we find situations that this check is not able to handle, it means the fix is incomplete.

[jkotas]

Thank you!

[AndyAyersMS]

You can try to delete all special-casing of JIT_RareDisableHelper

I was thinking the same thing. Will try it as a follow-up.