Improve stack overflow stack trace pretty print

[janvorli]

The current algorithm used for the stack overflow stactrace pretty printing isn't able to find repetitions that don't start at the first frame. That leads to huge stack traces when there are few non-repeated frames on the top of the stack, e.g. when a recursion in managed method calls to some common chain of methods and the stack overflow occurs down that chain and not in the recursive part.

This change fixes that. When no repeated sequence is found at the top of the stack, it tries to search from the next frame and so on until a repeated sequence is identified.

Close #118218

[janvorli]

With this change, the stack trace from the test code from #118218 looks like this:

Stack overflow.
   at System.RuntimeFieldInfoStub.FromPtr(IntPtr)
   at System.Buffers.Text.FormattingHelpers.CountDigits(UInt32)
   at System.Number.UInt32ToDecStr_NoSmallNumberCheck(UInt32)
   at System.Number.UInt32ToDecStr(UInt32)
   at System.Number.FormatInt32(Int32, Int32, System.String, System.IFormatProvider)
   at System.Int32.ToString(System.IFormatProvider)
   at System.IO.TextWriter.Write(Int32)
   at System.IO.TextWriter.WriteLine(Int32)
   at System.IO.TextWriter+SyncTextWriter.WriteLine(Int32)
   at System.Console.WriteLine(Int32)
Repeated 15721 times:
--------------------------------
   at Program.<<Main>$>g__M|0_0(Int32)
--------------------------------
   at Program.<Main>$(System.String[])

[Copilot]

Pull request overview

This PR improves the stack overflow stack trace pretty printing algorithm to detect repetitions that don't start at the first frame. Previously, the algorithm could only identify repeated sequences starting from the top of the stack, leading to excessively long stack traces when non-repeated frames appeared at the beginning.

Key changes:

• Refactored repetition detection from inline (during stack walk) to post-processing analysis
• Removed member variables (m_commonStartIndex, m_largestCommonStartLength, m_largestCommonStartRepeat) in favor of local variables in PrintStackTrace
• Implemented nested loop algorithm that searches for repeated sequences starting from different offsets in the stack

[jkotas]

Thanks

[janvorli]

Actually, let me test it on macOS arm64. The minimal stack frame is much smaller there and I want to check if we need to introduce the upper limit for the outerloop due to that.

[janvorli]

@jkotas test on macOS arm64 proves we need an upper limit. With your test case, there is over 350,000 frames on the stack, so the logging took 16 secs.