feat(crashdump): Add crash dump functionality for fatal errors #1594

slurmlord · 2025-09-18T09:33:24Z

This change introduces the option to generate crash dumps, aka. mini dumps, on fatal errors.
The main minidump functionality is done by explicitly loading the dbghelp.dll from the system directory, as the dbghelp.dll that is bundled with the game is an older version that does not include this functionality. There is an option to create small dumps or extended info dumps, currently both are created.

Small dumps

These mostly contain stacks for the process threads and some stack variables, or to create dumps with extended info. The use case for these is to quickly determine where a crash occured, the type of crash, if it was already fixed etc. In addition, if the memory allocation structures are corrupted enough, an extended info dump might not succeed while the small dump should. The size of these dumps are typically on the order of 250kB.

Extended info

These contain global values, along with the memory regions allocated via the memory pool factory and the dynamic memory allocator. This makes all in-game objects available to the person debugging the crash dump, so for example dt generalszh!TheWritableGlobalData in WinDbg will show the state at the time the dump was created.

An alternative option could be to not traverse the memory structures "manually" to get to the allocations and instead just specify the MiniDumpWithFullMemory flag to MiniDumpWriteDump, but that increases the file size considerably.

As an example, dump of the generalszh process in the main menu with the shell map in the background yields a ~140MB dump when traversing and ~420MB with MiniDumpWithFullMemory. Beyond that, the ~140MB file compresses to ~20MB with 7Z, so should be relatively easily transferable.

Storage Location

Crash dumps are stored in a new folder called 'CrashDumps' under the userDir ("Documents\Command and Conquer Generals Zero Hour Data"), and on startup it will create this directory if it doesn't exist and delete any older dumps so only the 10 newest small and 2 newest extended info dumps are left. This is to preserve disk space, as the extended info files can be several hundred MB.

Integration points

For VS2022 builds, unhandled exceptions end up in the UnhandledExceptionFilter in WinMain, which then get a reference to the actual exception that occurred and includes that in the dump.
For VC6 builds, unhandled exceptions are caught in the catch(...) blocks of GameEngine::execute which then calls RELEASE_CRASH. As there is no exception data available in this case to populate _EXCEPTION_POINTERS from, an intentional exception is triggered to get the trace of the current thread. This makes the stack traces for VC6 a bit more cryptic than VS2022 builds as the C++ exception handling gets included in the trace.

Limitations

In the longer run we'll probably want to replace this code with a more mature solution, like CrashPad, but that currently depends on a newer compiler than VC6.
As the code is intended to be temporary, it's kept behind a new CMake feature so it can be easily removed. There are also some other decisions made with this in mind:

Minidump is created in-process. Ideally, the dump should be performed by a process external to the crashing/failing process, but to avoid having to ship an extra binary, in-process was chosen instead. It's being performed in a separate thread to hopefully have a clean stack to work with.
Depends on RTS_BUILD_OPTION_VC6_FULL_DEBUG for VC6 builds. The PDBs generated with the default VC6 compile options are lacking in information, making the mini dumps less useful. The option RTS_BUILD_OPTION_VC6_FULL_DEBUG should be enabled for VC6 builds to ensure maximum usability. VS2022 builds produce better PDBs and require no extra options.
Directory management code is contained within the MiniDumper class, not re-usable by other components.
Many Win32-specific types and functions are used directly without regards for portability.
As the MiniDump feature is not available for VC6, a lot of headers have been borrowed from minidumpapiset.h and included in the MiniDumper_compat.h file.
Globals are used for storing the current exception info.
Only enabled for the games, tools are currently not included.

OmniBlade

Looks good, tested dump generation with a 2022 build.

slurmlord · 2025-10-09T12:41:24Z

Another approach I realized after publishing could be to move the GameMemory allocations from using GlobalAlloc to instead create a separate GameMemory heap and use HeapAlloc.
The benefit would be a lot less code required to traverse the allocations for inclusions the dump, as it could be done with HeapWalk instead for only the GameMemory heap.

xezon

Looks good overall. Just a bunch of small comments.

xezon · 2025-10-08T17:23:37Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+{
+	// Find the full path to the dbghelp.dll file in the system32 dir
+	GetSystemDirectory(m_sysDbgHelpPath, MAX_PATH);
+	strlcat(m_sysDbgHelpPath, "\\dbghelp.dll", MAX_PATH);


In #1066 I added a class DbgHelpLoader that takes care of all the loading and unloading. Would be good if we can reuse that after #1066 is merged.

Agreed, I'll hold off on this one (but push changes for the review comments) until after #1066 is merged, then replace the loading functionality here with DbgHelpLoader and extend it with the MiniDumpWriteDump function.

xezon · 2025-10-08T20:09:15Z

Core/GameEngine/Source/Common/System/Debug.cpp

+#ifdef RTS_ENABLE_CRASHDUMP
+#include "Common/MiniDumper.h"
+
+MiniDumper TheMiniDumper = MiniDumper();


I think = MiniDumper() is not necessary.

xezon · 2025-10-08T20:10:51Z

Core/GameEngine/Source/Common/System/Debug.cpp

+#ifdef RTS_ENABLE_CRASHDUMP
+#include "Common/MiniDumper.h"
+
+MiniDumper TheMiniDumper = MiniDumper();


Maybe move this into MiniDumper.cpp ? This is how EA does it.

xezon · 2025-10-08T20:11:20Z

Generals/Code/Main/WinMain.cpp

 Win32Mouse *TheWin32Mouse= NULL;  ///< for the WndProc() only
 DWORD TheMessageTime = 0;	///< For getting the time that a message was posted from Windows.
+#ifdef RTS_ENABLE_CRASHDUMP
+extern MiniDumper TheMiniDumper;


Maybe move this into MiniDumper.h (or Debug.h) ?

xezon · 2025-10-08T20:12:41Z

Core/GameEngine/Include/Common/MiniDumper.h

+class MiniDumper
+{
+public:
+	MiniDumper()


The function body can go to cpp.

xezon · 2025-10-09T18:28:33Z

Generals/Code/Main/WinMain.cpp

 Win32Mouse *TheWin32Mouse= NULL;  ///< for the WndProc() only
 DWORD TheMessageTime = 0;	///< For getting the time that a message was posted from Windows.
+#ifdef RTS_ENABLE_CRASHDUMP
+extern MiniDumper TheMiniDumper;


And maybe make this MiniDumper* TheMiniDumper ? This way we can control its lifetime better.

xezon · 2025-10-09T18:35:49Z

Core/GameEngine/Include/Common/GameMemory.h

+	AllocationRangeIterator operator++(int) { AllocationRangeIterator tmp = *this; ++(*this); return tmp; }
+
+	friend bool operator== (const AllocationRangeIterator& a, const AllocationRangeIterator& b) { return a.m_currentBlobInPool == b.m_currentBlobInPool; };
+	friend bool operator!= (const AllocationRangeIterator& a, const AllocationRangeIterator& b) { return a.m_currentBlobInPool != b.m_currentBlobInPool; };


xezon · 2025-10-09T18:38:08Z

Core/GameEngine/Source/Common/System/GameMemory.cpp

 		m_stacktrace[0] = NULL;
 	}
 #endif
 }


Is this end of scope correctly placed? Looks strange.

xezon · 2025-10-09T18:43:38Z

Core/GameEngine/Source/Common/System/GameMemory.cpp

+}
+void DynamicMemoryAllocator::fillAllocationRangeForRawBlockN(const Int n, MemoryPoolAllocatedRange& allocationRange) const
+{
+


Remove blank line

xezon · 2025-10-09T18:47:30Z

Core/GameEngine/Source/Common/System/MiniDumper.cpp

+
+void MiniDumper::CleanupResources()
+{
+	// NOTE: This method should not be called unless the dump thread is confirmed to not be running anymore.


Maybe can put an assert here for this assumption?

feat(crashdump): Add crash dump functionality

7187333

OmniBlade approved these changes Sep 23, 2025

View reviewed changes

xezon added Major Severity: Minor < Major < Critical < Blocker Gen Relates to Generals ZH Relates to Zero Hour Debug Is mostly debug functionality System Is Systems related labels Oct 8, 2025

xezon reviewed Oct 9, 2025

View reviewed changes

feat(crashdump): Add crash dump functionality for fatal errors #1594

Are you sure you want to change the base?

feat(crashdump): Add crash dump functionality for fatal errors #1594

Uh oh!

Conversation

slurmlord commented Sep 18, 2025

Small dumps

Extended info

Storage Location

Integration points

Limitations

Uh oh!

OmniBlade left a comment

Choose a reason for hiding this comment

Uh oh!

slurmlord commented Oct 9, 2025

Uh oh!

xezon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants