High Windows Committed Memory (Virtual Memory)

[RyanJDick]

The way that Windows handles virtual memory reservations causes some problems for Invoke as we try to squeeze the most out of a system’s available RAM and VRAM limits. This became increasingly evident in v5.6.0rc2, so I have taken the time to document the issue for reference. There will be a handful of PRs coming soon to mitigate the issues.

Background: Windows page file

Background reading:

• https://learn.microsoft.com/en-us/troubleshoot/windows-client/performance/introduction-to-the-page-file
• https://answers.microsoft.com/en-us/windows/forum/all/physical-and-virtual-memory-in-windows-10/e36fb5bc-9ac8-49af-951c-e7d39b979938

Key takeaways:

• “Committed Memory” == “Virtual Memory”
• If the pagefile is “system-managed” it will grow up to 3 times the physical memory, but no more than 1/8 of the disk volume size. (This assumes that there is enough disk space to support the pagefile.)

On Windows all reserved virtual memory space (aka ‘committed memory’) must be backed by a physical location (either physical memory or a page file). This differs from Linux, which allows virtual memory reservations to have no physical backing if they are never used.

Symptoms

On Windows, you could run out of page file space for the following reasons:

• The page file is set to a fixed size rather than “system-managed”, so does not grow dynamically.
• The page file has hit the upper size limit of the “system-managed” policy.
• There is insufficient disk space for the page file to grow any larger.

When you run out of page file space, you may see a variety of different crashes. The most common are:

• Invoke exits with windows error code 3221225477
• Invoke crashes without an error, but eventvwr.msc reveals an error with code 0xc0000005 (the hex equivalent of 3221225477).

Why is virtual memory usage high in Invoke?

Invoke does two things that result in high virtual memory usage:

• Allocate CUDA tensors using torch
• Load models from memory-mapped files using safetensors

torch CUDA Tensors

All CUDA memory requires virtual address space. Because of Windows’ policy that all virtual address space has a physical backing, this means that we must reserve physical disk space in the page file for all allocated CUDA memory, even though it will never be used! To put it differently: if we load a 24 GB model onto the GPU, Windows will reserve a 24GB virtual memory ‘placeholder’ in the page file (on disk) even though it will never be accessed!

Here is a minimal reproduction if you wish to see this in action:

1. Open your Windows Task Manager and check the current “Committed” memory.

2. Allocate a 10GB CUDA tensor:

PS C:\Users\Ryan\ryan_src\InvokeAI> python
Python 3.11.11 (main, Dec 19 2024, 14:36:07) [MSC v.1942 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> x = torch.rand([1, 10 * 2 ** 30 // 4], device="cuda")

3. Check the updated “Committed” memory.

4. Note that the “In use” memory has remained roughly unchanged, but the “Committed” memory (Virtual Memory) has grown by ~10GB.

safetensors mmap

When we load model files using safetensors (e.g. safetensors.torch.load_file(...)), it uses a memory-mapped file so that tensors can be lazy-loaded as needed from the file. In Windows, the mmap’ed file requires virtual address space, so once again Windows will unnecessarily reserve this space in the page file.

Here’s a minimal reproduction:

1. Open your Windows Task Manager and check the current “Committed” memory.

2. Load a safetensors file from disk. In this example, I am loading a ~22GB FLUX model.

PS C:\Users\Ryan\ryan_src\InvokeAI> python
Python 3.11.11 (main, Dec 19 2024, 14:36:07) [MSC v.1942 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from safetensors.torch import load_file
>>> sd = load_file("C:\\Users\\Ryan\\ryan_invoke\\models\\flux\\main\\FLUX Dev.safetensors")

3. Check the updated memory stats. At this point in time, Windows has reserved virtual address space for the mmap’ed file, but no weights have actually been loaded into memory. The “Committed” memory has grown by ~22GB, and the “In use” memory has stayed roughly the same.

4. Now, suppose that we try to do something with the mmap-ed state dict. The weights will be materialized in memory, but the mmap’ed file won’t be released until all references to it are gone. At the peak, we will have 2x the model size reserved in virtual memory!

>>> for k in sd.keys():
...     sd[k] = sd[k] * 2