Mypy/dmypy takes over 16GB of memory

[juandiegopalomino]

Bug Report
Hello!

Happy Friday folks! I am writing as I was hoping someone here might be help with my company's mypy usage. For us, most of the code is in a massive django monolith with hundreds of dependencies and tens of thousands of lines of code. We are trying to do a good job in adding typechecking in as many places in our code base as we can, but our running into a mypy/dmypy limitation. Dmypy is now taking over 10 minutes to run and when fully started is consuming over 16GB of memory. This slowness and memory consumption is causing serious problems for our engineers, even leading to multiple OOMs on their dev machines because of this memory usage. It has gotten so bad that we have started telling folks to stop running mypy.

I do want to see if there is any fix I can use, so I'm writing here to see if anyone in the community knows how to address these kinds of issues. Is there anyway to tell dmypy to use disk, or the max amount of memory it can use? Any speed hacks? I'm open to any idea.

Thank you for reading, and have a nice day!

To Reproduce

1. Have a huge monolith
2. Run dmypy on it

Expected Behavior
Don't take over 50% of the total available memory in a macbook or beefy ec2 🤣

Actual Behavior
Uses over 16GB of memory

Your Environment
Mypy: 1.4.1 (actually got worse in 1.9)
Command: dmypy start --log-file .dmypy.log -- --follow-imports=error --cache-dir=/dev/null --config-file /pyproject.toml --use-fine-grained-cache
mypy.ini:

[mypy]
no_implicit_optional = True
local_partial_types = True
follow_imports = normal
disallow_untyped_defs = True
disable_error_code = misc

Python version: 3.11.8

[juandiegopalomino]

Here are the perf stats:

platform                :      linux
python_version          :       3_11
roundtrip_time          :    630.699
stats                   : {'validate_meta_time': 0.03480100631713867, 'files_parsed': 33907, 'modules_parsed': 32467, 'stubs_parsed': 1440, 'parse_time': 63.488799810409546, 'find_module_time': 9.783974647521973, 'find_module_calls': 429156, 'graph_size': 33907, 'stubs_found': 1440, 'graph_load_time': 88.9201250076294, 'fm_cache_size': 98115, 'load_fg_deps_time': 2.7418136596679688e-05, 'sccs_left': 0, 'nodes_left': 0, 'cache_commit_time': 1.1920928955078125e-06}
status                  :          1

[solaluset]

Have you tried enabling cache? It should speed it up considerably.

[juandiegopalomino]

What is the command/flag to enable cache?
The problem however, is not speed, but memory usage

[solaluset]

If I'm not mistaken, it's enabled by default, and disabled by --cache-dir=/dev/null in your configuration.
I don't know how to reduce memory usage, sorry 😅

[sjsmith757]

I am seeing the same issue on our Linux server. Curious if there has been any update on this? Dmypy runs using ~15GB of memory