mypy daemon consumes an increasing amount of memory every time `run` or `recheck` is invoked

[bzoracler]

Bug Report

The amount of memory that the daemon process consumes doesn't drop when typing errors are fixed; the memory consumption appears to stay the same if there are no errors, and when more errors appear, the memory consumption just continues increasing.

This is problematic when running the daemon on initially untyped code bases with large amounts of typing errors (as usually done in code editors integrating with a type-checker), as dmypy becomes sluggish very quickly (because the same existing errors are repeatedly reported).

To Reproduce & Actual Behaviour

1. Repeat this line 2000 times in a file project/test/__init__.py,

   aaaaaaaaaaaaaaaaaaaaaaaa

   and create the following mypy configuration file at project/mypy.ini

   [mypy]
   files = test/
   

2. Navigate to project/, then run dmypy run > errors.txt multiple times without editing project/test/__init__.py; watch the memory consumption grow for the dmypy process.

3. Delete all code in project/test/__init__.py, then run dmypy run again multiple times. The memory consumption does not decrease.

Expected Behavior

I expect the memory consumption to be proportional to the number of errors, AST nodes, and size of a cache diff since the last time the checked code files have changed. If there's no code changes, it's surprising to me that memory consumption would grow for just repeated reports of the same errors.

Your Environment

• Mypy version used: 1.17.1
• Mypy command-line flags: None
• Mypy configuration options from mypy.ini (and other config files): See # project/mypy.ini
• Operating system: Ubuntu (Pop!_OS) 22.04 LTS
• Python version used: 3.11

[bzoracler]

A cursory look reveals that the following objects accumulate in memory after each run of dmypy run or recheck:

• lists:

  [_StoreTrueAction(option_strings=['--disallow-any-expr'], dest='disallow_any_expr', nargs=0, const=True, default=False, type=None, choices=None, required=False, help='Disallow all expressions that have type Any', metavar=None), _StoreTrueAction(option_strings=['--disallow-any-decorated'], dest='disallow_any_decorated', nargs=0, const=True, default=False, type=None, choices=None, required=False, help='Disallow functions that have Any in their signature after decorator transformation', metavar=None), ...]
  ['--disallow-subclassing-any']
  ['--allow-subclassing-any']
  ...
  

• dicts:

  {'action': {None: <class 'argparse._StoreAction'>, 'store': <class 'argparse._StoreAction'>, 'store_const': <class 'argparse._StoreConstAction'>, 'store_true': <class 'argparse._StoreTrueAction'>, ...}
  

---

EDIT

The accumulated objects above are caused by mypy's GC threshold tuning:

mypy/mypy/build.py

Lines 217 to 220 in 982853d

	if platform.python_implementation() == "CPython":
	# Run gc less frequently, as otherwise we can spent a large fraction of
	# cpu in gc. This seems the most reasonable place to tune garbage collection.
	gc.set_threshold(200 * 1000, 30, 30)

argparse.* objects tend to have a lot of circular references.

Unfortunately, this still doesn't seem to account for the large bumps in memory consumption that I've observed (for code-bases with a large amount of existing typing errors) when a file is edited back-and-forth.

[bzoracler]

A large contributor to the increasing memory consumption is that mypy.nodes.SymbolTable doesn't clear redefined symbol nodes between runs from the following:

• mypy/mypy/semanal.py

  Line 6815 in e30128b

  def add_redefinition(self, names: SymbolTable, name: str, symbol: SymbolTableNode) -> None:

• Possibly

  mypy/mypy/util.py

  Line 466 in e30128b

  def get_unique_redefinition_name(name: str, existing: Container[str]) -> str:

Editing back-and-forth and running dmypy run will cause <node name>-redefinition<N> to accumulate on the symbol table where N increases seemingly without bound.