Type hinting strategies for monkey patching

[jace]

I'm adding type hints to a codebase that had previously adopted a strategy of splitting a class across modules (context for why is below). A class decorator copies the attributes from a temporary class into an existing class, then returns the existing class, discarding the temporary class. The result is a code pattern that looks like this:

from .other_module import ImportedClass
from .helpers import reopen

@reopen(ImportedClass)
class TemporaryClass:
    def new_method(self) -> int:
        return self.existing_attribute + 1

assert TemporaryClass is ImportedClass

This code is equivalent to:

def new_method(self: ImportedClass) -> int:
    return self.existing_attribute + 1

ImportedClass.new_method = new_method

Using the reopen decorator allows for cleaner, less surprising code, and the decorator adds safety checks over direct patching. The decorator is also typed to indicate it replaces the decorated value with its parameter so that the assert passes type checking. The problem is that mypy does not actually recognise the patching, so I get other errors:

1. In new_method it will raise an error about TemporaryClass not containing existing_attribute.
2. If I change the signature to def new_method(self: ImportedClass) -> int, it will raise an error for the erased type of self ImportedClass not being a superclass of TemporaryClass.
3. I can workaround this by using cast(ImportedClass, self), but mypy also does not recognise ImportedClass.new_method to exist, so downstream code will get flagged.

I'm aware monkey patching and static type analysis don't get along, so I'm wondering how to reorganise without littering my code with cast and # type: ignore. Can this be addressed with a mypy plugin, as happens with enums and dataclasses, or is this sort of distributed class definition too complex for mypy?

Why do this?

The reopen mechanism came up as a coping strategy for SQLAlchemy models. I have ORM models representing one-to-many (or parent-child) relationships that are split across files. SQLAlchemy has back-references that allow a ChildModel model class to create a collection on a ParentModel model class (like ParentModel.children), but no mechanism for injecting methods. If I need a parent.refresh_children() method, I could:

1. Define it in the ParentModel class. However, the method itself depends on the structure of ChildModel, and if I have multiple types of child models, all with their own support functions, it becomes hard to understand as related functionality is now spread across files, apart from also causing circular imports.

2. Define a classmethod on ChildModel with a signature like def refresh_parent(cls, parent: ParentModel). This keeps related functionality in the same file and in the same class, but feels unpythonic in usage:

   @app.route('/refresh/<parent_id:int>', methods=['POST'])
   def refresh_parent(parent_id: int):
       parent = ParentModel.query.get(parent_id)
       ChildModel.refresh_parent(parent)
       # Nicer call:
       # parent.refresh_children()

3. Use the above approach with @reopen, keeping related functionality all in one file while also having a Pythonic API in usage.

But this approach no longer works with the introduction of type hinting, so I'm hoping for another way to organise that checks all the boxes: easy to read and maintain, nice API to use, and mypy-compatible.

---

Variations of this pattern have come up in the past, sometimes referring to Ruby's class extensions:

• Mypy issue from 2018, in which @JukkaL mentions having considered it: Allow adding attributes to classes after definition mypy#5363
• On the Python Ideas list: https://www.mail-archive.com/[email protected]/msg26034.html
• Ian Bicking in 2007: https://ianbicking.org/blog/2007/08/opening-python-classes.html