How can I type annotate a factory helper's **kwargs to match that of the __init__ of the type being created?

[cjw296]

So, for this code:

class Collection:
    def make(self, type_: Type[T], **attrs) -> T:
        ...

How can I annotate **attrs such that a type checker, mypy in particular, will correctly ensure that any keywords passed are a valid subset of those that could be passed to T's constructor?

[erictraut]

You can use a ParamSpec. There's no way to constrain it to only keyword parameters though. After all, the constructor may support positional parameters.

from typing import Callable

class Collection:
    def make[**P, T](
        self, type_: Callable[P, T], *args: P.args, **kwargs: P.kwargs
    ) -> T: ...

Here's the same thing using older (pre-3.12) syntax:

from typing import Callable, ParamSpec, TypeVar

P = ParamSpec("P")
T = TypeVar("T")

class Collection:
    def make(self, type_: Callable[P, T], *args: P.args, **kwargs: P.kwargs) -> T: ...

[Daverball]

The Python Discourse would be the current best place for typing related discussions: https://discuss.python.org/

ParamSpec adjacent topics and discussions have come up several times. Python function signatures are very versatile, but that also makes them very difficult to deal with in static typing contexts. One big issue is expressiveness, since we currently have no way to spell a complex signature in a pure type expression. We instead have to rely on callback protocols, which only gives us part of the functionality we need and some things are only very difficult or awkward to express. This is a relative recent discussion about a possible syntax extension to fix that part of the problem: https://discuss.python.org/t/reviving-the-hybrid-keyword-arrow-syntax-for-callables/52499

Once we can actually express all kinds of signatures in type expressions, it will be much easier to add the bound argument to ParamSpec, which would allow your use-case (although you'd still have to write *args: P.args, **kwargs: P.kwargs). TypeVarMapping has been discussed several times as well, but after how difficult ParamSpec and TypeVarTuple turned out to implement and how long it took for even just basic support to land in more than one type checker, the brakes were hit on adding more kinds of type parameters.

The main focus for Python typing currently is to write a complete typing spec and compliance test suite, so type checkers have to waste less time individually figuring out how certain features are supposed to work, because the wording in the original PEP was too loose/ambiguous.

[cjw296]

Has any progress been made on this?

(although you'd still have to write *args: P.args, **kwargs: P.kwargs).

Not sure what you mean by this? My requirement is to be able to explicitly say that only kwargs from P are accepted.

[Daverball]

No progress has been made on this, no. What I mean by that is that the restriction of "no positional arguments" would be handled through the bound parameter of the ParamSpec, but that wouldn't necessarily change the requirement that you always need to unpack both components of P.

So the type checker would ensure that no positional arguments can be passed in, but your generic signature would still need to include them anyways (unless that restriction is lifted for upper bounds that ensure that either one of the components is definitely always empty).

A TypeVarMapping might be an alternative avenue, although given that TypeVarTuple and ParamSpec still aren't fully supported in pytype to this day, it would probably be met with little enthusiasm, by the people that would have to implement that feature in their type checkers.

[cjw296]

but that wouldn't necessarily change the requirement that you always need to unpack both components of P.

Sorry, not following this, could you provide a code example of what you mean?

So the type checker would ensure that no positional arguments can be passed in, but your generic signature would still need to include them anyways

This sounds pretty sub-optimal :-/

What's pytype? That's not a phrase I've heard before...

[Daverball]

Sorry, not following this, could you provide a code example of what you mean?

The spec for ParamSpec currently states that if P.args is used in a function's signature P.kwargs must also be used and vice versa, there also can't be any arguments in between, the only thing you're allowed to do is prepend arguments, like you would with Concatenate. This is to ensure that no matter what P solves to, the call is still possible. Otherwise the same P would need to check every single place it's used in order to ensure consistency.

def allowed[**P](self, *args: P.args, **kwargs: P.kwargs): ...
def positional_only_prepend_allowed[**P](self, arg: int, /, *args: P.args, **kwargs: P.kwargs): ...
def no_kwargs_disallowed[**P](self, *args: P.args): ...
def no_args_disallowed[**P](self, **kwargs: P.kwargs): ...
def middle_extra_kwarg_disallowed[**P](self, *args: P.args, my_new_kwarg: int = 0, **kwargs: P.kwargs): ...

With upper bounds on P this restriction could potentially be relaxed in some cases, e.g. with an upper bound of def (**kwargs: Any) you may be allowed to omit P.args and define an arbitrary number of preceding positional only arguments. Although the required implementation complexity makes this probably almost certainly not worth it.

What's pytype? That's not a phrase I've heard before...

It's a Python type checker made by Google. It performs very deep type inference and allows a lot of things other type checkers would disallow because their inference is too shallow to ensure type safety without an explicit type annotation. So it can be useful in code that doesn't make heavy use of type annotations, since it will still be able to detect a lot of problems anyways.