Representation of ParamSpecs at runtime compared to Callable

[Gobot1234]

          I think a list representation is more consistent with the text of PEP 612 and the runtime representation of Callable. We should change the runtime repr of ParamSpec generics to be more like Callable, because right now they're inconsistent:

>>> Callable[[int, str], bool]
typing.Callable[[int, str], bool]
>>> P = ParamSpec("P")
>>> T = TypeVar("T")
>>> class MyCallable(Generic[P, T]): pass
... 
>>> MyCallable[[int, str], bool]
__main__.MyCallable[(<class 'int'>, <class 'str'>), bool]
>>> get_args(Callable[[int, str], bool])
([<class 'int'>, <class 'str'>], <class 'bool'>)
>>> get_args(MyCallable[[int, str], bool])
((<class 'int'>, <class 'str'>), <class 'bool'>)

Originally posted by @JelleZijlstra in python/typing#1274 (comment)

Linked PRs

• gh-102615: Use list instead of tuple in repr of paramspec #102637
• gh-102615: Fix type vars substitution of collections.abc.Callable and custom generics with ParamSpec #102681

[Fidget-Spinner]

Just clarifying for anyone who plans to contribute: underneath both __args__ are the same (a flat non-nested tuple of t ypes) get_args just shows the string representation of Callable to wrap [] around the args.

[sobolevn]

underneath both args are the same (a flat non-nested tuple of t ypes)

No, they are different:

>>> MyCallable[[int, str], bool].__args__
((<class 'int'>, <class 'str'>), <class 'bool'>)
>>> Callable[[int, str], bool].__args__
(<class 'int'>, <class 'str'>, <class 'bool'>)

I've sent a PR with the repr change.

[Fidget-Spinner]

underneath both args are the same (a flat non-nested tuple of t ypes)

No, they are different:

>>> MyCallable[[int, str], bool].__args__
((<class 'int'>, <class 'str'>), <class 'bool'>)
>>> Callable[[int, str], bool].__args__
(<class 'int'>, <class 'str'>, <class 'bool'>)

I've sent a PR with the repr change.

Oh woops, I'm wrong then :). That's a bug. The tuples should be flattened and they should be the same. I remember trying to flatten all the args representations when I implemented PEP 612.

>>> Callable[P, T][[int, str], bool].__args__
(<class 'int'>, <class 'str'>, <class 'bool'>)
>>> collections.abc.Callable[P, T][[int, str], bool].__args__
(<class 'int'>, <class 'str'>, <class 'bool'>)

There's a very important reason why they need to be flattened -- generic expects a list of types in __args__, it does not handle tuples. So double substitution will break if there are tuples.

E.g this works.

>>> Callable[P, T][[P, str], bool][int]
typing.Callable[[int, str], bool]

But ugh these two are broken

# Broken in collections.abc :(, needs fixing
>>> collections.abc.Callable[P, T][[P, str], bool][int]
TypeError: Callable must be used as Callable[[arg, ...], result].
# Broken in typing
>>> MyCallable[P, T][[P, str], bool][int]
TypeError: __main__.MyCallable[(~P, <class 'str'>), bool] is not a generic clas

[sobolevn]

That's a bug

Yeah, I was not sure if this is intentional or a bug. Got it now!
Will change my PR to address it soon :)

[sobolevn]

After reading quite a lot of tests, I think that

>>> MyCallable[[int, str], bool].__args__
((<class 'int'>, <class 'str'>), <class 'bool'>)

is correct. For example, take a look at this test:

def test_multiple_paramspecs_in_user_generics(self):
        P = ParamSpec("P")
        P2 = ParamSpec("P2")

        class X(Generic[P, P2]):
            f: Callable[P, int]
            g: Callable[P2, str]

        G1 = X[[int, str], [bytes]]
        G2 = X[[int], [str, bytes]]
        self.assertNotEqual(G1, G2)
        self.assertEqual(G1.__args__, ((int, str), (bytes,)))
        self.assertEqual(G2.__args__, ((int,), (str, bytes)))

If we are going to flatten G1 = X[[int, str], [bytes]] to have (int, str, bytes) - it won't be clear what we mean here.

So, I think it is intentional: args are not flattened for ParamSpec substitution.

@Fidget-Spinner, however, your examples are broken indeed.

[Fidget-Spinner]

Thanks for looking into this. I vaguely recall that for user defined generics with ParamSpec, we don't flatten because of the following:

Generic[P1, int, P2]

The subsitution of the above, if flattened, would be impossible to reconstruct with get_args.

I need to dig through the history. I think I left a comment somewhere about this.

[Fidget-Spinner]

I think the only thing that needs fixing here is the repr and collections.abc.Callable then. Thanks again Nikita!

[AlexWaygood]

But ugh these two are broken

>>> # Broken in collections.abc :(, needs fixing
>>> collections.abc.Callable[P, T][[P, str], bool][int]
TypeError: Callable must be used as Callable[[arg, ...], result].
>>> # Broken in typing
>>> MyCallable[P, T][[P, str], bool][int]
TypeError: __main__.MyCallable[(~P, <class 'str'>), bool] is not a generic clas

Are these two examples valid uses of ParamSpec? I thought you had to use Concatenate if you wanted to prepend parameters to a ParamSpec variable in a parameters list. And I thought appending parameters to a ParamSpec variable in a parameters list was disallowed altogether by PEP-612.

So surely the runtime is correct in disallowing both of these: they both involve a ParamSpec being substituted with an invalid parameters list?