@overload correctness

[dvoraen]

Hello! I'm looking for some advice on how to properly type functions via @overload. To supply context on the project I'm working on in my spare time, it's establishing type hints for Evennia, which is a Django based library that allows the creation of text-based games. After talking with one of the primary maintainers, it was decided to do so as a type stub submission rather than inline it into the code. Part of the reason for that has to do with the in development code supporting up to Python 3.10, whereas a few of the features I want to type with originate with Python 3.11. Thus, stubs. :) That about covers the madness behind the method.

Getting on to @overload, I have been hitting some mental snarls in my work in the file I'm currently working on. The following function is the case study for my questions which I would really appreciate advice on:

https://github.com/evennia/evennia/blob/cebd09e0d94af245b4b9b7c9a2d02687dfc91e3c/evennia/utils/utils.py#L124-L147

def pad(text, width=None, align="c", fillchar=" "):
    """
    Pads to a given width.

    Args:
        text (str): Text to pad.
        width (int, optional): The width to pad to, in characters.
        align (str, optional): This is one of 'c', 'l' or 'r' (center,
            left or right).
        fillchar (str, optional): The character to fill with.

    Returns:
        text (str): The padded text.

    """
    width = width if width else settings.CLIENT_DEFAULT_WIDTH
    align = align if align in ("c", "l", "r") else "c"
    fillchar = fillchar[0] if fillchar else " "
    if align == "l":
        return text.ljust(width, fillchar)
    elif align == "r":
        return text.rjust(width, fillchar)
    else:
        return text.center(width, fillchar)

And here is my current (probably naïve) typing for it. Explanations about my solution are baked into the queries below.

def pad(
    text: str, width: int | None = ..., align: Literal["c", "l", "r"] = ..., fillchar: str = ...
) -> str: ...

Here are my queries about typing it:

Should @overload cover all possible permutations, where they exist?

In typing pad(), I was led to wonder if I should stick with simply the Literal type for the align parameter, or if I should be including an overload for the case where a general str is passed in. The function's code itself will process other input (a general str) and default it to 'c' if it's not in the acceptable values tuple, but should typing a function care about that? Would it not be better to make it clear that only 'c'/'l'/'r' are what should be passed in for align?

In other words, should the correct typing for this function be what I have above, or should I be adding the following @overload?

@overload
def pad(
    text: str, width: int | None = ..., align: str = ..., fillchar: str = ...
) -> str: ...

I'm not sure I really understand the nuance here of creating an overload with a general str when it's heavily implied by the code that anything that isn't 'c'/'l'/'r' is not going to get you anywhere. The code will accept any str passed in for align, but logically this would be an error and a possible source of bugs, because pad() will default extraneous input to a centered approach. (For example, if someone passed in 'left' rather than 'l', you unexpectedly get centered text; that's a logic error that could potentially be caught by a type checker, because the acceptable values for the Literal were violated.)

My other point is that I'm not 100% clear on how "masked" arguments work here. By masked, I mean a value passed to a function via a variable or function return that would be passed on to pad() that a type checker can't say with certainty what it is. In other words, we're passing a non-literal value that could be compared to the Literal. Something like this:

def foo(text: str, align: str) -> str:
    return pad(text, align)

There is no indication of what foo()'s arguments are or could be; they're simply forwarded to pad(). Would this mean that a type checker would state that the call to pad() here is invalid, because the forwarded argument isn't necessarily a literal value? Would that then mean that I should be providing an overload so that pad() accepts any str even though it could lead to a potential logic error as explained above?

Is @overload meant to be used only for correct calls of functions?

I touched on this already, but I pose the formal question here. I'm not clear on the exact intentions behind the use of @overload when it concerns establishing all the appropriate typings for a function, even after reading PEP 484 and the readthedocs.io site for typing. One excerpt from the typing site on typing.readthedocs.io is the following about overloads:

The following (which would be used in the implementation) is wrong in type stubs:

@overload
def foo(x: str) -> str: ...
@overload
def foo(x: float) -> int: ...
def foo(x: str | float) -> Any: ...

Why is it "wrong" in type stubs? There isn't an explanation, though I think I can intuit why (you want to make it clear that a str parameter returns a str result, and a float gets you a float, but providing a catch-all muddies the water in this specific case plus you've already covered each acceptable usage of foo()). The clarification I'm looking for here is an explanation of when it's appropriate to have a catch-all or fallback overload. In other words, I'm looking for the "when and when nots" of @overload.

That's all I have for now. Thank you for your time in reading my book and for any responses I garner. :)

[srittau]

I have no experience with Evennia, so please take that in mind in my response.

While passing something other than "c", "l", or "r" into align is possible, this would most likely be unintended, i.e. a bug, and is exactly what type checking is supposed to catch. Personally I would use the Literal. One exception would be if passing anything else (for example an empty string) is a common idiom in existing code.

I would expect the author of foo() to use the same literal type for align as you are using in pad(). You could provide a type alias like _Alignment: TypeVar = Literal["c", "l", "r"] and use that. This alias could then be used by the author of foo() (although it requires a bit of an awkward idiom using if TYPE_CHECKING:).

The example in the docs is not supposed to showcase a catch-all overload. Instead, it's just meant to show that while in the implementation you have one non-overload version of the signature (the one which contains the actual implementation), in the stubs you don't include it.

Generally speaking, I would only include a catch-all overload if there are valid cases that can't be expressed with more specific overloads. Ideally, the stubs would only represent valid calls to the function. In practice, that is not always possible or practical, though.

[dvoraen]

Hello! Thank you for that answer. That does help me untangle my thoughts quite a bit. The exception you mention regarding common idioms has come up in my discussion with one of Evennia's maintainers. One such example I raised was code that basically says if not <argument>:, where the argument is intended to be a str. However, as you know if not implies it could also have been passed None, and the code will take that into account even though it doesn't feel intended (and isn't documented as a potential value in the docstring). To make sure I understand your points correctly, because I am extremely prone to overthinking simple matters:

Typing should aim for correct function calls (where able) over what the implementation does.

Is that an accurate summarization of your post or am I putting words into your mouth?

[srittau]

I would say: Typing should catch potential bugs, while not overcomplicating the life of authors.

In typeshed we tend to follow implementation rather documentation. For example, we often allow Iterable where the documentation talks about list, because why should we forbid passing a tuple if that works in practice and will do what the author intended it to do? But your literal example is one where not passing one of the predefined values is most likely a bug and will not do what the author intended it to do.

If I'm writing stubs for my libraries and not for typeshed, I tend to be a bit stricter. As library author I know better what I might want to refactor or change in the future. For example, while passing an Iterable might work at the moment, I might want to refactor my code in the future to use len() (which is not supported by all iterables). Therefore, I might opt to use the stricter Sequence in my own stubs.

So I would say there is no definite answer, which is probably not what you wanted to hear. :)

[dvoraen]

No that's fair. I'm trying to reverse engineer the trains of thought that go into properly typing functions so that I can synthesize them for my own use, rather than constantly try to Matrix-download what everyone knows. :) The Iterable vs Sequence example you raise is another thing that came up during my typing work on Evennia (including the use of len(), as it happens), but those functions required a little bit of library knowledge, so I stuck with the more general-use pad() for the sake of my queries. :)

Thank you for your time in explaining this to me. I greatly appreciate it.

[erictraut]

I think @srittau already provided a great answer, but I'll add one additional tip. The @overload capability is intended to handle polymorphic functions where the return type differs based on the types of its inputs. In cases where the return type is always the same (as in your example, which always returns str), then @overload is unnecessary and probably not the right tool for the job.

[dvoraen]

Thank you for that tip, Eric. That helped also with the cognitive tangle going on inside me. My mind had itself set on the idea that I should try to accommodate "all intended parameters" as far as @overload was concerned, even if doing so didn't change the return type for them.

By the way, thank you for TypeGuard. It has come up in the project I'm doing. :)

[JelleZijlstra]

It's not your primary question (@srittau already gave a great answer), but I want to react to this part of your post:

Part of the reason for that has to do with the in development code supporting up to Python 3.10, whereas a few of the features I want to type with originate with Python 3.11.

3.11 added a host of useful new features to typing, but all of them are usable in 3.10 too if you use the typing_extensions package. This would have to be added as a runtime dependency for your library, but it's a very light package that's already widely installed, so it shouldn't be a big burden to add it. Inline types are going to be easier to maintain than separate stub files, so I would recommend going with inline types instead.

[dvoraen]

While I agree with you on the subject of inline types, one of the primary maintainers of the Evennia library and I talked about how to go about typing it and we came to an agreement on type stubs being the least invasive while v1.0 is still in development (which is what I'm targeting with the typing). Over time, we might be able to move from stubs to inline, but I am not intending to revisit the topic until after v1.0 is done and stable, and I have a lot more experience and know-how under my belt. :)