Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[red-knot] Decompose bool to Literal[True, False] in unions and intersections #15738

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,22 @@ reveal_type(c >= d) # revealed: Literal[True]
#### Results with Ambiguity

```py
def _(x: bool, y: int):
class P:
def __lt__(self, other: "P") -> bool:
return True

def __le__(self, other: "P") -> bool:
return True

def __gt__(self, other: "P") -> bool:
return True

def __ge__(self, other: "P") -> bool:
return True

class Q(P): ...

def _(x: P, y: Q):
Comment on lines +61 to +76
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this because it doesn't look like we've implemented comparisons between unions inside tuples yet, and bool is now internally represented as a union, which meant that many reveal_type assertions in this test became TODOs rather than bools. Testing whether we knew how to compare a bool with a bool didn't seem to me to be the point of the test.

a = (x,)
b = (y,)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -455,9 +455,9 @@ else:
reveal_type(x) # revealed: slice
finally:
# TODO: should be `Literal[1] | str | bytes | bool | memoryview | float | range | slice`
reveal_type(x) # revealed: bool | float | slice
reveal_type(x) # revealed: bool | slice | float
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order here changes because I'm using swap_remove in Type::display to ensure that Literal[True, False] is displayed as bool to users. But maybe it's okay to use remove() in Type::display() rather than swap_remove? remove() is O(n) rather than O(1), but displaying a type shouldn't really be performance-sensitive: it's usually only done as part of printing a diagnostic to the terminal


reveal_type(x) # revealed: bool | float | slice
reveal_type(x) # revealed: bool | slice | float
```

## Nested `try`/`except` blocks
Expand Down Expand Up @@ -534,7 +534,7 @@ try:
reveal_type(x) # revealed: slice
finally:
# TODO: should be `Literal[1] | str | bytes | bool | memoryview | float | range | slice`
reveal_type(x) # revealed: bool | float | slice
reveal_type(x) # revealed: bool | slice | float
x = 2
reveal_type(x) # revealed: Literal[2]
reveal_type(x) # revealed: Literal[2]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,22 +21,22 @@ else:
if x and not x:
reveal_type(x) # revealed: Never
else:
reveal_type(x) # revealed: Literal[0, -1, "", "foo", b"", b"bar"] | bool | None | tuple[()]
reveal_type(x) # revealed: Literal[0, -1, "", "foo", b"", b"bar"] | bool | tuple[()] | None

if not (x and not x):
reveal_type(x) # revealed: Literal[0, -1, "", "foo", b"", b"bar"] | bool | None | tuple[()]
reveal_type(x) # revealed: Literal[0, -1, "", "foo", b"", b"bar"] | bool | tuple[()] | None
else:
reveal_type(x) # revealed: Never

if x or not x:
reveal_type(x) # revealed: Literal[0, -1, "", "foo", b"", b"bar"] | bool | None | tuple[()]
reveal_type(x) # revealed: Literal[0, -1, "", "foo", b"", b"bar"] | bool | tuple[()] | None
else:
reveal_type(x) # revealed: Never

if not (x or not x):
reveal_type(x) # revealed: Never
else:
reveal_type(x) # revealed: Literal[0, -1, "", "foo", b"", b"bar"] | bool | None | tuple[()]
reveal_type(x) # revealed: Literal[0, -1, "", "foo", b"", b"bar"] | bool | tuple[()] | None

if (isinstance(x, int) or isinstance(x, str)) and x:
reveal_type(x) # revealed: Literal[-1, True, "foo"]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -346,4 +346,16 @@ static_assert(is_assignable_to(Never, type[str]))
static_assert(is_assignable_to(Never, type[Any]))
```

### `bool` is assignable to unions that include `bool`

Since we decompose `bool` to `Literal[True, False]` in unions, it would be surprisingly easy to get
this wrong if we forgot to normalize `bool` to `Literal[True, False]` when it appeared on the
left-hand side in `Type::is_assignable_to()`.

```py
from knot_extensions import is_assignable_to, static_assert

static_assert(is_assignable_to(bool, str | bool))
```

[typing documentation]: https://typing.readthedocs.io/en/latest/spec/concepts.html#the-assignable-to-or-consistent-subtyping-relation
Original file line number Diff line number Diff line change
Expand Up @@ -118,4 +118,48 @@ class R: ...
static_assert(is_equivalent_to(Intersection[tuple[P | Q], R], Intersection[tuple[Q | P], R]))
```

## Unions containing tuples containing `bool`

```py
from knot_extensions import is_equivalent_to, static_assert
from typing_extensions import Literal

class P: ...

static_assert(is_equivalent_to(tuple[Literal[True, False]] | P, tuple[bool] | P))
static_assert(is_equivalent_to(P | tuple[bool], P | tuple[Literal[True, False]]))
```

## Unions and intersections involving `AlwaysTruthy`, `bool` and `AlwaysFalsy`

```py
from knot_extensions import AlwaysTruthy, AlwaysFalsy, static_assert, is_equivalent_to, Not
from typing_extensions import Literal

static_assert(is_equivalent_to(AlwaysTruthy | bool, Literal[False] | AlwaysTruthy))
static_assert(is_equivalent_to(AlwaysFalsy | bool, Literal[True] | AlwaysFalsy))
static_assert(is_equivalent_to(Not[AlwaysTruthy] | bool, Not[AlwaysTruthy] | Literal[True]))
static_assert(is_equivalent_to(Not[AlwaysFalsy] | bool, Literal[False] | Not[AlwaysFalsy]))
```

## Unions and intersections involving `AlwaysTruthy`, `LiteralString` and `AlwaysFalsy`

```py
from knot_extensions import AlwaysTruthy, AlwaysFalsy, static_assert, is_equivalent_to, Not, Intersection
from typing_extensions import Literal, LiteralString

# TODO: these should all pass!

# error: [static-assert-error]
static_assert(is_equivalent_to(AlwaysTruthy | LiteralString, Literal[""] | AlwaysTruthy))
# error: [static-assert-error]
static_assert(is_equivalent_to(AlwaysFalsy | LiteralString, Intersection[LiteralString, Not[Literal[""]]] | AlwaysFalsy))
# error: [static-assert-error]
static_assert(is_equivalent_to(Not[AlwaysFalsy] | LiteralString, Literal[""] | Not[AlwaysFalsy]))
# error: [static-assert-error]
static_assert(
is_equivalent_to(Not[AlwaysTruthy] | LiteralString, Not[AlwaysTruthy] | Intersection[LiteralString, Not[Literal[""]]])
)
```

[the equivalence relation]: https://typing.readthedocs.io/en/latest/spec/glossary.html#term-equivalent
Original file line number Diff line number Diff line change
Expand Up @@ -449,5 +449,31 @@ static_assert(not is_subtype_of(Intersection[Unknown, int], int))
static_assert(not is_subtype_of(tuple[int, int], tuple[int, Unknown]))
```

## `bool` is a subtype of `AlwaysTruthy | AlwaysFalsy`

`bool` is equivalent to `Literal[True] | Literal[False]`. `Literal[True]` is a subtype of
`AlwaysTruthy` and `Literal[False]` is a subtype of `AlwaysFalsy`; it therefore stands to reason
that `bool` is a subtype of `AlwaysTruthy | AlwaysFalsy`.

```py
from knot_extensions import AlwaysTruthy, AlwaysFalsy, is_subtype_of, static_assert, Not, is_disjoint_from
from typing_extensions import Literal

static_assert(is_subtype_of(bool, AlwaysTruthy | AlwaysFalsy))

# the inverse also applies -- TODO: this should pass!
# See the TODO comments in the `Type::Intersection` branch of `Type::is_disjoint_from()`.
static_assert(is_disjoint_from(bool, Not[AlwaysTruthy | AlwaysFalsy])) # error: [static-assert-error]

# `Type::is_subtype_of` delegates many questions of `bool` subtyping to `int`,
# but set-theoretic types like intersections and unions are still handled differently to `int`
static_assert(is_subtype_of(Literal[True], Not[Literal[2]]))
static_assert(is_subtype_of(bool, Not[Literal[2]]))
static_assert(is_subtype_of(Literal[True], bool | None))
static_assert(is_subtype_of(bool, bool | None))

static_assert(not is_subtype_of(int, Not[Literal[2]]))
```

[special case for float and complex]: https://typing.readthedocs.io/en/latest/spec/special-types.html#special-cases-for-float-and-complex
[typing documentation]: https://typing.readthedocs.io/en/latest/spec/concepts.html#subtype-supertype-and-type-equivalence
71 changes: 64 additions & 7 deletions crates/red_knot_python_semantic/src/types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -811,6 +811,31 @@ impl<'db> Type<'db> {
}
}

/// Normalize the type `bool` -> `Literal[True, False]`.
///
/// Using this method in various type-relational methods
/// ensures that the following invariants hold true:
///
/// - bool ≡ Literal[True, False]
/// - bool | T ≡ Literal[True, False] | T
/// - bool <: Literal[True, False]
/// - bool | T <: Literal[True, False] | T
/// - Literal[True, False] <: bool
/// - Literal[True, False] | T <: bool | T
#[must_use]
pub fn with_normalized_bools(self, db: &'db dyn Db) -> Self {
const LITERAL_BOOLS: [Type; 2] = [Type::BooleanLiteral(false), Type::BooleanLiteral(true)];

match self {
Type::Instance(InstanceType { class }) if class.is_known(db, KnownClass::Bool) => {
Type::Union(UnionType::new(db, Box::from(LITERAL_BOOLS)))
}
// TODO: decompose `LiteralString` into `Literal[""] | TruthyLiteralString`?
// We'd need to rename this method... --Alex
_ => self,
}
}

/// Return a normalized version of `self` in which all unions and intersections are sorted
/// according to a canonical order, no matter how "deeply" a union/intersection may be nested.
#[must_use]
Expand Down Expand Up @@ -859,6 +884,12 @@ impl<'db> Type<'db> {
return true;
}

let normalized_self = self.with_normalized_bools(db);
let normalized_target = target.with_normalized_bools(db);
if normalized_self != self || normalized_target != target {
return normalized_self.is_subtype_of(db, normalized_target);
}

// Non-fully-static types do not participate in subtyping.
//
// Type `A` can only be a subtype of type `B` if the set of possible runtime objects
Expand Down Expand Up @@ -961,7 +992,7 @@ impl<'db> Type<'db> {
KnownClass::Str.to_instance(db).is_subtype_of(db, target)
}
(Type::BooleanLiteral(_), _) => {
KnownClass::Bool.to_instance(db).is_subtype_of(db, target)
KnownClass::Int.to_instance(db).is_subtype_of(db, target)
Comment on lines -964 to +995
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a stack overflow for a looong time here before I realised I had to change this... was tearing my hair out 🙃

Copy link
Contributor

@sharkdp sharkdp Jan 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you say more why this is necessary? And maybe add a comment (unless I'm missing something obvious)?

Copy link
Member Author

@AlexWaygood AlexWaygood Jan 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first step in Type::is_subtype_of() is now (with this PR) to normalize bool to Literal[True] | Literal[False] for both self and target. So if self is bool, self.is_subtype_of(target) delegates to whether Literal[True] | Literal[False] is a subtype of target. That takes us to the Type::Union branch: a union type is a subtype of target iff all its members are a subtype of target. So then we try the first member of the union Literal[True] | Literal[False], and ask the question: "Is Literal[True] a subtype of target? If target is not equivalent to Literal[True], that takes us to this branch, and this branch says "Literal[True] is a subtype of target iff bool is a subtype of target". So then we ask the question "Is bool a subtype of target?", which is the very first question we started off with. And bool is normalized to Literal[True] | Literal[False]... the cycle repeats indefinitely, resulting in infinite recursion.

Copy link
Contributor

@sharkdp sharkdp Jan 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

I still don't understand why it's correct though? There are types which are supertypes of bool, but not of int. bool | None or ~Literal[2], for example. So why is it correct to replace the bool <: target check with a int <: target check? Because all special cases have been handled above?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't understand why it's correct though?

It might not be -- I have been known to make mistakes 😜

There are types which are supertypes of bool, but not of int. bool | None or ~Literal[2], for example. So why is it correct to replace the bool <: target check with a int <: target check? Because all special cases have been handled above?

Yeah, I think our set-theoretic types such as unions and intersections should be handled in the branches above? I just added more test cases demonstrating this.

(I will add a comment to the code reflecting this conversation, but first I'm experimenting with alternative ways of structuring the code that might make this clearer in the first place!)

Copy link
Member Author

@AlexWaygood AlexWaygood Jan 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(#15773 is the "alternative way of structuring the code", for anybody else reading this thread)

}
(Type::IntLiteral(_), _) => KnownClass::Int.to_instance(db).is_subtype_of(db, target),
(Type::BytesLiteral(_), _) => {
Expand Down Expand Up @@ -1077,6 +1108,11 @@ impl<'db> Type<'db> {
if self.is_gradual_equivalent_to(db, target) {
return true;
}
let normalized_self = self.with_normalized_bools(db);
let normalized_target = target.with_normalized_bools(db);
if normalized_self != self || normalized_target != target {
return normalized_self.is_assignable_to(db, normalized_target);
}
match (self, target) {
// Never can be assigned to any type.
(Type::Never, _) => true,
Expand Down Expand Up @@ -1177,6 +1213,13 @@ impl<'db> Type<'db> {
pub(crate) fn is_equivalent_to(self, db: &'db dyn Db, other: Type<'db>) -> bool {
// TODO equivalent but not identical types: TypedDicts, Protocols, type aliases, etc.

let normalized_self = self.with_normalized_bools(db);
let normalized_other = other.with_normalized_bools(db);

if normalized_self != self || normalized_other != other {
return normalized_self.is_equivalent_to(db, normalized_other);
}

match (self, other) {
(Type::Union(left), Type::Union(right)) => left.is_equivalent_to(db, right),
(Type::Intersection(left), Type::Intersection(right)) => {
Expand Down Expand Up @@ -1218,6 +1261,13 @@ impl<'db> Type<'db> {
///
/// [Summary of type relations]: https://typing.readthedocs.io/en/latest/spec/concepts.html#summary-of-type-relations
pub(crate) fn is_gradual_equivalent_to(self, db: &'db dyn Db, other: Type<'db>) -> bool {
let normalized_self = self.with_normalized_bools(db);
let normalized_other = other.with_normalized_bools(db);

if normalized_self != self || normalized_other != other {
return normalized_self.is_gradual_equivalent_to(db, normalized_other);
}

if self == other {
return true;
}
Expand Down Expand Up @@ -1250,6 +1300,12 @@ impl<'db> Type<'db> {
/// Note: This function aims to have no false positives, but might return
/// wrong `false` answers in some cases.
pub(crate) fn is_disjoint_from(self, db: &'db dyn Db, other: Type<'db>) -> bool {
let normalized_self = self.with_normalized_bools(db);
let normalized_other = other.with_normalized_bools(db);
if normalized_self != self || normalized_other != other {
return normalized_self.is_disjoint_from(db, normalized_other);
}

match (self, other) {
(Type::Never, _) | (_, Type::Never) => true,

Expand Down Expand Up @@ -4642,18 +4698,19 @@ pub struct TupleType<'db> {
}

impl<'db> TupleType<'db> {
pub fn from_elements<T: Into<Type<'db>>>(
db: &'db dyn Db,
types: impl IntoIterator<Item = T>,
) -> Type<'db> {
pub fn from_elements<I, T>(db: &'db dyn Db, types: I) -> Type<'db>
where
I: IntoIterator<Item = T>,
T: Into<Type<'db>>,
{
let mut elements = vec![];

for ty in types {
let ty = ty.into();
let ty: Type<'db> = ty.into();
if ty.is_never() {
return Type::Never;
}
elements.push(ty);
elements.push(ty.with_normalized_bools(db));
}

Type::Tuple(Self::new(db, elements.into_boxed_slice()))
Expand Down
Loading
Loading