bug in the linenumbers for generator/list/set/dict expressions (python 3.8-3.10)

[15r10nk]

The following code produces the wrong line number in the traceback.

script:

import dis

def f():
    return list(q+"" for q in [""])==list(

        q+"" for q in [5])



dis.dis(f.__code__)
f()

output (Python 3.8.12):

  4           0 LOAD_GLOBAL              0 (list)
              2 LOAD_CONST               1 (<code object <genexpr> at 0x7f6d135df9d0, file "genexpr.py", line 4>)
              4 LOAD_CONST               2 ('f.<locals>.<genexpr>')
              6 MAKE_FUNCTION            0
              8 LOAD_CONST               3 (('',))
             10 GET_ITER
             12 CALL_FUNCTION            1
             14 CALL_FUNCTION            1
             16 LOAD_GLOBAL              0 (list)
             18 LOAD_CONST               1 (<code object <genexpr> at 0x7f6d135df9d0, file "genexpr.py", line 4>)
             20 LOAD_CONST               2 ('f.<locals>.<genexpr>')
             22 MAKE_FUNCTION            0

  6          24 LOAD_CONST               4 ((5,))

  4          26 GET_ITER
             28 CALL_FUNCTION            1
             30 CALL_FUNCTION            1
             32 COMPARE_OP               2 (==)
             34 RETURN_VALUE

Disassembly of <code object <genexpr> at 0x7f6d135df9d0, file "genexpr.py", line 4>:
  4           0 LOAD_FAST                0 (.0)
        >>    2 FOR_ITER                14 (to 18)
              4 STORE_FAST               1 (q)
              6 LOAD_FAST                1 (q)
              8 LOAD_CONST               0 ('')
             10 BINARY_ADD
             12 YIELD_VALUE
             14 POP_TOP
             16 JUMP_ABSOLUTE            2
        >>   18 LOAD_CONST               1 (None)
             20 RETURN_VALUE
Traceback (most recent call last):
  File "genexpr.py", line 11, in <module>
    f()
  File "genexpr.py", line 4, in f
    return list(q+"" for q in [""])==list(
  File "genexpr.py", line 4, in <genexpr>
    return list(q+"" for q in [""])==list(                  
TypeError: unsupported operand type(s) for +: 'int' and 'str'

The traceback shows that q is iterating over [""] and the exception says that q is an int.

the expected output in the traceback would be:

  File "genexpr.py", line 6, in <genexpr>
    q+"" for q in [5])

The reason seems to be that the both gerator-expressions are using the same bytecode (it should not be the same bytecode, because the code is on different lines).

Adding one newline changes the situation (which I can not explain).

script:

import dis

def f():
    return list(
        q+"" for q in [""])==list(

        q+"" for q in [5])



dis.dis(f.__code__)
f()

output (Python 3.8.12):

  4           0 LOAD_GLOBAL              0 (list)
              2 LOAD_CONST               1 (<code object <genexpr> at 0x7f493201c9d0, file "genexpr.py", line 4>)
              4 LOAD_CONST               2 ('f.<locals>.<genexpr>')
              6 MAKE_FUNCTION            0

  5           8 LOAD_CONST               3 (('',))

  4          10 GET_ITER
             12 CALL_FUNCTION            1
             14 CALL_FUNCTION            1

  5          16 LOAD_GLOBAL              0 (list)
             18 LOAD_CONST               4 (<code object <genexpr> at 0x7f493201ca80, file "genexpr.py", line 5>)
             20 LOAD_CONST               2 ('f.<locals>.<genexpr>')
             22 MAKE_FUNCTION            0

  7          24 LOAD_CONST               5 ((5,))

  5          26 GET_ITER
             28 CALL_FUNCTION            1
             30 CALL_FUNCTION            1

  4          32 COMPARE_OP               2 (==)
             34 RETURN_VALUE

Disassembly of <code object <genexpr> at 0x7f493201c9d0, file "genexpr.py", line 4>:
  4           0 LOAD_FAST                0 (.0)
        >>    2 FOR_ITER                14 (to 18)

  5           4 STORE_FAST               1 (q)
              6 LOAD_FAST                1 (q)
              8 LOAD_CONST               0 ('')
             10 BINARY_ADD
             12 YIELD_VALUE
             14 POP_TOP
             16 JUMP_ABSOLUTE            2
        >>   18 LOAD_CONST               1 (None)
             20 RETURN_VALUE

Disassembly of <code object <genexpr> at 0x7f493201ca80, file "genexpr.py", line 5>:
  5           0 LOAD_FAST                0 (.0)
        >>    2 FOR_ITER                14 (to 18)

  7           4 STORE_FAST               1 (q)
              6 LOAD_FAST                1 (q)
              8 LOAD_CONST               0 ('')
             10 BINARY_ADD
             12 YIELD_VALUE
             14 POP_TOP
             16 JUMP_ABSOLUTE            2
        >>   18 LOAD_CONST               1 (None)
             20 RETURN_VALUE
Traceback (most recent call last):
  File "genexpr.py", line 12, in <module>
    f()
  File "genexpr.py", line 5, in f
    q+"" for q in [""])==list(
  File "genexpr.py", line 7, in <genexpr>
    q+"" for q in [5])
TypeError: unsupported operand type(s) for +: 'int' and 'str'

I bisected the bug town to this commit: b619b09

The same problem exists also for list/set/dict-expressions

[sobolevn]

Looks like it is fixed on main and 3.11:

Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/ex.py", line 12, in <module>
    f()
  File "/Users/sobolev/Desktop/cpython/ex.py", line 5, in f
    q+"" for q in [""])==list(
                         ^^^^^
  File "/Users/sobolev/Desktop/cpython/ex.py", line 7, in <genexpr>
    q+"" for q in [5])
    ~^~~
TypeError: unsupported operand type(s) for +: 'int' and 'str'

[15r10nk]

Yes, there is no problem in 3.11.

One similar issue I know there is #95150 (which I reported) but this was only a problem if the comprehension are on the same line.

There where some bigger changes in 3.11 for the line number code. It might be that this issue was fixed without knowing that it existed. But this is only speculation.

[iritkatriel]

3.10 is in security-fix only mode now, so this is out of date.