inspect.getsourcelines() is buggy on 3.12

[JelleZijlstra]

% cat gsl.py
import inspect

def test_list_add(self):
    def capybara() -> None:
        assert_is_value(
            [x] + [y],
            z,
        )
        assert_is_value(
            # in the binop implementation
            a,
        )

print(inspect.getsourcelines(test_list_add))
% python3.11 gsl.py 
(['def test_list_add(self):\n', '    def capybara() -> None:\n', '        assert_is_value(\n', '            [x] + [y],\n', '            z,\n', '        )\n', '        assert_is_value(\n', '            # in the binop implementation\n', '            a,\n', '        )\n'], 3)
% ~/py/cpython/python.exe gsl.py 
(['def test_list_add(self):\n', '    def capybara() -> None:\n', '        assert_is_value(\n', '            [x] + [y],\n', '            z,\n', '        )\n', '        assert_is_value(\n', '            # in the binop implementation\n'], 3)
% ~/py/cpython/python.exe -V
Python 3.12.0b1+

On 3.12, the last two lines (a, )) are not returned as part of the source lines. When I tried to minify by removing the first assert_is_value() call, it instead returned the print() line as part of the function.

This looks related to the tokenizer, cc @pablogsal @lysnikolaou.

Linked PRs

• gh-104866: Tokenize should emit NEWLINE after exiting block with comment #104870
• [3.12] gh-104866: Tokenize should emit NEWLINE after exiting block with comment (GH-104870) #104872

[lysnikolaou]

This is a bug in the implementation of the new tokenize module. The tokenizer checks whether something is in a parenthesized block in order to emit a NL token instead of a NEWLINE token. It also sets and unsets a flag about a comment-only line, in order to do the same. When both are true (a comment inside a parenthesized block), it nevers resets the comment flag. Here's a minimal example:

cpython on  main [$?] via C v14.0.3-clang via 🐍 pyenv 3.11.3
❯ cat hello.py              
[
    # hello
    1
]

cpython on  main [$?] via C v14.0.3-clang via 🐍 pyenv 3.11.3
❯ ./python.exe -m tokenize hello.py
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,1:            OP             '['            
1,1-1,2:            NL             '\n'           
2,4-2,11:           COMMENT        '# hello'      
2,11-2,12:          NL             '\n'           
3,4-3,5:            NUMBER         '1'            
3,5-3,6:            NL             '\n'           
4,0-4,1:            OP             ']'            
4,1-4,2:            NL             '\n'       # This should be a NEWLINE instead    
5,0-5,0:            ENDMARKER      ''             

I'm opening a PR shortly.