IDLE is unable to open any `.py` files

[AlexWaygood]

With a fresh CPython build (be0c106), IDLE is unable to open any .py files.

To reproduce:

1. Create an empty .py file with the name repro.py
2. Run python -m idlelib repro.py

IDLE still seems able to create new .py files and save them; it just can't open pre-existing .py files right now.

Traceback observed

C:\Users\alexw\coding\cpython>python -m idlelib repro.py
Running Debug|x64 interpreter...
Traceback (most recent call last):
  File "C:\Users\alexw\coding\cpython\Lib\runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\__main__.py", line 7, in <module>
    idlelib.pyshell.main()
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\pyshell.py", line 1640, in main
    if flist.open(filename) is None:
       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\filelist.py", line 37, in open
    edit = self.EditorWindow(self, filename, key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\pyshell.py", line 135, in __init__
    EditorWindow.__init__(self, *args)
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 289, in __init__
    self.set_indentation_params(is_py_src)
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 1327, in set_indentation_params
    i = self.guess_indent()
        ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 1574, in guess_indent
    opener, indented = IndentSearcher(self.text, self.tabwidth).run()
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 1646, in run
    save_tabsize = tokenize.tabsize
                   ^^^^^^^^^^^^^^^^
AttributeError: module 'tokenize' has no attribute 'tabsize'

Environment

(Given the cause of the bug, the environment details shouldn't really be relevant; but they're included here anyway, for completeness.)

Python 3.12.0a7+ (heads/main:be0c106789, May 21 2023, 12:00:27) [MSC v.1932 64 bit (AMD64)] on win32

Reproduces on a debug and non-debug build, FWIW.

Linked PRs

• gh-104719: Restore Tokenize module constants #104722
• gh-104719: IDLE - delete useless monkeypatch of tokenize #104726
• [3.11] gh-104719: IDLE - delete useless monkeypatch of tokenize (GH-104726) #104727
• gh-104719: IDLE - test existence of all tokenize references. #104767
• [3.12] gh-104719: IDLE - test existence of all tokenize references. (GH-104767) #104844
• [3.11] gh-104719: IDLE - test existence of all tokenize references. (GH-104767) #104845

[AlexWaygood]

Bisects to 6715f91 (unsurprisingly 😄)

Cc. @mgmacias95, @pablogsal, @lysnikolaou, @isidentical

[pablogsal]

Will check this today

[pablogsal]

Looks like IDLE is using a bunch of non documented attributes of the tokenize module that are no longer used internally. The easier course of action is probably just restore these but note that there will be no guarantee that whatever idle is going with them will continue working.

I ran the buildbots and the CI before merge so another question is why CI didn't catch this.

[AlexWaygood]

I ran the buildbots and the CI before merge so another question is why CI didn't catch this.

Do any of the buildbots run the test suite with -u gui (or -u all), I wonder? A bunch of IDLE's tests actually open and close GUI windows, so they're not run unless you run the test suite with that option.

[AlexWaygood]

I ran the buildbots and the CI before merge so another question is why CI didn't catch this.

Do any of the buildbots run the test suite with -u gui (or -u all), I wonder? A bunch of IDLE's tests actually open and close GUI windows, so they're not run unless you run the test suite with that option.

That doesn't seem relevant; python -m test test_idle -u gui passes on main, on both debug and non-debug builds. So it does seem like there's some missing test coverage here.

[pablogsal]

I checked a bit more how the constant that was eliminated is being used and seems that IDLE is monkey patching the value to modify the behaviour of the now deleted _tokenize private function to adapt something regarding tabsize.

Unfortunately I don't think this is going to work and was out of contract anyway so I think the best course of action here is to eliminate that code from idlelib as restoring the constant will not restore whatever was happening before.