Strange import errors with Python 3.12 on Windows

[pekkaklarck]

I tried to test our project with Python 3.12 beta 1 on Windows but everything failed. After some debugging I noticed that module imports seem to fail when modules aren't on my C-drive:

C:\Users\peke>echo print(1) > test312.py

C:\Users\peke>py -3.12 -c "import test312"
1

C:\Users\peke>e:

E:\>echo print(1) > test312.py

E:\>py -3.12 -c "import test312"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'test312'

No problems with earlier Python versions:

E:\>py -3.11 -c "import test312"
1

Not sure does it matter, but I'm running Windows on VirtualBox and that E-drive is mapped to a directory on the Linux host.

Linked PRs

• gh-104820: Fixes os.stat on Windows to better handle file systems that do not support FileIdInformation #104892
• [3.12] gh-104820: Fixes os.stat on Windows to better handle file systems that do not support FileIdInformation (GH-104892) #104905

[eryksun]

This is due to a bug in os.stat() for filesystems that lack support for FileIdInfo. The same bug is also the cause of the problem with pathlib.Path.is_dir() that's reported in gh-104806. pathlib.Path.is_dir() uses os.stat(), while ntpath.isdir() uses nt._path_isdir(). For example, volume "G:" on my system contains an exFAT filesystem, which doesn't support FileIdInfo.

>>> nt._path_isdir('G:\\')
True
>>> stat.S_ISDIR(os.stat('G:\\').st_mode)
False
>>> stat.S_ISBLK(os.stat('G:\\').st_mode)
True

>>> nt._path_isfile('G:\\spam.txt')
True
>>> stat.S_ISREG(os.stat('G:\\spam.txt').st_mode)
False
>>> stat.S_ISBLK(os.stat('G:\\spam.txt').st_mode)
True

As shown above, os.stat() is mistakenly reporting files and directories on this volume as block devices.

@zooba, in win32_xstat_slow_impl() in "Modules/posixmodule.c", the FileIdInfo request isn't universally supported by filesystem drivers. For example, it's not supported by FAT32/exFAT and, as demonstrated by this issue, it's not supported by the VirtualBox shared-folder filesystem.

        if (!GetFileInformationByHandle(hFile, &fileInfo) ||
            !GetFileInformationByHandleEx(hFile, FileBasicInfo,
                                          &basicInfo, sizeof(basicInfo)) ||
            !GetFileInformationByHandleEx(hFile, FileIdInfo,
                                          &idInfo, sizeof(idInfo))) {
            switch (GetLastError()) {
            case ERROR_INVALID_PARAMETER:
            case ERROR_INVALID_FUNCTION:
            case ERROR_NOT_SUPPORTED:
                /* Volumes and physical disks are block devices, e.g.
                   \\.\C: and \\.\PhysicalDrive0. */
                memset(result, 0, sizeof(*result));
                result->st_mode = 0x6000; /* S_IFBLK */
                goto cleanup;
            }
            retval = -1;
            goto cleanup;
        }

I'd add a new pointer variable, p_idInfo. If the request fails, set p_idInfo = NULL. Otherwise set p_idInfo = &idInfo. _Py_attribute_data_to_stat() in "Python/fileutils.c" falls back on the 64-bit file ID from the BY_HANDLE_FILE_INFORMATION if the id_info parameter is a NULL pointer.

Also, to err on the side of caution, _Py_attribute_data_to_stat() should fall back on the 64-bit file ID if the 128-bit file ID is 0 (i.e. both the low and high 64-bit parts are 0). The latter is the required value specified in [MS-FSCC] if a filesystem doesn't support a 128-bit file ID (even with the high 64-bit part set to 0) but for some reason the driver implements the FileIdInformation information class. I don't have an example of this. Usually it's either supported or requesting FileIdInformation fails, but the specification says we should be prepared to handle a zero value.

Also, when id_info is available, I'd prefer to use its 64-bit VolumeSerialNumber field for st_dev, which is consistent with the new by-name fast path. Else fall back on the 32-bit dwVolumeSerialNumber from the BY_HANDLE_FILE_INFORMATION. NTFS and ReFS have always supported a 64-bit volume serial number.

[eryksun]

@zooba

Also, when id_info is available, I'd prefer to use its 64-bit VolumeSerialNumber field for st_dev

Sorry, Steve. I missed that you had already implemented this when I scanned over the code yesterday. I should have read it more carefully.