Skip to content

Support getting files in the build tree from editable installs #808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Support getting files in the build tree from editable installs #808

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
999a675
Demonstrate where a potential fix could be and why it doesn't seem to…
vyasr Jul 13, 2024
26129bc
Fix or ignore all style errors
vyasr Apr 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions src/scikit_build_core/resources/_editable_redirect.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
from __future__ import annotations

import importlib.abc
import importlib.machinery

# TODO: importlib.readers is Python version specific
import importlib.readers # type: ignore[import-not-found]
Copy link
Collaborator

@LecrisUT LecrisUT Oct 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did some experimentation a while back. I believe what we should do here is to inject importlib-readers dependency and use a version check for this

if sys.version_info >= (3, 10):
    import importlib_resources
else:
    import importlib.resources as importlib_resources

I don't remember which versions we had issues with, but if we need a very specific version of importlib-resources we should use it. I previously had concerns with mixing these two libraries, but I found out it was not really an issue, the Loader or Spec or something else dictates which library will be used throughout the current file.

import importlib.util
import os
import subprocess
Expand All @@ -11,6 +15,7 @@
TYPE_CHECKING = False
if TYPE_CHECKING:
from importlib.machinery import ModuleSpec
from pathlib import Path


DIR = os.path.abspath(os.path.dirname(__file__))
Expand Down Expand Up @@ -77,6 +82,27 @@ def release(self) -> None:
os.close(self.lock_file_fd)


# Note: This solution relies on importlib's call stack in Python 3.11. Python 3.9 looks
# different, so might require a different solution, but I haven't gone deeper into that
# yet since I don't have a solution for the 3.11 case yet anyway.
class ScikitBuildRedirectingReader(importlib.readers.FileReader): # type: ignore[misc]
def files(self) -> Path:
# ATTENTION: This is where the problem is. The expectation is that this returns
# a Traversable object. We could hack together an object that satisfies that
# API, but methods like `joinpath` don't have sensible implementations if
# `files` could return multiple paths instead of a single one. We could do some
# hackery to figure out which paths exist on the backend by hiding some internal
# representation that knows both possible roots and checks for existence when
# necessary, but that seriously violates the principle of least surprise for the
# user so I'd be quite skeptical.
return self.path # type: ignore[no-any-return]


class ScikitBuildRedirectingLoader(importlib.machinery.SourceFileLoader):
def get_resource_reader(self, module: str) -> ScikitBuildRedirectingReader: # type: ignore[override]
return ScikitBuildRedirectingReader(self)


class ScikitBuildRedirectingFinder(importlib.abc.MetaPathFinder):
def __init__(
self,
Expand Down Expand Up @@ -153,6 +179,9 @@ def find_spec(
submodule_search_locations=submodule_search_locations
if redir.endswith(("__init__.py", "__init__.pyc"))
else None,
loader=ScikitBuildRedirectingLoader(
fullname, os.path.join(self.dir, redir)
),
)
if fullname in self.known_source_files:
redir = self.known_source_files[fullname]
Expand All @@ -162,6 +191,7 @@ def find_spec(
submodule_search_locations=submodule_search_locations
if redir.endswith(("__init__.py", "__init__.pyc"))
else None,
loader=ScikitBuildRedirectingLoader(fullname, redir),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that this is specifying a single directory (the redir) is perhaps part of where we could improve things. (Sub)packages are found in the known source files list by their __init__.py. As a result, that path is used as the root, which means that we cannot find files nested in the package but present in the build directory because the package does not exist in known_wheel_files.

)
return None

Expand Down
Loading