-
-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for constraints, install wheels before pyodide packages #181
Add support for constraints, install wheels before pyodide packages #181
Conversation
for more information, see https://pre-commit.ci
…yodidegh-175-constraints
…de/wheel install order
@ryanking13 @hoodmane this should Do The Thing without breaking as much! |
CHANGELOG.md
Outdated
- `micropip.install` now installs wheels from PyPI or custom indexes before built-in | ||
Pyodide packages, reversing the previous behavior. | ||
[#181](https://github.com/pyodide/micropip/pull/181) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a user-facing change related to this? I think the order of installation is an implementation detail that is not visible to the user. What a user sees whether the whole set of packages (with dependencies) are installed or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I don't know: seems like In interactive package management, which can be modified by about five different flags now, I think "reversing what we used to do," warrants at least a note. I am not sure if it's breaking, but things folk did before might not work now, and they might not need workarounds (like the double-install thing).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. Makes sense. Let me check the code more carefully and review.
I think the current wording could be a bit misleading to users though. When the same package exists in both the pyodide lock file and PyPI, the lockfile currently takes precedence (in terms of which package to choose, not the order of installation). This PR is not related to it, but maybe the user will think that this PR is changing that behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right: today, in the transaction phase once a dependency chain hits a pyodide package, further dependency resolution was skipped because the lockfile would take care of the rest. This was a performance win, for sure. In the install stage, they were all mixed together in wheel_tasks
, and there was really no way to make any claims about what would happen.
With this change, the transaction will keep going all the way down for every novel dependency. In the install phase, the PyPI wheels get installed (in parallel) before calling loadPackage
, which won't re-install (and re-download) a pyodide wheel.
This does mean there will be an extra add_dependency_inner
for each depends
item, but I couldn't figure out a way to do it that would be remotely correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
... but that's a lot to say in a CHANGELOG, maybe there's a terser way?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
tried some more things on 92baa16
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With this change, the transaction will keep going all the way down for every novel dependency. In the install phase, the PyPI wheels get installed (in parallel) before calling loadPackage, which won't re-install (and re-download) a pyodide wheel.
Hmm, I still don't quite get what which won't re-install
means. By default, during the dependency resolution step, micropip will check if the package is available in the lockfile, and select the one from the lockfile instead of the one from the PyPI.
So, in the installation step, we expect no overlap between transaction.pyodide_packages
and transaction.wheels
. Does this PR change this behavior?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it. I didn't consider putting multiple packages in install(...)
. Thanks.
Will work on merge conflicts. |
…raints-wheels-first
Merging
|
Ah, it looks like the
... but also no idea why it's not failing locally... |
Back to ✔️ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me overall. Thanks for your work!
if self.verbose and messages: | ||
for constraint, msg in messages.items(): | ||
logger.info("Transaction: constraint %s discarded: %s", constraint, msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the long run, it seems better to change it so that the installation fails when constraints resolution fails (just like pip's behavior). However, micropip's current dependency resolution is incomplete on its own, so let's leave it for now.
Thanks all! |
Released in https://github.com/pyodide/micropip/releases/tag/v0.9.0, thank you! |
references
changes
transaction.py
install.py
loadPackage
magically fulfilling the dependencies which may be inconstraints
breaking changes