RFC Add micropip redirect API

[rth]

Related to pyodide/pyodide#1986 the limit between,

• packages that work with a few patches
• packages that only work after a significant rewrite (i.e. lots of patches)

is not very clear. For instance, if pyodide/pyodide#1956 re-implements requests with WebAPI can we really say we are still using requests, as opposed to pyodide-requests a fork? Maybe we could define the limit as,

• if patches could one day be contributed upstream (with maybe a few exceptions for non-essential functionality e.g. file download in pandas): it's still a patched original package
• if there are lots of patches for the core functionality that will never be contributed upstream: it's a fork. If it's a fork it should have a separate name , as pyodide-<package-name>.

However, even in the second case, it might still make sense to install it e.g. as the original name (e.g. requests) since lots of packages would have this in dependencies, and won't bother with,

setup(
    ...,
    install_requires=[
        "pyodide-requests; platform_system=='Emscripten'",
        "requests; platform_system!='Emscripten'",
    ],
)

In which case, we could implement some redirect system in micropip, so that,

micropip.install('requests')

would install pyodide-requests instead.

It could be maybe just a dict,

micropip.redirects = {'requests': 'pyodide-requests'}

but that requires good documentation. Unless micropip.add_redirect / micripip.remove_redirect would be better?

What do you think?

[bollwyvl]

We've been fretting pretty hard about this over on jupyterlite, as we have some wontfix upstream stuff that we must patch around.

To do this, both for our first-party efforts like pyolite (or whatever we rename it to, eventually) and for various other use cases, we've started exploring a layered approach wrapped up in piplite, a wrapper around micropip. It basically offers the ability to provide a number of Warehouse-like canned JSON responses (one of which is ours that gets shipped in our webpack monster). It is likely this will grow some other features (e.g. a replacement for the %pip magic), but as suggested by @dsblank, maybe we can move some of this stuff upstream.

In the conda world, this is not dissimilar to channel_priority: strict whereby one could, say, always install pytorch (and any patched dependencies) from the pytorch channel, while being able to install everything else from conda-forge... looking to the day when we can offer %conda, it would be relatively easy for a wasm-forge to be a separate effort, at least to start, especially if maintainers weren't interested in these jumping jacks.

Anyhow, here is the most recent form of the monkeypatches, as part of the larger, long-winded PR, which includes docs and build-time concerns. At present it uses a _PIPLITE_URLS: List[str], but one could imagine it as an OrderedDict or something that maintained order and address-ability.

[ryanking13]

I also think that the best long term solution would be making a separate channel, which can be a all-in-one solution for not only redirections but also for a patched packages and binary wheels for Pyodide (#655).

micropip.redirects = {'requests': 'pyodide-requests'}

I'm +1 one this for a short-term solution for a redirection, which indeed needs a good documentation for packages (#1986).

Unless micropip.add_redirect / micripip.remove_redirect would be better?

I think we can make this API private, because most end users won't need this.

[bollwyvl]

binary wheels for Pyodide

I don't know if wheels are enough for a complete dependency solution without something else to handle non-python dependencies in a maintainable fashion. I really hope that the stars align to make conda-forge/conda-forge.github.io#1401 a reality!

most end users

Yet another place I've been getting tripped up over on jupyterlite: while someone writing python-in-the-browser might not need/want to do this kind of thing, I think a deployer of a pyodide-enabled site will greatly appreciate having stable, documented APIs for changing how imports work to make their magic sauce work. The work we're doing on piplite, aside from simplifying/codifying some of our (self-inflicted) build problems, is already enabling use cases a couple of degrees removed, namely kernels that re-use the ipykernel infrastructure to implement other languages entirely.

[rth]

I also think that the best long term solution would be making a separate channel, which can be a all-in-one solution for not only redirections but also for a patched packages and binary wheels for Pyodide

I'm all for having priority channels in micropip, and if you are interested in contributing that @bollwyvl it would be very welcome. Related pyodide/pyodide#575

I don't know if wheels are enough for a complete dependency solution without something else

Agreed, wheels are nice but it's probably not ideal to bundle C dependencies in each wheel as one would do with the standard Python build pyodide/pyodide#655 (comment)

[rth]

Though even with a separate channel, I feel that distributing significantly rewritten packages that might be partially working (e.g. no binary downloads in pyodide-requests for now) under the same name as the original package is problematic. In particular, a number of large packages have a trademark on their name, and if such a modification is considered a fork, I imagine using the same name for it could be considered a trademark violation. Particularly because the user experience would be worse due to things partially working, which would be negatively associated with the package name.

Anyway, I'm not a lawyer, but using a separate name in those packages makes more sense to me. In which case, this redirect mechanism would still be useful.

[bollwyvl]

priority channels in micropip, and if you are interested in contributing

The biggest reason I haven't is I've gone ahead and made up a new file format, namely one that puts every response from the warehouse API into a single, package namespaced file... partially, this is because I can't reliably serve responses from /json endpoints, but also to avoid 100s of requests. In this case, it's unlikely i'd ever be serving more than one version of a package. It's got a schema, but meh...

I guess I crave a countable number of files to deploy, sufficient to express:

• pyodide's own home-rolled packages.json
• (a better) warehouse API
• someday, conda's repodata.json

considered a trademark violation

While "official distributions" are nice, I am not aware of a single, full-featured packaging solution that doesn't permit, and in many cases require patches in order to function outside of the authors' original intent. I've never seen an open source package get angry about this... but then, some of these things we're having to do in the browser are rather nasty.

That being said, there is a long (and maybe questionable) history of a difference between the distribution and importable names of python packages... does overloading an importable also trigger warnings?

If we get to where micropip.check() works (which is good!), however, it would be required to match the distribution name as well.