Allow using maturin develop outside of virtualenv

[programmerjake]

This seems like something that would be useful to allow, perhaps requiring a --force-develop flag of some sort if retaining the current error message seems useful to direct beginners to using virtualenv.

[konstin]

When would this be required? I generally advise against installing anything but standalone cli tools into the global environment, and even that breaks easily. I wouldn't want to encourage using the user/system site packages with maturin, especially since it's still simple enough to use maturin build -i python && pip install <...>.

[programmerjake]

I submitted this bug report for my coworker who doesn't have a github account. Here's his response:

i did manage to use "pip3 install something.whl" however it was the
complete lack of documentation that that was even possible which
tripped me up. this gave the false impression that the only thing
possible was to use virtualenv.

So maybe adding documentation to the error message on how to install using maturin build && pip install ... is what's needed.

[g-braeunlich]

setuptools also has

./setup.py build_ext --inplace

This would build the .so and move it directly into the python package folder.
I personally would also like this.

[konstin]

I've updated the message to make the options clearer. I'm however not interested in supporting anything that could break the user's global environment.

[ntakouris]

I'm using docker for development so I don't really care about using venv or something else. We still need a --skip-venv-check option.

For now, I am doing pip install . but I am not sure if it does a release configuration.

[konstin]

Is your dev workflow to maturin develop into the global environment inside the docker container? If so, is there a reason not to have a virtualenv inside the container? It would be cached build step so it wouldn't incur any overhead

For now, I am doing pip install . but I am not sure if it does a release configuration.

It actually always does a release build, PEP 517 has no way to communicate whether you want dev or prod (https://discuss.python.org/t/pep-517-debug-vs-release-builds/1924)

[ntakouris]

There is no reason to have a virtual env with just one env. It's the dev docker container and contains everything needed for that project. You can revert to use virutalenv inside docker but

1. It's unnecessary; isolation of dev env is the whole point of using container for dev
2. I will have to migrate every other shell/ci/etl script to use that

I wouldn't force people to use virtualenv if features do not directly depend on venv specific internals. It's just a python env. Maybe remove the venv guard and print a warning without a --i-understand flag being present.

[thehappycheese]

I am trying to run maturin on repl.it but I can't do it because repl's package management system is preventing me from installing venv :(

All I want is for maturin to do its thing and put the dev binary in the python_src folder

See broken repl here: https://replit.com/@thehappycheese/Megamerge#pyproject.toml ... hopefully this link takes you to the explorer/editor view and not the stupid preview screen that repl.it sometimes shows when you are not logged in.

[konstin]

you can still python -m venv .venv and then activate that

[thehappycheese]

you can still python -m venv .venv and then activate that

Not true. Try it. repl.it prevents the installation of the venv package, and it is not in the preinstalled libraries as you might expect, so that is impossible.

To be fair, its more of a problem with repl.it than with maturin. I understand why maturin wants to be inside a venv. But still maturin seems at least 10% at fault for being so inflexible.

[udalrich]

I would like this feature for use within a CI job. We have a docker image that is used for python CI builds. It is based on something fairly old and was providing pip 18.x. So I updated the Dockerfile to update pip to the current version, which worked fine.

I then started trying to use it for a mixed rust/python project. Since I am in a single use container, there is no problem with installing into the global environment. So I tried

maturin build
python -m pip install target/wheels/my-project.whl[test]
python -m pytest my_project

which is apparently not the correct way to install the maturin wheel locally. After several more tries, I gave up on maturin build and tried maturin develop, since that does the right thing (tm). That failed, because there was not a venv.

So I created a venv, activated it and ran the commands inside the venv. Which then failed, because the venv was using pip 18, not the updated pip from the global environment.

Eventually, I got it working by creating the venv, activating the venv, updating pip, and then doing the stuff I actually wanted to do. Which is a lot of extra code to understand when looking at the CI config file, plus extra work the CI server is doing, making the job take longer. And if there is some other tool in the global workspace that was also updated, I'll need to update the in the venv as well.

So it is a lot harder to understand and maintain, and slower to run, that being able to do

maturin develop --without-venv-which-is-a-bad-idea-if-this-is-not-a-throwaway-environment

[konstin]

i don't know you're exact setup, but there's nothing that maturin develop can do and that maturin build followed by pip install doesn't.

Eventually, I got it working by creating the venv, activating the venv, updating pip, and then doing the stuff I actually wanted to do. Which is a lot of extra code to understand when looking at the CI config file, plus extra work the CI server is doing, making the job take longer. And if there is some other tool in the global workspace that was also updated, I'll need to update the in the venv as well.

if you do pip install virtualenv && virtualenv .venv, you get a venv with recent pip. i generally start by this and then install everything in the venv instead of the global environment. This one command at the beginning doesn't have a noticeable impact on build but greatly improves debuggability and avoids conflict with python libraries installed by the os. but again, just use maturin build followed by pip install if you want the global environment.

[stinodego]

I just want to add one thing to this discussion:

maturin develop does an editable install of the Python package, while building the wheel first with maturin build and then installing the wheel will always result in a non-editable install.

This is relevant, as for local development an editable install is needed, as well as for some other edge cases like running pytest-cov in your CI pipeline.

Requiring a virtual environment is just needlessly restrictive. The responsibility should lie with the developer to manage their environments correctly. Now maturin develop enforces virtual environments, which needlessly restricts some workflows, as well as getting this wrong by not supporting pyenv-virtualenv and anything that's not a standard Python venv / Conda venv.

Unless there is a technical reason maturin develop must be run in one of these specific virtual environment types, this requirement should be dropped.

[konstin]

This is relevant, as for local development an editable install is needed, as well as for some other edge cases like running pytest-dev/pytest-cov#388 in your CI pipeline.

When would you do local development in your global environment? Wouldn't this mean that all of your projects share one environment, potentially overwriting each other, and also interfering with globally pip installed tools.

Now maturin develop enforces virtual environments, which needlessly restricts some workflows, as well as getting this wrong by not supporting pyenv-virtualenv and anything that's not a standard Python venv / Conda venv.

How does this impact pyenv-virtualenv, i'd expect a tool called pyenv-virtualenv to create virtualenvs? And what are those not standard venv workflows?

i do understand that with docker using the global environment is not a problem, but for deployment cases you should use separate build stages for build and deploy anyway, and also in containers there can be os installed python tools and debugging the global environment is a nightmare.