Ubuntu 22.04, Python 3.10, and broken virtual environment layouts

[colindean]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu Jammy

• Poetry version: 1.2.0

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/colindean/27d36dcc7cf78f722f374ebd5bfd970f (also contains log output and a brief description of env setup steps)

Issue

root@80d0e404badc:/what# poetry self add poetry-plugin-bundle
Using version ^1.0.0 for poetry-plugin-bundle

Updating dependencies
Resolving dependencies... (3.8s)

[Errno 2] No such file or directory: 'python'

I believe the problem is in here:

poetry/src/poetry/utils/env.py

Lines 1231 to 1248 in f1af3f8

	def _find_python_executable(self) -> None:
	bin_dir = self._bin_dir
	if self._is_windows and self._is_conda:
	bin_dir = self._path
	python_executables = sorted(
	p.name
	for p in bin_dir.glob("python*")
	if re.match(r"python(?:\d+(?:\.\d+)?)?(?:\.exe)?$", p.name)
	)
	if python_executables:
	executable = python_executables[0]
	if executable.endswith(".exe"):
	executable = executable[:-4]
	self._executable = executable

Ubuntu doesn't provide a python command anymore, just python3:

root@80d0e404badc:/what# python --version
bash: python: command not found
root@80d0e404badc:/what# python3 --version
Python 3.10.4

Seeing as though all supported versions of Python on which Poetry runs should also provide a python3 executable, perhaps the fallback here should be python3?

poetry/src/poetry/utils/env.py

Line 1178 in f1af3f8

self._executable = "python"

[colindean]

N.b. I hadn't even run poetry install yet and just now went I did…

root@80d0e404badc:/what# poetry install
Creating virtualenv what--AkdOZe_-py3.10 in /root/.cache/pypoetry/virtualenvs

[Errno 2] No such file or directory: 'python'

[colindean]

I appear to be unable to tell poetry which Python to use, either:

root@80d0e404badc:/what# poetry env use $(which python3)

[Errno 2] No such file or directory: 'python'

[neersighted]

I think this is actually the old DEB_PYTHON_INSTALL_LAYOUT=deb issue -- can you set that in the environment and reinstall Poetry/poetry install from scratch? python will always exist as far as Poetry is concerned as it is called for by the virtual environment spec.

[colindean]

I used that envvar at Poetry installation time… see repro here. Does it need to be set all of the time, as in, added to the user's shellrc?

I can confirm that changing

poetry/src/poetry/utils/env.py

Line 1178 in f1af3f8

self._executable = "python"

to python3 restores expected functionality for poetry install and poetry env use … but not for poetry self add ….

[colindean]

root@80d0e404badc:/what# export DEB_PYTHON_INSTALL_LAYOUT=deb
root@80d0e404badc:/what# poetry self add poetry-plugin-bundle
Using version ^1.0.0 for poetry-plugin-bundle

Updating dependencies
Resolving dependencies... (16.5s)

[Errno 2] No such file or directory: 'python'

[neersighted]

We're still trying to fully understand how we can work around the regression in Debian/Ubuntu as they don't appear willing to walk back the changes. I am not sure why you're having that issue with self add, but I can confirm for you that on the ubuntu:22.04 container with the python3 package I can install Poetry, run the Poetry test suite (after poetry install), and poetry self add without issue if I set DEB_PYTHON_INSTALL_LAYOUT=deb before running any Python.

It is important to note that the issue is not in Poetry, but is in the virutualenv package which we consume. It is still considered unsolved there, as the changes made by Debian/Ubuntu result in an invalid virtual environment according to the spec: pypa/virtualenv#2350

[colindean]

OK. To confirm, does DEB_PYTHON_INSTALL_LAYOUT=deb need to stay forever, then? As in, rather that just needing for it to be set at installation time, it should be set the user's shell rc or profile?

[neersighted]

From a practical standpoint, it's the known workaround right now. Fundamentally this is unsolved and upstream seems intransigent or indifferent (see: https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1967920).

The deadsnakes PPA solves this with a patch on top of the Debian 3.10 patches. Ideally this would get solved in virtualenv, but things are stalled there too. We haven't put too much effort into figuring out how to work around it in Poetry (but the solution definitely is NOT just adapting to local/bin as that is a violation of the virtual env spec and other tools will break), if that is even possible without somehow trying to monkey-patch the virtualenv module.

The best solution is to use a Python.org Python (aka a build from source installed into /opt/ or using python-build, or pyenv or a python docker hub image, or the like), or to use a deadsnakes build, until some upstream solves this, or we figure out a not-awful way to work around it here in the downstream.