Switch to setuptools and find_packages

[Flamefire]

Any reasonably modern Python distribution includes setuptools. So the workaround, that was useful for Python 2.x, is no longer necessary and can be removed.
This allows using find_packages instead of maintaining an explicit list of packages which is error-prone.

I ran into this when adding the vendored tomli package

see easybuilders/easybuild-easyblocks#4006 & easybuilders/easybuild-easyconfigs#24747

[boegel]

I will need to check this thoroughly, since I don't want to me up a release due to this change...

[Flamefire]

I did a test locally: find_packages() and easybuild_packages were the same

[boegel]

Which Python/setuptools version was used here matters a lot...

I don't think we can generally assume that find_packages will behave the way we expect it to in all possible circumstances

[boegel]

I'm not happy with introducing a hard requirement on setuptools, since it's not part of the Python standard library, and often includes backwards-incompatible changes in new versions...

[Flamefire]

We already have a hard dependency on that for Python 3.12 where distutils doesn't exist anymore.

According to various sources setuptools should be available with any reasonably modern Python installation, i.e. Python 3.something. e.g. via ensurepip. And we don't have CI (for newer Pythons) without setuptools installed, so not sure if EB even works completely without it.

And as we don't use any advanced features I doubt that those setuptools changes affect us.

[Flamefire]

What exactly were the issues? I only found #2988 referencing https://lists.ugent.be/wws/arc/easybuild/2019-08/msg00057.html which doesn't open for me. Maybe >6yrs later those issues are gone/rare enough?

[boegel]

Copying @Flamefire's post from #5063 here to avoid splintering the discussion:

Without setuptools you'd need to list all "packages" in the setup call and it took me multiple tries to figure out which it actually requires.

What were those issues and are they still relevant? Our usage here might have changed enough that we are "standard" enough not to run into them (anymore). Especially if those issues were from the Python 2 days. Seems most advise using setuptools unconditionally (if not even pyproject.toml) for non-ancient Pythons

An alternative if you really want to play it safe:

* Import `setuptools.find_packages` if possible

* check output against our manual list and error on difference printing the correct list

* Have EASYBUILD_IGNORE_FINDPACKAGES=1 to skip that as a way out if it unexpectedly causes issues on some system.

But I'd really like to avoid error-prone adding (more) things to a hard-coded list if possible.

What exactly were the issues? I only found #2988 referencing https://lists.ugent.be/wws/arc/easybuild/2019-08/msg00057.html which doesn't open for me. Maybe >6yrs later those issues are gone/rare enough?

#2988 gives a hint, indeed. Unfortunately the EasyBuild mailing list isn't publicly available currently (requires UGent VPN), hopefully that can be restored soon.
Here are some links to public archives:

• https://www.mail-archive.com/[email protected]/msg04882.html
• https://www.mail-archive.com/[email protected]/msg05263.html

It's very hard to determine whether setuptools is now properly stable and won't cause a ton of confusing problems if we have a hard requirement on it for all Python versions we support (even if that's only in setup.py and nowhere else).

Generally speaking, setuptools has a reputation of happily making backwards-incompatible breaking changes, and expecting that people "just deal with it" by using a sufficiently recent version, or by adjusting their setup.py accordingly.
Somehow it didn't get an honorable mention in my "How To Make Package Managers Cry" talk from FOSDEM 2018, but it really should have. I'll get an opportunity to rectify that soon...
One example is pypa/setuptools#944, see also easybuilders/easybuild-easyblocks#1090.

I recall frequently being frustrated with setuptools several years ago for this type of reasons.
I don't recall recent instances, but that could be because EasyBuild hasn't had a hard requirement for setuptools for many years now (see also #2836).

In fact, more broadly speaking we have a standing rule that EasyBuild should not have a hard dependency on any Python package outside of the Python standard library. That's a rule that was imposed a while ago, after being fed up with dealing with a whole range of weird/annoying problems with various 3rd party dependencies.
I personally try to avoid hard 3rd party dependencies in the Python tools I develop...

So, introducing a hard requirement for setuptools.find_packages just so we don't need to maintain a (long) hardcoded list of easybuild.* packages anymore is not a good enough reason imho.
Depending on setuptools.find_packages may cause quite a bit of pain for years to come, without much benefit, especially since we don't really add additional easybuild.* packages frequently (#5063 most definitely is an exception here).
In fact, find_packages is even a bit of a foot-gun, see for example here.

@Flamefire Please rework #5063 so it doesn't require these changes.
It should boil down to adding easybuild.tools.tomli and maybe also easybuild.tools.tomli.tomllib to easybuild_packages in setup.py. Even if it somehow is more complex than that, then I still prefer not introducing a hard requirement for setuptools.find_packages.

[Flamefire]

I checked the first issue (https://www.mail-archive.com/[email protected]/msg04884.html) and it is not related to setuptools at all:

• EasyBuild requires vs-install
• vsc-install requires mock
• mock requires setuptools>=17.1
• Hence the failure without a recent enough setuptools

The other is:

[[ERROR]] Running 'easy_install --quiet --upgrade --prefix=/tmp/tmp0d0WpW/eb_stage1 vsc-install<0.11.4' failed: error: Could not find suitable distribution for Requirement.parse('vsc-install<0.11.4')

So also unrelated (and only mentions are distutils and easy_install) and even not relevant anymore given that vsc-* is no longer a dependency. In fact we don't have dependencies anymore.

In fact, find_packages is even a bit of a foot-gun, see for example here.

Can you point out what exactly is the foot-gun here? Where it is mentioned the most times is https://jwodder.github.io/kbits/posts/pypkg-mistakes/#top-level-tests-directory-in-wheel which we are currently(!) guilty! We install a test package which we should not (must not according to that) do ever!
The other mentions seem to rather be of the kind: Won't work at all. Which we can easily catch on CI.

It's very hard to determine whether setuptools is now properly stable

Given it is the tool that is recommended for a long time and our issues where with Python 2.x I'd say yes.
One comment to the above issue was "setuptools failed to isolate itself enough [...] which is fixed" (paraphrased) so it seems it is even independent of the OS version.

I tested the oldest/first setuptools (v17.0) that works with Python 3.6 and that worked. Same for 34.0.2 mentioned in the issue and the latest (59.6.0, for Python 3.6).

So I'd really switch from distutils right now for all Python versions, not just for 3.12+ as we currently do. Having to maintain 2 versions/utilities doesn't sound easier to me.
If you want to play it safe we could, until EB 6.0, provide a fallback to distutils via explicit opt-in, e.g. EASYBUILD_USE_DISTUTILS=1, and drop that in EB 6.0 as we would likely have discovered issues until then, and be using Python 3.9

[boegel]

Only requiring setuptools (in setup.py) with Python 3.12+ is a necessary evil, because distutils was removed from the Python standard library.
The approach taken in #4478 is very deliberate: only rely on setuptools when we have no other choice, and see how it pans out.

Not requiring setuptools (cfr. #2836) beyond setup.py was a very deliberate choice that I don't want to back-pedal on, but that's not really under discussion here (since this PR only affects setup.py).

We'll see how things work out with Python 3.12+, where we can't avoid setuptools.
I don't think enough people are already using that to have a goo view on whether or not there's trouble.
The people who do use Python 3.12+ already are perhaps a bit more tech-savvy.

The default Python in RHEL10 and derivatives (Rocky Linux 10, etc.) is Python 3.12, so we'll have a better view on things soon-ish.

For Python < 3.12, I'm strongly in favor of never requiring setuptools, because for those people who can't or don't want to use Python 3.12+, there will be surprises with setuptools at some point.

"Those Who Cannot Remember the Past Are Condemned To Repeat It"

[Flamefire]

We'll see how things work out with Python 3.12+, where we can't avoid setuptools.
I don't think enough people are already using that to have a goo view on whether or not there's trouble.

My take is that we should test using setuptools as long as most people are using a Python version where they have an alternative in the (now unlikely) case there is an issue with that.
I.e. use setuptools by default and provide distutils fallback by explicit opt-in.

This allows us to gather experience without (hard)breaking anyone.

For Python < 3.12, I'm strongly in favor of never requiring setuptools, because for those people who can't or don't want to use Python 3.12+, there will be surprises with setuptools at some point.

"Those Who Cannot Remember the Past Are Condemned To Repeat It"

As argued above: None of the previous issues is actually related to using setuptools for installation. The only issue related to setuptools was requiring setuptools as a runtime dependency in a specific version that wasn't available.
So I really can't see any potential for surprises, especially none that will only affect a subset of people, i.e. if there are issues with 3.12+ they are likely there in earlier versions, and vice-versa. The latter allows us to address those before people are forced to use 3.12+