Merge pull request #353 from easybuilders/develop

Sync `devlop` to `main`

Author: branfosj

docs/contributing.md

@@ -266,6 +266,94 @@ git checkout develop
 git pull upstream develop
 ```
 
+##### Running your development instance {: #running_your_development_instance }
+
+There are multiple ways to deploy a local development copy of EasyBuild. This
+section presents a method that is more relevant when modifying the framework and
+easyblock repositories.
+
+To deploy a local copy of EasyBuild clone the
+[easybuild-framework](https://github.com/easybuilders/easybuild-framework),
+[easybuild-easyblocks](https://github.com/easybuilders/easybuild-easyblocks), and
+[easybuild-easyconfigs](https://github.com/easybuilders/easybuild-easyconfigs)
+repositories in the same directory, for instance a directory name `EasyBuild`.
+After cloning these repositories the content of the `EasyBuild` directory
+should be the following.
+
+``` console
+$ tree -L 1 EasyBuild
+EasyBuild
+├── easybuild-easyblocks
+├── easybuild-easyconfigs
+└── easybuild-framework
+
+3 directories, 0 files
+```
+
+An environment variable with the path to the `EasyBuild` directory simplifies
+calls to the development instance of EasyBuild significantly; define this
+variable in the following manner.
+
+``` shell
+export EB_DEVEL_ROOT=/path/to/EasyBuild
+```
+
+To run the development instance, add the paths of the framework and easyblock
+repositories to the Python path, and the path of the easyconfig repository to
+the robot search path.
+
+For instance, a command to install `ReFrame-4.3.3.eb` and its dependencies with
+the necessary variables defined for the command environment would be the
+following:
+
+``` shell
+PYTHONPATH=${EB_DEVEL_ROOT}/easybuild-framework:${EB_DEVEL_ROOT}/easybuild-easyblocks:${PYTHONPATH} EASYBUILD_ROBOT_PATHS=${EB_DEVEL_ROOT}/easybuild-easyconfigs/easybuild/easyconfigs ${EB_DEVEL_ROOT}/easybuild-framework/eb --robot ReFrame-4.3.3.eb
+```
+
+To avoid adding the variables in the environment of every single command while
+maintaining a clean shell environment, an alias can be defined as follows.
+
+``` shell
+alias eb_devel='PYTHONPATH="${EB_DEVEL_ROOT}/easybuild-framework:${EB_DEVEL_ROOT}/easybuild-easyblocks:${PYTHONPATH}" EASYBUILD_ROBOT_PATHS="${EB_DEVEL_ROOT}/easybuild-easyconfigs/easybuild/easyconfigs" ${EB_DEVEL_ROOT}/easybuild-framework/eb'
+```
+
+With the alias defined, the command to install `ReFrame-4.3.3.eb` and its
+dependencies now becomes the following:
+
+``` shell
+eb_devel --robot ReFrame-4.3.3.eb
+```
+
+??? note "Controlling the robot search path in development instances"
+
+    There is no default value for the [robot search
+    path](/using-easybuild/#controlling_robot_search_path) in the development
+    instance. The robot search path is set explicitly using the
+    `EASYBUILD_ROBOT_PATHS` environment variable. As a result, expect that some
+    features of the `EASYBUILD_ROBOT_PATHS` and its accompanying command line
+    option, `--robot-paths` will not work with `eb_devel`, the alias for the
+    EasyBuild development version.
+
+    For instance,
+
+    - you cannot use `EASYBUILD_ROBOT_PATHS` to set the robot path as it is
+      overridden in the alias `eb_devel`, and
+    - you cannot append to the default robot search path, with `--robot-paths`,
+      as it overrides the `EASYBUILD_ROBOT_PATHS` environment option.
+
+    As a workaround,
+
+    - use `--robot-paths` to override the robot search path set via the alias, and
+    - use the environment variable `EASYBUILD_ROBOT_PATHS` to access the
+      the robot search path set by the alias so that you can append to it.
+
+    For instance to append to the development robot search path, use the
+    following command.
+
+    ``` shell
+    eb_devel --robot-paths=${EASYBUILD_ROBOT_PATHS}:/path/to/dir_0:/path/to/dir_1 --robot ReFrame-4.3.3.eb
+    ```
+
 ### Opening a new pull request {: #contributing_creating_pull_requests }
 
 !!! note

docs/partial-installations.md

@@ -10,9 +10,9 @@ naming scheme).
 To stop the installation procedure *after* a specific step in the installation procedure, the `-s`/`--stop`
 command line option can be used; the name of the step must be supplied as an argument.
 
-The following step names are recognized (listed in execution order): `fetch`, `ready`, `source`, `patch`,
-`prepare`, `configure`, `build`, `test`, `install`, `extensions`, `package`, `postproc`,
-`sanitycheck`, `cleanup`, `module`, `testcases`.
+The following step names are recognized (listed in execution order): `fetch`, `ready`, `extract`, `patch`,
+`prepare`, `configure`, `build`, `test`, `install`, `extensions`, `postiter`, `postproc`,
+`sanitycheck`, `cleanup`, `module`, `permissions`, `package`, `testcases`.
 
 Example usage:
 

docs/python-search-path.md

@@ -0,0 +1,57 @@
+# Customizing Python search path
+
+In order to locate Python packages in modules, before v5.0.0, EasyBuild conventionally used the `PYTHONPATH` environment variable. However, this has several issues:
+
+1. `PYTHONPATH` has highest priority; it prevents users from making a custom virtual environment on top of modules and shadowing packages.
+2. Packages are picked up even for incompatible python versions, e.g. an OS installed old python 3.6 will break if packages from 3.12 exists in `PYTHONPATH`.
+3. Modules can't have optional dependencies on different Python versions, as PYTHONPATH points directly to the `site-packages` subdirectory.
+4. Packages with `pth` files can't work with PYTHONPATH and requires being added to the site dir.
+
+Unfortunately, Python offers no environment variables to do the correct thing here.
+To solve this (initially for Python `multi_deps`) EasyBuild has for a long time supported the use of the custom `EBPYTHONPREFIXES` via a `sitecustomize.py` script for the Python modules we build. 
+It is included in standard Python installations made with EasyBuild. You can opt out of using `sitecustomize.py` if you have strong technical reasons to avoid it.
+
+The `sitecustomize.py` script does the correct thing by only considering the correct Python version, and puts Python packages provided by the environment module at the lowest priority, allowing a user venv to override them.
+
+## Using `--prefer-python-search-path` 
+
+Since v5.0.0 the new global configuration option `--prefer-python-search-path` can be used for EasyBuild to prefer the use of either `PYTHONPATH` or `EBPYTHONPREFIXES`.
+For backwards compatibility with existing modules, EasyBuild was unfortunately required to keep `PYTHONPATH` as the default.
+Note that the option is just the preferred option, if the package path doesn't follow the standard `lib/pythonY.X/site-packages` format then `PYTHONPATH` must be used. If multi-deps is used, then `EBPYTHONPREFIXES` is required.
+
+If you wish to switch to `EBPYTHONPREFIXES`, you should also traverse the existing Python bundles you have installed and make sure to rebuild the modules.
+Failure to do so might leave you with a few easyconfigs that require shadowing of older package versions not working correctly due to the import priority changing.
+
+If you are building a new software stack from scratch, you can safely switch this option to `EBPYTHONPREFIXES` and enjoy the benefits.
+
+## Fixing existing modules
+
+You can convert existing modules by rebuilding them with `eb --module-only --rebuild ...`.
+You can find the modules that use `PYTHONPATH` by e.g. grepping through your modules:
+
+```bash
+grep --include '*.lua' -Rl 'PYTHONPATH.*site-packages' /path/to/modules/all
+```
+
+If you use the default module naming scheme
+
+```bash
+cd $MODULEPATH
+grep -Rl PYTHONPATH */*.lua | grep -v EasyBuild | sed -e 's+/+-+' | sed -e 's+lua$+eb+' | xargs eb --rebuild --module-only
+```
+
+If done correctly, your old module files will have gone from
+
+```lua
+prepend_path("PYTHONPATH", pathJoin(root, "lib", "python3.12", "site-packages"))
+# or
+prepend_path("PYTHONPATH", pathJoin(root, "lib/python3.12/site-packages"))
+```
+
+to
+
+```lua
+prepend_path("EBPYTHONPREFIXES", root)
+```
+
+Remember to back up all your module files first before rebuilding.

mkdocs.yml

@@ -73,6 +73,7 @@ nav:
       - Cray support: cray-support.md
       - Customizing EasyBuild via hooks: hooks.md
       - Including Python modules: including-additional-python-modules.md
+      - Customizing Python search path: python-search-path.md
       - Packaging support: packaging-support.md
       - RPATH support: rpath-support.md
       - Using external modules: using-external-modules.md