sndsw: let alibuild figure out dependencies

[olantwin]

If we specify the dependencies manually, we prevent aliBuild from loading the correct dependencies (e.g. when adding a new one).

We also overwrite the module file in the incremental build recipe.

This PR fixes both of those issues by using alibuild's builtin tool for generating modules automatically and by only updating the SNDSW_HASH when doing an incremental rebuild of sndsw.

[ThomasRuf]

This looks like a huge change, but would of course simplify life enormously . Was it tested for all operating systems? Also with present CVMFS release? I mean running the simulation/digitization/reconstruction? Or would it require to build a new stack? I remember large problems of having stuff in the ROOT include path, python path didn't worked on some OS. Most of the settings of env variables are added because they were not present otherwise.

[olantwin]

Thank you for your comments!

Was it tested for all operating systems?

Not yet, I opened this PR to allow for wider testing.

Also with present CVMFS release?

Again, not yet. The one complication is that I'm not completely certain whether the module files are all correctly found when we force using "system" packages. But this would also apply to manually specifying all the dependencies.

I mean running the simulation/digitization/reconstruction?

For testing, it should be sufficient to compare the environments before and after the change, and see which modules are loaded in both cases.

As least on my machine, it made no difference (apart from also loading dependencies only specified in the yaml section of the recipe, which were ignored otherwise).

Or would it require to build a new stack?

As most users use $SNDDIST from CVMFS, the stack does not need to be rebuilt. It should however work with the current stack.

I remember large problems of having stuff in the ROOT include path, python path didn't worked on some OS.

Most of the settings of env variables are added because they were not present otherwise.

The ENV variables are not touched, with the exception of SNDSW_HASH for incremental builds. This modifies only which modules are loaded as dependencies. alibuild-generate-module might also add SNDSW libraries and binaries to the corresponding paths a second time, but this doesn't matter (and can be cleaned up in a second step).

[olantwin]

Thank you for your comments!

Was it tested for all operating systems?

Not yet, I opened this PR to allow for wider testing.

Also with present CVMFS release?

Again, not yet. The one complication is that I'm not completely certain whether the module files are all correctly found when we force using "system" packages. But this would also apply to manually specifying all the dependencies.

I mean running the simulation/digitization/reconstruction?

For testing, it should be sufficient to compare the environments before and after the change, and see which modules are loaded in both cases.

As least on my machine, it made no difference (apart from also loading dependencies only specified in the yaml section of the recipe, which were ignored otherwise).

Or would it require to build a new stack?

As most users use $SNDDIST from CVMFS, the stack does not need to be rebuilt. It should however work with the current stack.

Most of the settings of env variables are added because they were not present otherwise.

The ENV variables are not touched, with the exception of SNDSW_HASH for incremental builds. This modifies only which modules are loaded as dependencies. alibuild-generate-module might also add SNDSW libraries and binaries to the corresponding paths a second time, but this doesn't matter (and can be cleaned up in a second step).

I remember large problems of having stuff in the ROOT include path, python path didn't worked on some OS.

We should investigate these cases further, as the modules set the environment variables in a deterministic way (and cases per architecture can be added), but those should be separate issues.

[olantwin]

Just tested on lxplus, where a lot of scary output is produced, but the resulting env is the same as before.

The created sndsw module is:

#%Module1.0
proc ModulesHelp { } {
  global version
  puts stderr "ALICE Modulefile for sndsw master-local2"
}
set version master-local2
module-whatis "ALICE Modulefile for sndsw master-local2"
# Dependencies
if ![ is-loaded 'BASE/1.0' ] {
 module load BASE/1.0
}
if ![ is-loaded 'pythia6/-local1' ] { module load pythia6/-local1 }
if ![ is-loaded 'PHOTOSPP/-local1' ] { module load PHOTOSPP/-local1 }
if ![ is-loaded 'FreeType/-local1' ] { module load FreeType/-local1 }
if ![ is-loaded 'ofi/-local1' ] { module load ofi/-local1 }
if ![ is-loaded 'GCC-Toolchain/-local1' ] { module load GCC-Toolchain/-local1 }
if ![ is-loaded 'fmt/-local1' ] { module load fmt/-local1 }
if ![ is-loaded 'sndsw/-local1' ] { module load sndsw/-local1 }
if ![ is-loaded 'alpaca/-local1' ] { module load alpaca/-local1 }
if ![ is-loaded 'FEDRA/-local1' ] { module load FEDRA/-local1 }
if ![ is-loaded 'lhapdf/-local1' ] { module load lhapdf/-local1 }
if ![ is-loaded 'protobuf/-local1' ] { module load protobuf/-local1 }
if ![ is-loaded 'FairCMakeModules/-local1' ] { module load FairCMakeModules/-local1 }
if ![ is-loaded 'CMake/-local1' ] { module load CMake/-local1 }
if ![ is-loaded 'zlib/-local1' ] { module load zlib/-local1 }
if ![ is-loaded 'Python-modules/-local1' ] { module load Python-modules/-local1 }
if ![ is-loaded 'libpng/-local1' ] { module load libpng/-local1 }
if ![ is-loaded 'Python/-local1' ] { module load Python/-local1 }
if ![ is-loaded 'VMC/-local1' ] { module load VMC/-local1 }
if ![ is-loaded 'log4cpp/-local1' ] { module load log4cpp/-local1 }
if ![ is-loaded 'vgm/-local1' ] { module load vgm/-local1 }
if ![ is-loaded 'sqlite/-local1' ] { module load sqlite/-local1 }
if ![ is-loaded 'lzma/-local1' ] { module load lzma/-local1 }
if ![ is-loaded 'Tauolapp/-local1' ] { module load Tauolapp/-local1 }
if ![ is-loaded 'apfel/-local1' ] { module load apfel/-local1 }
if ![ is-loaded 'EvtGen/-local1' ] { module load EvtGen/-local1 }
if ![ is-loaded 'OpenSSL/-local1' ] { module load OpenSSL/-local1 }
if ![ is-loaded 'GEANT3/-local1' ] { module load GEANT3/-local1 }
if ![ is-loaded 'ROOT/-local1' ] { module load ROOT/-local1 }
if ![ is-loaded 'pythia/-local1' ] { module load pythia/-local1 }
if ![ is-loaded 'GSL/-local1' ] { module load GSL/-local1 }
if ![ is-loaded 'boost/-local1' ] { module load boost/-local1 }
if ![ is-loaded 'GENIE/-local1' ] { module load GENIE/-local1 }
if ![ is-loaded 'asiofi/-local1' ] { module load asiofi/-local1 }
if ![ is-loaded 'GEANT4/-local1' ] { module load GEANT4/-local1 }
if ![ is-loaded 'FairMQ/-local1' ] { module load FairMQ/-local1 }
if ![ is-loaded 'HepMC/-local1' ] { module load HepMC/-local1 }
if ![ is-loaded 'GEANT4_VMC/-local1' ] { module load GEANT4_VMC/-local1 }
if ![ is-loaded 'flatbuffers/-local1' ] { module load flatbuffers/-local1 }
if ![ is-loaded 'FairRoot/-local1' ] { module load FairRoot/-local1 }
if ![ is-loaded 'XRootD/-local1' ] { module load XRootD/-local1 }
if ![ is-loaded 'asio/-local1' ] { module load asio/-local1 }
if ![ is-loaded 'XercesC/-local1' ] { module load XercesC/-local1 }
if ![ is-loaded 'libffi/-local1' ] { module load libffi/-local1 }
if ![ is-loaded 'libxml2/-local1' ] { module load libxml2/-local1 }
if ![ is-loaded 'FairLogger/-local1' ] { module load FairLogger/-local1 }
if ![ is-loaded 'ZeroMQ/-local1' ] { module load ZeroMQ/-local1 }

set PKG_ROOT $::env(BASEDIR)/sndsw/$version
prepend-path PATH $PKG_ROOT/bin
prepend-path LD_LIBRARY_PATH $PKG_ROOT/lib

# Our environment
setenv EOSSHIP root://eospublic.cern.ch/
setenv SNDSW_ROOT $::env(BASEDIR)/sndsw/$version
setenv FAIRSHIP $::env(SNDSW_ROOT)
setenv FAIRSHIP_ROOT $::env(SNDSW_ROOT)
setenv SNDSW_HASH 3b9063ed41c39f8aa114dacf5b0117c7a0f2e5e8
setenv FAIRSHIP_HASH $::env(SNDSW_HASH)
setenv VMCWORKDIR $::env(SNDSW_ROOT)
setenv GEOMPATH $::env(SNDSW_ROOT)/geometry
setenv CONFIG_DIR $::env(SNDSW_ROOT)/gconfig
setenv GALGCONF $::env(SNDSW_ROOT)/shipgen/genie_config
prepend-path PATH $::env(SNDSW_ROOT)/bin
prepend-path LD_LIBRARY_PATH $::env(SNDSW_ROOT)/lib
setenv FAIRLIBDIR $::env(SNDSW_ROOT)/lib
prepend-path ROOT_INCLUDE_PATH $::env(SNDSW_ROOT)/include
append-path ROOT_INCLUDE_PATH $::env(GEANT4_ROOT)/include
append-path ROOT_INCLUDE_PATH $::env(GEANT4_ROOT)/include/Geant4
append-path ROOT_INCLUDE_PATH $::env(PYTHIA_ROOT)/include
append-path ROOT_INCLUDE_PATH $::env(PYTHIA_ROOT)/include/Pythia8
append-path ROOT_INCLUDE_PATH $::env(GEANT4_VMC_ROOT)/include
append-path ROOT_INCLUDE_PATH $::env(GEANT4_VMC_ROOT)/include/geant4vmc
prepend-path PYTHONPATH $::env(SNDSW_ROOT)/python
append-path PYTHONPATH $::env(SNDSW_ROOT)/shipLHC/scripts
append-path PYTHONPATH $::env(SNDSW_ROOT)/shipLHC/rawData
append-path PYTHONPATH $::env(XROOTD_ROOT)/lib/python/site-packages
# required for ubuntu22.04: don't know how to fix this more elegant
append-path PYTHONPATH  $::env(XROOTD_ROOT)/local/lib/python3.10/dist-packages

Note, how the only difference is that now the module loading fails loudly instead of quietly (but errors are still ignored), and dependencies are resolved to the leaves instead of only one level down.

[ThomasRuf]

Simona and I are making a new release for lxplus9. I would like to postpone this PR until this is finished.

[olantwin]

I've updated this as I had made a mistake in the incremental recipe.

The recipe now follows the ROOT procedure for generating and updating the modulefile for incremental builds.