Consider pixi for dependency management

[ashiklom]

Pixi is a great cross-platform dependency manager, from the author of mamba (which in turn is an improved version of conda). Pixi pulls packages from conda-forge, so it has a pretty solid collection of R packages and scientific libraries (NetCDF, HDF5, GDAL, etc.).

A major advantage of pixi is that, like conda/mamba, it tracks both R packages and their system dependencies. So, e.g., pixi add r-ncdf4 will automatically pull in NetCDF4 and HDF5 dependencies; similarly, r-sf will implicitly pull GDAL, Proj, etc.

Similarly to renv, pixi manages dependencies with two files: pixi.toml (which provides project metadata and defines overall package constraints (e.g., R = ">=4.0"; r-roxygen2 = "==7.3.2") and a machine-generated pixi.lock file that contains a complete list of the "solved" environment (including all mutual version constraints) and its precise version and origin. The general workflow is that you "re-solve" the lockfile periodically whenever you want to update packages to the latest combination of mutually supported versions, but for day-to-day usage, you pull packages from the lockfile and are guaranteed to have an identical installation. Typically, both pixi.toml and pixi.lock are added to version control.

Like renv, pixi accomplishes this by maintaining a complete copy of your project library in the .pixi folder in the project root. Note that this includes a full installation of R itself and its dependencies (gfortran, etc.), which gives you a very strong reproducibility guarantee.

After a bit of iteration, here's a pixi.toml file that should work for PEcAn on x86-64 Linux systems (I've left almost all versions constrained, but we can add those constraints if needed). The dependencies here are pulled from the PEcAn package DESCRIPTION files (minus packages that are not available through conda-forge --- we still have to install those by hand).

We can add additional platform support by modifying the platform field, but then we lose some conda-forge packages that aren't supported on those systems (e.g., duckdb, a dependency of neonstore, and taxize, a dependency of traits, don't t have working osx-arm64 (M-series Mac) builds; jags doesn't have a Windows build). That may be worthwhile to make it easier to do PEcAn development locally, at least on MacOS; something to consider.

Note also that pixi has a convenient "task" system for defining specific commands to run in the environment. For example, with the config below, a complete PEcAn install is just:

pixi run install_pecan

That will first make sure the entire environment is installed, then run the install_gh task to grab amerifluxr from GitHub, and then install PEcAn and remaining (non-conda) package dependencies.

Combining pixi + pak (for R package installation; pak is clever and aggressive about parallel build and install, which helps it go fast), I was able to get a complete PEcAn.all + PEcAn.SIPNET installation in around 8 minutes on my local machine (12 cores; ~6 GB RAM; WSL2 in Windows 11). The resulting .pixi folder is ~2.7 GB.

It may be possible to combine pixi with renv for complete reproducibility (by using renv to track non-conda-forge packages), but I haven't tried.

Details

[workspace]
authors = [
  "The PEcAn team"
]
channels = ["conda-forge"]
name = "workflows"
# NOTE: win-64 would also be nice, but `r-rjags` is not supported on it
platforms = ["linux-64"]
version = "0.1.0"

[tasks]
[tasks.install_gh]
args = [{ arg = "pkg" }, { arg = "repo" }]
cmd = "Rscript -e 'if (!requireNamespace(\"{{ pkg }}\")) {pak::pak(\"{{ repo }}\")}'"

[tasks.install_pecan]
depends-on = [{ task = "install_gh", args = [ { pkg = "amerifluxr" }, { repo = "chuhousen/amerifluxr" } ] }]
cmd = "Rscript -e 'pak::repo_add(\"pecanproject.r-universe.dev\")' -e 'pak::pak(c(\"PEcAn.all\", \"PEcAn.SIPNET\"))'"

[dependencies]
r = ">=4"
r-roxygen2 = "==7.3.2"
r-pak = ">=0.9.2,<0.10"
"r-abind" = "*"
"r-admisc" = "*"
"r-alabama" = "*"
"r-ape" = "*"
"r-arrow" = "*"
"r-askpass" = "*"
"r-assertthat" = "*"
"r-backports" = "*"
"r-base64enc" = "*"
"r-bit" = "*"
"r-bit64" = "*"
"r-blob" = "*"
"r-brew" = "*"
"r-bridgesampling" = "*"
"r-Brobdingnag" = "*"
"r-bslib" = "*"
"r-cachem" = "*"
"r-callr" = "*"
"r-classInt" = "*"
"r-cli" = "*"
"r-clipr" = "*"
"r-clue" = "*"
"r-coda" = "*"
"r-commonmark" = "*"
"r-config" = "*"
"r-corrplot" = "*"
"r-cpp11" = "*"
"r-crayon" = "*"
"r-crosstalk" = "*"
"r-curl" = "*"
"r-data.table" = "*"
"r-DBI" = "*"
"r-dbplyr" = "*"
"r-DEoptimR" = "*"
"r-desc" = "*"
"r-devtools" = "*"
"r-DHARMa" = "*"
"r-digest" = "*"
"r-doParallel" = "*"
"r-doSNOW" = "*"
"r-dotty" = "*"
"r-dplR" = "*"
"r-dplyr" = "*"
"r-dtw" = "*"
"r-dtwclust" = "*"
"r-duckdb" = "*"
"r-e1071" = "*"
"r-ellipse" = "*"
"r-emulator" = "*"
"r-evaluate" = "*"
"r-exactextractr" = "*"
"r-expm" = "*"
"r-farver" = "*"
"r-fastmap" = "*"
"r-filelock" = "*"
"r-flexclust" = "*"
"r-fontawesome" = "*"
"r-foreach" = "*"
"r-fs" = "*"
"r-furrr" = "*"
"r-future" = "*"
"r-gap" = "*"
"r-gap.datasets" = "*"
"r-generics" = "*"
"r-geometry" = "*"
"r-getPass" = "*"
"r-ggmcmc" = "*"
"r-ggplot2" = "*"
"r-ggpubr" = "*"
"r-ggrepel" = "*"
"r-glue" = "*"
"r-gmm" = "*"
"r-gridExtra" = "*"
"r-gtable" = "*"
"r-hdf5r" = "*"
"r-here" = "*"
"r-highr" = "*"
"r-hms" = "*"
"r-htmltools" = "*"
"r-htmlwidgets" = "*"
"r-httpuv" = "*"
"r-httr" = "*"
"r-IDPmisc" = "*"
"r-isoband" = "*"
"r-iterators" = "*"
"r-itertools" = "*"
"r-jquerylib" = "*"
"r-jsonlite" = "*"
"r-jsonvalidate" = "*"
"r-keyring" = "*"
"r-knitr" = "*"
"r-labeling" = "*"
"r-later" = "*"
"r-lazyeval" = "*"
"r-lifecycle" = "*"
"r-linprog" = "*"
"r-lme4" = "*"
"r-lmtest" = "*"
"r-lpSolve" = "*"
"r-lubridate" = "*"
"r-magic" = "*"
"r-magrittr" = "*"
"r-markdown" = "*"
"r-mclust" = "*"
"r-MCMCpack" = "*"
"r-memoise" = "*"
"r-mgcv" = "*"
"r-mime" = "*"
"r-minpack.lm" = "*"
"r-minqa" = "*"
"r-mockery" = "*"
"r-modeltools" = "*"
"r-msm" = "*"
"r-mvtnorm" = "*"
"r-ncdf4" = "*"
"r-neonUtilities" = "*"
"r-nimble" = "*"
"r-nleqslv" = "*"
"r-nloptr" = "*"
"r-numbers" = "*"
"r-numDeriv" = "*"
"r-openssl" = "*"
"r-optparse" = "*"
"r-otel" = "*"
"r-parsedate" = "*"
"r-pbapply" = "*"
"r-pillar" = "*"
"r-pkgbuild" = "*"
"r-pkgconfig" = "*"
"r-pkgload" = "*"
"r-plotly" = "*"
"r-plotrix" = "*"
"r-plyr" = "*"
"r-png" = "*"
"r-polycor" = "*"
"r-prettyunits" = "*"
"r-processx" = "*"
"r-prodlim" = "*"
"r-progress" = "*"
"r-promises" = "*"
"r-proxy" = "*"
"r-ps" = "*"
"r-purrr" = "*"
"r-pwr" = "*"
"r-qgam" = "*"
"r-qrng" = "*"
"r-R.methodsS3" = "*"
"r-R.oo" = "*"
"r-R.utils" = "*"
"r-R6" = "*"
"r-randomForest" = "*"
"r-randtoolbox" = "*"
"r-rappdirs" = "*"
"r-raster" = "*"
"r-rbibutils" = "*"
"r-RColorBrewer" = "*"
"r-Rcpp" = "*"
"r-RcppArmadillo" = "*"
"r-RcppEigen" = "*"
"r-RcppParallel" = "*"
"r-RcppProgress" = "*"
"r-RcppThread" = "*"
"r-RcppTOML" = "*"
"r-rcrossref" = "*"
"r-RCurl" = "*"
"r-Rdpack" = "*"
"r-readr" = "*"
"r-reformulas" = "*"
"r-reshape" = "*"
"r-reshape2" = "*"
"r-reticulate" = "*"
"r-rgl" = "*"
"r-rjags" = "*"
"r-rjson" = "*"
"r-rlang" = "*"
"r-rlist" = "*"
"r-rmarkdown" = "*"
"r-robustbase" = "*"
"r-RPostgres" = "*"
"r-rprojroot" = "*"
"r-RSpectra" = "*"
"r-RSQLite" = "*"
"r-rstudioapi" = "*"
"r-s2" = "*"
"r-S7" = "*"
"r-sandwich" = "*"
"r-sass" = "*"
"r-scales" = "*"
"r-sessioninfo" = "*"
"r-sf" = "*"
"r-shiny" = "*"
"r-shinyjs" = "*"
"r-sourcetools" = "*"
"r-sp" = "*"
"r-spacefillr" = "*"
"r-storr" = "*"
"r-stringi" = "*"
"r-stringr" = "*"
"r-sys" = "*"
"r-tensorflow" = "*"
"r-terra" = "*"
"r-testthat" = "*"
"r-tfautograph" = "*"
"r-tfruns" = "*"
"r-tibble" = "*"
"r-tictoc" = "*"
"r-tidyr" = "*"
"r-tidyselect" = "*"
"r-tidyverse" = "*"
"r-timechange" = "*"
"r-tinytex" = "*"
"r-tmvtnorm" = "*"
"r-traits" = "*"
"r-truncnorm" = "*"
"r-tzdb" = "*"
"r-units" = "*"
"r-urltools" = "*"
"r-utf8" = "*"
"r-uuid" = "*"
"r-vctrs" = "*"
"r-vdiffr" = "*"
"r-viridisLite" = "*"
"r-vroom" = "*"
"r-whisker" = "*"
"r-withr" = "*"
"r-wk" = "*"
"r-xfun" = "*"
"r-xgboost" = "*"
"r-XML" = "*"
"r-xml2" = "*"
"r-xslt" = "*"
"r-xtable" = "*"
"r-xts" = "*"
"r-yaml" = "*"
"r-zeallot" = "*"
"r-zip" = "*"
"r-zoo" = "*"

Here's a quick R script to build this pixi.toml file from the PEcAn package DESCRIPTIONs.

Details

#!/usr/bin/env Rscript

use("desc")
use("tidyverse")
use("pak")

base <- list.dirs("base", recursive = FALSE)
modules <- list.dirs("modules", recursive = FALSE)
models <- list.dirs("models", recursive = FALSE)

pkgs <- c(base, modules, models)
names(pkgs) <- pkgs

alldeps <- pkgs |>
  map(desc::desc_get_deps) |>
  bind_rows(.id = "source")

external_deps <- alldeps |>
  filter(!startsWith(package, "PEcAn."))

# Packages included with R
base_r <- c(
  "R",
  "boot",
  "class",
  "cluster",
  "codetools",
  "lattice",
  "grDevices",
  "graphics",
  "grid",
  "KernSmooth",
  "MASS",
  "Matrix",
  "methods",
  "nlme",
  "parallel",
  "stats",
  "survival",
  "tools",
  "utils"
)

# Packages not available on conda-forge
no_conda <- c(
  # Primary dependencies
  "BayesianTools",
  "BioCro",
  "MODISTools",
  "Maeswrap",
  "REddyProc",
  "RPostgreSQL",
  "Rpreles",
  "SimilarityMeasures",
  "SticsRFiles",
  "TruncatedNormal",
  "amerifluxr",
  "dataone",
  "datapack",
  "ecmwfr",
  "geonames",
  "keras3",
  "linkages",
  "lqmm",
  "mlegp",
  "mvbutils",
  "neonstore",
  "nneo",
  "redland",
  "sensitivity",
  "sirt",
  "suntools",
  # nested dependencies:
  "CDM",
  "SparseGrid",
  "TAM",
  "bigleaf",
  "duckdbfs",
  "pbv",
  "solartime",
  "thor"
  # Not available on macos-arm64, but work on linux-64
  # "dtw",
  # "dtwclust",
  # "duckdb"
  # "nimble",
  # "traits",   # Older versions are available
)

# Get the dependencies of the non-conda packages
get_deps <- function(pkg) {
  return(tryCatch(
    pak::pkg_deps(pkg),
    error = function(e) {
      message("Error for ", pkg, " : ", e)
      return(NULL)
    }
  ))
}

names(no_conda) <- no_conda
no_conda_deps <- no_conda |>
  map(get_deps) |>
  bind_rows(.id = "package")

no_conda_uniq <- no_conda_deps |>
  dplyr::filter(
    !(ref %in% no_conda),
    !(ref %in% base_r),
    !(startsWith(ref, "installed::")),
    ref != "roxygen2"
  ) |>
  distinct(ref) |>
  arrange(ref) |>
  pull()

uniq_deps <- external_deps |>
  filter(
    !(package %in% base_r),
    !(package %in% no_conda)
  ) |>
  distinct(package) |>
  pull()

# Additional packages
extras <- c(
  "RCurl"  # Needed by amerifluxr
)

all_uniq_deps <- sort(unique(c(uniq_deps, no_conda_uniq, extras)))
pixi_string <- sprintf("\"r-%s\" = \"*\"", all_uniq_deps)

pixi_lines <- c(
  "[workspace]",
  'authors = [ "Chris Black" ]',
  'channels = ["conda-forge"]',
  'name = "workflows"',
  # NOTE: Would be nice to get osx-64, osx-arm64, and win-64
  # However, some packages don't have builds for osx-arm64 and win-64.
  'platforms = ["linux-64"]',
  'version = "0.1.0"',
  "",
  "[dependencies]",
  'r = ">=4"',
  'r-roxygen2 = "==7.3.2"',
  pixi_string
)

outdir <- "~/scratch/test-pecan-pixi"
dir.create(outdir, showWarnings = FALSE)
writeLines(pixi_lines, file.path(outdir, "pixi.toml"))

I'm not sure this replaces Singularity/Apptainer/Docker for portability, but I find this way of working somewhat more convenient, especially for development workflows. I've been actively using pixi for lots of Python and R projects and I find it's incredibly effective for eliminating the headache of moving development around between operating systems.

Food for thought!

[mdietze]

As someone who continues to do mostly "bare metal" PEcAn installs, and has watched many a student spend their entire first week on a project to just getting PEcAn installed and running, I really like the idea of a build system that makes that easier (8 min would be amazing). That said, the need for new PEcAn developers to learn all of our system-specific tools/hacks (building documentation, updating the Docker dependencies, updating the build log files, etc) is already challenging and I'd hate to create another thing that people need to keep up-to-date. Is there a way for these scripts to be more integrated with things like docker/depends/pecan_package_dependencies.csv for building it's list of packages to install on-the-fly instead of being hard coded?

Also, for those of us operating on a HPC, is there a way to get pixi to NOT install duplicate copies of things that are already in the module system? Maybe I'm being silly, but I'm particularly hesitant to install entire copies of R itself (into a non-standard location and without sudo privileges).

[ashiklom]

I'd hate to create another thing that people need to keep up-to-date. Is there a way for these scripts to be more integrated with things like docker/depends/pecan_package_dependencies.csv for building it's list of packages to install on-the-fly instead of being hard coded?

100%. We should definitely do this. This specific implementation is just something I sketched out quickly and is not where we should land. We should have one authoritative list of dependencies, stored in version control, a single script that updates that list periodically, and some ability to translate that list to pixi or other things.

Also, for those of us operating on a HPC, is there a way to get pixi to NOT install duplicate copies of things that are already in the module system? Maybe I'm being silly, but I'm particularly hesitant to install entire copies of R itself (into a non-standard location and without sudo privileges).

Alas, no. The reason pixi works as quickly and consistently as it does is that it installs binaries of everything that are pre-compiled against each other. Your question isn't silly -- it highlights an important trade-off about the pixi-like approach of installing binaries. Besides duplicating installs of big things like R and Python, you also lose out on HPC hardware-specific optimizations and nonstandard compiler flags, which can be important for HPC development.

Where pixi shines is: Once you get an environment working on one system, it is guaranteed to be identical on every other system (with some caveats for major architecture differences, like x86-64 vs arm64 vs. windows). That is a huge benefit to cross-system development; it "just works"!

My idea would probably be to use something like pixi (or its close cousins, conda/mamba) as the default for PEcAn development (especially cross-platform) but to retain the ability to do a lower-level source install for HPC and other special cases.

But, we should discuss this as a team! I don't mean to push this as the best solution, just suggest it as something I personally like that has worked really well for me.