Portable Lockdirs

[gridbugs]

Using this issue to track work for making dune's lockdirs portable.

The context of this work is that currently a dune lock directory is specialized to a single machine (combination of arch, os, distro, os version). By default when you generate a lockdir it will be specific to the current machine, but it's possible to force dune to generate a lockdir for a different machine (in which case the lockdir will be specialized to that machine). The main problem this causes is that if one contributor to a project checks in their lockdir then other contributors to that project may be surprised when the lockdir does not work on their machines. A second problem is that specialized lockdirs contain a profile of the machine on which they were generated. Lockfiles in other ecosystems do not work this way, so this behaviour may lead to users unknowingly checking in details of their machine they would rather the public not know (e.g. if they are running an outdated version of their distro with known vulnerabilities).

The goal of this work is to make it so one user can generate a lockdir on their machine, and a second user can use the same lockdir to build the project on a different type of machine (where build/install commands and dependencies of some packages may be different from on the first user's machine).

A second goal of this work is to do it as quickly as possible so this change can be evaluated with the dune developer preview. As such I'll cut some corners to simplify the implementation and use feature flags to protect against breaking changes where appropriate.

There two problems that need to be solved:

1. Build/install commands are specialized to the machine where the solver ran (an opam package may specify that different commands should be run to build/install a package depending on the current platform)
2. The dependencies of each package are specialized to the machine where the solver ran (an opam package may specify that a package has different dependencies on different platforms)

Problem 1 can be solved by deferring evaluation of platform-specific variables to build-time. Here's a PR which makes this change.

Problem 2 is more complicated and will be the focus of the remainder of this issue. It's complicated because opam solvers make decisions about which packages to include in a solution based on a given environment. If a package depends on foo-linux { os = "linux" } and foo-macos { os = "macos" }, only one of these packages can be included in the solution while respecting those dependency filters. Further, it's possible that foo-linux and foo-macos conflict with each other, preventing a solver from including both in a solution. A portable lockdir must include both packages, as well as metadata to allow the appropriate package to be built at build-time.

It might help to divide this problem into two parts:

• Representation - How will dune represent the fact that the dependencies of a package are different on different platforms?
• Generation - How will dune choose which packages to include in the lockdir?

Representation

The lockfile format will be extended to allow each dependency of a package to have an associated condition, matching the corresponding opam filter. The example above would become:

(depends
 (foo-linux (= %{os} linux))
 (foo-macos (= %{os} macos)))

Dune already has a DSL for encoding opam filters (it's called "slang"), so adding conditions to dependencies won't be hard.

Different versions of the same packagge package for different platforms

Another problem is that the version of a dependency may differ depending on the platform. This is difficult to fix as it will require a separate lockfile for each version of a package (as each version may have different build commands, deps, etc), and dune would need to choose the correct version of each dependency before building. That sounds a lot like solving, albeit on a very small package universe. The solver could probably be generalized to work for this purpose, however it would be a lot of work. Since I expect this problem to be very rare, I'll ignore it for now and just throw an error instead.

Generation

For the initial version of portable lockfiles I propose simply running the solver multiple times and merging the results. Each run of the solver will be for a different platform, and the set of platforms will be hardcoded based on which are most common in the wild (probably each combination of {linux,macos,windows} x {x86_64,aarch64}). It would also be possible to allow the set of platforms to be changed via a new field in dune-workspace or dune-project, but I'm hesitant to add that at first as it commits us to the solution of running the entire solver multiple times for a concrete set of platforms.

Long term we could explore changes to the dependency problem or the solver to allow a general solution to be chosen in a single pass. One example might be to have the solver treat every dependency filter as true, and to ignore conflicts between dependencies.

"Merging the results"

Each package lockfile will only appear in the lockdir a single time. Build and install commands will be completely portable, so there is no need to update them once computed for the first time. However each time the solver is run for a new platform, the new conditional dependencies may need to be added to some package lockfiles, and existing dependencies present in some package lockfiles may need to be made conditional. A naive approach to updating dependencies is to make each dependency conditional on the disjunction of all the platforms where it was included in the solution.

Continuing the example above, if we first solve for linux-x86_64 and have:

(depends
 (foo-linux (and (= %{os} linux) (= %{arch} x86_64))))

...then solving for macos-aarch64 would update it to:

(depends
 (foo-linux (or (and (= %{os} linux) (= %{arch} x86_64)) (and (= %{os} macos) (= %{arch} aarch64))))

There are obvious improvements that could be made here but I claim this will be sufficient for initially evaluating portable lockdirs in the dev preview.