Windows Readiness

[leostera]

This is a tracking issue for the State of the OCaml Platform on Windows document.

Issue #1: OCaml is typically installed from source, which needs a C compiler

Unfortunately, Unix emulation cannot be entirely avoided. The OCaml compiler, along with nearly all libraries in its ecosystem, relies on Unix utilities such as sed, sh, make, and tar for the build process. As a result, a Unix environment is necessary at some stage to facilitate the compilation of dependencies, including the compiler itself. However, it is possible to produce final binaries that do not require any Unix emulation, ensuring they run natively on a clean Windows system.

The ongoing work on relocatability of the compiler is necessary to ship compiler binaries and would improve the installation time by a lot.

Issue #2: Standard library isn’t Windows friendly

Many things in the standard library were not built with Windows support in mind, or don’t reflect Windows preferences as they have evolved.

The most frequently recurring topics in this context are path manipulation and process launch and interaction. Other commonly mentioned issues include signal handling and end-of-line management. These core functionalities are provided by the Sys, Unix, and Filename modules included with OCaml.

The design philosophy behind Sys, Filename, and Unix was to offer the same high-level functions across all platforms. Most Sys functions were originally designed to align with ANSI C standards at the time. Signal handling in Sys and Unix attempted to emulate Unix behavior on Windows where feasible, though certain features, such as Unix.fork, were inherently incompatible. The Filename module was kept minimal, focusing primarily on compiler requirements and relative path manipulation. Consequently, these modules often fall short of meeting the full range of system interaction needs for many users.

This minimalist approach led to the development of alternative system libraries and overlays, such as Core_unix and BOS, as well as projects that vendor their own system libraries, including Dune and opam. However, not all of these system libraries have been developed with equal consideration for Windows compatibility.

Relevant issues and PRs

• Add Sys.runtime_executable and caml_sys_proc_self_exe ocaml/ocaml#13728
  • Symmetrize shared Sys and Unix functions ocaml/ocaml#13447
  • Stdlib - missing signals ocaml/ocaml#13825 (comment)

Issue #3: Complexity of external dependencies

The integration of non-OCaml software dependencies is currently managed primarily through opam's depext system. This mechanism is used to locate or install various types of external dependencies, including:

• General-purpose system tools (e.g., git, wget)
• External compilers and build tools (e.g., pkg-config, cargo)
• External binaries required at runtime (e.g., ffmpeg)
• Foreign libraries (e.g., openssl, libpng)

The depext system provides a single source of dependency specifications, which opam can either infer automatically or enforce based on user-defined opam variables. This centralized approach works well on certain platforms, such as Debian, where system tools and libraries—whether for end users or developers—are packaged uniformly.

However, for other environments, particularly Windows and cross-compilation setups, the current system has limitations and lacks flexibility. A more suitable approach would allow:

• Different sources for tools and foreign libraries (e.g., using winget for git, MinGW for compilers, MSYS2 for precompiled foreign libraries, and VCPKG for source-distributed libraries)
• Separate sources for host and cross-compiled foreign libraries (e.g., Debian for host dependencies and MXE for cross-compilation targets)

Unlike its initial implementation, the current depext system is now built directly into opam, meaning any extension requires modifications to opam itself. Adding new sources or introducing a depext source mixin system necessitates patching opam, which increases complexity as more systems and use cases need to be supported.

Issue #4: Core parts of the ecosystem are built for Unix

Several libraries in the ecosystem Lwt, Eio, Fpath, Bos, have been written with Unix conventions in mind, or have runtime dependencies on Unix tools.

Relevant issues/PRs

• Improve handling of Windows paths ocaml-multicore/eio#762
• Outreachy: Improve Windows Support in Eio ocaml-multicore/eio#758
• Windows support ocaml-multicore/eio#125
• fix setting Lwt_process env on Windows ocsigen/lwt#967

Issue #5: Cross-compilation considerations

With technically possible, there are no easy to install, readily available cross-compilers available that make cross-compiling to windows a matter of simply running dune build -x windows.

Managing native cross-compilation toolchains requires manual work.

Issue #6: Sharing and reusing OCaml packages

The lack of metadata on opam packages around what packages work (and to what degree they work) on different platforms, makes it difficult to know what works on windows.

Additionally, the lack of a standardized way for writing multi-platform code makes it difficult to write, publish, and consume multi-platform packages.

Issue #7: Performance considerations

Currently, opam update takes ages on Windows. This is in big part due to writing the new version to disk and calling diff. Windows and Windows Defender make working with so many files very slow.

[leostera]

@Sudha247 actually, could we transfer this issue to ocaml-windows/roadmap? I think this would be a better place to have it since these are more than small papercuts people will encounter.

[dra27]

It's a good summary, thanks! Two comments:

Unlike its initial implementation, the current depext system is now built directly into opam, meaning any extension requires modifications to opam itself. Adding new sources or introducing a depext source mixin system necessitates patching opam, which increases complexity as more systems and use cases need to be supported.

I disagree that the integration of depext into opam has increased the complexity - changes to the old opam-depext plugin were just as difficult to get through. opam-depext was integrated into opam to solve (pun intended) a very real solving issue across all systems. Trying to iterate a new depext system outside of opam will very likely hit the same problem that the old opam-depext plugin suffered from.

Currently, opam update takes ages on Windows.

Just for status, this is being actively worked on, and it's hoped it'll be addressed in opam 2.4 (to be released in the next few months)

[samoht]

@dra27 is there a plan to expose opam-depext as a separate library/tool? It's not very practical to have this built-in into opam (as it ties the release schedule and makes it hard to install depext from dune pkg). I understand the rationale of merging this in opam at the time, but I still do think it was somehow a local optimum :-)

[dra27]

The actual probing logic for depext is low-level in opam and can be used directly from the libraries, without needing an opam root (well, and if it turns out that's not possible, then that can/should be changed). When we were originally discussing this in the context of dune pkg, the expectation I think was that Dune was more likely to advise the user about what to run, rather than ever actually run the commands (in the way that opam is configured to by default)?

[leostera]

Thanks @dra27! And thanks for transferring the issue. I'll let @Sudha247 address the comments on the contents 🙏🏼

re: dune managing system dependencies, the dev preview of Dune already lists what system deps are missing, but will not try to install them for you.

The only ecosystem that gets somewhat close to this is Rust but they do not install system dependencies at all. Instead they rely on a feature-flag system at the package level and conditional compilation to offer support for linking to an externally built library, or a bundled mode that compiles the target native library as part of the project (see the sdl2 crate, sqlite3-sys, v8, etc).

[Sudha247]

Just for status, this is being actively worked on, and it's hoped it'll be addressed in opam 2.4 (to be released in the next few months)

Wonderful! Just curious: do we have any numbers for this? I understand if it's too early to tell.

I disagree that the integration of depext into opam has increased the complexity - changes to the old opam-depext plugin were just as difficult to get through. opam-depext was integrated into opam to solve (pun intended) a very real solving issue across all systems. Trying to iterate a new depext system outside of opam will very likely hit the same problem that the old opam-depext plugin suffered from.

Makes sense. I don't know enough about the history of depext, but I understand that integrating it with opam has been useful. However, the context here is the depext system on Windows. A quick scan of the Windows health check shows that many package build failures are due to dependencies on conf packages. The blocker here is addressing them. Would it accurately reflect the status if we removed the part about depext being directly built into opam?