Merge pull request #61 from hasufell/midstream-bindists

Add midstream bindist proposal

Author: jmct

proposals/0000-midstream-bindists.md

@@ -0,0 +1,281 @@
+# Midstream bindists
+
+## Abstract
+
+Bindists for the Haskell toolchain have been produced by upstream (the developers of each respective tool) for
+a long time and many tools rely on these "official" bindists (e.g. GHCup and stack).
+
+We propose here that bindists are built and maintained by the GHCup project, which
+provides the main installation experiences in the Haskell ecosystem, removing the hard
+dependency on upstream bindists entirely.
+
+## Background
+
+Historically, installers like GHCup and stack have used upstream bindists for mainly one reason: it's easy to do
+so and doesn't require further efforts.
+
+However, using upstream bindists directly is extremely rare in the Linux world of distribution. Most distributions
+build, package, test and curate binary packages themselves, not only because they have custom formats, but for
+reasons of control, trust and quality.
+
+The relationship of GHCup and bindists has also been described in the blog post
+[GHCup is not an installer](https://hasufell.github.io/posts/2023-11-14-ghcup-is-not-an-installer.html) recently.
+
+## Problem Statement
+
+From the perspective of a GHCup developer, there are several issues with relying on upstream bindists.
+
+### Platform support
+
+GHC and other tools have in the past dropped support for certain platforms either entirely or requested
+the community to step up and do the work (e.g. on GHC CI).
+
+E.g. [GHC ARMv7 support was dropped silently without any call for help](https://gitlab.haskell.org/ghc/ghc/-/issues/21177#note_470440). Similarly, FreeBSD support just ceased to exist when the GHC FreeBSD CI stopped working. Later the community asked for a [revival](https://gitlab.haskell.org/groups/ghc/-/epics/5), but nothing signifcant has happened so far.
+GHCup still produces bindists from time to time for FreeBSD, but e.g. the [HLS release manager for 2.5.0.0 recently refused to add FreeBSD bindists](https://github.com/haskell/ghcup-metadata/pull/159).
+
+Similarly, stack used to have issues with Aarch64 bindists:
+
+* https://github.com/commercialhaskell/stack/issues/5709
+* https://github.com/commercialhaskell/stack/issues/5854
+* https://github.com/commercialhaskell/stack/issues/5540
+* https://github.com/commercialhaskell/stack/issues/5610
+* https://github.com/commercialhaskell/stack/issues/5619
+* https://github.com/commercialhaskell/stack/issues/6141
+* https://github.com/commercialhaskell/stack/issues/6142
+
+Recently, cabal-install had issues with i386 binaries and alpine, delaying a GHCup metadata PR:
+
+* https://github.com/haskell/ghcup-metadata/pull/127#issuecomment-1766020410
+
+These issues are frequent and so far the GHCup developers used to single handedly fix all those missing bindists manually
+and provide them here: https://downloads.haskell.org/~ghcup/unofficial-bindists/
+
+This is unfunded and significant work.
+
+### Bindist maintenance
+
+Sometimes, bindists are broken, e.g. for GHC there are a couple of instances:
+
+* 9.0.2 shipping without profiling info: https://gitlab.haskell.org/ghc/ghc/-/issues/21841
+* DESTDIR variable ignored by `make install`: https://gitlab.haskell.org/ghc/ghc/-/issues/19646
+
+Sometimes, bindists have been built for very old version of linux distros and won't run well on newer linux versions.
+This is currently a problem since Debian has removed ncurses5:
+
+* https://github.com/haskell/ghcup-hs/issues/902
+* https://answers.launchpad.net/ubuntu/+source/ncurses/+question/707838
+* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025964
+
+For cabal, there has been the infamous hLock issue:
+
+* https://github.com/haskell/cabal/issues/7313
+* https://github.com/haskell/cabal/issues/7950
+
+This shows that bindists, for current and historical versions, need continuous maintenance. However, upstream developers
+so far have very rarely engaged in this type of maintenance work, pushing it down to GHCup. As an example, here are all the
+manually patched and re-packaged bindists that fix the DESTDIR bug outlined above: https://downloads.haskell.org/ghcup/unofficial-bindists/ghc/curated/
+
+This type of work also requires significant time.
+
+### Quality gateway
+
+There's something pleasing about upstream providing bindists: there's a perceived trust about it in the community,
+e.g. the [haskell.org committee expressed concerns about GHCup changing bindists in the past](https://github.com/haskell-infra/www.haskell.org/issues/212#issuecomment-1272312911).
+
+However, rarely do upstream developers have signifcant experience in redistribution, nor do they have the time to focus
+on all the issues that come with it. Bindists are mostly provided "as-is" and support beyond what the release CI outputs is
+left to midstream (GHCup, stack, ...).
+
+Conceptually, it is a good idea to separate concerns: upstream provides the sources and has tested it. Distributions build the binaries,
+validate that the program can be successfully built from source and make sure that the final artifacts pass the test suite.
+This is good, because we want to know whether end users can build e.g. a functioning GHC from source. If only the release CI outputs
+a GHC that passes the test suite, then something is fundamentally broken.
+
+Distributors are often closer to the end-users and can provide additional support and efforts for the installation experience.
+
+### Security backports
+
+Cabal has recently found to be vulnerable to [HSEC-2023-0015](https://github.com/haskell/security-advisories/blob/main/advisories/hackage/cabal-install/HSEC-2023-0015.md).
+
+This vulnerability [had not been communicated to the GHCup team](https://github.com/haskell/security-advisories/issues/129) prior to disclosure,
+causing high distress for a backport, since at the time of disclosure, cabal-3.6.2.0
+was still 'recommended' by GHCup, since [cabal-3.10.2.0 is still broken on windows](https://github.com/haskell/cabal/issues/9334).
+
+As such, GHCup developers needed a quick and efficient way to:
+
+* patch cabal
+* build release binaries for cabal
+* ship the binaries
+
+This had been done as a downstream release `3.6.2.0-p1` roughly a week after the disclosure.
+
+Meanwhile, cabal upstream still has not finished their backport due to issues
+with hackage dependencies: https://github.com/haskell/cabal/pull/9457
+
+This shows that a certain amount of independences from upstream CI and upstream workflow
+is essential to fulfill swift security backports to potentially under-maintained
+branches/versions.
+
+### GHC nightlies
+
+As a special case, I want to point out that GHC nightlies have been frequently broken beyond repair:
+
+* https://gitlab.haskell.org/ghc/ghcup-metadata/-/issues/2
+* https://gitlab.haskell.org/ghc/ghc/-/issues/24000
+
+The breakage was left unattended and some bindists completely vanished from gitlab CI artifacts, because of misconfiguration.
+Here's a graph of nightlies availability: https://grafana.gitlab.haskell.org/d/ab109e66-a8a1-4ae9-b976-40e2dfe281ab/availabilitie-of-ghc-nightlies-via-ghcup?orgId=2
+
+## Prior Art, Related Efforts and alternative solutions
+
+So far, GHCup developers have tried to close the gap, doing signifcant work on upstream CIs and building bindists manually where
+necessary.
+
+Bindists for cabal-install are now produced by GHCup's own CI: https://github.com/haskell/ghcup-metadata/blob/develop/.github/workflows/cabal-release.yaml
+
+HLS will likely follow shortly.
+
+As mentioned before, there have been attempts to improve the coordination and collaboration across the entire Haskell
+toolchain, view the tooling end-user experience in a holistic way and make decisions based on that end-user experience: https://github.com/haskellfoundation/tech-proposals/issues/48
+
+The proposers believe that the community structure at the moment does not allow such an approach and there needs to be
+significant work to align goals, perception and priorities. Otherwise there will be too much friction.
+
+The most important currency in the open source volunteer world is **energy**. It is not code or technical effort. As such we
+believe that the amount of saved energy by being more independent of upstream release processes and decisions far outweighs
+potential costs of technical redundancy/duplication.
+
+A future proposal may very well attempt to create a unified user experience across the entire Haskell toolchain
+through joint management and collaboration. But that is not in the scope of this proposal and we have no concrete
+idea how to achieve that.
+
+## Technical Content
+
+We propose here to start with the smallest step possible, to build the entire Haskell toolchain autonomically.
+The way this will be implemented is to start a central GitHub repository that builds bindists for releases of:
+
+- GHC
+- HLS
+- cabal
+- stack
+
+For the following platforms:
+
+- FreeBSD x86_64
+- Linux i386
+- Linux x86_64
+- Linux armv7
+- Linux aarch64
+- Darwin x86_64
+- Darwin aarch64
+- Windows x86_64
+
+For the following Linux x86_64 distros:
+
+- Debian
+- Ubuntu
+- Mint
+- Fedora
+- CentOS
+- RedHat
+- Rocky Linux
+- Void Linux
+- Amazon Linux
+- Alpine
+
+Linux i386, armv7 and aarch64 will be confined to Debian or Ubuntu.
+
+These private runners will be made available to the whole Haskell GitHub org and as such benefit
+other projects there as well (like HLS, Cabal, bytestring, etc.).
+
+## Future work
+
+The following ideas and goals are outside of the scope of this proposal, but are essential
+to understand the broader mission and roadmap that motivated this proposal.
+
+### Enhancements to bindist quality and installation experience
+
+Further goals are:
+
+* enhance the quality of the bindists by
+  - running the entire test suite for all of the tools
+  - having a mechanism to report test failures back upstream
+  - publishing test failures for end users to see
+  - communicte test status of bindists clearly through e.g. GHCup
+  - resolve GHC issues related to test bindists:
+    * https://gitlab.haskell.org/ghc/ghc/-/issues/22726
+    * https://gitlab.haskell.org/ghc/ghc/-/issues/22723
+    * https://gitlab.haskell.org/ghc/ghc/-/issues/22727
+* make fixing bindists easier
+  - implement revisions in GHCup: https://github.com/haskell/ghcup-hs/issues/361
+  - make it easy to update an older GHC branch and re-run the release pipeline
+- Make building upstream release binaries easier
+  * https://github.com/haskell/cabal/issues/9461
+  * https://github.com/haskell/haskell-language-server/issues/3878
+
+One main idea is that bindists should be primarily tested **on the users system**, because that is where they're going to run.
+It is great to know that e.g. the test suite passes on GHC CI, but that may have little value in different environments.
+Additionally, issues with tests can flow back to upstream developers and we may develop workflows and processes to streamline
+this type of feedback. Early release candidates can assist with this workflow.
+
+Another perception shift necessary is that upstream projects should consider that their build system
+are end-user interfaces, making it easier for both distributors and end-user to build release binaries correctly themselves.
+
+### Nightlies
+
+We also want to make nightlies available for GHC and cabal and HLS. This will require
+coming up with a permanent storage solution and very robust nightly pipelines.
+
+## Timeline
+
+* 6 months: proof of concept of a central GitHub CI building most of the toolchain
+* 12 months: building GHC via github actions
+
+## Funding
+
+### Who and what
+
+The GHCup project requests funding for private GitHub CI runners to power the midstream bindist release pipelines.
+It will receive and manage the funding in strict collaboration with the Haskell Foundation.
+
+Volunteers who want to collaborate in midstream bindists are welcome to look at the project structure and collaboration guidelines:
+
+* https://www.haskell.org/ghcup/about/#team
+* https://www.haskell.org/ghcup/dev/#contribution-process-and-expectations
+
+### Budget
+
+The following is an example/estimate of a budget (prices are in USD). The HF and the proposer will negotiate the exact terms in private.
+
+- Linux/FreeBSD x86_64 runner on Hetzner (AX52)
+  * monthly: $62.24
+  * yearly: $746.88
+- Linux aarch64 runner on Hetzner (RX170)
+  * monthly: $181.73
+  * yearly: $2,180.76
+- Darwin runner on Hetzner (Mac Mini M1)
+  * monthly: $56.59
+  * yearly: $679.08
+
+To host one private runner per all these platform, the yearly cost would be: **$3,606.72**
+
+There will likely also be an initial setup cost (as is usual for Hetzner).
+
+We may request more runners depending on the demand, so it may very well be 3 runners per platform, resulting in yearly cost of: $10,820.16
+
+**As such, the budget estimated is between 4 to 10k USD per year.**
+
+## Stakeholders
+
+* GHCup developers, who receive funding
+* GHC developers
+* cabal developers
+* stack developers
+* HLS developers
+* VSCode Haskell developers
+* Haskell toolchain end users
+
+## Success
+
+* reliable, continuously maintained bindists, readily available
+