Rewrite to use setup-toml-test #12
Conversation
This rewrites the lot to use setup-toml-test[1], rather than relying on the correct environment being set up on the local machine. This way the authors of implementations deal with the correct setup of the ecosystem and building of the required tools. Managing a few ecosystems is okay, but it's becoming unwieldy to manage over a dozen. Some ecosystems are fairly stable, others a lot more fickle. The main downside is that we lose the benchmarks. I might look into that in the future, but I'm okay with that for now. Another question is what to do with unmaintained implementations. I think I'll probably add a feature to take the tests from a fork. [1]: https://github.com/toml-lang/setup-toml-test
|
I'll be more blunt than before. I think this will not get enough desired participation and what results will be there will be of lower quality that this will severely hamper or kill this effort. I'm questioning whether I'd add the relevant jobs to my CI and I'm a regular participant with this. |
|
How would you suggest we proceed? Because me manually managing it all is a huge timesink and also not really an option. Right now I have time now to spend on it, but when that stops updates will stop too (like it did for about a year when I was busy). |
|
Also it doesn't need to be a binary thing: if some implementations need to be done manually as currently, then that can be okay. If we have 80% from GitHub actions, then that's already 80% better than doing it all manually. But obviously I'd like to delegate as much of the work as possible: people already have the correct environments in their CI, so why not run it there? |
|
Looking at this from an implementers perspective, I already run the tests in a way that is integrated with my workflow. I can advertise the results of that on my README. So how much value do I get out of adding this extra step? For me, its mostly from adding parsers I don't have control over to help people understand the costs of adopting those parsers. Requiring the projects to adopt this puts that benefit at risk. For end-users, this as the problem that the results will be untrustworthy as I called out in #11 because this is dependent on the projects to run. Yes, if the parsers put it on a schedule it can help so long as Github doesn't unschedule the job (iirc it takes just 3 months of inactivity to have it unscheduled). |
|
I don't think we should do this -- specifically, because we should NOT require TOML parser implementations to be hosted on GitHub / use GitHub Actions, to be listed on such a page.
Based on https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/disabling-and-enabling-a-workflow, it's 60 days. Based on my experience maintaining https://sphinx-themes.org/, the most reasonable thing that we can do here is have as much of the generation logic live in a single place and be as automated as possible. Ideally, that automation also allows working with a single project at a time as well as letting individual project's generation fail gracefully. Notably, this would also enable explicitly surfacing when individual projects are failing with some issue (and what that issue is) without needing site-maintainer intervention, as well as providing an auditable source for the performance descriptors being provided in the data. To that end, I think the direction that #9 is taking (having the CI run in this repository, for regularly regenerating the site) along with the logic from this PR's metadata management (having a human-editable file that contains the list of parsers to process) is a better way to handle this than to require the use of a specific GitHub Action in the destination projects. |
|
Adding some other integration should be easy enough. It's just that all the implementations on the wiki except one use GitHub now, so support for anything else isn't really required. It's all pretty trivial. Other than that, the Sphinx template repo is a different scenario because you don't have tons of toolchains and ecosystems to manage. Doing this in the CI is worse than on my local machine because everything will take forever. I'd rather stab a fork in my eye than deal with that. That's why I abandoned that (twice), because it retains all the same problems I'm running in to now and doesn't really fix anything. |
Also: to be blunt: I started working on this approach as I was just fed up dealing with all of this. I have zero interests continuing to work on this with how it works now, which is really just a thing I very quickly hacked up after a HN discussion. It's okay for a few implementations in a few languages I'm somewhat familiar-ish with, but it just doesn't scale at all. No solution is perfect here, but I don't see any other option. I appreciate that the current method might make things a bit easier for you Ed, but at the expense of making it much harder for me. |
Keep in mind that this is unlikely to apply to just me but to others maintainers. |
|
Wouldn't it be possible to leverage Docker, and the GitHub Container Registry to make the setup viable? Using a base image with just the toml-test binary, and then a bunch of images for the different ecosystems and libraries. Those images could be published to ghcr so re-runs are faster as images won't need to be re-built, if the image needs quite some time to build. It's very likely multiple libraries will be using a same Dockerfile (e.g. an image with Node LTS and Corepack installed/enabled is likely enough for all the JS libraries). And actually you probably won't even need to bother much and can use off-the-shelf stock images and copy/download the toml-test binary, so ghcr might not even be necessary. (node, maven, gradle, rust, gcc, ...) Then have a configuration file, that's specifying for each library which Dockerfile it needs to use for the run, and a script setting up the container, running the commands, and collecting the results. Admittedly it's not very far from the current setup, although having each ecosystem isolated in a (reusable) container seem to at least make it a lot more manageable for running anywhere (in CI, or locally without installing a bazillion different toolchains). |
|
Yes, I think that could work @cyyynthia. The question is: who is going to create and maintain those images, and ensure they actually work? For that, you'll need to run that in the repo's CI. Having greater reproducibility is good, but also a lot more complex to set up.
I don't know; only one way to find out I guess? To be honest I don't really understand why you wouldn't want to add it? The only thing that would change for you is the location of the toml-test integration binaries, and I would really prefer to keep those in the repo whenever possible regardless of this change, so they can be maintained, tested, etc. Adding some of them here was always a quick hack motivated by expediency more than anything else. While Rust/Cargo is generally fairly painless, the Rust integration did break a few weeks ago and I had to spend some time figuring that out. And this multiplies: it's Rust this week, Haskell the next week, JS the week after, etc. I appreciate that you did help out with that, but I don't think that me chasing after dozens of people to please fix things in this repo is really going to work in the long term. It's also not something I want to do, because pretty much everyone is doing this unpaid in their spare time, and by writing a TOML implementation doesn't mean you signed up to maintain an integration here. |
This rewrites the lot to use setup-toml-test1, rather than relying on the correct environment being set up on the local machine.
This way the authors of implementations deal with the correct setup of the ecosystem and building of the required tools. Managing a few ecosystems is okay, but it's becoming unwieldy to manage over a dozen. Some ecosystems are fairly stable, others a lot more fickle.
The main downside is that we lose the benchmarks. I might look into that in the future, but I'm okay with that for now.
Another question is what to do with unmaintained implementations. I think I'll probably add a feature to take the tests from a fork.