Tracking Issue for `shuffle` and `shuffle-seed`

[smoelius]

This is a tracking issue for unstable libtest options --shuffle and --shuffle-seed (#85440).

These options allow tests to be run in random order, as opposed to the default alphabetical order.

Steps / History

• Implementation: Implement #85440 (Random test ordering) #89082
• Final comment period (FCP)
• Stabilization PR

Unresolved Questions

• None yet.

[smoelius]

Following up on #89082 (comment) (cc: @joshtriplett).

This appears to be the commit that introduced lld's --shuffle-sections: llvm/llvm-project@d48d339

Here are some differences I noticed between their implementation and ours:

• They use a 32-bit seed, we use 64 bits. Using a 64-bit seed seems reasonable since, e.g., that is what Hasher::finish outputs. Note that Go also uses a 64-bit seed.
• They disallow 0 as a seed; we allow it. It is conceivable that a developer might test with a fixed set of seeds, and that 0 might be one of them. Hence, it seems preferable to allow 0 as a seed.
• They use Mersenne Twister for their random number generator; we use a kind of iterated SipHash13. Using a standard random number generator would be preferable, but we are limited by what code is available to a core library. See this comment for some discussion.

[epage]

Hmm, the original issue (#85440) was left open and not reused as the tracking issue, so unsure where discussion should go.

Pulling over my comment at #85440 (comment)

The testing-devex team is looking at defining a minimal API for any test harness to implement, shifting extra functionality to test runners, like cargo test, or custom test harnesses implementing extensions on top of that minimal API.

As we get to the point that cargo test is able better integrate with this API, I wonder if shuffling should be handled inside of cargo test. This would require just one small tweak to libtests behavior: tests are sorted alphabetically unless a filter is applied and then they are in filter order, followed by alphabetical (e.g. assert-rs/libtest2#96). cargo test could do a --list --format json operation on the tests, shuffle the results, and pass them down to the libtest binary with --exact. We'd likely blow up CLI lengths with this and need to support @argfiles.

Benefits

• Less for custom test harness to implement, reducing build times
• Being in Cargo offers some more flexibility, for example if we put this in config, then we get persistent, environment variable, and CLI support "for free".
• This scales to other potential needs, like sharding of tests (splitting tests across a matrix of jobs)
  • As an aside on sharding, what we shard on (package, binary, test) and whether we break down that category further would be dependent on the costs of each. By sharding on packages, jobs can avoid building dependencies they may not care about. By sharding on binaries, jobs can avoid building and linking all test binaries.