chore: use shared containers for integration tests

[gruuya]

Closes #923

This PR makes integration tests re-use the same set of docker containers, hence allowing them to be run concurrently

1. all existing tests are moved to a shared_tests module, with a single top-level shared.rs file, thus requiring only one compilation step (as opposed to compiling every test file individually)
2. set_test_fixture is made sync, since there's no async code in it
3. given 1 and 2, the test fixture is wrapped in a OnceLock, so that it is invoked only once for the resulting (shared) test binary; it also has a corresponding destructor to spin down the containers after the tests run
4. if there's a need for integration tests which don't use the shared docker containers, their test files should be placed alongside shared.rs (top-level); otherwise they should be created in the shared_tests module
5. one (shared) file can now contain an arbitrary number of tests that utilize the shared set of containers (without having to resort to decorating them with #[serial])

Finally, with this change my integration test runs go from taking ~3.5 minutes down to 30 seconds.

EDIT: The CI unit workflow run duration also seems ~10min shorter now.

[gruuya]

Since the tests now share this fixture but they're all executed in different runtimes, they can't share the same client, as those can get dropped prematurely when a test ends, resulting in dispatch task is gone: runtime dropped the dispatch task

[kevinjqliu]

generally lgtm. Since we're using a shared catalog, are there any concerns with side effects?
In pyiceberg we use a table identifier fixture to generate table names so they dont conflict

[gruuya]

generally lgtm. Since we're using a shared catalog, are there any concerns with side effects? In pyiceberg we use a table identifier fixture to generate table names so they dont conflict

Good point, I could also extract the namespace as a shared fixture among the tests (in the PR i just ignore the result of create_namespace assuming it was already created if it was an error).

EDIT: I've extracted two common fixtures now: the apple-ios namespace and the foo-bar-baz schema.

[xxchan]

Not sure if I understand it correctly:

• Before this PR, tests are compiled into separated binaries. In theory they can run concurrently (e.g., if use cargo nextest), but just cargo test will run them in serial.
• In this PR, we put them into 1 binary. And then they are run concurrently by #[tokio::tests].
  • We use "shared container" to also save the time of spinning up and down containers. But need to take care of potential conflicts.

[gruuya]

• Before this PR, tests are compiled into separated binaries. In theory they can run concurrently (e.g., if use cargo nextest), but just cargo test will run them in serial.

Correct; at present the tests can't be run concurrently because they depend on separate docker container sets. (Also even if they didn't, cargo nextest would compile each top-level test file separately.)

• In this PR, we put them into 1 binary. And then they are run concurrently by #[tokio::tests].

  • We use "shared container" to also save the time of spinning up and down containers. But need to take care of potential conflicts.

Yep, that's it in a nutshell. Using the shared container set is the biggest time-saver (eliminating the docker build-start-stop overhead for each test). Having made them use shared containers, the next simple improvement is to make them all compile to one test binary (thus eliminating multiple compilations), as is done through having a single top-level shared.rs file, and consequently run them concurrently as well.

[xxchan]

Code changes and the idea LGTM 👍 Maybe we could rename the PR title to "use shared container for integration tests" instead as it's the biggest time-saver?

[kevinjqliu]

thanks! generally lgtm.
Does this also affect CI runs? if so, would be great to also include how much time we saved there too.

[gruuya]

thanks! generally lgtm. Does this also affect CI runs? if so, would be great to also include how much time we saved there too.

Good question; a brief glance suggests it may have shaved about ~5 minutes from the unit CI workflow, e.g. by comparing the Test step duration from a job on this PR(3m 51s) and a job on another PR(8m 45s).

The difference would certainly grow as more and more integration tests are added.

EDIT: It's actually closer to 10 minutes, since the integration tests are run twice, once in the Test step and once in the Async-std Test step.

[kevinjqliu]

thats amazing! Thanks for looking into that, i like faster CI :)

[gruuya]

Hey @ZENOTME @Fokko, can you also take a look at this proposal?

[ZENOTME]

Hey @ZENOTME @Fokko, can you also take a look at this proposal?

Thanks! @gruuya It's a great job to improve our ci. the idea and code change LGTM.

[ZENOTME]

Each test is an independent test and I think it's common for them to have different schemas, e.g. scal_all_type. Do we really need to extract this as a public function? 🤔

[gruuya]

Hey, thanks for the review! So initially I found that append_data_file_test.rs, append_partition_data_file_test.rs and conflict_commit_test.rs use the same schema, and I figured it made sense to extract it simply for de-duplication (and allowing any future tests where schema isn't that important to re-use it as well).

I don't have any strong opinions on it though—would you prefer for the schemas to be constructed directly in each of those tests as before?

[ZENOTME]

I figured it made sense to extract it simply for de-duplication (and allowing any future tests where schema isn't that important to re-use it as well).

Both way looks good to me. I'm fine to keep it now for reuse by 3 case and we can remove them when need to evolve schema indenpently.

[ZENOTME]

cc @liurenjie1024 @Xuanwo @sdd