chore: make integration tests run concurrently #924
Conversation
| pub rest_catalog: RestCatalog, | ||
| pub catalog_config: RestCatalogConfig, |
There was a problem hiding this comment.
Since the tests now share this fixture but they're all executed in different runtimes, they can't share the same client, as those can get dropped prematurely when a test ends, resulting in dispatch task is gone: runtime dropped the dispatch task
| .name("t1".to_string()) | ||
| .name("t2".to_string()) |
There was a problem hiding this comment.
Making the write tests use different tables each allows for them to also use the shared docker stack.
By extension make them use the same set of docker containers.
gruuya
force-pushed
the
concurrent-integration-tests
branch
from
fd569a9 to
110ce33
Compare
There was a problem hiding this comment.
generally lgtm. Since we're using a shared catalog, are there any concerns with side effects?
In pyiceberg we use a table identifier fixture to generate table names so they dont conflict
Good point, I could also extract the namespace as a shared fixture among the tests (in the PR i just ignore the result of |
Closes #923
This PR makes integration tests re-use the same set of docker containers, hence allowing them to be run concurrently
sharedmodule, with a single top-levelmain.rsfile, thus requiring only one compilation step (as opposed to compiling every test file individuallyset_test_fixtureis made sync, since there's no async code in itOnceLock, so that it is invoked only once for the resulting (shared) test binary; it also has a corresponding destructor to spin down the containers after the tests runmain.rs(top-level); otherwise they should be created in thesharedmodule#[serial])Finally, with this change my integration test runs go from taking ~3.5 minutes down to 30 seconds.