Installation tests v3

[grusev]

Reference Issues/PRs

What does this implement or fix?

Those are installation tests targeted to be executed against different versions of arcticdb from conda and pypi without building our project.

GitHubWorkflow (not yet finished)

allows matrix execution of selected combinations of OS-es and Pytho versions. For each it install artcicdb from conda(mac-os) and pypi(other os-es), along with our test dependencies. We build also protobufs and execute our tests
Not finished: ability to select arcticdb version, ability to run on demand with selected options from user

Link to successfull run with LMDB tests only:
latest 5.3.3: https://github.com/man-group/ArcticDB/actions/runs/14494364845
5.1.2: https://github.com/man-group/ArcticDB/actions/runs/14509109094/job/40703855715
4.4.7: https://github.com/man-group/ArcticDB/actions/runs/14509267618

Added:

• @pytest.mark.installation - for marking only tests we want to execute
• ARCTICDB_STORAGE_LMDB - env var to allow LMDB to be selected only (if all local storages are disbaled ARCTICDB_LOCAL_STORAGE_TESTS_ENABLED=0 )

Modified:

• fixture 'lmdb_storage' is enhanced so that it can spawn other types of storages on demand, needed for resl storage installation tests where no arctic_* fixtures are available
• FixtureMarks class - currently holding only one decorator to helo easily expand installation tests that use lmdb-* fixtures to run also against real storages

Opened Questions:

1. The tests that we have in this PR can be executed on the master branch too. A workflow should be added for that purpose. However, if added such workflow perhaps only LMDB tests should execute, and the reason is that AWS S3 and GCP are costing quite much in that frequent execution along with a problem that they might fail due to reaching quota. Thus on master real storage tests can and should be added for execution at merge time in a separate workflow so that we keep track of failing tests and reasons.
2. What to do when old versions have changes in API that prevent current tests to run? (From here on we can safely assume that we can checkout also the same version of sources - at least from the point this PR is merged)
3. The tests we run against arcticdb requires whole current test framework, fixtures and its dependencies - this is too muc and could be reduced to bare minimum, See this PR: Installation tests #2316

Any other comments?

Checklist

Checklist for code changes...

• Have you updated the relevant docstrings, documentation and copyright notice?
• Is this contribution tested against all ArcticDB's features?
• Do all exceptions introduced raise appropriate error messages?
• Are API changes highlighted in the PR description?
• Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

[poodlewars]

We'll want to run these using the Linux Conda build too, not just Mac

[grusev]

Yes I havethought about it too and would propose to add as one of the combination (currently we have 3 linuxes + pypi, we can make one of linuxes not pypi bu conda)

[poodlewars]

I think it will be clearer to have a dedicated test suite for these installability tests because they will have to keep working with possibly very old ArcticDB versions, so will in a sense be "frozen" - whereas the core tests will change more over time.

I also don't think we should add this complexity to the core lmdb_storage fixture - a dedicated single purpose fixture would be clearer to me.

I understand why you've done it like this, it is quite appealing to just reuse existing tests, and the way you've done it is really neat, I just think it is best to be painfully obvious here (especially for the comprehensibility for new joiners etc).

[grusev]

I have testsed so far with 4.4.7 version (lowest) and there were no problems. But I do understand what you mean. An approach with what you suggest was the original one - #2316

The idea there was to reuse as little as possible, and therefore only shared_tests.py was the thing that both current tests and new tests shared, and the idea was once a test started to diverge to be no more shared but have two versions - one for the original and one for the installation tests (it was ok since the tests files/suites were different )

[poodlewars]

Regarding the open questions in the description,

1. I expect we can get most of the value by running against simulators (moto, azurite) rather than real storage backends. Seems to be the "goldilocks zone"
2. I think this is an argument for a dedicated suite, that can cope with any API changes by testing the installed arcticdb version, rather than putting this in to the existing test suites
3. Sounds good

Finally, it will be important to have some brief docs explaining this setup (and why we have it) for the benefit of future developers.

[poodlewars]

I think an important part of this will be having a way to constrain the dependencies used by earlier ArcticDB versions - we don't want tests to fail on ArcticDB vPREHISTORIC when numpy 3 is released, for example

[poodlewars]

Do we think it's valuable to do this on PyPi (where we bundle all our C++ deps) or perhaps only doing this on Conda would suffice?

[maxim-morozov]

1. I don't know if it makes sense to reuse some of the storage tests for installation tests. I think it makes it a bit hard to make sure that we are running only the version of tests from the branch when the release was cut. I think the goal of the installation tests is to make sure that ArcticDB basic functionality is still working when it comes down to the underlying dependencies. It probably can be sufficient to create a basic set of tests like writing and reading from different storages that can be run against any ArcticDB.
2. This is exactly my point as in 1. With this proposal, we are basically using master tests to run against previous versions of ArcticDB. ArcticDB does not ensure forward compatibility, so it can easily be broken. I think the options are:
   a. Do pure installation tests with very minimal functionality that we expect is not going to be changing, like reading and writing. Potentially have an option to specify the set of versions these tests are supposed to work with.
   b. If we want to reuse our integration tests, we should check out tests from the same branch and version ArcticDB release was cut from.
3. Yes, this is why we need a separate test suite for installation tests. Also, installation tests should be quick and run multiple times a day as the underlying dependencies can change at any time.

[maxim-morozov]

It looks like we are planning to run those tests on branch push. I think it's much better to run them periodically. We rarely need to update older branches, but we still need to keep the older versions workable.

[maxim-morozov]

Why do we need to check out? Is it for checking out tests? If we rely on latest tests with older versions of arcticdb, it might not have the up to date functionality.