test: add integration tests for stretched and lam use cases #217

VeraChristina · 2025-03-25T13:31:33Z

Do Not Merge

The added tests take 3 mins each when run locally but very long on the github runners, likely related to how we fetch the data. Need to resolve this before merging.

Description

Add two integration tests for small training cycles using
- a stretched grid
- a local area model
Add tests for config validation based on the same configs
Update file names for the global use case for consistency -- i.e. using the name of the template used (config.yaml)

Code Compatibility

I have performed a self-review of my code

Code Performance and Testing

I ran the complete Pytest test suite locally, and they pass
I have run the new integration tests locally, and they pass

Documentation

My code follows the style guidelines of this project
I have updated the documentation and docstrings to reflect the changes
I have added comments to my code, particularly in hard-to-understand areas

📚 Documentation preview 📚: https://anemoi-training--217.org.readthedocs.build/en/217/

📚 Documentation preview 📚: https://anemoi-graphs--217.org.readthedocs.build/en/217/

📚 Documentation preview 📚: https://anemoi-models--217.org.readthedocs.build/en/217/

for more information, see https://pre-commit.ci

VeraChristina · 2025-03-25T14:14:49Z

Tagging @anaprietonem because we wanted to revisit the file naming discussed on the first integration tests PR

for more information, see https://pre-commit.ci

VeraChristina · 2025-03-27T14:53:13Z

marking as draft while adding the lam use case to this PR as well

theissenhelen

Happy with these changes!

anaprietonem · 2025-03-31T12:46:20Z

training/tests/integration/config/test_lam.yaml

+  dataset:
+    cutout:
+      - dataset: ${hardware.paths.data}/${hardware.files.dataset}
+        thinning: 25 # ???-value in lam template


Is that a general value ? @JPXKQX views on this?

anaprietonem · 2025-03-31T12:48:27Z

training/tests/integration/config/test_lam.yaml

+    data: https://object-store.os-api.cci1.ecmwf.int/ml-tests/test-data/samples/anemoi-integration-tests
+  files:
+    dataset: regional-use-cases/cerra-rr-an-oper-0001-mars-5p5km-2017-2017-6h-v2-hmsi-testing.zarr
+    forcing_dataset: regional-use-cases/aifs-ea-an-oper-0001-mars-o96-2017-2017-6h-v8-testing.zarr


For the forcing dataset, how long does this train currently takes? I believe for the global use case we were using a o48 dataset so wonder if that could make things faster

anaprietonem · 2025-03-31T12:54:03Z

training/docs/contributing.rst

-`training/tests/integration/test_training_cycle.yaml`. It furthermore
-applies use-case-specific modifications as detailed in
-`training/tests/integration/test_basic.yaml` to provide the location of
+`training/tests/integration/config/testing_modifications.yaml`. It


Regarding the naming:

the 'test_config.yaml' is the one related to the global deterministic model right? Could we more meaningful to call it test_global.yaml.

Related to this, in the future we would have uses cases related to probabilistic model or tests for the temporal interpolator. Any view on how to scale the naming conventions for those? Just wondering it could be good to do it already now with test_[coverage][model_task][forecaster_class]. Where coverage would be global/lam/stretched grid, model_task could be forecaster/interpolator and class would refer to single or probabilistic

anaprietonem

Great work both! I left some comments regarding views on scaling the naming convention and pinned Mario as he might has some views on the settings for LAM & stretched grid

MeraX · 2025-03-31T14:48:04Z

Thank you for this inspiring PR. I have to minor questions, posted individually, and I'm wondering whether this test is supposed to run on a CPU?

MeraX · 2025-03-31T13:06:02Z

training/tests/integration/conftest.py

            overrides=overrides,
        )  # apply architecture overrides to template since they override a default
-        use_case_modifications = OmegaConf.load(Path.cwd() / "training/tests/integration/config/test_basic.yaml")
+        use_case_modifications = OmegaConf.load(Path.cwd() / "training/tests/integration/config/test_config.yaml")


Is Path.cwd() relative to the working directory when executing pytest?

MeraX · 2025-03-31T14:02:04Z

training/tests/integration/conftest.py

-    temp_dir = str(tmp_path)
-    testing_modifications.hardware.paths.output = temp_dir
-    return testing_modifications
+def stretched_config(testing_modifications_with_temp_dir: OmegaConf) -> None:


Is the return type correct?

add test for stretched grid use case

4b284f3

github-actions bot added training labels Mar 25, 2025

pre-commit-ci bot and others added 2 commits March 25, 2025 13:31

[pre-commit.ci] auto fixes from pre-commit.com hooks

601dacb

for more information, see https://pre-commit.ci

update file names for consistency

f2e28c1

Merge branch 'main' into test/add-integration-test-stretched-use-case

4b737de

VeraChristina marked this pull request as ready for review March 27, 2025 11:24

VeraChristina requested review from theissenhelen and anaprietonem March 27, 2025 11:24

VeraChristina and others added 8 commits March 27, 2025 14:10

add lam use case

ebfa253

Merge branch 'main' into test/add-integration-test-stretched-use-case

d446e50

update docs wrt config file renaming

f3d4f8a

[pre-commit.ci] auto fixes from pre-commit.com hooks

79e2502

for more information, see https://pre-commit.ci

debug lam test

c9c1bad

add test for lam use case

093234c

clean up

c047e29

merge

85858a0

VeraChristina marked this pull request as draft March 27, 2025 14:49

VeraChristina changed the title ~~test: add integration test for stretched grid use case~~ test: add integration tests for stretched and lam use cases Mar 27, 2025

VeraChristina marked this pull request as ready for review March 27, 2025 14:59

theissenhelen previously approved these changes Mar 27, 2025

View reviewed changes

update test data dir name in ewc

2518e5b

VeraChristina dismissed theissenhelen’s stale review via 2518e5b March 27, 2025 15:37

Merge branch 'main' into test/add-integration-test-stretched-use-case

2a095b1

anaprietonem reviewed Mar 31, 2025

View reviewed changes

MeraX reviewed Mar 31, 2025

View reviewed changes

VeraChristina requested a review from dietervdb-meteo April 1, 2025 21:11

VeraChristina marked this pull request as draft April 4, 2025 09:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add integration tests for stretched and lam use cases #217

test: add integration tests for stretched and lam use cases #217

VeraChristina commented Mar 25, 2025 •

edited

Loading

VeraChristina commented Mar 25, 2025

VeraChristina commented Mar 27, 2025

theissenhelen left a comment

anaprietonem Mar 31, 2025 •

edited

Loading

anaprietonem Mar 31, 2025

anaprietonem Mar 31, 2025

anaprietonem left a comment

MeraX commented Mar 31, 2025

MeraX Mar 31, 2025

MeraX Mar 31, 2025

test: add integration tests for stretched and lam use cases #217

Are you sure you want to change the base?

test: add integration tests for stretched and lam use cases #217

Conversation

VeraChristina commented Mar 25, 2025 • edited Loading

Do Not Merge

Description

Code Compatibility

Code Performance and Testing

Documentation

VeraChristina commented Mar 25, 2025

VeraChristina commented Mar 27, 2025

theissenhelen left a comment

Choose a reason for hiding this comment

anaprietonem Mar 31, 2025 • edited Loading

Choose a reason for hiding this comment

anaprietonem Mar 31, 2025

Choose a reason for hiding this comment

anaprietonem Mar 31, 2025

Choose a reason for hiding this comment

anaprietonem left a comment

Choose a reason for hiding this comment

MeraX commented Mar 31, 2025

MeraX Mar 31, 2025

Choose a reason for hiding this comment

MeraX Mar 31, 2025

Choose a reason for hiding this comment

VeraChristina commented Mar 25, 2025 •

edited

Loading

anaprietonem Mar 31, 2025 •

edited

Loading