Centralised readme (#52)

Co-authored-by: Aaron Hopkinson <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Vera <[email protected]>

Author: 4 people

.github/actions/deploy/action.yaml

@@ -43,7 +43,7 @@ runs:
           module load wellies
 
           # Clone the repository
-          PACKAGE_NAME=anemoi-docs
+          PACKAGE_NAME=anemoi
           PACKAGE_BRANCH=${{ github.head_ref || github.ref_name }}
           git clone -b $PACKAGE_BRANCH https://${{ inputs.github_token }}@github.com/ecmwf/${PACKAGE_NAME}.git
           cd $PACKAGE_NAME

.github/workflows/nightly_ci_hpc.yaml

@@ -20,7 +20,7 @@ jobs:
       - name: Check for recent commits on main in anemoi repos
         id: check-commits
         run: |
-          REPOS=("anemoi-core" "anemoi-datasets" "anemoi-docs")
+          REPOS=("anemoi-core" "anemoi-datasets" "anemoi")
           SHOULD_RUN=false
 
           for REPO in "${REPOS[@]}"; do

.github/workflows/on_demand_ci_hpc.yaml

@@ -3,8 +3,8 @@ name: on-demand-system-level-test
 on:
   workflow_dispatch:
     inputs:
-      anemoi_docs_branch:
-        description: Branch for anemoi-docs to be used to deploy the anemoi_tests suite.
+      anemoi_branch:
+        description: Branch for the anemoi repo to be used to deploy the anemoi_tests suite.
         required: false
       anemoi_datasets_branch:
         description: Branch for anemoi-datasets to be used to build the datasets environment for testing.
@@ -34,16 +34,16 @@ jobs:
           build_options=""
 
           if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
-            docs_branch="${{ github.event.inputs.anemoi_docs_branch }}"
+            anemoi_branch="${{ github.event.inputs.anemoi_branch }}"
             datasets_branch="${{ github.event.inputs.anemoi_datasets_branch }}"
             training_branch="${{ github.event.inputs.anemoi_training_branch }}"
 
             # Default to ref name if docs branch not specified
             if [[ -z "$docs_branch" ]]; then
-              docs_branch="${{ github.ref_name }}"
+              anemoi_branch="${{ github.ref_name }}"
             fi
 
-            build_options="anemoi_docs_branch=$docs_branch anemoi_datasets_branch=$datasets_branch anemoi_training_branch=$training_branch"
+            build_options="anemoi_branch=$anemoi_branch anemoi_datasets_branch=$datasets_branch anemoi_training_branch=$training_branch"
           fi
 
           echo "build_options=$build_options" >> "$GITHUB_OUTPUT"

README.md

@@ -1,20 +1,117 @@
-# anemoi-docs
+Anemoi Framework
+================
 
-This repo builds the main [documentation](https://anemoi.readthedocs.io/) for the Anemoi project.
+[**Overview**](#overview) | [**Features**](#features) | [**Installation**](#installation) | [**Contributing**](#contributing) | [**Roadmap**](#roadmap) | [**Community**](#community) | [**License**](#license) |  [**Other resources & References**](#other-resources--references)
 
-To build it yourself clone this repo and pip install:
 
-```bash
-git clone [email protected]:ecmwf/anemoi-docs.git
-cd anemoi-docs
-python -m venv .venv
-source .venv/bin/activate
-pip install -e .
-```
+Anemoi is an open-source, Python-based framework developed collaboratively by ECMWF and several European national meteorological services.  It is designed to facilitate the development, training, and deployment of machine learning (ML) models for weather forecasting. As an 'end to end' framework, it provides a comprehensive toolkit that spans data preparation, model training, and inference, enabling meteorological organizations to leverage their own data for ML-based weather prediction.
 
-Then, build the documentation:
+![Anemoi Logo](docs/_static/logo.png)
 
-```bash
-cd docs
-make html
-```
+- [Anemoi - A new framework for Weather Forecasting based on Machine Learning](https://www.ecmwf.int/en/about/media-centre/news/2024/anemoi-new-framework-weather-forecasting-based-machine-learning)
+
+
+Overview
+--------
+The `anemoi` repository aims, for now, to provide a centralized overview of the entire framework.
+Additionally, it contains the [Anemoi Documentation](https://anemoi.readthedocs.io/en/latest/) and system-level tests that assess the interfaces and ensure all components work well with each other.
+
+Anemoi offers a modular architecture composed of several packages, each targeting different components necessary for constructing data-driven weather models:
+
+- **[anemoi-datasets](https://github.com/ecmwf/anemoi-datasets)**: Tools for handling and preprocessing meteorological datasets.
+- **[anemoi-graphs](https://github.com/ecmwf/anemoi-core/tree/main/graphs)**: Utilities for constructing graph representations of meteorological data.
+- **[anemoi-models](https://github.com/ecmwf/anemoi-core/tree/main/models)**: Core components for defining and training ML models, including graph neural networks (GNNs) and graph transformers.
+- **[anemoi-training](https://github.com/ecmwf/anemoi-core/tree/main/training)**: Framework for training ML models, including support for distributed training.
+- **[anemoi-inference](https://github.com/ecmwf/anemoi-inference)**: Tools for deploying trained models and performing inference.
+- **[anemoi-transform](https://github.com/ecmwf/anemoi-transform)**: Utilities for data transformation and augmentation.
+- **[anemoi-utils](https://github.com/ecmwf/anemoi-utils)**: Miscellaneous utility functions.
+- **[anemoi-registry](https://github.com/ecmwf/anemoi-registry)**: Centralized registry for storing and accessing model configurations and metadata.
+
+Each package collects metadata that can be used by subsequent packages, facilitating a seamless workflow from data ingestion to operational model deployment.
+
+The diagram below shows the relationship of these packages in terms of dependencies.\
+![Anemoi Dependencies](docs/_static/dependencies.png)
+
+The diagram below represents the main packages in the Anemoi framework and the general workflow for using them:
+
+![Anemoi Workflow](docs/_static/anemoi_packages_workflow.png)
+
+Summary of workflow:
+- Start by building a dataset with `anemoi-datasets`.
+- Train a model using `anemoi-core` (central yellow boxes inside a black rectangle): This repository groups three distinct but related packages to support model development:
+    - `anemoi-modes`: Contains definitions for different machine learning models.
+    - `anemoi-training`: Provides the training framework and utilities for fitting models on prepared datasets.
+    - `anemoi-graphs`: Supports graph-based representations for modeling meteorological data.
+These three packages are separated to make development easier, but they all live under the [anemoi-core GitHub repository](https://github.com/ecmwf/anemoi-core).
+- Perform predictions with `anemoi-inference`.
+- Any domain-specific transformations and code that can be shared across datasets, training, and inference is handled by `anemoi-transform`
+- Helper functions and domain-agnostic general purpose utilities are included in `anemoi-utils`.
+
+This structure separates concerns clearly, promotes reusability, and makes development modular while keeping core components organized under `anemoi-core`.
+
+Each package collects metadata that can be used by subsequent packages, facilitating a seamless workflow from data ingestion to operational model deployment.
+
+
+Features
+--------
+
+- **Modular Design**: Easily extendable components for various stages of the ML pipeline.
+- **Customizable Configurations**: Flexible configuration system using Hydra for model and training parameters.
+- **Distributed Training Support**: Leverage PyTorch Lightning for scalable and efficient training.
+- **Interoperability**: Built on top of established Python tools including PyTorch, Lightning, Hydra, Zarr, Xarray, and Earthkit.
+- **Open-Source Collaboration**: Developed in collaboration with multiple European meteorological services.
+
+Installation
+------------
+
+To install the Anemoi framework, clone the repository and install the required dependencies (e.g. for anemoi-transform) :
+
+    $ git clone https://github.com/ecmwf/anemoi-transform.git
+$ cd anemoi-transform
+    $ pip install -e
+
+Or one can also install each of the packages directly from PyPI
+
+    $ pip install anemoi-transform
+
+
+For detailed installation instructions and environment setup, refer to the [Installation Guide](https://anemoi.readthedocs.io/en/latest/installation/).
+
+Contributing
+------------
+
+## Contributing
+
+We welcome contributions to the Anemoi framework! If you’d like to get involved, please follow the [Contributing Guide](https://anemoi.readthedocs.io/projects/training/en/stable/dev/contributing.html#), which outlines the full development workflow — from setting up your environment, to branching and committing, to opening pull requests and going through review. We encourage opening issues as way to discuss any ideas or new features as a first-step!
+
+If you’re new to Anemoi, we recommend starting with the [Getting Started Tour](https://anemoi.readthedocs.io/en/latest/getting-started/tour.html) for a quick overview of the framework before diving into development.
+
+
+## Roadmap
+
+The development of the Anemoi framework follows a structured roadmap to ensure continuous improvement and feature development.
+For a detailed view of planned milestones, priorities, and upcoming features, see the [Development Roadmap](https://anemoi.readthedocs.io/en/latest/contributing/roadmap.html#development-roadmap).
+
+
+## Community
+
+The Anemoi framework fosters an open and collaborative community. Governance is guided by clear principles to ensure transparency and inclusiveness, described in the [Governance Documentation](https://anemoi.readthedocs.io/en/latest/contributing/governance.html).
+
+
+Community members are encouraged to participate in regular interactions such as the [Anemoi Community Meetings](https://anemoi.readthedocs.io/en/latest/contributing/roadmap.html#anemoi-community-meeting), where updates, discussions take place.
+
+## License
+
+Anemoi is licensed under the Apache License 2.0.
+See the [LICENSE](https://github.com/ecmwf/anemoi-core/blob/main/LICENSE) file for more details.
+
+For more information, visit the [Anemoi Documentation](https://anemoi.readthedocs.io/en/latest/).
+
+## Other resources & References
+
+Additional resources for using and understanding Anemoi:
+
+- Documentation for each package within the framework: [Anemoi Packages Docs](https://anemoi.readthedocs.io/en/latest/)
+- Anemoi webinars: [Anemoi Webinars](https://events.ecmwf.int/event/446/)
+- `anemoi-configs` package for example configurations: [Anemoi Configs](https://github.com/ecmwf/anemoi-configs)
+- Citation information: [How to Cite Anemoi](https://anemoi.readthedocs.io/en/latest/index.html#how-to-cite-anemoi)

docs/_static/anemoi_packages_workflow.png

@@ -24,7 +24,7 @@ Steps for contributing documentation:
    creating an issue and submitting a Pull Request.
 
 -  Identify the most appropriate place for your addition (e.g., overall
-   `Anemoi documentation <https://github.com/ecmwf/anemoi-docs>`_,
+   `Anemoi documentation <https://github.com/ecmwf/anemoi>`_,
    package-level documentation), and the relevant section (e.g., user
    guide, getting started, or developer guide).
 

docs/contributing/documentation.rst

@@ -40,10 +40,10 @@ Anemoi is an open-source machine learning framework co-developed by a
 number of national meteorological institutions in collaboration with
 ECMWF. Details about individual contributors can be found in the
 `CONTRIBUTORS.md
-<https://github.com/ecmwf/anemoi-docs/blob/main/CONTRIBUTORS.md>`_ file
-in each package. Inspired by the governance structures of other
-open-source projects such as PyTorch and TensorFlow, **Anemoi is guided
-by the Anemoi Technical Subgroup (ATS)**, a committee composed of
+<https://github.com/ecmwf/anemoi/blob/main/CONTRIBUTORS.md>`_ file in
+each package. Inspired by the governance structures of other open-source
+projects such as PyTorch and TensorFlow, **Anemoi is guided by the
+Anemoi Technical Subgroup (ATS)**, a committee composed of
 representatives from each contributing institution. The ATS meets weekly
 in what is known as the ATS meeting, which serves as a forum to
 coordinate the project's technical direction. During these meetings, we

docs/contributing/governance.rst

@@ -38,8 +38,8 @@ System-level Tests
 -  Test the anemoi packages as a whole, including end-to-end workflows,
    from dataset creation to model training and inference.
 -  These tests ensure that the packages work together as expected.
--  The system-level test suite is located in the `anemoi-docs`
-   repository in the `tests/system-level` directory.
+-  The system-level test suite is located in the `anemoi` repository in
+   the `tests/system-level` directory.
 
 ***************
  Running Tests
@@ -100,8 +100,8 @@ docs for more information.
 
    <br><br>
 
-To run **system-level tests**, navigate to the `anemoi-docs` repository
-on github and trigger the workflow `on-demand-system-level-test` via the
+To run **system-level tests**, navigate to the `anemoi` repository on
+github and trigger the workflow `on-demand-system-level-test` via the
 GitHub Actions tab.
 
 .. note::
@@ -272,8 +272,8 @@ package-level documentation.
 ***************************************************
 
 To add a test case in the system-level test suite, you need to add
-config files in the relevant directory in the `anemoi-docs` repository.
-The config files should be placed in the `tests/system-level/configs`
+config files in the relevant directory in the `anemoi` repository. The
+config files should be placed in the `tests/system-level/configs`
 directory as explained below. They will constitute tasks in the
 system-level test suite. No pyflow knowledge is required to add new test
 cases.
@@ -297,7 +297,7 @@ In the folder add
    the `dataset_config.yaml` file) will be set in the suite.
 
 If you need additional flexibility in configuring your test case, please
-open an issue in the `anemoi-docs` repository.
+open an issue in the `anemoi` repository.
 
 Model Training Test cases
 =========================

docs/contributing/testing.rst

@@ -44,4 +44,4 @@ e.g. building a new dataset to train a model using new data. See the
 We would love to make Anemoi even easier for everyone to use. If there
 are any use-cases you would like to see covered in the future, please
 consider contacting the developers or creating a pull request to the
-`anemoi-docs repository <https://github.com/ecmwf/anemoi-docs>`_.
+`anemoi` repository <https://github.com/ecmwf/anemoi>`_.

docs/getting-started/tour.rst

@@ -24,14 +24,14 @@ This directory is used to store the suite's outputs and to build the virtual env
 
 ### Testing Configuration Changes
 
-If you modify configuration files in `anemoi_test/configs`, you need to point to a committed branch of `anemoi-docs` so that the deployed suite can pull your config files from that branch.
+If you modify configuration files in `anemoi_test/configs`, you need to point to a committed branch of the `anemoi` repo so that the deployed suite can pull your config files from that branch.
 
 - Commit your changes to a branch.
-- Push the branch to anemoi-docs.
+- Push the branch to anemoi.
 - Run the build script pointing to that branch:
 
 ```
-./build.sh -s OUTPUT_ROOT_SUITE=$SCRATCH/workdir anemoi_docs_branch=name-of-your-branch
+./build.sh -s OUTPUT_ROOT_SUITE=$SCRATCH/workdir anemoi_branch=name-of-your-branch
 ```
 
 ### Additional Build Options