Skip to content

Adding new observation types for the RDASApp ctests

Samuel Degelia edited this page Oct 17, 2024 · 1 revision

RDASApp uses a method for merging individual observation-space yamls into one "super yaml" for its ctests. If you have validated a new observation type and wish to add it to the RDASApp ctests, you will need to make a few changes in addition to adding the yaml for the new observation type. Please see the detailed list below for additional developments that need to be made during your PR.

1. Stage the data under RDAS_DATA

This step can be done by contacting someone on the fix file management team. You can also include the path to this data in your PR so that we can update the staged data on each machine.

2. Add the new observation-space yaml file under rrfs-test/validated_yamls/templates/obtype_config

Please just include the lines of the yaml corresponding to the new observation you are adding (i.e., start the yaml with the line - obs space:). Also note that additional sections need to be included in the yaml such that the yaml can be used for both Var-based DA or EnKF. These include the distribution and obs localizations headers. Please see the example yamls already included in the rrfs-test/validated_yamls/templates/obtype_config directory.

Additionally, please do not include the input obsfile name or distribution name directly in the yaml. Instead, you will need to pass in strings that can be replaced when merging the yamls. See below for an example for mesonet yamls:

     - obs space:
         name: msonet
         distribution:
           name: "@DISTRIBUTION@"
           halo size: 100e3
         obsdatain:
           engine:
             type: H5File
             obsfile: "@OBSFILE@"

3. Add the new observation type to the merging script in rrfs-test/validated_yamls/gen_yaml_ctest.sh and run the script

This script controls the observation-space yamls that are merged together for the ctests. First, you need to add any new yamls to the lists with the names *_obtype_configs. Separate lists are created for each set of yamls that will use a different observation file. For example, if you are adding a new yaml that uses the same observation file as msonet_uv_288.yaml, then you can add the new yaml under msonet_obtype_configs. But if you are adding a yaml that uses a new observation type, you will need to create a new list along the lines of ${OBNAME}_obtype_configs.

If you are creating a new list, you will also need to add a new line that calls the process_obtypes function for that observation. This function replaces the @OBSFILE@ string described in the above step with the observation file name. Please see the existing calls to process_obtypes for examples of what to add.

After adding the new lines, please run this script to generate the new "super yamls" inside rrfs-test/testinput. This mega yaml should also be included in your PR.

4. Run the ctest and generate new test reference data

Once your "super yaml" has been generated, you are ready to rerun the ctests. We expect the ctest to fail initially due to the output not matching the previous version of the ctest reference output in RDASApp. The ctest will generate new test output that will become the updated test reference data.

Navigate to the individual ctest run directories (e.g., build/rrfs-test/rundir-rrfs_mpasjedi_2024052700_Ens3Dvar) and locate the rrfs-mpasjedi-*.out file. You will then need to rename this file to rrfs-mpasjedi-*.ref and copy it into build/rrfs-test/testoutput.