Add QC checks for ICE source files

[DeniseWorthen]

DESCRIPTION OF CHANGES:

Implements some QC checks on the source CICE6 restart files to fix possible issues known to cause run-time failures in the coupled model. A unit test is added for the QC test functionality.

TESTS CONDUCTED:

If there are changes to the build or source code, the tests below must be conducted. Contact a repository manager if you need assistance.

• Compile branch on all Tier 1 machines using Intel (Orion, Jet, Hera, Hercules and WCOSS2). Done using fed6cd9.
• Compile branch on Hera using GNU. Done using fed6cd9.
• Compile branch in 'Debug' mode on WCOSS2. Done using fed6cd9.
• Run unit tests locally on any Tier 1 machine. Done on Hera using fed6cd9. All tests, including the new ftst_qcice test passed.
• Run ocnice_prep consistency tests locally on all Tier 1 machines using new baseline data here. Done using fed6cd9.
• Check doxygen. Done on Hera using fed6cd9. No warnings associated with the ocnice_prep code.

Describe any additional tests performed.

1. Baselines were run on Hera. The downscaled ocean warmstarts are B4B, as expected. The 1/2 deg and 1 deg ice files fail comparison. The only differences were seen in fields which are QC'd (category ice coverage, volume and snow volume and surface temperature).

2. The initial, unfixed IC restart file used during the investigation of the negative dvice issue was QC'd using the ocnice-prep utility. Using the debug=.true. setting for the ocnice-prep, the QC'd fields were dumped to a netCDF file. The four fields QC'd (aicen, vicen, vsnon, Tsfcn) were identical to those generated via ncap2 commands previously identified as fixing the negative dvice issue. For completeness, the script using ncap2 is

#!/bin/bash

set -x

maskfile=/scratch1/NCEPDEV/global/glopara/fix/cice/20240416/025/kmtu_cice_NEMS_mx025.nc
src=cice_model.res.nc
dst=ncap2.fixes.nc

cp ${src} ${dst}
ncks -A ${maskfile} ${dst}
ncap2 -O -s 'where(kmt==0) aicen=0.0' ${dst} ${dst}
ncap2 -O -s 'where(kmt==0) vicen=0.0' ${dst} ${dst}
ncap2 -O -s 'where(kmt==0) vsnon=0.0' ${dst} ${dst}
ncap2 -O -s 'where(kmt==0) Tsfcn=0.0' ${dst} ${dst}
ncap2 -O -s 'where(aicen.total($ncat) == 0.0) vicen=0.0' ${dst} ${dst}
ncap2 -O -s 'where(aicen.total($ncat) == 0.0) vsnon=0.0' ${dst} ${dst}
ncks -O -x -v kmt ${dst} ${dst}

3. A similar test was done for the down-scaled product of the ocnice-prep utility. The generated file via QC was identical to using the ncap2 on a file generated w/o the use of QC.

ISSUE:

• fixes downscaling utility needs to check for phantom ice and ice-on-land #1019
• related to Model crash, negative dvice ufs-weather-model#2562

[GeorgeGayno-NOAA]

Do you want this merged before #1013?

[DeniseWorthen]

They're independent. Either can go first. We use the warmstart files for one of the UWM RTs; the runs which created the failures are using the source files at their native resolution, so are not "fixed" with the QC in this utility. They'll need to implement something at point-of-use for those native resolution files w/ issues.

[DeniseWorthen]

Baselines are located here:

hera: /scratch1/NCEPDEV/stmp4/Denise.Worthen/OCNICE_PREP/BASELINE
orion: /work/noaa/stmp/dworthen/OCNICE_PREP/BASELINE
hercules: /work2/noaa/stmp/dworthen/OCNICE_PREP/BASELINE/
jet: /lfs5/HFIP/h-nems/Denise.Worthen/OCNICE_PREP/BASELINE
wcoss2: /lfs/h2/emc/nems/noscrub/denise.worthen/OCNICE_PREP/BASELINE

[GeorgeGayno-NOAA]

Baselines are located here:

hera: /scratch1/NCEPDEV/stmp4/Denise.Worthen/OCNICE_PREP/BASELINE orion: /work/noaa/stmp/dworthen/OCNICE_PREP/BASELINE hercules: /work2/noaa/stmp/dworthen/OCNICE_PREP/BASELINE/ jet: /lfs5/HFIP/h-nems/Denise.Worthen/OCNICE_PREP/BASELINE wcoss2: /lfs/h2/emc/nems/noscrub/denise.worthen/OCNICE_PREP/BASELINE

Thanks. I will host them in the official location.

[GeorgeGayno-NOAA]

Looks good. Will merge, then update the baselines on all machines.