MPAS fcst crashed at test of the workflow

[HuiLiu-NOAA]

I just tested the Retro case provided by Guoqing. The 1st cycle finished fine. But, the fcst step at 03h failed as below:

Beginning MPAS-atmosphere Error Log File for task 0 of 80
Opened at 2025/01/24 19:33:31

ERROR: At least one fields to be read from the 'input' stream is dimensioned
ERROR: by 'nscat', but the 'nscat' dimension is not defined
ERROR: in the file mpasin.nc
CRITICAL ERROR: Please check the input file(s) to be read by the 'input' input stream.
Logging complete. Closing file at 2025/01/24 19:33:31

/scratch1/NCEPDEV/fv3-cam/Hui.Liu/RRFSv2/v2.0.8/conus12km/stmp/rrfs/v2.0.8/rrfs.20240527/03/rrfsdet_fcst_03

Suggestions are appreciated.

[guoqing-noaa]

@HuiLiu-NOAA, Thanks for creating the issue. I can confirm that I got a similar error. I think it may be due to the wrong MPAS version or namelist being used in the workflow for the warm-start runs.
@chunhuazhou Could you check whether the Jet real-time runs use the same MPAS version/namelist as that in the workflow?

A side request for @HuiLiu-NOAA: Could you change the title to "MPAS forecast crashed" and change my name from "Quoqing" to "Guoqing"? Thanks!

[HuiLiu-NOAA]

@HuiLiu-NOAA, Thanks for creating the issue. I can confirm that I got a similar error. I think it may be due to the wrong MPAS version or namelist being used in the workflow for the warm-start runs. @chunhuazhou Could you check whether the Jet real-time runs use the same MPAS version/namelist as that in the workflow?

A side request for @HuiLiu-NOAA: Could you change the title to "MPAS forecast crashed" and change my name from "Quoqing" to "Guoqing"? Thanks!

Thanks. fixed the typos.

[chunhuazhou]

@HuiLiu-NOAA Are you using the GSL verison of MPAS Model? We are using the GSL version of MPAS model for the real-time runs on Jet, but with a local change to the registry file (/mnt/lfs5/BMC/wrfruc/mhu/rrfs2/v2.0.7/rrfs-workflow/sorc/MPAS-Model/src/core_atmosphere/Registry.xml). The default registry are missing some lsm fields and this updated registry file works.

[guoqing-noaa]

Thanks, @chunhuazhou for the information!

I will create a PR to fix this soon.

[chunhuazhou]

I am not sure what is the latest registry.xml like. It might have more fields than the one from @hu5970 so I am posting the main changes from Ming, for reference.

$ git diff
diff --git a/src/core_atmosphere/Registry.xml b/src/core_atmosphere/Registry.xml
index df27e9c9..38317e43 100644
--- a/src/core_atmosphere/Registry.xml
+++ b/src/core_atmosphere/Registry.xml
@@ -1307,10 +1307,6 @@
                        <var_array name="scalars"/>
                        <var name="initial_time"/>
                        <var name="xtime"/>
-                       <var name="cldfrac"/>
-                       <var name="re_cloud" packages="mp_thompson_in;mp_wsm6_in;mp_tempo_in"/>
-                       <var name="re_ice" packages="mp_thompson_in;mp_wsm6_in;mp_tempo_in"/>
-                       <var name="re_snow" packages="mp_thompson_in;mp_wsm6_in;mp_tempo_in"/>
                        <var name="u"/>
                        <var name="w"/>
                        <var name="rho"/>
@@ -1324,6 +1320,20 @@
                        <var name="uReconstructMeridional"/>
                        <var name="surface_pressure"/>
 #ifdef DO_PHYSICS
+                       <var name="isltyp"/>
+                       <var name="soilf"/>
+                       <var name="ivgtyp"/>
+                       <var name="landusef"/>
+                       <var name="mminlu"/>
+                       <var name="isice_lu"/>
+                       <var name="iswater_lu"/>
+                       <var name="landmask"/>
+                       <var name="shdmin"/>
+                       <var name="shdmax"/>
+                       <var name="snoalb"/>
+                       <var name="albedo12m"/>
+                       <var name="greenfrac"/>
+                       <var name="lai12m"/>
                        <var name="cldfrac"/>
                        <var name="re_cloud" packages="mp_thompson_in;mp_wsm6_in"/>
                        <var name="re_ice" packages="mp_thompson_in;mp_wsm6_in"/>

[HuiLiu-NOAA]

Thanks @guoqing-noaa and @chunhuazhou! The test did run through more cycles. But got an error at cycle 2806z:

Beginning MPAS-atmosphere Error Log File for task 0 of 80
Opened at 2025/01/29 06:01:17

CRITICAL ERROR: NaN detected in 'w' field.
Logging complete. Closing file at 2025/01/29 06:01:17

The details are in /scratch1/NCEPDEV/fv3-cam/Hui.Liu/RRFSv2/conus12km/stmp/rrfs/v2.0.8/rrfs.20240528/06/rrfsztd_fcst_06.

[guoqing-noaa]

@HuiLiu-NOAA
Thanks for reporting this error.
I have not run that far yet. It looks like a Model problem.

Could you check whether 0528_00z successfully made forecasts to 0528_12z?

[HuiLiu-NOAA]

@HuiLiu-NOAA Thanks for reporting this error. I have not run that far yet. It looks like a Model problem.

Could you check whether 0528_00z successfully made forecasts to 0528_12z?

Yes, that forecast was successful.

[guoqing-noaa]

@HuiLiu-NOAA
Could you make a standalone case for this model crash at 0528_06z?
Then we can submit this to our model team for them to debug.
Thanks!