cam6_4_091: Implement support for moving mountain gravity wave scheme in MPAS dycore

[kuanchihwang]

This PR implements support for moving mountain gravity wave scheme in MPAS dycore.

The use_gw_movmtn_pbl namelist option will now default to .true. when MPAS dycore and CAM7 physics are both selected, which matches the behavior as in SE dycore.

The moving mountain gravity wave scheme needs relative vorticities at cell points as input. However, because MPAS uses staggered C-grid for spatial discretization, where wind vectors are located at edge points, it calculates relative vorticities at vertex points instead. As a result, this PR introduces a new functionality in MPAS subdriver to regrid vertex values to cell values. The regridding functionality is also generalized so that it will work with all variables at vertex points.

Subsequently, relative vorticities are passed to physics buffer during dynamics-physics coupling, after which the moving mountain gravity wave scheme can query and use them as input.

Closes #1253
Closes #1277

Additional notes about tests:

• This PR is expected to be answer-changing for MPAS dycore with CAM7 physics, but all relevant regression tests still pass because they do not test this particular configuration.
• I manually ran the "F2000dev" compset with MPAS dycore. I inspected and plotted the "VORT4GW" variable in physics buffer. I can confirm that relative vorticities are correctly passed to physics by visual inspection.
• MPAS dycore has variables that are located at both cell and vertex points (e.g., absolute vorticities). I used the regridding functionality in this PR to calculate absolute vorticities at cell points, and compared the results with the values calculated internally by MPAS. The global maximum absolute error is in the order of $10^{-19}$, which shows that the implementation is provably correct.

[kuanchihwang]

I do not have the permission to either assign myself or request reviews. Tagging @PeterHjortLauritzen and @mgduda to take a look at this PR to ensure scientific correctness.

[mgduda]

I do not have the permission to either assign myself or request reviews. Tagging @PeterHjortLauritzen and @mgduda to take a look at this PR to ensure scientific correctness.

Thanks for the tag! I've just added myself and @PeterHjortLauritzen as reviewers.

[PeterHjortLauritzen]

Thank you for implementing this!

As a sanity check I ran an SE CAM7 regression test:

./create_test --output-root /glade/derecho/scratch/pel/ --project P93300042 ERP_D_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_intel.cam-outfrq9s

And "hacked" an MPAS CAM7 regression test:

./create_test --output-root /glade/derecho/scratch/pel/ --project P93300042 ERS_Ln9_P288x1.mpasa120_mpasa120.F2000climo.derecho_intel.cam-outfrq9s_mpasa120

with

./xmlchange CAM_CONFIG_OPTS="-phys CAM7"

and I had to change dust scheme due to an error message:

dust_emis_method = 'Zender_2003'

Note that this run uses L32. I ran the MPAS test for 5 days to avoid any spin-up issues.

Here is VORT4GW for time-step 9 for SE (left) comparred to day 5 for MPAS (right) using same countour interval [-0.0003,0.0003] in the surface level:

Same as above but for top level and top level minus 1, respectively:

The latter two plots seem to show sponge layer damping differences.

In all I think the VORT4GW is scientifically correct. PR approved!

[PeterHjortLauritzen]

Looks great ... thank you!

We really need a CAM7 (L58) low top regression test for MPAS. Something like this one:

ERP_D_Ln9.ne30pg3_ne30pg3_mt232.FHISTC_LTso.derecho_intel.cam-outfrq9s

Would you be willing to add that to this PR? (Jesse and/or Cheryl can help you out if you have questions)

[nusbaume]

Thanks @kuanchihwang! I just had one ChangeLog update reminder, but otherwise the PR looks good to me!

[cacraigucar]

This is the reminder (more to me than to you) that a regression test needs to be added or an existing one needs to be modified to test this new feature.

[kuanchihwang]

@cacraigucar I pushed 9fd166e to add a new test for CAM7 low top with MPAS dynamical core. It is an ERS (Exact Restart from Startup) test with the "FHISTC_LTso" compset ^[1].

The test is quite heavy to run, requiring ~15 minutes of wall time on 128 CPU cores and ~110 GB of memory, so I did not configure it on Izumi.

Let me know if you have other review comments!

[cacraigucar]

@cacraigucar I pushed 9fd166e to add a new test for CAM7 low top with MPAS dynamical core. It is an ERS (Exact Restart from Startup) test with the "FHISTC_LTso" compset [1].

The test is quite heavy to run, requiring ~15 minutes of wall time on 128 CPU cores and ~110 GB of memory, so I did not configure it on Izumi.

Let me know if you have other review comments!

@kuanchihwang - We typically try to run most of our testing at the coarsest grids and only have a very few at the "realistic" grids. Could this test use your mpasa480_mpasa480 grid instead?

[kuanchihwang]

@cacraigucar CTSM is active in the "FHISTC_LTso" compset. Unfortunately, CTSM does not provide the "flanduse_timeseries" input data ^[1] for MPAS on the "mpasa480_mpasa480" grid.

Attempting to configure such test will result in an error that shows "No default value found for flanduse_timeseries. Are defaults provided for this resolution and land mask?".

[cacraigucar]

@cacraigucar CTSM is active in the "FHISTC_LTso" compset. Unfortunately, CTSM does not provide the "flanduse_timeseries" input data [1] for MPAS on the "mpasa480_mpasa480" grid.

Attempting to configure such test will result in an error that shows "No default value found for flanduse_timeseries. Are defaults provided for this resolution and land mask?".

@ekluzek and/or @adamrher - I believe we need to have a CTSM supporting dataset for the mpasa480_mpasa480 grid. Is it possible for us to get this?

[ekluzek]

@cacraigucar we didn't provide support for a landuse.timeseries file for mpas480 as it isn't in the list in this file:

https://docs.google.com/spreadsheets/d/1Osq56e423CF107zhoNQ0VS7-iH_JXLF9AtCvBdXyfJ4

But, if needed we can add it.

For your testing you might just add the following to user_nl_clm for your tests:

flanduse_timeseries = ' '

[adamrher]

@ekluzek does setting flanduse_timeseries to ' ' tell the model to grab all the surface properties from the fsurdat file?

If so, I think it's fine to do this for testing purposes.

[ekluzek]

@ekluzek does setting flanduse_timeseries to ' ' tell the model to grab all the surface properties from the fsurdat file?

If so, I think it's fine to do this for testing purposes.

Yes, exactly. It'll just keep surface properties fixed at whatever the fsurdat file has. For testing you probably aren't testing the whole historical period for example. And it just means that if you do run the whole historical period the surface will stay with 1850 conditions. And that's not likely to be a problem for the things you care about in testing CAM.

[kuanchihwang]

@cacraigucar

1. As discussed above, I pushed 729ca63 to work around missing input data for CTSM on mpasa480 grid.
2. I pushed bd8f971 to change the test to run on mpasa480 grid instead. As a result, the test requirements have reduced to ~5 minutes of wall time on 128 CPU cores and ~55 GB of memory, which we can now afford to run on Izumi as well.