Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Moving nest test case #1237

Closed
danielabdi-noaa opened this issue Sep 12, 2023 · 26 comments
Closed

Moving nest test case #1237

danielabdi-noaa opened this issue Sep 12, 2023 · 26 comments

Comments

@danielabdi-noaa
Copy link

Hi,

I am trying to run an adaptive moving nest test case preferably one that tracks a tropical cyclone.
Is there a test case somewhere that I can use?

Also, I could not find a test case among the RegTests where amr.max_level is set above 0. I tried modifying
the DensityCurrent test case for this purpose to do static AMR but it fails complaining about division by 0.
Turning the FPE trap off does help a little bit but it fails after a few time steps. I do not know how to setup
an AMR test case and probably missed setting an AMR variable. I have attached the modified DensityCurrent problem.

DensityCurrent.txt

Could you please help me setup either a tropical cyclone test case, if possible, or the density current problem with dynamic AMR
capability -- e.g. one that is able to track the temperature field or some other criteria?

This is not an issue so please let me know if I should move the question elsewhere.

Thanks
Daniel
@asalmgren
Copy link
Collaborator

@danielabdi-noaa -- let me start by apologizing -- I really dropped the ball on this one. @AMLattanzi has been doing a lot of work on the data structures so let me ask him to pick up the ball and run with it (to complete my sports analogy!) I know he's run test cases with max_level > 0 but they may not be in the repo right now. In the meantime I'll try to get the DensityCurrent case working.

@asalmgren
Copy link
Collaborator

@danielabdi-noaa -- I just realized @AMLattanzi is on vacation this week, and I'm on travel as well. Let me look at that DC case though.

@danielabdi-noaa
Copy link
Author

@asalmgren No problem, it can wait until both of you are back to work. Thanks for looking into it!

@asalmgren
Copy link
Collaborator

@danielabdi-noaa Ok now I'm confused -- did you look at Exec/RegTests/DynamicRefinement? That has two inputs files with refinement -- one with max_level = 1 (inputs_onelevel) and one with max_level = 2 (inputs_twolevel). It appears to "just work" -- this is the one I set up for you a while back. I'm curious you didn't see this directory -- are you working with an old version of the code?

@danielabdi-noaa
Copy link
Author

@asalmgren Indeed it looks like I am working with an older clone that did not have the DynamicRefinement test case. The last commit I have is from Jul 20 , PR #1172 "Towards dynamic mesh refinement". I will update the branch and try to run that test case and the modified density current test case too. Thanks for the quick turnaround!

@asalmgren
Copy link
Collaborator

Unfortunately I didn't fix the DensityCurrent case yet but go ahead and play with DynamicRefinement and I'll let you know when the other cases are working with refinement...

@danielabdi-noaa
Copy link
Author

Ok thanks! It looks like my modified density current test case works with the newly built binaries! I can see the statically refined mesh at the center and the simulation runs for the whole 900 steps. However, the DynamicRefinement test case doesn't seem to work. First of all the test case is not part of Exec/CMakeLists.txt so doesn't build the init binary for it, nor does it install it. Manually doing that and running the test case fails with an error message but I am sure I have done something wrong. Anyway, I have the DensityCurrent test case to work with now! Thanks again

@danielabdi-noaa
Copy link
Author

Nevermind, DynamicRefinement works too. I just had to turn off amrex.fpe_trap_invalid = 0 just like I did for the DensityCurrent problem. The run produce a moving nest that tracks a diagonally moving bubble in a periodic boundary problem. All good now!

@asalmgren
Copy link
Collaborator

@danielabdi-noaa -- I'm very surprised you're seeing different behavior with amrex.fpe_trap_invalid set to 0 vs 1. Locally, DynamicRefinement runs fine with either option for me. What behavior do you see when it fails?

@danielabdi-noaa
Copy link
Author

I am getting erroneous operation error when it is set to 1

...
TIME= 0 MASS        = 330.4117604
TIME= 0 SCALAR      = 301.9301818
Based on cfl of 1.0 
Fast  dt at level 0 would be:  0.0001456388997
Slow  dt at level 0 would be:  0.0002380348181
Fixed dt at level 0       is:  1.5e-05
smallest even ratio is: 2
Based on cfl of 1.0 
Fast  dt at level 1 would be:  7.277866935e-05
Slow  dt at level 1 would be:  0.0001187135426
Fixed dt at level 1       is:  1.5e-05
smallest even ratio is: 2
Based on cfl of 1.0 
Fast  dt at level 2 would be:  1.84022468e-05
Slow  dt at level 2 would be:  2.989658757e-05
Fixed dt at level 2       is:  1.5e-05
smallest even ratio is: 2
Erroneous arithmetic operation
See Backtrace.0 file for details
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 8.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

This is actually better than what I got for the modified DensityCurrent problem where it displayed and undefined Slow dt value as shown below

Fast  dt at level 1 would be:  0.03603275208
Slow  dt at level 1 would be undefined 
Fixed dt at level 1       is:  1
Fixed fast dt at level 1       is:  0.25
smallest even ratio is: 4
Erroneous arithmetic operation
See Backtrace.0 file for details

The log in Backtrace.0 does not seem to be helpful, the last lines of which are

 7: ../install/bin/erf_dynamic_refinement(+0x50055) [0x55aee2d81055]
    ?? ??:0

 8: ../install/bin/erf_dynamic_refinement(+0x2fdee) [0x55aee2d60dee]
    ?? ??:0

 9: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fd0c7c0ad90]

10: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fd0c7c0ae40]

11: ../install/bin/erf_dynamic_refinement(+0x32845) [0x55aee2d63845]
    ?? ??:0

@asalmgren
Copy link
Collaborator

That comment about "slow dt would be undefined" is only telling us that velocity == 0 so dt / velocity would be undefined -- so that is correct and not an issue.

The error that I see with the DensityCurrent run is in the refluxing, which I'm in the process of fixing. Could you try setting erf.coupling_type = OneWay and see if that "just works"?

Also -- could you build the DynamicRefinement case in Debug mode? That will give a more detailed backtrace

@danielabdi-noaa
Copy link
Author

Here is the Backtrace file for DynamicRefinement test case after rebuilding binaries in Debug mode.
Backtrace.0.txt

For the DensityCurrent setting erf.coupling_type = OneWay does not seem to help. It still fails unles I turn off the FPE trap.

@asalmgren
Copy link
Collaborator

Ok I don't know what's going on with that flag -- I'll need to play around to see if I can get it to fail locally. What kind of machine are you running on? With/without mpi and if with, how many ranks?

@danielabdi-noaa
Copy link
Author 

System:
   AMD Ryzen 9 3900X 12-Core Processor
   Ubuntu 22.04.2 LTS
   Built with OpenMPI 4.1.2

I run the test case with 1 mpi rank like this

$ mpirun -n 1 ../build/Exec/RegTests/DynamicRefinement/erf_dynamic_refinement inputs_twolevel 

MPI initialized with 1 MPI processes
MPI initialized with thread support level 0
AMReX (23.07-23-g5a90c264fad8) initialized
Successfully read inputs file ... 

ERF git hash: JOSS_paper-117-gbebd76f93d29-dirty
AMReX git hash: 23.07-23-g5a90c264fad8

Using default dycore_horiz_adv_type
Using default dycore_vert_adv_type
Using default dryscal_horiz_adv_type
Using default dryscal_vert_adv_type
SOLVER CHOICE: 
no_substepping              : 0
force_stage1_single_substep : 1
incompressible              : 0
use_coriolis                : 0
use_rayleigh_damping        : 0
use_gravity                 : 0
ABL Driver Type: None
No ABL driver selected 
Advection Choices: 
dycore_horiz_adv_type       : Centered_2nd
dycore_vert_adv_type        : Centered_2nd
dryscal_horiz_adv_type      : Centered_2nd
dryscal_vert_adv_type       : Centered_2nd
Diffusion choices: 
rho0_trans                  : 1
alpha_T                     : 0
alpha_C                     : 0
dynamicViscosity            : 0
Not using any molecular diffusivity, i.e. using the modeled turbulent diffusivity
Sponge choices: 
Turbulence Settings at level 0
Using DNS model at level 0
Turbulence Settings at level 1
Using DNS model at level 1
Turbulence Settings at level 2
Using DNS model at level 2
  vortex initialized at (0, 0)
  reference pressure = 100000 Pa
  reference temperature = 300 K
  reference potential temperature (not used) = 300 K
  calculated freestream air density = 1.161440186 kg/m^3
  calculated speed of sound, a = 347.1887095 m/s
  freestream u/a = 0.8451542547
  freestream v/a = 0.8451542547
Refinement ratio at level 1 set to be 2 2 1
Refinement ratio at level 2 set to be 4 4 1
Initializing uniform fields rho=1.161440186 theta=300 -- this probably only makes sense with gravity turned off
Initializing uniform fields rho=1.161440186 theta=300 -- this probably only makes sense with gravity turned off
Initializing uniform fields rho=1.161440186 theta=300 -- this probably only makes sense with gravity turned off

TIME= 0 MASS        = 330.4117604
TIME= 0 SCALAR      = 301.9301818
Based on cfl of 1.0 
Fast  dt at level 0 would be:  0.0001456388997
Slow  dt at level 0 would be:  0.0002380348181
Fixed dt at level 0       is:  1.5e-05
smallest even ratio is: 2
Based on cfl of 1.0 
Fast  dt at level 1 would be:  7.277866935e-05
Slow  dt at level 1 would be:  0.0001187135426
Fixed dt at level 1       is:  1.5e-05
smallest even ratio is: 2
Based on cfl of 1.0 
Fast  dt at level 2 would be:  1.84022468e-05
Slow  dt at level 2 would be:  2.989658757e-05
Fixed dt at level 2       is:  1.5e-05
smallest even ratio is: 2
Erroneous arithmetic operation
See Backtrace.0 file for details
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 8.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

@asalmgren
Copy link
Collaborator

I'm really curious what's going on -- I also see that you and I get slightly different initial mass (I have 330.4109109 instead of your 330.4117604) -- which suggests that something is ever so slightly different in our initialization.

@danielabdi-noaa
Copy link
Author

danielabdi-noaa commented Sep 13, 2023 •
edited
Loading

@asalmgren It looks like when I turn off amrex.fpe_trap_invalid = 0 I get the exact same initial mass as yours in Release modes and the run completed, but with it set to 1 the initial mass is different and the run fails. However, for the Debug build, the initial mass is 330.4117604 whether I set the FPE trap to 0 or 1. Setting it to 0 and with the incorrect initial mass still runs fine in Debug mode. I will rebuild with Release mode to confirm this.

@danielabdi-noaa
Copy link
Author

I can not reproduce the 330.4109109 initial mass with the Release mode any more. Both Release/Debug modes now give initial mass 330.4117604 whether the FPE trap is set to 0 or 1, and both run successfully with FPE trap set to 0.

However, I did see an initial mass of 330.4109109 for a successful run I did in release mode earlier. Unfortunately I have lost that run now.

@danielabdi-noaa
Copy link
Author

Ooops this was all for nothing. The input_onelevel file gives an initial mass of 330.4109109 and the input_twolevel file gives 330.4117604.

@asalmgren
Copy link
Collaborator

asalmgren commented Sep 13, 2023 via email

@danielabdi-noaa
Copy link
Author

Both test cases fail if I don't set the variable to 0.

@asalmgren
Copy link
Collaborator

asalmgren commented Sep 13, 2023 via email

@asalmgren
Copy link
Collaborator

Success! In the sense that I can now reproduce the erroneous arithmetic error on a different machine -- will let you know when I've got a fix for you to try out

@asalmgren
Copy link
Collaborator

@danielabdi-noaa -- so the failure I got was because on the second machine I was using an old version of amrex in the submodule. Can you see what version of amrex you have in your repo? You may need to explicitly try "git submodule update" to get the most recent version. Then see if that fixes the issue?

@danielabdi-noaa
Copy link
Author

@asalmgren Thank you that solved the problem! I had 23.07 before and after the update now have 23.09. Both test cases for DynamicRefinement run fine now with amrex.fpe_trap_invalid = 1. Thanks a lot!

@asalmgren
Copy link
Collaborator

awesome! Shall we close this issue?

@danielabdi-noaa
Copy link
Author

danielabdi-noaa commented Sep 13, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@asalmgren @danielabdi-noaa