AMR with Vorticity, amrex::out of bound

[Adiram97]

Hello,

I am trying to use ERF for the purpose of Adaptive Mesh Refinement using AMReX submodules. Specifically, I am trying to do atmospheric simulations for the case of Convective boundary layer.

I am trying to regrid based on Vorticity. I have attached the screenshot and as you can see, it gives an error saying the accessed array is out of bounds. I read through the backtrace and found the function derived::dervortz which is trying to compute the vorticity through the velocity gradients with a centered scheme. I see that the array on the LHS has bounds (-1, 0..n-1, n) and therefore it is trying to compute vorticity on the point -1. But this means, it has to use the velocity on the point -2 based on the centered algorithm. Then if I try to run a parallel simulation, this issue further propagates to the ghost nodes of every block of the parallelization. I compiled the executable with cmake_with_fft.sh

Backtrace message:

0: ./erf_abl(+0xe5fef2) [0x59bdef032ef2]
amrex::BLBackTrace::print_backtrace_info(_IO_FILE*) at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_BLBackTrace.cpp:203

1: ./erf_abl(+0xe5f8c6) [0x59bdef0328c6]
amrex::BLBackTrace::handler(int) at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_BLBackTrace.cpp:101

2: ./erf_abl(+0xe1e6c9) [0x59bdeeff16c9]
amrex::ParallelDescriptor::Abort(int, bool) at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_ParallelDescriptor.cpp:222

3: ./erf_abl(+0xd8fea7) [0x59bdeef62ea7]
amrex::Error_host(char const*, char const*) at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX.cpp:261

4: ./erf_abl(+0xd8fdd0) [0x59bdeef62dd0]
amrex::Abort(char const*) at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX.H:176
(inlined by) amrex::Abort(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX.cpp:232

5: ./erf_abl(+0x66695) [0x59bdee239695]
amrex::Array4::index_assert(int, int, int, int) const at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Array4.H:279 (discriminator 53)

6: ./erf_abl(+0xffda2) [0x59bdee2d2da2]
double const& amrex::Array4::operator()<double const, 0>(int, int, int, int) const at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Array4.H:144
(inlined by) operator() at /home/adi/ERF/Source/ERF_Derive.cpp:267

7: ./erf_abl(+0x106f12) [0x59bdee2d9f12]
void amrex::ParallelFor<derived::erf_dervorty(amrex::BoxND<3> const&, amrex::FArrayBox&, int, int, amrex::FArrayBox const&, amrex::Geometry const&, double, int const*, int)::{lambda(int, int, int)#1}, 3>(amrex::BoxND<3> const&, derived::erf_dervorty(amrex::BoxND<3> const&, amrex::FArrayBox&, int, int, amrex::FArrayBox const&, amrex::Geometry const&, double, int const*, int)::{lambda(int, int, int)#1} const&) at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_GpuLaunchFunctsC.H:59
(inlined by) call_f_intvect_handler<derived::erf_dervorty(const Box&, amrex::FArrayBox&, int, int, const amrex::FArrayBox&, const amrex::Geometry&, amrex::Real, int const*, int)::<lambda(int, int, int)>, 3> at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_GpuLaunchFunctsC.H:78
(inlined by) ParallelFor_impND<3, derived::erf_dervorty(const Box&, amrex::FArrayBox&, int, int, const amrex::FArrayBox&, const amrex::Geometry&, amrex::Real, int const*, int)::<lambda(int, int, int)>, 3> at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_GpuLaunchFunctsC.H:273
(inlined by) ParallelFor<derived::erf_dervorty(const Box&, amrex::FArrayBox&, int, int, const amrex::FArrayBox&, const amrex::Geometry&, amrex::Real, int const*, int)::<lambda(int, int, int)>, 3> at /home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_GpuLaunchFunctsC.H:291

8: ./erf_abl(+0x1008cd) [0x59bdee2d38cd]
derived::erf_dervorty(amrex::BoxND<3> const&, amrex::FArrayBox&, int, int, amrex::FArrayBox const&, amrex::Geometry const&, double, int const*, int) at /home/adi/ERF/Source/ERF_Derive.cpp:274

9: ./erf_abl(+0x21f8d1) [0x59bdee3f28d1]
ERF::ErrorEst(int, amrex::TagBoxArray&, double, int) at /home/adi/ERF/Source/ERF_Tagging.cpp:71

10: ./erf_abl(+0x1009845) [0x59bdef1dc845]
amrex::AmrMesh::MakeNewGrids(int, double, int&, amrex::Vector<amrex::BoxArray, std::allocatoramrex::BoxArray >&) at /home/adi/ERF/Submodules/AMReX/Src/AmrCore/AMReX_AmrMesh.cpp:680

11: ./erf_abl(+0x100ac77) [0x59bdef1ddc77]
amrex::AmrMesh::MakeNewGrids(double) at /home/adi/ERF/Submodules/AMReX/Src/AmrCore/AMReX_AmrMesh.cpp:995

12: ./erf_abl(+0xf63f6c) [0x59bdef136f6c]
amrex::AmrCore::InitFromScratch(double) at /home/adi/ERF/Submodules/AMReX/Src/AmrCore/AMReX_AmrCore.cpp:82

13: ./erf_abl(+0x11299c) [0x59bdee2e599c]
ERF::InitData_pre() at /home/adi/ERF/Source/ERF.cpp:678

14: ./erf_abl(+0x112810) [0x59bdee2e5810]
ERF::InitData() at /home/adi/ERF/Source/ERF.cpp:653

15: ./erf_abl(+0x68adc) [0x59bdee23badc]
main at /home/adi/ERF/Source/main.cpp:126

16: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7e89a0c29d90]

17: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7e89a0c29e40]

18: ./erf_abl(+0x580b5) [0x59bdee22b0b5]
?? ??:0

Input file
#------------------ INPUTS TO MAIN PROGRAM -------------------
stop_time = 400

erf.anelastic = 1
erf.use_fft = true

amrex.fpe_trap_invalid = 1
amrex.fpe_trap_zero = 1
amrex.fpe_trap_overflow = 1

fabarray.mfiter_tile_size = 1024 1024 1024

#===============PROBLEM SIZE & GEOMETRY========================
geometry.prob_extent = 2 2 0.8306048416557245
amr.n_cell = 256 256 96

#============Boundary Conditions================

geometry.is_periodic = 1 1 0
zlo.type = "NoSlipWall"
zhi.type = "SlipWall"
zlo.theta = 315.2905
zhi.theta_grad = 91.7431

#===================TIME STEP CONTROL========================

#erf.fixed_dt = 0.005 # fixed time step depending on grid resolution
erf.init_shrink = 0.2
erf.cfl = 0.2

#================== DIAGNOSTICS & VERBOSITY==========================
erf.sum_interval = 1 # timesteps between computing mass
erf.v = 1 # verbosity in ERF.cpp
amr.v = 1 # verbosity in Amr.cpp

#============== REFINEMENT / REGRIDDING=======================
erf.refinement_indicators = vorthi vortlo
amr.max_level = 1 # maximum level number allowed

erf.vorthi.max_level = 1
erf.vorthi.field_name = vorticity
erf.vorthi.value_greater = 0.9

erf.vortlo.field_name = vorticity
erf.vortlo.max_level = 1
erf.vortlo.value_less = -0.9

erf.regrid_int = 5

#==========CHECKPOINT FILES==========
erf.check_file = chk # root name of checkpoint file
erf.check_int = 40 # number of timesteps between checkpoints

#============= PLOTFILES================
erf.plot_file_1 = plt # prefix of plotfile name
erf.plot_int_1 = 20 # number of timesteps between plotfiles
erf.plot_vars_1 = density rhoadv_0 x_velocity y_velocity z_velocity pressure temp theta

#============== SOLVER CHOICE==============
erf.use_gravity = true

#=========DNS Parameters=====================
erf.molec_diff_type = "Constant"
erf.dynamic_viscosity = 1e-5
erf.alpha_C = 1e-5
erf.alpha_T = 1e-5

#==========Input/Initialisation==============
erf.init_type = "Input_Sounding"
erf.init_sounding_ideal = false
erf.input_sounding_file = "input_sounding"

#====================PROBLEM PARAMETERS=================
prob.rho_0 = 1.0
prob.A_0 = 1.0
prob.U_0 = 0.0
prob.V_0 = 0.0
prob.W_0 = 0.0

#===========Perturbation==================
prob.U_0_Pert_Mag = 0.01
prob.V_0_Pert_Mag = 0.01
prob.W_0_Pert_Mag = 0.01
prob.T_0_Pert_Mag = 0.0
prob.pert_ref_height = 0.1

[asalmgren]

@Adiram97 -- thanks so much for pointing out that error and sharing the inputs file so I could replicate it quickly. I believe that PR 2209 #2209 fixes this problem but I'd like my colleague @AMLattanzi to look it over before we merge it . I'm going to close this now but please re-open (or open a new issue) if you have any more problems! Best, Ann

[Adiram97]

Hi, Thank you for your investigation. Although I have a new error with the assertation involved in array bounds in AMReX Vector. I have attached the error message from the Debug mode:

BA FROM SCRATCH AT LEVEL 0 (BoxArray maxbox(16)
m_ref->m_hash_sig(0)
((0,0,0) (63,63,95) (0,0,0)) ((64,0,0) (127,63,95) (0,0,0)) ((128,0,0) (191,63,95) (0,0,0)) ((192,0,0) (255,63,95) (0,0,0)) ((0,64,0) (63,127,95) (0,0,0)) ((64,64,0) (127,127,95) (0,0,0)) ((128,64,0) (191,127,95) (0,0,0)) ((192,64,0) (255,127,95) (0,0,0)) ((0,128,0) (63,191,95) (0,0,0)) ((64,128,0) (127,191,95) (0,0,0)) ((128,128,0) (191,191,95) (0,0,0)) ((192,128,0) (255,191,95) (0,0,0)) ((0,192,0) (63,255,95) (0,0,0)) ((64,192,0) (127,255,95) (0,0,0)) ((128,192,0) (191,255,95) (0,0,0)) ((192,192,0) (255,255,95) (0,0,0)) )

max of zphys_nd 0.8306048417
max of zlevels 0.8306048417
input_sounding file location : input_sounding
Successfully opened the input_sounding file. Now reading...
Successfully read the 0th input_sounding file...
Adding random x-velocity perturbations
Adding random y-velocity perturbations
Creating new distribution map on level: 1
BA FROM SCRATCH AT LEVEL 1 (BoxArray maxbox(4)
m_ref->m_hash_sig(0)
((0,0,0) (255,255,23) (0,0,0)) ((256,0,0) (511,255,23) (0,0,0)) ((0,256,0) (255,511,23) (0,0,0)) ((256,256,0) (511,511,23) (0,0,0)) )

Adding random x-velocity perturbations
Adding random y-velocity perturbations
1::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!! SIGABRT 4::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!!
SIGABRT
0::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!! SIGABRT 5::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!!
SIGABRT
15::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!! SIGABRT 7::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!!
SIGABRT
13::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!! SIGABRT 6::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!!
SIGABRT
10::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!! SIGABRT 14::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!!
SIGABRT
2::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!! SIGABRT 3::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!!
SIGABRT
8::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!! SIGABRT 11::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!!
SIGABRT
9::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!! 12::Assertion i < (this->std::vector<T, Allocator>::size())' failed, file "/home/adi/ERF/Submodules/AMReX/Src/Base/AMReX_Vector.H", line 35 !!!
SIGABRT
SIGABRT

[AMLattanzi]

@Adiram97 I believe I was able to replicate the error and have a correction in PR 2212. @asalmgren will review and confirm the patch before merging. Please feel free to test the PR immediately and provide any feedback.

[Adiram97]

sorry for the late message, but it works now. Thank you very much.