inflow-outflow boundary condition and unphysical flow

[lawrenceccheung]

Bug description

We might have to take a look at the fix that was discussed in #1496 and the AMReX PR AMReX-Fluids/AMReX-Hydro#162.

To be clear, using the new AMReX library does allow the nodal projections to converge as expected, but it introduces regions of nonphysical flow that are not connected to the inflow/outflow boundaries. In this case, inflow/outflow boundaries are applied to the xlo/xhi/ylo/yhi faces, but after a few seconds of running, there is a region of flow in the middle of the domain which develops with really high vertical velocity, and it looks like a big rift has been developed in the flow with smaller than necessary velocities:

(Block dots correspond to turbine locations). Notice that this region of flow does not necessarily correspond to boundaries of where the mesh refinements are:

If you set max_level=0 or you use an older amr-wind, then the flow behaves as normal, although the nodal_projections do slow down the calculation. This is the expected flow behavior at the same time:

Steps to reproduce

Steps to reproduce the behavior:

1. Compiler used
   • GCC
   • LLVM
   • oneapi (Intel)
   • nvcc (NVIDIA)
   • rocm (AMD)
   • with MPI
   • other:
2. Operating system
   • Linux
   • OSX
   • Windows
   • other (do tell ;)):
3. Hardware:
   • CPU
   • GPU
4. Machine details ():

Sandia Flight

5. Input file attachments

Input file available at https://gist.github.com/lawrenceccheung/4784364af6c1d0125fba00bec51e029f

6. Error (paste or attach):

No errors, but AMR-Wind will eventually blow up due to excessive CFL/flow velocities developing.

7. If this is a segfault, a stack trace from a debug build (paste or attach):
   N/A

Expected behavior

See images above from level 0 only simulation.

AMR-Wind information

The AMR-Wind/AMReX version which has problems is

==============================================================================
                AMR-Wind (https://github.com/exawind/amr-wind)

  AMR-Wind version :: v3.4.0-16-g26063277
  AMR-Wind Git SHA :: 26063277b57415e735274c0d366ff702ca14fc14
  AMReX version    :: 25.02-23-g06b4a5b105f5

  Exec. time       :: Fri Mar 14 11:04:49 2025
  Build time       :: Mar  4 2025 14:43:50
  C++ compiler     :: GNU 12.1.0

  MPI              :: ON    (Num. ranks = 1344)
  GPU              :: OFF
  OpenMP           :: OFF

  Enabled third-party libraries: 
    NetCDF    4.9.2
    OpenFAST  

           This software is released under the BSD 3-clause license.           
 See https://github.com/Exawind/amr-wind/blob/development/LICENSE for details. 
------------------------------------------------------------------------------

The version which does work (but is slower due to the nodal_projections) is

==============================================================================
                AMR-Wind (https://github.com/exawind/amr-wind)

  AMR-Wind version :: v3.3.1-44-gd7046e1c-DIRTY
  AMR-Wind Git SHA :: d7046e1c70246f5c461f3244bc85958b60398310-DIRTY
  AMReX version    :: 25.01-16-g92d35c2c8163

  Exec. time       :: Fri Mar 14 21:12:42 2025
  Build time       :: Feb  6 2025 21:45:27
  C++ compiler     :: GNU 12.1.0

  MPI              :: ON    (Num. ranks = 2240)
  GPU              :: OFF
  OpenMP           :: OFF

  Enabled third-party libraries: 
    NetCDF    4.9.2
    OpenFAST  

           This software is released under the BSD 3-clause license.           
 See https://github.com/Exawind/amr-wind/blob/development/LICENSE for details. 
------------------------------------------------------------------------------

[lawrenceccheung]

Tagging @asalmgren and @mukul1992 on this issue

[asalmgren]

@lawrenceccheung -- is it easy to run this case with inflow on some faces or outflow on others, or is the flow actually inflowing and outflowing on the same face? That would be another data point -- if it only happens with both refinement and this particular bc type.

The inflow/outflow option shouldn't be having any effect in the middle of the domain but given that it only appears with refinement, I'd suspect that somehow we're labeling the coarse/fine boundaries as inflow/outflow by mistake? I think that labeling is happening internal to AMR-Wind, not in AMReX-Hydro ....?

Mukul is heading off for a new position and I'm on foreign travel right now, so I'm wondering if there's an AMR-WInd developer who could look into this right now? @marchdf -- could you take a look?

[lawrenceccheung]

Hi @asalmgren,

Yes, for this problem we can switch out the mass_inflow_outflow BC for mass_inflow/pressure_outflow on some or all of the faces. It's not a long term stable solution because eventually there will be faces with both inflow and outflow, but it should be enough to diagnose this problem.

Lawrence

[marchdf]

Hi Lawrence, thanks for running those checks. While you are doing that, would it be possible to share the boundary planes and the restart chk file?

[hgopalan]

Following up yesterday's discussion, I had run a LES case in the past with flow turning. However, I did not use mass inflow-outflow. I instead set pressure outflow on all the horizontal boundaries and added a fringe zone to the boundaries to force the 1-D vertical profile. The simulation was for a smaller WFIP 2 site with up to 2-3 levels of refinement. There is flow turning in the span wise velocity component.

1-D Profile

Streamwise - Wind

Vertical - Wind

[lawrenceccheung]

@marchdf The checkpoint, boundary files, and the MMC tendency files are now at Kestrel at /scratch/lcheung/precursors/MMC_unstable_BigPrecursor3. Let me know if you have any problems accessing them.

Also, confirming what @hgopalan mentioned earlier -- this version of AMR-Wind and AMReX works fine with cases that use mass_inflow (the normal boundary plane I/O) BC with constant body forces (see the NREL5MW benchmark case). We can switch those cases to mass_inflow_outflow and see if anything bad happens there, but let me try with this case first since I know there is a problem here for sure.

[mbkuhn]

I think the issue that Lawrence is having has to do with the bug revealed here: AMReX-Fluids/AMReX-Hydro#164

As it is in the code now, it checks the higher levels to see if they line up with the domain boundary of lev = 0. Because Lawrence's refinement zones are near the middle of the domain, their indices could possibly line up with the domain boundary of level = 0. (indices on lev = 1 are 2 times the indices on lev = 0 due to refinement, which means that halfway through the domain, (2*nx)/2 = nx, which would be the outflow boundary indices on lev = 0.)

These would also be erroneously "corrected" by the routine, likely leading to unphysical flow. Furthermore, the nature of the boundary index check changed between the two implementations discussed here, possibly introducing the bug.

However, I have no proof yet that this AMReX-Hydro bug (which definitely exists) is causing the problem we see in this AMR-Wind issue.

[marchdf]

Amazing work, @mbkuhn!

[mbkuhn]

@lawrenceccheung Now that #1554 is merged with the bug fixed in AMReX-Hydro, could you try to rerun this case and see if the bad behavior is gone?