JOSS review: Failing tests

[pratikvn]

Hello,

As a part of the JOSS review process (#1010), I am running the test suite by compiling with the provided basic cmake.sh script in the Build/ folder.

I run the Tests with the ctest command and the following tests fail.

UpdateCTestConfiguration  from :$HOME/00Sources/ERF/build-nocuda/DartConfiguration.tcl
Parse Config file:$HOME/00Sources/ERF/build-nocuda/DartConfiguration.tcl
UpdateCTestConfiguration  from :$HOME/00Sources/ERF/build-nocuda/DartConfiguration.tcl
Parse Config file:$HOME/00Sources/ERF/build-nocuda/DartConfiguration.tcl
Test project $HOME/00Sources/ERF/build-nocuda
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 22
    Start 22: MovingTerrain_nosub

22: Test command: /usr/bin/sh "-c" "/software/all/mpi/openmpi/4.1_gnu_8/bin/mpiexec -n 4 --oversubscribe $HOME/00Sources/ERF/build-nocuda/Exec/MovingTerrain/moving_terrain $HOME/00Sources/ERF/build-nocuda/Tests/test_files/MovingTerrain_nosub/MovingTerrain_nosub.i  > MovingTerrain_nosub.log && $HOME/00Sources/ERF/build-nocuda/Submodules/AMReX/Tools/Plotfile/fcompare -a -r 1e-12 --abs_tol 1.0e-12 $HOME/00Sources/ERF/Tests/ERF-WindGoldFiles/MovingTerrain_nosub $HOME/00Sources/ERF/build-nocuda/Tests/test_files/MovingTerrain_nosub/plt00020"
22: Test timeout computed to be: 5400
22: 
22:             variable name            absolute error            relative error
22:                                         (||A - B||)         (||A - B||/||A||)
22:  ----------------------------------------------------------------------------
22:  level = 0
22:  density                            4.440892099e-15           3.824348191e-15
22:  x_velocity                         5.563021173e-13           5.306799119e-12
22:  y_velocity                           1.1509322e-19              0.4057820202
22:  z_velocity                         4.203707897e-13           3.931109855e-12
22:  pressure                           4.656612873e-10            4.65780326e-15
22:  theta                              7.389644452e-13             2.4631878e-15
22:  pres_hse                           3.783497959e-10           3.784522763e-15
22:  dens_hse                           3.774758284e-15           3.250695837e-15
22:  pert_pres                          4.399822728e-10           2.848124617e-10
22:  pert_dens                          5.329070518e-15           2.081808556e-09
22:  dpdx                               7.858076013e-11             8.0625449e-10
22:  dpdy                               1.206521105e-14              0.1279824109
22:  pres_hse_x                         5.820981694e-11           2.307965928e-07
22:  pres_hse_y                         1.206615185e-14              0.1279923904
22:  z_phys                             3.552713679e-15           8.938356067e-18
22:  detJ                               2.886579864e-15           2.883216085e-15
1/2 Test #22: MovingTerrain_nosub ..............***Failed    8.32 sec
test 23
    Start 23: MovingTerrain_sub

23: Test command: /usr/bin/sh "-c" "/software/all/mpi/openmpi/4.1_gnu_8/bin/mpiexec -n 4 --oversubscribe $HOME/00Sources/ERF/build-nocuda/Exec/MovingTerrain/moving_terrain $HOME/00Sources/ERF/build-nocuda/Tests/test_files/MovingTerrain_sub/MovingTerrain_sub.i  > MovingTerrain_sub.log && $HOME/00Sources/ERF/build-nocuda/Submodules/AMReX/Tools/Plotfile/fcompare -a -r 1e-12 --abs_tol 1.0e-12 $HOME/00Sources/ERF/Tests/ERF-WindGoldFiles/MovingTerrain_sub $HOME/00Sources/ERF/build-nocuda/Tests/test_files/MovingTerrain_sub/plt00010"
23: Test timeout computed to be: 5400
23: 
23:             variable name            absolute error            relative error
23:                                         (||A - B||)         (||A - B||/||A||)
23:  ----------------------------------------------------------------------------
23:  level = 0
23:  density                            8.659739592e-15           7.457449344e-15
23:  x_velocity                         7.119054017e-13           6.682107789e-12
23:  y_velocity                         2.477377881e-14              0.9998576579
23:  z_velocity                         5.657830975e-13           5.280345392e-12
23:  pressure                           7.130438462e-10           7.132271236e-15
23:  theta                              1.818989404e-12           6.063231461e-15
23:  pres_hse                           5.384208634e-10           5.385667378e-15
23:  dens_hse                           5.773159728e-15           4.971656958e-15
23:  pert_pres                          4.585629654e-10           3.086508208e-10
23:  pert_dens                          8.437694987e-15           5.430548735e-10
23:  dpdx                               8.586018268e-11           9.404887579e-10
23:  dpdy                               4.365574569e-12               0.994538118
23:  pres_hse_x                         6.113835926e-11           2.663532726e-08
23:  pres_hse_y                         1.606825666e-14                 0.1702319
23:  z_phys                             2.842170943e-14           7.150684854e-17
23:  detJ                               5.551115123e-15           5.544626322e-15
2/2 Test #23: MovingTerrain_sub ................***Failed    9.58 sec

0% tests passed, 2 tests failed out of 2

Label Time Summary:
regression    =  71.58 sec*proc (2 tests)

Total Test time (real) =  17.91 sec

The following tests FAILED:
	 22 - MovingTerrain_nosub (Failed)
	 23 - MovingTerrain_sub (Failed)

My CMakeCache.txt file is attached for your reference:

CMakeCache.txt

[asalmgren]

I have updated the CI tests to not ask for more precision than is reasonable -- I didn't see this before because my diffs were returning 0 but asking for certain perturbational quantities / derivatives to be less than the threshold was not reasonable. The CI tests should now pass.

[asalmgren]

Going ahead and closing this issue because I believe this is resolved.