merge_hack1

README.md

@@ -1,6 +1,6 @@
 # MHIT36
 
-Code for direct numerical simulation of Navier-Stokes equation coupled with a phase-field method (ACDI) for interface description.
+Code for direct numerical simulation of Navier-Stokes equations coupled with a phase-field method (ACDI) for interface description.
 
 ~~~text
 ███    ███ ██   ██ ██ ████████ ██████   ██████       
@@ -17,7 +17,7 @@ If you use this code, please cite the following work:
   @article{roccon2025,
   title   = {MHIT36: A Phase-Field Code for Gpu Simulations of Multiphase Homogeneous Isotropic Turbulence},
   author  = {Roccon, A. and Enzenberger, L. and Zaza, D. and Soldati, A.},
-  journal = {SSRN},
+  journal = {Computer Physics Communications (in press)},
   year    = {2025},
   doi     = {http://dx.doi.org/10.2139/ssrn.5264052}
 }
@@ -58,15 +58,36 @@ If you use this code, please cite the following work:
 
 ## Reference performance
 
-Performance (NS only) - from last push
-* 512 x 512 x 512 | 4 x A100@Leonardo  |  19 ms/timestep
-* 1024 x 1024 x 1024 | 4 x GH200@Nvidia | 161.9 ms/timestep (out of date)
-* 1024 x 1024 x 1024 | 8 x A100@Leonardo | 150.1 ms/timestep
-* 512 x 512 x 512 | 4 x H100@MN5-ACC   |  18 ms/timestep (out of date)
+Performance (NS only)
+* 256 x 256 x 256 | 4 x A100@Leonardo  |   16 ms/timestep
+* 512 x 512 x 512 | 4 x A100@Leonardo  |  270 ms/timestep 
+* 1024 x 1024 x 1024 | 32 x A100@Leonardo | 357 ms/timestep 
+* 2048 x 2048 x 2048 | 128 x A100@Leonardo | 820 ms/timestep
+* 256 x 256 x 256 | 4 x H100@MN5-ACC   |   13 ms/timestep
+* 512 x 512 x 512 | 4 x H100@MN5-ACC   |  230 ms/timestep
+* 1024 x 1024 x 1024 | 32 x H100@MN5-ACC  | 320 ms/timestep
+* 2048 x 2048 x 2048 | 512 x H100@MN5-ACC  | 259 ms/timestep
 
-Performance (NS + ACDI) - from last push
-* 512 x 512 x 512 | 4 x A100@Leonardo  |  28 ms/timestep
-* 1024 x 1024 x 1024 | 8 x A100@Leonardo | 288 ms/timestep
+Phase-field introduces about 15% of overhead compared to NS only.
+
+## Scaling
+
+Strong scaling results obtained on Leonardo (4 x A100 64 GB x node) and MN5 (4 x H100 40 GB x node)
+* Tested from 1 node up to 128 nodes (Leonardo)
+* Tested from 1 node up to 256 nodes (MN5-ACC)
+* Grid from 64 x 64 x 64 up to 4096 x 4096 x 4096
+* Very similar scaling for both NS and NS+ACDI
+
+![Scal](val/scaling.png)
+
+
+## Validation
+
+Benchamrk present in "W.M.VanRees, A.Leonard, D.Pullin, P.Koumoutsakos, A comparison of vortex and pseudo-spectral methods for the simulation of periodic vortical flows at high Reynolds numbers,J. Comput. Phys.2 30(8)(2011)2794–2805" and also Used in CaNS.
+
+Time evolution of the viscous dissipation:
+
+![Test](val/val.png)
 
 
 ## Contributing

make_lib_leo.sh

@@ -6,4 +6,6 @@ module load profile/candidate
 module load nvhpc/25.3
 module load hpcx-mpi/2.19
 cmake ..
+#enable nvshmem
+#cmake -DCUDECOMP_BUILD_EXTRAS=1 -DCUDECOMP_ENABLE_NVSHMEM=1 ..
 make -j

multi/Makefile_local

@@ -4,7 +4,7 @@ LD	= $(FC)
 RM	= /bin/rm -f
 
 # Paths (Modify if necessary)
-CUDECOMP_DIR = /home/milton/MHIT36_cuDecomp/cuDecomp/build/
+CUDECOMP_DIR = /home/milton/MHIT36/cuDecomp/build/
 
 # Compiler and Linker Flags
 FFLAGS = -fast -acc -gpu=managed -Minfo=accel -Mfree -Mpreprocess -cpp -Minfo=accel -cuda  -I$(CUDECOMP_DIR)/include