f/gcl#5
Open
gdeskos wants to merge 861 commits into
Open
Conversation
* New GPU kernel Turb Kinetic Energy Rodi Second try. Since the test is over one element change the old non-GPU test to be over one element so the reference solution can be the same between the two. * Combine mesh utilities.
- Add an ABL source interpolator capable of execution on GPUs - Implement NGP ready versions of the nodal source algorithms
Compile on my mac with clang to make sure that there are no warnings and the unit tests run.
…lass (see issue kynema#249) (kynema#265) * TpetraSegregatedSystem: adding the new TpetraSegregatedLinearSystem class (see issue kynema#249) Create the new class and adds the necessary hooks in LinearSystem calss and CMakeList.txt Running the code on dgNonConformalThreeBlade goes through the simulation but more check probabaly would be good to make sure that the solution is accurate enough. A comparison with the segregated system using Hypre would be good too! * TpetraSegregatedLinearSystem: adding regression test ablNeutralEdgeSegregated Adding regression test to cover the new feature with the nightly tests.
Use MomentumSymmetry BC as an example to demonstrate conversion of edge BC algorithms. Add necessary KOKKOS_FUNCTION decorators to MasterElement methods.
The check to determine the ABL stability was incorrectly implemented in `ABLWallFunction` solver algorithms and, therefore, the code always ended up choosing the Stable profile function even for zero heat flux (neutral conditions). The logic was implemented correctly in `ComputeABLWallFrictionVelocityAlgorithm.C` (see line 285). Fixing this allows a much tighter tolerance in the ABLWallFunction unit tests. Another inconsistency is noted in the two implementations. ComputeABLWallFriction checks heatFlux whereas `AssembleABL*` checks the temperature flux (thereby dependent on density and specific heat). So there could be situations where friction velocity is computed assuming neutral conditions, where as the linear system is solved with a small contribution of psi_m from Stable profile function.
- Add necessary infrastructure to unit test the algorithm
Generates a lot of output during production runs.
Sets a tighter tolerance when using custom gold norms to check for change
The behavior of `pass_fail.sh` is modified such that it reports both the absolute difference (original behavior) and the corresponding relative difference for failing tests. In addition, it also introduces a few other modifications: - Add padding between the test name and the timing outputs - Refactor the code that uses `bc` for doing calculations - Fix an issue where the script was computing diffs of timesteps and time per timestep
…ynema#277) This pull request replaces the current non-NGP versions of Edge and nodal source algorithms in the following equation systems with their NGP counterparts (see kynema#238): - Momentum - Continuity - Enthalpy - TurbKineticEnergy - SpecificDissipationRate - WallDistance Other comments: - `AssembleElemSolverAlgorithm::initialize_connectivity()` calls `TpetraLinearSystem::buildFaceToNodeGraph()` when dealing with a boundary kernel. Without this, edge flavor of BC kernel (e.g., `MomentumABLWallFuncEdgeKernel`) segfaults. - The changes introduce some diffs in the regression tests due to order of operation changes, but all appear to be within expected range. * Transition to NGP edge kernels in certain equation systems Equation systems changed: - Momentum - Continuity - Enthalpy - TurbKineticEnergy - SpecificDissipationRate * Transition Node source kernels in MomentumEquationSystem to NGP versions * EnthalpyEQS: Introduce NGP version of NodeSource kernels Nodal source algorithms changed: - Time derivative (MassBDF) - ABL Forcing * SSTEQS: Add NGP node kernels Update TurbKineticEnergy and SpecificDissipationRate equations. Only time derivative algorithm for now. * MomentumEQS: Update symmetry BC to NGP version * WallDistEQS: Convert edge/node algorithms to NGP version * Fix initialize_connectivity for AssembleElemSolverAlgorithm Invoke buildFaceToNodeGraph if entityRank is side_rank() when using this class with Boundary kernels * Convert Edge ABL Wall function to NGP version * Cleanup unused code
* Convert ContinuityGclNodeSuppAlg to NGP. * Making some variable names consistent. * Adding div_mesh_velocity field to TestKernelHex8Mesh and additional state to dualNodalVolume in TestKernelHex8Mesh.
* Convert MomentumGclSrcNodeSuppAlg to NGP. * Adding missing private keyword to header. * Changing momentum gcl node kernel test name and adding gamma values and dt values to unit test. * Adding gold values to momentum gcl node kernel unit test.
* Convert ScalarGclNodeSuppAlg to NGP. * Fixing scalar gcl node kernel unit test with correct rhs values.
Introduce wrapper functions that accept a C++ lambda or functor to abstract out the looping process over buckets, and the handling of ScratchViews for element, face/element based looping constructs. These constructs will be used to replace current algorithms that deal with field modification so that the can occur on device instead of host. - NgpTypes: Introduce data structures to ease handling NGP data. In preparation for replacing logic within the code that currently use MetaData/Bulkdata with ngp::Mesh and ngp::Field<T>, start introducing some convenience types. Replace handling of ngp data in Realm with the one using wrapper. - Check looping strategy and field modification using mass flow rate computation in unit tests. - Convert massFlowRate calculator in unit tests to use NGP loops
Doesn't work yet. Pushing for debugging.
Also fixing a bug in the divergence calculation to not divide by volume.
To show it doesn't work
Also fix a bug in MdotEdgeAlg from earlier commit that left out multiplication of face velocity magnitude by density at face
gantech
pushed a commit
that referenced
this pull request
Sep 22, 2020
…ynema#473) * Separate graph and solver types Add new CrsGraphTypes namespace that LinearSolverTypes pulls in. * Rename linear system helper file Rename file holding Crs graph helper functions. The new name reflects that the functions are for the matrix graph. * Add new CrsGraph class This is a step towards removing the matrix graph creation/initialization that is currently in TpetraLinearSystem. Currently, CrsGraph duplicates functionality TpetraLinearSystem. Everything compiles. * Move graph from TpetraLinearSystem to CrsGraph This commit adds a CrsGraph object to TpetraLinearSystem and forwards graph requests to the CrsGraph class. NaluWind compiles and regression test "AblNeutralEdge" runs. * Code clean up remove code that has been moved to CrsGraph * Fix unit test compilation Reenable methods getRowLID and getColLID, which are used in unit testing. Change header inclusion. * CrsGraph: initialize inConstruction flag in ctor uninitialized bool was causing initialization to be skipped With this commit, all tests pass: Test project /ascldap/users/jhu/exawind/build-naluwind-gcc-7.2.0-openmpi-1.10.1-opt Start 75: oversetHybrid Start 33: heatedBackStep 1/77 Test kynema#33: heatedBackStep ......................... Passed 227.44 sec Start 38: hoVortex Start 63: quad9HC 2/77 Test kynema#63: quad9HC ................................ Passed 71.32 sec Start 66: variableDensNonIso 3/77 Test kynema#38: hoVortex ............................... Passed 117.75 sec Start 67: variableDensNonUniform 4/77 Test kynema#67: variableDensNonUniform ................. Passed 77.28 sec Start 73: unitTest1 Start 57: periodic3dElemNp1 5/77 Test kynema#66: variableDensNonIso ..................... Passed 139.80 sec Start 47: nonConformalWithPeriodic 6/77 Test kynema#57: periodic3dElemNp1 ...................... Passed 18.53 sec Start 42: kovasznay_P5 7/77 Test kynema#73: unitTest1 .............................. Passed 23.09 sec Start 60: periodic3dEdgeNp1 8/77 Test kynema#60: periodic3dEdgeNp1 ...................... Passed 8.00 sec Start 41: karmanVortex 9/77 Test kynema#42: kovasznay_P5 ........................... Passed 13.79 sec 10/77 Test kynema#41: karmanVortex ........................... Passed 1.96 sec Start 74: unitTest2 11/77 Test kynema#47: nonConformalWithPeriodic ............... Passed 18.18 sec Start 48: nonConformalWithPeriodicConsolidated 12/77 Test kynema#74: unitTest2 .............................. Passed 13.63 sec Start 31: femHCGL 13/77 Test kynema#48: nonConformalWithPeriodicConsolidated ... Passed 14.57 sec Start 26: elemClosedDomain 14/77 Test kynema#31: femHCGL ................................ Passed 3.85 sec Start 30: femHC 15/77 Test kynema#26: elemClosedDomain ....................... Passed 3.95 sec Start 19: ductElemWedge 16/77 Test kynema#30: femHC .................................. Passed 3.90 sec Start 20: ductWedge 17/77 Test kynema#19: ductElemWedge .......................... Passed 2.54 sec 18/77 Test kynema#20: ductWedge .............................. Passed 1.73 sec Start 53: nonIsoNonUniformElemOpenJet 19/77 Test kynema#53: nonIsoNonUniformElemOpenJet ............ Passed 165.31 sec Start 43: milestoneRun 20/77 Test kynema#43: milestoneRun ........................... Passed 115.86 sec Start 4: ablUnstableEdge 21/77 Test #4: ablUnstableEdge ........................ Passed 89.04 sec Start 64: steadyTaylorVortex 22/77 Test kynema#64: steadyTaylorVortex ..................... Passed 121.05 sec Start 44: milestoneRunConsolidated 23/77 Test kynema#75: oversetHybrid .......................... Passed 1017.76 sec Start 77: waleElemXflowMixFrac3.5m 24/77 Test kynema#44: milestoneRunConsolidated ............... Passed 104.43 sec Start 3: ablStableElem 25/77 Test #3: ablStableElem .......................... Passed 87.88 sec Start 52: nonIsoNonUniformEdgeOpenJet 26/77 Test kynema#52: nonIsoNonUniformEdgeOpenJet ............ Passed 63.72 sec Start 16: dgNonConformalFluids 27/77 Test kynema#16: dgNonConformalFluids ................... Passed 61.76 sec Start 50: nonIsoElemOpenJet 28/77 Test kynema#50: nonIsoElemOpenJet ...................... Passed 61.97 sec Start 17: dgNonConformalFluidsEdge 29/77 Test kynema#17: dgNonConformalFluidsEdge ............... Passed 50.90 sec Start 51: nonIsoElemOpenJetConsolidated 30/77 Test kynema#51: nonIsoElemOpenJetConsolidated .......... Passed 52.57 sec Start 5: ablUnstableEdge_ra 31/77 Test #5: ablUnstableEdge_ra ..................... Passed 37.09 sec Start 24: ekmanSpiralConsolidated 32/77 Test kynema#24: ekmanSpiralConsolidated ................ Passed 43.84 sec Start 40: inputFireElem 33/77 Test kynema#77: waleElemXflowMixFrac3.5m ............... Passed 516.90 sec Start 37: hoHelium 34/77 Test kynema#40: inputFireElem .......................... Passed 42.54 sec Start 34: heatedWaterChannelEdge 35/77 Test kynema#34: heatedWaterChannelEdge ................. Passed 37.97 sec Start 18: dgNonConformalThreeBlade 36/77 Test kynema#18: dgNonConformalThreeBlade ............... Passed 245.03 sec Start 49: nonIsoEdgeOpenJet 37/77 Test kynema#49: nonIsoEdgeOpenJet ...................... Passed 23.74 sec Start 6: concentricRad 38/77 Test #6: concentricRad .......................... Passed 25.52 sec Start 11: dgNonConformal3dFluidsHexTet 39/77 Test kynema#11: dgNonConformal3dFluidsHexTet ........... Passed 134.40 sec Start 29: elemPipeCHT 40/77 Test kynema#29: elemPipeCHT ............................ Passed 18.82 sec Start 23: ekmanSpiral 41/77 Test kynema#37: hoHelium ............................... Passed 539.10 sec Start 71: hoVortex_P2 42/77 Test kynema#23: ekmanSpiral ............................ Passed 14.30 sec Start 10: dgNonConformal3dFluids 43/77 Test kynema#10: dgNonConformal3dFluids ................. Passed 117.65 sec Start 25: elemBackStepLRSST 44/77 Test kynema#25: elemBackStepLRSST ...................... Passed 11.00 sec Start 46: movingCylinder 45/77 Test kynema#46: movingCylinder ......................... Passed 12.13 sec Start 22: edgePipeCHT 46/77 Test kynema#22: edgePipeCHT ............................ Passed 10.90 sec Start 39: inputFireEdgeUpwind 47/77 Test kynema#39: inputFireEdgeUpwind .................... Passed 7.94 sec Start 9: dgNonConformal 48/77 Test kynema#9: dgNonConformal ......................... Passed 8.07 sec Start 58: periodic3dElemNp4 49/77 Test kynema#58: periodic3dElemNp4 ...................... Passed 5.47 sec Start 61: periodic3dEdgeNp4 50/77 Test kynema#61: periodic3dEdgeNp4 ...................... Passed 3.82 sec Start 35: heatedWaterChannelElem 51/77 Test kynema#35: heatedWaterChannelElem ................. Passed 20.46 sec Start 13: dgNonConformalEdge 52/77 Test kynema#13: dgNonConformalEdge ..................... Passed 7.60 sec 53/77 Test kynema#71: hoVortex_P2 ............................ Passed 384.64 sec Start 36: heliumPlume 54/77 Test kynema#36: heliumPlume ............................ Passed 226.47 sec Start 76: uqSlidingMeshDG 55/77 Test kynema#76: uqSlidingMeshDG ........................ Passed 216.66 sec Start 70: cvfemHexHC_P3 56/77 Test kynema#70: cvfemHexHC_P3 .......................... Passed 235.98 sec Start 32: fluidsPmrChtPeriodic 57/77 Test kynema#32: fluidsPmrChtPeriodic ................... Passed 99.94 sec Start 72: steadyTaylorVortex_P4 58/77 Test kynema#72: steadyTaylorVortex_P4 .................. Passed 154.11 sec Start 68: variableDensNonUniform_P5 59/77 Test kynema#68: variableDensNonUniform_P5 .............. Passed 187.59 sec Start 1: ablNeutralEdge 60/77 Test #1: ablNeutralEdge ......................... Passed 78.58 sec Start 27: elemHybridFluids 61/77 Test kynema#27: elemHybridFluids ....................... Passed 94.55 sec Start 69: BoussinesqNonIso 62/77 Test kynema#69: BoussinesqNonIso ....................... Passed 90.32 sec Start 28: elemHybridFluidsShift 63/77 Test kynema#28: elemHybridFluidsShift .................. Passed 88.30 sec Start 65: tensor_tg_mms 64/77 Test kynema#65: tensor_tg_mms .......................... Passed 98.70 sec Start 15: dgNonConformalElemCylinder 65/77 Test kynema#15: dgNonConformalElemCylinder ............. Passed 103.07 sec Start 21: edgeHybridFluids 66/77 Test kynema#21: edgeHybridFluids ....................... Passed 61.73 sec Start 8: dgMMS Start 55: oversetFluids 67/77 Test kynema#55: oversetFluids .......................... Passed 23.22 sec Start 56: oversetFluidsEdge 68/77 Test kynema#56: oversetFluidsEdge ...................... Passed 21.07 sec Start 54: overset 69/77 Test kynema#54: overset ................................ Passed 2.16 sec 70/77 Test kynema#8: dgMMS .................................. Passed 51.45 sec Start 12: dgNonConformal3dFluidsP1P2 71/77 Test kynema#12: dgNonConformal3dFluidsP1P2 ............. Passed 294.81 sec Start 45: mixedTetPipe 72/77 Test kynema#45: mixedTetPipe ........................... Passed 30.65 sec Start 14: dgNonConformalEdgeCylinder 73/77 Test kynema#14: dgNonConformalEdgeCylinder ............. Passed 65.57 sec Start 7: cvfemHC 74/77 Test #7: cvfemHC ................................ Passed 10.66 sec Start 59: periodic3dElemNp8 75/77 Test kynema#59: periodic3dElemNp8 ...................... Passed 3.86 sec Start 62: periodic3dEdgeNp8 76/77 Test kynema#62: periodic3dEdgeNp8 ...................... Passed 2.54 sec Start 2: ablNeutralEdgeSegregated 77/77 Test #2: ablNeutralEdgeSegregated ............... Passed 67.66 sec 100% tests passed, 0 tests failed out of 77 Label Time Summary: performance = 14010.66 sec*proc (3 tests) regression = 26090.50 sec*proc (68 tests) unit = 50.35 sec*proc (2 tests) verification = 6920.38 sec*proc (4 tests) Total Test time (real) = 4722.03 sec * Add header to fix compile error. * Code cleanup remove ifdef'd out code * TpetraSegregatedLinearSystem uses CrsGraph class Code compiles, ablNeutralEdgeSegregated test passes. * TpetraSegregatedLinearSystem: code cleanup remove inactive code protected by macros * TpetraSegregatedLinearSystem: use Tpetra GIDs in copy_stk_to_tpetra(), use realm_.tpetGlobalId_ instead of realm_.naluGlobalId_ 1/1 Test #2: ablNeutralEdgeSegregated ......... Passed 68.39 sec * Fix cmake error from merge * Add missing beginConstruction * Add matrix graph reuse between physics phases * Add forgotten header. Oops. * add parser option to rebalance mesh using stk_balance (kynema#471) * add parser option to rebalance mesh using stk_balance * Move call to doBalanceNodes after rebalanceMesh * Add throw when rebalance is called with parmetis but Zoltan2 is not built with parmetis enabled * Remove unused non-NGP code (kynema#486) * Remove unused classes after transition to NGP * Remove deprecated code used during NGP transition * Remove OversetManagerSTK in preparation for NGP transition of overset capability (kynema#487) * Prepare for deprecation of STK-search based overset manager * Remove deprecated OversetManagerSTK * Remove deprecated methods related to overset logic - Remove `LinearSystem::prepareConstraints` and its overrides - Remove `SolverAlgorithm::apply_coeff` methods that have NGP versions * Remove the tests from CTestList.cmake also * Hypre regression tests moved into guarded block Regression tests using Hypre as one of their solvers need to be guarded with if(ENABLE_HYPRE) endif() * Airfoil tests using Trilinos solver stack, see issue kynema#492 (kynema#494) * Airfoil tests using Trilinos solver stack, see issue kynema#492 This adds two new tests that use MueLu as preconditioner for the continuity eqs. The convergence behavior is reasonable and sets things well for future GPU testing of inflow/outflow BCs. * Fixing input decks to use the segregated solvers for momentum Fixes kynema#492 * NGP conversion of TpetraLinearSystem::applyDirichletBCs (kynema#493) * NGP conversion of TpetraLinearSystem::applyDirichletBCs with unit-test. * Minor fixes to ensure fields are sync'd correctly (host vs device) * Use NGP_ThrowAssert instead of NGP_ThrowRequire * Fix accidental revert of the mesh submodule. * CrsGraph: ensure finalizeGraph called only once * CrsGraph: initialize variable in ctor * Updates for CrsGraph These commits add the calls buildElemToNodeGraph and buildFaceElemToNodeGraph to various "initalize_connectivity" methods. This effectively creates a superset of graph connectivity for different physics. Also added a check to avoid calling CrsGraph::finalizeGraph multiple times (suggested by RK). Currently, these tests are still failing: 1 - ablNeutralEdge (Failed) 4 - ablUnstableEdge (Failed) 5 - ablUnstableEdge_ra (Failed) 6 - airfoilRANSEdgeTrilinos (Failed) 22 - ductWedge (Failed) 23 - edgeHybridFluids (Failed) 24 - edgePipeCHT (Failed) 43 - karmanVortex (Failed) 50 - nonIsoEdgeOpenJet (Failed) 53 - nonIsoNonUniformEdgeOpenJet (Failed) * Prepare equation systems for common graph * Revert "Updates for CrsGraph" This reverts commit 880555d. * Modify EquationSystems to use a common graph for Tpetra * Common graph initialization refactor - Only initialize graphs if they are required by TpetraLinearSystems - Make sure both hypre and tpetra pathways work correctly * Add timers to track init costs missing from EquationSystem timers * Consolidate buildNodeGraph * Propagate changes to TpetraSegregatedLinearSystem * Fix regression test failures Remove call to finalizeLinearSystem, as was done for other physics * Update unit test helpers Avoids referencing uninitialized maps. However, some tests are still not passing. * Improve unit-testing for sparse-matrix values. (kynema#544) * Improve unit-testing for sparse-matrix values. It used to just compare each matrix column and coefficient to the expected gold values, expecting position to match as well. Now it searchs the matrix row for each gold column and checks if the coefficient matches, and doesn't fail if there are extra columns in the matrix. * Add a check to make sure non-gold columns have coeffs of zero. * Fixing build error after updating Trilinos * NaluWind: fix nonlinear residual Make the nonlinear residual convergence tolerance for momentum and continuity consistent with the tolerance set by the user. * NaluWind: fix compile errors * RegTests: update linear solver input deck Tpetra matrices now potentially store hard zeros, due to refactoring to use pre-computed CrsGraph's. The test fluidsPmrChtPeriodic is sensitive to these zeros, so I've created a new MueLu input deck that drops the hard zeros during multigrid coarsening. * RegTests: new MueLu settings for fluidsPmrChtPeriodic Add aggregation dropping threshold to remove hard zeros due to CrsGraph refactor. * Remove commented out code. * Fix inheritance of nonlinear tolerances This is a much better fix, suggested by Shreyas. Co-authored-by: Johnathan Vo <jvo1012@users.noreply.github.com> Co-authored-by: Shreyas Ananthan <shreyas@umd.edu> Co-authored-by: Luc Berger-Vergiat <lberge@sandia.gov> Co-authored-by: Alan Williams <william@sandia.gov> Co-authored-by: Jon Rood <jon.rood@nrel.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.