coll/han: disable alltoall for device buffers and MPI_IN_PLACE #13200
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The han alltoall algorithm was never intended for an in-place alltoall and will crash if used in that configuration. Additionally it is not an efficient choice for device buffers. Fall back to another algorithm if either condition is met. Signed-off-by: Luke Robison <[email protected]>
As an optimization, when the han alltoall algorithm is forced to allocate a bounce buffer, and that bounce buffer is small enough, then use a buffer from the han_component.pack_buffers free list. Signed-off-by: Luke Robison <[email protected]>
|
@hppritcha This should address the performance problem you shared with me on main. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
also a small optimization, but note that it doesn't impact applications which provide contiguous buffers.
Testing: confirmed in-place bug and fix using osu micro benchmarks. Tested correctness using the alltoall mode of https://github.com/open-mpi/ompi-tests-public/tree/master/alltoallv_validation. I was not able to perform performance tests using device buffers.