opencl: add set_rows for f16 and f32
#14547
Open
+247
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
ggml
ggerganov
reviewed
Jul 6, 2025
ggml/src/ggml-opencl/ggml-opencl.cpp
Outdated
Comment on lines
3482
to
3486
|
|
||
| int nth0 = 256; | ||
| size_t global_work_size[] = {(size_t)ne01*nth0, (size_t)ne02, (size_t)ne03}; | ||
| size_t local_work_size[] = {(size_t)nth0, 1, 1}; | ||
|
|
There was a problem hiding this comment.
Note that implementing it like this won't be very efficient. This dedicates 256 threads for each row of data. So for small rows with less than 256 elements, there will be wasted resources. For example, when FA is disabled, ggml_set_rows() is used with rows of 1 element (due to the V cache being transposed), so 255 out of the 256 local threads will be idle.
That's why in the Metal implementation I did "threadgroup batching" so that the local threads can work on multiple rows. Might want to consider implementing it here too for improved performance.
There was a problem hiding this comment.
Thanks a lot for the suggestion - it makes a good point. Looking into this.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Following a70c8a0, this PR adds
set_rowsfor f16 and f32.Make sure to read the contributing guidelines before submitting a PR