feat: prefer torch stable abi for cuda fa2#997
Conversation
|
/kernel-bot build flash-attn2 |
|
Build request processed. Command: Dispatched (1):
|
Signed-off-by: David Holtz <david.richard.holtz@gmail.com>
|
/kernel-bot build flash-attn2 |
|
Build request processed. Command: Dispatched (1):
|
drbh
added
abi-migration
|
/kernel-bot build flash-attn2 |
1 similar comment
|
/kernel-bot build flash-attn2 |
|
Build request processed. Command: Dispatched (1):
|
|
/kernel-bot build flash-attn2 |
|
Build request processed. Command: Dispatched (1):
|
|
/kernel-bot build-and-stage flash-attn2 |
|
Build request processed. Command: Dispatched (1):
|
|
/kernel-bot build-and-stage flash-attn2 |
|
Build request processed. Command: Dispatched (1):
|
| [general] | ||
| name = "flash-attn2" | ||
| version = 2 | ||
| version = 3 |
There was a problem hiding this comment.
yes we ignore an argument and drop support for dropout
| version = 2 | ||
| version = 3 | ||
| license = "BSD-3-Clause" | ||
| edition = 5 |
There was a problem hiding this comment.
Remind me why do we need the edition field? Didn't find too much explanation under https://huggingface.co/docs/kernels/main/en/builder/writing-kernels#general.
There was a problem hiding this comment.
this was just added for backend scoped stable abis, its basically a versioning for the metadata, an earlier discussion is here huggingface/kernels#682 (comment)
| "int window_size_right, " | ||
| "float softcap, " | ||
| "bool deterministic, " | ||
| "Generator? gen_, " |
There was a problem hiding this comment.
I don't think Generator is supported in the schema as of 2.10--I added it in 2.13! But I suspect this works because gen_ is always None? Could you call out the supported/no longer supported features in this implementation? I've also been trying to get upstream to adopt stable dropoutless FA2 through a more explicit approach: Dao-AILab/flash-attention#2691
There was a problem hiding this comment.
ah thanks for pointing out, yea the Generator is always unused in these changes and was silently dropped. I've updated to throw if a generator is passed to be more explicit about not supporting.
and I've updated the pr description to more clearly indicate that dropout is disabled in this change set.
also longer term it would be great to replace the current approach with your changes once they are upstreamed. I'll keep and eye for the merge and circle back!
thanks for the help @janeyx99!
|
/kernel-bot merge-and-upload flash-attn2 |
|
Build request processed. Command: Merge result: PR is already merged. Continuing with build/upload. Dispatched (1):
|
3 participants
work in progress to move fa2 to torch stable abi
Warning
stable abi 2.10 does not support RNG so dropout is disabled.
related PR Dao-AILab/flash-attention#2691