Add support for Int4GroupwisePreshuffleTensor for fbgemm #2421
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2421
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit 851dc01 with merge base 5a50667 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
jerryzh168
force-pushed
the
jerryzh168/stack/1
branch
from
565e596 to
65a1373
Compare
facebook-github-bot
added
the
CLA Signed
jerryzh168
added
the
topic: new feature
jerryzh168
force-pushed
the
jerryzh168/stack/1
branch
from
65a1373 to
8dcecf4
Compare
jerryzh168
changed the title
jerryzh168
force-pushed
the
jerryzh168/stack/1
branch
3 times, most recently
from
44b79dd to
6ce4c7b
Compare
jerryzh168
force-pushed
the
jerryzh168/stack/1
branch
from
6ce4c7b to
027648f
Compare
| if importlib.util.find_spec("fbgemm_gpu") is None: | ||
| quantize_int4_preshuffle = None | ||
| else: | ||
| from fbgemm_gpu.experimental.gen_ai.quantize import quantize_int4_preshuffle |
There was a problem hiding this comment.
is this a prototype API? If yes, should the torchao version also be prototype? what does "experimental" mean in the folder structure here?
There was a problem hiding this comment.
it is stable and production ready, and used in production. it's just bad naming according to @jwfromm, and they have a plan to get rid of it
jerryzh168
force-pushed
the
jerryzh168/stack/1
branch
6 times, most recently
from
69a215e to
e664a1e
Compare
| shape: shape of the original Tensor | ||
|
|
||
| Note: | ||
| preshuffle means the weight is rearranged for more efficient use of loading instructions |
There was a problem hiding this comment.
it would be good to share the specifics of the preshuffle transformation, either here or via a link a user can follow
| Int4GroupwisePreshuffleTensor, | ||
| ) | ||
|
|
||
| Int4GroupwisePreshuffleTensor.__module__ = "torchao.quantization" |
There was a problem hiding this comment.
can we confirm (by actually testing it) that we can change the directory location later without breaking BC?
There was a problem hiding this comment.
I have a test that verifies the loaded weight have module path torchao.quantization.Int4GroupwisePreshuffleTensor, this (type(tensor))is used in the load code path: https://github.com/pytorch/pytorch/blob/d4b8857e51a089b7e0e722689398c5c3ada274c9/torch/_tensor.py#L262 which gives us good confidence that it would work as long as we do this
but I can do a e2e test a bit later by uploading the file in huggingface hub and change the path locally to verify as well
Summary: Note: slice is not working yet, others are working Test Plan: python test/dtypes/test_int4_groupwise_preshuffle.py Reviewers: Subscribers: Tasks: Tags: stack-info: PR: #2421, branch: jerryzh168/stack/1
jerryzh168
force-pushed
the
jerryzh168/stack/1
branch
from
e664a1e to
851dc01
Compare
Stacked PRs:
transpose_inputfrom fbgemm configs #2422Add support for Int4GroupwisePreshuffleTensor for fbgemm
Summary:
Note: slice is not working yet, others are working
Test Plan:
python test/dtypes/test_int4_groupwise_preshuffle.py
Reviewers:
Subscribers:
Tasks:
Tags: