Extend on-device sampling support for dual QPC VLMs #597
+203
−52
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
quic-xiyushi
requested review from
ochougul,
quic-amitraj,
quic-hemagnih and
quic-rishinr
as code owners
quic-xiyushi
force-pushed
the
on-device-sampling-vlm
branch
2 times, most recently
from
af8e673 to
df3501a
Compare
quic-hemagnih
requested changes
Oct 30, 2025
Contributor
quic-hemagnih
left a comment
quic-hemagnih
left a comment
There was a problem hiding this comment.
Can you please add the CI test cases.
Signed-off-by: quic-xiyushi <[email protected]>
quic-xiyushi
force-pushed
the
on-device-sampling-vlm
branch
from
df3501a to
d722a5a
Compare
Signed-off-by: quic-xiyushi <[email protected]>
quic-xiyushi
force-pushed
the
on-device-sampling-vlm
branch
from
d722a5a to
e06e175
Compare
Signed-off-by: quic-sanising <[email protected]> Signed-off-by: sanising <[email protected]>
quic-sanising
force-pushed
the
on-device-sampling-vlm
branch
from
900aee5 to
3e242ce
Compare
Signed-off-by: quic-xiyushi <[email protected]>
Signed-off-by: sanising <[email protected]>
ochougul
requested changes
Nov 12, 2025
Comment on lines
+1606
to
+1607
| if kwargs.pop("qaic_config", None): | ||
| raise NotImplementedError("On-device sampling is not supported for single QPC multimodal models yet.") |
Contributor
There was a problem hiding this comment.
Isn't qaic_config used for spec decoding as well?
Are we not supporting spec decode for single qpc?
Should we just error out or check what's passed in the config?
| self.config = model.config | ||
| self.vision_model = QEffVisionEncoderForTextImageToTextModel(model, **kwargs) | ||
| self.lang_model = QEffCausalLMForTextImageToTextModel(model, **kwargs) | ||
| self.lang_model = QEffCausalLMForTextImageToTextModel(model, continuous_batching=continuous_batching, **kwargs) |
Contributor
There was a problem hiding this comment.
replace line 984-985 with following
if kwargs.pop("full_batch_size", None):
continuous_batching = True
warnings.warn(
"full_batch_size argument is deprecated. Use continuous_batching=True instead.", DeprecationWarning, 2
)
Comment on lines
+785
to
+790
| def get_sampling_inputs_and_outputs( | ||
| self, | ||
| example_inputs: Dict[str, torch.Tensor], | ||
| output_names: List[str], | ||
| dynamic_axes: Dict[str, Dict[int, str]], | ||
| ): |
Contributor
There was a problem hiding this comment.
how is this method different from QEFFAutoModelForCausalLM.get_sampling_inputs_and_outputs
Can we combine these and create a different method in utils or spd_utils for this?
remove code duplication?
Contributor
There was a problem hiding this comment.
can you add a test for intern model i,e VLM in dual qpc mode?
| QEffGPTJForCausalLM, | ||
| QEffGraniteForCausalLM, | ||
| QEffGraniteMoeForCausalLM, | ||
| QEffInternDecoderWrapper, |
Contributor
There was a problem hiding this comment.
Does this mean we are enabling sampling only for intern model?
Will other VLMs also be supported?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.