[Sampling] Softmax Predicate patch #3383
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Issue is that the config read in the compile phase is a model-specific config object, that is defined in each models' implementation. Maybe the solution is not to add another field, but rather to overload / overwrite the vocab size parameter
babusid
commented
Nov 24, 2025
Comment on lines
+130
to
132
| logger.info("TOP LEVEL MODEL CONFIG BEFORE OVERRIDES: %s", str(model_config)) | ||
| _kwargs = getattr(model_config, "kwargs", {}) | ||
| model_config = args.overrides.apply(model_config) |
Contributor
Author
There was a problem hiding this comment.
Just a note here, i noticed that this override wipes out any kwargs in the original model_config. This PR isn't the place to address it probably, but I just wanted to call it out.
MasterJH5574
approved these changes
Nov 25, 2025
Member
MasterJH5574
left a comment
MasterJH5574
left a comment
There was a problem hiding this comment.
Thank you @babusid for the enhancement!
MasterJH5574
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Solves the issue of softmax assigning probability to reserved tokens and messing up the sampling kernel. This patch extracts the active vocabulary from the HF tokenizer, and uses that to zero out the reserved tokens, and only generate probabilities for valid "active" tokens.