fix(graph): correct shift handling in quantize_float / dequantize roundtrip#1027
Open
abhicris wants to merge 1 commit into
Open
fix(graph): correct shift handling in quantize_float / dequantize roundtrip#1027abhicris wants to merge 1 commit into
abhicris wants to merge 1 commit into
Conversation
…ndtrip
`quantize_float(elem, shift, scale)` defines `q = round(elem * 2^scale + shift)`,
but `dequantize(felt, scale, shift)` previously computed `q / 2^scale - shift`,
which is the inverse of `q = (elem + shift) * 2^scale` instead. The two
formulas agree only when `shift == 0` (the path used internally today), so the
bug was dormant for in-tree callers but `dequantize` is a public function and
silently returns wrong values for any external caller that passes a non-zero
shift.
The correct inverse is `(q - shift) / 2^scale`. With this change,
`dequantize(quantize_float(x, s, scale), scale, s) ≈ x` for every `s`, not
only `s = 0`.
While in this region I also fixed a related bound in `quantize_float`. The
representable range of `elem` is asymmetric whenever `shift != 0`:
(IntegerRep::MIN - shift) / mult <= elem <= (IntegerRep::MAX - shift) / mult
The old check used a symmetric bound `[-max_value, max_value]`, which under-
or over-rejected by `shift / mult` on each side. Replaced with the correct
asymmetric bounds.
Tests added in `src/graph/utilities.rs`:
- test_quantize_dequantize_roundtrip_zero_shift (regression-safe baseline)
- test_quantize_dequantize_roundtrip_with_shift (exercises the fix)
- test_quantize_dequantize_known_value_with_shift (hand-checked numeric case)
- test_quantize_float_asymmetric_bounds_with_shift (bound-check correctness)
Out of scope: behaviour of `quantize_float` on `f64::NAN` (still relies on the
`f64 as i128` saturating cast returning 0, as the existing test documents);
overflow behaviour at the absolute `i128::MAX` boundary, where the `as f64`
cast already loses precision. Both are pre-existing and orthogonal to the
shift-roundtrip bug fixed here.
Validated locally with the project's pinned nightly toolchain:
cargo check -> ok
cargo test --lib graph::utilities::tests::test_quantize -> 6 passed, 0 failed
cargo test --lib graph:: -> 10 passed, 0 failed
cargo test --lib fieldutils:: -> 4 passed, 0 failed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
quantize_float(elem, shift, scale)anddequantize(felt, scale, shift)insrc/graph/utilities.rsare documented as inverses, but their formulas onlyagree when
shift == 0.quantize_floatcomputesq = round(elem * 2^scale + shift).dequantizecomputeselem ≈ q / 2^scale - shift, which is the inverse ofq = (elem + shift) * 2^scale— a different fixed-point convention.So for any caller that passes a non-zero
shift,dequantizesilently returnswrong values. The bug is dormant for in-tree callers today (every internal use
of
dequantizepassesshift = 0.0— seesrc/graph/mod.rs:188,204), butdequantizeispub, theshiftparameter is part of its signature, and thedocstring promises it works for the general case.
While in this region I also fixed an asymmetric-range bug in
quantize_float'sbound check. The valid range of
elemisbut the code used a symmetric
[-max_value, max_value]check, which is correctonly when
shift == 0. For non-zeroshiftit both over- and under-rejects byshift / multon each side. Now uses the correct asymmetric bounds.Why this bug exists
The two functions were almost certainly written assuming
shift = 0(which isthe only path internally exercised today), and the asymmetric
shiftsemanticswere never re-derived when
shiftwas added to the public API. There is noround-trip test in the repo that exercises non-zero
shift, which is how theinconsistency stayed undetected.
What the fix does
dequantizenow computes(int_rep - shift) / multiplier, which is thetrue mathematical inverse of
q = elem * mult + shift.quantize_floatnow computes bothmax_valueandmin_valuefrom theactual
IntegerRep::MAX/IntegerRep::MINbounds and uses them in therange check.
the invariant is harder to forget next time.
src/graph/utilities.rs:test_quantize_dequantize_roundtrip_zero_shift— baseline; ensures thecommon
shift = 0path still roundtrips (would have stayed green witheither formula).
test_quantize_dequantize_roundtrip_with_shift— exercises the fix; thistest fails on
masterand passes here.test_quantize_dequantize_known_value_with_shift— hand-checked numericcase (
scale = 4,shift = 5.0,x = 2.0→q = 37→ recovered2.0).Pre-fix,
dequantizereturned-2.6875for the same inputs.test_quantize_float_asymmetric_bounds_with_shift— covers the bound-checkcorrection.
Validation
Tested with the repo's pinned
nightly-2025-12-01toolchain on aarch64-darwin:Behaviour-change risk
Internal callers all pass
shift = 0.0, where the new and old formulascoincide bit-for-bit. No in-tree behaviour changes.
External callers that were passing non-zero
shifttodequantizewillstart receiving correct values instead of the previous incorrect ones. That
is the intended fix, but worth flagging in the changelog.
Out of scope
quantize_float's handling off64::NAN(it relies on thef64 as i128saturating cast returning 0, as the existing
test_quantize_edge_casestest pins). Not changed.
i128::MAXboundary, whereIntegerRep::MAX as f64already loses precision. Not changed.src/graph/mod.rsflagged by chore(graph): remove panics in input conversion and batching checks #1011. Orthogonal.Test plan
cargo check— greencargo test --lib graph::utilities::tests::test_quantize— 6 passedcargo test --lib graph::— 10 passedcargo test --lib fieldutils::— 4 passed