What it is
A new pluggable transform (strategy/transforms.py) — call it power-rsvd — that
extends the built-in rsvd range finder with q steps of randomized subspace
iteration (Halko–Martinsson–Tropp 2011, Alg. 4.4). Where rsvd takes a single
sketch Y = AΩ (and AᵀΩ, BᵀΩ) and orthonormalizes, power-rsvd sharpens each
sketch toward the dominant singular directions before building the basis:
Y = AΩ
repeat q times: # re-orthonormalize between products for stability
Y = orth( A (Aᵀ Y) )
Q_colA = Y
(and the analogous (AᵀA)^q AᵀΩ, (BᵀB)^q BᵀΩ for the row spaces rsvd already
sketches). Concatenate the three blocks, thin-QR → the (n, m) orthonormal Q.
Why it should score (target track: decaying-spectrum, floor 0.90)
The reference regime the README calls out is full-rank random, where no subspace
method wins. But the decaying-spectrum track (singular values ~ k^-alpha, floor
0.90) is exactly where a single-shot sketch leaks tail energy into Q and rsvd
struggles to clear the floor. Each power step multiplies the captured spectrum by an
extra factor of the singular values, so after q iterations the subspace aligns with
the top directions far better — strictly lower Frobenius error at the same M, with
no change to the reconstruction Ĉ = P A P B P.
The cost is honest and bounded: q extra passes over A/B add ~4q·n²m FLOPs to basis
construction (reported in basis_flops), and q extra O(n²m) matmuls to latency —
both still O(N²M) ≪ O(N³), so power-rsvd still dominates exact on FLOPs,
latency, and VRAM while lifting accuracy above the floor rsvd can't clear. That is a
real improvement, not accuracy-for-speed laundering.
Sketch of the implementation
- Subclass
Transform, name = "power-rsvd", register it.
basis(self, n, m, backend, dtype, A, B, frac): split M across col(A)/row(A)/row(B)
as rsvd does; for each, run q re-orthonormalized subspace-iteration steps using the
existing streaming stream_gemm_right / stream_gemm_left_t helpers (pass frac
through, so it honors --vram-fraction); concatenate + thin-QR.
basis_flops(self, n, m): rsvd's 2n²m + 2nm² plus ~4q·n²m for the q steps —
reported so the FLOP savings are not overstated.
q as a constructor knob (default q=2), seeded like the other transforms.
Acceptance
python -m eval --fill decaying-spectrum --data-rank 64 --rank-m 256 should show
power-rsvd clearing the 0.90 floor with accuracy above rsvd at the same M, and
still faster/less-vram/fewer-flops than exact (an admitted improvement). Full-rank
behavior is unchanged in spirit — it will still gate there (as everything does), which
is expected.
References
- Halko, Martinsson, Tropp, "Finding Structure with Randomness" (2011), Alg. 4.3–4.4
(power iteration / subspace iteration for range finders).
What it is
A new pluggable transform (
strategy/transforms.py) — call itpower-rsvd— thatextends the built-in
rsvdrange finder with q steps of randomized subspaceiteration (Halko–Martinsson–Tropp 2011, Alg. 4.4). Where
rsvdtakes a singlesketch
Y = AΩ(andAᵀΩ,BᵀΩ) and orthonormalizes,power-rsvdsharpens eachsketch toward the dominant singular directions before building the basis:
(and the analogous
(AᵀA)^q AᵀΩ,(BᵀB)^q BᵀΩfor the row spacesrsvdalreadysketches). Concatenate the three blocks, thin-QR → the
(n, m)orthonormalQ.Why it should score (target track: decaying-spectrum, floor 0.90)
The reference regime the README calls out is full-rank random, where no subspace
method wins. But the decaying-spectrum track (singular values ~
k^-alpha, floor0.90) is exactly where a single-shot sketch leaks tail energy into
Qandrsvdstruggles to clear the floor. Each power step multiplies the captured spectrum by an
extra factor of the singular values, so after q iterations the subspace aligns with
the top directions far better — strictly lower Frobenius error at the same M, with
no change to the reconstruction
Ĉ = P A P B P.The cost is honest and bounded: q extra passes over A/B add
~4q·n²mFLOPs to basisconstruction (reported in
basis_flops), and q extraO(n²m)matmuls to latency —both still
O(N²M) ≪ O(N³), sopower-rsvdstill dominates exact on FLOPs,latency, and VRAM while lifting accuracy above the floor
rsvdcan't clear. That is areal improvement, not accuracy-for-speed laundering.
Sketch of the implementation
Transform,name = "power-rsvd", register it.basis(self, n, m, backend, dtype, A, B, frac): split M across col(A)/row(A)/row(B)as
rsvddoes; for each, run q re-orthonormalized subspace-iteration steps using theexisting streaming
stream_gemm_right/stream_gemm_left_thelpers (passfracthrough, so it honors
--vram-fraction); concatenate + thin-QR.basis_flops(self, n, m):rsvd's2n²m + 2nm²plus~4q·n²mfor the q steps —reported so the FLOP savings are not overstated.
qas a constructor knob (defaultq=2), seeded like the other transforms.Acceptance
python -m eval --fill decaying-spectrum --data-rank 64 --rank-m 256should showpower-rsvdclearing the 0.90 floor with accuracy aboversvdat the same M, andstill
faster/less-vram/fewer-flops than exact(an admitted improvement). Full-rankbehavior is unchanged in spirit — it will still gate there (as everything does), which
is expected.
References
(power iteration / subspace iteration for range finders).