Skip to content

[strategy] power-iteration range finder: q subspace-iteration steps to beat rsvd on decaying spectra #263

Description

@real-venus

What it is

A new pluggable transform (strategy/transforms.py) — call it power-rsvd — that
extends the built-in rsvd range finder with q steps of randomized subspace
iteration
(Halko–Martinsson–Tropp 2011, Alg. 4.4). Where rsvd takes a single
sketch Y = AΩ (and AᵀΩ, BᵀΩ) and orthonormalizes, power-rsvd sharpens each
sketch toward the dominant singular directions before building the basis:

Y = AΩ
repeat q times:            # re-orthonormalize between products for stability
    Y = orth( A (Aᵀ Y) )
Q_colA = Y

(and the analogous (AᵀA)^q AᵀΩ, (BᵀB)^q BᵀΩ for the row spaces rsvd already
sketches). Concatenate the three blocks, thin-QR → the (n, m) orthonormal Q.

Why it should score (target track: decaying-spectrum, floor 0.90)

The reference regime the README calls out is full-rank random, where no subspace
method wins. But the decaying-spectrum track (singular values ~ k^-alpha, floor
0.90) is exactly where a single-shot sketch leaks tail energy into Q and rsvd
struggles to clear the floor. Each power step multiplies the captured spectrum by an
extra factor of the singular values, so after q iterations the subspace aligns with
the top directions far better — strictly lower Frobenius error at the same M, with
no change to the reconstruction Ĉ = P A P B P.

The cost is honest and bounded: q extra passes over A/B add ~4q·n²m FLOPs to basis
construction (reported in basis_flops), and q extra O(n²m) matmuls to latency —
both still O(N²M) ≪ O(N³), so power-rsvd still dominates exact on FLOPs,
latency, and VRAM while lifting accuracy above the floor rsvd can't clear. That is a
real improvement, not accuracy-for-speed laundering.

Sketch of the implementation

  • Subclass Transform, name = "power-rsvd", register it.
  • basis(self, n, m, backend, dtype, A, B, frac): split M across col(A)/row(A)/row(B)
    as rsvd does; for each, run q re-orthonormalized subspace-iteration steps using the
    existing streaming stream_gemm_right / stream_gemm_left_t helpers (pass frac
    through, so it honors --vram-fraction); concatenate + thin-QR.
  • basis_flops(self, n, m): rsvd's 2n²m + 2nm² plus ~4q·n²m for the q steps —
    reported so the FLOP savings are not overstated.
  • q as a constructor knob (default q=2), seeded like the other transforms.

Acceptance

python -m eval --fill decaying-spectrum --data-rank 64 --rank-m 256 should show
power-rsvd clearing the 0.90 floor with accuracy above rsvd at the same M, and
still faster/less-vram/fewer-flops than exact (an admitted improvement). Full-rank
behavior is unchanged in spirit — it will still gate there (as everything does), which
is expected.

References

  • Halko, Martinsson, Tropp, "Finding Structure with Randomness" (2011), Alg. 4.3–4.4
    (power iteration / subspace iteration for range finders).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:evalProtected evaluator, metrics, artifacts, and policy (eval/, benchmarks/)area:strategySmart strategies / transforms (strategy/)status:triageNewly opened, not yet triagedtype:docsDocumentation-only change; not an accepted miner PR lanetype:strategyNew smart strategy / transform proposal

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions