fix(su2): zero-allocation measurement loops — stop the ~5GB/measurement retention - #24
Merged
ThinkOffApp merged 1 commit intoJul 7, 2026
Conversation
…nt retention Root cause of the 9101 wedge (78GB compressor, freed only on process exit): wilson_loop_trace/wilson_loop_field/the flux shift+product path allocate a fresh full-lattice field on every g(expr)/g.cshift step — thousands of ~96-191MB alloc/free cycles per single_measurement. Python frees them, but the churn retains memory below the Python layer (gc.collect, gpt cache clears and GRID_ALLOC_NCACHE=0 all measured ineffective), linear in sub-measurement count exactly as observed (2x1 fits, 8x2 OOMs). Fix: a per-grid _LatticeWorkspace of 5 matrix + 4 complex preallocated buffers; _loop_matrix builds the rectangle with ping-pong eval/cshift into fixed destinations — identical leg order and shift decomposition, zero steady-state allocation. Flux probe shifts and W*P products use the same workspace. Holds ~1GB once instead of unbounded growth. Also: - save-cfg-every default 1 -> 0 (chain tool + app): at every-1 a 4-seed x200 24^4 chain writes 76-153GB of per-measurement configs nothing consumes; resume uses the bounded checkpoint file. - dashboard SSE: 30s socket timeout so dead clients can't pin multi-MB payload threads for the run's lifetime. - tools/test_loop_workspace_parity.py: old-vs-new numerical parity (traces, fields, shifts; threshold 1e-11). MUST pass on the mb before merge — the Mini has no working cgpt build to run it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
ThinkOffApp
added a commit
that referenced
this pull request
Jul 7, 2026
Addresses the residual ~17MB/substep growth @claudemb measured at 24^4 full volume (climbed 41->52GB at 21% of measurement 0), which is SEPARATE from the per-step allocation churn PR #24 already fixed (that part is bit-identical + flat at 1GB on 8^4). Two changes: - clear_gpt_caches() now also clears gpt.core.foundation.lattice.cshift_plans — the one gpt cache the function was missing. It's bounded (str(grid) is dimension-based), so not the unbounded leak, but each plan can hold a full-lattice halo buffer on 24^4, pinning a few GB per measurement. - maybe_mem_report() (SU2_MEM_REPORT=1): per-tdir gpt.mem_report census of every live gpt lattice + its creation stack. This LOCALIZES the residual: if the live-lattice total climbs across tdirs, the grower is a retained gpt lattice and its stack names the line; if it stays flat while RSS climbs, the retention is below gpt (a cgpt/Grid mempool that never shrinks) and the fix belongs in Grid allocator config, not this script. Static analysis ruled out the Python layer: accumulators (loops_acc/ flux_profiles_acc) hold only scalars, and the full-volume sampler.mean is a scalar g.sum. So the residual is C++-side — this instruments it precisely. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (matches @claudemb's localization exactly)
wilson_loop_trace/wilson_loop_field/ the flux shift+product path allocate a fresh full-lattice field on every step — thousands of ~96–191 MB alloc/free cycles persingle_measurement. Python frees the objects, but the churn retains memory below the Python layer: ~5 GB per measurement, linear in sub-measurement count (2×1 fits, 8×2 OOMs), recovered only at process exit. That's whygc.collect(), gpt cache clears andGRID_ALLOC_NCACHE=0all did nothing — there is no Python-level object leaking; it's allocator churn.Fix
Per-grid
_LatticeWorkspace(5 matrix + 4 complex buffers, ~1 GB held once on 24^4 SU(3) double)._loop_matrixbuilds the rectangle via ping-pongg.eval/dst-formg.cshiftinto fixed destinations — identical leg order and shift decomposition to the old code. Flux probe shifts and W·P products reuse the same workspace. Steady-state per-measurement allocation: zero.Also:
save-cfg-everydefault 1→0 (a 4-seed ×200 chain writes 76–153 GB of configs nothing consumes — the independent leak-hunt's top finding); dashboard SSE 30 s socket timeout (dead clients pinned multi-MB payload threads).Gate before merge — @claudemb
python3 tools/test_loop_workspace_parity.pyin your working env — must print PARITY PASS (thresh 1e-11).single_measurementon the 8×2 config: peak compressor should stay flat instead of +5 GB per sub-measurement.wilson_loop_fieldnow returns a shared buffer (valid until the next loop call on that grid) — flux uses it within-iteration only, but flag anything else that stores it.🤖 Generated with Claude Code