v0.3.0 release train: SU(2) production pipeline — GPU generation, leak-free measurement, live dashboard - #30
Merged
Merged
Conversation
Config GENERATION now runs on the GPU (was: CPU GCD sweeps at ~4 min/sweep on 24^4). Metal kernel keeps the whole gauge field GPU-resident as quaternions, philox4x32-10 RNG, exact mirror of gpt's Kennedy-Pendleton su2_heat_bath conventions. Validation (gpu-metal-wip/su2_hb_validate.py, all PASS): - plaquette parity GPU vs gpt: 1.1e-07 on random config - staple kernel vs action.staple: max 1.8e-06 (float32) all sites/dirs - independent chains 8^4 beta=2.4: GPU 0.62928(35) vs CPU 0.62993(40), 1.2 sigma - 24^4 production resume: 180 therm sweeps in 40 s (~1300x vs CPU), P=0.6301 vs literature 0.6303 Driver: --gauge-gpu 1 routes one_sweep through the GPU generator (pack -> sweep -> unpack per call, so measurements/checkpoints always see a current field). Worker: pin interpreter to python3.12 (SU2_PYTHON); bare python3 resolved to system 3.9 after reboot and imported a stale cgpt. Build: swiftc -O -emit-library -o gpu-metal-wip/libcwsu2gen.dylib gpu-metal-wip/su2gen_gpu.swift Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the Jul 5-6 SIGKILLs and the Jul 6 machine panic: the measurement pipeline leaks C++ memory that macOS compresses out of RSS (compressor at ~76 GB at kill time while the process showed 28 GB RSS); memorystatus then kills the process with no crash report. - per-tdir clear_gpt_caches() in single_measurement (cuts peak RSS 95 GB -> 40 GB but does not stop total footprint growth) - --max-meas-per-run N: exit cleanly every N checkpointed measurements - tools/su2_run_loop.sh: supervisor that resumes until done=true, auto-heals SIGKILL, gives up on 3 fast failures - tools/killcatch.sh: unified-log kill-event capture used for diagnosis Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The --max-meas-per-run break fell into the final-analysis code, which wrote a bogus complete/200 checkpoint and final-results file after 6 measurements. Early exit now writes an honest production/done=false progress + checkpoint and returns before the completion block. State files repaired (checkpoint meas_done=6, bogus su2_2q_signal_9101.json quarantined). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reduces (does not eliminate) measurement-phase memory growth. Root cause found: ~5GB/sub-measurement leaks in cgpt/Grid C++ and only frees on process exit (killing the worker drops the macOS compressor 78GB->0.9GB), so 2x1 estimator (2 subs) fits but 8x2 (16 subs) OOMs. GRID_ALLOC_NCACHE=0 and Python-level clear_gpt_caches between iterations both tested, neither stops the compressor fill - the leak is below the Python cache layer. Real fix (TODO) = split estimator sub-measurements across process restarts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Even the 2x1 estimator can wedge: on Jul 7 a single measurement ran 2h at 78GB macOS compressor (all CPU spent on page decompression, ~0 progress) before manual kill. Watchdog polls compressor every 60s and SIGKILLs the worker above 60GB so the supervisor resumes from checkpoint in seconds instead of losing hours. Reinforces that the process-split estimator (the approved real fix) is needed; this keeps the interim baseline alive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…memory alone v1 killed measurements that were progressing fine during system-wide compressor spikes, creating a kill-resume loop that never cleared meas 30. v2 discriminator: THRASH = compressor > threshold AND meas_cfg_substep_done frozen for STALL_SEC. A progressing measurement is never killed regardless of memory; a true wedge (frozen substep + ballooning compressor) dies fast. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nt retention Root cause of the 9101 wedge (78GB compressor, freed only on process exit): wilson_loop_trace/wilson_loop_field/the flux shift+product path allocate a fresh full-lattice field on every g(expr)/g.cshift step — thousands of ~96-191MB alloc/free cycles per single_measurement. Python frees them, but the churn retains memory below the Python layer (gc.collect, gpt cache clears and GRID_ALLOC_NCACHE=0 all measured ineffective), linear in sub-measurement count exactly as observed (2x1 fits, 8x2 OOMs). Fix: a per-grid _LatticeWorkspace of 5 matrix + 4 complex preallocated buffers; _loop_matrix builds the rectangle with ping-pong eval/cshift into fixed destinations — identical leg order and shift decomposition, zero steady-state allocation. Flux probe shifts and W*P products use the same workspace. Holds ~1GB once instead of unbounded growth. Also: - save-cfg-every default 1 -> 0 (chain tool + app): at every-1 a 4-seed x200 24^4 chain writes 76-153GB of per-measurement configs nothing consumes; resume uses the bounded checkpoint file. - dashboard SSE: 30s socket timeout so dead clients can't pin multi-MB payload threads for the run's lifetime. - tools/test_loop_workspace_parity.py: old-vs-new numerical parity (traces, fields, shifts; threshold 1e-11). MUST pass on the mb before merge — the Mini has no working cgpt build to run it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the residual ~17MB/substep growth @claudemb measured at 24^4 full volume (climbed 41->52GB at 21% of measurement 0), which is SEPARATE from the per-step allocation churn PR #24 already fixed (that part is bit-identical + flat at 1GB on 8^4). Two changes: - clear_gpt_caches() now also clears gpt.core.foundation.lattice.cshift_plans — the one gpt cache the function was missing. It's bounded (str(grid) is dimension-based), so not the unbounded leak, but each plan can hold a full-lattice halo buffer on 24^4, pinning a few GB per measurement. - maybe_mem_report() (SU2_MEM_REPORT=1): per-tdir gpt.mem_report census of every live gpt lattice + its creation stack. This LOCALIZES the residual: if the live-lattice total climbs across tdirs, the grower is a retained gpt lattice and its stack names the line; if it stays flat while RSS climbs, the retention is below gpt (a cgpt/Grid mempool that never shrinks) and the fix belongs in Grid allocator config, not this script. Static analysis ruled out the Python layer: accumulators (loops_acc/ flux_profiles_acc) hold only scalars, and the full-volume sampler.mean is a scalar g.sum. So the residual is C++-side — this instruments it precisely. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…flation in mem_report getrusage(2): ru_maxrss is kilobytes on Linux but bytes on Darwin. The blanket *1024 made the census print 'Resident memory per rank 66675.3 GB' for a 66.7 GB process on the M4 MacBook. Print-only; no numerics touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er covariance - fitCornellPotential / fitCornellPotentialFixedE now return cov, covSingular and fitErrFn(R) = sqrt(g·Cov·g) with g = dV/dparams ([1,R,-1/R] free, [1,R] fixed-e) - drawXYError renders the ±1σ band (clipped to the plot rect so the ~1/R divergence near R->0 cannot stretch the y-scale) and reports bandDrawn so the axes caption only mentions the band when it is actually drawn - fmtPM gains a singular flag: covariance-derived errors display '± n/a (singular cov)' instead of a silent '± -'; covSingular also gates potential-stability so agreements soften when errors are unavailable - Fixed e=π/12 test row now shows σ and V₀ with their errors instead of discarding them Band math verified against an independent covariance computation to machine precision; live render verified on seed 9110 with zero console errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dashboard: ±1σ error band on Cornell V(R) fit from full parameter covariance
A parameter-less dashboard URL defaults to seed petrus-su2-signal-e; on hosts without that run (e.g. the Mini) the page silently rendered empty, with no fit and no errors. When the URL names no run and the default progress path 404s, fall back to the newest entry from /api/runs (mtime descending) and reload. Hosts without /api/runs (plain http.server) keep the old behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d edge lines On high-stats runs the 1σ band shrinks below a couple of pixels and the 0.16-alpha fill disappears entirely (petrus could not see it on the Mini). Raise fill alpha to 0.26 and stroke dashed edge lines at +/-1σ so the band reads at any width. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ctor Adds a V_eff(R,T) = -ln[W(R,T+1)/W(R,T)] vs T panel for a selected R, overlaid with the fitted constant plateau + shaded ±1σ band (the value that becomes the V(R) point). Lets you eyeball whether a plateau is real or still sloping (excited-state contamination) — the visual behind the Cornell fit's chi2/dof. One shared R selector drives BOTH R-dependent panels at once: picking R updates the Selected Wilson Loop panel (uses that R with flux_t, else its largest T) and the new plateau panel together. Reuses the existing all-sample jackknife pair estimator and the plateau fit that already produce V(R) internally, so the panel just visualizes what the fit does; band drawn via drawXYError's fitFn/fitErrFn. Verified live on mb_live/9110: V(R=6)=1.035±0.009 (chi2/dof=0.08), R=12=1.606±0.062, selector drives both panels, zero console errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dashboard: effective-potential plateau panel with shared R selector
…nder memoization Linear y made W(R,T>=2) visually sit on zero even at S/N>13 (W drops exponentially, 0.33 -> 0.0017 across T) — claudeMB's diagnosis of Petrus's 'signal only at t=1' confusion. The toggle plots log10 |running mean Re W|, clamped at the smallest positive |v| so early sign flips can't produce -Infinity; SEM band dropped in log mode (running-mean bands can cross zero). Also fixes a real memoization bug found while wiring it: renderLive's renderSig early-return didn't include selectedR or the new checkbox, so UI-only changes (R switch, log toggle) skipped re-render with stale panels until the next data tick. Signature now includes both. Verified in-browser on mb_live/9110: toggle flips both directions, labels update (clamp value shown), R-switch now updates loop axes/key immediately (previously stale until next measurement), zero console errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dashboard: log-y toggle for the Wilson-loop panel + render-memoization fix
All three probe arrows (vertex (-1,-1,-1), edge (0,0,-1), face (1,-1,-1)) pointed OUTSIDE the fundamental domain selected by tetSymLevel (dx+dy>=0, dx>=dy, dy+dz>=0, dy>=dz), i.e. into the symmetry-removed region. Replace with the in-wedge representatives verified against the full 24-element T_d orbit: vertex (1,1,-1) toward Q2, edge (1,0,0) (Q2-Q3 midpoint), face (1,1,1) (Q2Q3Q4 centroid). The reduction logic itself is exact (Monte-Carlo check: every T_d orbit has exactly one member in the wedge; volume fraction 1/24). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Viewer: fix tetrahedron probe arrows to lie inside the kept 1/24 wedge
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release train approved by Petrus in thinkoff-development 2026-07-07 20:34 UTC ("release go"), proposed 20:22 UTC. Brings main (last moved May 14) up to the production state that has been running the 24⁴ campaign.
Generation
Measurement reliability
Dashboard
Geometry viewer
🤖 Generated with Claude Code