[MISC] Fix broken Apple Metal CI. by hughperkins · Pull Request #3054 · Genesis-Embodied-AI/genesis-world

hughperkins · 2026-07-16T10:55:59Z

The macOS CI runners expose a virtualized 'Apple Paravirtual device' GPU. Running test_deformable_uv_textures (FEM Elastic + PBD Cloth deformables rendered through the rasterizer) drives that virtualized Metal driver into a persistent, VM-wide broken state: afterwards newComputePipelineStateWithFunction returns 'Compilation failed (code=2)' for every kernel - even a trivial copy kernel. The wedge is unrecoverable within the VM (survives MTLCompilerService restart and process exit) and cascades to every later test in the job, turning the Metal GPU job permanently red.

The failure was isolated to exactly this test. Skip it on the macOS Metal backend until the underlying Apple driver bug is resolved.

Description

Full analysis by AI:

Metal Genesis CI failure: experiments and results

Author: AI (2026-07-15).
Subject: Genesis CI job macos-15-3.12-gpu-ndarray failing on pull requests.
Debug branches: hp/metal-ci-debug (workflow .github/workflows/metal-debug.yml,
scripts scripts/metal_repro.py, scripts/metal_scene.py) and hp/metal-limit
(workflow .github/workflows/metal-limit.yml, scripts
scripts/metal_device_info.swift, scripts/metal_buffer_sweep.py,
scripts/metal_adstack.py, scripts/metal_compiler_stress.py) on
hughperkins/genesis-world.

This document records the experiments run and their observed results. Sections
0-9 record experiments with no analysis. Sections 10-16 (added 2026-07-15, second
session) isolate the trigger.

Sections 18-24 (added 2026-07-15, third session) CORRECT the earlier
conclusion. The second-session conclusion (concurrency-driven GPU-compiler
contention, fixable by retrying newComputePipelineStateWithFunction) is
disproven by direct experiment: an in-process retry fails 6/6 despite firing
120x/run, a single worker fails 5/5 on every host, and the offline cache is not
the discriminator. The real cause is a persistent, VM-wide, unrecoverable wedge
of the Apple Paravirtual GPU device, triggered by exactly one test
(test_deformable_uv_textures). See the revised "## Conclusion".

0. Observed CI failure

Genesis org repo Genesis-Embodied-AI/genesis-world.

Run 29365292338 (PR [BUG FIX] Fix fastcache support. #3034). Job conclusions: all jobs success except
macos-15-3.12-gpu-ndarray = failure. The other macOS jobs
(macos-15-3.12-cpu-ndarray, macos-15-3.12-cpu-field,
macos-15-3.10/3.11/3.13-cpu-ndarray), the Linux jobs, and the Windows job
all passed.

In the failing job, the repeated error text is:

RuntimeError: [runtime.cpp:CompiledQuadrantsKernel@306] Failed to create
pipeline for kernel task 'ext_arr_to_ndarray_matrix_c20_0_0_t00'
(RhiResult=-1). The SPIR-V shader was rejected by the backend driver; see the
preceding RHI log for the underlying diagnostic. On Metal, a common cause is
exceeding Apple's MSL per-thread Function-scope footprint in reverse-mode AD
kernels that use the adstack pipeline.

Job environment banner: Running on [Apple M1 (Virtual)] with backend gs.metal. Device memory: 7.00 GB.
Job software versions: quadrants 1.1.1 (llvm 22.1.0, commit 35b1a060),
torch 2.13.0, python 3.12.10, runner image macos-15-arm64 20260706.0213.1.

Environment variables set on the failing job include GS_ENABLE_NDARRAY=1,
GS_TORCH_FORCE_CPU_DEVICE=1, QD_OFFLINE_CACHE=1,
QD_OFFLINE_CACHE_FILE_PATH=.cache/quadrants, QD_ENABLE_METAL=1, QD_DEBUG=0.
The test command is
pytest -v -ra --logical --dev --backend gpu -m 'required and not slow' --forked ./tests.

The generic workflow restores .cache under the key
${OS_FAMILY}-${MACHINE_ARCH}-${QUADRANTS_VERSION} (= Darwin-arm64-1.1.1), and
saves that same key from the macos-15 cpu / ndarray / py3.12 job only.

0.0 Underlying RHI diagnostic (from org logs)

Grepping the org PR #3042 failing job log for the quadrants RHI log line that
precedes each runtime.cpp exception yields, for every failing kernel:

[metal_device.mm:create_compute_pipeline@181] cannot create compute pipeline
state for kernel 'ext_arr_to_ndarray_matrix_c20_0_0_t00' (msl_bytes=1641):
Compilation failed (code=2)

The three distinct failing kernels report msl_bytes of 1641, 1360, 1020.
Count of cannot create compute pipeline state for kernel lines: 181 (for the
ext_arr_to_ndarray_matrix_c20 kernel) plus one each for the other two.
Immediately preceding each failure, the Genesis log shows
Materializing kernel ext_arr_to_ndarray_matrix_c20_0 in AutodiffMode.NONE,
i.e. the failure occurs on the first kernel materialized during
Building scene inside a freshly-forked test process.
The quadrants source (quadrants/rhi/metal/metal_device.mm,
create_compute_pipeline) attributes this line to
newComputePipelineStateWithFunction returning nil with the NSError
localizedDescription = Compilation failed and code = 2.

0.1 Other PRs

Reading the macos-15-3.12-gpu-ndarray check-run conclusion on each PR head SHA:

PR #3042 failing job (run 29423732829, job 87380938028) log contents:

Distinct failing kernel tasks:
ext_arr_to_ndarray_matrix_c20_0_0_t00, matrix_to_ext_arr_c22_42_0_t00,
ndarray_to_ext_arr_c8_3_0_t00.
549 lines containing SPIR-V shader was rejected; 549 lines containing
RhiResult=-1.
Terminal per-test result counts: 304 PASSED, 300 FAILED.
Sequence of per-test results (run-length encoded, PASSED/FAILED): the run
begins with 176 consecutive PASSED, after which PASSED and FAILED runs
interleave for the remainder (e.g. 13 FAILED, 3 PASSED, 1 FAILED, 6 FAILED, ..., and later 70 PASSED, 2 FAILED). There is no single point after which
every subsequent test fails.
Device banner: Running on [Apple M1 (Virtual)] ... Device memory: 7.00 GB;
pytest reports created: 2/2 workers.

0.2 Date range (from PR head-SHA check-runs)

PR [MISC] Add loop_config kernel names for profiling. #2950 (merged 2026-06-15): macos-15-*-gpu-ndarray = success.
PR [BUG FIX] Render each heterogeneous variant in its own environment. #2958 (created 2026-06-16) and PRs up to Revert "[MISC] Significantly improve non-batched CPU simulation speed (up to 30%). (#2991)" #3000+: failure.

1. Local, real Apple Silicon Metal (isolated copy kernel)

Machine: local Apple Silicon Mac (darwin 25.5.0).

Ran qd.init(arch=qd.metal, debug=True) then, for element types matching the CI
failing kernel names, allocated a quadrants ndarray, called .from_numpy(host)
and .to_numpy(), and compared:

qd.Matrix.ndarray(4, 5) (20 components, ext_arr_to_ndarray_matrix_c20)
qd.Matrix.ndarray(2, 11) (22 components)
qd.Vector.ndarray(8) (8 components)
qd.Vector.ndarray(3) (3 components)

Result with quadrants 0.8.0: all pass, roundtrip max error 0.0.

Result with quadrants 1.1.1 (separate py3.10 venv): all pass, roundtrip max
error 0.0.

In both, quadrants logged Out-of-bound access checking is not supported on arch=metal.

quadrants 1.1.1 provides wheels for cp310, cp311, cp312, cp313; no cp314 wheel
exists.

2. Fork CI, isolated copy kernel, macos-15 virtualized runner

Repo hughperkins/genesis-world, runs-on: macos-15. sysctl -n machdep.cpu.brand_string prints Apple M1 (Virtual). Installed
quadrants==1.1.1 and numpy only. Script scripts/metal_repro.py allocates
qd.Matrix.ndarray(4, 5) and runs .from_numpy / .to_numpy.

2.1 Init-flag sweep (run `29439255683`)

Configs: baseline, no_advanced_opt, no_scalarize, no_fast_math,
cfg_opt. Result: all 5 jobs success, each printing
RESULT ext_arr_to_ndarray_matrix_c20 OK roundtrip_err=0.000e+00.

2.2 Genesis-exact init flags plus ad-stack sweep (run `29439514762`)

Init mirrored genesis/__init__.py on Metal: enable_fallback=False,
debug=False, check_out_of_bound=False, force_scalarize_matrix=True,
advanced_optimization=True, cfg_optimization=False, fast_math=False,
cpu_max_num_threads=1, ad_stack_experimental_enabled=True,
ad_stack_sparse_threshold_bytes=200_000_000.

Configs: genesis_exact, no_ad_stack, ad_stack_small_bytes
(ad_stack_sparse_threshold_bytes=1_000_000), no_advanced_opt,
no_scalarize. Result: all 5 jobs success, roundtrip error 0.0.

3. Fork CI, full Genesis install, trivial scene and single test

Installed pip install ".[dev,render,usd]" plus torch --upgrade
(index download.pytorch.org/whl/cpu) plus matplotlib<3.11. quadrants 1.1.1.

Attempt with pytest tests/test_rigid_physics.py::test_gravity (run
29439730976): pytest reported no tests ran / exit code 5. On this branch
the file is tests/rigid/test_dynamics.py (tests were reorganized into
per-component folders); the old path collects 0 items.
Corrected attempt (run 29440166086), job success:
- scripts/metal_scene.py: gs.init(backend=gs.metal, debug=True, logging_level="debug"), scene with Plane + Sphere, build(), 3
  step()s. Ran to SCENE_STATUS=PASS.
- pytest tests/rigid/test_dynamics.py::test_gravity --logical --dev --backend gpu --forked: passed.

4. Fork CI, full test file, CI-like parallelism (run `29440658358`)

Command per config:
pytest -v -ra --logical --dev --backend gpu --forked -m 'required and not slow' --numprocesses=$NPROC tests/rigid/test_collision.py.

ci_like (QD_OFFLINE_CACHE=1, --numprocesses=auto): completed, job
success.
serial (QD_OFFLINE_CACHE=1, --numprocesses=0), no_cache
(QD_OFFLINE_CACHE=0, --numprocesses=auto), serial_no_cache
(QD_OFFLINE_CACHE=0, --numprocesses=0): cancelled before completion (each
was still running at ~15 min). No result recorded.

5. Fork CI, cross-backend offline cache (run `29442109831`)

Two configs, both QD_OFFLINE_CACHE=1,
QD_OFFLINE_CACHE_FILE_PATH=.cache/quadrants, running
pytest ... --backend gpu --forked -m 'required and not slow' tests/rigid/test_collision.py:

metal_only_cache: Metal run only, starting from an empty cache.
cross_backend_cache: first runs tests/rigid/test_collision.py with
--backend cpu to populate .cache/quadrants, then runs the same file with
--backend gpu.

Result: both jobs success.

cross_backend_cache: the CPU warm step populated
.cache/quadrants/python_side_cache/*.cache.txt; the subsequent Metal run
reported 62 passed, 1 xfailed.
metal_only_cache: Metal run reported 62 passed, 1 xfailed.

5b. Runner comparison (org vs fork)

The org failing job (29423732829) and the fork jobs both print the same device
banner Running on [Apple M1 (Virtual)] ... Device memory: 7.00 GB and pytest
created: 2/2 workers.

6. Fork CI, full `./tests` suite mirroring the org GPU job (run `29443749352`, commit `a9eb95cc`, branch `hp/metal-ci-debug`)

Command mirrors the org generic.yml gpu job:
pytest -v -ra --logical --dev --backend gpu -m 'required and not slow' --forked ./tests, with the same env (QD_OFFLINE_CACHE=1,
QD_OFFLINE_CACHE_FILE_PATH=.cache/quadrants, QD_ENABLE_METAL=1,
GS_ENABLE_NDARRAY=1, GS_TORCH_FORCE_CPU_DEVICE=1, QD_DEBUG=0), starting from
an empty cache.

Job completed in about 70 minutes. Result: 304 passed, 150 failed. The failing
kernels and their exact RHI diagnostics match the org run:

182  cannot create compute pipeline state for kernel 'ext_arr_to_ndarray_matrix_c20_0_0_t00' (msl_bytes=1641): Compilation failed (code=2)
  1  cannot create compute pipeline state for kernel 'kernel_clear_external_force_c284_0_0_t00' (msl_bytes=1968): Compilation failed (code=2)
  1  cannot create compute pipeline state for kernel 'fill_ndarray_c2_71_0_t00' (msl_bytes=708): Compilation failed (code=2)

All three are small copy/fill/clear kernels (msl_bytes 708-1968). This run did
not capture the system log.

7. Fork CI, Metal device limits and buffer/adstack size sweeps (run `29444269087`, branch `hp/metal-limit`)

runs-on: macos-15.

7.1 Device limits (`scripts/metal_device_info.swift`)

name                          Apple Paravirtual device
maxBufferLength_bytes         3758096384      (3584 MB)
recommendedMaxWorkingSetSize  5010800640      (4778 MB)
maxThreadgroupMemoryLength    32768           (32 KB)
hasUnifiedMemory              true
supportsFamily(.apple7)       false
supportsFamily(.apple8)       false
supportsFamily(.mac2)         true

7.2 Single-buffer size sweep (`scripts/metal_buffer_sweep.py`, quadrants 1.1.1)

Allocates one qd.field(qd.f32, shape=n) of the given size on Metal, fills it in
a kernel, sums via to_numpy. Sizes 50, 100, 150, 200, 300, 500, 1000 MB: all
jobs success (each prints RESULT MB=<n> OK).

7.3 Reverse-mode adstack size sweep (`scripts/metal_adstack.py`, quadrants 1.1.1)

Runs a reverse-mode autodiff kernel whose backward pass uses a dynamic-loop-bound
adstack, with ad_stack_experimental_enabled=True and
ad_stack_sparse_threshold_bytes set to 10M, 50M, 100M, 150M, 200M: all jobs
success (each prints RESULT ad_stack_bytes=<n> OK grad0=2.000).

8. Fork CI, synthetic Metal compiler-service stress (run `29447591837`, branch `hp/metal-limit`)

scripts/metal_compiler_stress.py compiles structurally distinct kernels (unique
compile-time unroll count per iteration, offline_cache=False) so each requires
a fresh MSL translation and newComputePipelineStateWithFunction. Two configs:

single_process: one process compiling 4000 distinct kernels. Job success
(RESULT compiled 4000 distinct kernels OK).
repeated_process: 80 sequential processes, each compiling 60 distinct
kernels. Job success.

9. Fork CI, real heavy folders (rigid + rendering) with system-log capture (run `29448183249`, branch `hp/metal-limit`)

Result: 267 passed, 19 failed. The reproduction matches the org and the full-suite
run:

18  cannot create compute pipeline state for kernel 'ext_arr_to_ndarray_matrix_c20_0_0_t00' (msl_bytes=1641): Compilation failed (code=2)
 2  cannot create compute pipeline state for kernel 'kernel_clear_external_force_c274_0_0_t00' (msl_bytes=1968): Compilation failed (code=2)
 2  cannot create compute pipeline state for kernel 'fill_ndarray_c2_105_0_t00' (msl_bytes=735): Compilation failed (code=2)

System-log observations for the run window (job start to end):

EXC_RESOURCE, high watermark, phys_footprint: no matching log lines (the
only occurrences of these tokens in the job log are the echoed grep filter
command itself).
The jetsam / memorystatus filter returned only runningboardd background
lines: Ignoring jetsam update because this process is not memory-managed
(for anon<Python> and various osservice processes) and
memorystatus_control error: MEMORYSTATUS_CMD_CONVERT_MEMLIMIT_MB(-1|0) returned -1 22 (Invalid argument). None reference a memory limit being crossed
by a test process.
MTLCompiler|GPUCompiler|AGXCompiler lines matching
fail|exit|crash|kill|interrupt: the only matches are at 21:24:12 (job
teardown), where launchd SIGKILLs the com.apple.MTLCompilerService
instances with the note sent by launchd during teardown of process-scoped services after host exited, ran for ~3440000 ms, and the XPC connection is
invalidated because the client process ... either cancelled the connection or exited. No MTLCompilerService crash, kill, or compilation-failure log line
appears during the pytest run itself.
The XPC + interrupt|invalid|CONNECTION filter returned no matching log
lines from the run.

10. Fork CI, repeated init/teardown in one long-lived process (run `29453644680`, branch `hp/metal-reinit`)

Motivation: on macOS the Genesis conftest.py forces config.option.forked = False (fork is only honored on Linux), so each pytest-xdist worker runs many
tests in a single long-lived process, calling gs.init(metal) / gs.destroy()
(hence qd.init / qd.reset) once per test. None of sections 1-9 exercised
that repeated-init axis in one process. Two scripts mirror it.

scripts/metal_reinit_qd.py (pure quadrants 1.1.1): loop qd.init(metal)
with the genesis-exact flags of section 2.2, do a qd.Matrix.ndarray(4, 5)
from_numpy/to_numpy roundtrip (the ext_arr_to_ndarray_matrix_c20 kernel),
qd.reset(), for 600 cycles. Matrix nocache_1p (cache off, 1 proc),
cache_1p (cache on, 1 proc), cache_2p (cache on, 2 concurrent procs).
- Result: 0 Metal pipeline-creation failures in any config (no
  Compilation failed (code=2) / RhiResult=-1 in any log); the Metal
  roundtrip succeeded on every cycle. (The script's own check used exact
  float equality float(out)==-9.81 against a float32 roundtrip and so
  reported AssertionError on every cycle; that is a bug in the check, not a
  Metal failure. Corrected in a later commit.)
scripts/metal_reinit_gs.py (full Genesis): loop gs.init(metal) -> build a
Plane+Sphere rigid scene -> 3 step()s -> gs.destroy(), for 300 cycles.
- Result: 0/300 failures, 0 compile failures. Cycle 0 took ~112 s (cold
  kernel compilation); each later cycle ~2 s.

11. Fork CI, init/reset x distinct-kernel churn in one process (run `29454911916`, branch `hp/metal-churn`)

scripts/metal_churn_qd.py (pure quadrants 1.1.1), 800 cycles, one process,
combining the two axes that sections 8 and 10 tested only separately:

reinit_distinct: qd.init/qd.reset every cycle AND a structurally
distinct kernel (unique compile-time unroll) compiled every cycle.
reinit_same: qd.init/qd.reset every cycle, same kernel.
noreinit_distinct: single qd.init, a distinct kernel every cycle.

Result: all three modes 0/800, 0 compile failures. Repeated init/teardown,
distinct-kernel compilation, and their combination in a single process do not
reproduce.

12. Fork CI, concurrency synthetic + real-folder split (run `29455302366`, branch `hp/metal-conc`)

Two job families.

conc-np{1,2,3}: launch N concurrent processes (each its own offline-cache
dir), each running metal_churn_qd.py reinit_distinct for 800 cycles on the
one Apple Paravirtual GPU. Result: all procs 0/800, 0 compile failures at
1, 2, and 3 concurrent processes. Concurrent Metal compilation alone (no
rendering) does not reproduce.
suite-*: real Genesis install, org gpu-job env, default parallelism
(created: 2/2 workers).
- rendering_auto (tests/rendering): reproduces - 25 failed, 4 passed,
  14 skipped in ~170 s. Failing kernels: ext_arr_to_ndarray_matrix_c20
  (dominant), kernel_init_geom_fields_c254, kernel_pbd_rigid_collide_c84.
  Ordered outcomes show a few RASTERIZER tests pass, then after the first
  failure in a worker essentially every later test in that worker fails.
- rigid_auto (tests/rigid, -n auto, 2 workers, no rendering):
  258 passed, 0 compile failures. Two-worker concurrency without rendering
  does not reproduce.

13. Fork CI, single-process rasterizer isolation (run `29455788553`, branch `hp/metal-raster`)

scripts/metal_raster_gs.py, one process, gs.init(metal) per cycle, 40
cycles, three modes:

render: build a Sphere scene + camera, camera.render(rgb/depth/seg/normal).
build_only: build a rasterizer-camera scene (creates the offscreen GL
context) + step(), but no render().
physics: no camera, just step().

Result: all three modes 0/40, 0 compile failures. A single process doing
offscreen rasterizer rendering (or GL-context creation) repeatedly does not
reproduce.

14. Fork CI, rendering per-file + serial/parallel folder bisection (run `29456194019`, branch `hp/metal-bisect`)

Real Genesis install, org gpu-job env.

Per file, -n auto (2 workers): test_debug_draw (3 passed),
test_imgui_overlay (2 passed), test_interactive_viewer (9 passed),
test_offscreen (10 passed) - all pass in isolation;
test_dynamic_meshes - reproduces (ext_arr_to_ndarray_matrix_c20,
kernel_clear_external_force_c274).
Whole tests/rendering folder:
- -n auto (2 workers): reproduces - 4 passed, 26 pipeline-creation
  failures.
- -n 0 (1 process): 26 passed, 14 skipped, 3 xfailed, 0 failures.

test_dynamic_meshes.py is the unique rendering file that triggers the failure,
and the whole folder passes when run in a single process.

15. Fork CI, dynamic_meshes serial vs parallel (run `29456993054`, branch `hp/metal-serial`)

pytest ... tests/rendering/test_dynamic_meshes.py, real Genesis install, org
gpu-job env, differing only in worker count. The same three RASTERIZER tests
under each:

test	`-n 0` (1 proc)	`-n auto` (2 workers)
`test_deformable_uv_textures[RASTERIZER]`	PASSED	FAILED
`test_set_vverts[RASTERIZER]`	PASSED	FAILED
`test_set_vverts_sphere_to_box[RASTERIZER]`	PASSED	FAILED

Single process: 3 passed, 0 compile failures. Two workers: all three fail
with Failed to create pipeline ... (RhiResult=-1) ... Compilation failed (code=2). In this pair of jobs the only changed variable is the number of
concurrent worker processes; section 17 shows single-process jobs are not always
clean, so 2 workers guarantee the failure rather than being the sole cause.

16. Fork CI, two independent single-worker processes run concurrently (run `29457579661`, branch `hp/metal-cap`)

Capstone to test whether the trigger is pytest-xdist-specific or plain
cross-process contention. After a single-process warm step, the job launches two
independent pytest ... -n 0 tests/rendering/test_dynamic_meshes.py processes
(each with its own GS_CACHE_FILE_PATH and QD_OFFLINE_CACHE_FILE_PATH, so no
shared cache file) at the same time and waits for both.

Both concurrent processes reproduced (each: 3 failed, RhiResult=-1
present). So the trigger is ordinary OS-level concurrency of two processes on
the one GPU, not anything internal to xdist.
Note: the single-process warm step in this same job also failed (3 failed
in 337 s), whereas the identical single-process run in section 15 passed. So a
lone process is intermittent, not a guaranteed pass; concurrency raises the
failure probability toward certainty.

17. Fork CI, failure-rate over repeated runs (run `29458328690` / re-run `29459447352`, branch `hp/metal-stats`)

Same file, org gpu-job env, cache warmed once, then the identical command
repeated N times back-to-back (fresh process each), counting how many runs emit
RhiResult=-1.

parallel_auto (-n auto, 2 workers, run 29458328690): 8/8 runs
reproduced (9 RhiResult=-1 lines each). Two-worker reproduction is
deterministic.
serial_n0 (-n 0, 1 process): the outcome is uniform per job, not per run:
- run 29458328690: warm step passed and the first repeat was clean (0
  RhiResult=-1).
- run 29459447352: 10/10 repeats reproduced (9 RhiResult=-1 each).
- sections 14, 15 (empty cache) and the section 16 warm step (empty cache)
  show the same split: some whole single-process jobs pass, others fail.
So a lone process does not have a stable per-run failure probability; instead
an entire job either passes or fails. The offline cache is not the
discriminator (both stats jobs warmed the cache first, yet one passed and one
failed 10/10). A whole-job pass/fail split points to a per-VM/host factor -
contention on the shared, virtualized Apple Paravirtual GPU compiler from
whatever else is co-scheduled on that physical host - rather than anything in
the test itself. Two xdist workers add a guaranteed concurrent load of their
own, which is why the real 2-worker CI fails consistently. Single-process
failures are the same RhiResult=-1 / Compilation failed (code=2)
pipeline-creation error.

18. Third session: is the failure recoverable in-process? Retry fix disproven (branch `hp/metal-prove`)

Goal: prove the second-session mitigation #1 (retry newComputePipelineStateWithFunction).
The retry was emulated at the nearest Python boundary,
quadrants.lang.kernel.Kernel.__call__: "Failed to create pipeline" is raised
inside launch_kernel during prog.compile_kernel, and compiled_kernel_data
is cached only on success, so re-invoking the kernel re-drives compilation +
pipeline creation - exactly what a C++ retry would do. Loaded as a pytest plugin
in each worker, gated by QD_PIPELINE_RETRY. Three arms interleaved on ONE host
(controls for per-host state): 0 baseline (no patch), 1 retry, 2
passthrough wrapper (identical extra frame, no retry - a heisenbug control).

Run 29465439295 (valid; 6 iterations, warm cache, -n auto
test_dynamic_meshes.py):
- baseline: FAIL ~5/6, passthrough (patched=3): FAIL ~5/6, retry
  (patched=3, retry_events=120): FAIL 6/6.
- The retry fired 120x/run (40 attempts x 3 kernels) and never recovered.
  Passthrough failing like baseline rules out a timing/heisenbug artifact.
(Earlier runs 29462466840/29463513156 gave a false "retry passes": the
plugin hit INTERNALERROR: 'function' object has no attribute 'Kernel' -
quadrants.lang.__init__ does from quadrants.lang.kernel_impl import *, which
rebinds the attribute quadrants.lang.kernel to the kernel decorator; the
submodule must be fetched with importlib.import_module. Tests never ran, so
RhiResult=-1 count was 0 = a bogus pass. Fixed, then the real result above.)

Conclusion: pipeline creation stays broken for the whole process; an in-process
retry cannot recover. Mitigation #1 is refuted.

19. Single worker vs two workers, across four hosts (run `29466875938`, branch `hp/metal-prove2`)

test_dynamic_meshes.py, warm cache, interleaved -n auto (2 workers) and -n 0
(1 process), 5 iterations, on 4 independent runners (h1-h4).

-n 0 (single worker): FAIL 5/5 on all four hosts - just as reliably as
-n auto (also 5/5 on all four).

This demolishes the concurrency hypothesis (sections 15-17): a single process
with no concurrency at all fails identically. The only thing these runs share and
the earlier "serial passed" runs (section 15) lacked is that they run after a
prior rendering run on the same VM (the warm-up step) - i.e. run-order, not
worker count.

20. Cold cache vs warm cache, single worker (run `29468572792`, branch `hp/metal-prove3`)

test_dynamic_meshes.py, -n 0, interleaved QD_OFFLINE_CACHE=0 (always
recompile) vs a pre-populated warm cache.

Both FAIL 3/3 on both hosts. The offline cache is not the discriminator.
The revealing detail is run-order: on h2 the first pytest invocation
(cold, 401 s) passed 3/3, then the next 6 fresh processes (cold and warm
alike) all failed on the first kernel, over 3+ minutes. On h1 even the
first run failed. So the real variable is simply "has a rendering run already
happened on this VM", not cache/concurrency. The failing kernel is a trivial
~1.6 KB copy kernel (ext_arr_to_ndarray_matrix_c20_0_0_t00, msl_bytes=1641)
- so the compiler is wedged, not overloaded by any specific shader.

21. Is the wedge recoverable? killall / cache-clear (run `29470334885`, branch `hp/metal-fix`)

A fast no-render probe (scripts/metal_probe.py: gs.init(metal) +
qd.Matrix.ndarray(...).from_numpy -> the exact failing kernel, no OpenGL). Each
job: run test_dynamic_meshes.py until the probe fails (VM poisoned), then apply
a recovery and re-probe x3. Matrix: none / killall / clearcache / both.

Poison confirmed VM-wide and needs no rendering in the victim: once
poisoned, the no-GL probe process crashes with no output (hard native abort
in gs.init/pipeline creation).
Poison threshold is low: 3/4 arms poisoned after the 2nd rendering run (first
passed), one after the 1st.
No recovery works: killall -9 MTLCompilerService -> 0/3, clear Metal
shader caches (~/Library/Caches, /private/var/folders/**/com.apple.metal*)
-> 0/3, both -> 0/3.

So the wedge is not in the MTLCompilerService process (respawning it does not
help) and not in the on-disk shader cache. It is persistent state in the Apple
Paravirtual GPU device / VM, unrecoverable short of a fresh VM.

22. What triggers the wedge? render vs GL-context vs physics (run `29471301393`, branch `hp/metal-trigger`)

scripts/metal_trigger.py run repeatedly in fresh processes, probing VM health
between, 8 runs x 3 replicate hosts per mode:

render (simple sphere + camera + cam.render): never wedged (0/8 x3).
glbuild (camera scene built, GL context created, no render): never wedged.
physics (no camera, 40 steps of Metal compute): never wedged.

72 clean runs. Generic rasterization, GL-context creation, and Metal compute do
NOT wedge the VM. This matches section 13 (simple-sphere render 0/40). The
trigger is specific to test_dynamic_meshes, not rendering in general.

23. Which test wedges? (run `29472155741`, branch `hp/metal-triggerb`)

Each of the three test_dynamic_meshes.py RASTERIZER tests run alone, repeated in
fresh processes, probe between, 6 runs x 2 hosts:

test_deformable_uv_textures (PBD Cloth + FEM Elastic + render): wedges the
VM on run 1 (and the test itself fails); every subsequent run fails; probe
crashes.
test_set_vverts (custom vverts + n_envs=3 + set_vverts + render):
6/6 pass, VM healthy both hosts.
test_set_vverts_sphere_to_box: 6/6 pass, VM healthy both hosts.

The sole trigger is test_deformable_uv_textures - the FEM/PBD-deformable +
UV-texture-render combination. The dynamic-mesh set_vverts path (initially
suspected) is not the trigger.

24. Prove the fix: deselect the trigger (run `29473010006`, branch `hp/metal-provefix`)

Fresh VM per arm (baseline poisons its VM, so arms cannot share one), 3 replicate
hosts each, probe between runs:

baseline = test_dynamic_meshes.py full.
fixed = same, -k 'not deformable_uv_textures'.
fixedfolder = whole tests/rendering, -n auto, -k 'not deformable_uv_textures'.

RESULT (baseline from run 29473010006; fixed / fixedfolder from run
29476315670 after fixing a -k-quoting bug that had made the fixed arms run 0
tests - caught by an explicit "no tests ran" guard):

baseline (deformable included), 3 fresh VMs: the VM wedges within 1-2 runs;
2/3, 2/3, 3/3 runs hit RhiResult=-1 (the one clean run per VM is always the
VM's first, cold run). Matches the org cascade.
fixed (-k 'not deformable_uv_textures'), 3 VMs x 3 runs = 9/9 clean:
every run 2 passed, 3 deselected, rhi=0, and the between-run probe is
PROBE_OK - the VM never wedges.
fixedfolder (whole tests/rendering, -n auto, minus deformable), 3 VMs x
3 runs = 9/9 clean: every run 25-26 passed, 13 skipped, 2-3 xfailed,
rhi=0, VM healthy throughout.

So deselecting the single trigger test_deformable_uv_textures makes the entire
rendering suite pass reliably and keeps the VM's Metal compiler healthy, on every
host tried. This is the proven, reliable fix; retry (section 18), single-worker
(section 19), and cache changes (section 20) are all disproven.

Conclusion

Root cause (CORRECTED; supersedes the second-session conclusion below): the
macos-15-*-gpu-ndarray failures are a persistent, VM-wide wedge of the GitHub
macos-15 runner's Apple Paravirtual GPU device, triggered by
test_deformable_uv_textures (FEM/PBD deformables rendered through the
rasterizer). Once that test runs on a VM, the VM's Metal compiler can no longer
create a compute pipeline for any kernel - even a trivial ~1.6 KB copy kernel
(ext_arr_to_ndarray_matrix_c20, msl_bytes=1641) - returning Compilation failed (code=2) from newComputePipelineStateWithFunction. The state is
unrecoverable within the VM (surviving MTLCompilerService restarts, Metal cache
clears, and process exit) and crashes even a no-OpenGL probe process. This
produces the observed org-CI cascade: the suite runs fine until the deformable
render test executes, then all remaining kernel compilations fail (the "176 pass,
then everything fails" pattern of section 0.1).

Evidence chain (third session):

Not in-process-recoverable: an in-process retry of pipeline creation fails
6/6 despite firing 120x/run (section 18).
Not concurrency: a single worker fails 5/5 on all four hosts, identical to
two workers (section 19); the second-session "2 workers required" reading
(sections 15-17) was a run-order confound (the 2-worker jobs always ran the
trigger; the "serial passed" jobs happened to be the VM's first run).
Not the offline cache: cold (recompile-everything) fails identically to
warm; the discriminator is run-order on the VM (section 20).
VM-wide and unrecoverable: a no-render probe crashes on a poisoned VM;
killing MTLCompilerService and clearing Metal caches do not recover it
(section 21).
Trigger is specific: generic render / GL-context / physics never wedge (72
clean runs, section 22); of the three dynamic-mesh tests only
test_deformable_uv_textures wedges (section 23).

This is almost certainly a bug in Apple's virtualized-GPU (Paravirtual device)
Metal driver on the CI image, provoked by the deformable-render workload - not a
Genesis or quadrants logic bug (the same kernels compile fine before the trigger
runs, and on real hardware). The actionable outputs are therefore a CI mitigation
plus an upstream report.

Fix / mitigation (CORRECTED - the second-session list below is superseded):

Reliable CI fix: quarantine the trigger on virtualized macOS. Skip /
xfail test_deformable_uv_textures (and, defensively, any FEM/PBD-deformable
- rasterizer-render test) on the macos-15 Metal GPU job - e.g. a
  @pytest.mark.skipif keyed on Metal + the paravirtual device, or deselect it
  in the job. Section 24 proves the rest of the rendering suite then passes
  reliably and the VM never wedges. This is deterministic because the
  VM-poisoning workload never runs. (It does not "fix" the deformable render on
  virtualized macOS - that path is broken by the driver bug - it stops one
  broken test from taking down the other ~180.)
If deformable render must be exercised on macOS CI, isolate it in its own
throwaway job/VM so its wedge cannot cascade. Note even alone it is flaky on
this driver (fails ~on the first run on some hosts), so prefer skip.
Report upstream to Apple (Feedback Assistant): virtualized Apple GPU on
the macos-15 runner image is driven into a permanent
newComputePipelineStateWithFunction code=2 state by a specific Metal +
rendering workload; include the minimal repro (test_deformable_uv_textures
-> then any trivial pipeline creation fails process-wide and VM-wide).
Longer term / real hardware: investigate what in the FEM/PBD deformable
render path provokes the driver (e.g. a specific texture / buffer / compute
interaction) and evaluate a headless or Metal-native render path.

Superseded second-session conclusion (kept for the record; disproven by sections 18-24)

Root cause (diagnosed): the macos-15-*-gpu-ndarray failures are GPU-compiler
contention on the shared Apple Paravirtual device of the GitHub macos-15
runner, triggered specifically by Genesis's rasterizer (offscreen OpenGL via
pyglet) rendering running concurrently with Metal (quadrants) compute-pipeline
compilation. When two pytest-xdist workers execute rendering tests at the
same time, Metal's newComputePipelineStateWithFunction starts returning nil
with Compilation failed (code=2); once a worker hits this, its Metal state
stays poisoned and every later kernel it compiles fails, producing the observed
"many pass, then a worker cascades to all-fail" pattern (section 0.1's 176
consecutive passes then interleaved failures).

Evidence chain:

Rendering is necessary. Every reproduction involves the rasterizer.
tests/rendering reproduces (section 12); of its files only
test_dynamic_meshes.py (mesh-from-file + rasterizer render) reproduces in
isolation (section 14); tests/rigid with 2 workers and no rendering does not
(section 12, 258 passed); pure-Metal synthetics never fail (sections 8, 10,
11).
A guaranteed concurrent load makes it deterministic. Two xdist workers
fail in 8/8 runs (section 17); two independent single-worker processes run
concurrently both fail (section 16), so it is cross-process contention on the
shared GPU, not an xdist internal.
Concurrency alone is not sufficient. 2-worker rigid (no rendering) and 1-3
concurrent pure-Metal churn processes never fail (sections 12, 11). The
in-process rasterizer(OpenGL) + Metal combination is the fragility; concurrent
GPU-compiler load is the stressor that tips it over.
A single process passes or fails per job, not per run (section 17: one
serial job 10/10 fail, another clean; sections 14/15 clean), independent of
the offline cache. A whole-job split is what you expect if the stressor is
contention on the shared, virtualized GPU compiler from other tenants
co-scheduled on the same physical macOS host - a sibling xdist worker is
simply a guaranteed, in-job source of that same contention.
It is not codegen (the same small copy/fill kernels compile fine in
isolation, sections 1, 2, 8), not PR [BUG FIX] Fix fastcache support. #3034-specific (section 0.1, many
PRs and main affected), not the offline cache (section 5), and not
memory/resource exhaustion (no EXC_RESOURCE/jetsam/compiler-crash in syslog,
sections 8, 9).

Suggested mitigations (in rough order of preference) [DISPROVEN - see sections
18-24 and the corrected list above]:

Make create_compute_pipeline retry newComputePipelineStateWithFunction
on transient Compilation failed (code=2). Disproven (section 18): the
failure is persistent, not transient; retry fails 6/6 despite 120 retries/run.
The "identical MSL succeeds when recompiled without contention" premise was
never actually tested and is false here.
~~Run macOS Metal rendering tests with a single worker.~~ Disproven (section
19): a single worker fails 5/5 on every host.
~~Isolate the rasterizer's OpenGL context from Metal compute.~~ Not the
mechanism: generic GL-context creation and rendering never wedge the VM
(section 22).
Headless/Metal-native rendering path - still worth evaluating long-term, but
the immediate cause is the deformable-render driver wedge, not OpenGL per se.

Appendix: run index (fork `hughperkins/genesis-world`; branches `hp/metal-ci-debug` and `hp/metal-limit`)

run	branch	experiment	result
29439255683	metal-ci-debug	2.1 init-flag sweep (isolated kernel)	success (5/5)
29439514762	metal-ci-debug	2.2 genesis-exact + ad-stack sweep	success (5/5)
29439730976	metal-ci-debug	3 full genesis, wrong test path	exit 5 (0 tests)
29440166086	metal-ci-debug	3 full genesis, scene + test_gravity	success
29440658358	metal-ci-debug	4 parallelism/cache matrix on test_collision.py	ci_like success; others cancelled
29442109831	metal-ci-debug	5 cross-backend offline cache	both success (62 passed each)
29443749352	metal-ci-debug	6 full suite mirroring org gpu job	304 passed, 150 failed (reproduces org)
29444269087	metal-limit	7 device limits + buffer/adstack sweeps	all success
29447591837	metal-limit	8 synthetic compiler-service stress	both success; no EXC_RESOURCE in syslog
29448183249	metal-limit	9 real heavy folders + syslog capture	267 passed, 19 failed; no EXC_RESOURCE/watermark/mid-run compiler kill in syslog
29453644680	metal-reinit	10 repeated init/teardown, one process	qd 0 compile-fails/600; gs 0/300
29454911916	metal-churn	11 init/reset x distinct-kernel churn	all 3 modes 0/800
29455302366	metal-conc	12 concurrency synthetic + real folders	conc 1/2/3 procs 0/800; rendering(-n auto) reproduces 25 failed; rigid(-n auto) see below
29455788553	metal-raster	13 single-process rasterizer isolation	render/build_only/physics all 0/40
29456194019	metal-bisect	14 per-file + serial/parallel folder	only test_dynamic_meshes reproduces; folder -n auto reproduces, -n 0 passes (26 passed)
29456993054	metal-serial	15 dynamic_meshes -n 0 vs -n auto	-n 0: 3 passed; -n auto: 3 failed
29457579661	metal-cap	16 two independent serial procs concurrently	both procs reproduce; single-proc warm step also failed
29458328690	metal-stats	17 rate: parallel + serial (buggy grep)	parallel 8/8 reproduced; serial job clean
29459447352	metal-stats	17 rate: serial 10x (fixed)	serial 10/10 reproduced (different job/host)
29465439295	metal-prove	18 retry vs passthrough vs baseline (valid)	baseline/passthrough ~5/6 fail; retry 6/6 fail (120 retries/run) - retry DISPROVEN
29466875938	metal-prove2	19 single vs two workers x4 hosts	-n 0 fails 5/5 on all 4 hosts (= -n auto) - concurrency DISPROVEN
29468572792	metal-prove3	20 cold vs warm cache, single worker	both fail 3/3; run-order (not cache) is the variable
29470334885	metal-fix	21 recovery: killall / clearcache	none recover (0/3 each); no-render probe crashes; VM-wide wedge
29471301393	metal-trigger	22 render vs glbuild vs physics x3 hosts	none wedge (72 clean runs) - generic render/GL/physics not the trigger
29472155741	metal-triggerb	23 which dynamic_meshes test wedges	only test_deformable_uv_textures wedges; set_vverts / sphere_to_box 6/6 pass
29473010006	metal-provefix	24 baseline (deformable included) x3 hosts	wedges within 1-2 runs; 2/3, 2/3, 3/3 runs hit RhiResult=-1
29476315670	metal-provefix	24 fixed + fixedfolder (deselect deformable) x3 hosts	9/9 clean each; fixed 2 passed/run, fixedfolder 25-26 passed/run; VM never wedges

Related Issue

Resolves Genesis-Embodied-AI/Genesis#

Motivation and Context

How Has This Been / Can This Be Tested?

Screenshots (if appropriate):

Checklist:

I read the CONTRIBUTING document.
I followed the Submitting Code Changes section of CONTRIBUTING document.
I tagged the title correctly (including BUG FIX/FEATURE/MISC/BREAKING)
I updated the documentation accordingly or no change is needed.
I tested my changes and added instructions on how to test it for reviewers.

I have added tests to cover my changes.
All new and existing tests passed.

The macOS CI runners expose a virtualized 'Apple Paravirtual device' GPU. Running test_deformable_uv_textures drives that virtualized Metal driver into a persistent, VM-wide broken state: afterwards newComputePipelineStateWithFunction returns 'Compilation failed (code=2)' for every kernel - even a trivial copy kernel. The wedge is unrecoverable within the VM (survives MTLCompilerService restart and process exit) and cascades to every later test in the job, turning the Metal GPU job permanently red. Component isolation narrowed the trigger to the test's FEM Elastic implicit (Newton/PCG) solver run for many PCG iterations: the rasterizer render path, the UV textures, the PBD Cloth solver, and the explicit / low-PCG FEM solvers do not wedge the GPU. Skip this test on the macOS Metal backend until the underlying Apple driver bug is resolved. Co-authored-by: Cursor <cursoragent@cursor.com>

hughperkins · 2026-07-16T20:12:37Z

🙌

hughperkins marked this pull request as ready for review July 16, 2026 13:01

hughperkins requested review from YilingQiao and duburcqa as code owners July 16, 2026 13:01

hughperkins force-pushed the hp/skip-metal-deformable-render branch from 07b435a to 96eaedc Compare July 16, 2026 13:02

Inline the Metal deformable-render skip reason.

80a78fe

duburcqa force-pushed the hp/skip-metal-deformable-render branch from 50ca09a to 80a78fe Compare July 16, 2026 20:09

duburcqa approved these changes Jul 16, 2026

View reviewed changes

duburcqa merged commit 7d81b27 into Genesis-Embodied-AI:main Jul 16, 2026
4 checks passed

duburcqa changed the title ~~[CI] Skip test_deformable_uv_textures on macOS Metal~~ [MISC] Fix broken Apple Metal CI. Jul 16, 2026

hughperkins deleted the hp/skip-metal-deformable-render branch July 16, 2026 20:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[MISC] Fix broken Apple Metal CI.#3054

[MISC] Fix broken Apple Metal CI.#3054
duburcqa merged 2 commits into
Genesis-Embodied-AI:mainfrom
hughperkins:hp/skip-metal-deformable-render

hughperkins commented Jul 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

hughperkins commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

hughperkins commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Metal Genesis CI failure: experiments and results

0. Observed CI failure

0.0 Underlying RHI diagnostic (from org logs)

0.1 Other PRs

0.2 Date range (from PR head-SHA check-runs)

1. Local, real Apple Silicon Metal (isolated copy kernel)

2. Fork CI, isolated copy kernel, macos-15 virtualized runner

2.1 Init-flag sweep (run 29439255683)

2.2 Genesis-exact init flags plus ad-stack sweep (run 29439514762)

3. Fork CI, full Genesis install, trivial scene and single test

4. Fork CI, full test file, CI-like parallelism (run 29440658358)

5. Fork CI, cross-backend offline cache (run 29442109831)

5b. Runner comparison (org vs fork)

6. Fork CI, full ./tests suite mirroring the org GPU job (run 29443749352, commit a9eb95cc, branch hp/metal-ci-debug)

7. Fork CI, Metal device limits and buffer/adstack size sweeps (run 29444269087, branch hp/metal-limit)

7.1 Device limits (scripts/metal_device_info.swift)

7.2 Single-buffer size sweep (scripts/metal_buffer_sweep.py, quadrants 1.1.1)

7.3 Reverse-mode adstack size sweep (scripts/metal_adstack.py, quadrants 1.1.1)

8. Fork CI, synthetic Metal compiler-service stress (run 29447591837, branch hp/metal-limit)

9. Fork CI, real heavy folders (rigid + rendering) with system-log capture (run 29448183249, branch hp/metal-limit)

10. Fork CI, repeated init/teardown in one long-lived process (run 29453644680, branch hp/metal-reinit)

11. Fork CI, init/reset x distinct-kernel churn in one process (run 29454911916, branch hp/metal-churn)

12. Fork CI, concurrency synthetic + real-folder split (run 29455302366, branch hp/metal-conc)

13. Fork CI, single-process rasterizer isolation (run 29455788553, branch hp/metal-raster)

14. Fork CI, rendering per-file + serial/parallel folder bisection (run 29456194019, branch hp/metal-bisect)

15. Fork CI, dynamic_meshes serial vs parallel (run 29456993054, branch hp/metal-serial)

16. Fork CI, two independent single-worker processes run concurrently (run 29457579661, branch hp/metal-cap)

17. Fork CI, failure-rate over repeated runs (run 29458328690 / re-run 29459447352, branch hp/metal-stats)

18. Third session: is the failure recoverable in-process? Retry fix disproven (branch hp/metal-prove)

19. Single worker vs two workers, across four hosts (run 29466875938, branch hp/metal-prove2)

20. Cold cache vs warm cache, single worker (run 29468572792, branch hp/metal-prove3)

21. Is the wedge recoverable? killall / cache-clear (run 29470334885, branch hp/metal-fix)

22. What triggers the wedge? render vs GL-context vs physics (run 29471301393, branch hp/metal-trigger)

23. Which test wedges? (run 29472155741, branch hp/metal-triggerb)

24. Prove the fix: deselect the trigger (run 29473010006, branch hp/metal-provefix)

Conclusion

Superseded second-session conclusion (kept for the record; disproven by sections 18-24)

Appendix: run index (fork hughperkins/genesis-world; branches hp/metal-ci-debug and hp/metal-limit)

Related Issue

Motivation and Context

How Has This Been / Can This Be Tested?

Screenshots (if appropriate):

Checklist:

Uh oh!

Uh oh!

hughperkins commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hughperkins commented Jul 16, 2026 •

edited

Loading

2.1 Init-flag sweep (run `29439255683`)

2.2 Genesis-exact init flags plus ad-stack sweep (run `29439514762`)

4. Fork CI, full test file, CI-like parallelism (run `29440658358`)

5. Fork CI, cross-backend offline cache (run `29442109831`)

6. Fork CI, full `./tests` suite mirroring the org GPU job (run `29443749352`, commit `a9eb95cc`, branch `hp/metal-ci-debug`)

7. Fork CI, Metal device limits and buffer/adstack size sweeps (run `29444269087`, branch `hp/metal-limit`)

7.1 Device limits (`scripts/metal_device_info.swift`)

7.2 Single-buffer size sweep (`scripts/metal_buffer_sweep.py`, quadrants 1.1.1)

7.3 Reverse-mode adstack size sweep (`scripts/metal_adstack.py`, quadrants 1.1.1)

8. Fork CI, synthetic Metal compiler-service stress (run `29447591837`, branch `hp/metal-limit`)

9. Fork CI, real heavy folders (rigid + rendering) with system-log capture (run `29448183249`, branch `hp/metal-limit`)

10. Fork CI, repeated init/teardown in one long-lived process (run `29453644680`, branch `hp/metal-reinit`)

11. Fork CI, init/reset x distinct-kernel churn in one process (run `29454911916`, branch `hp/metal-churn`)

12. Fork CI, concurrency synthetic + real-folder split (run `29455302366`, branch `hp/metal-conc`)

13. Fork CI, single-process rasterizer isolation (run `29455788553`, branch `hp/metal-raster`)

14. Fork CI, rendering per-file + serial/parallel folder bisection (run `29456194019`, branch `hp/metal-bisect`)

15. Fork CI, dynamic_meshes serial vs parallel (run `29456993054`, branch `hp/metal-serial`)

16. Fork CI, two independent single-worker processes run concurrently (run `29457579661`, branch `hp/metal-cap`)

17. Fork CI, failure-rate over repeated runs (run `29458328690` / re-run `29459447352`, branch `hp/metal-stats`)

18. Third session: is the failure recoverable in-process? Retry fix disproven (branch `hp/metal-prove`)

19. Single worker vs two workers, across four hosts (run `29466875938`, branch `hp/metal-prove2`)

20. Cold cache vs warm cache, single worker (run `29468572792`, branch `hp/metal-prove3`)

21. Is the wedge recoverable? killall / cache-clear (run `29470334885`, branch `hp/metal-fix`)

22. What triggers the wedge? render vs GL-context vs physics (run `29471301393`, branch `hp/metal-trigger`)

23. Which test wedges? (run `29472155741`, branch `hp/metal-triggerb`)

24. Prove the fix: deselect the trigger (run `29473010006`, branch `hp/metal-provefix`)

Appendix: run index (fork `hughperkins/genesis-world`; branches `hp/metal-ci-debug` and `hp/metal-limit`)