[AMDGPU] Fix: increase Quadrants device memory pool on ROCm (hipMemset)

[gayathiri-venkataraman]

Summary

On ROCm/AMDGPU, Genesis init now sets Quadrants device_memory_GB from GS_DEVICE_MEMORY_GB when set, otherwise ~80% of total device memory. The default 1 GB pool is too small for large-scale sims and can cause hipMemset allocation failures.

Also adjusts the default PYOPENGL_PLATFORM handling in the rasterizer so an empty env var does not force an invalid platform string on headless nodes.

Correctness

No numerical/physics change — memory pool sizing and renderer platform selection only.

Verification

Tested on AMD MI300X/MI325X (gfx942) vs baseline.

Made with Cursor

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dcfb4567f0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Limit the pool expansion to AMDGPU

This branch now runs for every non-CPU backend, including CUDA and Metal selected by init, even though the change is meant to address ROCm hipMemset failures. On CUDA/Metal jobs that already rely on PyTorch, rendering, or other processes sharing VRAM, raising Quadrants from its previous 1 GB default to 80% of device memory at gs.init() can starve those allocations before any ROCm workload is involved; guard this to _gs_backend.amdgpu or make it opt-in for the other GPU backends.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Use GPU memory before forcing Torch to CPU

When GS_TORCH_FORCE_CPU_DEVICE=1 is set with a GPU backend, the fallback above overwrites total_mem with host RAM while leaving backend as the GPU backend. This new default then passes round(total_mem * 0.8, 1) as Quadrants GPU device_memory_GB, so machines with much more system RAM than VRAM can request an impossible GPU pool and fail initialization; preserve the original GPU total or skip this override in that mode.

Useful? React with 👍 / 👎.

[hughperkins]

I dont have a major concern with enabling an environment variable to control this, so if you make this PR just about that, I'm fairly comfortable merging that.

Changing the default allocation is much more concerning to me, and would need a lot more investigation and discussion, since you are the only person who has raised this issue, and memory allocation is a big deal.

Are you able to consider scoping this PR to just honoring the env var, or do you want to initiate a longer discussion about the goals and use-cases for the change in the default?

My concerns about changing the default by the way are:

• why has this issue only arisen now? It seems like a pretty major change, and I would expect it to affect a lot more people if it is as serious as you are asserting
• why is this change intended to be AMD specific? ie why dont other platforms have this issue?
• if the change is in fact global (I didnt read in detail, but one of the codex comments above suggests it is), why do we need to change this default across all platforms?
• why are other people not encountering this issue? (since it seems like a really fundamental issue).

[hughperkins]

I checked with Opus. Here is what it said:

Summary:

• it looks like there are already existing env vars QD_DEVICE_MEMORY_GB, QD_DEVICE_MEMORY_FRACTION, that you can use (from Quadrants)
• Quadrants has two memory modes:
  • mem pool
  • no mem pool
• these env vars only work on the no-mem-pool path, as does your PR proposal
• a key question is whether you are using mem pool, or no mem pool
  • on modern GPUS, such as MI300X, mem pool should be supported, and none of these env vars, nor this PR should have any effect
• there is no way to directly know if you are using non-pooled or pooled memory
• you can infer which you are using by running:

QD_DEVICE_MEMORY_GB=8 python - <<'PY'
import torch, genesis as gs
free0, total = torch.cuda.mem_get_info()
gs.init(backend=gs.gpu)          # no scene built yet
free1, _ = torch.cuda.mem_get_info()
print(f"grabbed at init: {(free0-free1)/1e9:.2f} GB of {total/1e9:.1f} GB")
PY

• Jump of ~8 GB -> no-pool path (the fixed preallocation is active, and device_memory_GB is the issue you are encountering).
• Jump of ~0 (tens/low-hundreds of MB of context + runtime objects) -> pool path (device_memory_GB is a no-op; the hipMemset failure is coming from somewhere else)

Could you:

• provide the output from the short script above please, and
• if you are using non-pooled path, state whether using QD_DEVICE_MEMORY_GB or QD_DEVICE_MEMORY_FRACTION are able to resolve your issue?

[gayathiri-venkataraman]

Ran your mem script on MI300X: default, QD_DEVICE_MEMORY_GB=8, and GS_DEVICE_MEMORY_GB=8 all grab ~1.24 GB at init (pool path) — env vars don't change allocation here; dropped the 80% default, PR now AMDGPU-only GS_DEVICE_MEMORY_GB when set.

[hughperkins]

Ok, so we are concurring that GS_DEVICE_MEMORY_GB has no effect I think?

So the problem you are facing lies elsewhere.

Could you post a minimum reproducible example script to reprdocue your issue please? (I have access to an MI300X I can run your reproduction script on).