Multi-round inference leads to insufficient memory crash

[Edsuns]

Comparing Qwen3.6 35B 4bit mlx to Qwen3.6 35B q4_k_m gguf, after running in Claude Code for a while, the MLX version consumes significantly more memory than the GGUF version, and the MLX version crashed due to insufficient memory.

Run reproduction long_prompt_demo.py on Mac mini M4 32GB:

lms get mlx-community/Qwen3.6-35B-A3B-4bit
python3 long_prompt_demo.py --model mlx-community/Qwen3.6-35B-A3B-4bit --max-kv-size 65536 --rounds 5 --prompt-length 100000

I have tested models that exhibit this problem:

• Qwen3.6 35B 4bit mlx
• Qwen3.5 35B mxfp4 MLX

Error log using lms:

2026-04-18 12:08:57  [INFO]
 [unn/qwen3.6-35b-a3b-4bit] Running Anthropic messages API on conversation with 19 messages.
2026-04-18 12:08:57  [INFO]
 [unn/qwen3.6-35b-a3b-4bit] Streaming Anthropic response...
2026-04-18 12:08:58 [DEBUG]
 [cache_wrapper][INFO]: Prompt cache: using 36384/36871 tokens from cache
2026-04-18 12:08:58  [INFO]
 [unn/qwen3.6-35b-a3b-4bit] Prompt processing progress: 0.0%
2026-04-18 12:09:04  [INFO]
 [unn/qwen3.6-35b-a3b-4bit] Prompt processing progress: 97.7%
2026-04-18 12:09:04  [INFO]
 [unn/qwen3.6-35b-a3b-4bit] Prompt processing progress: 99.8%
2026-04-18 12:09:04  [INFO]
 [unn/qwen3.6-35b-a3b-4bit] Prompt processing progress: 99.8%
2026-04-18 12:09:05 [DEBUG]
 libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
2026-04-18 12:09:05 [DEBUG]
 Fatal Python error: Aborted

Thread 0x
2026-04-18 12:09:05 [DEBUG]
 0000000de2c57000 (most recent call first):
  File "/Users/xxx/.lmstudio/extensions/backends/vendor/_amphibian/app-mlx-generate-mac14-arm64@22/lib/python3.11/site-packages/mlx_lm/generate.py", line 455 in generate_step
  File "/Users/xxx/.lmstudio/extensions/backends/vendor/_amphibian/app-mlx-generate-mac14-arm64@22/lib/python3.11/site-packages/mlx_lm/generate.py", line 705 in <genexpr>
  File "/Users/xxx/.lmstudio/extensions/backends/vendor/_amphibian/app-mlx-generate-mac14-arm64@22/lib/python3.11/site-packages/mlx_lm/generate.py", line 716 in stream_generate
2026-04-18 12:09:05 [DEBUG]
 File "/Users/xxx/.lmstudio/extensions/backends/vendor/_amphibian/app-mlx-generate-mac14-arm64@22/lib/python3.11/site-packages/mlx_engine/generate.py", line 543 in _sequential_generation

Thread 0x00000001f7d6d8c0 (most recent call first):
  <no Python frame>
2026-04-18 12:09:05 [DEBUG]
 
Extension modules: yaml._yaml
2026-04-18 12:09:05 [DEBUG]
 , regex._regex, numpy._core._multiarray_umath
2026-04-18 12:09:05 [DEBUG]
 , numpy.linalg._umath_linalg, markupsafe._speedups
2026-04-18 12:09:05 [DEBUG]
 , PIL._imaging, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special
2026-04-18 12:09:05 [DEBUG]
 , numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand
2026-04-18 12:09:05 [DEBUG]
 , numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator
2026-04-18 12:09:05 [DEBUG]
 , sentencepiece._sentencepiece
2026-04-18 12:09:05 [DEBUG]
 , PIL._imagingft
2026-04-18 12:09:05 [DEBUG]
 , charset_normalizer.md, charset_normalizer.cd, requests.packages.charset_normalizer.md, requests.packages.chardet.md, requests.packages.charset_normalizer.cd, requests.packages.chardet.cd
2026-04-18 12:09:05 [DEBUG]
 , xxhash._xxhash
2026-04-18 12:09:06 [DEBUG]
  (total: 35)
2026-04-18 12:09:06 [ERROR]
 [unn/qwen3.6-35b-a3b-4bit] Anthropic streaming error: The model has crashed without additional information. (Exit code: null)
2026-04-18 12:09:06  [INFO]
 [unn/qwen3.6-35b-a3b-4bit] Finished streaming Anthropic response

[Edsuns]

Gemma 4 26B 4bit MLX (context length: 131072) also has the same problem. It seems to only occur at the end of the prompt processing.

2026-04-18 21:08:36  [INFO]
 [gemma-4-26b-a4b-it-heretic] Running Anthropic messages API on conversation with 109 messages.
2026-04-18 21:08:36  [INFO]
 [gemma-4-26b-a4b-it-heretic] Streaming Anthropic response...
2026-04-18 21:08:37 [DEBUG]
 [cache_wrapper][INFO]: Prompt cache: using 66006/66219 tokens from cache
2026-04-18 21:08:37  [INFO]
 [gemma-4-26b-a4b-it-heretic] Prompt processing progress: 0.0%
2026-04-18 21:08:42  [INFO]
 [gemma-4-26b-a4b-it-heretic] Prompt processing progress: 94.8%
2026-04-18 21:08:47  [INFO]
 [gemma-4-26b-a4b-it-heretic] Prompt processing progress: 99.5%
2026-04-18 21:08:47  [INFO]
 [gemma-4-26b-a4b-it-heretic] Prompt processing progress: 99.5%
2026-04-18 21:08:53 [DEBUG]
 libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
2026-04-18 21:08:53 [DEBUG]
 Fatal Python error: Aborted

Thread 0x0000000cc3ab3000 (most recent call first):
  File "/Users/xxx/.lmstudio/extensions/backends/vendor/_amphibian/app-mlx-generate-mac14-arm64@22/lib/python3.11/site-packages/mlx_lm/generate.py", line 455 in generate_step
  File "/Users/xxx/.lmstudio/extensions/backends/vendor/_amphibian/app-mlx-generate-mac14-arm64@22/lib/python3.11/site-packages/mlx_lm/generate.py", line 705 in <genexpr>
  File "/Users/xxx/.lmstudio/extensions/backends/vendor/_amphibian/app-mlx-generate-mac14-arm64@22/lib/python3.11/site-packages/mlx_lm/generate.py", line 716 in stream_generate
  File "/Users/xxx/.lmstudio/extensions/backends/vendor/_amphibian/app-mlx-generate-mac14-arm64@22/lib/python3.11/site-packages/mlx_engine/generate.py", line 543 in _sequential_generation

Thread 0x00000001edd7d8c0 (most recent call first):
  <no Python frame>
2026-04-18 21:08:53 [DEBUG]
 
Extension modules: yaml._yaml
2026-04-18 21:08:53 [DEBUG]
 , regex._regex
2026-04-18 21:08:53 [DEBUG]
 , numpy._core._multiarray_umath
2026-04-18 21:08:53 [DEBUG]
 , numpy.linalg._umath_linalg
2026-04-18 21:08:53 [DEBUG]
 , markupsafe._speedups
2026-04-18 21:08:53 [DEBUG]
 , PIL._imaging
2026-04-18 21:08:53 [DEBUG]
 , torch._C
2026-04-18 21:08:53 [DEBUG]
 , torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils
2026-04-18 21:08:53 [DEBUG]
 , torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special
2026-04-18 21:08:53 [DEBUG]
 , numpy.random._common
2026-04-18 21:08:53 [DEBUG]
 , numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator
2026-04-18 21:08:53 [DEBUG]
 , sentencepiece._sentencepiece
2026-04-18 21:08:53 [DEBUG]
 , PIL._imagingft
2026-04-18 21:08:53 [DEBUG]
 , charset_normalizer.md
2026-04-18 21:08:53 [DEBUG]
 , charset_normalizer.cd
2026-04-18 21:08:53 [DEBUG]
 , requests.packages.charset_normalizer.md, requests.packages.chardet.md, requests.packages.charset_normalizer.cd, requests.packages.chardet.cd
2026-04-18 21:08:53 [DEBUG]
 , xxhash._xxhash
2026-04-18 21:08:53 [DEBUG]
  (total: 35)
2026-04-18 21:08:54 [ERROR]
 [gemma-4-26b-a4b-it-heretic] Anthropic streaming error: The model has crashed without additional information. (Exit code: null)
2026-04-18 21:08:54  [INFO]
 [gemma-4-26b-a4b-it-heretic] Finished streaming Anthropic response

[Bustycat]

The current workaround is using LM Studio MLX v1.5.0 as v1.6.0 and v1.7.0 both crash.

[lustfeind]

I can't count how often in the past crashed I get "The model has crashed without additional information" when using LM Studio MLX. Why this is happening over and over again?

[Bustycat]

I haven't experienced since v1.8.x. Still needs further observation.