mesh-llm crashes when loading with --gguf argument

[sofiageo]

Problem

mesh-llm crashes when loading with --gguf argument.

Steps to reproduce

1. run mesh-llm serve --gguf ./model --discover

Diagnostics

ROCm crash: https://gist.github.com/sofiageo/b78645924258f7f2355dddb5c53afb33
Vulkan crash: https://gist.github.com/sofiageo/df9d328ac96ced048a5896482ee70d07

PC: Strix Halo with ROCm / Vulkan.
mesh-llm version: 0.75rc1

[ndizazzo]

I traced this further against the v0.75.0-rc1 code path and the repository’s pinned llama.cpp revision (8190848bb36c7df4251db4352bd81bc07d0a4385).

Model-name selection is intentional and does not appear to be the cause. The name is resolved to the local GGUF and its associated projector sidecar before the native runtime is started.

The likely failure path is:

Skippy skippy_model_open
  → llama_model_load_from_file(text model)
  → llama_init_from_model(text context)
  → mtmd_init_from_file(mmproj, llama_model)
  → clip_init
  → clip_model_loader::load_tensors
  → GPU projector buffer allocation

In Skippy, the projector is loaded after the text model has already been opened. MediaProjector::open calls mtmd_init_from_file with GPU loading enabled.

The ROCm coredump is consistent with this path: it ends in clip_init → mtmd_context → mtmd_init_from_file. The Vulkan coredump is symbol-stripped, but it occurs during the same native model-startup phase:

• ROCm coredump
• Vulkan coredump

The relevant upstream code appears to have an unsafe allocation failure path:

1. clip.cpp selects a GPU backend and calls ggml_backend_alloc_ctx_tensors_from_buft.
2. The allocator returns NULL if the backend cannot allocate the projector buffer.
3. clip.cpp immediately calls ggml_backend_buffer_set_usage without checking for NULL.
4. ggml_backend_buffer_set_usage asserts that the buffer is valid, producing a hard abort instead of returning an error.

Relevant upstream code:

• clip.cpp projector allocation
• ggml-alloc.c allocation failure
• ggml-backend.cpp null assertion

There is also a closely related upstream report with the same general stack shape: ggml_abort → ggml_backend_buffer_set_usage → clip_model_loader::load_tensors → clip_init → mtmd_init_from_file (llama.cpp#21402).

The Skippy-specific concern is memory planning. Upstream llama-server estimates projector memory with mtmd_get_memory_usage before loading/fitting the text model, then adds the projector’s per-device memory requirement to the fit target:

• server-context.cpp projector memory preflight
• common.h fit parameters enabled by default

Skippy currently resolves a fit_target_mib based on available memory and a safety margin, but the direct GGUF translation does not pass an equivalent projector-aware fit target into native llama.cpp loading. It passes the GPU layer count, mmap/mlock settings, selected device, and projector path, but does not preflight or reserve projector memory.

That creates this likely sequence on Strix Halo:

text model loads successfully
  → text model/context consume most available shared GPU memory
  → projector allocation fails
  → upstream mtmd dereferences a null buffer
  → process aborts

ROCm and Vulkan failing in the same way would be plausible because both backends are using the same constrained unified-memory pool.

Things that appear less likely:

• Model-name/path resolution: intentional and completes before the native load.
• params.use_gpu = true: upstream mtmd_context_params_default() already defaults this to true.
• FFI ABI mismatch: the Rust mtmd_context_params layout matches the native structure.
• Skippy tensor filtering: direct --gguf loading uses a full-model configuration with tensor filtering disabled.
• The Skippy patch queue: the relevant patches do not alter the mtmd context ABI or projector allocation path.

There is one secondary integration issue worth checking: Skippy can pin the text model to an exact backend device, while mtmd independently chooses its GPU backend using MTMD_BACKEND_DEVICE or the first available GPU. On a multi-device system, the projector could therefore be placed on a different device from the text model. This is less likely on a single-device Strix Halo, but the logs should compare Skippy’s selected device with mtmd’s CLIP using ... backend message.

Suggested validation:

1. Reproduce with the exact GGUF and mmproj paths recorded.
2. Temporarily load the model without its projector sidecar. If the text model starts successfully, that isolates the failure to projector loading.
3. Capture logs immediately before the crash and look for:
   • CLIP using ... backend
   • failed to allocate ... buffer
   • the selected Skippy backend device
4. Compare the same model/mmproj with the pinned upstream llama-server, both with its normal fit behavior and with fitting disabled.
5. As a diagnostic, force the projector to CPU in a local build. If that avoids the crash, it strongly supports GPU projector allocation failure.
6. If no allocation failure is reported, the next suspects are an unsupported/corrupt projector or a model/mmproj embedding-dimension mismatch. The exact model and projector names are needed to distinguish those cases.

At this point, the evidence supports a combination of a Skippy memory-planning gap and an upstream mtmd allocation-error handling bug. The exact model/mmproj and allocator log are still needed to confirm that this is the specific failure for issue #1166.