Windows Vulkan runtime bundles a mismatched MinGW libstdc++

[ApexDevelopment]

Summary

The Windows x86_64 Vulkan native runtime ships a libstdc++-6.dll built for a
different MinGW-w64 configuration than the DLLs that import it. As a result
libllama.dll cannot be loaded at all: LoadLibraryEx fails with
ERROR_PROC_NOT_FOUND (127). No GPU backend is registered and the node falls
back to CPU only.

Affected versions

• 0.74.0
• 0.75.0-rc1

sha256(lib/libstdc++-6.dll) is
3529d11c422b2aaf0bbad7221bf62ba7b5a39f854e89a015bb58b6b55a677da1 in both
releases.

Environment

• Windows 10 (10.0.19044), x86_64
• Runtime: meshllm-native-runtime-windows-x86_64-vulkan

Steps to reproduce

1. Install the Windows Vulkan native runtime.
2. From the runtime lib directory, call LoadLibraryExW on libllama.dll
   with LOAD_WITH_ALTERED_SEARCH_PATH.
3. The call fails with error 127.

Expected

libllama.dll loads and the Vulkan backend registers its devices.

Actual

Load fails with error 127. Diffing the PE import tables of the shipped DLLs
against the export table of the shipped libstdc++-6.dll shows 16 imported
symbols that the bundled libstdc++-6.dll does not export.

ggml.dll (5 missing):

_ZNKSt25__codecvt_utf8_utf16_baseIwE10do_unshiftER9_MbstatetPcS3_RS3_
_ZNKSt25__codecvt_utf8_utf16_baseIwE5do_inER9_MbstatetPKcS4_RS4_PwS6_RS6_
_ZNKSt25__codecvt_utf8_utf16_baseIwE6do_outER9_MbstatetPKwS4_RS4_PcS6_RS6_
_ZNKSt25__codecvt_utf8_utf16_baseIwE9do_lengthER9_MbstatetPKcS4_y
_ZNSt7codecvtIwc9_MbstatetEC2Ey

libllama.dll (3 missing):

_ZNSo5seekpESt4fposI9_MbstatetE
__emutls_v._ZSt11__once_call
__emutls_v._ZSt15__once_callable

libllama-common.dll (8 missing): the five above plus

_ZNSt15basic_streambufIcSt11char_traitsIcEE7seekposESt4fposI9_MbstatetESt13_Ios_Openmode
__emutls_v._ZSt11__once_call
__emutls_v._ZSt15__once_callable

Analysis

The missing symbols identify a mismatch on two independent axes.

1. Threading model. The importers reference __emutls_v._ZSt11__once_call and
   __emutls_v._ZSt15__once_callable, which a posix-threads libstdc++ emits.
   The bundled libstdc++-6.dll instead exports __get_once_callv and
   __get_once_callablev, which is the win32-threads form.

2. C runtime. The importers reference symbols mangled with 9_Mbstatet, which
   is the UCRT mbstate_t. The bundled libstdc++-6.dll imports msvcrt.dll,
   not the api-ms-win-crt-* UCRT stubs that the other runtime DLLs import.

So the ggml and llama DLLs are built with a UCRT posix-threads toolchain while
the bundled support DLL is an MSVCRT win32-threads build.

Confirmed by replacing only libstdc++-6.dll, libgcc_s_seh-1.dll and
libwinpthread-1.dll with the UCRT posix-threads equivalents (WinLibs
winlibs-x86_64-posix-seh-gcc-16.1.0-mingw-w64ucrt-14.0.0-r2). After the swap
the import diff reports zero missing symbols for every DLL in the package, all
11 DLLs load, and the Vulkan backend enumerates the device correctly.

Likely origin

scripts/windows-native-runtime-deps.py:

• default_search_dirs() puts $VULKAN_SDK/Bin and $VULKAN_SDK/Bin32 ahead
  of every PATH entry.
• collect_dependencies() builds a filename keyed index and copies the first
  match, with no check that the resolved DLL is ABI compatible with the
  importer.

The LunarG SDK ships its own GCC runtime DLLs, so it wins that lookup.

scripts/build-llama.sh already documents this exact hazard around line 166,
including the resulting STATUS_ENTRYPOINT_NOT_FOUND (0xc0000139), and works
around it by putting the selected compiler's runtime directory first on PATH.
That workaround is applied to the build step only. The packaging script never
received the equivalent fix.

verify_dependencies() only reports missing files, so a resolved but ABI
incompatible DLL passes verification and ships.

Suggested fix

1. In windows-native-runtime-deps.py, resolve MinGW runtime DLLs from the
   toolchain that built the artifacts (for example the directory of the g++
   used by build-llama.sh) and search that before $VULKAN_SDK.
2. Extend verify_dependencies() to check that every imported symbol is
   exported by the DLL that was actually bundled. That turns this class of
   failure into a build error instead of a broken release.

[rileyorscheln]

Independent reproduction on v0.74.0 with symbol-level detail, plus two findings
I don't think are in the thread yet: the CPU artifact is broken too, and the
packaging cause is localizable to two specific lines.

Environment — MeshLLM 0.74.0, Skippy ABI 0.1.32, Windows 11 Pro 26200
x86_64, AMD Radeon RX 9070 XT (Vulkan 1.4.349, driver 26.7.1 LLPC). Consumer is
Buzz Desktop v0.5.5 via the embedded mesh-llm-host-runtime
(initialize_host_runtime()).

Artifact meshllm-native-runtime-windows-x86_64-vulkan, SHA256
7fd3561550c34ed674d00698595304e46f725c065837a6dcc81bd68c708da802 — matches
native-runtimes.json (1ef2565126027370ee580c3e8bfee4084acf4e2cef38f5f6c07c574c11dc4b91).
A freshly downloaded copy is byte-identical to the installed one, so this is not
local corruption.

This fails before any GPU work. mesh-llm gpus enumerates the card
correctly (Vulkan0, 16 GB). No model is loaded and no Vulkan device is used.

Failure

mesh native runtime failed to install or load: load native runtime
meshllm-native-runtime-windows-x86_64-vulkan from <cache>: LoadLibraryExW failed

Loading each library by full path in topological order (so this is not a DLL
search-path issue):

libwinpthread-1.dll  OK     ggml-base.dll    OK
libgcc_s_seh-1.dll   OK     ggml-cpu.dll     OK
libstdc++-6.dll      OK     ggml-vulkan.dll  OK
libgomp-1.dll        OK
ggml.dll             FAIL  GetLastError=127  ERROR_PROC_NOT_FOUND
libllama-common.dll  FAIL  126 (cascade)
libmtmd.dll          FAIL  126 (cascade)
libllama.dll         FAIL  126 (cascade)

ggml.dll is the first failing module.

Symbols

ggml.dll imports :  _ZNSt7codecvtIwc9_MbstatetEC2Ey   codecvt<wchar_t,char,_Mbstatet>
libstdc++ exports:  _ZNSt7codecvtIwciEC2Ey            codecvt<wchar_t,char,int>

bundled libstdc++-6.dll:  "9_Mbstatet" forms = 0 , int-mbstate forms = 8

Unresolved against the bundled libstdc++-6.dll (7,052 exports):

DLL	libstdc++ imports	unresolved
ggml.dll	54	5
libllama.dll	169	1 — _ZNSo5seekpESt4fposI9_MbstatetE
libllama-common.dll	270	6
ggml-base / ggml-cpu / ggml-vulkan / libmtmd	35/17/92/58	0

Every unresolved symbol contains 9_Mbstatet or fposI9_MbstatetE.

The CPU artifact is also broken, differently

Artifact	Toolchain	MinGW support DLLs	Probe
vulkan	MinGW (lib*)	bundled, wrong ABI	ggml.dll → 127
cpu	MinGW (lib*)	none shipped	ggml-base.dll → 127
rocm	MSVC (MSVCP140/VCRUNTIME140)	n/a	base/cpu OK; ggml-hip.dll 126 (no HIP SDK, expected)

The CPU artifact's DLLs import libstdc++-6.dll, libgcc_s_seh-1.dll, and
libwinpthread-1.dll, but the package ships none of them. On a host with any
unrelated MinGW runtime on PATH it binds to that and fails the same way; on a
clean host it should fail with 126. Only the MSVC-built ROCm package is
internally self-consistent.

(CPU e83f054d836ac75a6ceb0298d72672fb07052fce806cd201727d02981b4b939b,
ROCm b599faf77758a9bbf575e2ba0e817dd661426f7003a89ce379fab338fdaf04e5 — both
checksum-verified.)

Where the bad DLL comes from

scripts/package-native-runtime.sh:379 runs collection only for Vulkan,
which is why CPU ships nothing:

if [[ "$runtime_os" == "windows" && "$BACKEND" == "vulkan" ]]; then

scripts/windows-native-runtime-deps.py:131-143 then prefers the LunarG SDK
over the compiler's own runtime:

def default_search_dirs() -> list[pathlib.Path]:
    vulkan_sdk = os.environ.get("VULKAN_SDK")
    if vulkan_sdk:
        candidates.extend([Path(vulkan_sdk)/"Bin", Path(vulkan_sdk)/"Bin32"])   # first
    candidates.extend(PATH entries)                                              # after

_dll_index() (line 157) uses setdefault — first match wins — and
collect_dependencies() (line 186) searches
[lib_dir, *search_dirs, *default_search_dirs()], so the compiler's runtime is
only reachable after $VULKAN_SDK/Bin.

This is the hazard scripts/build-llama.sh:133-149 already documents:

# The LunarG SDK ships GCC runtime DLLs in its Bin directories. GitHub Actions
# prepends those directories to PATH, so a MinGW-built vulkan-shaders-gen can
# load the SDK's older libstdc++ instead of the runtime matching its compiler
# and fail with STATUS_ENTRYPOINT_NOT_FOUND (0xc0000139).

PR #963 applied that guard to the build step. The packaging collector was
not covered — and explicitly prioritizes the SDK.

Why checksum and doctor checks didn't catch it

scripts/verify-native-runtime-package.sh:18 verifies that "Windows
non-system DLL imports are present in the artifact" — dependency_gaps()
compares imported DLL names against packaged filenames and never resolves a
symbol. Right filenames + wrong ABI passes cleanly.

Checksum verification passes for the same reason: the artifact is exactly what
was published; what was published is internally inconsistent.

mesh-llm doctor compounds it — it reports
status: ok / flavor: vulkan / installed: 1 from manifest metadata and never
calls LoadLibrary, so it declares healthy a runtime that cannot load. That
sent me looking at the host application first.

Proposed packaging fix

1. Stage MinGW support DLLs from the compiler's own runtime directory
   (dirname $(command -v g++)), not $VULKAN_SDK/Bin — drop $VULKAN_SDK
   from default_search_dirs(), or order it after PATH, or pass the compiler
   runtime dir as the first --search-dir. Same fix PR Fix Windows Vulkan native runtime release build #963 applied to the
   build step.
2. For CPU: either run the same collection step for non-Vulkan MinGW backends,
   or link CPU/Vulkan with -static-libstdc++ -static-libgcc and ship no MinGW
   runtime at all — which removes this bug class entirely.
3. Include the Win32 error in the host message:
   LoadLibraryExW failed: <path>: GetLastError=127 (The specified procedure could not be found). Without it, identifying this took a dependency-ordered
   probe plus symbol-level import/export diffing.

Proposed CI loadability test

Manifest and checksum verification both pass today, so neither is sufficient.
The test must load the packaged runtime and resolve its entry point:

# after packaging, on a clean Windows runner with no MinGW/MSYS2 on PATH
import ctypes, json, pathlib
root = pathlib.Path(artifact_dir)
manifest = json.load(open(root / "manifest.json"))
for rel in manifest["runtime"]["libraries"]:          # topological order
    ctypes.WinDLL(str(root / rel))
primary = ctypes.WinDLL(str(root / manifest["build"]["primary_library"]))
assert primary.llama_backend_init                      # resolve a real export

Two properties matter: a clean runner (unrelated MinGW on PATH masks the CPU
defect), and resolving an exported symbol rather than merely loading.

Happy to test a corrected artifact on this hardware.

[i386]

Investigated against current main. The CPU collector is now unconditional, but the remaining ABI bug is reproducible in the resolver's search order: same-named MinGW DLLs from the Vulkan SDK can win over the compiler runtime.

Draft fix: #1201 — compiler runtime discovery is prioritized before PATH/SDK candidates, and Windows CPU/Vulkan packaging passes the compiler directory before SDK directories. Added a collision regression test; the full script suite passes (386 tests, 7 skipped). Clean Windows artifact loadability remains to be exercised on CI.