TVM FFI Error with flashinfer-python==0.4.0

[trevor-m]

After upgrading from flashinfer-python 0.4.0rc3 to 0.4.0, I'm getting this error when trying to use Sglang. It appears to occur during DeepGemm JIT.

Traceback (most recent call last):
  File "./dlfcn/dlopen.c", line 81, in ___dlopen
  File "./dlfcn/dlopen.c", line 71, in dlopen_implementation
  File "./dlfcn/dlerror.c", line 138, in _dlerror_run
  File "./elf/dl-error-skeleton.c", line 227, in __GI__dl_catch_error
  File "./elf/dl-error-skeleton.c", line 208, in __GI__dl_catch_exception
  File "./dlfcn/dlopen.c", line 56, in dlopen_doit
  File "./elf/dl-open.c", line 883, in _dl_open
  File "./elf/dl-error-skeleton.c", line 208, in __GI__dl_catch_exception
  File "./elf/dl-open.c", line 771, in dl_open_worker
  File "./elf/dl-open.c", line 808, in dl_open_worker
  File "./elf/dl-error-skeleton.c", line 182, in __GI__dl_catch_exception
  File "./elf/dl-init.c", line 117, in _dl_init
  File "./elf/dl-init.c", line 33, in call_init
  File "./elf/dl-init.c", line 70, in call_init
  File "<unknown>", line 0, in _GLOBAL__sub_I_analyzer.cc
  File "<unknown>", line 0, in tvm::ffi::NDArrayObj::_GetOrAllocRuntimeTypeIndex()
  File "<unknown>", line 0, in TVMFFITypeGetOrAllocIndex
  File "src/ffi/object.cc", line 133, in tvm::ffi::TypeTable::GetOrAllocTypeIndex(tvm::ffi::String, int32_t, int32_t, int32_t, bool, int32_t)::<lambda()>
InternalError: Check failed: (type_table_[static_type_index] == nullptr) is false: Conflicting static index 72 between ffi.Map and ffi.NDArray

Repro

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 --enable-dp-attention --host 0.0.0.0 --port 30000

[tqchen]

this is likely due to an older version of ffi that was used during rc, perhaps due to jit cache that contains libraries compiled in older version. We are stablizing tvm ffi abi this week, so hopefully same issue won't occur in future ones. please clear the jit cache and try

[trevor-m]

Thanks @tqchen for the explanation. Since I'm using a docker container (added repro command above) I don't think there is a stale jit cache, but could you please let me know how I can clear it just in case?

[yzh119]

You can try flashinfer clear-cache in command line to clear cache.

What's the output of python -c "import tvm_ffi; print(tvm_ffi.__version__) btw.

[trevor-m]

Thanks @yzh119 I will try that. The version is 0.1.0b15
Edit: Clearing cache didn't help.

[tqchen]

do u mind dig a bit File "./dlfcn/dlopen.c", line 81, in ___dlopen which dll exactly triggered the error, that might help locate the issue. Background on the error. the latest tvm_ffi version have map type code as 72,

that update was switched in in this commit (a mo ago), where previously NDArray type code was 72

apache/tvm-ffi@777cf8d

So likely what happens in your case is that we are loading a dynamic library(so) that was compiled on the older version, and triggers this error when registering back to the system

[trevor-m]

@tqchen LD_DEBUG=libs shows that /usr/local/lib/python3.12/dist-packages/tilelang/lib/libtvm.so is being loaded right as the error occurs. I'm guessing that lib is on the older FFI version?

[tqchen]

ah yes, that is likely the cause. cc @LeiWang1999 , @Hzfengsy would be great if tilelang can bump soon as we can sync to the latest stable abi

[trevor-m]

FYI Sglang has removed it's dependency on tilelang so it is no longer affected by this issue: sgl-project/sglang#11455 sgl-project/sglang#11450

[yzh119]

Closed for now, we need to address the tvm-ffi version discrepancy between tilelang and flashinfer, at TVM-side, the version bumped in apache/tvm#18371 and I believe it will take some time to for tilelang to update.