Skip to content

Eval bug: Granite Vision 3.1 and 3.2 Surgery Script Found 0 Tensors to Extract #12202

Closed
@taronaeo

Description

@taronaeo

Name and Version

$ build/bin/llama-cli --version
version: 4752 (586d5fe6)
built with Homebrew clang version 19.1.7 for arm64-apple-darwin24.1.0

Operating systems

Mac, Other? (Please let us know in description)

GGML backends

Metal, CPU

Hardware

For MacBook,

$ sysctl -a | grep machdep.cpu
machdep.cpu.cores_per_package: 10
machdep.cpu.core_count: 10
machdep.cpu.logical_per_package: 10
machdep.cpu.thread_count: 10
machdep.cpu.brand_string: Apple M1 Pro

For IBM Z & LinuxONE:

IBM z15 8 IFLs / 64 GB RAM / NOSMT / LPAR

Models

  1. ibm-granite/granite-vision-3.1-2b-preview
  2. ibm-granite/granite-vision-3.2-2b

Problem description & steps to reproduce

While following the LLaVa Granite Vision README steps, the surgery script python llava_surgery_v2.py -C -m $GRANITE_MODEL fails to detect tensors to extract.

  1. git clone https://huggingface.co/ibm-granite/granite-vision-3.1-2b-preview
  2. cd granite-vision-3.1-2b-preview
  3. git lfs pull
  4. export GRANITE_MODEL=/absolute/path/to/granite-vision-3.1-2b-preview
  5. python3 llava_surgery_v2.py -C -m $GRANITE_MODEL
  6. The script fails to surgically split llava.clip and llava.projector

As a result, no llava.clip and llava.projector files were generated from the operation.

Refer to the logs below for a copy. Tested on both M1 MacBook Pro and IBM z15, and the same problem occurs.

Please feel free to contact me on IBM Slack at @/Aaron Teo for quicker communication (IBM-internal only)

First Bad Commit

No response

Relevant log output

$ ls -la $GRANITE_MODEL
total 23264488
drwxr-xr-x  20 taronaeo  staff         640 Mar  5 22:09 .
drwxr-xr-x  26 taronaeo  staff         832 Mar  5 22:09 ..
drwxr-xr-x   3 taronaeo  staff          96 Mar  5 22:07 .cache
-rw-r--r--   1 taronaeo  staff        1567 Mar  5 22:07 .gitattributes
-rw-r--r--   1 taronaeo  staff        9870 Mar  5 22:07 README.md
-rw-r--r--   1 taronaeo  staff         116 Mar  5 22:07 added_tokens.json
-rw-r--r--   1 taronaeo  staff        1643 Mar  5 22:07 chat_template.json
-rw-r--r--   1 taronaeo  staff        3107 Mar  5 22:07 config.json
-rw-r--r--   1 taronaeo  staff      361043 Mar  5 22:07 example.png
-rw-r--r--   1 taronaeo  staff         137 Mar  5 22:07 generation_config.json
-rw-r--r--   1 taronaeo  staff      441810 Mar  5 22:07 merges.txt
-rw-r--r--   1 taronaeo  staff  9999947128 Mar  5 22:09 model-00001-of-00002.safetensors
-rw-r--r--   1 taronaeo  staff  1901748968 Mar  5 22:08 model-00002-of-00002.safetensors
-rw-r--r--   1 taronaeo  staff       86365 Mar  5 22:07 model.safetensors.index.json
-rw-r--r--   1 taronaeo  staff        1993 Mar  5 22:07 preprocessor_config.json
-rw-r--r--   1 taronaeo  staff         115 Mar  5 22:07 processor_config.json
-rw-r--r--   1 taronaeo  staff         701 Mar  5 22:07 special_tokens_map.json
-rw-r--r--   1 taronaeo  staff     2057967 Mar  5 22:07 tokenizer.json
-rw-r--r--   1 taronaeo  staff        6974 Mar  5 22:07 tokenizer_config.json
-rw-r--r--   1 taronaeo  staff      776995 Mar  5 22:07 vocab.json

$ python3 ~/Documents/llama.cpp/examples/llava/llava_surgery_v2.py -C -m $GRANITE_MODEL
Cleaning ./granite-vision-3.1-2b-preview/model-00001-of-00002.safetensors
image_newline : torch.Size([2048])
... truncating ...
vision_tower.vision_model.head.probe : torch.Size([1, 1, 1152])
vision_tower.vision_model.post_layernorm.bias : torch.Size([1152])
vision_tower.vision_model.post_layernorm.weight : torch.Size([1152])
Taking projector from None
Found 0 tensors to extract out of 0 tensors.
No tensors found. Is this a LLaVA model?

$ pip freeze
certifi==2025.1.31
charset-normalizer==3.4.1
filelock==3.17.0
fsspec==2025.2.0
gguf==0.14.0
huggingface-hub==0.29.1
idna==3.10
Jinja2==3.1.5
MarkupSafe==3.0.2
mpmath==1.3.0
networkx==3.4.2
numpy==1.26.4
packaging==24.2
protobuf==4.25.6
PyYAML==6.0.2
regex==2024.11.6
requests==2.32.3
safetensors==0.5.3
sentencepiece==0.2.0
sympy==1.13.3
tokenizers==0.21.0
torch==2.2.2
tqdm==4.67.1
transformers==4.49.0
typing_extensions==4.12.2
urllib3==2.3.0

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions