Closed
Description
Name and Version
$ build/bin/llama-cli --version
version: 4752 (586d5fe6)
built with Homebrew clang version 19.1.7 for arm64-apple-darwin24.1.0
Operating systems
Mac, Other? (Please let us know in description)
GGML backends
Metal, CPU
Hardware
For MacBook,
$ sysctl -a | grep machdep.cpu
machdep.cpu.cores_per_package: 10
machdep.cpu.core_count: 10
machdep.cpu.logical_per_package: 10
machdep.cpu.thread_count: 10
machdep.cpu.brand_string: Apple M1 Pro
For IBM Z & LinuxONE:
IBM z15 8 IFLs / 64 GB RAM / NOSMT / LPAR
Models
- ibm-granite/granite-vision-3.1-2b-preview
- ibm-granite/granite-vision-3.2-2b
Problem description & steps to reproduce
While following the LLaVa Granite Vision README steps, the surgery script python llava_surgery_v2.py -C -m $GRANITE_MODEL
fails to detect tensors to extract.
- git clone https://huggingface.co/ibm-granite/granite-vision-3.1-2b-preview
- cd granite-vision-3.1-2b-preview
- git lfs pull
- export GRANITE_MODEL=/absolute/path/to/granite-vision-3.1-2b-preview
- python3 llava_surgery_v2.py -C -m $GRANITE_MODEL
- The script fails to surgically split
llava.clip
andllava.projector
As a result, no llava.clip
and llava.projector
files were generated from the operation.
Refer to the logs below for a copy. Tested on both M1 MacBook Pro and IBM z15, and the same problem occurs.
Please feel free to contact me on IBM Slack at @/Aaron Teo for quicker communication (IBM-internal only)
First Bad Commit
No response
Relevant log output
$ ls -la $GRANITE_MODEL
total 23264488
drwxr-xr-x 20 taronaeo staff 640 Mar 5 22:09 .
drwxr-xr-x 26 taronaeo staff 832 Mar 5 22:09 ..
drwxr-xr-x 3 taronaeo staff 96 Mar 5 22:07 .cache
-rw-r--r-- 1 taronaeo staff 1567 Mar 5 22:07 .gitattributes
-rw-r--r-- 1 taronaeo staff 9870 Mar 5 22:07 README.md
-rw-r--r-- 1 taronaeo staff 116 Mar 5 22:07 added_tokens.json
-rw-r--r-- 1 taronaeo staff 1643 Mar 5 22:07 chat_template.json
-rw-r--r-- 1 taronaeo staff 3107 Mar 5 22:07 config.json
-rw-r--r-- 1 taronaeo staff 361043 Mar 5 22:07 example.png
-rw-r--r-- 1 taronaeo staff 137 Mar 5 22:07 generation_config.json
-rw-r--r-- 1 taronaeo staff 441810 Mar 5 22:07 merges.txt
-rw-r--r-- 1 taronaeo staff 9999947128 Mar 5 22:09 model-00001-of-00002.safetensors
-rw-r--r-- 1 taronaeo staff 1901748968 Mar 5 22:08 model-00002-of-00002.safetensors
-rw-r--r-- 1 taronaeo staff 86365 Mar 5 22:07 model.safetensors.index.json
-rw-r--r-- 1 taronaeo staff 1993 Mar 5 22:07 preprocessor_config.json
-rw-r--r-- 1 taronaeo staff 115 Mar 5 22:07 processor_config.json
-rw-r--r-- 1 taronaeo staff 701 Mar 5 22:07 special_tokens_map.json
-rw-r--r-- 1 taronaeo staff 2057967 Mar 5 22:07 tokenizer.json
-rw-r--r-- 1 taronaeo staff 6974 Mar 5 22:07 tokenizer_config.json
-rw-r--r-- 1 taronaeo staff 776995 Mar 5 22:07 vocab.json
$ python3 ~/Documents/llama.cpp/examples/llava/llava_surgery_v2.py -C -m $GRANITE_MODEL
Cleaning ./granite-vision-3.1-2b-preview/model-00001-of-00002.safetensors
image_newline : torch.Size([2048])
... truncating ...
vision_tower.vision_model.head.probe : torch.Size([1, 1, 1152])
vision_tower.vision_model.post_layernorm.bias : torch.Size([1152])
vision_tower.vision_model.post_layernorm.weight : torch.Size([1152])
Taking projector from None
Found 0 tensors to extract out of 0 tensors.
No tensors found. Is this a LLaVA model?
$ pip freeze
certifi==2025.1.31
charset-normalizer==3.4.1
filelock==3.17.0
fsspec==2025.2.0
gguf==0.14.0
huggingface-hub==0.29.1
idna==3.10
Jinja2==3.1.5
MarkupSafe==3.0.2
mpmath==1.3.0
networkx==3.4.2
numpy==1.26.4
packaging==24.2
protobuf==4.25.6
PyYAML==6.0.2
regex==2024.11.6
requests==2.32.3
safetensors==0.5.3
sentencepiece==0.2.0
sympy==1.13.3
tokenizers==0.21.0
torch==2.2.2
tqdm==4.67.1
transformers==4.49.0
typing_extensions==4.12.2
urllib3==2.3.0