uv installation broken because of submodule error

[andrei-apollo]

When installing tinyBenchmarks as a Python dependency, pip and poetry work fine, but uv fails with

   Updating https://github.com/felipemaiapolo/tinyBenchmarks.git (9c7e20302301ad531bfdfd9a7288e6e916bf22e9)
  × Failed to download and build `tinybenchmarks @ git+https://github.com/felipemaiapolo/tinyBenchmarks.git@9c7e20302301ad531bfdfd9a7288e6e916bf22e9`
  ├─▶ Git operation failed
  ╰─▶ process didn't exit successfully: `/opt/homebrew/bin/git submodule update --recursive --init` (exit status: 128)
      --- stderr
      fatal: No url found for submodule path 'tutorials/py-irt' in .gitmodules

[elichen3051]

Hi @andrei-apollo ,

This is how I use TinyBenchmarks with uv installation.

1. Install the prerequisite and tinybenchmarks

# Use python=3.12
uv pip install torch torchvision torchaudio
uv pip install transformers datasets==2.21.0 accelerate
uv pip install sentencepiece protobuf

# Install lm-evaluation harness locally
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
git checkout v0.4.3
uv pip install -e .
cd ..

# Install the flash-attn you want.
uv pip install flash-attn==2.7.2.post2 --no-build-isolation

# Install the TinyBenchmarks locally.
git clone https://github.com/felipemaiapolo/tinyBenchmarks.git
uv pip install -e tinyBenchmarks

2. Modify the importation of the evaluator of tinybenchmarks in lm-evalution-harness

path: .venv/lib/python3.12/site-packages/lm_eval/tasks/tinyBenchmarks/agg_functions.py
try:
    import tinyBenchmarks.tinyBenchmarks as tb

3. Run the lm_eval, here is an example:

export MODEL="meta-llama/Llama-3.1-8B"
export OUT_DIR="resutls/$MODEL/"
echo $MODEL

accelerate launch -m lm_eval --model hf \
    --model_args pretrained=$MODEL,attn_implementation=flash_attention_2,dtype="bfloat16" \
    --tasks tinyMMLU \
    --num_fewshot 5 \
    --output_path $OUT_DIR/mmlu-5-shot \
    --batch_size 1

4. You can see the result like this:

[andrei-apollo]

Hi @elichen3051 thanks for the instructions!

Let me share my solution as well. I'm using pyproject.toml and install all dependencies with uv sync. So I wanted to avoid any manual steps and instead created a fork of tinyBenchmarks, which is completely equivalent except for the fact that it does not have the broken submodule. Now I'm able to directly import tinyBenchmarks like this:

dependencies = [
    ...
    "tinyBenchmarks @ git+https://github.com/andrei-apollo/tinyBenchmarks.git@0b769cf382704eb0bc5c1fb84084c075ccb93229",
]

[couragec]

你好@andrei-apollo，

这就是我使用带有 uv 安装的 TinyBenchmarks 的方法。

1. 安装先决条件和 tinybenchmarks

# Use python=3.12
uv pip install torch torchvision torchaudio
uv pip install transformers datasets==2.21.0 accelerate
uv pip install sentencepiece protobuf

# Install lm-evaluation harness locally
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
git checkout v0.4.3
uv pip install -e .
cd ..

# Install the flash-attn you want.
uv pip install flash-attn==2.7.2.post2 --no-build-isolation

# Install the TinyBenchmarks locally.
git clone https://github.com/felipemaiapolo/tinyBenchmarks.git
uv pip install -e tinyBenchmarks

2. 修改 lm-evalution-harness 中 tinybenchmarks 评估器的导入

path: .venv/lib/python3.12/site-packages/lm_eval/tasks/tinyBenchmarks/agg_functions.py
try:
    import tinyBenchmarks.tinyBenchmarks as tb

3. 运行 lm_eval，下面是一个例子：

export MODEL="meta-llama/Llama-3.1-8B"
export OUT_DIR="resutls/$MODEL/"
echo $MODEL

accelerate launch -m lm_eval --model hf \
    --model_args pretrained=$MODEL,attn_implementation=flash_attention_2,dtype="bfloat16" \
    --tasks tinyMMLU \
    --num_fewshot 5 \
    --output_path $OUT_DIR/mmlu-5-shot \
    --batch_size 1

4. 你可以看到这样的结果：

Hi @andrei-apollo ,

This is how I use TinyBenchmarks with uv installation.

1. Install the prerequisite and tinybenchmarks

# Use python=3.12
uv pip install torch torchvision torchaudio
uv pip install transformers datasets==2.21.0 accelerate
uv pip install sentencepiece protobuf

# Install lm-evaluation harness locally
git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
git checkout v0.4.3
uv pip install -e .
cd ..

# Install the flash-attn you want.
uv pip install flash-attn==2.7.2.post2 --no-build-isolation

# Install the TinyBenchmarks locally.
git clone https://github.com/felipemaiapolo/tinyBenchmarks.git
uv pip install -e tinyBenchmarks

2. Modify the importation of the evaluator of tinybenchmarks in lm-evalution-harness

path: .venv/lib/python3.12/site-packages/lm_eval/tasks/tinyBenchmarks/agg_functions.py
try:
    import tinyBenchmarks.tinyBenchmarks as tb

3. Run the lm_eval, here is an example:

export MODEL="meta-llama/Llama-3.1-8B"
export OUT_DIR="resutls/$MODEL/"
echo $MODEL

accelerate launch -m lm_eval --model hf \
    --model_args pretrained=$MODEL,attn_implementation=flash_attention_2,dtype="bfloat16" \
    --tasks tinyMMLU \
    --num_fewshot 5 \
    --output_path $OUT_DIR/mmlu-5-shot \
    --batch_size 1

4. You can see the result like this:

你好，请问支持本地模型吗，比如qwen-7b，我遇到一个问题：safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge