DeepSeek-R1-Distill-Qwen-7B with uv + SGLang

This repo runs DeepSeek-R1-Distill-Qwen-7B with SGLang and evaluates GSM8K.

1) Install dependencies

cd /home/zhj/Paper
uv sync

2) Start SGLang server

FLASHINFER_WORKSPACE_DIR=/tmp/flashinfer \
UV_CACHE_DIR=/tmp/uv-cache \
uv run python -m sglang.launch_server \
  --model-path /home/zhj/models/DeepSeek-R1-Distill-Qwen-7B \
  --trust-remote-code \
  --dtype bfloat16 \
  --host 0.0.0.0 \
  --port 30000 \
  --context-length 8192 \
  --mem-fraction-static 0.85 \
  --tensor-parallel-size 1

For multi-GPU, set --tensor-parallel-size to the GPU count (for example 4). If you explicitly enable --reasoning-parser deepseek-r1, outputs may appear in reasoning_content.

3) Run one inference request

UV_CACHE_DIR=/tmp/uv-cache \
uv run python infer_local.py \
  --base-url http://127.0.0.1:30000/v1 \
  --model /home/zhj/models/DeepSeek-R1-Distill-Qwen-7B \
  --prompt "用中文解释注意力机制为什么有效"

4) Evaluate GSM8K accuracy

UV_CACHE_DIR=/tmp/uv-cache \
uv run python eval_gsm8k.py \
  --base-url http://127.0.0.1:30000/v1 \
  --model /home/zhj/models/DeepSeek-R1-Distill-Qwen-7B \
  --dataset-dir /home/zhj/Paper/gsm8k_main \
  --split test \
  --prompt-style final_only \
  --batch-size 16 \
  --concurrency 4 \
  --max-new-tokens 256 \
  --save-jsonl /home/zhj/Paper/results/gsm8k_sglang_qwen7b.jsonl

The script prints final accuracy and saves per-sample outputs to results/.