diff --git a/docs/user-guide/dine-in/get-started.md b/docs/user-guide/dine-in/get-started.md index a2826c06..7e0ee0a1 100644 --- a/docs/user-guide/dine-in/get-started.md +++ b/docs/user-guide/dine-in/get-started.md @@ -60,7 +60,7 @@ cd ../dine-in > **Note:** Only needed once. Model files are shared between Dine-In and Take-Away. -This downloads Qwen2.5-VL-7B-Instruct (~7 GB) and converts it to OpenVINO™ INT8 format. This is only needed once — the model files are shared with Take-Away. +This downloads MiniCPM-V-4.5 and converts it to OpenVINO™ INT4 format. This is only needed once — the model files are shared with Take-Away. ### Step 4: Prepare Test Data diff --git a/docs/user-guide/dine-in/get-started/system-requirements.md b/docs/user-guide/dine-in/get-started/system-requirements.md index 61e5a007..3db5ec10 100644 --- a/docs/user-guide/dine-in/get-started/system-requirements.md +++ b/docs/user-guide/dine-in/get-started/system-requirements.md @@ -24,7 +24,7 @@ Hardware, software, and network requirements for deploying Dine-In Order Accurac | GPU | Intel® Data Center GPU (for concurrent validation) | | Storage | 200 GB NVMe SSD | -**GPU VRAM guidance:** The Qwen2.5-VL-7B INT8 model requires ~8 GB of VRAM. +**GPU VRAM guidance:** The MiniCPM-V-4.5 INT4 model requires ~5 GB of VRAM. The default `cache_size=4` reserves an additional 4 GB VRAM for the KV cache. Total VRAM needed is around 12 GB, which fits in an Intel® Arc™ A770 16 GB. On **integrated GPU** (iGPU) platforms such as Wildcat Lake and Meteor Lake, the KV cache is drawn from **system RAM** diff --git a/docs/user-guide/dine-in/how-it-works.md b/docs/user-guide/dine-in/how-it-works.md index c720b682..87349b6c 100644 --- a/docs/user-guide/dine-in/how-it-works.md +++ b/docs/user-guide/dine-in/how-it-works.md @@ -36,7 +36,7 @@ This document provides a comprehensive technical overview of the system architec ┌─────────────────┐ ┌─────────────────┐ │ │ │ │ │ OVMS VLM │ │ Semantic │ - │ (Qwen2.5-VL) │ │ Service │ + │ (MiniCPM-V4.5) │ │ Service │ │ Port 8000 │ │ Port 8080 │ │ │ │ │ └─────────────────┘ └─────────────────┘ diff --git a/docs/user-guide/dine-in/index.md b/docs/user-guide/dine-in/index.md index b872d1d5..c8b04268 100644 --- a/docs/user-guide/dine-in/index.md +++ b/docs/user-guide/dine-in/index.md @@ -30,7 +30,7 @@ In a full-service restaurant: | Feature | Description | | ---------------------------- | -------------------------------------------------------------------------------- | -| **Zero-Training Deployment** | Uses pre-trained Qwen2.5-VL-7B model - no fine-tuning required | +| **Zero-Training Deployment** | Uses pre-trained MiniCPM-V-4.5 model - no fine-tuning required | | **Semantic Matching** | Fuzzy item matching handles naming variations (e.g., "Big Mac" ↔ "Maharaja Mac") | | **Real-Time Validation** | Sub-15-second end-to-end latency for operational efficiency | | **Circuit Breaker Pattern** | Fault-tolerant with automatic service recovery | diff --git a/docs/user-guide/get-started/advanced.md b/docs/user-guide/get-started/advanced.md index 24ba581f..2f29466f 100644 --- a/docs/user-guide/get-started/advanced.md +++ b/docs/user-guide/get-started/advanced.md @@ -66,7 +66,8 @@ LOG_LEVEL=INFO # Service Endpoints # ============================================================================= OVMS_ENDPOINT=http://ovms-vlm:8000 -OVMS_MODEL_NAME=Qwen/Qwen2.5-VL-7B-Instruct +OVMS_MODEL_NAME=openbmb/MiniCPM-V-4_5-int4 +VLM_PRECISION=int4 # must match the -int4 suffix in OVMS_MODEL_NAME SEMANTIC_SERVICE_ENDPOINT=http://semantic-service:8080 API_TIMEOUT=60 ``` @@ -106,7 +107,10 @@ API_TIMEOUT=60 # ============================================================================= VLM_BACKEND=ovms OVMS_ENDPOINT=http://ovms-vlm:8000 -OVMS_MODEL_NAME=Qwen/Qwen2.5-VL-7B-Instruct +OVMS_MODEL_NAME=openbmb/MiniCPM-V-4_5-int4 +VLM_PRECISION=int4 # must match the -int4 suffix in OVMS_MODEL_NAME +VLM_IMAGE_MAX_SIZE=448 # square canvas size submitted to the VLM +VLM_ENABLE_THINKING=false # keep MiniCPM's reasoning off OPENVINO_DEVICE=GPU # 'GPU', 'CPU', or 'AUTO' # ============================================================================= diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 27d62324..05a2136f 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -52,7 +52,7 @@ Optimized for validating food trays at serving stations before delivery to table - REST API for POS integration - Gradio web interface for manual validation - Hybrid semantic matching -- Zero-training deployment with pre-trained Qwen2.5-VL-7B model +- Zero-training deployment with pre-trained MiniCPM-V-4.5 model #### Use Case @@ -109,7 +109,7 @@ Optimized for high-throughput drive-through environments with multiple camera st ### VLM Backend (OVMS) -Both applications use OpenVINO™ Model Server with Qwen2.5-VL for vision-language inference. +Both applications use OpenVINO™ Model Server with MiniCPM-V-4.5 (INT4) for vision-language inference. ### Semantic Comparison Service diff --git a/docs/user-guide/oa-get-started.md b/docs/user-guide/oa-get-started.md index 02ae5b9f..470d0573 100644 --- a/docs/user-guide/oa-get-started.md +++ b/docs/user-guide/oa-get-started.md @@ -49,7 +49,7 @@ cd ../dine-in ``` - This downloads and converts the Qwen2.5-VL-7B model (~7GB). This only needs to be done once. + This downloads and converts the MiniCPM-V-4.5 model to OpenVINO™ INT4. This only needs to be done once. 3. **Prepare Test Data** - Add your food tray/plate images to the `images/` folder diff --git a/docs/user-guide/oa-how-it-works.md b/docs/user-guide/oa-how-it-works.md index 32cee075..4a06b200 100644 --- a/docs/user-guide/oa-how-it-works.md +++ b/docs/user-guide/oa-how-it-works.md @@ -14,7 +14,7 @@ The Order Accuracy platform is an enterprise AI vision system designed for real- ### Key Features -- **VLM-Powered Detection**: Uses Qwen2.5-VL-7B for accurate item identification +- **VLM-Powered Detection**: Uses MiniCPM-V-4.5 (INT4) for accurate item identification - **Intel Hardware Optimization**: Optimized for Intel CPUs and GPUs via OpenVINO - **Dual Application Support**: Dine-In (image-based) and Take-Away (video stream-based) - **Semantic Matching**: Fuzzy matching for item name variations @@ -44,7 +44,7 @@ graph TB end subgraph "Shared Services" - OVMS[OVMS VLM
Qwen2.5-VL-7B] + OVMS[OVMS VLM
MiniCPM-V-4.5] SEM[Semantic Service] MINIO[MinIO Storage] end @@ -76,7 +76,7 @@ flowchart TB DVS --> DMET["Metrics Collector"] end - DVLM --> OVMS["OVMS VLM
(Qwen2.5-VL)"] + DVLM --> OVMS["OVMS VLM
(MiniCPM-V-4.5)"] DSEM --> SEM["Semantic Service"] ``` @@ -91,7 +91,7 @@ flowchart TB minio --> selector["Frame Selector
(YOLO11n-CPU)"] selector -->|top 3 frames| scheduler["VLM Scheduler
(ThreadPool)"] selector -->|top 3 frames| validation["Validation Agent"] - scheduler --> ovms["OVMS VLM
(Qwen2.5-VL, GPU-INT8)"] + scheduler --> ovms["OVMS VLM
(MiniCPM-V-4.5, GPU-INT4)"] validation --> semantic["Semantic Service"] end @@ -105,12 +105,12 @@ flowchart TB #### 1. VLM Backend (OVMS) -OpenVINO™ Model Server hosting Qwen2.5-VL-7B for vision-language inference. +OpenVINO™ Model Server hosting MiniCPM-V-4.5 for vision-language inference. **Features:** - OpenAI-compatible API (`/v3/chat/completions`) -- INT8 quantization for optimized performance +- INT4 weight compression for optimized performance - GPU acceleration via Intel/NVIDIA hardware - Shared model instance for both applications @@ -120,7 +120,7 @@ OpenVINO™ Model Server hosting Qwen2.5-VL-7B for vision-language inference. response = requests.post( f"{OVMS_ENDPOINT}/v3/chat/completions", json={ - "model": "Qwen/Qwen2.5-VL-7B-Instruct", + "model": "openbmb/MiniCPM-V-4_5-int4", "messages": [ { "role": "user", @@ -235,7 +235,7 @@ Request batching scheduler optimizing OVMS throughput. - Store selected frames in MinIO 3. **VLM Processing**: - - VLM Scheduler → OVMS (Qwen2.5-VL): + - VLM Scheduler → OVMS (MiniCPM-V-4.5): - Batch frames by time window - Send to OVMS with detection prompt - Parse structured item response diff --git a/docs/user-guide/take-away/get-started.md b/docs/user-guide/take-away/get-started.md index 383674da..d1e35426 100644 --- a/docs/user-guide/take-away/get-started.md +++ b/docs/user-guide/take-away/get-started.md @@ -82,7 +82,7 @@ cd ../take-away This downloads and exports: -- Qwen2.5-VL-7B-Instruct (OpenVINO™ format) +- MiniCPM-V-4_5 INT4 (OpenVINO™ format) - YOLOv11 model (INT8 OpenVINO™) - EasyOCR detection and recognition models @@ -112,9 +112,16 @@ make up # ============================================================================= VLM_BACKEND=ovms OVMS_ENDPOINT=http://ovms-vlm:8000 -OVMS_MODEL_NAME=Qwen/Qwen2.5-VL-7B-Instruct +OVMS_MODEL_NAME=openbmb/MiniCPM-V-4_5-int4 +VLM_PRECISION=int4 # must match the -int4 suffix in OVMS_MODEL_NAME TARGET_DEVICE=GPU # 'GPU' or 'CPU' — also set OPENVINO_DEVICE to match +# ============================================================================= +# VLM Image / Decoding Options +# ============================================================================= +VLM_IMAGE_MAX_SIZE=448 # frames are fitted into a 448x448 square canvas +VLM_ENABLE_THINKING=false # keep MiniCPM's reasoning disabled + # ============================================================================= # Inference Device (must match TARGET_DEVICE) # ============================================================================= @@ -138,6 +145,15 @@ MINIO_ENDPOINT=minio:9000 > **Changing the inference device:** Set both `TARGET_DEVICE` and `OPENVINO_DEVICE` to the same value (`GPU` or `CPU`), then re-run `./setup_models.sh --app take-away` to re-export the model for that device. +> **`VLM_ENABLE_THINKING`:** MiniCPM-V-4.5 is a hybrid reasoning model. Leave this +> `false`. When enabled, the model spends the whole token budget on a `` +> reasoning block and the detected-item list is truncated, which shows up as +> missing items and ~4x higher latency. + +> **`VLM_IMAGE_MAX_SIZE`:** Each selected frame is fitted into a square canvas of +> this size (aspect ratio preserved, white padding), matching the dine-in +> pipeline. Larger values increase detail but also prompt tokens and latency. + ### Validate Configuration ```bash diff --git a/docs/user-guide/take-away/get-started/build-from-source.md b/docs/user-guide/take-away/get-started/build-from-source.md index 67c2372c..ff834cad 100644 --- a/docs/user-guide/take-away/get-started/build-from-source.md +++ b/docs/user-guide/take-away/get-started/build-from-source.md @@ -78,7 +78,7 @@ cd ../take-away This downloads and exports: -- **Qwen2.5-VL-7B-Instruct** (OpenVINO™ INT8) → `ovms-service/models/` +- **MiniCPM-V-4_5** (OpenVINO™ INT4) → `ovms-service/models/` - **EasyOCR** models → `take-away/models/easyocr/` - **YOLO11n** (FP32 + INT8 OpenVINO) → `take-away/models/` diff --git a/docs/user-guide/take-away/get-started/system-requirements.md b/docs/user-guide/take-away/get-started/system-requirements.md index 23889b24..db4714ec 100644 --- a/docs/user-guide/take-away/get-started/system-requirements.md +++ b/docs/user-guide/take-away/get-started/system-requirements.md @@ -24,7 +24,7 @@ Hardware, software, and network requirements for deploying Take-Away Order Accur | **GPU** | Intel® Data Center GPU Max (48 GB) — for 4+ concurrent stations | | **Storage** | 200 GB NVMe SSD | -**GPU VRAM guidance:** The Qwen2.5-VL-7B INT8 model requires ~8 GB of VRAM. +**GPU VRAM guidance:** The MiniCPM-V-4_5 INT4 model requires ~5 GB of VRAM. The default `cache_size=4` reserves an additional 4 GB VRAM for the KV cache. Total VRAM needed is around 12 GB, which fits in an Intel® Arc™ A770 16 GB. On **integrated GPU** (iGPU) platforms such as Wildcat Lake and Meteor Lake, the KV cache is drawn from **system RAM** diff --git a/docs/user-guide/take-away/how-it-works.md b/docs/user-guide/take-away/how-it-works.md index 2718656f..74907d24 100644 --- a/docs/user-guide/take-away/how-it-works.md +++ b/docs/user-guide/take-away/how-it-works.md @@ -15,7 +15,7 @@ flowchart TB minio --> selector["Frame Selector
(YOLO11n-CPU)"] selector -->|top 3 frames| scheduler["VLM Scheduler
(ThreadPool)"] selector -->|top 3 frames| validation["Validation Agent"] - scheduler --> ovms["OVMS VLM
(Qwen2.5-VL, GPU-INT8)"] + scheduler --> ovms["OVMS VLM
(MiniCPM-V-4.5, GPU-INT4)"] validation --> semantic["Semantic Service"] end @@ -107,7 +107,7 @@ WORKERS=0 │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────┐ │ │ │ OVMS VLM (GPU) │ │ -│ │ Qwen2.5-VL-7B / Continuous Batching │ │ +│ │ MiniCPM-V-4.5 INT4 / Continuous Batching │ │ │ └──────────────────────────────────────────────────────────────────────┘ │ │ │ │ Characteristics: │ @@ -317,7 +317,7 @@ response = requests.post( - Store selected frames in MinIO 3. **VLM Processing**: - - VLM Scheduler → OVMS (Qwen2.5-VL): + - VLM Scheduler → OVMS (MiniCPM-V-4.5): - Batch frames by time window - Send to OVMS with detection prompt - Parse structured item response @@ -387,7 +387,7 @@ rtspsrc location= latency=0 buffer-mode=0 protocols=tcp ntp-sync=false do-r ┌─────────────────────────────────────────────────────────────────────────────────┐ │ OVMS VLM INTEGRATION │ │ │ -│ Model: Qwen/Qwen2.5-VL-7B-Instruct │ +│ Model: openbmb/MiniCPM-V-4_5-int4 │ │ │ │ ┌────────────────────────────────────────────────────────────────────────┐ │ │ │ OVMS Model Server │ │ @@ -395,12 +395,12 @@ rtspsrc location= latency=0 buffer-mode=0 protocols=tcp ntp-sync=false do-r │ │ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │ │ │ │ │ Vision │ │ Language │ │ Output │ │ │ │ │ │ Encoder │───▶│ Model │───▶│ Decoder │ │ │ -│ │ │ (ViT-based) │ │ (Qwen2.5) │ │ (JSON) │ │ │ +│ │ │ (SigLIP) │ │ (MiniCPM) │ │ (JSON) │ │ │ │ │ └────────────────┘ └────────────────┘ └────────────────┘ │ │ │ │ │ │ │ │ API: OpenAI-compatible /v3/chat/completions │ │ │ │ Port: 8001 (configurable) │ │ -│ │ Precision: INT8 (optimized for inference) │ │ +│ │ Precision: INT4 (optimized for inference) │ │ │ └────────────────────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────────────────────┘ @@ -412,7 +412,7 @@ rtspsrc location= latency=0 buffer-mode=0 protocols=tcp ntp-sync=false do-r ```json { - "model": "Qwen/Qwen2.5-VL-7B-Instruct", + "model": "openbmb/MiniCPM-V-4_5-int4", "messages": [ { "role": "user", @@ -426,10 +426,17 @@ rtspsrc location= latency=0 buffer-mode=0 protocols=tcp ntp-sync=false do-r } ], "max_completion_tokens": 100, - "temperature": 0.2 + "temperature": 0.2, + "chat_template_kwargs": { "enable_thinking": false } } ``` +> **`enable_thinking`:** MiniCPM-V-4.5 is a hybrid reasoning model. Unless +> thinking is explicitly disabled, its chat template opens a `` block and +> the entire `max_completion_tokens` budget is spent on reasoning, leaving the +> item list truncated. The service always sends `enable_thinking: false` +> (configurable via `VLM_ENABLE_THINKING`). + **Response:** ```json diff --git a/docs/user-guide/take-away/release-notes.md b/docs/user-guide/take-away/release-notes.md index e019f42b..5c94c8ad 100644 --- a/docs/user-guide/take-away/release-notes.md +++ b/docs/user-guide/take-away/release-notes.md @@ -4,6 +4,36 @@ Version history and changelog for Take-Away Order Accuracy. --- +## Version 2026.1.0 (Unreleased) + +### What's New + +- **VLM switched to MiniCPM-V-4.5 (INT4)** — Take-Away now uses + `openbmb/MiniCPM-V-4_5-int4` instead of `Qwen/Qwen2.5-VL-7B-Instruct`, aligning it + with Dine-In so both applications share a single OVMS model instance. + `VLM_PRECISION` defaults to `int4`. +- **`setup_models.sh` per-app defaults** — `--app take-away` and `--app dine-in` both + default to MiniCPM-V-4.5 INT4. `VLM_PRECISION` is now derived from the precision + suffix of `OVMS_MODEL_NAME`, and a mismatch between the two emits a warning. +- **Reasoning disabled by default** — MiniCPM-V-4.5 is a hybrid reasoning model; the + service now sends `chat_template_kwargs.enable_thinking=false` (configurable via + `VLM_ENABLE_THINKING`) so the token budget is spent on the answer instead of a + `` block. Any stray `` output is stripped before parsing. +- **Aspect-preserving image preprocessing** — frames are fitted into a 448x448 square + canvas (LANCZOS downscale, white padding, light contrast/sharpening) matching the + Dine-In pipeline, configurable via `VLM_IMAGE_MAX_SIZE`. This replaces the previous + fixed 512x512 resize, which distorted the aspect ratio. + +### Fixed + +- Frames are no longer stretched to 512x512, which distorted product shapes. +- Detections are no longer truncated by reasoning output, which previously caused + missing items and roughly 4x higher VLM latency. +- `setup_models.sh` no longer migrates legacy Qwen weights into a directory named + after a different model. + +--- + ## Version 2026.0.0 (March 2026) **General Availability Release** diff --git a/docs/user-guide/take-away/ta-benchmarking.md b/docs/user-guide/take-away/ta-benchmarking.md index ee85d639..706ee541 100644 --- a/docs/user-guide/take-away/ta-benchmarking.md +++ b/docs/user-guide/take-away/ta-benchmarking.md @@ -62,6 +62,13 @@ make up > make download-sample-video > ``` +> **Order manifests:** The benchmark validates the orders detected in the video +> against `config/orders.json`. Order IDs must match the order numbers shown in +> the video, and each order's expected items must match what is actually visible. +> A stale manifest reports correct detections as mismatches; an unparseable one +> (for example a trailing comma) makes every order fail and the benchmark reports +> zero transactions. + ## Benchmark Commands ### Fixed Workers Benchmark diff --git a/docs/user-guide/take-away/troubleshooting.md b/docs/user-guide/take-away/troubleshooting.md index 1e4df966..8b876f5c 100644 --- a/docs/user-guide/take-away/troubleshooting.md +++ b/docs/user-guide/take-away/troubleshooting.md @@ -63,3 +63,48 @@ OPENVINO_DEVICE=CPU cd ../ovms-service && ./setup_models.sh --app take-away cd ../take-away && make down && make up ``` + +## OVMS Returns HTTP 404 for Every Request + +`OVMS_MODEL_NAME` must exactly match the name registered in +`ovms-service/models/config.json`, which `setup_models.sh` generates including the +precision suffix (for example `openbmb/MiniCPM-V-4_5-int4`). Any mismatch makes +every inference request return 404. + +```bash +# Compare the registered name with the configured one +grep '"name"' ../ovms-service/models/config.json +grep OVMS_MODEL_NAME .env +``` + +## Items Missing from Detections / High Latency + +MiniCPM-V-4.5 is a hybrid reasoning model. If `VLM_ENABLE_THINKING` is enabled, the +model emits a `` reasoning block that consumes the entire +`max_completion_tokens` budget, so the answer is truncated — orders come back with +items missing and latency is several times higher. + +```bash +# Confirm thinking is disabled +grep VLM_ENABLE_THINKING .env # expected: false +docker exec oa_service env | grep VLM_ENABLE_THINKING +``` + +Set `VLM_ENABLE_THINKING=false` in `.env` and restart (`make down && make up`). + +## Benchmark Reports Zero Transactions + +An all-zero result (`total_transactions: 0`, `No vlm_metrics_logger files found`) +usually means every order failed before inference. The most common cause is an +invalid or edited `config/orders.json` — a trailing comma or a missing entry makes +the file unparseable, and the service logs `Order not found in orders.json`. + +```bash +# Validate the file before benchmarking +python3 -c "import json; print(json.load(open('config/orders.json')).keys())" +docker logs oa_service 2>&1 | grep -i "orders.json" +``` + +Order IDs in `config/orders.json` must also match the order numbers that appear in +the benchmark video, and each order's expected items must reflect what is actually +visible — otherwise correct detections are still reported as mismatches. diff --git a/ovms-service/README.md b/ovms-service/README.md index c9345f3e..4859b36c 100755 --- a/ovms-service/README.md +++ b/ovms-service/README.md @@ -37,9 +37,10 @@ bash ovms-service/setup_models.sh --app dine-in This will: - Download `export_model.py` and install its dependencies automatically -- Read `OVMS_MODEL_NAME` from the selected app `.env` (`take-away/.env` or `dine-in/.env`) +- Read `OVMS_MODEL_NAME` from the selected app `.env` (`take-away/.env` or `dine-in/.env`), + falling back to the per-app default (`openbmb/MiniCPM-V-4_5-int4` for both apps) - Download that model from HuggingFace -- Convert to OpenVINO™ IR format with int8 quantization +- Convert to OpenVINO™ IR format at the `VLM_PRECISION` weight format - Save to `ovms-service/models//` - Generate `graph.pbtxt` for OVMS configuration @@ -54,12 +55,27 @@ Supported values for `OVMS_MODEL_NAME`: > This list mirrors `SUPPORTED_MODEL_SOURCES` in `ovms-service/setup_models.sh`, > which is the source of truth — check that script if this list appears out of date. +### Precision (`VLM_PRECISION`) + +`VLM_PRECISION` is the weight format passed to the export (`--weight-format`). +When it is not set, the default is derived from the precision suffix of +`OVMS_MODEL_NAME` (`-int4` → `int4`, `-int8` → `int8`, otherwise `int8`). + +The suffix is only a directory label, so it does not by itself change the exported +weights. Setting `VLM_PRECISION=int8` while naming the model `...-int4` writes INT8 +weights into an `-int4` directory; the script warns when it detects this mismatch. + +> `openbmb/MiniCPM-V-4_5-int4` is exported from the full-precision +> `openbmb/MiniCPM-V-4_5` source with NNCF INT4 weight compression. The +> HuggingFace repository of the same name is a bitsandbytes checkpoint that +> optimum-intel cannot import directly. + Optional authentication for gated Hugging Face models: - Add `HF_TOKEN=` (or `HUGGINGFACE_HUB_TOKEN=`) to the selected app `.env` - Or run `huggingface-cli login` -Public models (for example Qwen) can be downloaded without authentication. +Public models (for example MiniCPM-V-4.5) can be downloaded without authentication. > **ℹ Low-RAM systems:** Set `export CACHE_SIZE=2` before running `setup_models.sh` if you are on a 16 GB system. For first-time export, a 48–64 GB host is recommended to avoid OOM. See [Tuning the KV Cache Size](#tuning-the-kv-cache-size). diff --git a/ovms-service/setup_models.sh b/ovms-service/setup_models.sh index 1bc1c834..4cb84359 100755 --- a/ovms-service/setup_models.sh +++ b/ovms-service/setup_models.sh @@ -66,13 +66,20 @@ read_env_value() { grep -E "^${key}=" "${file}" 2>/dev/null | head -1 | cut -d'=' -f2- | tr -d '"\r' } +# Per-app default model. Both dine-in and take-away run MiniCPM-V-4_5 at INT4; +# these defaults are only used when OVMS_MODEL_NAME is not set in the shell +# environment or in the app's .env file. +declare -A APP_DEFAULT_MODEL +APP_DEFAULT_MODEL["take-away"]="openbmb/MiniCPM-V-4_5-int4" +APP_DEFAULT_MODEL["dine-in"]="openbmb/MiniCPM-V-4_5-int4" + ENV_FILE="${PROJECT_ROOT}/${APP}/.env" echo "App: ${APP} → reading config from ${ENV_FILE}" if [ -f "${ENV_FILE}" ]; then _OVMS_MODEL_NAME_FILE=$(read_env_value "OVMS_MODEL_NAME" "${ENV_FILE}") fi # Read OVMS_MODEL_NAME: shell env var takes precedence over app .env -OVMS_MODEL_NAME_ENV="${OVMS_MODEL_NAME:-${_OVMS_MODEL_NAME_FILE:-Qwen/Qwen2.5-VL-7B-Instruct}}" +OVMS_MODEL_NAME_ENV="${OVMS_MODEL_NAME:-${_OVMS_MODEL_NAME_FILE:-${APP_DEFAULT_MODEL[${APP}]}}}" if [ -z "${SUPPORTED_MODEL_SOURCES[${OVMS_MODEL_NAME_ENV}]+x}" ]; then echo "ERROR: Unsupported OVMS_MODEL_NAME='${OVMS_MODEL_NAME_ENV}'" @@ -107,13 +114,36 @@ else echo "Hugging Face authentication: no token found; using anonymous access" fi -# Read TARGET_DEVICE: shell env var takes precedence over take-away/.env, default GPU +# Read TARGET_DEVICE: shell env var takes precedence over the app .env, default GPU _TARGET_DEVICE_FILE=$(read_env_value "TARGET_DEVICE" "${ENV_FILE}") TARGET_DEVICE_ENV="${TARGET_DEVICE:-${_TARGET_DEVICE_FILE:-GPU}}" -# Read VLM_PRECISION: shell env var takes precedence over take-away/.env, default int8 +# Read VLM_PRECISION: shell env var takes precedence over the app .env. +# The default is derived from the precision suffix of OVMS_MODEL_NAME +# (e.g. "-int4" → int4), falling back to int8 when there is no suffix. _VLM_PRECISION_FILE=$(read_env_value "VLM_PRECISION" "${ENV_FILE}") -VLM_PRECISION_ENV="${VLM_PRECISION:-${_VLM_PRECISION_FILE:-int8}}" +case "${OVMS_MODEL_NAME_ENV}" in + *-int4) _VLM_PRECISION_DEFAULT="int4" ;; + *-int8) _VLM_PRECISION_DEFAULT="int8" ;; + *) _VLM_PRECISION_DEFAULT="int8" ;; +esac +VLM_PRECISION_ENV="${VLM_PRECISION:-${_VLM_PRECISION_FILE:-${_VLM_PRECISION_DEFAULT}}}" + +# The precision suffix in the model name is only a directory label, so an +# int8 export written into an "-int4" directory silently produces slower +# weights than the name advertises. Warn loudly on a mismatch. +case "${OVMS_MODEL_NAME_ENV}" in + *-int4|*-int8) + if [ "${VLM_PRECISION_ENV}" != "${_VLM_PRECISION_DEFAULT}" ]; then + echo "" + echo " ⚠ WARNING: OVMS_MODEL_NAME='${OVMS_MODEL_NAME_ENV}' implies" + echo " ${_VLM_PRECISION_DEFAULT} weights, but VLM_PRECISION='${VLM_PRECISION_ENV}'." + echo " The exported weights will NOT match the model directory name." + echo " Set VLM_PRECISION=${_VLM_PRECISION_DEFAULT} in ${ENV_FILE} to align them." + echo "" + fi + ;; +esac # Read CACHE_SIZE: KV cache size in GB for OVMS. Default is 4 GB which is # adequate for a single-station app (max_num_seqs=4). On GPU the KV cache @@ -244,6 +274,13 @@ migrate_legacy_model() { return 0 # Already at the new location, nothing to do fi + # The legacy layout only ever contained Qwen2.5-VL-7B-Instruct, so never + # migrate it into a directory named after a different model (e.g. MiniCPM) + # — that would serve Qwen weights under the wrong model name. + if [ "${target_path}" != "${MODELS_DIR}/Qwen/Qwen2.5-VL-7B-Instruct" ]; then + return 1 + fi + local legacy_paths=( "${MODELS_DIR}/Qwen/Qwen2.5-VL-7B-Instruct-ov-int8" "${MODELS_DIR}/Qwen2.5-VL-7B-Instruct-ov-int8" @@ -507,8 +544,8 @@ done # export_model.py generates a model_config_list config which does NOT support # the /v3/chat/completions endpoint. OVMS LLM/VLM serving requires a # mediapipe_config_list entry (pointing at the graph.pbtxt). The model name -# is read from OVMS_MODEL_NAME in take-away/.env so it always matches what -# the OA service requests (e.g. "Qwen/Qwen2.5-VL-7B-Instruct"). +# is read from OVMS_MODEL_NAME in the selected app's .env so it always matches +# what the OA service requests (e.g. "openbmb/MiniCPM-V-4_5-int4"). ############################################### generate_ovms_config() { echo "" diff --git a/take-away/.env.example b/take-away/.env.example index 863ada9d..6c19a52a 100644 --- a/take-away/.env.example +++ b/take-away/.env.example @@ -28,18 +28,35 @@ VLM_BACKEND=ovms # OVMS connection settings OVMS_ENDPOINT=http://ovms-vlm:8000 -OVMS_MODEL_NAME=Qwen/Qwen2.5-VL-7B-Instruct -VLM_PRECISION=int8 +# Must match the model name registered in ovms-service/models/config.json, +# which setup_models.sh generates with the precision suffix (e.g. -int4). +# A mismatch makes every OVMS request return HTTP 404. +OVMS_MODEL_NAME=openbmb/MiniCPM-V-4_5-int4 +# Weight format passed to the model export (--weight-format). This must agree +# with the precision suffix in OVMS_MODEL_NAME above: the suffix is only a +# directory label, so setting int8 here while naming the model -int4 produces +# INT8 weights in an "-int4" directory (~1.6x slower decode). +VLM_PRECISION=int4 TARGET_DEVICE=GPU OVMS_TIMEOUT=120 +# VLM_IMAGE_MAX_SIZE: frames are fitted into a square canvas of this size +# (aspect ratio preserved, white padding) before being sent to the VLM. +# 448 matches the dine-in pipeline. +VLM_IMAGE_MAX_SIZE=448 + +# VLM_ENABLE_THINKING: MiniCPM-V-4.5 is a hybrid reasoning model. Leave this +# false — when enabled, the model spends the whole token budget on a +# block and the item list is truncated. +VLM_ENABLE_THINKING=false + # Optional Hugging Face authentication for gated models. -# Leave empty for public models (for example Qwen/Qwen2.5-VL-7B-Instruct). +# Leave empty for public models (for example openbmb/MiniCPM-V-4_5). # setup_models.sh also accepts HUGGINGFACE_HUB_TOKEN as an alternative name. HF_TOKEN= # OpenVINO local settings (when VLM_BACKEND=openvino) -VLM_MODEL_PATH=/model/Qwen2.5-VL-7B-Instruct +VLM_MODEL_PATH=/model/MiniCPM-V-4_5-int4 # ----------------------------------------------------------------------------- # Semantic Search Agent Configuration @@ -63,7 +80,7 @@ SEMANTIC_LOG_LEVEL=INFO CACHE_TTL=3600 # OpenVINO model path for semantic service (when SEMANTIC_VLM_BACKEND=openvino) -OPENVINO_MODEL_PATH=/models/Qwen2.5-VL-7B-Instruct +OPENVINO_MODEL_PATH=/models/MiniCPM-V-4_5-int4 # Metrics logging CONTAINER_RESULTS_PATH=/results @@ -169,7 +186,7 @@ BENCHMARK_MIN_TRANSACTIONS=1 # Safety limit to prevent runaway tests MAX_ITERATIONS=50 -BENCHMARK_DURATION=150 +BENCHMARK_DURATION=30 # RESULTS_DIR: Directory for benchmark results output RESULTS_DIR=./results diff --git a/take-away/Makefile b/take-away/Makefile index 02e2078d..05104172 100644 --- a/take-away/Makefile +++ b/take-away/Makefile @@ -31,7 +31,7 @@ SCALING_MODE ?= fixed # VLM Backend VLM_BACKEND ?= ovms OVMS_ENDPOINT ?= http://ovms-vlm:8000 -OVMS_MODEL_NAME ?= Qwen/Qwen2.5-VL-7B-Instruct +OVMS_MODEL_NAME ?= openbmb/MiniCPM-V-4_5-int4 OPENVINO_DEVICE ?= $(TARGET_DEVICE) # Automatically select CPU-only OVMS image when TARGET_DEVICE=CPU OVMS_IMAGE ?= $(if $(filter CPU,$(TARGET_DEVICE)),openvino/model_server:latest,openvino/model_server:latest-gpu) diff --git a/take-away/README.md b/take-away/README.md index 0a16d511..7aa3fa7b 100755 --- a/take-away/README.md +++ b/take-away/README.md @@ -17,7 +17,7 @@ Take-Away Order Accuracy is an AI-powered vision system that validates drive-thr - **Real-Time Video Processing**: GStreamer-based pipeline with RTSP support - **Multi-Station Parallel Processing**: Concurrent order validation across multiple stations -- **VLM-Based Item Detection**: Qwen2.5-VL-7B for visual product identification +- **VLM-Based Item Detection**: MiniCPM-V-4.5 (INT4) for visual product identification - **Intelligent Frame Selection**: YOLO-powered frame selection for optimal VLM input - **Semantic Matching**: Hybrid exact/semantic matching for robust item comparison - **Production-Ready Architecture**: Circuit breaker, exponential backoff, health monitoring @@ -71,7 +71,7 @@ cd ../take-away This downloads and exports: -- Qwen2.5-VL-7B-Instruct (OpenVINO™ format) +- MiniCPM-V-4_5 INT4 (OpenVINO™ format) - YOLOv11 model (INT8 OpenVINO™) - EasyOCR detection and recognition models diff --git a/take-away/config/application.yaml b/take-away/config/application.yaml index 80fb4167..091b006f 100755 --- a/take-away/config/application.yaml +++ b/take-away/config/application.yaml @@ -37,12 +37,12 @@ vlm: backend: ovms # Using OVMS backend for VLM inference # Embedded backend configuration (used when backend: embedded) - model_path: /model/Qwen2.5-VL-7B-Instruct + model_path: /model/MiniCPM-V-4_5-int4 device: GPU # OVMS backend configuration (used when backend: ovms) ovms_endpoint: http://ovms-vlm:8000 - ovms_model: Qwen/Qwen2.5-VL-7B-Instruct + ovms_model: openbmb/MiniCPM-V-4_5-int4 # Common configuration endpoint: http://order-accuracy:8000/run_vlm @@ -50,8 +50,8 @@ vlm: timeout_sec: 360 # Increased: 7B VLM on iGPU takes ~300s per inference max_new_tokens: 100 # Item list is ~30 tokens max; 512 caused KV-cache overflow → OVMS crash temperature: 0.2 - model: Qwen/Qwen2.5-VL-7B-Instruct - precision: int8 + model: openbmb/MiniCPM-V-4_5-int4 + precision: int4 # Semantic Comparison Service Configuration semantic_service: diff --git a/take-away/config/orders.json b/take-away/config/orders.json index c99686e8..e890863b 100755 --- a/take-away/config/orders.json +++ b/take-away/config/orders.json @@ -21,8 +21,7 @@ ], "651": [ { "name": "banana", "quantity": 2 }, - { "name": "coke 2 liter bottle", "quantity": 1 }, - { "name": "water bottle", "quantity": 1 } + { "name": "coke 2 liter bottle", "quantity": 1 } ], "892": [ { "name": "apple", "quantity": 1 }, diff --git a/take-away/docker-compose.yaml b/take-away/docker-compose.yaml index 592baef5..5524bfed 100755 --- a/take-away/docker-compose.yaml +++ b/take-away/docker-compose.yaml @@ -58,7 +58,7 @@ services: # SINGLE Station (per-container): docker compose up -d --scale order-accuracy=4 # MULTI Station (multi-process in one container): Set SERVICE_MODE=parallel order-accuracy: - image: intel/order-accuracy-take-away:${TAG} + image: ${TAKEAWAY_IMAGE:-intel/order-accuracy-take-away:latest} build: context: . args: @@ -98,15 +98,22 @@ services: # VLM Backend (embedded or ovms) VLM_BACKEND: ${VLM_BACKEND:-ovms} - VLM_MODEL_PATH: ${VLM_MODEL_PATH:-/model/Qwen2.5-VL-7B-Instruct} + VLM_MODEL_PATH: ${VLM_MODEL_PATH:-/model/MiniCPM-V-4_5-int4} OPENVINO_DEVICE: ${TARGET_DEVICE:-GPU} # OVMS settings (when VLM_BACKEND=ovms) OVMS_ENDPOINT: ${OVMS_ENDPOINT:-http://ovms-vlm:8000} - OVMS_MODEL_NAME: ${OVMS_MODEL_NAME:-Qwen/Qwen2.5-VL-7B-Instruct} - VLM_PRECISION: ${VLM_PRECISION:-int8} + OVMS_MODEL_NAME: ${OVMS_MODEL_NAME:-openbmb/MiniCPM-V-4_5-int4} + VLM_PRECISION: ${VLM_PRECISION:-int4} VLM_DEVICE: ${TARGET_DEVICE:-GPU} OVMS_TIMEOUT: ${OVMS_TIMEOUT:-120} + # Square canvas size submitted to the VLM (matches dine-in) + VLM_IMAGE_MAX_SIZE: ${VLM_IMAGE_MAX_SIZE:-448} + # Keep MiniCPM's reasoning disabled so the token budget is + # spent on the answer rather than on reasoning. + VLM_ENABLE_THINKING: ${VLM_ENABLE_THINKING:-false} + # Debug: save the exact frames submitted to the VLM under /results/vlm-in + SAVE_VLM_INPUT: ${SAVE_VLM_INPUT:-false} # Semantic service SEMANTIC_SERVICE_ENDPOINT: ${SEMANTIC_SERVICE_ENDPOINT:-http://semantic-service:8080} @@ -188,7 +195,7 @@ services: - order-accuracy-net frame-selector: - image: intel/order-accuracy-frame-selector:${TAG} + image: ${FRAME_SELECTOR_IMAGE:-intel/order-accuracy-frame-selector:latest} build: context: ./frame-selector-service args: @@ -236,7 +243,7 @@ services: - order-accuracy-net gradio-ui: - image: intel/order-accuracy-take-away-ui:${TAG} + image: ${TAKEAWAY_UI_IMAGE:-intel/order-accuracy-take-away-ui:latest} build: context: ./gradio-ui args: @@ -285,12 +292,12 @@ services: - VLM_BACKEND=${SEMANTIC_VLM_BACKEND:-ovms} # OVMS settings (when VLM_BACKEND=ovms) - OVMS_ENDPOINT=http://ovms-vlm:8000 - - OVMS_MODEL_NAME=${OVMS_MODEL_NAME:-Qwen/Qwen2.5-VL-7B-Instruct} - - VLM_PRECISION=${VLM_PRECISION:-int8} + - OVMS_MODEL_NAME=${OVMS_MODEL_NAME:-openbmb/MiniCPM-V-4_5-int4} + - VLM_PRECISION=${VLM_PRECISION:-int4} - VLM_DEVICE=${TARGET_DEVICE:-GPU} - OVMS_TIMEOUT=${OVMS_TIMEOUT:-60} # OpenVINO settings (when VLM_BACKEND=openvino) - - OPENVINO_MODEL_PATH=${OPENVINO_MODEL_PATH:-/models/Qwen2.5-VL-7B-Instruct} + - OPENVINO_MODEL_PATH=${OPENVINO_MODEL_PATH:-/models/MiniCPM-V-4_5-int4} - OPENVINO_DEVICE=${TARGET_DEVICE:-GPU} # Cache settings - CACHE_ENABLED=true @@ -315,7 +322,7 @@ services: # RTSP Streamer - On-demand video streaming via MediaMTX # Streams start only when a GStreamer client connects — no sync needed. rtsp-streamer: - image: intel/order-accuracy-take-away-rtsp:${TAG} + image: ${TAKEAWAY_RTSP_IMAGE:-intel/order-accuracy-take-away-rtsp:latest} build: context: ./rtsp-streamer args: diff --git a/take-away/gradio-ui/gradio_app.py b/take-away/gradio-ui/gradio_app.py index 234f0282..d78a8719 100755 --- a/take-away/gradio-ui/gradio_app.py +++ b/take-away/gradio-ui/gradio_app.py @@ -1551,7 +1551,7 @@ def _load_mode_status():
intel
AI Solutions • Take-Away Order Accuracy System
-
Powered by Qwen2.5-VL Vision-Language Model on OpenVINO™
+
Powered by MiniCPM-V-4.5 Vision-Language Model on OpenVINO™
''') diff --git a/take-away/src/core/ovms_client.py b/take-away/src/core/ovms_client.py index 66b11ac0..60bf6c59 100755 --- a/take-away/src/core/ovms_client.py +++ b/take-away/src/core/ovms_client.py @@ -2,6 +2,7 @@ import requests import base64 +import re import time import logging import os @@ -9,7 +10,7 @@ from typing import List, Optional from pathlib import Path import numpy as np -from PIL import Image +from PIL import Image, ImageEnhance, ImageFilter from vlm_metrics_logger import ( log_start_time, log_end_time, @@ -23,6 +24,34 @@ VLM_INPUT_DIR = os.environ.get('VLM_INPUT_DIR', '/results/vlm-in') SAVE_VLM_INPUT = os.environ.get('SAVE_VLM_INPUT', 'false').lower() in ('true', '1', 'yes') +# Square canvas size fed to the VLM. Matches the dine-in pipeline so both +# applications submit identically sized images to the same MiniCPM model. +VLM_IMAGE_MAX_SIZE = int(os.environ.get('VLM_IMAGE_MAX_SIZE', '448')) +VLM_IMAGE_MIN_SIZE = 224 +VLM_JPEG_QUALITY = int(os.environ.get('VLM_JPEG_QUALITY', '82')) + +# MiniCPM-V-4.5 is a hybrid reasoning model: unless thinking is explicitly +# disabled its chat template opens a block and the whole +# max_completion_tokens budget is spent on reasoning, leaving the answer +# truncated. Setting enable_thinking=false makes the template pre-fill an +# empty pair so generation starts with the answer. +VLM_ENABLE_THINKING = os.environ.get('VLM_ENABLE_THINKING', 'false').lower() in ('true', '1', 'yes') + +_THINK_BLOCK_RE = re.compile(r".*?", re.DOTALL | re.IGNORECASE) + + +def strip_think_block(text: str) -> str: + """Remove ... reasoning emitted by hybrid reasoning VLMs.""" + if not text: + return text + cleaned = _THINK_BLOCK_RE.sub("", text) + # An unterminated block means generation was cut off mid-reasoning; there + # is no usable answer after it, so drop everything from onwards. + idx = cleaned.lower().find("") + if idx != -1: + cleaned = cleaned[:idx] + return cleaned.strip() + class OVMSVLMClient: """ @@ -33,7 +62,7 @@ class OVMSVLMClient: def __init__( self, endpoint: str, - model_name: str = "Qwen/Qwen2-VL-2B-Instruct", + model_name: str = "openbmb/MiniCPM-V-4_5-int4", timeout: int = 120, max_new_tokens: int = 512, temperature: float = 0.2, @@ -68,12 +97,42 @@ def _encode_image(self, image: np.ndarray) -> str: """ # Images arrive as RGB (loaded via PIL in vlm_service.py) — no channel flip needed. pil_img = Image.fromarray(image.astype('uint8')) + pil_img = self._preprocess_image(pil_img) buffer = BytesIO() - pil_img.save(buffer, format="JPEG", quality=82, optimize=True) + pil_img.save(buffer, format="JPEG", quality=VLM_JPEG_QUALITY, optimize=True) img_b64 = base64.b64encode(buffer.getvalue()).decode('utf-8') return f"data:image/jpeg;base64,{img_b64}" + @staticmethod + def _preprocess_image(img: Image.Image) -> Image.Image: + """ + Fit the frame into a VLM_IMAGE_MAX_SIZE square canvas, mirroring the + dine-in ImagePreprocessor so both applications feed the VLM images of + identical dimensions. + + Aspect ratio is preserved (LANCZOS downscale) and the remaining canvas + is padded white; light contrast enhancement and sharpening improve + product-label legibility at this reduced resolution. + """ + if img.mode != 'RGB': + img = img.convert('RGB') + + target = VLM_IMAGE_MAX_SIZE + width, height = img.size + scale = min(target / width, target / height) + + new_width = min(max(int(round(width * scale)), VLM_IMAGE_MIN_SIZE), target) + new_height = min(max(int(round(height * scale)), VLM_IMAGE_MIN_SIZE), target) + + resized = img.resize((new_width, new_height), Image.Resampling.LANCZOS) + resized = ImageEnhance.Contrast(resized).enhance(1.15) + resized = resized.filter(ImageFilter.UnsharpMask(radius=1.0, percent=60, threshold=3)) + + canvas = Image.new('RGB', (target, target), (255, 255, 255)) + canvas.paste(resized, ((target - new_width) // 2, (target - new_height) // 2)) + return canvas + def _save_input_frames(self, images: List, unique_id: str) -> str: """ Save input frames to disk for debugging/analysis. @@ -169,6 +228,9 @@ def generate( ], "max_completion_tokens": self.max_new_tokens, "temperature": self.temperature, + # Hybrid reasoning models (MiniCPM-V-4.5) must be told not to emit a + # block, otherwise the token budget is consumed by reasoning. + "chat_template_kwargs": {"enable_thinking": VLM_ENABLE_THINKING}, } # Save input frames for debugging (before sending request) @@ -204,6 +266,7 @@ def generate( # Extract text from response text = result.get("choices", [{}])[0].get("message", {}).get("content", "") + text = strip_think_block(text) logger.info(f"[OVMS-CLIENT] Response received in {total_latency:.2f}s") logger.debug(f"[OVMS-CLIENT] Generated text: {text[:200]}...") diff --git a/take-away/src/core/vlm_backend_factory.py b/take-away/src/core/vlm_backend_factory.py index ea267cee..5a70707a 100755 --- a/take-away/src/core/vlm_backend_factory.py +++ b/take-away/src/core/vlm_backend_factory.py @@ -44,7 +44,7 @@ def _create_ovms_backend(config: dict, max_new_tokens: int, temperature: float): from .ovms_client import OVMSVLMClient, MockGenerationConfig endpoint = config.get("ovms_endpoint", "http://ovms-vlm:8000") - model_name = config.get("ovms_model", "Qwen/Qwen2-VL-2B-Instruct") + model_name = config.get("ovms_model", "openbmb/MiniCPM-V-4_5-int4") timeout = config.get("timeout_sec", 120) logger.info(f"[BACKEND-FACTORY] Creating OVMS backend: {endpoint}") diff --git a/take-away/src/core/vlm_service.py b/take-away/src/core/vlm_service.py index b24134d2..a64e20c4 100755 --- a/take-away/src/core/vlm_service.py +++ b/take-away/src/core/vlm_service.py @@ -272,7 +272,7 @@ def process(self, images: list[np.ndarray], unique_id: str = None, expected_item backend_config = { "ovms_endpoint": os.getenv("OVMS_ENDPOINT", VLM_CFG.get("ovms_endpoint", "http://" \ ":8000")), - "ovms_model": os.getenv("OVMS_MODEL_NAME", VLM_CFG.get("ovms_model", "Qwen/Qwen2.5-VL-7B-Instruct")), + "ovms_model": os.getenv("OVMS_MODEL_NAME", VLM_CFG.get("ovms_model", "openbmb/MiniCPM-V-4_5-int4")), "timeout_sec": VLM_CFG.get("timeout_sec", 300), } @@ -393,7 +393,10 @@ async def _run_vlm_internal(order_id: str, station_id: str): for f in frames: logger.debug(f"[INTERNAL] Loading frame: {f}") data = client.get_object(SELECTED_BUCKET, f) - img = Image.open(data).convert("RGB").resize((512, 512)) + # No resize here: OVMSVLMClient fits each frame into a 448x448 + # square with the aspect ratio preserved (same as dine-in). + # Resizing to a fixed 512x512 first distorted the aspect ratio. + img = Image.open(data).convert("RGB") images.append(np.array(img)) logger.info(f"[INTERNAL] Loaded {len(images)} images, starting VLM inference for order_id={order_id}") diff --git a/take-away/src/main.py b/take-away/src/main.py index f4dfce9c..db01d432 100755 --- a/take-away/src/main.py +++ b/take-away/src/main.py @@ -151,7 +151,7 @@ def run_parallel_mode(): scheduler = VLMScheduler( queue_manager=queue_mgr, ovms_url=os.getenv('OVMS_ENDPOINT', 'http://ovms-vlm:8000'), - model_name=os.getenv('OVMS_MODEL_NAME', 'Qwen/Qwen2.5-VL-7B-Instruct'), + model_name=os.getenv('OVMS_MODEL_NAME', 'openbmb/MiniCPM-V-4_5-int4'), batch_window_ms=100, max_batch_size=16, max_workers=vlm_workers diff --git a/take-away/src/parallel/config.py b/take-away/src/parallel/config.py index 975230c1..2f2a65b8 100755 --- a/take-away/src/parallel/config.py +++ b/take-away/src/parallel/config.py @@ -47,7 +47,7 @@ class ScalingConfig: class VLMConfig: """VLM inference configuration""" ovms_url: str = "http://localhost:8000" - model_name: str = os.getenv('OVMS_MODEL_NAME', 'Qwen/Qwen2.5-VL-7B-Instruct') # Read from env + model_name: str = os.getenv('OVMS_MODEL_NAME', 'openbmb/MiniCPM-V-4_5-int4') # Read from env batch_window_ms: int = 100 max_batch_size: int = 16 max_workers: int = 4