From d84c316f5c3321c418c53288dd6594ee26d4ba12 Mon Sep 17 00:00:00 2001 From: Wiktor Iwaszko Date: Tue, 14 Jul 2026 14:30:53 +0200 Subject: [PATCH] [DOCS] Align unified OA Benchmarking/GS to DI/TA Benchmarking/GS --- docs/user-guide/dine-in/di-benchmarking.md | 2 +- docs/user-guide/dine-in/get-started.md | 2 +- docs/user-guide/oa-benchmarking.md | 107 ++++++++++++++++----- docs/user-guide/oa-get-started.md | 52 +++++++--- docs/user-guide/take-away/get-started.md | 2 +- 5 files changed, 122 insertions(+), 43 deletions(-) diff --git a/docs/user-guide/dine-in/di-benchmarking.md b/docs/user-guide/dine-in/di-benchmarking.md index 7dbf6c0e..95ce38a5 100644 --- a/docs/user-guide/dine-in/di-benchmarking.md +++ b/docs/user-guide/dine-in/di-benchmarking.md @@ -71,7 +71,7 @@ make benchmark-stream-density make benchmark-stream-density BENCHMARK_TARGET_LATENCY_MS=20000 BENCHMARK_INIT_DURATION=30 ``` -> **Note:** `make benchmark-density` runs a Python script locally that sends concurrent HTTP requests to the running `dine-in` API. No separate worker containers are needed for this mode. +> **Note:** `make benchmark-stream-density` runs a Python script locally that sends concurrent HTTP requests to the running `dine-in` API. No separate worker containers are needed for this mode. ## Metrics Processing diff --git a/docs/user-guide/dine-in/get-started.md b/docs/user-guide/dine-in/get-started.md index a2826c06..c40b6dc3 100644 --- a/docs/user-guide/dine-in/get-started.md +++ b/docs/user-guide/dine-in/get-started.md @@ -67,7 +67,7 @@ This downloads Qwen2.5-VL-7B-Instruct (~7 GB) and converts it to OpenVINO™ INT Before running the application, you must prepare your test data: 1. **Add Images**: Place your food tray images in the `images/` folder - - Supported formats: `.jpg`, `.jpeg`, `.png` + - Supported formats: `.jpg`, `.jpeg` or `.png` - Images should clearly show the food items on the tray 2. **Update Orders**: Edit `configs/orders.json` with your test orders diff --git a/docs/user-guide/oa-benchmarking.md b/docs/user-guide/oa-benchmarking.md index c6bd6296..907cf0db 100644 --- a/docs/user-guide/oa-benchmarking.md +++ b/docs/user-guide/oa-benchmarking.md @@ -6,10 +6,46 @@ Test your Order Accuracy pipeline performance on various hardware configurations **Goal**: Run a basic performance test to verify your system works correctly +> **Note — Inference Device:** The default device is `GPU`. To switch to `CPU`, you must do **both** steps below, otherwise the model will be exported for the wrong device: +> +> 1. Set **both** variables in your `.env` file: +> +> ```bash +> TARGET_DEVICE=GPU # used by setup_models.sh and docker-compose +> OPENVINO_DEVICE=GPU # used by the Makefile benchmark targets +> ``` +> +> 2. Re-export the model for the new device: +> +> **Dine-In** +> +> +> ```bash +> cd ../ovms-service && ./setup_models.sh --app dine-in +> ``` +> +> **Take-Away** +> +> +> ```bash +> cd ../ovms-service && ./setup_models.sh --app take-away +> ``` +> +> +> +> `TARGET_DEVICE` is what `setup_models.sh` reads to export the model in the correct format. `OPENVINO_DEVICE` is what the Makefile passes to the benchmark script. Both must match. + ### 1. Initialize Performance Tools ```bash +# 1. Initialize git submodules (first time only) make update-submodules + +# 2. Start services +make up ``` ### 2. Run Quick Benchmark @@ -27,8 +63,15 @@ make benchmark :::{tab-item}hide_directive--> **Take-Away** +> **Important:** Before running benchmarks, ensure a test video file is present at `storage/videos/test.mp4`. You can download a sample video using: +> +> ```bash +> make download-sample-video +> ``` + ```bash cd take-away +# Default run make benchmark ``` @@ -53,7 +96,7 @@ make benchmark ```bash -make benchmark +make benchmark-single IMAGE_ID=MCD-1001 ``` Tests single image validation latency: @@ -68,7 +111,10 @@ Tests single image validation latency: ```bash -make benchmark-density +make benchmark-stream-density + +# With overrides +make benchmark-stream-density BENCHMARK_TARGET_LATENCY_MS=20000 BENCHMARK_INIT_DURATION=30 ``` Finds maximum concurrent requests the system can handle under latency constraints: @@ -103,7 +149,10 @@ Tests end-to-end latency for single order validation: :::{tab-item}hide_directive--> **Fixed Workers Benchmark** ```bash -make benchmark-oa BENCHMARK_WORKERS=4 BENCHMARK_DURATION=300 +make benchmark \ + BENCHMARK_WORKERS=4 \ + BENCHMARK_DURATION=300 \ + BENCHMARK_INIT_DURATION=30 ``` Tests system with fixed number of concurrent workers: @@ -118,7 +167,16 @@ Tests system with fixed number of concurrent workers: ```bash +# Default run make benchmark-stream-density + +# Custom run +make benchmark-stream-density \ + BENCHMARK_TARGET_LATENCY_MS=25000 \ + BENCHMARK_LATENCY_METRIC=avg \ + BENCHMARK_INIT_DURATION=30 \ + BENCHMARK_MIN_TRANSACTIONS=3 \ + BENCHMARK_WORKER_INCREMENT=1 ``` Finds maximum sustainable worker count under latency constraints: @@ -140,32 +198,31 @@ Finds maximum sustainable worker count under latency constraints: :::{tab-item}hide_directive--> **Dine-In Configuration** -| Variable | Default | Description | -| ------------------- | ----------------------- | ------------------------------------ | -| `TARGET_LATENCY_MS` | 15000 | Target latency threshold (ms) | -| `LATENCY_METRIC` | avg | 'avg', 'p95', or 'max' | -| `DENSITY_INCREMENT` | 1 | Concurrent images per iteration | -| `INIT_DURATION` | 60 | Warmup time (seconds) | -| `MIN_REQUESTS` | 3 | Min requests before measuring | -| `REQUEST_TIMEOUT` | 300 | Individual request timeout (seconds) | -| `API_ENDPOINT` | `http://localhost:8083` | API endpoint URL | -| `RESULTS_DIR` | `./results` | Results output directory | +| Variable | Default | Description | +| ----------------------------- | ----------------------- | ------------------------------------ | +| `BENCHMARK_TARGET_LATENCY_MS` | 25000 | Target latency threshold (ms) | +| `BENCHMARK_LATENCY_METRIC` | avg | 'avg', 'p95', or 'max' | +| `BENCHMARK_DENSITY_INCREMENT` | 1 | Concurrent images per iteration | +| `BENCHMARK_INIT_DURATION` | 60 | Warmup time (seconds) | +| `BENCHMARK_MIN_REQUESTS` | 3 | Min requests before measuring | +| `BENCHMARK_REQUEST_TIMEOUT` | 300 | Individual request timeout (seconds) | +| `BENCHMARK_API_ENDPOINT` | `http://localhost:8083` | API endpoint URL | +| `RESULTS_DIR` | `./results` | Results output directory | **Take-Away Configuration** -| Variable | Default | Description | -| -------------------- | ------- | --------------------------------- | -| `TARGET_LATENCY_MS` | 25000 | Target latency threshold (ms) | -| `LATENCY_METRIC` | avg | 'avg', 'p95', or 'max' | -| `WORKER_INCREMENT` | 1 | Workers added per iteration | -| `INIT_DURATION` | 10 | Warmup time (seconds) | -| `MIN_TRANSACTIONS` | 3 | Min transactions before measuring | -| `MAX_ITERATIONS` | 50 | Max scaling iterations | -| `MAX_WAIT_SEC` | 600 | Max wait per iteration (seconds) | -| `BENCHMARK_WORKERS` | 1 | Number of workers (fixed mode) | -| `BENCHMARK_DURATION` | 60 | Test duration (seconds) | +| Variable | Default | Description | +| ----------------------------- | ------- | ------------------------------------------------------ | +| `BENCHMARK_TARGET_LATENCY_MS` | 25000 | Target latency threshold (ms) | +| `BENCHMARK_LATENCY_METRIC` | avg | 'avg', 'p95' | +| `BENCHMARK_WORKER_INCREMENT` | 1 | Workers added per iteration | +| `BENCHMARK_INIT_DURATION` | 10 | Warmup time (seconds) | +| `BENCHMARK_MIN_TRANSACTIONS` | 1 | Min transactions before measuring | +| `BENCHMARK_WORKERS` | 1 | Number of workers (fixed mode) | +| `BENCHMARK_DURATION` | 200 | Test duration (seconds) | +| `OOM_PROTECTION` | 1 | Set to `0` to disable OOM protection (not recommended) | @@ -254,7 +311,7 @@ ls -la results/ ```bash make consolidate-metrics -cat results/metrics_summary.csv +cat results/consolidated_metrics.csv ``` ## Expected Performance diff --git a/docs/user-guide/oa-get-started.md b/docs/user-guide/oa-get-started.md index 02ae5b9f..85be1da5 100644 --- a/docs/user-guide/oa-get-started.md +++ b/docs/user-guide/oa-get-started.md @@ -41,22 +41,35 @@ cd order-accuracy/dine-in ``` -2. **Setup OVMS Models (First Time Only)** +2. **Configure the Environment** + + ```bash + # Create .env from template + make init-env + # Edit .env if needed — defaults work for most setups + + # Initialize git submodules (for benchmark tools) + make update-submodules + ``` + +3. **Setup OVMS Models (First Time Only)** + + The setup script reads model configuration (device, precision, model name) from `dine-in/.env` (created in Step 2), so **complete Step 2 before running this step**. ```bash cd ../ovms-service - ./setup_models.sh -app dine-in + ./setup_models.sh --app dine-in cd ../dine-in ``` This downloads and converts the Qwen2.5-VL-7B model (~7GB). This only needs to be done once. -3. **Prepare Test Data** - - Add your food tray/plate images to the `images/` folder +4. **Prepare Test Data** + - Add your food tray/plate images to the `images/` folder (`.jpg`, `.jpeg` or `.png`) - Update `configs/orders.json` with test orders - Update `configs/inventory.json` with your menu items -4. **Build and Start Services** +5. **Build and Start Services** ```bash # Using pre-built images (recommended for first run) @@ -68,7 +81,7 @@ make up ``` -5. **Access the Application** +6. **Access the Application** - **Gradio UI**: `http://localhost:7861` - **REST API Docs**: `http://localhost:8083/docs` @@ -89,22 +102,31 @@ cd order-accuracy/take-away ``` -2. **Setup OVMS Models (First Time Only)** +2. **Configure the Environment** ```bash - cd ../ovms-service - ./setup_models.sh -app take-away - cd ../take-away + # Create .env from template + make init-env + # Edit .env if needed — defaults work for most setups + + # Initialize git submodules (for benchmark tools) + make update-submodules ``` - This downloads the VLM and EasyOCR models. This only needs to be done once. +3. **Setup OVMS Models (First Time Only)** -3. **Initialize Environment** + Set `TARGET_DEVICE` in your `.env` **before** running this step. The script reads that value to export the model in the correct format for the target device. ```bash - make init-env + cd ../ovms-service + ./setup_models.sh --app take-away # Downloads and exports model (~30-60 min first time) + cd ../take-away ``` + This downloads the VLM and EasyOCR models. This only needs to be done once. + + > **Note:** Re-run this step any time you change `TARGET_DEVICE` in `.env`. + 4. **Build and Start Services** ```bash @@ -182,7 +204,7 @@ make logs make down # Stop and remove volumes (clean restart) -make down-volumes +make clean ``` ## Quick Start Reference @@ -196,7 +218,7 @@ make down-volumes | **Start Services** | `make up` | Start all Dine-In services | | **Build Locally** | `make build REGISTRY=false` | Build images from source | | **View Logs** | `make logs` | View service logs | -| **Stop Services** | `make down` | Stop all containers | +| **Stop Services** | `make down` | Stop all services | **Take-Away Commands** diff --git a/docs/user-guide/take-away/get-started.md b/docs/user-guide/take-away/get-started.md index 383674da..39b3ea34 100644 --- a/docs/user-guide/take-away/get-started.md +++ b/docs/user-guide/take-away/get-started.md @@ -256,7 +256,7 @@ make status # Check service status make logs # View logs make test-api # Test API health make down # Stop services -make clean # Stop and remove volumes +make clean # Stop containers and remove volumes make benchmark # Run fixed-workers benchmark make benchmark-stream-density # Run stream density benchmark ```