Skip to content

[DOCS] Align unified OA Benchmarking/GS to DI/TA Benchmarking/GS #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wiwaszko-intel wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[DOCS] Align unified OA Benchmarking/GS to DI/TA Benchmarking/GS #167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/user-guide/dine-in/di-benchmarking.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ make benchmark-stream-density
make benchmark-stream-density BENCHMARK_TARGET_LATENCY_MS=20000 BENCHMARK_INIT_DURATION=30
```

> **Note:** `make benchmark-density` runs a Python script locally that sends concurrent HTTP requests to the running `dine-in` API. No separate worker containers are needed for this mode.
> **Note:** `make benchmark-stream-density` runs a Python script locally that sends concurrent HTTP requests to the running `dine-in` API. No separate worker containers are needed for this mode.

## Metrics Processing

Expand Down
2 changes: 1 addition & 1 deletion docs/user-guide/dine-in/get-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ This downloads Qwen2.5-VL-7B-Instruct (~7 GB) and converts it to OpenVINO™ INT
Before running the application, you must prepare your test data:

1. **Add Images**: Place your food tray images in the `images/` folder
- Supported formats: `.jpg`, `.jpeg`, `.png`
- Supported formats: `.jpg`, `.jpeg` or `.png`
- Images should clearly show the food items on the tray

2. **Update Orders**: Edit `configs/orders.json` with your test orders
Expand Down
107 changes: 82 additions & 25 deletions docs/user-guide/oa-benchmarking.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,46 @@ Test your Order Accuracy pipeline performance on various hardware configurations

**Goal**: Run a basic performance test to verify your system works correctly

> **Note — Inference Device:** The default device is `GPU`. To switch to `CPU`, you must do **both** steps below, otherwise the model will be exported for the wrong device:
>
> 1. Set **both** variables in your `.env` file:
>
> ```bash
> TARGET_DEVICE=GPU # used by setup_models.sh and docker-compose
> OPENVINO_DEVICE=GPU # used by the Makefile benchmark targets
> ```
>
> 2. Re-export the model for the new device:
>
> <!--hide_directive::::{tab-set}
> :::{tab-item}hide_directive--> **Dine-In**
> <!--hide_directive:sync: dine-in hide_directive-->
>
> ```bash
> cd ../ovms-service && ./setup_models.sh --app dine-in
> ```
>
> <!--hide_directive:::
> :::{tab-item}hide_directive--> **Take-Away**
> <!--hide_directive:sync: take-away hide_directive-->
>
> ```bash
> cd ../ovms-service && ./setup_models.sh --app take-away
> ```
>
> <!--hide_directive:::
> ::::hide_directive-->
>
> `TARGET_DEVICE` is what `setup_models.sh` reads to export the model in the correct format. `OPENVINO_DEVICE` is what the Makefile passes to the benchmark script. Both must match.

### 1. Initialize Performance Tools

```bash
# 1. Initialize git submodules (first time only)
make update-submodules

# 2. Start services
make up
```

### 2. Run Quick Benchmark
Expand All @@ -27,8 +63,15 @@ make benchmark
:::{tab-item}hide_directive--> **Take-Away**
<!--hide_directive:sync: take-away hide_directive-->

> **Important:** Before running benchmarks, ensure a test video file is present at `storage/videos/test.mp4`. You can download a sample video using:
>
> ```bash
> make download-sample-video
> ```

```bash
cd take-away
# Default run
make benchmark
```

Expand All @@ -53,7 +96,7 @@ make benchmark
<!--hide_directive:sync: singlehide_directive-->

```bash
make benchmark
make benchmark-single IMAGE_ID=MCD-1001
```

Tests single image validation latency:
Expand All @@ -68,7 +111,10 @@ Tests single image validation latency:
<!--hide_directive:sync: density hide_directive-->

```bash
make benchmark-density
make benchmark-stream-density

# With overrides
make benchmark-stream-density BENCHMARK_TARGET_LATENCY_MS=20000 BENCHMARK_INIT_DURATION=30
```

Finds maximum concurrent requests the system can handle under latency constraints:
Expand Down Expand Up @@ -103,7 +149,10 @@ Tests end-to-end latency for single order validation:
:::{tab-item}hide_directive--> **Fixed Workers Benchmark**

```bash
make benchmark-oa BENCHMARK_WORKERS=4 BENCHMARK_DURATION=300
make benchmark \
BENCHMARK_WORKERS=4 \
BENCHMARK_DURATION=300 \
BENCHMARK_INIT_DURATION=30
```

Tests system with fixed number of concurrent workers:
Expand All @@ -118,7 +167,16 @@ Tests system with fixed number of concurrent workers:
<!--hide_directive:sync: density hide_directive-->

```bash
# Default run
make benchmark-stream-density

# Custom run
make benchmark-stream-density \
BENCHMARK_TARGET_LATENCY_MS=25000 \
BENCHMARK_LATENCY_METRIC=avg \
BENCHMARK_INIT_DURATION=30 \
BENCHMARK_MIN_TRANSACTIONS=3 \
BENCHMARK_WORKER_INCREMENT=1
```

Finds maximum sustainable worker count under latency constraints:
Expand All @@ -140,32 +198,31 @@ Finds maximum sustainable worker count under latency constraints:
:::{tab-item}hide_directive--> **Dine-In Configuration**
<!--hide_directive:sync: dine-in hide_directive-->

| Variable | Default | Description |
| ------------------- | ----------------------- | ------------------------------------ |
| `TARGET_LATENCY_MS` | 15000 | Target latency threshold (ms) |
| `LATENCY_METRIC` | avg | 'avg', 'p95', or 'max' |
| `DENSITY_INCREMENT` | 1 | Concurrent images per iteration |
| `INIT_DURATION` | 60 | Warmup time (seconds) |
| `MIN_REQUESTS` | 3 | Min requests before measuring |
| `REQUEST_TIMEOUT` | 300 | Individual request timeout (seconds) |
| `API_ENDPOINT` | `http://localhost:8083` | API endpoint URL |
| `RESULTS_DIR` | `./results` | Results output directory |
| Variable | Default | Description |
| ----------------------------- | ----------------------- | ------------------------------------ |
| `BENCHMARK_TARGET_LATENCY_MS` | 25000 | Target latency threshold (ms) |
| `BENCHMARK_LATENCY_METRIC` | avg | 'avg', 'p95', or 'max' |
| `BENCHMARK_DENSITY_INCREMENT` | 1 | Concurrent images per iteration |
| `BENCHMARK_INIT_DURATION` | 60 | Warmup time (seconds) |
| `BENCHMARK_MIN_REQUESTS` | 3 | Min requests before measuring |
| `BENCHMARK_REQUEST_TIMEOUT` | 300 | Individual request timeout (seconds) |
| `BENCHMARK_API_ENDPOINT` | `http://localhost:8083` | API endpoint URL |
| `RESULTS_DIR` | `./results` | Results output directory |

<!--hide_directive:::
:::{tab-item}hide_directive--> **Take-Away Configuration**
<!--hide_directive:sync: take-away hide_directive-->

| Variable | Default | Description |
| -------------------- | ------- | --------------------------------- |
| `TARGET_LATENCY_MS` | 25000 | Target latency threshold (ms) |
| `LATENCY_METRIC` | avg | 'avg', 'p95', or 'max' |
| `WORKER_INCREMENT` | 1 | Workers added per iteration |
| `INIT_DURATION` | 10 | Warmup time (seconds) |
| `MIN_TRANSACTIONS` | 3 | Min transactions before measuring |
| `MAX_ITERATIONS` | 50 | Max scaling iterations |
| `MAX_WAIT_SEC` | 600 | Max wait per iteration (seconds) |
| `BENCHMARK_WORKERS` | 1 | Number of workers (fixed mode) |
| `BENCHMARK_DURATION` | 60 | Test duration (seconds) |
| Variable | Default | Description |
| ----------------------------- | ------- | ------------------------------------------------------ |
| `BENCHMARK_TARGET_LATENCY_MS` | 25000 | Target latency threshold (ms) |
| `BENCHMARK_LATENCY_METRIC` | avg | 'avg', 'p95' |
| `BENCHMARK_WORKER_INCREMENT` | 1 | Workers added per iteration |
| `BENCHMARK_INIT_DURATION` | 10 | Warmup time (seconds) |
| `BENCHMARK_MIN_TRANSACTIONS` | 1 | Min transactions before measuring |
| `BENCHMARK_WORKERS` | 1 | Number of workers (fixed mode) |
| `BENCHMARK_DURATION` | 200 | Test duration (seconds) |
| `OOM_PROTECTION` | 1 | Set to `0` to disable OOM protection (not recommended) |

<!--hide_directive:::
::::hide_directive-->
Expand Down Expand Up @@ -254,7 +311,7 @@ ls -la results/

```bash
make consolidate-metrics
cat results/metrics_summary.csv
cat results/consolidated_metrics.csv
```

## Expected Performance
Expand Down
52 changes: 37 additions & 15 deletions docs/user-guide/oa-get-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,22 +41,35 @@
cd order-accuracy/dine-in
```

2. **Setup OVMS Models (First Time Only)**
2. **Configure the Environment**

```bash
# Create .env from template
make init-env
# Edit .env if needed — defaults work for most setups

# Initialize git submodules (for benchmark tools)
make update-submodules
```

3. **Setup OVMS Models (First Time Only)**

The setup script reads model configuration (device, precision, model name) from `dine-in/.env` (created in Step 2), so **complete Step 2 before running this step**.

```bash
cd ../ovms-service
./setup_models.sh -app dine-in
./setup_models.sh --app dine-in
cd ../dine-in
```

This downloads and converts the Qwen2.5-VL-7B model (~7GB). This only needs to be done once.

3. **Prepare Test Data**
- Add your food tray/plate images to the `images/` folder
4. **Prepare Test Data**
- Add your food tray/plate images to the `images/` folder (`.jpg`, `.jpeg` or `.png`)
- Update `configs/orders.json` with test orders
- Update `configs/inventory.json` with your menu items

4. **Build and Start Services**
5. **Build and Start Services**

```bash
# Using pre-built images (recommended for first run)
Expand All @@ -68,7 +81,7 @@
make up
```

5. **Access the Application**
6. **Access the Application**
- **Gradio UI**: `http://localhost:7861`
- **REST API Docs**: `http://localhost:8083/docs`

Expand All @@ -89,22 +102,31 @@
cd order-accuracy/take-away
```

2. **Setup OVMS Models (First Time Only)**
2. **Configure the Environment**

```bash
cd ../ovms-service
./setup_models.sh -app take-away
cd ../take-away
# Create .env from template
make init-env
# Edit .env if needed — defaults work for most setups

# Initialize git submodules (for benchmark tools)
make update-submodules
```

This downloads the VLM and EasyOCR models. This only needs to be done once.
3. **Setup OVMS Models (First Time Only)**

3. **Initialize Environment**
Set `TARGET_DEVICE` in your `.env` **before** running this step. The script reads that value to export the model in the correct format for the target device.

```bash
make init-env
cd ../ovms-service
./setup_models.sh --app take-away # Downloads and exports model (~30-60 min first time)
cd ../take-away
```

This downloads the VLM and EasyOCR models. This only needs to be done once.

> **Note:** Re-run this step any time you change `TARGET_DEVICE` in `.env`.

4. **Build and Start Services**

```bash
Expand Down Expand Up @@ -182,7 +204,7 @@ make logs
make down

# Stop and remove volumes (clean restart)
make down-volumes
make clean
```

## Quick Start Reference
Expand All @@ -196,7 +218,7 @@ make down-volumes
| **Start Services** | `make up` | Start all Dine-In services |
| **Build Locally** | `make build REGISTRY=false` | Build images from source |
| **View Logs** | `make logs` | View service logs |
| **Stop Services** | `make down` | Stop all containers |
| **Stop Services** | `make down` | Stop all services |

<!--hide_directive:::
:::{tab-item}hide_directive--> **Take-Away Commands**
Expand Down
2 changes: 1 addition & 1 deletion docs/user-guide/take-away/get-started.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ make status # Check service status
make logs # View logs
make test-api # Test API health
make down # Stop services
make clean # Stop and remove volumes
make clean # Stop containers and remove volumes
make benchmark # Run fixed-workers benchmark
make benchmark-stream-density # Run stream density benchmark
```
Expand Down
Loading