pipeline enhancement

[sainijit]

No description provided.

[Copilot]

Pull request overview

This PR enhances the video/VLM pipeline by tuning RTSP GStreamer parameters for stability, expanding OVMS model setup/configurability (including Hugging Face auth), and adding VLM prompt/response improvements plus debug observability (prompt config, structured decoding, warmup, and JSONL debug logs).

Changes:

• Adjust RTSP pipeline defaults (latency/timeout/retry, queue tuning, h264parse) in both take-away station worker and core runner.
• Expand setup_models.sh to support multiple model sources, validate OVMS_MODEL_NAME, and load optional HF auth tokens from app .env.
• Improve dine-in VLM client behavior (square letterbox preprocessing, prompt config file, guided decoding schema, warmup) and add prediction/validation debug logging with added request metadata.

Reviewed changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
take-away/src/parallel/station_worker.py	Adjust RTSP defaults and persistent pipeline string (latency/timeout/drop-on-latency, queue params, h264parse).
take-away/src/core/pipeline_runner.py	Make RTSP latency and FPS configurable via env; update queue settings and RTSP element properties.
take-away/.env.example	Document optional HF token for gated model downloads.
ovms-service/setup_models.sh	Add supported-model registry, validate model selection, load HF tokens, and improve export error reporting.
ovms-service/README.md	Update model export instructions and document OVMS_MODEL_NAME behavior and HF auth.
dine-in/src/worker.py	Pass image filename through to validation for richer diagnostics.
dine-in/src/services/vlm_client.py	Add prompt config loading, guided decoding schema/max token cap, warmup call, square resize behavior, and debug logging metadata.
dine-in/src/services/validation_service.py	Propagate image filename into VLM inference and log validation debug records.
dine-in/src/services/prediction_debug_logger.py	Introduce JSONL debug logger for prediction/validation artifacts.
dine-in/src/api.py	Add VLM warmup during startup; include filename in single/batch validation calls.
dine-in/README.md	Update setup documentation to reflect model selection and optional HF auth.
dine-in/Makefile	Increase benchmark iterations and add cleanup of stale containers before benchmarking.
dine-in/configs/prompt_config.json	Add runtime-editable prompt template/config for VLM prompting and schema examples.
dine-in/configs/orders.json	Update example orders used by the app/benchmarks.
dine-in/.env.example	Change default model and document optional HF auth token.
.gitignore	Ignore additional results dir and .github/skills/ session artifacts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

take-away/src/parallel/station_worker.py:936

• _verify_rtsp_stream_quick() hard-codes latency=500 and derives its own timeout= value, which can drift from PipelineConfig (e.g., if latency/timeout are tuned via config). Use self.pipeline_config values (and cap the rtspsrc timeout to the overall process timeout) so verification behavior matches the actual pipeline settings.

                    'timeout', str(timeout_sec),
                    'gst-launch-1.0', '-e',
                    'rtspsrc', f'location={self.rtsp_url}', 'protocols=tcp', 'latency=500',
                    f'timeout={timeout_sec * 1000000}',  # rtspsrc timeout is in microseconds

dine-in/src/services/vlm_client.py:520

• max_output_tokens comes from JSON config and is used as the OpenAI-compatible max_tokens payload field. If the config is edited to a non-integer (string/null/etc.), the request will fail at runtime. Coerce/validate this value and fall back to MAX_OUTPUT_TOKENS on invalid input.

        # Instance-level max_tokens/response_format so configs/prompt_config.json
        # controls these too (falls back to the class-level defaults above if the
        # config doesn't specify them).
        self.max_output_tokens = self.prompt_config.get("max_output_tokens", self.MAX_OUTPUT_TOKENS)
        self.response_format = json.loads(json.dumps(self.RESPONSE_FORMAT))  # deep copy

dine-in/src/services/vlm_client.py:807

• _load_prompt_config() claims a bad edit of configs/prompt_config.json can’t take the service down, but _build_prompt() assumes the loaded templates are strings and calls .format() unconditionally. If a user accidentally sets a template key to null/non-string, this will raise and break inference. Add type-safe fallbacks to the built-in templates before calling .format().

        if self.model_name.startswith("OpenVINO/Phi"):
            inventory_list = "\n".join(self.inventory_items) if self.inventory_items else ""
            template = cfg.get("phi", {}).get("with_inventory_template", "")
            prompt = template.format(
                inventory_list=inventory_list,

[Copilot]

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

ovms-service/setup_models.sh:88

• Associative-array value access uses an unquoted subscript, which can break when OVMS_MODEL_NAME contains characters that undergo word-splitting/globbing. Quote the subscript to make the lookup robust.

MODEL_SOURCES["${OVMS_MODEL_NAME_ENV}"]="${SUPPORTED_MODEL_SOURCES[${OVMS_MODEL_NAME_ENV}]}"

ovms-service/setup_models.sh:77

• Associative-array lookup uses an unquoted subscript, which can trigger word-splitting/globbing if OVMS_MODEL_NAME contains whitespace or glob characters, and can fail for unexpected values. Quote the subscript when checking membership.

This issue also appears on line 88 of the same file.

if [ -z "${SUPPORTED_MODEL_SOURCES[${OVMS_MODEL_NAME_ENV}]+x}" ]; then

dine-in/src/services/vlm_client.py:261

• _smart_resize() now upscales images that are already smaller than max_size (scale_factor > 1), increasing preprocessing cost without adding information. If the intent is letterboxing without upscaling, cap the scale factor at 1.0 and only downscale when needed.

        target_size = self.max_size
        scale_factor = min(target_size / original_width, target_size / original_height)