AI: Enforce Q5_K_M minimum quantization for SGLang production inference

[anchapin]

Summary

Audit the current SGLang self-hosted inference deployment (#1203) to verify the serving model uses at least Q5_K_M (5-bit) quantization. Models quantized to Q4 or below produce measurable increases in syntax errors in generated code — mismatched brackets, truncated JSON, invalid JavaScript — which directly degrades PortKit's Bedrock output quality.

Problem

The SGLang deployment (PR #1297) went live without a documented quantization floor. The inference quality literature for code generation is unambiguous:

"Q5_K_M (5-bit) quantization is the absolute minimum threshold for reliable code generation; 4-bit and below often introduce syntax errors (e.g., mismatched brackets) that break compilation."

Bedrock Add-on output is structured code: manifest.json must be valid JSON with exact field names, and the Scripting API .js files must parse without syntax errors. Q4 quantization artifacts (off-by-one token predictions, truncated outputs) would silently produce invalid output that fails downstream validation.

What to do

1. Check current serving format: SSH into the SGLang RunPod instance and verify which GGUF/AWQ/EXL2 model file is being served. Document the quantization level.

2. If below Q5_K_M: Download and serve the Q5_K_M or Q6_K GGUF of the current model instead. For Qwen2.5-Coder-7B, the Q5_K_M GGUF is available on HuggingFace from Bartowski's quantized models.

3. For GPU-only inference paths (AWQ/EXL2 via vLLM): The equivalent quality floor is AWQ 4-bit with group size 128 (which preserves more precision than standard Q4). If using AWQ, ensure group_size ≤ 128.

4. Add a startup assertion to the SGLang launch script that logs the quantization level and warnings if it detects a model below the 5-bit threshold.

5. Document the standard in the SGLang deployment runbook: minimum Q5_K_M for GGUF, minimum AWQ-4bit-gs128 for GPU-only paths.

Acceptance Criteria

• Current quantization level documented
• Model serving at ≥ Q5_K_M (GGUF) or ≥ AWQ-4bit-gs128 (GPU)
• JSON validity rate on eval set ≥ 90% (run the eval script from docs/ml_intern_finetuning_prompt.md)
• JavaScript syntax pass rate ≥ 85% on eval set

References

• Source document: AI for Game and Mod Coding — Section 6 (Quantization and Deployment), "Technical Advice"
• SGLang deployment: PR feat: Self-hosted LLM inference deployment via SGLang (#1203) #1297
• Related: ml-intern fine-tuning — docs/ml_intern_finetuning_prompt.md

[anchapin]

Investigation Summary

Finding: SGLang is not yet deployed. The codebase has client integration (PR #1297) but no running SGLang/vLLM server. The architecture doc marks SGLang as "Phase 3 — post-beta."

Actions Taken

1. Quantization floor enforcement added to inference client

Added to ai-engine/utils/self_hosted_inference.py:

• New function: check_quantization_floor(model_name, quant_type, awq_group_size) — validates GGUF ≥ Q5_K_M (5-bit) or AWQ ≥ 4-bit with group_size ≤ 128
• New env vars: MODEL_QUANT_TYPE (gguf/awq/exl2) and AWQ_GROUP_SIZE (default 128)
• Startup warning: Quantization floor is checked and logged on client init; models below floor log a warning

2. Documentation updated

Updated docs/INFERENCE_DEPLOYMENT.md with:

• Quantization standards table (GGUF ≥ Q5_K_M, AWQ ≥ 4-bit/gs128)
• Rationale (Q4 produces syntax errors in code generation)
• Configuration env vars
• Recommended model variants for Qwen2.5-Coder-7B

What's Still Needed

When SGLang is actually deployed:

1. Verify the GGUF file served matches the Q5_K_M (or higher) floor
2. Run eval metrics: JSON validity ≥ 90%, JS syntax ≥ 85%

The acceptance criteria in the issue (JSON validity ≥ 90%, JS syntax ≥ 85%) require a live server to measure — tracked as part of the Phase 3 SGLang deployment.

References

• ai-engine/utils/self_hosted_inference.py — quantization validation
• docs/INFERENCE_DEPLOYMENT.md — Quantization Standards section
• AWQ equivalent: 4-bit with group_size ≤ 128 (not 4-bit with default group_size)