Feature: Add vLLM Integration and Upgrade HARM System for VRAM Optimization

[Ewan-Dkhar]

Context

To achieve high-throughput, low-latency code generation, we are introducing a vLLM container to run alongside our local SearXNG instance. However, because our target environment utilizes a limited GPU resources, we cannot let vLLM run with its default configurations. vLLM aggressively pre-allocates VRAM for its KV Cache, which will cause the PyTorch DockerSandboxRunner to instantly encounter CUDA Out of Memory (OOM) errors during the execution phase.

Problem

• VRAM Starvation: Running vLLM without memory constraints will lock up the host GPU, leaving no VRAM available for the execution/testing sandbox.
• Lack of API Unload: Unlike Ollama, vLLM does not support an API endpoint to explicitly evict/unload a model (keep_alive: 0). The API server must remain active, meaning it permanently retains its assigned VRAM while running.

Proposed Solution

1. Spin up vLLM via Docker Compose using strict memory limiters (--gpu-memory-utilization and --enforce-eager).
2. Upgrade the Hardware Resource Manager (HARM) to use the Python docker SDK to physically pause and unpause the vLLM container. This safely freezes the process and releases its VRAM allocations right before the PyTorch sandbox runs, restoring the container once execution completes.

Acceptance Criteria

• Add the vllm-server service to the docker-compose.yml mapping Qwen/Qwen2.5-Coder-3B-Instruct.
• Hard-cap the service configuration with --gpu-memory-utilization 0.5, --max-model-len 4096, and --enforce-eager to protect the host architecture.
• Add the docker Python SDK to our environment dependencies via uv add docker.
• Implement evict_llm() and restore_llm() methods inside a dedicated HardwareResourceManager class using container pause/unpause hooks.
• Update the main graph's execute_code_node to trigger the HARM isolation routine wrapping the DockerSandboxRunner.
• Verify a full integration loop using test_self_healing.py under the vLLM backend configuration.

Proposed Service & Code Architecture

docker-compose.yml Snippet:

vllm-server:
  image: vllm/vllm-openai:latest
  container_name: hephaestus_vllm
  runtime: nvidia
  ports:
    - "8000:8000"
  volumes:
    - ~/.cache/huggingface:/root/.cache/huggingface
  ipc: host
  command: >
    --model Qwen/Qwen2.5-Coder-3B-Instruct
    --gpu-memory-utilization 0.5
    --max-model-len 4096
    --enforce-eager
  networks:
    - hephaestus_net

HARM Logic Blueprint:

import docker
import time

class HardwareResourceManager:
    def __init__(self):
        self.docker_client = docker.from_env()
        self.vllm_container_name = "hephaestus_vllm"

    async def evict_llm(self):
        try:
            container = self.docker_client.containers.get(self.vllm_container_name)
            if container.status == "running":
                print("[HARM] Pausing vLLM to reclaim VRAM...")
                container.pause()
                time.sleep(2)
        except docker.errors.NotFound:
            pass

    async def restore_llm(self):
        try:
            container = self.docker_client.containers.get(self.vllm_container_name)
            if container.status == "paused":
                print("[HARM] Resuming vLLM service...")
                container.unpause()
        except docker.errors.NotFound:
            pass