Update GPU Inference and Managed Model Selection - #72
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds GPU compatibility manifests, CUDA 12.9 runtime verification, pinned vLLM dependencies, GPU and Slurm capacity discovery, qualified model selection, and tensor-parallel launch planning. It introduces shared coding-agent acceptance scripts with direct inference, approval handling, replay, audit verification, and qualification reports. Model catalogs and persisted selections gain expanded metadata and schema v2 support. CLI and Web UI downloads now require explicit confirmation, while workflows collect qualification artifacts and documentation describes the new GPU compatibility and runtime paths. Sequence Diagram(s)sequenceDiagram
participant CLI
participant SessionGateway
participant GPUEnvironment
participant ModelCatalog
participant VLLMRuntime
CLI->>SessionGateway: request managed model plan
SessionGateway->>GPUEnvironment: inspect visible GPUs or Slurm capacity
SessionGateway->>ModelCatalog: recommend qualified compatible snapshot
ModelCatalog-->>SessionGateway: model configuration and resource requirements
SessionGateway-->>CLI: launch plan and download requirement
CLI->>VLLMRuntime: start tensor-parallel vLLM runtime
VLLMRuntime-->>CLI: health readiness
Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
documentation/reference/glossary.md (2)
75-80: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMaintain alphabetical order in the glossary.
As per coding guidelines, glossary entries should be kept roughly alphabetical.
Tensor parallelism(T-e-n) should appear beforeTerra(T-e-r).♻️ Proposed fix to reorder entries
-**Terra** -: A cloud platform for biomedical research workspaces, data, workflows, and interactive Jupyter compute. - **Tensor parallelism** : A runtime layout that divides one model across a fixed number of GPUs. +**Terra** +: A cloud platform for biomedical research workspaces, data, workflows, and interactive Jupyter compute. +🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@documentation/reference/glossary.md` around lines 75 - 80, Reorder the glossary entries so “Tensor parallelism” appears before “Terra,” preserving the existing entry text and formatting.Source: Coding guidelines
30-34: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMaintain alphabetical order in the glossary.
As per coding guidelines, glossary entries should be kept roughly alphabetical.
CUDA(C-u) should appear afterCredential binding(C-r).♻️ Proposed fix to reorder entries
-**CUDA** -: NVIDIA's software platform and application binary interface for GPU computing. - **Credential binding** : A non-secret identifier that tells Heartwood where an authorized process or platform supplies a provider credential. +**CUDA** +: NVIDIA's software platform and application binary interface for GPU computing. +🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@documentation/reference/glossary.md` around lines 30 - 34, Reorder the glossary entries so “Credential binding” appears before “CUDA,” preserving the existing definitions and formatting.Source: Coding guidelines
packages/gateway/src/heartwood/gateway/_gateway.py (1)
843-845: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winConsider memoizing the GPU inventory to avoid blocking subprocesses on hot paths.
gpu_environment()runsinspect_gpu_environment, which shells out tonvidia-smi(10s timeout) and, on Carina login nodes,sinfo(15s timeout). This executes on everymodel_artifacts()call (Line 774) and again on the vLLM-availability fallback in_local_model_choice_dict(gpu_environment or self.gpu_environment()), reached frominspect_model_repository(Line 909) and_require_local_model_runtime(Line 1475, only to build an error message). Hardware/scheduler inventory is effectively static within a gateway lifetime, so repeated synchronous discovery adds avoidable latency to interactive model-listing requests.Caching the result per gateway instance (with an explicit refresh if needed) would keep listing responsive.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/gateway/src/heartwood/gateway/_gateway.py` around lines 843 - 845, Memoize the result of gpu_environment() per gateway instance so repeated model_artifacts(), _local_model_choice_dict(), inspect_model_repository(), and _require_local_model_runtime() calls reuse the discovered inventory instead of rerunning blocking subprocesses. Add an explicit refresh mechanism that invalidates and recomputes the cached GpuEnvironment when requested, while preserving the existing inspect_gpu_environment inputs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@images/generic/scripts/verify_coding_agent_e2e.py`:
- Around line 93-94: Update the tool execution validation using the existing
event.payload access so a missing exit_code defaults to 0 before comparing
against the failure value. Preserve raising ValueError for explicitly nonzero
exit codes while allowing successful executions that omit exit_code.
- Around line 147-154: Update the artifact_path handling before the
sensitive-value validation loop so the path is resolved to an absolute path,
then use its resolved parent directory when checking audit_text. Preserve the
existing checks for the task-content strings and the unsanitized-content
ValueError.
In `@packages/cli/src/heartwood/cli/__init__.py`:
- Around line 430-435: Update the combined validation error in the
argument-validation flow to remove --cpus from its message, since --cpus is
handled by its own check. Keep the validation condition for --startup-timeout
and --port unchanged and report only those options in the associated
parser.error message.
In `@packages/gateway/src/heartwood/gateway/_project_config.py`:
- Around line 716-717: Update the catalog_source fallback in the project
configuration construction to use "catalog" instead of the superseded
"recommended" value. Keep it consistent with LocalModelSelection.validate(), the
dataclass default, and select_local_model’s default, without retaining a
compatibility path for "recommended".
---
Nitpick comments:
In `@documentation/reference/glossary.md`:
- Around line 75-80: Reorder the glossary entries so “Tensor parallelism”
appears before “Terra,” preserving the existing entry text and formatting.
- Around line 30-34: Reorder the glossary entries so “Credential binding”
appears before “CUDA,” preserving the existing definitions and formatting.
In `@packages/gateway/src/heartwood/gateway/_gateway.py`:
- Around line 843-845: Memoize the result of gpu_environment() per gateway
instance so repeated model_artifacts(), _local_model_choice_dict(),
inspect_model_repository(), and _require_local_model_runtime() calls reuse the
discovered inventory instead of rerunning blocking subprocesses. Add an explicit
refresh mechanism that invalidates and recomputes the cached GpuEnvironment when
requested, while preserving the existing inspect_gpu_environment inputs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: f5762379-d423-4db1-a9c4-e71c53114b93
📒 Files selected for processing (72)
.github/workflows/container-smoke.yml.github/workflows/create-release.yml.github/workflows/dependency-review.yml.github/workflows/gpu-container-image.yml.github/workflows/main-validation.yml.github/workflows/validate.ymlAGENTS.mddeploy/carina/bootstrap.shdeploy/tests/native_installer_smoke.shdeploy/verify_gpu_compatibility.pydeploy/verify_model_sources.pydocumentation/architecture/testing.mddocumentation/models/choose-managed.mddocumentation/models/run-with-heartwood.mddocumentation/operate/index.mddocumentation/platforms/carina.mddocumentation/platforms/containers.mddocumentation/platforms/index.mddocumentation/platforms/terra.mddocumentation/reference/cli.mddocumentation/reference/glossary.mddocumentation/reference/gpu-compatibility.mddocumentation/reference/troubleshooting.mdimages/Dockerfileimages/generic/README.mdimages/generic/local-runtime/snapshots.tomlimages/generic/scripts/capable_model_e2e.shimages/generic/scripts/coding_agent_e2e.shimages/generic/scripts/verify_coding_agent_e2e.pyimages/gpu/coding_agent_e2e.shimages/gpu/compatibility.tomlimages/gpu/heartwood-vllmimages/gpu/heartwood_vllm.pyimages/gpu/install_runtime.shimages/gpu/qualification_config.pyimages/gpu/sitecustomize.pyimages/gpu/start_vllm.shimages/gpu/verify_runtime.shimages/gpu/verify_vllm.pyimages/gpu/vllm-exclusions.txtimages/gpu/vllm-overrides.txtimages/gpu/vllm-requirements.txtimages/gpu/vllm.inimages/platforms.tomlpackages/cli/src/heartwood/cli/__init__.pypackages/cli/src/heartwood/cli/_launch.pypackages/cli/tests/test_carina_launch_integration.pypackages/cli/tests/test_cli.pypackages/cli/tests/test_launch.pypackages/compliance/tests/test_coding_agent_qualification.pypackages/compliance/tests/test_container_assets.pypackages/compliance/tests/test_model_source_verification.pypackages/gateway/src/heartwood/gateway/__init__.pypackages/gateway/src/heartwood/gateway/_gateway.pypackages/gateway/src/heartwood/gateway/_gpu_environment.pypackages/gateway/src/heartwood/gateway/_local_import.pypackages/gateway/src/heartwood/gateway/_local_models.pypackages/gateway/src/heartwood/gateway/_model_snapshots.pypackages/gateway/src/heartwood/gateway/_project_config.pypackages/gateway/tests/test_gateway_contract.pypackages/gateway/tests/test_gpu_environment.pypackages/gateway/tests/test_local_models.pypackages/gateway/tests/test_model_artifacts.pypackages/gateway/tests/test_model_snapshots.pypackages/gateway/tests/test_project_config.pypackages/gateway/tests/test_readiness.pypackages/webui/src/App.test.tsxpackages/webui/src/client.test.tspackages/webui/src/components/UtilitySheet.tsxpackages/webui/src/e2e/app.spec.tspackages/webui/src/types.tszensical.toml
💤 Files with no reviewable changes (3)
- images/gpu/sitecustomize.py
- images/gpu/heartwood_vllm.py
- deploy/carina/bootstrap.sh
|
@coderabbitai review |
✅ Action performedReview finished.
|
♻️ Current situation & Problem
Heartwood's GPU runtime and managed-model metadata did not provide a reproducible CUDA 12.9 stack, platform-aware resource planning, or an end-to-end qualification gate for coding models.
Related: #25, #34, #42, #44, #49, #51, #53.
⚙️ Release Notes
0.25.1+cu129with PyTorch2.11.0+cu129in an isolated, fully hashed runtime and prevent CUDA 13 dependency substitution.The new platform configurations remain evaluation candidates until the exact released image and model revision pass the protected acceptance task on Terra T4 or Carina L40S hardware. Heartwood does not label or automatically select an unqualified candidate as recommended.
📚 Documentation
Adds a release-owned GPU compatibility matrix and expands the model, Terra, Carina, container, CLI, testing, and recovery guidance with pinned revisions, resource envelopes, expected download sizes, startup ranges, context defaults, tensor parallelism, and tool parsers.
✅ Testing
777 passedwith 90.08% Python coverage; Ruff, mypy, actionlint, shellcheck, yamllint, and strict documentation build pass.Live Terra and Carina model qualification requires their managed GPU environments and therefore remains gated by the protected workflow rather than claimed from CPU-only CI.
Code of Conduct & Contributing Guidelines
By creating and submitting this pull request, you agree to follow our Code of Conduct and Contributing Guidelines: