Validate Terra Runtime And Local Model Integration

.github/workflows/create-release.yml

@@ -67,6 +67,8 @@ jobs:
             echo "tag already exists: ${RELEASE_VERSION}" >&2
             exit 1
           fi
+      - name: Verify recommended model sources
+        run: python3 deploy/verify_model_sources.py --source-root .
       - name: Verify release readiness on the exact commit
         env:
           GH_TOKEN: ${{ github.token }}

.github/workflows/validate.yml

@@ -20,6 +20,16 @@ concurrency:
   cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 jobs:
+  model-sources:
+    name: Verify Recommended Model Sources
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      - name: Verify immutable Hugging Face revisions
+        run: python3 deploy/verify_model_sources.py --source-root . --allow-unavailable
   reuse:
     name: REUSE Compliance
     uses: SchmiedmayerLab/.github/.github/workflows/reuse.yml@v0.2

deploy/carina/bootstrap.sh

@@ -75,6 +75,7 @@ printf 'Installing the locked vLLM environment.\n'
 "${root}/bootstrap/bin/uv" pip sync \
   --require-hashes --python "${root}/vllm/bin/python" images/gpu/vllm-requirements.txt
 install -m 0444 images/gpu/heartwood_vllm.py "${root}/vllm/bin/heartwood_vllm.py"
+install -m 0444 images/gpu/sitecustomize.py "${root}/vllm/bin/sitecustomize.py"
 install -m 0555 images/gpu/heartwood-vllm "${root}/vllm/bin/heartwood-vllm"
 
 export PATH="${root}/bootstrap/bin:${PATH}"

deploy/verify_model_sources.py

@@ -0,0 +1,147 @@
+# This source file is part of the Heartwood open-source project
+#
+# SPDX-FileCopyrightText: 2026 Stanford University and the project authors (see CONTRIBUTORS.md)
+#
+# SPDX-License-Identifier: MIT
+
+"""Verify that recommended Hugging Face model revisions still resolve exactly."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import time
+import tomllib
+import urllib.error
+import urllib.parse
+import urllib.request
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+_SNAPSHOT_CATALOG = Path("images/generic/local-runtime/snapshots.toml")
+_COMMIT_PATTERN = re.compile(r"^[0-9a-f]{40}$")
+
+
+class ModelSourceVerificationError(RuntimeError):
+    """Raised when a recommended model source cannot be verified."""
+
+
+class ModelSourceUnavailableError(RuntimeError):
+    """Raised when the provider cannot be reached after bounded retries."""
+
+
+@dataclass(frozen=True)
+class ModelSource:
+    """One immutable recommended model source."""
+
+    model_id: str
+    repository: str
+    revision: str
+
+    @property
+    def api_url(self) -> str:
+        """Return the Hugging Face revision endpoint."""
+        repository = urllib.parse.quote(self.repository, safe="/")
+        revision = urllib.parse.quote(self.revision, safe="")
+        return f"https://huggingface.co/api/models/{repository}/revision/{revision}"
+
+
+JsonFetcher = Callable[[str, float], dict[str, Any]]
+
+
+def load_model_sources(source_root: Path) -> tuple[ModelSource, ...]:
+    """Load and validate every recommended model pin from the catalog."""
+    catalog_path = source_root / _SNAPSHOT_CATALOG
+    with catalog_path.open("rb") as file:
+        catalog = tomllib.load(file)
+    snapshots = catalog.get("snapshots")
+    if not isinstance(snapshots, dict) or not snapshots:
+        raise ModelSourceVerificationError("model snapshot catalog contains no snapshots")
+    sources: list[ModelSource] = []
+    for model_id, value in sorted(snapshots.items()):
+        if not isinstance(model_id, str) or not isinstance(value, dict):
+            raise ModelSourceVerificationError("model snapshot catalog is malformed")
+        repository = value.get("source_repository")
+        revision = value.get("source_revision")
+        if not isinstance(repository, str) or repository.count("/") != 1:
+            raise ModelSourceVerificationError(f"{model_id}: invalid Hugging Face repository")
+        if not isinstance(revision, str) or _COMMIT_PATTERN.fullmatch(revision) is None:
+            raise ModelSourceVerificationError(f"{model_id}: revision must be a commit SHA")
+        sources.append(ModelSource(model_id, repository, revision))
+    return tuple(sources)
+
+
+def verify_model_source(source: ModelSource, *, fetch_json: JsonFetcher) -> None:
+    """Require the remote repository and immutable commit to match the catalog."""
+    payload = fetch_json(source.api_url, 20)
+    repository = payload.get("id")
+    if repository is None:
+        repository = payload.get("modelId")
+    revision = payload.get("sha")
+    if repository != source.repository:
+        raise ModelSourceVerificationError(
+            f"{source.model_id}: expected repository {source.repository}, got {repository!r}"
+        )
+    if revision != source.revision:
+        raise ModelSourceVerificationError(
+            f"{source.model_id}: expected revision {source.revision}, got {revision!r}"
+        )
+
+
+def _fetch_json(url: str, timeout: float) -> dict[str, Any]:
+    request = urllib.request.Request(url, headers={"User-Agent": "heartwood-release-verifier"})
+    last_error: Exception | None = None
+    for attempt in range(3):
+        try:
+            with urllib.request.urlopen(request, timeout=timeout) as response:
+                payload = json.load(response)
+            if not isinstance(payload, dict):
+                raise ModelSourceVerificationError("Hugging Face returned a non-object response")
+            return payload
+        except urllib.error.HTTPError as error:
+            if error.code < 500 and error.code not in {408, 425, 429}:
+                raise ModelSourceVerificationError(
+                    f"Hugging Face rejected the immutable revision request with HTTP {error.code}"
+                ) from error
+            last_error = error
+            if attempt < 2:
+                time.sleep(attempt + 1)
+        except (OSError, ValueError, urllib.error.URLError) as error:
+            last_error = error
+            if attempt < 2:
+                time.sleep(attempt + 1)
+    raise ModelSourceUnavailableError(f"Hugging Face is unavailable: {last_error}")
+
+
+def main() -> int:
+    """Verify all catalog model sources."""
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--source-root", type=Path, default=Path.cwd())
+    parser.add_argument(
+        "--allow-unavailable",
+        action="store_true",
+        help="warn instead of failing when the provider is temporarily unavailable",
+    )
+    args = parser.parse_args()
+    try:
+        sources = load_model_sources(args.source_root.resolve())
+        for source in sources:
+            try:
+                verify_model_source(source, fetch_json=_fetch_json)
+                print(f"Verified {source.repository}@{source.revision}")
+            except ModelSourceUnavailableError as error:
+                if not args.allow_unavailable:
+                    raise
+                print(f"WARNING: {source.model_id}: {error}")
+    except (OSError, tomllib.TOMLDecodeError, ModelSourceVerificationError) as error:
+        parser.error(str(error))
+    except ModelSourceUnavailableError as error:
+        parser.error(str(error))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

design/03-architecture.md

@@ -116,7 +116,7 @@ The gateway owns one model-catalog service. OpenAI and Anthropic catalogs use th
 
 A model profile remains the single non-secret execution configuration consumed by policy and OpenHands: a stable profile id, LiteLLM model identifier, base URL when needed, declared normalized completion endpoint, capability tier, and one credential binding. Selecting a catalog entry materializes or replaces one profile for that connection. Researcher interfaces do not require profile ids, credential-storage details, policy endpoints, or LiteLLM prefixes.
 
-Hosted model limits remain upstream-owned. The catalog projects context metadata reported by the provider and the pinned OpenHands/LiteLLM stack, while OpenHands owns provider-specific request shaping. Heartwood does not maintain a second cloud-model limit table. A Heartwood-managed local profile additionally persists the selected runtime context, passes the total capacity to the inference server, and divides it into an input allowance and bounded output budget through OpenHands' native token options. OpenHands' per-event character guard is derived from that input allowance so code and tool output are not prematurely truncated by its smaller default. The runtime, agent, CLI, browser, and notebook therefore describe one capacity.
+Hosted model limits remain upstream-owned. The catalog projects context metadata reported by the provider and the pinned OpenHands/LiteLLM stack, while OpenHands owns provider-specific request shaping. Heartwood does not maintain a second cloud-model limit table. A Heartwood-managed local model records its source-declared capacity, while launch selects an effective power-of-two tier from 16,384 through 1,048,576 tokens bounded by that capacity and conservative observed RAM or GPU-memory headroom. The 1,048,576-token validation ceiling accommodates current long-context metadata while bounding implausibly large values; it is not a default or a runtime guarantee. The launcher passes the effective capacity to the inference server, persists it in the active profile, and divides it into an input allowance and bounded output budget through OpenHands' native token options. OpenHands' per-event character guard is derived from that input allowance so code and tool output are not prematurely truncated by its smaller default. The runtime, agent, CLI, browser, and notebook therefore describe one effective capacity.
 
 A credential binding identifies a managed identity, environment-backed platform secret, mounted file, or session-only value. Project configuration stores the binding, never the secret. The CLI and same-origin web interface use the same gateway operation to establish a selection; a submitted token is never returned, logged, audited, written to project state, or accepted as a command-line argument. Session-only values must be entered again after the gateway restarts; durable bindings are owned and protected by the deployment.
 
@@ -130,7 +130,9 @@ Local profiles use the same contract. The local connection lists models reported
 
 An **Other model** path accepts a Hugging Face `owner/model` identifier and an optional revision. The gateway uses the maintained Hugging Face client to inspect repository metadata and resolve the request to a full immutable commit. When an NVIDIA vLLM deployment is available, it selects a complete snapshot only when the repository has standard configuration and weights and reports a text-generation architecture supported by the packaged tool-call parser; embedding models and unknown parser families fail before transfer. Otherwise, when the portable llama.cpp runtime is available, it selects one single-file GGUF and prefers a uniquely identifiable balanced quantization such as Q4_K_M. It rejects split GGUF files, ambiguous choices, repositories without complete size or digest metadata, unsupported formats, and runtime mismatches with a not-yet-supported message linked to the repository issue chooser. This is intentionally a best-effort compatibility layer rather than a model-format framework.
 
-Before transfer, the normalized plan contains the chosen runtime, immutable source, selected file when applicable, expected bytes, minimum free space, a conservative CPU/RAM/GPU estimate, context window, source-reported license posture, and `recommended` or `user-selected` catalog provenance. Reviewed recommendations declare a 32,768-token local context. User-selected models use supported repository metadata up to that same cap and fall back to 16,384 tokens when no usable value is available. Heartwood keeps this deterministic instead of silently shrinking context according to the current machine; the launcher compares the persisted choice with observed RAM and GPU memory and warns before startup when the conservative estimate exceeds available resources.
+Before transfer, the normalized plan contains the chosen runtime, immutable source, selected file when applicable, expected bytes, minimum free space, a conservative CPU/RAM/GPU estimate, context capacity, source-reported license posture, and `recommended` or `user-selected` catalog provenance. Reviewed recommendations currently declare a 32,768-token local capacity. User-selected models use supported repository metadata up to 1,048,576 tokens and default to 32,768 tokens when no usable value is available. At launch, Heartwood applies the same deterministic tier planner across platforms: it reserves resource headroom, selects the largest supported tier that fits the estimate, reports the reason, and uses the 32,768-token safe tier when model-size or memory evidence is unavailable. This explicit effective value keeps the runtime and all interaction surfaces aligned; runtime-native model validation remains the final compatibility check. A model with less than the normal 16,384-token agent minimum remains eligible only for compatibility fixtures and fails normal capability validation.
+
+Every OpenHands agent receives the upstream `LLMSummarizingCondenser`. When a profile declares an input budget, token-based condensation begins at 75% of that allowance, leaving room for system instructions, tool schemas, the next response, and summary recovery. The upstream 240-event threshold remains as a soft fallback when token metadata is unavailable or unusually sparse, and OpenHands' context-exceeded request remains a hard recovery trigger. The condenser uses the same policy-authorized model route under a separate usage identifier, preserves the first two and recent events, and lets OpenHands own token counting, overflow recovery, and summary construction. Heartwood does not maintain a parallel transcript summarizer. Condensation extends a session but cannot guarantee verbatim retention of old detail, so durable facts and outputs remain in project files and audit records rather than relying on model history alone.
 
 The plan and verified download metadata persist in `.heartwood/config.toml`, so a new gateway process projects the same state. Single-file GGUF content is checked against its source SHA-256; snapshots are staged atomically and receive an exact local manifest. The image may include a portable llama.cpp server binary or an explicit GPU vLLM runtime, but model weights live in project storage or a platform disk. NVIDIA images and the Carina native bundle isolate the same hash-locked CUDA 11.8 vLLM environment. Its launcher retains patched Transformers 5 behavior while applying legacy subclass initialization only to vLLM's bundled configuration classes, preserving the current Terra driver baseline without carrying a vulnerable Transformers 4 release. The launcher backports the upstream fix for [GHSA-8fr4-5q9j-m8gm](https://github.com/vllm-project/vllm/security/advisories/GHSA-8fr4-5q9j-m8gm) by removing the affected configuration from vLLM's private registry before any model configuration is loaded. The lock also overrides vLLM's vulnerable exact xgrammar pin with the first patched compatible release, overrides the legacy HTTP graph with a fully patched idna release, and verifies both dependencies with vLLM's structured-output backend. Image construction and native installation run these security regressions in addition to verifying the CUDA build and normal model-configuration integration. A live launch repeats the integration preflight and initializes CUDA before starting the server so a missing GPU, incompatible driver, vulnerable raw launcher, or dependency mismatch fails before model loading begins. Background downloads report transferred and expected bytes through the gateway; successful verification records the standard loopback model profile. Download completion produces `compute-required`, not runtime readiness or a model-capability claim, until `heartwood launch` starts and verifies the selected server.