evidence(containers): the SBSA image RUNS on Tegra -- Orin generates on the GPU

FOLLOWING_AGENTS_PROTOCOL

W6's Tegra determination, and it contradicts what this spec predicted. The risk
entry said Thor and Orin "are Tegra/L4T with a different CUDA runtime" and that
the arm64 image made no claim there. It runs. There is no CUDA-version wall,
and no separate Tegra lane is needed: ONE SBSA image serves both arm64
families.

Measured on a Jetson AGX Orin Developer Kit (sm_87, L4T R36.4.3, Docker 27.5.1,
Kairos immutable OS), running the image built on GB10:

  validator  config + layout verified; /health 200, /version 200, declared
             healthcheck passed, clean SIGTERM, on --runtime nvidia --gpus all
  model      Qwen3-0.6B (rev c1899de289a04d12100db370d81485cdf75e47ca, 1.5 GB)
             loads and GENERATES: "The capital of France is" -> " Paris. ..."
             24 completion tokens
  gpu        tegrastats GR3D_FREQ 95-97% during a 120-token generation, against
             14-15% idle -- decode runs on the Orin GPU, not a CPU fallback

That last line is the one /health could not make, on Orin or on GB10.

THE INVOCATION DIFFERS, and all three cases fail differently:

  --gpus all alone            REFUSED by the runtime hook
  --runtime nvidia alone      starts, no driver, dies on libcuda.so.1 missing
  --runtime nvidia --gpus all works

The middle case is the trap: the container comes up and dies on a missing
library, which reads as a broken image rather than a wrong flag. docs/USAGE.md
documented only `--gpus all` and was therefore wrong for every Jetson.

Two portability defects the second box exposed, both invisible on GB10 because
that box happened to match:

- validate-container-image.py used `docker stop --timeout`, which is
  newer-Docker only; Orin's 27.5.1 rejects it. `-t` is accepted by both.
- /models must be READABLE BY UID 1000. On GB10 the model was owned by uid 1000
  so mode 0600 worked; on Orin (files owned by 65535) the server initialised
  CUDA and died on `safetensors: cannot open file`, which reads like a corrupt
  checkpoint. Now documented.

A wrong hypothesis, killed by test rather than shipped: I assumed the injected
driver path just was not on the loader search path and tried patching
LD_LIBRARY_PATH to include /usr/lib/aarch64-linux-gnu/nvidia. Still not found --
because with --runtime alone the driver is not mounted at all. No Dockerfile
change is needed; the nvidia runtime handles ldconfig itself in csv mode.

Scope: Orin (sm_87) only. Thor (sm_110) has never been probed and inherits
nothing from this. Nothing is published to GHCR.

Issues: #170

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude:claude-opus-5 [Claude Code]

Author: mudler

.agents/engine-matrix.md

@@ -305,12 +305,14 @@ gate passes, and no publish step uses a wildcard.
 
 ## Risks and open questions
 
-- **Tegra versus SBSA.** A hosted `ubuntu-24.04-arm` runner produces an SBSA
-  CUDA build. GB10 is SBSA and is expected to run it; **Thor (`sm_110`) and
-  Orin (`sm_87`) are Tegra/L4T with a different CUDA runtime**. The arm64 cuda
-  image is claimed to run on Tegra only once it has run on Thor. If it cannot,
-  the honest outcome is a recorded boundary or a separate Tegra lane, never a
-  quiet widening of the arm64 tag.
+- **Tegra versus SBSA: MEASURED 2026-08-11, and the earlier prediction here
+  was WRONG.** This entry used to say the arm64 image made no Tegra claim,
+  on the reasoning that Thor and Orin run Tegra/L4T with a different CUDA
+  runtime. The SBSA image RUNS on Jetson AGX Orin (`sm_87`, L4T R36.4.3):
+  `/health` 200, `/version` 200, in-container healthcheck, clean SIGTERM.
+  There is no CUDA-version wall. What differs is the INVOCATION, not the
+  image -- see the Tegra section below. Thor (`sm_110`) is still unprobed
+  and inherits nothing from this.
 - **CUDA runtime redistribution.** Copying `libcudart.so.12` and
   `libcublasLt.so.12` out of the toolkit is permitted under the CUDA EULA's
   redistributable list, but the exact file list, version, and notice text are a
@@ -486,6 +488,64 @@ Lock discipline, since the box is shared: the build ran outside `gpu.lock`
 because it needs no GPU, and only the container run took the lock, blocking --
 it queued 16:58:38 -> 17:50:43 behind other users rather than jumping them.
 
+### W6 Tegra result: the SBSA image runs on Orin, but not the same way
+
+Measured 2026-08-11 on a Jetson AGX Orin Developer Kit (`sm_87`, L4T R36.4.3,
+Docker 27.5.1, Kairos immutable OS), running the SBSA image built on GB10:
+
+```
+container image OK  lane=cuda
+  config, layout: verified
+  boot: /health 200, /version 200, declared healthcheck passed, clean SIGTERM,
+        on --runtime nvidia --gpus all
+```
+
+So one image serves both arm64 families and no separate Tegra lane is needed.
+`libcuda.so.1` resolves to `/usr/lib/aarch64-linux-gnu/nvidia/libcuda.so.1`
+there, against `/usr/lib/aarch64-linux-gnu/libcuda.so.1` on SBSA; the nvidia
+runtime handles that itself, and the image needs no `LD_LIBRARY_PATH` change
+(tried, and it was the wrong hypothesis -- the driver simply was not mounted).
+
+**The invocation differs, and all three cases fail differently:**
+
+| flags | Tegra behaviour |
+|---|---|
+| `--gpus all` alone | REFUSED: "invoking the NVIDIA Container Runtime Hook directly ... is not supported" |
+| `--runtime nvidia` alone | starts, no driver mounted, dies on `libcuda.so.1: cannot open shared object file` |
+| `--runtime nvidia --gpus all` | driver injected, server runs |
+
+The middle case is the trap: the container comes up and dies on a missing
+library, which reads as a broken image rather than a wrong flag.
+
+**Two operator requirements this exposed**, both invisible on GB10 because that
+box happened to match:
+
+- `/models` must be READABLE BY UID 1000. The image runs as uid 1000; on GB10
+  the model was owned by uid 1000 so mode 0600 worked, while on Orin (files
+  owned by 65535) the server initialised CUDA and then died on
+  `safetensors: cannot open file`, which looks like a corrupt checkpoint.
+- `docker stop --timeout` is newer-Docker only. The validator used it and broke
+  on Docker 27.5.1; `-t` is accepted by both.
+
+**It generates, and the GPU does the work.** `/health` alone would only prove
+the engine came up, so the Orin run was taken further with a real model:
+Qwen3-0.6B (HF `Qwen/Qwen3-0.6B`, revision `c1899de289a04d12100db370d81485cdf75e47ca`,
+1.5 GB bf16) loaded in the container and served `/v1/completions`:
+
+```
+prompt      "The capital of France is"
+completion  " Paris. The capital of France is also the capital of the French Republic..."
+usage       prompt_tokens 5, completion_tokens 24
+```
+
+`tegrastats` during a 120-token generation reads **GR3D_FREQ 95-97%**, against
+14-15% at idle -- so decode runs on the Orin GPU rather than falling back to CPU
+paths. That is the distinction `/health` cannot make, and the GB10 result did
+not make either.
+
+**Scope.** Orin (`sm_87`) only. Thor (`sm_110`) has never been probed and
+inherits nothing from this.
+
 ### Pull-request scope, and why it is not a hole
 
 A release run builds every lane on both architectures. A pull request builds a

.agents/specs/container-images.md

@@ -8,7 +8,7 @@
 | **Structured state record (active)** | v1 scalar + relational + Git-history contracts | No benchmark. At `776c56f1`: 157 imports = 3,231,342 exact bytes; append preserved all 156 wrappers/rows. 95 tests: validator/core 44 (checker 20 + core 24), NOW 18, migration 22, cutover 11. New raw-row mutation guard. | n/a |
 | **Binary release matrix (ACTIVE; required W1-W11/W13 implemented in #196)** | Eight primary CPU/CUDA/Vulkan/Metal/MLX host tuples | Adaptive x86 tiers, Vulkan 35/35 + cross-device 11/11, and metadata/mutation gates green. **PENDING:** hosted full matrix, matching hardware, tagged publish | n/a |
 | **Binary release delivery topology** | #196: read-only build/verify, OIDC attest, protected publish; generated indexes and explicit handoff-authenticated assets | Fixes the zero-binary release path by attaching all eight archive/checksum/provenance triplets plus indexes. Hosted proof pending; W12 diagnostics optional | n/a |
-| **Container images (ACTIVE; arm64 cuda RUNTIME-VERIFIED 2026-08-11)** | `ENG-RELEASE-CONTAINERS` ([spec](../.agents/specs/container-images.md)) | cpu amd64 **783 MB**; cuda arm64 **1.71 GB**, `/health`+`/version`+healthcheck+clean SIGTERM **on `--gpus all`** on GB10 `sm_121a`; SIGTERM **0.25 s exit 0** (was 30 s SIGKILL, #312) | n/a |
+| **Container images (ACTIVE; arm64 cuda verified on GB10 + Orin 2026-08-11)** | `ENG-RELEASE-CONTAINERS` ([spec](../.agents/specs/container-images.md)) | cpu amd64 783 MB; cuda arm64 **1.71 GB**. GB10 `sm_121a`: `/health`+`/version`+SIGTERM on `--gpus all`. Orin `sm_87` (Tegra): Qwen3-0.6B **generates**, GPU **GR3D 95-97%** | n/a |
 | **Developer/row protocol** | Contribution entry point; `ENG-NOW-DERIVED` #374 @`dbd0d51c` | Entry-point gates retained. #374 W1-W5 DONE; benchmark/runtime/parity `VOID`; row specs now carry `## Now` | n/a |
 | **LoRA runtime W2** (`LORA-RUNTIME`, #278) | **No number owed:** correctness-only; a grid PENDS the W7 model gate |
 | **ARCH audit: ABI is text-only** | 4 capabilities (H3 video, Laguna, Kimi-Linear, DeepSeek-V4) reachable only from `examples/`, none registry-backed. No gate asks whether a CONSUMER can reach a capability. Documentation only |

docs/BENCHMARKS.md

@@ -38,9 +38,10 @@ Cold start: `MEASURED`. Load (#150): 27B bf16 loads **1.54x warm / 1.61x cold**,
 Releases: W1-W11/W13 are implemented in #196; local CPU, Vulkan, archive,
 metadata and mutation gates are green, while the hosted eight-tuple dry run and
 tagged publication stay pending, so no binary is published. Container
-images (#170): the cpu (amd64) and cuda (arm64) lanes pass their gates, the
-latter RUNTIME-VERIFIED on GB10 `sm_121a` with `--gpus all`; Tegra (Thor, Orin)
-is untested and nothing is published.
+images (#170): the cpu (amd64) and cuda (arm64) lanes pass their gates. One SBSA
+arm64 image serves BOTH families: GB10 `sm_121a`, and Jetson AGX Orin `sm_87`
+where Qwen3-0.6B generates with the GPU at 95-97% (Tegra needs
+`--runtime nvidia --gpus all`). Thor is unprobed; nothing is published.
 See [RELEASES](RELEASES.md).
 
 Protocol (2026-08-09): `776c56f1` has 157 imports = 3,231,342 exact bytes;

docs/STATUS.md

@@ -549,9 +549,28 @@ docker run --rm --gpus all -p 8000:8000 \
   --model /models/Qwen3.6-35B-A3B
 ```
 
-`/models` is the weights mount and `/cache` is the tokenizer/HF cache; the
-container runs as uid 1000, so `/cache` must be writable by it if you bind-mount
-one. `ffmpeg` is installed in every lane, so `/v1/videos` works out of the box —
+`/models` is the weights mount and `/cache` is the tokenizer/HF cache. The
+container runs as **uid 1000**, so `/cache` must be writable by it and the
+weights under `/models` must be READABLE by it. A model file with mode `0600`
+owned by another uid fails as `safetensors: cannot open file`, which reads like
+a corrupt checkpoint rather than a permissions problem.
+
+### On Jetson (Tegra/L4T)
+
+The cuda image is one SBSA build and it runs on Jetson too -- verified on AGX
+Orin (`sm_87`, L4T R36.4.3) -- but Tegra needs a **different invocation**:
+
+```sh
+docker run --rm --runtime nvidia --gpus all -p 8000:8000 \
+  -v /path/to/models:/models:ro \
+  ghcr.io/mudler/vllm.cpp:latest-cuda \
+  --model /models/your-model
+```
+
+`--gpus all` on its own is refused there ("invoking the NVIDIA Container Runtime
+Hook directly ... is not supported"), and `--runtime nvidia` on its own starts a
+container with no driver, which dies on `libcuda.so.1: cannot open shared object
+file`. Both flags together are what works. `ffmpeg` is installed in every lane, so `/v1/videos` works out of the box —
 a deliberate difference from the tarballs, which never vendor it because they
 are extracted onto a host that already has a `PATH`.
 

docs/USAGE.md

@@ -241,7 +241,12 @@ def wait_for_health(url: str, timeout: float) -> tuple[bool, str]:
 
 
 def check_boot(
-    image: str, model: Path, port: int, timeout: float, gpus: str | None = None
+    image: str,
+    model: Path,
+    port: int,
+    timeout: float,
+    gpus: str | None = None,
+    runtime: str | None = None,
 ) -> list[str]:
     errors: list[str] = []
     name = f"vllm-cpp-smoke-{port}"
@@ -251,6 +256,12 @@ def check_boot(
     # nothing about the GPU it was built for. The driver still comes from the
     # host through the container runtime; the image never carries one.
     gpu_args = ["--gpus", gpus] if gpus else []
+    # Tegra/L4T needs --runtime nvidia and REJECTS --gpus outright:
+    #   "invoking the NVIDIA Container Runtime Hook directly (e.g. specifying
+    #    the docker --gpus flag) is not supported"
+    # so a GPU lane cannot be validated on Jetson through --gpus alone.
+    if runtime:
+        gpu_args = ["--runtime", runtime, *gpu_args]
 
     run(["docker", "rm", "--force", name])
     code, output = run(
@@ -292,7 +303,9 @@ def check_boot(
 
         # SIGTERM, not SIGKILL: an image that has to be killed loses in-flight work
         # on every ordinary orchestrator restart.
-        code, output = run(["docker", "stop", "--timeout", "30", name])
+        # `-t`, not `--timeout`: the long form only exists on newer Docker (the
+        # Jetson node runs 27.5.1 and rejects it), and `-t` is accepted by both.
+        code, output = run(["docker", "stop", "-t", "30", name])
         if code != 0:
             errors.append(f"docker stop failed: {output.strip()}")
         else:
@@ -316,6 +329,11 @@ def main() -> int:
     parser.add_argument("--version", required=True)
     parser.add_argument("--expect-revision")
     parser.add_argument("--model", type=Path, help="model directory for the boot smoke")
+    parser.add_argument(
+        "--docker-runtime",
+        help="pass through to `docker run --runtime` (e.g. nvidia). Required on "
+        "Tegra/L4T, which rejects --gpus",
+    )
     parser.add_argument(
         "--gpus",
         help="pass through to `docker run --gpus` (e.g. all) so the boot smoke is "
@@ -335,7 +353,12 @@ def main() -> int:
             errors.append(f"--model {args.model} is not a directory")
         else:
             boot_errors = check_boot(
-                args.image, args.model.resolve(), args.port, args.boot_timeout, args.gpus
+                args.image,
+                args.model.resolve(),
+                args.port,
+                args.boot_timeout,
+                args.gpus,
+                args.docker_runtime,
             )
             errors += boot_errors
             runtime_verified = not boot_errors
@@ -349,7 +372,12 @@ def main() -> int:
     print(f"container image OK: {args.image} lane={args.lane} version={args.version}")
     print(f"  config, layout: verified")
     if runtime_verified:
-        where = f"on --gpus {args.gpus}" if args.gpus else "on CPU paths only (no --gpus)"
+        selectors = []
+        if args.docker_runtime:
+            selectors.append(f"--runtime {args.docker_runtime}")
+        if args.gpus:
+            selectors.append(f"--gpus {args.gpus}")
+        where = f"on {' '.join(selectors)}" if selectors else "on CPU paths only (no GPU)"
         print(
             f"  boot: /health 200, /version 200, declared healthcheck passed, clean "
             f"SIGTERM, {where}"