From 5059396bed3d14d060d07ca1790f7ada384c71dc Mon Sep 17 00:00:00 2001 From: Pedro Larroy Date: Fri, 14 Aug 2026 22:23:33 -0700 Subject: [PATCH 1/7] docs(skills): add build-deps setup skill Wraps the existing scripts/setup-dev.ps1 rather than reimplementing prerequisite detection. Documents the winget package IDs and the PATH refresh step needed before re-verifying, since freshly installed tools are not visible to an already-open shell. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/build-deps/SKILL.md | 66 ++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 .claude/skills/build-deps/SKILL.md diff --git a/.claude/skills/build-deps/SKILL.md b/.claude/skills/build-deps/SKILL.md new file mode 100644 index 000000000..ac4f812f9 --- /dev/null +++ b/.claude/skills/build-deps/SKILL.md @@ -0,0 +1,66 @@ +--- +name: build-deps +description: Install or repair the build prerequisites for this repo (.NET 10 SDK, Node.js/npm, Windows 10 SDK, Git, WebView2). Use when `build.ps1` reports "issue(s) found", when a build fails with a missing SDK/toolchain, or when setting up a fresh Windows checkout. +--- + +# Build dependencies + +This repo already ships the tooling. Do not write new detection or install logic. + +- `scripts\setup-dev.ps1` detects and installs prerequisites via winget, refreshes PATH, and trusts the checkout for GitVersion. +- `build.ps1 -CheckOnly` reports prerequisites only. This is what prints `N issue(s) found`. + +## Procedure + +1. **Diagnose** (never installs, never touches git config): + + ```powershell + .\scripts\setup-dev.ps1 -CheckOnly + ``` + +2. **Install what is missing.** Report the missing list to the user and confirm before installing, since winget changes machine state and may prompt for elevation. + + ```powershell + .\scripts\setup-dev.ps1 + ``` + + Run this in an elevated PowerShell if winget reports it needs admin. If winget itself is missing, tell the user to install "App Installer" from the Microsoft Store, then rerun. + +3. **Refresh PATH, then re-verify.** Newly installed tools are not on the PATH of any shell that was already open, including this session's shell. Pull the current machine and user PATH into the process before re-checking: + + ```powershell + $env:Path = @( + [Environment]::GetEnvironmentVariable("Path", "Machine"), + [Environment]::GetEnvironmentVariable("Path", "User") + ) -join ";" + .\scripts\setup-dev.ps1 -CheckOnly + ``` + + If a tool still is not found after this, the install needs a fresh terminal (or a reboot for the Windows SDK). Say so rather than looping on retries. + +4. **Confirm the build works** once the check is clean: + + ```powershell + .\build.ps1 + ``` + +## Package IDs (for reference and manual fallback) + +| Missing item | winget id | +| --- | --- | +| .NET SDK / .NET 10 SDK | `Microsoft.DotNet.SDK.10` | +| Node.js (and npm) | `OpenJS.NodeJS.LTS` | +| Windows 10 SDK | `Microsoft.WindowsSDK.10.0.26100` | +| Git | `Git.Git` | +| WebView2 Runtime | `Microsoft.EdgeWebView2Runtime` | + +Manual install: `winget install --id -e`. + +## Notes + +- The Windows SDK can also come from the Visual Studio Installer ("Desktop development with C++" or the standalone SDK component). Detection just looks for a versioned directory under `%ProgramFiles(x86)%\Windows Kits\10\Include`. +- Node.js is required even for the WinUI build: it runs `npm ci` to restore `@microsoft/mxc-sdk` and copy `wxc-exec.exe` into the output. +- Git is required at *build* time, not just for version control, because GitVersion reads repository metadata. `setup-dev.ps1` adds the checkout to `git config --global safe.directory`; pass `-NoTrustRepository` to skip that. +- The .NET version floor is pinned in `global.json` (`10.0.100`, `rollForward: latestFeature`). +- `setup-dev.ps1 -RunValidation` additionally runs the full build plus the shared and tray test projects required by `AGENTS.md` closeout. +- Keep this file free of em dashes: `scripts\validate-docs.ps1` runs during `build.ps1` and fails the build on them. From cbeb376db6131fc8432b920c7a1949b3ff4ba46e Mon Sep 17 00:00:00 2001 From: Pedro Larroy Date: Mon, 17 Aug 2026 16:43:26 -0700 Subject: [PATCH 2/7] feat(inference): add hardware probe, backend selection, and model catalog First phase of local llama.cpp inference: the pure decision layer that maps a host's hardware onto a llama.cpp build and a model it can run. No download, process, or UI code yet. New in src/OpenClaw.Shared/Inference/: - HardwareProbe / NvidiaSmiParser / HostHardwareInfo detect CPU architecture, installed RAM, and graphics adapters. NVIDIA VRAM and CUDA version come from nvidia-smi via the injectable ICommandRunner seam, so tests need no GPU. The probe never throws; every failure degrades to unknown so an unclassifiable host lands on the CPU build. - LlamaBackendCatalog pins llama.cpp release b10472 and its six Windows variants (CUDA 12.4/13.3 x64, CUDA 13.4 arm64, Vulkan x64, CPU x64/arm64), each with a verified SHA-256 and size. - BackendSelector maps hardware to a preferred variant plus an ordered fallback chain, since a CUDA build can fail to launch for reasons the probe cannot observe. - LocalModelCatalog holds Qwen3.6-35B-A3B, DeepSeek V4 Flash 0731, and a placeholder for the unreleased Qwen3.8-27B, each with its tuned llama-server run recipe stored as a structured argument list. - ModelRecommender assesses fit against VRAM then system RAM and returns no recommendation rather than guessing when nothing fits. Notable decisions: - Unknown CUDA version degrades to the CUDA 12 build. A CUDA 12 runtime works on newer drivers; a CUDA 13 runtime does not work on older ones. - Win32_VideoController.AdapterRAM is never used as a VRAM number. It is a 32-bit field that wraps above 4 GB, exactly the range that matters, so the non-NVIDIA fallback reports vendor and name only. - Vulkan is preferred only when a Vulkan loader is present. Shipping a Vulkan build to a host without one turns "no acceleration" into "the server will not start". - DeepSeek V4 Flash is never auto-recommended. A 155 GB download must be an explicit choice. PhysicalMemoryProbe takes ownership of the GlobalMemoryStatusEx interop that DeviceStatusProvider held; DeviceStatusProvider now calls it, with identical behavior, so the two copies cannot drift. Asset integrity follows the existing audio-asset policy: an entry without a pinned hash is not downloadable. AssetHashPinningTests is extended to guard both new catalogs. GGUF hashes are the HuggingFace LFS object ids; llama.cpp hashes were computed from the downloaded archives with sizes cross-checked against the releases API. See docs/LOCAL_INFERENCE_ASSETS.md for provenance and its limits. Validation: ./build.ps1 (all projects, 49 docs), OpenClaw.Shared.Tests 3759 passed, OpenClaw.Tray.Tests 2469 passed. Co-Authored-By: Claude Opus 5 (1M context) --- docs/LOCAL_INFERENCE_ASSETS.md | 85 ++++++ docs/LOCAL_INFERENCE_PLAN.md | 272 ++++++++++++++++++ .../Inference/BackendSelector.cs | 159 ++++++++++ .../Inference/HardwareProbe.cs | 216 ++++++++++++++ .../Inference/HostHardwareInfo.cs | 126 ++++++++ .../Inference/LlamaBackendCatalog.cs | 183 ++++++++++++ .../Inference/LocalModelCatalog.cs | 192 +++++++++++++ .../Inference/ModelRecommender.cs | 173 +++++++++++ .../Inference/NvidiaSmiParser.cs | 139 +++++++++ .../Inference/PhysicalMemoryProbe.cs | 63 ++++ .../Services/DeviceStatusProvider.cs | 38 +-- .../AssetHashPinningTests.cs | 48 ++++ .../Inference/BackendSelectorTests.cs | 205 +++++++++++++ .../Inference/HardwareProbeTests.cs | 168 +++++++++++ .../Inference/ModelRecommenderTests.cs | 185 ++++++++++++ .../Inference/NvidiaSmiParserTests.cs | 107 +++++++ 16 files changed, 2326 insertions(+), 33 deletions(-) create mode 100644 docs/LOCAL_INFERENCE_ASSETS.md create mode 100644 docs/LOCAL_INFERENCE_PLAN.md create mode 100644 src/OpenClaw.Shared/Inference/BackendSelector.cs create mode 100644 src/OpenClaw.Shared/Inference/HardwareProbe.cs create mode 100644 src/OpenClaw.Shared/Inference/HostHardwareInfo.cs create mode 100644 src/OpenClaw.Shared/Inference/LlamaBackendCatalog.cs create mode 100644 src/OpenClaw.Shared/Inference/LocalModelCatalog.cs create mode 100644 src/OpenClaw.Shared/Inference/ModelRecommender.cs create mode 100644 src/OpenClaw.Shared/Inference/NvidiaSmiParser.cs create mode 100644 src/OpenClaw.Shared/Inference/PhysicalMemoryProbe.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/BackendSelectorTests.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/HardwareProbeTests.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/ModelRecommenderTests.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/NvidiaSmiParserTests.cs diff --git a/docs/LOCAL_INFERENCE_ASSETS.md b/docs/LOCAL_INFERENCE_ASSETS.md new file mode 100644 index 000000000..88160efb7 --- /dev/null +++ b/docs/LOCAL_INFERENCE_ASSETS.md @@ -0,0 +1,85 @@ +# Local Inference Asset Integrity + +Local inference downloads two kinds of executable-adjacent assets at runtime: a +prebuilt `llama.cpp` server for the detected hardware, and GGUF model weights. +Both run on the user's machine with the user's privileges, so both follow the +same fail-closed rules as the audio assets described in +[AUDIO_MODEL_ASSETS.md](AUDIO_MODEL_ASSETS.md). + +## Authoritative catalogs + +| Asset | Source of truth | Runtime storage | +| --- | --- | --- | +| llama.cpp Windows builds | `LlamaBackendCatalog.Variants` | `\llama\runtimes\\` | +| GGUF checkpoints | `LocalModelCatalog.Models` | `\llama\models\\` | + +The source catalogs hold the download URL, pinned SHA-256, and exact size. Do +not duplicate those values here. + +## Rules + +1. The `llama.cpp` release tag is pinned in `LlamaBackendCatalog.ReleaseTag`. + Only the backend *variant* is chosen at runtime, from detected hardware. We + do not resolve "latest" at runtime: an unpinned binary cannot be + integrity-checked, and an upstream flag change would silently break the + per-model run recipes. +2. Every asset must carry a lowercase 64-character SHA-256 and an HTTPS URL. + An entry missing either is not downloadable + (`LlamaBackendVariant.IsDownloadable` / `LocalModelInfo.IsDownloadable` are + false) and the runtime refuses to fetch it. +3. Downloads stage to a temporary file, verify the hash, and only then move or + extract. A mismatch deletes the partial file and surfaces an error. +4. Archive extraction rejects any entry whose resolved path escapes the + destination directory. + +## Provenance + +**GGUF checkpoints.** HuggingFace publishes each LFS object's id, which is the +file's SHA-256. The catalog hashes are those published values, read from +`https://huggingface.co/api/models//tree/`. Re-read that endpoint to +re-verify rather than trusting a locally computed hash of a file you already +downloaded through the same channel. + +**llama.cpp builds.** GitHub does not publish release-asset hashes, so these +must be computed from the downloaded archive: + +```powershell +Get-FileHash .\llama--bin-win-cuda-12.4-x64.zip -Algorithm SHA256 +``` + +Record the release tag, the date, and who verified it in the change description. + +**Current pinning.** Release `b10472`, all nine Windows assets, verified +2026-08-17. Each archive was downloaded from the release URL, its byte length +cross-checked against the size the GitHub releases API reports for that asset, +and its SHA-256 computed from the downloaded bytes. Both the hash and the size +are recorded in `LlamaBackendCatalog`, so a future re-verification that produces +a different length fails before the hash comparison. + +Note the limit of that check: the API size and the archive come from the same +origin, so this establishes that the bytes we hashed are the bytes GitHub serves +for that release, not that the release itself is authentic. Independent +provenance for upstream binaries would require a signed upstream manifest, which +llama.cpp does not currently publish. + +## Custom local builds + +A user may point the app at their own `llama-server.exe` via the custom runtime +path setting. That path bypasses the catalog and the hash check entirely, by +design: the binary is the user's own. The UI must show an explicit "custom build, +not verified" state whenever it is in use, so the bypass is never silent. + +## Bumping the pinned release + +1. Download every Windows asset listed in `LlamaBackendCatalog.Variants` for the + new tag. +2. Compute and record each SHA-256. +3. Update `ReleaseTag` and all hashes in one commit. +4. Re-verify the run recipes in `LocalModelCatalog` still parse against the new + build. Speculative-decoding flags such as `--spec-type` are the ones most + likely to change. +5. Run `dotnet test .\tests\OpenClaw.Shared.Tests\OpenClaw.Shared.Tests.csproj --filter Inference`. +6. Launch one real model end to end and confirm a completion. + +Re-verify every shipped asset hash before each public release and record the +evidence for release review. diff --git a/docs/LOCAL_INFERENCE_PLAN.md b/docs/LOCAL_INFERENCE_PLAN.md new file mode 100644 index 000000000..dfee0ae26 --- /dev/null +++ b/docs/LOCAL_INFERENCE_PLAN.md @@ -0,0 +1,272 @@ +# Local Inference via llama.cpp: Implementation Plan + +Working plan for running models locally on the Windows host. Status is tracked +inline; update it as phases land. + +| Phase | Scope | Status | +| --- | --- | --- | +| 1 | Hardware probe, backend selection, model recommender | Landed | +| 2 | Runtime and GGUF download managers | Not started (catalog hashes pinned, unblocked) | +| 3 | Server process and settings UI | Not started | +| 4 | Gateway provider registration | Not started (blocked on live schema) | +| 5 | Optional `localinference.status` node capability | Not started | + +## Context + +Every model the Companion can talk to is owned by the gateway: the tray calls +`models.list` and renders whatever the gateway reports +(`src/OpenClaw.Chat/ChatModelChoice.cs`). There is no way to run a model on the +user's own machine, so a workstation with a large NVIDIA GPU sits idle while +every turn goes to a remote provider. + +This work makes the Windows app detect the host's hardware (NVIDIA GPU plus +VRAM, system RAM, CPU architecture), download a matching prebuilt `llama.cpp` +server, download a GGUF checkpoint the machine can hold, launch `llama-server` +with the checkpoint's tuned run recipe, and register the resulting +OpenAI-compatible endpoint with the gateway so the models appear in the normal +chat model picker. + +Two existing subsystems are reused rather than reinvented: the hash-pinned, +single-flight, fail-closed asset downloader +(`src/OpenClaw.Shared/Audio/WhisperModelManager.cs`, `SingleFlightDownload.cs`, +and `PiperVoiceManager.cs` for the archive case) and the safe whole-config patch +builder (`src/OpenClaw.Shared/ChannelConfigPatchBuilder.cs`). + +Decisions taken with the maintainer: + +- The endpoint is auto-registered in the gateway config so models appear in the + existing picker. +- Backends: CUDA (x64 and arm64), Vulkan, CPU. No ROCm, SYCL, OpenVINO, or + OpenCL-Adreno. +- llama.cpp binaries come from a pinned release tag with pinned per-asset + SHA-256; only the variant is chosen at runtime. A user-supplied custom local + build is also supported and bypasses download entirely. +- DeepSeek V4 Flash is in the catalog but gated: never auto-recommended, and its + roughly 155 GB download is confirmed explicitly. + +## Architecture + +New pure code lives in `src/OpenClaw.Shared/Inference/`, with a thin tray-side +service and settings page on top. Nothing is added to `App.xaml.cs` or +`ConnectionPage.xaml.cs` beyond construction and wiring; both are active +god-file reduction targets per `ARCHITECTURE.md`. + +``` +HardwareProbe ──► BackendSelector ──► LlamaRuntimeManager (download + extract) + │ │ + └──────► ModelRecommender ──► GgufModelManager (download GGUF shards) + │ + LlamaServerProcess (spawn, health, shutdown) + │ + LocalInferenceProviderRegistrar (config.patch) +``` + +## Phase 1: probe, selection, recommender (landed) + +Pure, testable, no UI and no network. + +| File | Role | +| --- | --- | +| `Inference/HostHardwareInfo.cs` | Hardware snapshot. `TotalNvidiaVramBytes` sums across adapters because llama.cpp's default `--split-mode layer` spreads a model over every visible device. | +| `Inference/PhysicalMemoryProbe.cs` | Single owner of the `GlobalMemoryStatusEx` interop, lifted out of `DeviceStatusProvider`, which now calls it. | +| `Inference/NvidiaSmiParser.cs` | Pure parser for the `nvidia-smi` CSV query and version banner, plus vendor classification. | +| `Inference/HardwareProbe.cs` | Orchestrates detection, caches, exposes `RefreshAsync`. | +| `Inference/LlamaBackendCatalog.cs` | Pinned release tag and the six Windows variants. | +| `Inference/BackendSelector.cs` | Hardware to backend plan with an ordered fallback chain. | +| `Inference/LocalModelCatalog.cs` | The three models with sizes, hashes, and run recipes. | +| `Inference/ModelRecommender.cs` | Pure fit assessment and recommendation. | + +Decisions worth preserving: + +- **The probe never throws.** Every source is best-effort and every failure + degrades to null or empty, so an unclassifiable host lands on the CPU backend + instead of breaking the settings page. +- **Unknown CUDA version degrades to the CUDA 12 build.** A CUDA 12 runtime + works on newer drivers; a CUDA 13 runtime does not work on older ones, so the + unknown case must degrade downward. +- **`Win32_VideoController.AdapterRAM` is never used as a VRAM number.** It is a + 32-bit field that wraps above 4 GB, which is exactly the range that matters. + Only `nvidia-smi` populates a size; the platform fallback supplies vendor and + name only. That fallback is injected as a delegate from the tray because + reading the display-adapter registry needs a Windows-targeted TFM and + `OpenClaw.Shared` targets plain `net10.0`. +- **Vulkan is preferred only when `vulkan-1.dll` is present.** Shipping a Vulkan + build to a host without a loader turns "no acceleration" into "the server will + not start", which is strictly worse. +- **An unclassified adapter is not treated as Vulkan-capable.** Seeing an + adapter we cannot name is not evidence that a Vulkan build will drive it. + +### Backend selection matrix + +Pinned release: `b10472`. + +| Condition | Assets | +| --- | --- | +| NVIDIA, x64, CUDA 13 or newer | `llama-b10472-bin-win-cuda-13.3-x64.zip` plus `cudart-llama-bin-win-cuda-13.3-x64.zip` | +| NVIDIA, x64, CUDA 12.x or unknown | `llama-b10472-bin-win-cuda-12.4-x64.zip` plus `cudart-llama-bin-win-cuda-12.4-x64.zip` | +| NVIDIA, arm64 | `llama-b10472-bin-win-cuda-13.4-arm64.zip` plus `cudart-llama-bin-win-cuda-13.4-arm64.zip` | +| Non-NVIDIA adapter, x64, Vulkan loader present | `llama-b10472-bin-win-vulkan-x64.zip` | +| Otherwise | `llama-b10472-bin-win-cpu-x64.zip` or `llama-b10472-bin-win-cpu-arm64.zip` | + +CUDA variants need two archives extracted into the same directory. A missing +`cudart` produces a missing-DLL startup failure rather than anything +diagnostic, so the pairing is asserted structurally in tests. + +### Model catalog + +| Model | Size | Status | +| --- | --- | --- | +| Qwen3.6-35B-A3B UD-Q4_K_M | 22,663,387,424 bytes, single file | Default recommendation | +| Qwen3.8-27B | Unpublished | Entry present with no shards, so not downloadable | +| DeepSeek V4 Flash 0731 UD-Q4_K_XL | About 155 GB across 5 shards | Gated, never auto-recommended | + +Shard hashes are the HuggingFace LFS object ids, which are the files' SHA-256. +Run recipes are stored as structured argument lists and deliberately exclude +`-m`, `--host`, and `--port`, which the process launcher owns; a test enforces +that separation. + +## Phase 2: download managers (blocked on hash pinning) + +`Inference/LlamaRuntimeManager.cs` and `Inference/GgufModelManager.cs`, modelled +on `PiperVoiceManager` and `WhisperModelManager`: + +- Per-key single flight via the existing `SingleFlightDownload.RunAsync`. +- Stage to a `.tmp` file, verify SHA-256, and only then move or extract. +- Delete the partial file on any failure. +- Zip extraction rejects entries whose resolved path escapes the destination. +- Runtimes land in `\llama\runtimes\\`, models in + `\llama\models\\`. +- GGUF downloads need aggregate cross-shard progress, a free-disk-space + precheck, and `Range`-based resume. Restarting a 50 GB shard from zero after a + dropped connection is not acceptable. + +**Custom local build.** When `LocalInferenceCustomRuntimePath` is set it wins +over the catalog: validate the path, skip download and hashing entirely, and +show an explicit "custom build, not verified" state so the bypass is never +silent. + +GitHub does not publish release-asset hashes, so all nine `b10472` archives were +downloaded, size-checked against the releases API, and hashed on 2026-08-17. +Those values are pinned in `LlamaBackendCatalog` and guarded by +`AssetHashPinningTests`, so this phase is unblocked. See +`LOCAL_INFERENCE_ASSETS.md` for the provenance and its limits. + +## Phase 3: server process and UI + +`OpenClawTray.Services.LlamaServerProcess` spawns `llama-server.exe` with +`--port

--host 127.0.0.1 -m ` plus the recipe args. + +- Port comes from a free-port scan; reuse `PortDiagnosticsService` and + `WindowsTcpListenerSnapshot` for conflict reporting. +- Bind to `127.0.0.1` by default. Binding beyond loopback exposes an + unauthenticated inference endpoint to the LAN and must be an explicit, warned + opt-in. +- Health: poll `GET /health` until ready or timeout, and surface the stderr tail + on failure. A recipe flag an older build does not know fails here, and the + user needs to see why. +- Assign the child to a Win32 job object with + `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` so a tray crash cannot orphan a process + holding tens of gigabytes of VRAM. Graceful stop via `AppShutdownCoordinator`. + +`Pages/LocalInferencePage.xaml{,.cs}` follows `VoiceSettingsPage`, the closest +analogue: catalog combo, download button, progress bar driven by +`IProgress<(long downloaded, long total)>`, page-held `CancellationTokenSource`, +status text from resources. Register it in `Presentation/HubPageRegistry.cs` +(enum value, tag string, type map) and wire `Initialize` in +`Windows/HubWindow.xaml.cs`. All strings go through `LocalizationHelper` and +`Strings/en-us/Resources.resw`, with no em dashes per `AGENTS.md`. + +Settings added to `SettingsData`: `LocalInferenceEnabled`, +`LocalInferenceModelId`, `LocalInferenceBackendOverride`, +`LocalInferenceCustomRuntimePath`, `LocalInferencePort`, +`LocalInferenceAutoStart`, `LocalInferenceRegisterWithGateway`, +`LocalInferenceBindBeyondLoopback`. Change effects route through +`SettingsChangeCoordinator` and `SettingsChangeEffects`. + +## Phase 4: gateway registration (blocked on live schema) + +Once the server is healthy, patch the gateway config to add an +OpenAI-compatible provider pointing at it, via +`IOperatorGatewayClient.PatchConfigDetailedAsync(fullConfig, baseHash)`. + +Reuse `ChannelConfigPatchBuilder`'s `SetNestedValue` and +`FindRedactionSentinel`, which are `internal static` in the same assembly, and +apply the same **redaction-sentinel safety rail**: if the cached config holds +`[REDACTED]` or `***` outside the paths being written, refuse the patch and +route the user to the Config page. Silently clobbering real API keys with +redaction placeholders while enabling local inference would be a severe +regression. + +**Blocker.** This repo does not contain the gateway's config schema; +`ConfigPage` fetches it at runtime from `config.get`. The dot-path and shape for +registering an OpenAI-compatible provider is therefore not verifiable from this +checkout. First step of this phase: connect to a real gateway, inspect the +schema the Config page renders, and write the builder against the real shape. Do +not guess it. + +**WSL reachability.** When the gateway runs in WSL, `127.0.0.1` inside the +distro is not the Windows host. `WINDOWS_NODE_ARCHITECTURE.md` records this: +NAT-mode WSL2 reaches the host via `$(hostname).local` or +`host.docker.internal`, while mirrored networking can use `localhost`. Use +`GatewayHostAccessClassifier` and `GatewayRecord` to detect a WSL-managed +gateway and resolve the base URL accordingly. NAT mode requires binding beyond +loopback, which must be an explicit consent step. + +## Phase 5: optional node capability + +A `localinference.status` command. Per `AGENTS.md`, any new Windows node call +must be registered in the capability registry, added to +`McpToolBridge.CommandDescriptions`, documented in +`src/OpenClaw.WinNode.Cli/skill.md`, and covered by +`OpenClaw.WinNode.Cli.Tests`. That is a real slice of work, not a footnote. + +## Docs + +- `LOCAL_INFERENCE_ASSETS.md` covers the fail-closed download rules, hash + provenance, and the release-bump procedure. Added. +- `ARCHITECTURE.md` needs ownership rows for the new services once they exist. +- A user-facing `LOCAL_INFERENCE.md` should cover the hardware matrix, the model + catalog, the custom-build escape hatch, and the WSL caveat. + +## Verification + +Unit tests in `tests/OpenClaw.Shared.Tests/Inference/`: + +- Backend selection across architecture, vendor, and CUDA-version tuples, plus + the CUDA-and-cudart pairing invariant. +- Model fit across host shapes: large-VRAM workstation, small GPU with large + RAM, no GPU, small laptop, undetectable hardware; DeepSeek never + auto-selected; the unpublished checkpoint reported as pending. +- Catalog integrity: HTTPS URLs, pinned lowercase 64-character SHA-256 on + everything downloadable, unique path-safe ids and runtime keys, shard + ordering, and the reserved-argument separation. +- nvidia-smi parsing against captured real output, including the multi-GPU, + `[N/A]` memory, and missing-banner cases. + +Still to add in later phases: download managers against an `HttpMessageHandler` +fake (corrupt body rejected, `.tmp` deleted, nothing at the final path, +concurrent callers coalesced), zip traversal rejection, and the provider patch +builder preserving unrelated config while blocking on a redaction sentinel. + +Required repo validation per `AGENTS.md`: `./build.ps1`, then the Shared and +Tray test projects. In this linked worktree, set `OPENCLAW_REPO_ROOT` first or +`ReadmeValidationTests` fails on repo-root discovery. + +Real behavior proof, which CI cannot establish: + +1. On an NVIDIA host, screenshot the detected hardware and confirm it matches + `nvidia-smi`. +2. Download the CUDA runtime, confirm hash verification passes, and run + `llama-server.exe --version` from the extracted directory. +3. Download Qwen3.6-35B-A3B, start the server, and prove readiness with + `GET /health` and a real `POST /v1/chat/completions`. +4. Corrupt a downloaded GGUF and confirm the app refuses it, deletes the partial + file, and shows a clear error. +5. Kill the tray and confirm `llama-server.exe` dies with it. +6. With registration enabled, confirm the local models appear in the chat model + picker and that a turn reaches the local server. +7. Cover both mirrored and NAT WSL networking, or record which was not covered. + +Not verifiable in the current environment and explicitly deferred: the gateway +provider config schema, the Vulkan and arm64 CUDA paths, and the DeepSeek path. +State these as blockers in the PR rather than implying coverage. diff --git a/src/OpenClaw.Shared/Inference/BackendSelector.cs b/src/OpenClaw.Shared/Inference/BackendSelector.cs new file mode 100644 index 000000000..7c50ab8b6 --- /dev/null +++ b/src/OpenClaw.Shared/Inference/BackendSelector.cs @@ -0,0 +1,159 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Runtime.InteropServices; + +namespace OpenClaw.Shared.Inference; + +///

+/// The backend plan for a host: the preferred variant plus the ordered +/// fallbacks to try if it fails to launch. +/// +/// +/// Best variant for this hardware, or null when the pinned release has no build +/// for this architecture at all. +/// +/// +/// Variants to try, in order, after fails. A CUDA build +/// can fail at launch for reasons the probe cannot see (driver too old for the +/// CUDA runtime, GPU in TCC mode, VRAM already claimed), so having a real +/// degradation path matters more here than in most download flows. +/// +/// Short PII-free explanation of the choice, shown in the UI. +public sealed record BackendPlan( + LlamaBackendVariant? Preferred, + IReadOnlyList Fallbacks, + string Reason) +{ + /// Preferred first, then each fallback. Empty when nothing is available. + public IEnumerable InPreferenceOrder + { + get + { + if (Preferred is not null) yield return Preferred; + foreach (var fallback in Fallbacks) yield return fallback; + } + } +} + +/// +/// Maps detected hardware onto a llama.cpp backend variant from +/// . +/// +/// Pure and total: no I/O, no throwing, and every input shape produces a +/// plan (possibly an empty one). This is the piece that decides whether a user +/// gets GPU inference at all, so it is exhaustively unit tested rather than +/// discovered in the field. +/// +public static class BackendSelector +{ + /// + /// Choose the backend plan for . + /// + /// Probe result. yields the CPU plan. + /// + /// Explicit user choice from settings. When it names a variant that exists for + /// this architecture it wins outright, and the auto-selected plan becomes the + /// fallback chain. An override naming a build this release does not have for + /// this architecture is ignored rather than honored into a dead end. + /// + public static BackendPlan Select(HostHardwareInfo hardware, LlamaBackend? userOverride = null) + { + ArgumentNullException.ThrowIfNull(hardware); + + var arch = NormalizeArchitecture(hardware.CpuArchitecture); + var auto = SelectAutomatic(hardware, arch); + + if (userOverride is not { } requested) return auto; + + var overrideVariant = LlamaBackendCatalog.Find(requested, arch); + if (overrideVariant is null) return auto with { Reason = $"{auto.Reason} (override {requested} unavailable for {arch})" }; + + var fallbacks = auto.InPreferenceOrder + .Where(v => !ReferenceEquals(v, overrideVariant)) + .ToArray(); + + return new BackendPlan(overrideVariant, fallbacks, $"Backend overridden to {overrideVariant.DisplayName}."); + } + + private static BackendPlan SelectAutomatic(HostHardwareInfo hardware, Architecture arch) + { + var cpu = LlamaBackendCatalog.Find(LlamaBackend.Cpu, arch); + var vulkan = LlamaBackendCatalog.Find(LlamaBackend.Vulkan, arch); + + if (hardware.HasNvidiaGpu) + { + // CUDA 13 builds require a driver new enough for the CUDA 13 runtime. + // When nvidia-smi did not report a version we take the CUDA 12 build: + // a CUDA 12 runtime is forward-compatible with newer drivers, while the + // reverse is not true, so the unknown case must degrade downward. + var wantsCuda13 = hardware.MaxCudaMajorVersion is >= 13; + var primary = LlamaBackendCatalog.Find(wantsCuda13 ? LlamaBackend.Cuda13 : LlamaBackend.Cuda12, arch); + var secondary = LlamaBackendCatalog.Find(wantsCuda13 ? LlamaBackend.Cuda12 : LlamaBackend.Cuda13, arch); + + var chosen = primary ?? secondary; + if (chosen is not null) + { + var chain = new List(); + if (secondary is not null && !ReferenceEquals(secondary, chosen)) chain.Add(secondary); + if (vulkan is not null && hardware.VulkanAvailable) chain.Add(vulkan); + if (cpu is not null) chain.Add(cpu); + + var cudaLabel = hardware.MaxCudaMajorVersion is { } major ? $"CUDA {major}.x" : "CUDA version unknown"; + return new BackendPlan( + chosen, + chain, + $"NVIDIA GPU detected ({cudaLabel}); using {chosen.DisplayName}."); + } + + // NVIDIA hardware but no CUDA build for this architecture. + return BuildNonCudaPlan(hardware, vulkan, cpu, "NVIDIA GPU detected but this release has no CUDA build for " + arch + "."); + } + + if (hardware.HasNonNvidiaGpu) + { + return BuildNonCudaPlan( + hardware, + vulkan, + cpu, + hardware.VulkanAvailable + ? "Non-NVIDIA GPU detected with a Vulkan loader present." + : "Non-NVIDIA GPU detected but no Vulkan loader is installed."); + } + + return cpu is null + ? new BackendPlan(null, Array.Empty(), $"No llama.cpp build is available for {arch}.") + : new BackendPlan(cpu, Array.Empty(), "No supported GPU detected; using the CPU build."); + } + + private static BackendPlan BuildNonCudaPlan( + HostHardwareInfo hardware, + LlamaBackendVariant? vulkan, + LlamaBackendVariant? cpu, + string reason) + { + // Vulkan is only preferred when a loader is actually installed. Shipping a + // Vulkan build to a machine without vulkan-1.dll just moves the failure + // from "no GPU acceleration" to "server will not start". + if (vulkan is not null && hardware.VulkanAvailable) + { + IReadOnlyList fallbacks = cpu is null ? [] : [cpu]; + return new BackendPlan(vulkan, fallbacks, $"{reason} Using {vulkan.DisplayName}."); + } + + return cpu is null + ? new BackendPlan(null, Array.Empty(), reason) + : new BackendPlan(cpu, Array.Empty(), $"{reason} Falling back to the CPU build."); + } + + /// + /// Collapse the architectures we do not ship builds for onto the ones we do. + /// x86 hosts run the x64 build under WOW64 emulation rather than getting + /// nothing, and any unrecognized architecture is treated as x64. + /// + private static Architecture NormalizeArchitecture(Architecture architecture) => architecture switch + { + Architecture.Arm64 => Architecture.Arm64, + _ => Architecture.X64, + }; +} diff --git a/src/OpenClaw.Shared/Inference/HardwareProbe.cs b/src/OpenClaw.Shared/Inference/HardwareProbe.cs new file mode 100644 index 000000000..6e3e7992b --- /dev/null +++ b/src/OpenClaw.Shared/Inference/HardwareProbe.cs @@ -0,0 +1,216 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Runtime.InteropServices; +using System.Threading; +using System.Threading.Tasks; + +namespace OpenClaw.Shared.Inference; + +/// +/// Detects the inference-relevant hardware on this host: CPU architecture, +/// installed RAM, and graphics adapters (with NVIDIA VRAM and CUDA version). +/// +/// Contract: this probe never throws. Every source is best-effort and +/// every failure degrades to a null/empty field. A host we cannot classify must +/// end up on the CPU backend, not crash the settings page. +/// +/// NVIDIA detection goes through nvidia-smi, which is installed into +/// System32 by the display driver. It is the only source that reports a correct +/// VRAM size: Win32_VideoController.AdapterRAM is a signed 32-bit field +/// that wraps above 4 GB, which is exactly the range we care about. +/// +public sealed class HardwareProbe +{ + /// nvidia-smi is slow to start on some systems but never this slow. + private const int NvidiaSmiTimeoutMs = 10_000; + + private readonly ICommandRunner _commandRunner; + private readonly IOpenClawLogger _logger; + private readonly Func> _fallbackGpuEnumerator; + private readonly Func _vulkanLoaderProbe; + private readonly SemaphoreSlim _refreshGate = new(1, 1); + + private HostHardwareInfo? _cached; + + /// Used to invoke nvidia-smi. Injectable so tests need no GPU. + /// Diagnostics sink. + /// + /// Optional platform-specific adapter enumeration used when nvidia-smi + /// is absent. Lives outside this assembly because reading the display-adapter + /// registry class requires a Windows-targeted TFM. Defaults to "no adapters". + /// + /// + /// Optional override for Vulkan loader detection. Defaults to checking for + /// vulkan-1.dll in the system directory. + /// + public HardwareProbe( + ICommandRunner commandRunner, + IOpenClawLogger logger, + Func>? fallbackGpuEnumerator = null, + Func? vulkanLoaderProbe = null) + { + _commandRunner = commandRunner ?? throw new ArgumentNullException(nameof(commandRunner)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _fallbackGpuEnumerator = fallbackGpuEnumerator ?? (static () => Array.Empty()); + _vulkanLoaderProbe = vulkanLoaderProbe ?? DefaultVulkanLoaderProbe; + } + + /// + /// The most recent probe result, or null if has never + /// completed. Non-blocking; for UI that wants to render before the first probe. + /// + public HostHardwareInfo? Cached => _cached; + + /// + /// Returns the cached hardware snapshot, probing once on first call. + /// Concurrent callers share a single probe. + /// + public async Task GetAsync(CancellationToken cancellationToken = default) + { + if (_cached is { } cached) return cached; + return await RefreshAsync(cancellationToken).ConfigureAwait(false); + } + + /// + /// Re-runs detection and replaces the cache. Backs the settings page's + /// "Re-detect" action, which exists because a driver install or an eGPU + /// hotplug changes the answer without an app restart. + /// + public async Task RefreshAsync(CancellationToken cancellationToken = default) + { + await _refreshGate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + var info = await ProbeCoreAsync(cancellationToken).ConfigureAwait(false); + _cached = info; + return info; + } + finally + { + _refreshGate.Release(); + } + } + + private async Task ProbeCoreAsync(CancellationToken cancellationToken) + { + var memory = PhysicalMemoryProbe.TryRead(); + var gpus = await ProbeGpusAsync(cancellationToken).ConfigureAwait(false); + + bool vulkan; + // slopwatch-ignore: SW003 Probe is best-effort by contract; an unreadable system directory means "no Vulkan". + try { vulkan = _vulkanLoaderProbe(); } catch { vulkan = false; } + + var info = new HostHardwareInfo( + RuntimeInformation.OSArchitecture, + memory?.TotalBytes, + memory?.AvailableBytes, + gpus, + vulkan); + + _logger.Info( + $"[HardwareProbe] arch={info.CpuArchitecture} ram={FormatGib(info.TotalPhysicalMemoryBytes)} " + + $"gpus={info.Gpus.Count} nvidiaVram={FormatGib(info.TotalNvidiaVramBytes)} " + + $"cuda={info.MaxCudaMajorVersion?.ToString() ?? "unknown"} vulkan={vulkan}"); + + return info; + } + + private async Task> ProbeGpusAsync(CancellationToken cancellationToken) + { + // Pass 1: plain `nvidia-smi` for the CUDA version banner. Its absence is the + // normal, expected result on a machine with no NVIDIA driver, so a failure + // here is logged at debug volume, not as a warning. + var banner = await RunNvidiaSmiAsync(Array.Empty(), cancellationToken).ConfigureAwait(false); + var cudaMajor = NvidiaSmiParser.TryParseCudaMajorVersion(banner); + + // Pass 2: the machine-readable per-GPU query. + var query = await RunNvidiaSmiAsync(NvidiaSmiParser.QueryGpuArgs, cancellationToken).ConfigureAwait(false); + var nvidiaGpus = NvidiaSmiParser.ParseQueryGpu(query, cudaMajor); + + if (nvidiaGpus.Count > 0) return nvidiaGpus; + + // No NVIDIA driver. Fall back to platform adapter enumeration so we can + // still tell "AMD/Intel GPU present, try Vulkan" from "no GPU at all". + try + { + return _fallbackGpuEnumerator(); + } + catch (Exception ex) + { + _logger.Warn($"[HardwareProbe] Adapter fallback enumeration failed: {ex.Message}"); + return Array.Empty(); + } + } + + /// + /// Invoke nvidia-smi and return stdout, or null when it is missing or failed. + /// Tries PATH first, then the System32 copy the driver installs, because a + /// PATH-less service context is a real deployment shape. + /// + private async Task RunNvidiaSmiAsync(IReadOnlyList args, CancellationToken cancellationToken) + { + foreach (var executable in EnumerateNvidiaSmiCandidates()) + { + var argv = new List(args.Count + 1) { executable }; + argv.AddRange(args); + + CommandResult result; + try + { + result = await _commandRunner.RunAsync( + new CommandRequest { Argv = argv, TimeoutMs = NvidiaSmiTimeoutMs }, + cancellationToken).ConfigureAwait(false); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception ex) + { + _logger.Info($"[HardwareProbe] nvidia-smi ({executable}) not usable: {ex.Message}"); + continue; + } + + if (result.TimedOut) + { + _logger.Warn($"[HardwareProbe] nvidia-smi ({executable}) timed out after {NvidiaSmiTimeoutMs} ms"); + continue; + } + + if (result.ExitCode != 0) + { + _logger.Info($"[HardwareProbe] nvidia-smi ({executable}) exited {result.ExitCode}"); + continue; + } + + if (!string.IsNullOrWhiteSpace(result.Stdout)) return result.Stdout; + } + + return null; + } + + private static IEnumerable EnumerateNvidiaSmiCandidates() + { + yield return "nvidia-smi"; + + string? system32 = null; + // slopwatch-ignore: SW003 Best-effort path resolution; the PATH candidate above already covers the normal case. + try { system32 = Environment.SystemDirectory; } catch { /* ignore */ } + + if (!string.IsNullOrWhiteSpace(system32)) + yield return Path.Combine(system32, "nvidia-smi.exe"); + } + + private static bool DefaultVulkanLoaderProbe() + { + var system32 = Environment.SystemDirectory; + return !string.IsNullOrWhiteSpace(system32) + && File.Exists(Path.Combine(system32, "vulkan-1.dll")); + } + + private static string FormatGib(long? bytes) => + bytes is { } value && value > 0 + ? $"{value / (1024.0 * 1024 * 1024):F1}GiB" + : "unknown"; +} diff --git a/src/OpenClaw.Shared/Inference/HostHardwareInfo.cs b/src/OpenClaw.Shared/Inference/HostHardwareInfo.cs new file mode 100644 index 000000000..b06fb41fa --- /dev/null +++ b/src/OpenClaw.Shared/Inference/HostHardwareInfo.cs @@ -0,0 +1,126 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Runtime.InteropServices; + +namespace OpenClaw.Shared.Inference; + +/// +/// GPU vendor, as far as the probe could determine it. +/// means "we saw an adapter but could not classify it" and must be treated the +/// same as "no usable accelerator" by the backend selector. +/// +public enum GpuVendor +{ + Unknown = 0, + Nvidia = 1, + Amd = 2, + Intel = 3, + Other = 4, +} + +/// +/// One detected graphics adapter. +/// +/// Classified vendor. +/// Adapter name as reported by the source (e.g. "NVIDIA RTX 6000 Ada Generation"). +/// +/// Dedicated video memory in bytes, or null when unknown. Only ever populated +/// from a trustworthy source (nvidia-smi). WMI's Win32_VideoController.AdapterRAM +/// is a 32-bit field that wraps above 4 GB, so the WMI fallback deliberately +/// leaves this null rather than reporting a wrong number. +/// +/// Display driver version, when known. +/// +/// Major version of the CUDA runtime the driver supports, when known. Drives the +/// choice between the CUDA 12.x and CUDA 13.x llama.cpp builds. +/// +public sealed record GpuInfo( + GpuVendor Vendor, + string Name, + long? DedicatedMemoryBytes = null, + string? DriverVersion = null, + int? CudaMajorVersion = null); + +/// +/// Snapshot of the host's inference-relevant hardware. Every field is optional: +/// the probe never throws, and unknown values degrade to null so the backend +/// selector falls through to CPU rather than guessing. +/// +/// OS architecture (x64 / Arm64 in practice). +/// Installed system RAM, or null when the query failed. +/// Currently free system RAM, or null when the query failed. +/// All detected adapters, in the order the source reported them. +/// True when a Vulkan loader is present on the machine. +public sealed record HostHardwareInfo( + Architecture CpuArchitecture, + long? TotalPhysicalMemoryBytes, + long? AvailablePhysicalMemoryBytes, + IReadOnlyList Gpus, + bool VulkanAvailable) +{ + /// + /// The "we learned nothing" result. Used when every probe path failed; the + /// selector maps this to the CPU backend. + /// + public static HostHardwareInfo Unknown { get; } = new( + RuntimeInformation.OSArchitecture, + null, + null, + Array.Empty(), + false); + + /// All adapters classified as NVIDIA. + public IEnumerable NvidiaGpus => Gpus.Where(g => g.Vendor == GpuVendor.Nvidia); + + /// True when at least one NVIDIA adapter was detected. + public bool HasNvidiaGpu => Gpus.Any(g => g.Vendor == GpuVendor.Nvidia); + + /// + /// True when a non-NVIDIA adapter that a Vulkan build could drive was detected. + /// does not count: an unclassified adapter is + /// not evidence that a Vulkan build will work. + /// + public bool HasNonNvidiaGpu => + Gpus.Any(g => g.Vendor is GpuVendor.Amd or GpuVendor.Intel or GpuVendor.Other); + + /// + /// Combined dedicated VRAM across all NVIDIA adapters whose size is known, or + /// null when no NVIDIA adapter reported a size. llama.cpp's default + /// --split-mode layer spreads a model across every visible device, so + /// the sum (not the maximum) is the capacity that matters for model fit. + /// + public long? TotalNvidiaVramBytes + { + get + { + long total = 0; + var sawAny = false; + foreach (var gpu in NvidiaGpus) + { + if (gpu.DedicatedMemoryBytes is not { } bytes || bytes <= 0) continue; + total += bytes; + sawAny = true; + } + return sawAny ? total : null; + } + } + + /// + /// Highest CUDA major version reported by any NVIDIA adapter, or null when + /// unknown. Null must be treated as "assume the older CUDA build". + /// + public int? MaxCudaMajorVersion + { + get + { + int? best = null; + foreach (var gpu in NvidiaGpus) + { + if (gpu.CudaMajorVersion is not { } major) continue; + if (best is null || major > best) best = major; + } + return best; + } + } +} diff --git a/src/OpenClaw.Shared/Inference/LlamaBackendCatalog.cs b/src/OpenClaw.Shared/Inference/LlamaBackendCatalog.cs new file mode 100644 index 000000000..05ded1db0 --- /dev/null +++ b/src/OpenClaw.Shared/Inference/LlamaBackendCatalog.cs @@ -0,0 +1,183 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace OpenClaw.Shared.Inference; + +/// +/// A llama.cpp compute backend variant we ship. The variant is chosen at +/// runtime from detected hardware; the release it comes from is pinned. +/// +public enum LlamaBackend +{ + /// Portable CPU build. Always usable, always the last resort. + Cpu = 0, + /// NVIDIA CUDA 12.x build. Widest driver compatibility. + Cuda12 = 1, + /// NVIDIA CUDA 13.x build. + Cuda13 = 2, + /// Vendor-neutral Vulkan build, for AMD/Intel adapters. + Vulkan = 3, +} + +/// +/// One downloadable archive belonging to a backend variant. +/// +/// Release asset file name. +/// +/// Pinned lowercase hex SHA-256 of the archive. Null means the runtime must +/// refuse to download it, exactly as the audio catalogs behave. See +/// docs/LOCAL_INFERENCE_ASSETS.md. +/// +/// Size hint for the UI's progress display. 0 when unknown. +public sealed record LlamaBackendAsset( + string FileName, + string? Sha256, + long ApproximateSizeBytes = 0) +{ + /// Full HTTPS download URL, derived from the pinned release tag. + public string DownloadUrl => $"{LlamaBackendCatalog.ReleaseDownloadBase}/{FileName}"; +} + +/// +/// A backend variant plus every archive that has to be extracted for it to run. +/// +/// Which compute backend this entry provides. +/// CPU architecture the archive targets. +/// +/// All archives to extract into the same runtime directory, in extraction order. +/// The CUDA variants need two: the llama.cpp binaries plus the matching CUDA +/// runtime redistributable, without which llama-server.exe fails to start +/// with a missing-DLL error rather than anything diagnostic. +/// +/// Label for the backend override dropdown. +public sealed record LlamaBackendVariant( + LlamaBackend Backend, + System.Runtime.InteropServices.Architecture Architecture, + IReadOnlyList Assets, + string DisplayName) +{ + /// + /// Stable directory-safe key for this variant's extracted runtime, e.g. + /// b10472-cuda12-X64. Includes the release tag so a catalog bump + /// installs alongside the old runtime instead of half-overwriting it. + /// + public string RuntimeKey => + $"{LlamaBackendCatalog.ReleaseTag}-{Backend.ToString().ToLowerInvariant()}-{Architecture.ToString().ToLowerInvariant()}"; + + /// + /// True when every asset carries a pinned hash, so the download manager is + /// allowed to fetch it. False entries are visible in the UI but not installable. + /// + public bool IsDownloadable => Assets.Count > 0 && Assets.All(a => !string.IsNullOrWhiteSpace(a.Sha256)); + + /// Combined size hint across all archives. + public long ApproximateSizeBytes => Assets.Sum(a => a.ApproximateSizeBytes); +} + +/// +/// The pinned llama.cpp release and the Windows backend variants we ship from it. +/// +/// Pinning policy. The release tag and every asset hash are compiled +/// into the signed application; only the variant is decided at runtime, +/// from . We deliberately do not resolve "latest" at +/// runtime: an unpinned binary cannot be integrity-checked, and upstream flag +/// changes would silently break the per-model run recipes. +/// +/// Bumping the release. Download each asset, compute +/// Get-FileHash -Algorithm SHA256, update and every +/// hash below in the same commit, and re-verify the run recipes still parse against +/// the new build. See docs/LOCAL_INFERENCE_ASSETS.md. +/// +public static class LlamaBackendCatalog +{ + /// Pinned upstream release tag. + public const string ReleaseTag = "b10472"; + + /// Base URL for this release's assets. + public const string ReleaseDownloadBase = + "https://github.com/ggml-org/llama.cpp/releases/download/" + ReleaseTag; + + private const System.Runtime.InteropServices.Architecture X64 = + System.Runtime.InteropServices.Architecture.X64; + private const System.Runtime.InteropServices.Architecture Arm64 = + System.Runtime.InteropServices.Architecture.Arm64; + + // SECURITY - pinned SHA-256 hashes (lowercase hex), computed from the + // archives published at the b10472 release and cross-checked against the + // sizes the GitHub releases API reports for each asset. Verified 2026-08-17. + // Downloads with a different hash are rejected and the partial file is + // deleted. Re-verify before every public release and record the provenance. + // See docs/LOCAL_INFERENCE_ASSETS.md. + public static readonly IReadOnlyList Variants = + [ + new(LlamaBackend.Cuda13, X64, + [ + new($"llama-{ReleaseTag}-bin-win-cuda-13.3-x64.zip", + "ce7ca842c1400a85457e6c7ce844f21e52f187e6f0364b7daf3d2fd1ccf6db3b", + 146_707_438), + new("cudart-llama-bin-win-cuda-13.3-x64.zip", + "1462a050eb4c684921ba51dcc4cc488a036674c3e73e9945ee705b854808d03e", + 390_970_417), + ], + "NVIDIA CUDA 13.3 (x64)"), + + new(LlamaBackend.Cuda12, X64, + [ + new($"llama-{ReleaseTag}-bin-win-cuda-12.4-x64.zip", + "aadc171ddb4ed1822bc1730bff447068b529718cef886437866e3bd536eda143", + 250_798_945), + new("cudart-llama-bin-win-cuda-12.4-x64.zip", + "8c79a9b226de4b3cacfd1f83d24f962d0773be79f1e7b75c6af4ded7e32ae1d6", + 391_443_627), + ], + "NVIDIA CUDA 12.4 (x64)"), + + new(LlamaBackend.Cuda13, Arm64, + [ + new($"llama-{ReleaseTag}-bin-win-cuda-13.4-arm64.zip", + "1ce04088513dcbea5c172d529032cf0eb405c2bc74df761921bfb7bef3fa28b4", + 140_341_800), + new("cudart-llama-bin-win-cuda-13.4-arm64.zip", + "5a40dc7c5fa3d0a80ceeba4f16f9e8d25d87bcf1399c9233588953c43436c33c", + 153_318_797), + ], + "NVIDIA CUDA 13.4 (ARM64)"), + + new(LlamaBackend.Vulkan, X64, + [ + new($"llama-{ReleaseTag}-bin-win-vulkan-x64.zip", + "2104e62c7e5237f2190240cdc987d8c3946a77051f696771d03b8d762a9d2fae", + 34_813_404), + ], + "Vulkan (x64)"), + + new(LlamaBackend.Cpu, X64, + [ + new($"llama-{ReleaseTag}-bin-win-cpu-x64.zip", + "ef495329c85c171991972fd3226a179c1900368cab66e2ebba8b21a7471a74e5", + 18_470_168), + ], + "CPU (x64)"), + + new(LlamaBackend.Cpu, Arm64, + [ + new($"llama-{ReleaseTag}-bin-win-cpu-arm64.zip", + "6de7a00ad19fa3c5a772575d8a4fc75b265fcc2b875a2206b437af7d925b29b1", + 12_229_653), + ], + "CPU (ARM64)"), + ]; + + /// The server executable inside every extracted archive. + public const string ServerExecutableName = "llama-server.exe"; + + /// + /// Look up a variant by backend and architecture, or null when this release + /// has no such build (e.g. Vulkan on ARM64). + /// + public static LlamaBackendVariant? Find( + LlamaBackend backend, + System.Runtime.InteropServices.Architecture architecture) => + Variants.FirstOrDefault(v => v.Backend == backend && v.Architecture == architecture); +} diff --git a/src/OpenClaw.Shared/Inference/LocalModelCatalog.cs b/src/OpenClaw.Shared/Inference/LocalModelCatalog.cs new file mode 100644 index 000000000..20698092c --- /dev/null +++ b/src/OpenClaw.Shared/Inference/LocalModelCatalog.cs @@ -0,0 +1,192 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace OpenClaw.Shared.Inference; + +/// +/// One GGUF file belonging to a model. Large models are published as numbered +/// shards; llama.cpp is pointed at shard 1 and loads the rest itself, but every +/// shard has to be present in the same directory first. +/// +/// On-disk name. Must match the upstream name for shard discovery to work. +/// HTTPS URL of the file. +/// Exact size in bytes, from the HuggingFace file listing. +/// +/// Pinned lowercase hex SHA-256. HuggingFace reports this as the LFS object id, +/// so the values here are the upstream-published hashes rather than ones we +/// computed locally. Null means the download manager must refuse the file. +/// +public sealed record GgufShard( + string FileName, + string DownloadUrl, + long SizeBytes, + string? Sha256); + +/// +/// A model we offer for local inference, with the run recipe it needs. +/// +/// Stable catalog id used in settings and on disk. +/// Human-facing name. +/// GGUF files, shard 1 first. Empty for entries that are not published yet. +/// +/// Model-specific llama-server arguments. Excludes -m, --host, +/// and --port, which the process launcher owns. These are sampler and +/// speculative-decoding settings tuned per checkpoint; changing them changes +/// output quality, so they live with the checkpoint rather than in the launcher. +/// +/// +/// Memory needed for a comfortable run: model weights plus KV cache and runtime +/// overhead. Compared against VRAM first, then system RAM. +/// +/// +/// True for models whose download is large enough that starting it by accident is +/// a real harm. Never auto-recommended; the UI must confirm the size explicitly. +/// +/// Short PII-free note shown under the entry in the picker. +public sealed record LocalModelInfo( + string Id, + string DisplayName, + IReadOnlyList Shards, + IReadOnlyList RecipeArgs, + long MinimumRecommendedMemoryBytes, + bool RequiresConfirmation = false, + string? Notes = null) +{ + /// Total download and on-disk size across all shards. + public long TotalSizeBytes => Shards.Sum(s => s.SizeBytes); + + /// + /// False when the checkpoint is not published yet (no shards) or any shard + /// lacks a pinned hash. Such entries render as unavailable and the download + /// manager refuses them. + /// + public bool IsDownloadable => + Shards.Count > 0 && Shards.All(s => !string.IsNullOrWhiteSpace(s.Sha256) && !string.IsNullOrWhiteSpace(s.DownloadUrl)); + + /// The shard llama-server is launched against. + public GgufShard? PrimaryShard => Shards.Count > 0 ? Shards[0] : null; +} + +/// +/// The models we offer for local inference, with their tuned run recipes. +/// +/// Integrity. Every shard hash is the HuggingFace LFS object id, +/// which is the file's SHA-256. Downloads verify against it and fail closed on a +/// mismatch, matching the audio-asset policy in +/// docs/AUDIO_MODEL_ASSETS.md. +/// +public static class LocalModelCatalog +{ + /// Rough multiplier for KV cache and runtime overhead on top of the weights. + private const double OverheadFactor = 1.15; + + private const long Gib = 1024L * 1024 * 1024; + + public const string Qwen35BId = "qwen3.6-35b-a3b"; + public const string Qwen27BId = "qwen3.8-27b"; + public const string DeepSeekV4FlashId = "deepseek-v4-flash-0731"; + + private const string QwenRepoBase = + "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF/resolve/main"; + private const string DeepSeekRepoBase = + "https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF/resolve/main/UD-Q4_K_XL"; + + public static readonly IReadOnlyList Models = + [ + new( + Qwen35BId, + "Qwen3.6 35B A3B (UD-Q4_K_M)", + [ + new("Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", + $"{QwenRepoBase}/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf", + 22_663_387_424, + "0b21525e972670ed59e1812e170b27c26355381f0656ecc4e25617ece7dac58b"), + ], + // Multi-token-prediction speculative decoding plus the sampler settings + // Unsloth publishes for this checkpoint. + [ + "-b", "4096", + "-ub", "4096", + "--spec-type", "draft-mtp", + "--temp", "1.0", + "--top-k", "20", + "--top-p", "0.95", + "--min-p", "0.0", + "--repeat-penalty", "1", + "--presence-penalty", "1.5", + "-np", "1", + "-dio", + ], + MinimumRecommendedMemoryBytes: (long)(22_663_387_424 * OverheadFactor), + Notes: "A3B mixture-of-experts. Fits a single 32 GB or larger GPU."), + + new( + Qwen27BId, + "Qwen3.8 27B", + // Checkpoint not published upstream yet. The entry exists so the UI can + // show it as pending and so the recipe is reviewed alongside its sibling; + // with no shards it is not downloadable and cannot be selected. + [], + [ + "-b", "4096", + "-ub", "4096", + "--spec-type", "draft-mtp", + "--temp", "1.0", + "--top-k", "20", + "--top-p", "0.95", + "--min-p", "0", + "--repeat-penalty", "1", + "--presence-penalty", "0", + "-np", "1", + "-dio", + ], + MinimumRecommendedMemoryBytes: 20 * Gib, + Notes: "Checkpoint not released yet. The recipe is expected to match Qwen3.6 and will be confirmed on release."), + + new( + DeepSeekV4FlashId, + "DeepSeek V4 Flash 0731 (UD-Q4_K_XL)", + [ + new("DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf", + $"{DeepSeekRepoBase}/DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf", + 5_257_408, + "d13ce8f90855547bdaebe7312f531a1f2c4f822178d3103951f27fe884395cfa"), + new("DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00002-of-00005.gguf", + $"{DeepSeekRepoBase}/DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00002-of-00005.gguf", + 48_935_523_072, + "d5b61668950f4743aacd677675d7fcf7507dbe1db6d304e8ff97ed1f00827bee"), + new("DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00003-of-00005.gguf", + $"{DeepSeekRepoBase}/DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00003-of-00005.gguf", + 48_980_787_136, + "9705db7e589f360685ca7bd48100b270d78d228d4f5aa980508f3b2778af5494"), + new("DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00004-of-00005.gguf", + $"{DeepSeekRepoBase}/DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00004-of-00005.gguf", + 49_999_168_416, + "7f13a68e3ca64208454c4ba32cc2757c0cbe78e3e5576c3142bf7007ca97da42"), + new("DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00005-of-00005.gguf", + $"{DeepSeekRepoBase}/DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00005-of-00005.gguf", + 7_174_505_088, + "ed0d93164d3784968d6ce40d6d201ba98337f16e7db1b31fe495b2b0f334cc09"), + ], + [ + "--temp", "1.0", + "--top-p", "1.0", + "--min-p", "0.01", + "--spec-type", "draft-dspark", + "--spec-draft-n-max", "3", + "-ngl", "99", + "-ngld", "99", + ], + // ~155 GiB of weights. Only a very-large-memory host runs this usefully. + MinimumRecommendedMemoryBytes: 170 * Gib, + RequiresConfirmation: true, + Notes: "About 155 GB across 5 files. Requires a very large memory host and a long download."), + ]; + + /// Look up a model by catalog id, or null when unknown. + public static LocalModelInfo? Find(string? id) => + string.IsNullOrWhiteSpace(id) + ? null + : Models.FirstOrDefault(m => string.Equals(m.Id, id, StringComparison.OrdinalIgnoreCase)); +} diff --git a/src/OpenClaw.Shared/Inference/ModelRecommender.cs b/src/OpenClaw.Shared/Inference/ModelRecommender.cs new file mode 100644 index 000000000..aa4707233 --- /dev/null +++ b/src/OpenClaw.Shared/Inference/ModelRecommender.cs @@ -0,0 +1,173 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace OpenClaw.Shared.Inference; + +/// How well a model fits the host it would run on. +public enum ModelFit +{ + /// Will not run usefully here. + WontFit = 0, + /// Runs, but slowly: the weights spill out of VRAM into system RAM. + Tight = 1, + /// Fits in VRAM with headroom. + Fits = 2, +} + +/// +/// One model evaluated against a host. +/// +/// The catalog entry. +/// Verdict for this host. +/// Short PII-free explanation, shown next to the entry. +/// +/// False for entries the recommender must never pick on the user's behalf: +/// unpublished checkpoints, missing hashes, and confirmation-gated giants. +/// +public sealed record ModelFitAssessment( + LocalModelInfo Model, + ModelFit Fit, + string Reason, + bool IsEligibleForAutoSelection); + +/// +/// The recommender's answer for a host. +/// +/// Best auto-selectable model, or null when none fits. +/// Every catalog entry with its verdict, in catalog order, for the UI. +/// One-line PII-free explanation of the outcome. +public sealed record LocalModelRecommendation( + LocalModelInfo? Recommended, + IReadOnlyList Assessments, + string Summary); + +/// +/// Picks the local model a host can actually run. +/// +/// Pure and total: no I/O, no throwing. The rules are deliberately +/// conservative. Recommending a model that does not fit produces a download of +/// tens of gigabytes followed by a server that either refuses to start or runs +/// at unusable speed, so "no recommendation" is a better answer than a guess. +/// +public static class ModelRecommender +{ + /// + /// Fraction of VRAM we are willing to fill. The display driver, the desktop + /// compositor, and any other GPU client need the remainder; sizing to 100% + /// of reported VRAM reliably produces an out-of-memory failure at load. + /// + private const double VramHeadroomFactor = 0.90; + + /// + /// Fraction of installed RAM we are willing to fill for a CPU/spill run. The + /// OS and the rest of the app still have to fit. + /// + private const double SystemRamHeadroomFactor = 0.75; + + /// + /// Evaluate every catalog entry against and pick + /// the largest model that fits. + /// + /// Probe result. + /// Catalog to evaluate. Defaults to . + public static LocalModelRecommendation Recommend( + HostHardwareInfo hardware, + IReadOnlyList? models = null) + { + ArgumentNullException.ThrowIfNull(hardware); + models ??= LocalModelCatalog.Models; + + var vramBudget = Budget(hardware.TotalNvidiaVramBytes, VramHeadroomFactor); + var ramBudget = Budget(hardware.TotalPhysicalMemoryBytes, SystemRamHeadroomFactor); + + var assessments = models.Select(m => Assess(m, vramBudget, ramBudget)).ToArray(); + + // Prefer a true VRAM fit; only then accept a slow RAM-backed run. Within a + // tier, take the largest model, which is the most capable one that fits. + var recommended = assessments + .Where(a => a.IsEligibleForAutoSelection && a.Fit != ModelFit.WontFit) + .OrderByDescending(a => a.Fit) + .ThenByDescending(a => a.Model.TotalSizeBytes) + .Select(a => a.Model) + .FirstOrDefault(); + + var summary = BuildSummary(recommended, assessments, vramBudget, ramBudget); + return new LocalModelRecommendation(recommended, assessments, summary); + } + + private static ModelFitAssessment Assess(LocalModelInfo model, long? vramBudget, long? ramBudget) + { + if (!model.IsDownloadable) + { + return new ModelFitAssessment( + model, + ModelFit.WontFit, + model.Shards.Count == 0 + ? "Checkpoint is not published yet." + : "Checkpoint has no pinned hash, so it cannot be downloaded.", + IsEligibleForAutoSelection: false); + } + + var needed = model.MinimumRecommendedMemoryBytes; + var eligible = !model.RequiresConfirmation; + + if (vramBudget is { } vram && needed <= vram) + { + return new ModelFitAssessment( + model, + ModelFit.Fits, + $"Fits in {FormatGib(vram)} of usable VRAM.", + eligible); + } + + if (ramBudget is { } ram && needed <= ram) + { + var why = vramBudget is null + ? "No NVIDIA VRAM detected" + : $"Larger than the {FormatGib(vramBudget.Value)} of usable VRAM"; + return new ModelFitAssessment( + model, + ModelFit.Tight, + $"{why}, so it runs from system RAM. Expect slow generation.", + eligible); + } + + var largest = Math.Max(vramBudget ?? 0, ramBudget ?? 0); + return new ModelFitAssessment( + model, + ModelFit.WontFit, + largest > 0 + ? $"Needs about {FormatGib(needed)} but only {FormatGib(largest)} is usable on this host." + : $"Needs about {FormatGib(needed)}; available memory could not be determined.", + eligible); + } + + private static string BuildSummary( + LocalModelInfo? recommended, + IReadOnlyList assessments, + long? vramBudget, + long? ramBudget) + { + if (recommended is not null) + { + var fit = assessments.First(a => ReferenceEquals(a.Model, recommended)).Fit; + return fit == ModelFit.Fits + ? $"Recommended: {recommended.DisplayName}." + : $"Recommended: {recommended.DisplayName}. It will run from system RAM and be slow."; + } + + if (vramBudget is null && ramBudget is null) + return "Hardware could not be detected, so no model can be recommended."; + + var gated = assessments.Any(a => a.Model.RequiresConfirmation && a.Fit != ModelFit.WontFit); + return gated + ? "No model is recommended automatically. A large gated model would fit but must be chosen explicitly." + : "No catalog model fits this host's memory."; + } + + private static long? Budget(long? total, double factor) => + total is { } value && value > 0 ? (long)(value * factor) : null; + + private static string FormatGib(long bytes) => $"{bytes / (1024.0 * 1024 * 1024):F1} GB"; +} diff --git a/src/OpenClaw.Shared/Inference/NvidiaSmiParser.cs b/src/OpenClaw.Shared/Inference/NvidiaSmiParser.cs new file mode 100644 index 000000000..c2141384c --- /dev/null +++ b/src/OpenClaw.Shared/Inference/NvidiaSmiParser.cs @@ -0,0 +1,139 @@ +using System; +using System.Collections.Generic; +using System.Globalization; + +namespace OpenClaw.Shared.Inference; + +/// +/// Pure parser for the two nvidia-smi queries the hardware probe runs. +/// Kept separate from the probe so the (fiddly, vendor-controlled) text formats +/// can be unit tested against captured real output without spawning a process. +/// +public static class NvidiaSmiParser +{ + /// + /// Arguments for the per-GPU query. CSV with no header and no units keeps the + /// output machine-readable across driver versions. + /// + public static readonly string[] QueryGpuArgs = + [ + "--query-gpu=name,memory.total,driver_version", + "--format=csv,noheader,nounits", + ]; + + /// + /// Parse --query-gpu=name,memory.total,driver_version --format=csv,noheader,nounits + /// output into one per line. + /// + /// Raw stdout. Null/empty yields an empty list. + /// + /// CUDA major version to stamp on every returned adapter, from + /// . Null when unknown. + /// + /// + /// memory.total is reported in MiB. A row whose memory field is absent, + /// non-numeric, or the literal [N/A] (which the driver emits for some + /// virtualized adapters) yields a null size rather than a zero, so the caller + /// can tell "no VRAM" apart from "unknown VRAM". + /// + public static IReadOnlyList ParseQueryGpu(string? stdout, int? cudaMajorVersion = null) + { + if (string.IsNullOrWhiteSpace(stdout)) return Array.Empty(); + + var results = new List(); + foreach (var rawLine in stdout.Split('\n')) + { + var line = rawLine.Trim(); + if (line.Length == 0) continue; + + var fields = line.Split(','); + if (fields.Length < 1) continue; + + var name = fields[0].Trim(); + if (name.Length == 0) continue; + + var memoryBytes = fields.Length > 1 ? ParseMebibytes(fields[1]) : null; + var driver = fields.Length > 2 ? NullIfBlank(fields[2]) : null; + + results.Add(new GpuInfo( + GpuVendor.Nvidia, + name, + memoryBytes, + driver, + cudaMajorVersion)); + } + + return results; + } + + /// + /// Extract the CUDA major version from plain nvidia-smi output, whose + /// header line reads e.g. + /// | NVIDIA-SMI 570.86.10 Driver Version: 570.86.10 CUDA Version: 12.8 |. + /// Returns null when the marker is missing or unparseable. + /// + public static int? TryParseCudaMajorVersion(string? stdout) + { + if (string.IsNullOrWhiteSpace(stdout)) return null; + + const string marker = "CUDA Version:"; + var index = stdout.IndexOf(marker, StringComparison.OrdinalIgnoreCase); + if (index < 0) return null; + + var rest = stdout.AsSpan(index + marker.Length).TrimStart(); + + // Take the leading digit run; "12.8" and "13" both yield the major part. + var end = 0; + while (end < rest.Length && char.IsAsciiDigit(rest[end])) end++; + if (end == 0) return null; + + return int.TryParse(rest[..end], NumberStyles.None, CultureInfo.InvariantCulture, out var major) + ? major + : null; + } + + /// + /// Classify an adapter name reported by a non-NVIDIA source (WMI) into a vendor. + /// Deliberately conservative: anything unrecognized stays + /// so the selector does not assume a Vulkan + /// build will drive it. + /// + public static GpuVendor ClassifyVendor(string? adapterName) + { + if (string.IsNullOrWhiteSpace(adapterName)) return GpuVendor.Unknown; + + var name = adapterName.Trim(); + if (Contains(name, "nvidia") || Contains(name, "geforce") || Contains(name, "quadro") || Contains(name, "tesla")) + return GpuVendor.Nvidia; + if (Contains(name, "amd") || Contains(name, "radeon") || Contains(name, "advanced micro devices")) + return GpuVendor.Amd; + if (Contains(name, "intel") || Contains(name, "arc(tm)")) + return GpuVendor.Intel; + + return GpuVendor.Unknown; + + static bool Contains(string haystack, string needle) => + haystack.Contains(needle, StringComparison.OrdinalIgnoreCase); + } + + private static long? ParseMebibytes(string field) + { + var text = field.Trim(); + if (text.Length == 0 || text.Equals("[N/A]", StringComparison.OrdinalIgnoreCase)) return null; + + // Tolerate a stray unit suffix if a future driver stops honoring `nounits`. + if (text.EndsWith("MiB", StringComparison.OrdinalIgnoreCase)) + text = text[..^3].Trim(); + + if (!long.TryParse(text, NumberStyles.Integer, CultureInfo.InvariantCulture, out var mib) || mib <= 0) + return null; + + return mib * 1024L * 1024L; + } + + private static string? NullIfBlank(string field) + { + var trimmed = field.Trim(); + return trimmed.Length == 0 ? null : trimmed; + } +} diff --git a/src/OpenClaw.Shared/Inference/PhysicalMemoryProbe.cs b/src/OpenClaw.Shared/Inference/PhysicalMemoryProbe.cs new file mode 100644 index 000000000..50ec23b2b --- /dev/null +++ b/src/OpenClaw.Shared/Inference/PhysicalMemoryProbe.cs @@ -0,0 +1,63 @@ +using System; +using System.Runtime.InteropServices; + +namespace OpenClaw.Shared.Inference; + +/// Installed and currently-free physical memory, in bytes. +public readonly record struct PhysicalMemorySnapshot(long TotalBytes, long AvailableBytes) +{ + /// Percentage of physical memory currently in use, rounded to one decimal. + public double UsagePercent => TotalBytes > 0 + ? Math.Round((1.0 - (double)AvailableBytes / TotalBytes) * 100, 1) + : 0.0; +} + +/// +/// Single owner of the GlobalMemoryStatusEx P/Invoke. Both the device +/// status capability and the local-inference hardware probe need installed RAM, +/// and duplicating the interop struct in two assemblies is how the two copies +/// drift apart. +/// +public static class PhysicalMemoryProbe +{ + /// + /// Read installed and available physical memory. + /// + /// The Win32 query failed. + public static PhysicalMemorySnapshot Read() + { + var status = new MEMORYSTATUSEX { dwLength = (uint)Marshal.SizeOf() }; + if (!GlobalMemoryStatusEx(ref status)) + throw new InvalidOperationException("GlobalMemoryStatusEx failed"); + + return new PhysicalMemorySnapshot((long)status.ullTotalPhys, (long)status.ullAvailPhys); + } + + /// + /// Non-throwing variant for callers (like the hardware probe) that must + /// degrade to "unknown" rather than fail. + /// + public static PhysicalMemorySnapshot? TryRead() + { + // slopwatch-ignore: SW003 Probe is best-effort by contract; callers treat null as "unknown memory". + try { return Read(); } catch { return null; } + } + + [StructLayout(LayoutKind.Sequential)] + private struct MEMORYSTATUSEX + { + public uint dwLength; + public uint dwMemoryLoad; + public ulong ullTotalPhys; + public ulong ullAvailPhys; + public ulong ullTotalPageFile; + public ulong ullAvailPageFile; + public ulong ullTotalVirtual; + public ulong ullAvailVirtual; + public ulong ullAvailExtendedVirtual; + } + + [DllImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool GlobalMemoryStatusEx(ref MEMORYSTATUSEX lpBuffer); +} diff --git a/src/OpenClaw.Tray.WinUI/Services/DeviceStatusProvider.cs b/src/OpenClaw.Tray.WinUI/Services/DeviceStatusProvider.cs index bec25fe3d..add6f7211 100644 --- a/src/OpenClaw.Tray.WinUI/Services/DeviceStatusProvider.cs +++ b/src/OpenClaw.Tray.WinUI/Services/DeviceStatusProvider.cs @@ -6,6 +6,7 @@ using System.Threading; using System.Threading.Tasks; using OpenClaw.Shared; +using OpenClaw.Shared.Inference; using Microsoft.Win32; namespace OpenClawTray.Services; @@ -95,21 +96,13 @@ public Task GetCpuInfoAsync() public object GetMemoryInfo() { - var status = new MEMORYSTATUSEX { dwLength = (uint)Marshal.SizeOf() }; - if (!GlobalMemoryStatusEx(ref status)) - throw new InvalidOperationException("GlobalMemoryStatusEx failed"); - - var totalBytes = (long)status.ullTotalPhys; - var availableBytes = (long)status.ullAvailPhys; - var usagePercent = totalBytes > 0 - ? Math.Round((1.0 - (double)availableBytes / totalBytes) * 100, 1) - : 0.0; + var snapshot = PhysicalMemoryProbe.Read(); return new { - totalBytes, - availableBytes, - usagePercent + totalBytes = snapshot.TotalBytes, + availableBytes = snapshot.AvailableBytes, + usagePercent = snapshot.UsagePercent }; } @@ -228,25 +221,4 @@ public void Dispose() _cpuCounter = null; } - #region P/Invoke - - [StructLayout(LayoutKind.Sequential)] - private struct MEMORYSTATUSEX - { - public uint dwLength; - public uint dwMemoryLoad; - public ulong ullTotalPhys; - public ulong ullAvailPhys; - public ulong ullTotalPageFile; - public ulong ullAvailPageFile; - public ulong ullTotalVirtual; - public ulong ullAvailVirtual; - public ulong ullAvailExtendedVirtual; - } - - [DllImport("kernel32.dll", SetLastError = true)] - [return: MarshalAs(UnmanagedType.Bool)] - private static extern bool GlobalMemoryStatusEx(ref MEMORYSTATUSEX lpBuffer); - - #endregion } diff --git a/tests/OpenClaw.Shared.Tests/AssetHashPinningTests.cs b/tests/OpenClaw.Shared.Tests/AssetHashPinningTests.cs index be7e18016..86559181d 100644 --- a/tests/OpenClaw.Shared.Tests/AssetHashPinningTests.cs +++ b/tests/OpenClaw.Shared.Tests/AssetHashPinningTests.cs @@ -1,5 +1,7 @@ +using System.Linq; using System.Text.RegularExpressions; using OpenClaw.Shared.Audio; +using OpenClaw.Shared.Inference; using Xunit; namespace OpenClaw.Shared.Tests; @@ -67,4 +69,50 @@ public void SileroVadModel_HasPinnedSha256() Assert.Matches(Sha256Hex, SileroVadModelManifest.Sha256); Assert.StartsWith("https://", SileroVadModelManifest.DownloadUrl); } + + [Fact] + public void EveryLlamaBackendAsset_HasPinnedSha256AndSize() + { + Assert.NotEmpty(LlamaBackendCatalog.Variants); + foreach (var variant in LlamaBackendCatalog.Variants) + { + Assert.True(variant.IsDownloadable, + $"llama.cpp variant '{variant.RuntimeKey}' is not downloadable. " + + "Every shipped variant must have a pinned SHA-256 on all of its assets."); + + foreach (var asset in variant.Assets) + { + Assert.False(string.IsNullOrWhiteSpace(asset.Sha256), + $"llama.cpp asset '{asset.FileName}' is missing a pinned SHA-256 hash."); + Assert.Matches(Sha256Hex, asset.Sha256!); + Assert.StartsWith("https://", asset.DownloadUrl); + Assert.True(asset.ApproximateSizeBytes > 0, + $"llama.cpp asset '{asset.FileName}' is missing its size. " + + "The size is cross-checked against the release API when pinning the hash."); + } + } + } + + [Fact] + public void EveryPublishedLocalModel_HasPinnedSha256() + { + // Unpublished checkpoints legitimately carry no shards; they are not + // downloadable and cannot be selected. Anything with shards must be pinned. + var published = LocalModelCatalog.Models.Where(m => m.Shards.Count > 0).ToArray(); + Assert.NotEmpty(published); + + foreach (var model in published) + { + foreach (var shard in model.Shards) + { + Assert.False(string.IsNullOrWhiteSpace(shard.Sha256), + $"GGUF shard '{shard.FileName}' of model '{model.Id}' is missing a pinned SHA-256 hash."); + Assert.Matches(Sha256Hex, shard.Sha256!); + Assert.StartsWith("https://", shard.DownloadUrl); + Assert.True(shard.SizeBytes > 0, $"GGUF shard '{shard.FileName}' is missing its size."); + } + + Assert.True(model.IsDownloadable); + } + } } diff --git a/tests/OpenClaw.Shared.Tests/Inference/BackendSelectorTests.cs b/tests/OpenClaw.Shared.Tests/Inference/BackendSelectorTests.cs new file mode 100644 index 000000000..e11892587 --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/BackendSelectorTests.cs @@ -0,0 +1,205 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using OpenClaw.Shared.Inference; +using Xunit; + +// Disambiguates from the OpenClaw.Shared.Tests.Architecture namespace (the +// architecture-guard test folder), which otherwise shadows the enum. +using Arch = System.Runtime.InteropServices.Architecture; + +namespace OpenClaw.Shared.Tests.Inference; + +/// +/// The selector decides whether a user gets GPU inference at all, and its inputs +/// (driver versions, adapter vendors, architectures) are impossible to cover on +/// one CI machine. Every branch is therefore pinned here against synthetic +/// values. +/// +public class BackendSelectorTests +{ + private static HostHardwareInfo Host( + Arch arch = Arch.X64, + long? ram = 64L * 1024 * 1024 * 1024, + IReadOnlyList? gpus = null, + bool vulkan = false) => + new(arch, ram, ram, gpus ?? Array.Empty(), vulkan); + + private static GpuInfo Nvidia(long vramGib = 24, int? cudaMajor = 12) => + new(GpuVendor.Nvidia, "NVIDIA Test GPU", vramGib * 1024 * 1024 * 1024, "999.99", cudaMajor); + + [Fact] + public void NvidiaWithCuda13_PrefersTheCuda13Build() + { + var plan = BackendSelector.Select(Host(gpus: [Nvidia(cudaMajor: 13)])); + + Assert.Equal(LlamaBackend.Cuda13, plan.Preferred!.Backend); + Assert.Equal(Arch.X64, plan.Preferred.Architecture); + } + + [Fact] + public void NvidiaWithCuda12_PrefersTheCuda12Build() + { + var plan = BackendSelector.Select(Host(gpus: [Nvidia(cudaMajor: 12)])); + + Assert.Equal(LlamaBackend.Cuda12, plan.Preferred!.Backend); + } + + [Fact] + public void NvidiaWithUnknownCudaVersion_DegradesToTheCuda12Build() + { + // A CUDA 12 runtime works on newer drivers; a CUDA 13 runtime does not + // work on older ones. The unknown case must degrade downward. + var plan = BackendSelector.Select(Host(gpus: [Nvidia(cudaMajor: null)])); + + Assert.Equal(LlamaBackend.Cuda12, plan.Preferred!.Backend); + Assert.Contains("CUDA version unknown", plan.Reason); + } + + [Fact] + public void CudaVariantsAlwaysCarryTheirCudartArchive() + { + // Missing cudart is the classic "llama-server.exe won't start and the + // error names a DLL" failure, so the pairing is asserted structurally. + foreach (var variant in LlamaBackendCatalog.Variants + .Where(v => v.Backend is LlamaBackend.Cuda12 or LlamaBackend.Cuda13)) + { + Assert.Equal(2, variant.Assets.Count); + Assert.Contains(variant.Assets, a => a.FileName.StartsWith("cudart-", StringComparison.Ordinal)); + Assert.Contains(variant.Assets, a => a.FileName.StartsWith("llama-", StringComparison.Ordinal)); + } + } + + [Fact] + public void NvidiaOnArm64_SelectsTheArm64CudaBuild() + { + var plan = BackendSelector.Select(Host(Arch.Arm64, gpus: [Nvidia(cudaMajor: 13)])); + + Assert.Equal(LlamaBackend.Cuda13, plan.Preferred!.Backend); + Assert.Equal(Arch.Arm64, plan.Preferred.Architecture); + } + + [Fact] + public void NvidiaPlan_FallsBackThroughTheOtherCudaBuildThenCpu() + { + var plan = BackendSelector.Select(Host(gpus: [Nvidia(cudaMajor: 13)], vulkan: true)); + + var order = plan.InPreferenceOrder.Select(v => v.Backend).ToArray(); + Assert.Equal( + [LlamaBackend.Cuda13, LlamaBackend.Cuda12, LlamaBackend.Vulkan, LlamaBackend.Cpu], + order); + } + + [Fact] + public void AmdGpuWithVulkanLoader_SelectsVulkan() + { + var gpus = new[] { new GpuInfo(GpuVendor.Amd, "AMD Radeon RX 7900 XTX") }; + + var plan = BackendSelector.Select(Host(gpus: gpus, vulkan: true)); + + Assert.Equal(LlamaBackend.Vulkan, plan.Preferred!.Backend); + Assert.Equal(LlamaBackend.Cpu, Assert.Single(plan.Fallbacks).Backend); + } + + [Fact] + public void AmdGpuWithoutVulkanLoader_FallsBackToCpu() + { + // Shipping a Vulkan build to a host with no loader turns "no acceleration" + // into "the server refuses to start", which is strictly worse. + var gpus = new[] { new GpuInfo(GpuVendor.Amd, "AMD Radeon RX 7900 XTX") }; + + var plan = BackendSelector.Select(Host(gpus: gpus, vulkan: false)); + + Assert.Equal(LlamaBackend.Cpu, plan.Preferred!.Backend); + } + + [Fact] + public void UnclassifiedAdapter_IsNotTreatedAsVulkanCapable() + { + var gpus = new[] { new GpuInfo(GpuVendor.Unknown, "Microsoft Basic Display Adapter") }; + + var plan = BackendSelector.Select(Host(gpus: gpus, vulkan: true)); + + Assert.Equal(LlamaBackend.Cpu, plan.Preferred!.Backend); + } + + [Fact] + public void NoGpu_SelectsCpuForTheHostArchitecture() + { + Assert.Equal(LlamaBackend.Cpu, BackendSelector.Select(Host()).Preferred!.Backend); + + var arm = BackendSelector.Select(Host(Arch.Arm64)); + Assert.Equal(LlamaBackend.Cpu, arm.Preferred!.Backend); + Assert.Equal(Arch.Arm64, arm.Preferred.Architecture); + } + + [Fact] + public void UnknownHardware_SelectsCpu() + { + var plan = BackendSelector.Select(HostHardwareInfo.Unknown); + + Assert.Equal(LlamaBackend.Cpu, plan.Preferred!.Backend); + } + + [Fact] + public void X86Host_IsCollapsedOntoTheX64Build() + { + var plan = BackendSelector.Select(Host(Arch.X86)); + + Assert.Equal(Arch.X64, plan.Preferred!.Architecture); + } + + [Fact] + public void UserOverride_WinsAndKeepsTheAutoPlanAsFallback() + { + var plan = BackendSelector.Select(Host(gpus: [Nvidia()]), LlamaBackend.Cpu); + + Assert.Equal(LlamaBackend.Cpu, plan.Preferred!.Backend); + Assert.Contains(plan.Fallbacks, v => v.Backend == LlamaBackend.Cuda12); + Assert.DoesNotContain(plan.Fallbacks, v => ReferenceEquals(v, plan.Preferred)); + } + + [Fact] + public void UserOverride_IsIgnoredWhenTheReleaseHasNoSuchBuildForTheArchitecture() + { + // This release ships no ARM64 Vulkan build. Honoring the override would + // produce a plan that can never launch. + var plan = BackendSelector.Select(Host(Arch.Arm64), LlamaBackend.Vulkan); + + Assert.Equal(LlamaBackend.Cpu, plan.Preferred!.Backend); + Assert.Contains("override", plan.Reason, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void EveryCatalogAssetUsesAnHttpsUrlUnderThePinnedReleaseTag() + { + Assert.NotEmpty(LlamaBackendCatalog.Variants); + foreach (var asset in LlamaBackendCatalog.Variants.SelectMany(v => v.Assets)) + { + Assert.StartsWith("https://", asset.DownloadUrl, StringComparison.Ordinal); + Assert.Contains($"/{LlamaBackendCatalog.ReleaseTag}/", asset.DownloadUrl, StringComparison.Ordinal); + Assert.EndsWith(".zip", asset.FileName, StringComparison.Ordinal); + } + } + + [Fact] + public void VariantsWithoutAPinnedHashAreNotDownloadable() + { + // Fail-closed guard, mirroring the audio catalogs: an entry that has not + // been hashed yet must be visible but never installable. + foreach (var variant in LlamaBackendCatalog.Variants) + { + var allPinned = variant.Assets.All(a => !string.IsNullOrWhiteSpace(a.Sha256)); + Assert.Equal(allPinned, variant.IsDownloadable); + } + } + + [Fact] + public void RuntimeKeysAreUniqueAndPathSafe() + { + var keys = LlamaBackendCatalog.Variants.Select(v => v.RuntimeKey).ToArray(); + + Assert.Equal(keys.Length, keys.Distinct(StringComparer.Ordinal).Count()); + Assert.All(keys, k => Assert.Equal(-1, k.IndexOfAny(System.IO.Path.GetInvalidFileNameChars()))); + } +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/HardwareProbeTests.cs b/tests/OpenClaw.Shared.Tests/Inference/HardwareProbeTests.cs new file mode 100644 index 000000000..1ee021355 --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/HardwareProbeTests.cs @@ -0,0 +1,168 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using OpenClaw.Shared; +using OpenClaw.Shared.Inference; +using Xunit; + +namespace OpenClaw.Shared.Tests.Inference; + +/// +/// The probe's contract is that it never throws and always degrades to +/// "unknown", because a machine we cannot classify has to land on the CPU +/// backend rather than break the settings page. +/// +public class HardwareProbeTests +{ + private const string Banner = + "| NVIDIA-SMI 570.86.10 Driver Version: 570.86.10 CUDA Version: 13.0 |"; + private const string QueryOutput = "NVIDIA RTX 6000 Ada Generation, 49140, 570.86.10"; + + [Fact] + public async Task DetectsNvidiaGpuWithVramAndCudaVersion() + { + var runner = new ScriptedRunner(argv => + argv.Any(a => a.StartsWith("--query-gpu", StringComparison.Ordinal)) + ? Ok(QueryOutput) + : Ok(Banner)); + + var info = await new HardwareProbe(runner, NullLogger.Instance, vulkanLoaderProbe: () => false).GetAsync(); + + var gpu = Assert.Single(info.Gpus); + Assert.Equal(GpuVendor.Nvidia, gpu.Vendor); + Assert.Equal(49140L * 1024 * 1024, gpu.DedicatedMemoryBytes); + Assert.Equal(13, info.MaxCudaMajorVersion); + Assert.True(info.HasNvidiaGpu); + } + + [Fact] + public async Task SumsVramAcrossMultipleNvidiaGpus() + { + // llama.cpp's default split-mode spreads a model over every visible + // device, so total VRAM (not the largest single card) is the capacity. + var runner = new ScriptedRunner(argv => + argv.Any(a => a.StartsWith("--query-gpu", StringComparison.Ordinal)) + ? Ok("NVIDIA GeForce RTX 4090, 24564, 566.36\nNVIDIA GeForce RTX 4090, 24564, 566.36") + : Ok(Banner)); + + var info = await new HardwareProbe(runner, NullLogger.Instance, vulkanLoaderProbe: () => false).GetAsync(); + + Assert.Equal(2 * 24564L * 1024 * 1024, info.TotalNvidiaVramBytes); + } + + [Fact] + public async Task FallsBackToAdapterEnumerationWhenNvidiaSmiIsAbsent() + { + var runner = new ScriptedRunner(_ => throw new InvalidOperationException("nvidia-smi not found")); + IReadOnlyList fallback = [new(GpuVendor.Amd, "AMD Radeon RX 7900 XTX")]; + + var info = await new HardwareProbe( + runner, + NullLogger.Instance, + fallbackGpuEnumerator: () => fallback, + vulkanLoaderProbe: () => true).GetAsync(); + + Assert.False(info.HasNvidiaGpu); + Assert.True(info.HasNonNvidiaGpu); + Assert.True(info.VulkanAvailable); + Assert.Null(info.TotalNvidiaVramBytes); + } + + [Fact] + public async Task ReportsNoGpusWhenEverySourceFails() + { + var runner = new ScriptedRunner(_ => throw new InvalidOperationException("boom")); + + var info = await new HardwareProbe( + runner, + NullLogger.Instance, + fallbackGpuEnumerator: () => throw new InvalidOperationException("also boom"), + vulkanLoaderProbe: () => throw new InvalidOperationException("and boom")).GetAsync(); + + Assert.Empty(info.Gpus); + Assert.False(info.VulkanAvailable); + Assert.Null(info.MaxCudaMajorVersion); + } + + [Fact] + public async Task TreatsANonZeroExitAsNoNvidiaGpu() + { + var runner = new ScriptedRunner(_ => new CommandResult { ExitCode = 9, Stdout = "" }); + + var info = await new HardwareProbe(runner, NullLogger.Instance, vulkanLoaderProbe: () => false).GetAsync(); + + Assert.Empty(info.Gpus); + } + + [Fact] + public async Task TreatsATimeoutAsNoNvidiaGpu() + { + var runner = new ScriptedRunner(_ => new CommandResult { TimedOut = true, Stdout = QueryOutput }); + + var info = await new HardwareProbe(runner, NullLogger.Instance, vulkanLoaderProbe: () => false).GetAsync(); + + Assert.Empty(info.Gpus); + } + + [Fact] + public async Task CachesTheResultAndRefreshOnDemandReprobes() + { + var runner = new ScriptedRunner(argv => + argv.Any(a => a.StartsWith("--query-gpu", StringComparison.Ordinal)) + ? Ok(QueryOutput) + : Ok(Banner)); + var probe = new HardwareProbe(runner, NullLogger.Instance, vulkanLoaderProbe: () => false); + + Assert.Null(probe.Cached); + + await probe.GetAsync(); + var callsAfterFirst = runner.CallCount; + Assert.NotNull(probe.Cached); + + await probe.GetAsync(); + Assert.Equal(callsAfterFirst, runner.CallCount); + + await probe.RefreshAsync(); + Assert.True(runner.CallCount > callsAfterFirst); + } + + [Fact] + public async Task FallsBackToTheSystem32CopyWhenThePathLookupFails() + { + // A PATH-less service context is a real deployment shape; the driver + // installs nvidia-smi.exe into System32 regardless. + var runner = new ScriptedRunner(argv => + { + if (argv[0] == "nvidia-smi") throw new InvalidOperationException("not on PATH"); + return argv.Any(a => a.StartsWith("--query-gpu", StringComparison.Ordinal)) ? Ok(QueryOutput) : Ok(Banner); + }); + + var info = await new HardwareProbe(runner, NullLogger.Instance, vulkanLoaderProbe: () => false).GetAsync(); + + Assert.True(info.HasNvidiaGpu); + Assert.Contains(runner.Invocations, argv => argv[0].EndsWith("nvidia-smi.exe", StringComparison.OrdinalIgnoreCase)); + } + + private static CommandResult Ok(string stdout) => new() { ExitCode = 0, Stdout = stdout }; + + private sealed class ScriptedRunner : ICommandRunner + { + private readonly Func, CommandResult> _respond; + private readonly List> _invocations = []; + + public ScriptedRunner(Func, CommandResult> respond) => _respond = respond; + + public string Name => "scripted"; + public int CallCount => _invocations.Count; + public IReadOnlyList> Invocations => _invocations; + + public Task RunAsync(CommandRequest request, CancellationToken ct = default) + { + var argv = request.Argv ?? throw new InvalidOperationException("Probe must use direct argv."); + _invocations.Add(argv); + return Task.FromResult(_respond(argv)); + } + } +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/ModelRecommenderTests.cs b/tests/OpenClaw.Shared.Tests/Inference/ModelRecommenderTests.cs new file mode 100644 index 000000000..cd9e1e128 --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/ModelRecommenderTests.cs @@ -0,0 +1,185 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text.RegularExpressions; +using OpenClaw.Shared.Inference; +using Xunit; + +// Disambiguates from the OpenClaw.Shared.Tests.Architecture namespace. +using Arch = System.Runtime.InteropServices.Architecture; + +namespace OpenClaw.Shared.Tests.Inference; + +/// +/// Recommending a model that does not fit costs the user a multi-gigabyte +/// download followed by a server that will not start, so the fit rules are +/// pinned across the range of hosts we expect. +/// +public class ModelRecommenderTests +{ + private const long Gib = 1024L * 1024 * 1024; + private static readonly Regex Sha256Hex = new("^[0-9a-f]{64}$", RegexOptions.Compiled); + + private static HostHardwareInfo Host(long ramGib, long? vramGib, int cudaMajor = 12) + { + IReadOnlyList gpus = vramGib is { } vram + ? [new GpuInfo(GpuVendor.Nvidia, "NVIDIA Test GPU", vram * Gib, "999.99", cudaMajor)] + : Array.Empty(); + + return new HostHardwareInfo(Arch.X64, ramGib * Gib, ramGib * Gib, gpus, false); + } + + [Fact] + public void LargeVramWorkstation_RecommendsQwen35B() + { + var result = ModelRecommender.Recommend(Host(ramGib: 256, vramGib: 96)); + + Assert.NotNull(result.Recommended); + Assert.Equal(LocalModelCatalog.Qwen35BId, result.Recommended!.Id); + Assert.Equal( + ModelFit.Fits, + result.Assessments.Single(a => a.Model.Id == LocalModelCatalog.Qwen35BId).Fit); + } + + [Fact] + public void SmallGpuWithLargeRam_FallsBackToASlowRamBackedRun() + { + var result = ModelRecommender.Recommend(Host(ramGib: 64, vramGib: 8)); + + Assert.Equal(LocalModelCatalog.Qwen35BId, result.Recommended!.Id); + + var assessment = result.Assessments.Single(a => a.Model.Id == LocalModelCatalog.Qwen35BId); + Assert.Equal(ModelFit.Tight, assessment.Fit); + Assert.Contains("slow", assessment.Reason, StringComparison.OrdinalIgnoreCase); + Assert.Contains("slow", result.Summary, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void MachineWithNoGpuButAmpleRam_StillGetsARecommendation() + { + var result = ModelRecommender.Recommend(Host(ramGib: 64, vramGib: null)); + + Assert.Equal(LocalModelCatalog.Qwen35BId, result.Recommended!.Id); + Assert.Contains( + "No NVIDIA VRAM", + result.Assessments.Single(a => a.Model.Id == LocalModelCatalog.Qwen35BId).Reason, + StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void SmallLaptop_GetsNoRecommendation() + { + var result = ModelRecommender.Recommend(Host(ramGib: 16, vramGib: null)); + + Assert.Null(result.Recommended); + Assert.All(result.Assessments, a => Assert.Equal(ModelFit.WontFit, a.Fit)); + } + + [Fact] + public void UndetectableHardware_GetsNoRecommendationRatherThanAGuess() + { + var result = ModelRecommender.Recommend(HostHardwareInfo.Unknown); + + Assert.Null(result.Recommended); + Assert.Contains("could not be detected", result.Summary, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void DeepSeek_IsNeverAutoSelectedEvenOnAHostThatCouldRunIt() + { + // A 155 GB download must be an explicit choice, never a default. + var result = ModelRecommender.Recommend(Host(ramGib: 1024, vramGib: 512)); + + Assert.Equal(LocalModelCatalog.Qwen35BId, result.Recommended!.Id); + + var deepSeek = result.Assessments.Single(a => a.Model.Id == LocalModelCatalog.DeepSeekV4FlashId); + Assert.Equal(ModelFit.Fits, deepSeek.Fit); + Assert.False(deepSeek.IsEligibleForAutoSelection); + } + + [Fact] + public void UnpublishedCheckpoint_IsReportedAsPendingAndNotSelected() + { + var result = ModelRecommender.Recommend(Host(ramGib: 256, vramGib: 96)); + + var pending = result.Assessments.Single(a => a.Model.Id == LocalModelCatalog.Qwen27BId); + Assert.Equal(ModelFit.WontFit, pending.Fit); + Assert.False(pending.IsEligibleForAutoSelection); + Assert.Contains("not published", pending.Reason, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void EveryCatalogEntryIsAssessed() + { + var result = ModelRecommender.Recommend(Host(ramGib: 64, vramGib: 24)); + + Assert.Equal(LocalModelCatalog.Models.Count, result.Assessments.Count); + Assert.Equal( + LocalModelCatalog.Models.Select(m => m.Id), + result.Assessments.Select(a => a.Model.Id)); + } + + [Fact] + public void EveryDownloadableShardHasAPinnedSha256AndHttpsUrl() + { + // Same fail-closed contract as the audio catalogs: an entry we are willing + // to download must be verifiable. See docs/AUDIO_MODEL_ASSETS.md. + foreach (var model in LocalModelCatalog.Models.Where(m => m.IsDownloadable)) + { + Assert.NotEmpty(model.Shards); + foreach (var shard in model.Shards) + { + Assert.Matches(Sha256Hex, shard.Sha256!); + Assert.StartsWith("https://", shard.DownloadUrl, StringComparison.Ordinal); + Assert.True(shard.SizeBytes > 0, $"Shard '{shard.FileName}' has no size."); + } + } + } + + [Fact] + public void ModelsWithoutPinnedHashesAreNotDownloadable() + { + foreach (var model in LocalModelCatalog.Models) + { + var allPinned = model.Shards.Count > 0 + && model.Shards.All(s => !string.IsNullOrWhiteSpace(s.Sha256)); + Assert.Equal(allPinned, model.IsDownloadable); + } + } + + [Fact] + public void RecipesDoNotSetArgumentsTheLauncherOwns() + { + // -m / --host / --port are set by the process launcher from runtime state. + // A recipe that also sets them would silently win or duplicate. + string[] reserved = ["-m", "--model", "--host", "--port"]; + + foreach (var model in LocalModelCatalog.Models) + { + Assert.DoesNotContain(model.RecipeArgs, arg => reserved.Contains(arg, StringComparer.Ordinal)); + } + } + + [Fact] + public void MultiShardModelsAreOrderedWithShardOneFirst() + { + // llama-server is launched against the first shard and discovers the rest + // by name, so ordering is load-bearing, not cosmetic. + foreach (var model in LocalModelCatalog.Models.Where(m => m.Shards.Count > 1)) + { + Assert.Contains("00001-of-", model.PrimaryShard!.FileName, StringComparison.Ordinal); + Assert.Equal( + model.Shards.Select(s => s.FileName).OrderBy(f => f, StringComparer.Ordinal), + model.Shards.Select(s => s.FileName)); + } + } + + [Fact] + public void CatalogIdsAreUniqueAndPathSafe() + { + var ids = LocalModelCatalog.Models.Select(m => m.Id).ToArray(); + + Assert.Equal(ids.Length, ids.Distinct(StringComparer.OrdinalIgnoreCase).Count()); + Assert.All(ids, id => Assert.Equal(-1, id.IndexOfAny(System.IO.Path.GetInvalidFileNameChars()))); + } +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/NvidiaSmiParserTests.cs b/tests/OpenClaw.Shared.Tests/Inference/NvidiaSmiParserTests.cs new file mode 100644 index 000000000..dd074028a --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/NvidiaSmiParserTests.cs @@ -0,0 +1,107 @@ +using OpenClaw.Shared.Inference; +using Xunit; + +namespace OpenClaw.Shared.Tests.Inference; + +/// +/// The nvidia-smi text formats are vendor-controlled and the only source of a +/// trustworthy VRAM number, so they are pinned against captured real output. +/// +public class NvidiaSmiParserTests +{ + [Fact] + public void ParseQueryGpu_ReadsNameMemoryAndDriver() + { + const string stdout = "NVIDIA RTX 6000 Ada Generation, 49140, 570.86.10\n"; + + var gpus = NvidiaSmiParser.ParseQueryGpu(stdout, cudaMajorVersion: 12); + + var gpu = Assert.Single(gpus); + Assert.Equal(GpuVendor.Nvidia, gpu.Vendor); + Assert.Equal("NVIDIA RTX 6000 Ada Generation", gpu.Name); + Assert.Equal(49140L * 1024 * 1024, gpu.DedicatedMemoryBytes); + Assert.Equal("570.86.10", gpu.DriverVersion); + Assert.Equal(12, gpu.CudaMajorVersion); + } + + [Fact] + public void ParseQueryGpu_ReadsEveryGpuOnMultiGpuHosts() + { + const string stdout = + "NVIDIA GeForce RTX 4090, 24564, 566.36\r\n" + + "NVIDIA GeForce RTX 4090, 24564, 566.36\r\n"; + + var gpus = NvidiaSmiParser.ParseQueryGpu(stdout); + + Assert.Equal(2, gpus.Count); + Assert.All(gpus, g => Assert.Equal(24564L * 1024 * 1024, g.DedicatedMemoryBytes)); + } + + [Theory] + [InlineData("")] + [InlineData(" \n \n")] + [InlineData(null)] + public void ParseQueryGpu_ReturnsEmptyForNoOutput(string? stdout) + { + Assert.Empty(NvidiaSmiParser.ParseQueryGpu(stdout)); + } + + [Fact] + public void ParseQueryGpu_LeavesMemoryNullWhenDriverReportsNotAvailable() + { + // Some virtualized adapters report [N/A]. Null must be distinguishable + // from zero so the recommender treats it as "unknown", not "no VRAM". + var gpus = NvidiaSmiParser.ParseQueryGpu("NVIDIA A100-SXM4-40GB, [N/A], 535.104.05"); + + Assert.Null(Assert.Single(gpus).DedicatedMemoryBytes); + } + + [Fact] + public void ParseQueryGpu_ToleratesAUnitSuffixIfNounitsIsIgnored() + { + var gpus = NvidiaSmiParser.ParseQueryGpu("NVIDIA L40S, 46068 MiB, 550.54.15"); + + Assert.Equal(46068L * 1024 * 1024, Assert.Single(gpus).DedicatedMemoryBytes); + } + + [Fact] + public void TryParseCudaMajorVersion_ReadsTheBannerLine() + { + const string banner = + "Thu Aug 14 09:12:03 2026\n" + + "+-----------------------------------------------------------------------------+\n" + + "| NVIDIA-SMI 570.86.10 Driver Version: 570.86.10 CUDA Version: 12.8 |\n"; + + Assert.Equal(12, NvidiaSmiParser.TryParseCudaMajorVersion(banner)); + } + + [Fact] + public void TryParseCudaMajorVersion_HandlesAMajorOnlyVersion() + { + Assert.Equal(13, NvidiaSmiParser.TryParseCudaMajorVersion("CUDA Version: 13")); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData("no such marker here")] + [InlineData("CUDA Version: N/A")] + public void TryParseCudaMajorVersion_ReturnsNullWhenUnavailable(string? stdout) + { + Assert.Null(NvidiaSmiParser.TryParseCudaMajorVersion(stdout)); + } + + [Theory] + [InlineData("NVIDIA GeForce RTX 4090", GpuVendor.Nvidia)] + [InlineData("Quadro P2000", GpuVendor.Nvidia)] + [InlineData("AMD Radeon RX 7900 XTX", GpuVendor.Amd)] + [InlineData("Advanced Micro Devices, Inc. [AMD/ATI]", GpuVendor.Amd)] + [InlineData("Intel(R) Arc(TM) A770 Graphics", GpuVendor.Intel)] + [InlineData("Microsoft Basic Display Adapter", GpuVendor.Unknown)] + [InlineData("", GpuVendor.Unknown)] + [InlineData(null, GpuVendor.Unknown)] + public void ClassifyVendor_MapsKnownAdapterNames(string? name, GpuVendor expected) + { + Assert.Equal(expected, NvidiaSmiParser.ClassifyVendor(name)); + } +} From 76192ac0f1f8d74233468d2e94b996ba6e2d154a Mon Sep 17 00:00:00 2001 From: Pedro Larroy Date: Tue, 18 Aug 2026 15:51:41 -0700 Subject: [PATCH 3/7] feat(inference): add llama.cpp runtime and GGUF download managers Second phase of local inference: fetch and install what the Phase 1 decision layer selected. Still no process launch or UI. New in src/OpenClaw.Shared/Inference/: - VerifiedFileDownloader stages to a .part file, verifies SHA-256, and only then moves into place. Shared by both managers. - SafeZipExtractor extracts with an explicit path-traversal guard. - LlamaRuntimeManager installs a backend variant into llama/runtimes// and resolves llama-server.exe, or resolves a user-supplied custom build. - GgufModelManager downloads multi-shard checkpoints into llama/models//, preserving upstream shard names. Notable decisions: - Nothing unverified reaches a final path. A missing pinned hash fails before any network traffic; a hash mismatch, a length disagreement with the catalog, or a truncated response deletes the partial file and throws. The error never echoes the computed hash, which would be a confirmation oracle. - Resume is opt-in per request. GGUF shards run to tens of gigabytes, so a dropped connection has to be recoverable. A server that ignores Range and answers 200 triggers a clean restart rather than appending a full body onto an existing prefix. A partial file at or past the expected size is discarded, since it is the residue of an attempt that already failed verification and resuming from its end would loop. - A runtime directory is only trusted with its completion marker. A CUDA variant has two archives, and an interrupted install can leave a directory holding llama-server.exe but none of its CUDA DLLs, which would look installed and then fail at launch with a missing-DLL error. - Archives are deleted as they extract; a CUDA pair is close to 800 MB and keeping both would double peak disk use. - Free space is checked before starting, counting only missing shards so resuming a mostly-complete model is not blocked by the full size. Unknown free space proceeds rather than refusing. - A custom build skips download and hashing by design, and is reported through LlamaRuntime.IsUnverified so the UI can never show the bypass silently. Progress reporting uses a new InlineProgress rather than System.Progress. Progress posts each report to the thread pool with no ordering guarantee, so the per-file to aggregate translation could deliver counts out of order and make a bound progress bar rewind. This surfaced as an intermittent test failure and is a real defect, not a test artifact. Tests run against an in-memory HttpMessageHandler: tampered bodies and length disagreements rejected with no residue, resume via Range, clean restart when Range is ignored, truncated-then-retried downloads, monotonic aggregate progress, zip traversal and sibling-prefix rejection, interrupted-install rebuild, and the free-space precheck. Validation: ./build.ps1 (all projects, 49 docs), OpenClaw.Shared.Tests 3805 passed across five consecutive runs, OpenClaw.Tray.Tests 2469 passed. Co-Authored-By: Claude Opus 5 (1M context) --- docs/LOCAL_INFERENCE_PLAN.md | 74 ++-- .../Inference/GgufModelManager.cs | 275 +++++++++++++++ .../Inference/InlineProgress.cs | 25 ++ .../Inference/LlamaRuntimeManager.cs | 322 +++++++++++++++++ .../Inference/SafeZipExtractor.cs | 71 ++++ .../Inference/VerifiedFileDownloader.cs | 277 +++++++++++++++ .../Inference/FakeHttpTransport.cs | 95 +++++ .../Inference/GgufModelManagerTests.cs | 324 ++++++++++++++++++ .../Inference/LlamaRuntimeManagerTests.cs | 296 ++++++++++++++++ .../Inference/SafeZipExtractorTests.cs | 109 ++++++ .../Inference/VerifiedFileDownloaderTests.cs | 273 +++++++++++++++ 11 files changed, 2119 insertions(+), 22 deletions(-) create mode 100644 src/OpenClaw.Shared/Inference/GgufModelManager.cs create mode 100644 src/OpenClaw.Shared/Inference/InlineProgress.cs create mode 100644 src/OpenClaw.Shared/Inference/LlamaRuntimeManager.cs create mode 100644 src/OpenClaw.Shared/Inference/SafeZipExtractor.cs create mode 100644 src/OpenClaw.Shared/Inference/VerifiedFileDownloader.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/FakeHttpTransport.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/GgufModelManagerTests.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/LlamaRuntimeManagerTests.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/SafeZipExtractorTests.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/VerifiedFileDownloaderTests.cs diff --git a/docs/LOCAL_INFERENCE_PLAN.md b/docs/LOCAL_INFERENCE_PLAN.md index dfee0ae26..b53d3e8ca 100644 --- a/docs/LOCAL_INFERENCE_PLAN.md +++ b/docs/LOCAL_INFERENCE_PLAN.md @@ -6,7 +6,7 @@ inline; update it as phases land. | Phase | Scope | Status | | --- | --- | --- | | 1 | Hardware probe, backend selection, model recommender | Landed | -| 2 | Runtime and GGUF download managers | Not started (catalog hashes pinned, unblocked) | +| 2 | Runtime and GGUF download managers | Landed | | 3 | Server process and settings UI | Not started | | 4 | Gateway provider registration | Not started (blocked on live schema) | | 5 | Optional `localinference.status` node capability | Not started | @@ -125,31 +125,57 @@ Run recipes are stored as structured argument lists and deliberately exclude `-m`, `--host`, and `--port`, which the process launcher owns; a test enforces that separation. -## Phase 2: download managers (blocked on hash pinning) +## Phase 2: download managers (landed) -`Inference/LlamaRuntimeManager.cs` and `Inference/GgufModelManager.cs`, modelled -on `PiperVoiceManager` and `WhisperModelManager`: +| File | Role | +| --- | --- | +| `Inference/VerifiedFileDownloader.cs` | Fetch to `.part`, verify SHA-256, then move. Shared by both managers. | +| `Inference/SafeZipExtractor.cs` | Zip extraction with an explicit path-traversal guard. | +| `Inference/LlamaRuntimeManager.cs` | Install and resolve a backend variant, or a custom build. | +| `Inference/GgufModelManager.cs` | Download and manage multi-shard checkpoints. | -- Per-key single flight via the existing `SingleFlightDownload.RunAsync`. -- Stage to a `.tmp` file, verify SHA-256, and only then move or extract. -- Delete the partial file on any failure. -- Zip extraction rejects entries whose resolved path escapes the destination. -- Runtimes land in `\llama\runtimes\\`, models in - `\llama\models\\`. -- GGUF downloads need aggregate cross-shard progress, a free-disk-space - precheck, and `Range`-based resume. Restarting a 50 GB shard from zero after a - dropped connection is not acceptable. +Decisions worth preserving: + +- **Nothing unverified reaches a final path.** A missing pinned hash fails before + any network traffic. A mismatch, a length disagreement, or a truncated + response deletes the partial file and throws. The error never echoes the + computed hash, which would be a confirmation oracle. +- **Resume is opt-in per request.** GGUF shards run to tens of gigabytes, so a + dropped connection must be recoverable; small archives just restart. A server + that ignores `Range` and answers 200 triggers a clean restart rather than + appending a full body onto an existing prefix and corrupting the file. A + partial file at or past the expected size is discarded, since it is the + residue of an attempt that already failed verification and resuming from its + end would loop forever. +- **A runtime directory is only trusted with its completion marker.** A CUDA + variant has two archives. An interrupted install can leave a directory holding + llama-server.exe but none of its CUDA DLLs, which would look installed and + then fail at launch with a missing-DLL error. The marker is written only after + every archive extracted and the executable was found; without it the directory + is torn down and rebuilt. +- **Archives are deleted as they extract.** A CUDA pair is close to 800 MB and + keeping both would double peak disk use for no benefit. +- **The server executable is located recursively.** Upstream has moved between a + flat layout and a build/bin layout across releases. +- **Free space is checked before starting, counting only missing shards**, so + resuming a mostly-complete model is not blocked by the full model size. + Unknown free space proceeds rather than refusing. + +Both managers keep the existing per-key single-flight gate +(`SingleFlightDownload.RunAsync`). Runtimes land under +`llama/runtimes//` and models under `llama/models//` in +the tray data directory, with upstream shard names preserved because llama.cpp +discovers the remaining shards by name from the first one. **Custom local build.** When `LocalInferenceCustomRuntimePath` is set it wins over the catalog: validate the path, skip download and hashing entirely, and show an explicit "custom build, not verified" state so the bypass is never silent. -GitHub does not publish release-asset hashes, so all nine `b10472` archives were -downloaded, size-checked against the releases API, and hashed on 2026-08-17. -Those values are pinned in `LlamaBackendCatalog` and guarded by -`AssetHashPinningTests`, so this phase is unblocked. See -`LOCAL_INFERENCE_ASSETS.md` for the provenance and its limits. +All nine `b10472` archives were downloaded, size-checked against the releases +API, and hashed on 2026-08-17; those values are pinned in `LlamaBackendCatalog` +and guarded by `AssetHashPinningTests`. See `LOCAL_INFERENCE_ASSETS.md` for the +provenance and its limits. ## Phase 3: server process and UI @@ -243,10 +269,14 @@ Unit tests in `tests/OpenClaw.Shared.Tests/Inference/`: - nvidia-smi parsing against captured real output, including the multi-GPU, `[N/A]` memory, and missing-banner cases. -Still to add in later phases: download managers against an `HttpMessageHandler` -fake (corrupt body rejected, `.tmp` deleted, nothing at the final path, -concurrent callers coalesced), zip traversal rejection, and the provider patch -builder preserving unrelated config while blocking on a redaction sentinel. +Phase 2 adds, against an in-memory `HttpMessageHandler` fake: tampered bodies +and length disagreements rejected with no residue, resume via `Range`, clean +restart when a server ignores `Range`, truncated-then-retried downloads, +monotonic aggregate progress, traversal and sibling-prefix rejection in zip +extraction, interrupted-install rebuild, and the free-space precheck. + +Still to add: the provider patch builder preserving unrelated config while +blocking on a redaction sentinel. Required repo validation per `AGENTS.md`: `./build.ps1`, then the Shared and Tray test projects. In this linked worktree, set `OPENCLAW_REPO_ROOT` first or diff --git a/src/OpenClaw.Shared/Inference/GgufModelManager.cs b/src/OpenClaw.Shared/Inference/GgufModelManager.cs new file mode 100644 index 000000000..d4fbde7c4 --- /dev/null +++ b/src/OpenClaw.Shared/Inference/GgufModelManager.cs @@ -0,0 +1,275 @@ +using System; +using System.Collections.Concurrent; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using OpenClaw.Shared.Audio; + +namespace OpenClaw.Shared.Inference; + +/// +/// On-disk state of a catalog model. +/// +/// Every shard is present at its final path. +/// Total bytes of the shards that are present. +/// Total size of the model when complete. +/// How many shards are fully downloaded. +/// How many shards the model has. +public sealed record LocalModelDownloadState( + bool IsComplete, + long BytesOnDisk, + long TotalBytes, + int ShardsPresent, + int ShardCount) +{ + /// Bytes still to fetch. Zero when complete. + public long RemainingBytes => Math.Max(0, TotalBytes - BytesOnDisk); +} + +/// +/// Downloads and manages GGUF checkpoints. +/// +/// Layout: <data>\llama\models\<model-id>\, with every +/// shard keeping its upstream file name. llama.cpp is launched against shard one +/// and discovers the rest by name, so the names and the directory grouping are +/// load-bearing rather than cosmetic. +/// +/// Same fail-closed contract as the audio managers: no pinned hash, no +/// download; verification happens before a shard reaches its final name; a +/// failure leaves no partial file behind. +/// +public sealed class GgufModelManager +{ + /// + /// Slack required on top of the model size before a download starts. Filling + /// a disk to the last byte breaks the whole machine, and a 155 GB download + /// that dies at 90 percent on a full disk is the worst possible outcome. + /// + private const long FreeSpaceMarginBytes = 2L * 1024 * 1024 * 1024; + + private static readonly ConcurrentDictionary> InFlightDownloads = + new(StringComparer.OrdinalIgnoreCase); + + private readonly string _modelsDirectory; + private readonly IOpenClawLogger _logger; + private readonly VerifiedFileDownloader _downloader; + private readonly Func _freeSpaceProbe; + + /// Tray data directory; models go under llama\models. + /// Diagnostics sink. + /// Optional override so tests can inject a fake transport. + /// + /// Optional override returning free bytes for a path, or null when unknown. + /// Injectable so the precheck can be exercised without filling a real disk. + /// + public GgufModelManager( + string dataDirectory, + IOpenClawLogger logger, + VerifiedFileDownloader? downloader = null, + Func? freeSpaceProbe = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(dataDirectory); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _modelsDirectory = Path.Combine(dataDirectory, "llama", "models"); + _downloader = downloader ?? new VerifiedFileDownloader(logger); + _freeSpaceProbe = freeSpaceProbe ?? DefaultFreeSpaceProbe; + Directory.CreateDirectory(_modelsDirectory); + } + + /// Directory a model downloads into, whether or not it exists yet. + public string GetModelDirectory(LocalModelInfo model) + { + ArgumentNullException.ThrowIfNull(model); + return Path.Combine(_modelsDirectory, model.Id); + } + + /// + /// Path llama-server should be launched against: the model's first shard. + /// Returns null when the model has no shards (an unpublished checkpoint). + /// + public string? GetPrimaryShardPath(LocalModelInfo model) + { + ArgumentNullException.ThrowIfNull(model); + return model.PrimaryShard is { } shard + ? Path.Combine(GetModelDirectory(model), shard.FileName) + : null; + } + + /// Inspect what is on disk for a model. + public LocalModelDownloadState GetState(LocalModelInfo model) + { + ArgumentNullException.ThrowIfNull(model); + + var directory = GetModelDirectory(model); + var present = 0; + var bytes = 0L; + + foreach (var shard in model.Shards) + { + var path = Path.Combine(directory, shard.FileName); + if (!File.Exists(path)) continue; + + present++; + // The stored size, not the on-disk length: a shard only reaches its + // final name after passing both the length and hash checks, so the + // two agree and the catalog value avoids a stat per shard. + bytes += shard.SizeBytes; + } + + return new LocalModelDownloadState( + IsComplete: model.Shards.Count > 0 && present == model.Shards.Count, + BytesOnDisk: bytes, + TotalBytes: model.TotalSizeBytes, + ShardsPresent: present, + ShardCount: model.Shards.Count); + } + + /// True when every shard of the model is present. + public bool IsDownloaded(LocalModelInfo model) => GetState(model).IsComplete; + + /// + /// Download every missing shard of a model, verifying each one. + /// Concurrent calls for the same model share a single download. + /// + /// Catalog entry. + /// Aggregate bytes downloaded and total across all shards. + public Task DownloadAsync( + LocalModelInfo model, + IProgress<(long downloaded, long total)>? progress = null, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(model); + + if (IsDownloaded(model)) + { + _logger.Info($"[Inference] Model '{model.Id}' already downloaded"); + progress?.Report((model.TotalSizeBytes, model.TotalSizeBytes)); + return Task.CompletedTask; + } + + return SingleFlightDownload.RunAsync( + InFlightDownloads, + model.Id, + token => DownloadCoreAsync(model, progress, token), + cancellationToken); + } + + private async Task DownloadCoreAsync( + LocalModelInfo model, + IProgress<(long downloaded, long total)>? progress, + CancellationToken cancellationToken) + { + // SECURITY: fail closed. An unpublished checkpoint has no shards and an + // unpinned one has no hash; neither may be fetched. + if (!model.IsDownloadable) + { + throw new InvalidOperationException( + model.Shards.Count == 0 + ? $"Model '{model.Id}' has no published checkpoint yet; nothing to download." + : $"Model '{model.Id}' has shards without a pinned SHA-256; refusing to download."); + } + + var directory = GetModelDirectory(model); + Directory.CreateDirectory(directory); + + EnsureEnoughFreeSpace(model, directory); + + // Shards already on disk count toward progress from the start, so a + // resumed multi-shard download does not appear to restart at zero. + var completedBytes = model.Shards + .Where(s => File.Exists(Path.Combine(directory, s.FileName))) + .Sum(s => s.SizeBytes); + + progress?.Report((completedBytes, model.TotalSizeBytes)); + + foreach (var shard in model.Shards) + { + cancellationToken.ThrowIfCancellationRequested(); + + var destination = Path.Combine(directory, shard.FileName); + if (File.Exists(destination)) continue; + + var baseBytes = completedBytes; + var shardProgress = progress is null + ? null + : new InlineProgress(bytes => progress.Report((baseBytes + bytes, model.TotalSizeBytes))); + + await _downloader.DownloadAsync( + new VerifiedDownloadRequest( + shard.DownloadUrl, + destination, + shard.Sha256, + shard.SizeBytes, + // Shards run to tens of gigabytes; losing one to a dropped + // connection has to be recoverable. + AllowResume: true, + DisplayName: $"{model.Id}/{shard.FileName}"), + shardProgress, + cancellationToken).ConfigureAwait(false); + + completedBytes += shard.SizeBytes; + progress?.Report((completedBytes, model.TotalSizeBytes)); + } + + _logger.Info($"[Inference] Model '{model.Id}' downloaded and verified ({model.Shards.Count} shard(s))"); + } + + /// + /// Refuse to start when the volume cannot hold what is left to download. + /// Only the missing shards are counted, so resuming a mostly-complete model + /// is not blocked by the full model size. + /// + private void EnsureEnoughFreeSpace(LocalModelInfo model, string directory) + { + var required = model.Shards + .Where(s => !File.Exists(Path.Combine(directory, s.FileName))) + .Sum(s => s.SizeBytes) + FreeSpaceMarginBytes; + + var available = _freeSpaceProbe(directory); + if (available is null) + { + // Unknown free space is not a reason to refuse; the download will + // fail loudly on a full disk instead. + _logger.Debug($"[Inference] Free space for '{directory}' is unknown; skipping the precheck"); + return; + } + + if (available < required) + { + throw new IOException( + $"Model '{model.Id}' needs about {FormatGib(required)} free on the target volume " + + $"but only {FormatGib(available.Value)} is available."); + } + } + + /// Delete every downloaded shard of a model. Returns false when nothing was present. + public bool Delete(LocalModelInfo model) + { + ArgumentNullException.ThrowIfNull(model); + + var directory = GetModelDirectory(model); + if (!Directory.Exists(directory)) return false; + + Directory.Delete(directory, recursive: true); + _logger.Info($"[Inference] Deleted model '{model.Id}'"); + return true; + } + + private static long? DefaultFreeSpaceProbe(string path) + { + try + { + var root = Path.GetPathRoot(Path.GetFullPath(path)); + return string.IsNullOrEmpty(root) ? null : new DriveInfo(root).AvailableFreeSpace; + } + catch (Exception) + { + // Network paths and unusual mounts can throw here. Unknown is a valid + // answer; the caller skips the precheck rather than failing. + return null; + } + } + + private static string FormatGib(long bytes) => $"{bytes / (1024.0 * 1024 * 1024):F1} GB"; +} diff --git a/src/OpenClaw.Shared/Inference/InlineProgress.cs b/src/OpenClaw.Shared/Inference/InlineProgress.cs new file mode 100644 index 000000000..8f7e10bec --- /dev/null +++ b/src/OpenClaw.Shared/Inference/InlineProgress.cs @@ -0,0 +1,25 @@ +using System; + +namespace OpenClaw.Shared.Inference; + +/// +/// An that invokes its handler synchronously on the +/// reporting thread. +/// +/// posts each report to the captured +/// synchronization context, or to the thread pool when there is none, so two +/// reports can be delivered out of order. The download managers translate +/// per-file byte counts into a running aggregate, and an out-of-order delivery +/// there makes the aggregate jump backwards: a progress bar that visibly +/// rewinds, and in the worst case a "downloaded" figure that briefly exceeds or +/// undershoots reality. Forwarding inline keeps the sequence monotonic. +/// +/// The handler runs on whichever thread reported, so callers that touch UI +/// state are still responsible for marshalling. +/// +internal sealed class InlineProgress(Action handler) : IProgress +{ + private readonly Action _handler = handler ?? throw new ArgumentNullException(nameof(handler)); + + public void Report(T value) => _handler(value); +} diff --git a/src/OpenClaw.Shared/Inference/LlamaRuntimeManager.cs b/src/OpenClaw.Shared/Inference/LlamaRuntimeManager.cs new file mode 100644 index 000000000..fa8bf76fb --- /dev/null +++ b/src/OpenClaw.Shared/Inference/LlamaRuntimeManager.cs @@ -0,0 +1,322 @@ +using System; +using System.Collections.Concurrent; +using System.IO; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using OpenClaw.Shared.Audio; + +namespace OpenClaw.Shared.Inference; + +/// +/// Where a usable llama-server.exe came from. +/// +public enum LlamaRuntimeSource +{ + /// Extracted from a hash-verified archive in the pinned catalog. + Catalog = 0, + /// A build the user supplied. Not hash-verified, by design. + CustomBuild = 1, +} + +/// +/// A resolved llama.cpp runtime. +/// +/// Absolute path to llama-server.exe. +/// Whether this came from the catalog or a user-supplied build. +/// Catalog variant, or null for a custom build. +public sealed record LlamaRuntime( + string ServerExecutablePath, + LlamaRuntimeSource Source, + LlamaBackendVariant? Variant) +{ + /// Directory holding the server and its native dependencies. + public string Directory => Path.GetDirectoryName(ServerExecutablePath) ?? string.Empty; + + /// + /// True when this runtime bypassed integrity verification. The UI must show + /// this state explicitly so the bypass is never silent. + /// + public bool IsUnverified => Source == LlamaRuntimeSource.CustomBuild; +} + +/// +/// Downloads, extracts, and resolves llama.cpp runtimes. +/// +/// Layout: <data>\llama\runtimes\<runtime-key>\. The key +/// includes the pinned release tag, so bumping the catalog installs alongside the +/// old runtime rather than half-overwriting a directory another process may be +/// running from. +/// +/// A CUDA variant carries two archives (the binaries and the CUDA runtime +/// redistributable) that both extract into that one directory. Both must land +/// before the runtime is usable, so the install is only marked complete after all +/// of them are extracted and the server executable is found. +/// +public sealed class LlamaRuntimeManager +{ + /// + /// Written into a runtime directory once every archive has been extracted. + /// Its absence means a previous install was interrupted, so the directory is + /// torn down and rebuilt rather than trusted. Without this marker a partial + /// extraction that happened to contain llama-server.exe but not its CUDA DLLs + /// would look installed and fail at launch. + /// + private const string CompletionMarkerName = ".install-complete"; + + private static readonly ConcurrentDictionary> InFlightInstalls = + new(StringComparer.OrdinalIgnoreCase); + + private readonly string _runtimesDirectory; + private readonly IOpenClawLogger _logger; + private readonly VerifiedFileDownloader _downloader; + + /// Tray data directory; runtimes go under llama\runtimes. + /// Diagnostics sink. + /// Optional override so tests can inject a fake transport. + public LlamaRuntimeManager( + string dataDirectory, + IOpenClawLogger logger, + VerifiedFileDownloader? downloader = null) + { + ArgumentException.ThrowIfNullOrWhiteSpace(dataDirectory); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _runtimesDirectory = Path.Combine(dataDirectory, "llama", "runtimes"); + _downloader = downloader ?? new VerifiedFileDownloader(logger); + Directory.CreateDirectory(_runtimesDirectory); + } + + /// Directory a variant installs into, whether or not it exists yet. + public string GetRuntimeDirectory(LlamaBackendVariant variant) + { + ArgumentNullException.ThrowIfNull(variant); + return Path.Combine(_runtimesDirectory, variant.RuntimeKey); + } + + /// + /// True when the variant is fully installed: the completion marker is present + /// and the server executable exists. + /// + public bool IsInstalled(LlamaBackendVariant variant) + { + var directory = GetRuntimeDirectory(variant); + return File.Exists(Path.Combine(directory, CompletionMarkerName)) + && TryFindServerExecutable(directory) is not null; + } + + /// + /// Resolve a user-supplied build without downloading anything. + /// + /// + /// Path to llama-server.exe or to a directory containing it. + /// + /// No server executable at that path. + public LlamaRuntime ResolveCustomBuild(string customPath) + { + ArgumentException.ThrowIfNullOrWhiteSpace(customPath); + + var executable = Directory.Exists(customPath) + ? TryFindServerExecutable(customPath) + : File.Exists(customPath) ? customPath : null; + + if (executable is null) + { + throw new FileNotFoundException( + $"No {LlamaBackendCatalog.ServerExecutableName} found at '{customPath}'. " + + "Point the custom build setting at the executable or the directory containing it."); + } + + if (!Path.GetFileName(executable).Equals(LlamaBackendCatalog.ServerExecutableName, StringComparison.OrdinalIgnoreCase)) + { + throw new FileNotFoundException( + $"'{Path.GetFileName(executable)}' is not {LlamaBackendCatalog.ServerExecutableName}."); + } + + // Deliberately no hash check: this binary is the user's own. The caller + // is responsible for surfacing LlamaRuntime.IsUnverified in the UI. + _logger.Warn($"[Inference] Using a custom llama.cpp build at '{executable}'. It is not integrity verified."); + return new LlamaRuntime(Path.GetFullPath(executable), LlamaRuntimeSource.CustomBuild, null); + } + + /// + /// Ensure a variant is installed, downloading and extracting its archives if + /// needed, and return the resolved runtime. + /// Concurrent calls for the same variant share one install. + /// + /// Variant from . + /// Aggregate bytes downloaded and total across all archives. + public async Task EnsureInstalledAsync( + LlamaBackendVariant variant, + IProgress<(long downloaded, long total)>? progress = null, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(variant); + + if (!IsInstalled(variant)) + { + await SingleFlightDownload.RunAsync( + InFlightInstalls, + GetRuntimeDirectory(variant), + token => InstallCoreAsync(variant, progress, token), + cancellationToken).ConfigureAwait(false); + } + + var directory = GetRuntimeDirectory(variant); + var executable = TryFindServerExecutable(directory) + ?? throw new InvalidOperationException( + $"Runtime '{variant.RuntimeKey}' installed but {LlamaBackendCatalog.ServerExecutableName} was not found under '{directory}'."); + + return new LlamaRuntime(executable, LlamaRuntimeSource.Catalog, variant); + } + + private async Task InstallCoreAsync( + LlamaBackendVariant variant, + IProgress<(long downloaded, long total)>? progress, + CancellationToken cancellationToken) + { + // SECURITY: fail closed on an unpinned variant. VerifiedFileDownloader + // enforces this per asset too; checking up front avoids downloading one + // archive of a pair before discovering the other is unverifiable. + if (!variant.IsDownloadable) + { + throw new InvalidOperationException( + $"llama.cpp variant '{variant.RuntimeKey}' has assets without a pinned SHA-256; refusing to install."); + } + + var directory = GetRuntimeDirectory(variant); + + // A previous interrupted attempt leaves a directory with no completion + // marker. Rebuild from scratch: reusing it risks mixing archives from + // two different releases. + if (Directory.Exists(directory)) DeleteDirectory(directory); + Directory.CreateDirectory(directory); + + var archivesDirectory = Path.Combine(directory, ".archives"); + Directory.CreateDirectory(archivesDirectory); + + try + { + var totalBytes = variant.ApproximateSizeBytes; + var completedBytes = 0L; + + foreach (var asset in variant.Assets) + { + cancellationToken.ThrowIfCancellationRequested(); + + var archivePath = Path.Combine(archivesDirectory, asset.FileName); + var baseBytes = completedBytes; + var assetProgress = progress is null + ? null + : new InlineProgress(bytes => progress.Report((baseBytes + bytes, totalBytes))); + + await _downloader.DownloadAsync( + new VerifiedDownloadRequest( + asset.DownloadUrl, + archivePath, + asset.Sha256, + asset.ApproximateSizeBytes, + AllowResume: false, + DisplayName: asset.FileName), + assetProgress, + cancellationToken).ConfigureAwait(false); + + completedBytes += asset.ApproximateSizeBytes; + progress?.Report((completedBytes, totalBytes)); + + _logger.Info($"[Inference] Extracting '{asset.FileName}'"); + SafeZipExtractor.ExtractTo(archivePath, directory, cancellationToken); + + // Free the archive as we go. A CUDA pair is close to 800 MB and + // keeping both around doubles the peak disk requirement for no + // benefit once extraction succeeded. + TryDeleteFile(archivePath); + } + + if (TryFindServerExecutable(directory) is null) + { + throw new InvalidOperationException( + $"Archives for '{variant.RuntimeKey}' extracted but contained no {LlamaBackendCatalog.ServerExecutableName}."); + } + + TryDeleteDirectory(archivesDirectory); + await File.WriteAllTextAsync( + Path.Combine(directory, CompletionMarkerName), + variant.RuntimeKey, + cancellationToken).ConfigureAwait(false); + + _logger.Info($"[Inference] Runtime '{variant.RuntimeKey}' installed"); + } + catch + { + // Leave nothing half-installed: without the completion marker the + // directory would be rebuilt anyway, and a stale tree wastes disk. + DeleteDirectorySafely(directory); + throw; + } + } + + /// Delete an installed runtime. Returns false when it was not present. + public bool Uninstall(LlamaBackendVariant variant) + { + var directory = GetRuntimeDirectory(variant); + if (!Directory.Exists(directory)) return false; + + DeleteDirectory(directory); + _logger.Info($"[Inference] Removed runtime '{variant.RuntimeKey}'"); + return true; + } + + /// + /// Locate llama-server.exe under a directory. Searched recursively + /// because upstream archives have changed between a flat layout and a + /// build\bin layout across releases. + /// + private static string? TryFindServerExecutable(string directory) + { + if (!Directory.Exists(directory)) return null; + + var direct = Path.Combine(directory, LlamaBackendCatalog.ServerExecutableName); + if (File.Exists(direct)) return direct; + + try + { + return Directory + .EnumerateFiles(directory, LlamaBackendCatalog.ServerExecutableName, SearchOption.AllDirectories) + .FirstOrDefault(); + } + catch (Exception) + { + // An unreadable subtree means "not found" for our purposes; the + // caller turns that into an explicit install failure. + return null; + } + } + + private static void DeleteDirectory(string directory) => Directory.Delete(directory, recursive: true); + + private void DeleteDirectorySafely(string directory) + { + try + { + if (Directory.Exists(directory)) DeleteDirectory(directory); + } + catch (Exception ex) + { + _logger.Debug($"[Inference] Could not clean up '{directory}': {ex.Message}"); + } + } + + private void TryDeleteDirectory(string directory) => DeleteDirectorySafely(directory); + + private void TryDeleteFile(string path) + { + try + { + if (File.Exists(path)) File.Delete(path); + } + catch (Exception ex) + { + _logger.Debug($"[Inference] Could not delete archive '{Path.GetFileName(path)}': {ex.Message}"); + } + } +} diff --git a/src/OpenClaw.Shared/Inference/SafeZipExtractor.cs b/src/OpenClaw.Shared/Inference/SafeZipExtractor.cs new file mode 100644 index 000000000..069925bcf --- /dev/null +++ b/src/OpenClaw.Shared/Inference/SafeZipExtractor.cs @@ -0,0 +1,71 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.IO.Compression; +using System.Threading; + +namespace OpenClaw.Shared.Inference; + +/// +/// Extracts a zip archive with a path-traversal guard. +/// +/// already rejects +/// escaping entries on modern .NET, but these archives carry native executables +/// that we then launch, so the check is made explicit and testable here rather +/// than inherited from a framework implementation detail. An entry that resolves +/// outside the destination aborts the whole extraction. +/// +public static class SafeZipExtractor +{ + /// + /// Extract every entry of into + /// , overwriting existing files. + /// + /// Relative paths of the files written, in archive order. + /// + /// An entry resolves outside the destination directory. + /// + public static IReadOnlyList ExtractTo( + string archivePath, + string destinationDirectory, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(archivePath); + ArgumentException.ThrowIfNullOrWhiteSpace(destinationDirectory); + + Directory.CreateDirectory(destinationDirectory); + + // Trailing separator matters: without it "C:\dir" would be accepted as a + // prefix of "C:\dir-evil\payload.exe". + var destinationRoot = Path.TrimEndingDirectorySeparator(Path.GetFullPath(destinationDirectory)) + + Path.DirectorySeparatorChar; + + var written = new List(); + using var archive = ZipFile.OpenRead(archivePath); + + foreach (var entry in archive.Entries) + { + cancellationToken.ThrowIfCancellationRequested(); + + // A directory entry has an empty Name. Nothing to write; the file + // entries below create their own parent directories. + if (entry.Name.Length == 0) continue; + + var targetPath = Path.GetFullPath(Path.Combine(destinationDirectory, entry.FullName)); + if (!targetPath.StartsWith(destinationRoot, StringComparison.OrdinalIgnoreCase)) + { + throw new InvalidDataException( + $"Archive entry '{entry.FullName}' resolves outside the destination directory. " + + "Refusing to extract."); + } + + var parent = Path.GetDirectoryName(targetPath); + if (!string.IsNullOrEmpty(parent)) Directory.CreateDirectory(parent); + + entry.ExtractToFile(targetPath, overwrite: true); + written.Add(Path.GetRelativePath(destinationDirectory, targetPath)); + } + + return written; + } +} diff --git a/src/OpenClaw.Shared/Inference/VerifiedFileDownloader.cs b/src/OpenClaw.Shared/Inference/VerifiedFileDownloader.cs new file mode 100644 index 000000000..8f0ed5061 --- /dev/null +++ b/src/OpenClaw.Shared/Inference/VerifiedFileDownloader.cs @@ -0,0 +1,277 @@ +using System; +using System.IO; +using System.Net; +using System.Net.Http; +using System.Net.Http.Headers; +using System.Security; +using System.Security.Cryptography; +using System.Threading; +using System.Threading.Tasks; + +namespace OpenClaw.Shared.Inference; + +/// +/// One file to fetch and verify. +/// +/// HTTPS source. +/// Final path. Written only after verification. +/// +/// Pinned lowercase hex SHA-256. Required: a null or blank value is a hard +/// failure, so an unpinned asset can never install. +/// +/// Exact expected length, or 0 when unknown. +/// +/// Whether a partial .part file may be continued with a range request. +/// Worth it for the multi-gigabyte GGUF shards, pointless for small archives. +/// +/// Name used in log and error messages. +public sealed record VerifiedDownloadRequest( + string Url, + string DestinationPath, + string? Sha256, + long ExpectedSizeBytes = 0, + bool AllowResume = false, + string? DisplayName = null) +{ + /// Label for diagnostics; falls back to the destination file name. + public string Label => DisplayName ?? Path.GetFileName(DestinationPath); +} + +/// +/// Downloads a file to a temporary .part path, verifies its SHA-256, and +/// only then moves it into place. +/// +/// Fail closed. No pinned hash means no download. A hash mismatch +/// deletes the partial file and throws; nothing unverified ever reaches the +/// destination path. The error deliberately does not echo the computed hash, +/// which would hand an attacker a confirmation oracle. This mirrors the audio +/// asset managers; see docs/LOCAL_INFERENCE_ASSETS.md. +/// +/// Resume. With , +/// an existing .part shorter than the expected size is continued with a +/// Range request. Restarting a 50 GB shard from zero after a dropped +/// connection is not an acceptable failure mode. A server that ignores the range +/// (answers 200 instead of 206) restarts the file rather than corrupting it, and +/// a resumed file whose bytes turn out to be bad still fails the hash check and +/// is deleted, so a retry starts clean instead of looping. +/// +public sealed class VerifiedFileDownloader +{ + private const int BufferSize = 81920; + + private readonly IOpenClawLogger _logger; + private readonly Func _httpClientFactory; + + /// Diagnostics sink. + /// + /// Optional override so tests can inject a fake handler. Each call gets a + /// client that the downloader disposes. + /// + public VerifiedFileDownloader(IOpenClawLogger logger, Func? httpClientFactory = null) + { + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _httpClientFactory = httpClientFactory ?? CreateDefaultClient; + } + + private static HttpClient CreateDefaultClient() => + // Large weights over a slow link legitimately take hours. The timeout + // that matters is enforced by the caller's cancellation token, not here. + new() { Timeout = Timeout.InfiniteTimeSpan }; + + /// + /// Fetch and verify one file. Does nothing if the destination already exists. + /// + /// What to fetch. + /// + /// Optional progress reporting total bytes present for this file so far, + /// including any resumed prefix. Callers aggregating several files can sum + /// these directly. + /// + public async Task DownloadAsync( + VerifiedDownloadRequest request, + IProgress? bytesCompleted = null, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(request); + + // SECURITY: refuse anything without a pinned hash before touching the + // network. This is the single gate that keeps an unverifiable catalog + // entry from installing. + if (string.IsNullOrWhiteSpace(request.Sha256)) + { + throw new InvalidOperationException( + $"Asset '{request.Label}' has no pinned SHA-256; refusing to download. " + + "Add a verified hash to the catalog first."); + } + + if (File.Exists(request.DestinationPath)) + { + _logger.Info($"[Inference] '{request.Label}' already present"); + bytesCompleted?.Report(new FileInfo(request.DestinationPath).Length); + return; + } + + var directory = Path.GetDirectoryName(request.DestinationPath); + if (!string.IsNullOrEmpty(directory)) Directory.CreateDirectory(directory); + + var partPath = request.DestinationPath + ".part"; + + try + { + await FetchToPartFileAsync(request, partPath, bytesCompleted, cancellationToken).ConfigureAwait(false); + + if (request.ExpectedSizeBytes > 0) + { + var actualSize = new FileInfo(partPath).Length; + if (actualSize != request.ExpectedSizeBytes) + { + throw new InvalidDataException( + $"Asset '{request.Label}' downloaded {actualSize} bytes but the catalog expects " + + $"{request.ExpectedSizeBytes}. The partial file was discarded."); + } + } + + await VerifyHashAsync(partPath, request.Sha256!, request.Label, cancellationToken).ConfigureAwait(false); + + File.Move(partPath, request.DestinationPath, overwrite: true); + _logger.Info($"[Inference] '{request.Label}' downloaded and verified"); + } + catch + { + // Any failure discards the partial file. Keeping a mismatched or + // truncated .part would make the next resume attempt repeat the same + // failure forever. + TryDelete(partPath); + throw; + } + } + + private async Task FetchToPartFileAsync( + VerifiedDownloadRequest request, + string partPath, + IProgress? bytesCompleted, + CancellationToken cancellationToken) + { + var resumeFrom = ResolveResumeOffset(request, partPath); + + using var httpClient = _httpClientFactory(); + using var httpRequest = new HttpRequestMessage(HttpMethod.Get, request.Url); + if (resumeFrom > 0) + { + httpRequest.Headers.Range = new RangeHeaderValue(resumeFrom, null); + _logger.Info($"[Inference] Resuming '{request.Label}' at {resumeFrom} bytes"); + } + + using var response = await httpClient + .SendAsync(httpRequest, HttpCompletionOption.ResponseHeadersRead, cancellationToken) + .ConfigureAwait(false); + response.EnsureSuccessStatusCode(); + + // A server that ignores the Range header answers 200 with the whole body. + // Appending that to the existing prefix would silently corrupt the file, + // so fall back to a clean restart instead. + var appending = resumeFrom > 0 && response.StatusCode == HttpStatusCode.PartialContent; + if (resumeFrom > 0 && !appending) + { + _logger.Warn($"[Inference] Range request for '{request.Label}' was not honored; restarting the download"); + resumeFrom = 0; + } + + var written = appending ? resumeFrom : 0L; + bytesCompleted?.Report(written); + + await using var contentStream = await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false); + await using (var fileStream = new FileStream( + partPath, + appending ? FileMode.Append : FileMode.Create, + FileAccess.Write, + FileShare.None, + BufferSize)) + { + var buffer = new byte[BufferSize]; + int read; + while ((read = await contentStream.ReadAsync(buffer, cancellationToken).ConfigureAwait(false)) > 0) + { + await fileStream.WriteAsync(buffer.AsMemory(0, read), cancellationToken).ConfigureAwait(false); + written += read; + bytesCompleted?.Report(written); + } + + await fileStream.FlushAsync(cancellationToken).ConfigureAwait(false); + } + } + + /// + /// Byte offset to resume from, or 0 for a fresh download. Only a partial file + /// strictly shorter than the known expected size is resumable: a longer or + /// equal-length .part means the previous attempt already failed + /// verification, so it is discarded rather than continued. + /// + private long ResolveResumeOffset(VerifiedDownloadRequest request, string partPath) + { + if (!File.Exists(partPath)) return 0; + + if (!request.AllowResume || request.ExpectedSizeBytes <= 0) + { + TryDelete(partPath); + return 0; + } + + long existing; + try + { + existing = new FileInfo(partPath).Length; + } + catch (Exception ex) + { + _logger.Debug($"[Inference] Could not stat partial file for '{request.Label}': {ex.Message}"); + TryDelete(partPath); + return 0; + } + + if (existing <= 0 || existing >= request.ExpectedSizeBytes) + { + TryDelete(partPath); + return 0; + } + + return existing; + } + + /// + /// Compare the file's SHA-256 to the pinned value. Throws on mismatch without + /// echoing the computed hash, which would confirm to an attacker how close a + /// forgery got. + /// + private static async Task VerifyHashAsync( + string filePath, + string expectedHex, + string label, + CancellationToken cancellationToken) + { + using var sha = SHA256.Create(); + await using var stream = new FileStream( + filePath, FileMode.Open, FileAccess.Read, FileShare.Read, BufferSize, useAsync: true); + + var actual = await sha.ComputeHashAsync(stream, cancellationToken).ConfigureAwait(false); + var actualHex = Convert.ToHexString(actual).ToLowerInvariant(); + + if (!string.Equals(actualHex, expectedHex, StringComparison.OrdinalIgnoreCase)) + { + throw new SecurityException( + $"Asset '{label}' failed its integrity check. The downloaded file does not match the pinned SHA-256."); + } + } + + private void TryDelete(string path) + { + try + { + if (File.Exists(path)) File.Delete(path); + } + catch (Exception ex) + { + _logger.Debug($"[Inference] Could not delete '{Path.GetFileName(path)}': {ex.Message}"); + } + } +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/FakeHttpTransport.cs b/tests/OpenClaw.Shared.Tests/Inference/FakeHttpTransport.cs new file mode 100644 index 000000000..bd7866d5e --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/FakeHttpTransport.cs @@ -0,0 +1,95 @@ +using System; +using System.Collections.Generic; +using System.Net; +using System.Net.Http; +using System.Security.Cryptography; +using System.Threading; +using System.Threading.Tasks; + +namespace OpenClaw.Shared.Tests.Inference; + +/// +/// In-memory HTTP transport for the download managers: serves registered byte +/// bodies, honours (or deliberately ignores) Range requests, and can be told to +/// truncate a response mid-body so resume and integrity paths are exercisable +/// without a network. +/// +internal sealed class FakeHttpTransport : HttpMessageHandler +{ + private readonly Dictionary _entries = new(StringComparer.OrdinalIgnoreCase); + + /// Every absolute URL requested, in order, including repeats. + public List Requests { get; } = []; + + /// Range header values seen, in order. Null entries mean no Range header. + public List RangeHeaders { get; } = []; + + public void Add(string url, byte[] body, bool supportsRange = true, int? truncateAfterBytes = null) => + _entries[url] = new Entry(body, supportsRange, truncateAfterBytes); + + /// Stop truncating a previously-truncated entry, so a retry succeeds. + public void HealTruncation(string url) => + _entries[url] = _entries[url] with { TruncateAfterBytes = null }; + + protected override Task SendAsync( + HttpRequestMessage request, + CancellationToken cancellationToken) + { + var url = request.RequestUri!.ToString(); + Requests.Add(url); + RangeHeaders.Add(request.Headers.Range?.ToString()); + + if (!_entries.TryGetValue(url, out var entry)) + return Task.FromResult(new HttpResponseMessage(HttpStatusCode.NotFound)); + + long? rangeFrom = null; + if (request.Headers.Range is { } rangeHeader) + { + foreach (var range in rangeHeader.Ranges) + { + rangeFrom = range.From; + break; + } + } + + byte[] body; + HttpStatusCode status; + + if (rangeFrom is { } offset && entry.SupportsRange) + { + if (offset > entry.Body.Length) + return Task.FromResult(new HttpResponseMessage(HttpStatusCode.RequestedRangeNotSatisfiable)); + + body = entry.Body[(int)offset..]; + status = HttpStatusCode.PartialContent; + } + else + { + // Either no range was asked for, or this entry pretends not to + // understand Range and answers with the whole body. + body = entry.Body; + status = HttpStatusCode.OK; + } + + if (entry.TruncateAfterBytes is { } limit && body.Length > limit) + body = body[..limit]; + + return Task.FromResult(new HttpResponseMessage(status) { Content = new ByteArrayContent(body) }); + } + + /// Factory to hand to the downloader under test. + public Func ClientFactory => () => new HttpClient(this, disposeHandler: false); + + /// Deterministic pseudo-random body plus its SHA-256, for catalog fixtures. + public static (byte[] Body, string Sha256) MakeBody(int length, int seed) + { + var body = new byte[length]; + new Random(seed).NextBytes(body); + return (body, Convert.ToHexString(SHA256.HashData(body)).ToLowerInvariant()); + } + + public static string Sha256Of(byte[] body) => + Convert.ToHexString(SHA256.HashData(body)).ToLowerInvariant(); + + private sealed record Entry(byte[] Body, bool SupportsRange, int? TruncateAfterBytes); +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/GgufModelManagerTests.cs b/tests/OpenClaw.Shared.Tests/Inference/GgufModelManagerTests.cs new file mode 100644 index 000000000..367688976 --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/GgufModelManagerTests.cs @@ -0,0 +1,324 @@ +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Runtime.CompilerServices; +using System.Threading.Tasks; +using OpenClaw.Shared; +using OpenClaw.Shared.Inference; +using OpenClaw.TestSupport; +using Xunit; + +namespace OpenClaw.Shared.Tests.Inference; + +public class GgufModelManagerTests +{ + private const long Gib = 1024L * 1024 * 1024; + + [Fact] + public async Task DownloadsEveryShardOfAMultiShardModel() + { + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [500, 700, 300]); + var manager = NewManager(temp, transport); + + await manager.DownloadAsync(model); + + Assert.True(manager.IsDownloaded(model)); + var directory = manager.GetModelDirectory(model); + foreach (var shard in model.Shards) + Assert.True(File.Exists(Path.Combine(directory, shard.FileName)), $"{shard.FileName} missing"); + } + + [Fact] + public async Task KeepsUpstreamShardFileNames() + { + // llama.cpp finds the remaining shards by name from the first one, so + // renaming them on disk would break loading. + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [100, 100]); + var manager = NewManager(temp, transport); + + await manager.DownloadAsync(model); + + var onDisk = Directory.GetFiles(manager.GetModelDirectory(model)).Select(Path.GetFileName).Order().ToArray(); + Assert.Equal(model.Shards.Select(s => s.FileName).Order().ToArray(), onDisk); + } + + [Fact] + public async Task PrimaryShardPathPointsAtShardOne() + { + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [100, 100, 100]); + var manager = NewManager(temp, transport); + await manager.DownloadAsync(model); + + var primary = manager.GetPrimaryShardPath(model); + + Assert.NotNull(primary); + Assert.Equal(model.Shards[0].FileName, Path.GetFileName(primary)); + Assert.True(File.Exists(primary)); + } + + [Fact] + public async Task ResumesAfterAPartiallyDownloadedModelAndSkipsPresentShards() + { + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [200, 300, 400]); + var manager = NewManager(temp, transport); + + // Pretend shard 1 landed during an earlier run. + var directory = manager.GetModelDirectory(model); + Directory.CreateDirectory(directory); + File.WriteAllBytes(Path.Combine(directory, model.Shards[0].FileName), new byte[200]); + + var state = manager.GetState(model); + Assert.False(state.IsComplete); + Assert.Equal(1, state.ShardsPresent); + Assert.Equal(200, state.BytesOnDisk); + Assert.Equal(700, state.RemainingBytes); + + await manager.DownloadAsync(model); + + Assert.True(manager.IsDownloaded(model)); + // Only the two missing shards were fetched. + Assert.Equal(2, transport.Requests.Count); + } + + [Fact] + public async Task RefusesAModelWithNoPublishedCheckpoint() + { + using var temp = new TempDirectory(); + var transport = new FakeHttpTransport(); + var unpublished = new LocalModelInfo("pending-refuses-download", "Pending model", [], ["--temp", "1.0"], 1); + + var ex = await Assert.ThrowsAsync(() => + NewManager(temp, transport).DownloadAsync(unpublished)); + + Assert.Contains("no published checkpoint", ex.Message, StringComparison.OrdinalIgnoreCase); + Assert.Empty(transport.Requests); + } + + [Fact] + public async Task RefusesAModelWithAnUnpinnedShard() + { + using var temp = new TempDirectory(); + var transport = new FakeHttpTransport(); + var unpinned = new LocalModelInfo( + "unpinned", + "Unpinned model", + [new GgufShard("a.gguf", "https://example.test/a.gguf", 10, null)], + ["--temp", "1.0"], + 1); + + var ex = await Assert.ThrowsAsync(() => + NewManager(temp, transport).DownloadAsync(unpinned)); + + Assert.Contains("pinned SHA-256", ex.Message, StringComparison.OrdinalIgnoreCase); + Assert.Empty(transport.Requests); + } + + [Fact] + public async Task RefusesToStartWhenTheVolumeCannotHoldTheModel() + { + // A 155 GB download that dies at 90 percent on a full disk is the worst + // possible outcome, so the check happens before any bytes move. + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [1000, 1000]); + + var manager = new GgufModelManager( + temp.Path, + NullLogger.Instance, + new VerifiedFileDownloader(NullLogger.Instance, transport.ClientFactory), + freeSpaceProbe: _ => 100); + + var ex = await Assert.ThrowsAsync(() => manager.DownloadAsync(model)); + + Assert.Contains("free", ex.Message, StringComparison.OrdinalIgnoreCase); + Assert.Empty(transport.Requests); + } + + [Fact] + public async Task OnlyCountsMissingShardsInTheFreeSpaceCheck() + { + // Resuming a mostly-complete model must not be blocked by the full model + // size. Free space here is deliberately set below the whole model but + // above what the one missing shard still needs. + using var temp = new TempDirectory(); + var transport = new FakeHttpTransport(); + + var (smallBody, smallHash) = FakeHttpTransport.MakeBody(512, seed: 42); + const string smallUrl = "https://example.test/big-model-00002-of-00002.gguf"; + transport.Add(smallUrl, smallBody); + + // Shard one is declared huge but is already on disk, so it is never + // fetched and no body needs to exist for it. + var model = new LocalModelInfo( + "big-model", + "Big model", + [ + new GgufShard("big-model-00001-of-00002.gguf", "https://example.test/never-fetched.gguf", 8 * Gib, new string('a', 64)), + new GgufShard("big-model-00002-of-00002.gguf", smallUrl, smallBody.Length, smallHash), + ], + ["--temp", "1.0"], + MinimumRecommendedMemoryBytes: 8 * Gib); + + var manager = new GgufModelManager( + temp.Path, + NullLogger.Instance, + new VerifiedFileDownloader(NullLogger.Instance, transport.ClientFactory), + // Below the 8 GB model plus margin, above the 512 bytes plus margin. + freeSpaceProbe: _ => 3 * Gib); + + var directory = manager.GetModelDirectory(model); + Directory.CreateDirectory(directory); + await File.WriteAllBytesAsync(Path.Combine(directory, model.Shards[0].FileName), new byte[16]); + + await manager.DownloadAsync(model); + + Assert.True(manager.IsDownloaded(model)); + Assert.Equal(smallUrl, Assert.Single(transport.Requests)); + } + + [Fact] + public async Task ProceedsWhenFreeSpaceCannotBeDetermined() + { + // Network paths and unusual mounts cannot answer. Unknown must not be + // treated as "no space"; the download fails loudly later if it matters. + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [100]); + + var manager = new GgufModelManager( + temp.Path, + NullLogger.Instance, + new VerifiedFileDownloader(NullLogger.Instance, transport.ClientFactory), + freeSpaceProbe: _ => null); + + await manager.DownloadAsync(model); + + Assert.True(manager.IsDownloaded(model)); + } + + [Fact] + public async Task RejectsATamperedShardAndLeavesTheModelIncomplete() + { + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [100, 100]); + + // Serve the wrong bytes for the second shard. + transport.Add(model.Shards[1].DownloadUrl, new byte[100]); + + var manager = NewManager(temp, transport); + await Assert.ThrowsAsync(() => manager.DownloadAsync(model)); + + Assert.False(manager.IsDownloaded(model)); + var directory = manager.GetModelDirectory(model); + Assert.True(File.Exists(Path.Combine(directory, model.Shards[0].FileName))); + Assert.False(File.Exists(Path.Combine(directory, model.Shards[1].FileName))); + Assert.Empty(Directory.GetFiles(directory, "*.part")); + } + + [Fact] + public async Task SecondDownloadOfACompleteModelFetchesNothing() + { + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [100, 100]); + var manager = NewManager(temp, transport); + + await manager.DownloadAsync(model); + var afterFirst = transport.Requests.Count; + + await manager.DownloadAsync(model); + + Assert.Equal(afterFirst, transport.Requests.Count); + } + + [Fact] + public async Task ReportsAggregateProgressAcrossShardsEndingAtTheTotal() + { + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [1000, 2000, 500]); + + var reports = new List<(long downloaded, long total)>(); + var progress = new InlineProgress<(long downloaded, long total)>(reports.Add); + + await NewManager(temp, transport).DownloadAsync(model, progress); + + Assert.NotEmpty(reports); + Assert.All(reports, r => Assert.Equal(3500, r.total)); + Assert.Equal(3500, reports[^1].downloaded); + for (var i = 1; i < reports.Count; i++) + Assert.True(reports[i].downloaded >= reports[i - 1].downloaded, "Aggregate progress went backwards."); + } + + [Fact] + public async Task DeleteRemovesEveryShard() + { + using var temp = new TempDirectory(); + var (model, transport) = MakeModel(shardSizes: [100, 100]); + var manager = NewManager(temp, transport); + await manager.DownloadAsync(model); + + Assert.True(manager.Delete(model)); + Assert.False(manager.IsDownloaded(model)); + Assert.False(manager.Delete(model)); + } + + [Fact] + public void PrimaryShardPathIsNullForAnUnpublishedModel() + { + using var temp = new TempDirectory(); + var unpublished = new LocalModelInfo("pending-primary-shard", "Pending", [], ["--temp", "1.0"], 1); + + Assert.Null(NewManager(temp, new FakeHttpTransport()).GetPrimaryShardPath(unpublished)); + } + + /// + /// Build a synthetic multi-shard model plus a transport that serves it, with + /// upstream-style shard names so the ordering invariants are exercised. + /// + /// The model id is unique per call. GgufModelManager coalesces + /// concurrent downloads of the same id through a process-wide single-flight + /// gate whose completed entries are evicted by an asynchronous continuation, + /// so a shared id lets one test latch onto the previous test's finished task + /// and skip its own download. + /// + private static (LocalModelInfo Model, FakeHttpTransport Transport) MakeModel( + int[] shardSizes, + [CallerMemberName] string caller = "") + { + var transport = new FakeHttpTransport(); + var shards = new List(); + var id = $"test-model-{caller}"; + + for (var i = 0; i < shardSizes.Length; i++) + { + var name = $"{id}-{i + 1:00000}-of-{shardSizes.Length:00000}.gguf"; + var url = $"https://example.test/{name}"; + var (body, hash) = FakeHttpTransport.MakeBody(shardSizes[i], seed: 100 + i); + transport.Add(url, body); + shards.Add(new GgufShard(name, url, body.Length, hash)); + } + + var model = new LocalModelInfo( + id, + "Test model", + shards, + ["--temp", "1.0"], + MinimumRecommendedMemoryBytes: shardSizes.Sum()); + + return (model, transport); + } + + private static GgufModelManager NewManager(TempDirectory temp, FakeHttpTransport transport) => + new(temp.Path, + NullLogger.Instance, + new VerifiedFileDownloader(NullLogger.Instance, transport.ClientFactory), + freeSpaceProbe: _ => 100 * Gib); + + /// Reports on the calling thread so assertions see every value. + private sealed class InlineProgress(Action handler) : IProgress + { + public void Report(T value) => handler(value); + } +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/LlamaRuntimeManagerTests.cs b/tests/OpenClaw.Shared.Tests/Inference/LlamaRuntimeManagerTests.cs new file mode 100644 index 000000000..4ac69db7d --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/LlamaRuntimeManagerTests.cs @@ -0,0 +1,296 @@ +using System; +using System.IO; +using System.IO.Compression; +using System.Linq; +using System.Text; +using System.Threading.Tasks; +using OpenClaw.Shared; +using OpenClaw.Shared.Inference; +using OpenClaw.TestSupport; +using Xunit; + +// Disambiguates from the OpenClaw.Shared.Tests.Architecture namespace. +using Arch = System.Runtime.InteropServices.Architecture; + +namespace OpenClaw.Shared.Tests.Inference; + +public class LlamaRuntimeManagerTests +{ + [Fact] + public async Task InstallsAVariantAndResolvesTheServerExecutable() + { + using var temp = new TempDirectory(); + var (transport, variant) = SetUpTwoArchiveVariant(); + + var runtime = await NewManager(temp, transport).EnsureInstalledAsync(variant); + + Assert.Equal(LlamaRuntimeSource.Catalog, runtime.Source); + Assert.False(runtime.IsUnverified); + Assert.EndsWith(LlamaBackendCatalog.ServerExecutableName, runtime.ServerExecutablePath, StringComparison.OrdinalIgnoreCase); + Assert.True(File.Exists(runtime.ServerExecutablePath)); + + // Both archives of a CUDA pair have to land in the same directory. + Assert.True(File.Exists(Path.Combine(runtime.Directory, "cudart64_12.dll"))); + } + + [Fact] + public async Task DeletesTheDownloadedArchivesAfterExtracting() + { + // A CUDA pair is close to 800 MB; keeping the zips doubles peak disk use. + using var temp = new TempDirectory(); + var (transport, variant) = SetUpTwoArchiveVariant(); + var manager = NewManager(temp, transport); + + var runtime = await manager.EnsureInstalledAsync(variant); + + Assert.Empty(Directory.EnumerateFiles(runtime.Directory, "*.zip", SearchOption.AllDirectories)); + } + + [Fact] + public async Task SecondCallIsANoOpOnceInstalled() + { + using var temp = new TempDirectory(); + var (transport, variant) = SetUpTwoArchiveVariant(); + var manager = NewManager(temp, transport); + + await manager.EnsureInstalledAsync(variant); + var requestsAfterInstall = transport.Requests.Count; + Assert.True(manager.IsInstalled(variant)); + + await manager.EnsureInstalledAsync(variant); + + Assert.Equal(requestsAfterInstall, transport.Requests.Count); + } + + [Fact] + public async Task RefusesAVariantWithoutPinnedHashes() + { + using var temp = new TempDirectory(); + var transport = new FakeHttpTransport(); + var unpinned = new LlamaBackendVariant( + LlamaBackend.Cpu, + Arch.X64, + [new LlamaBackendAsset("llama-bin.zip", null, 10)], + "Unpinned test variant"); + + var ex = await Assert.ThrowsAsync(() => + NewManager(temp, transport).EnsureInstalledAsync(unpinned)); + + Assert.Contains("pinned SHA-256", ex.Message, StringComparison.OrdinalIgnoreCase); + Assert.Empty(transport.Requests); + } + + [Fact] + public async Task AFailedInstallLeavesNothingBehindAndIsNotReportedAsInstalled() + { + using var temp = new TempDirectory(); + var goodZip = MakeZip([(LlamaBackendCatalog.ServerExecutableName, "server")]); + var variant = new LlamaBackendVariant( + LlamaBackend.Cuda12, + Arch.X64, + [ + new LlamaBackendAsset("llama-bin.zip", FakeHttpTransport.Sha256Of(goodZip), goodZip.Length), + new LlamaBackendAsset("cudart.zip", new string('a', 64), 10), + ], + "Half-broken test variant"); + + var transport = new FakeHttpTransport(); + // Only the first archive is served; the second 404s partway through. + transport.Add(variant.Assets[0].DownloadUrl, goodZip); + + var manager = NewManager(temp, transport); + await Assert.ThrowsAnyAsync(() => manager.EnsureInstalledAsync(variant)); + + // Critical: the first archive did contain llama-server.exe. Without the + // completion marker this would look installed and then fail at launch + // with a missing CUDA DLL. + Assert.False(manager.IsInstalled(variant)); + Assert.False(Directory.Exists(manager.GetRuntimeDirectory(variant))); + } + + [Fact] + public async Task RebuildsARuntimeDirectoryLeftOverFromAnInterruptedInstall() + { + using var temp = new TempDirectory(); + var (transport, variant) = SetUpTwoArchiveVariant(); + var manager = NewManager(temp, transport); + + // Simulate an interrupted attempt: files present, no completion marker. + var directory = manager.GetRuntimeDirectory(variant); + Directory.CreateDirectory(directory); + await File.WriteAllTextAsync(Path.Combine(directory, "stale-from-old-release.dll"), "stale"); + + Assert.False(manager.IsInstalled(variant)); + var runtime = await manager.EnsureInstalledAsync(variant); + + Assert.True(manager.IsInstalled(variant)); + Assert.False(File.Exists(Path.Combine(runtime.Directory, "stale-from-old-release.dll"))); + } + + [Fact] + public async Task FailsWhenTheArchivesContainNoServerExecutable() + { + using var temp = new TempDirectory(); + var zip = MakeZip([("readme.txt", "no server here")]); + var variant = new LlamaBackendVariant( + LlamaBackend.Cpu, + Arch.X64, + [new LlamaBackendAsset("llama-bin.zip", FakeHttpTransport.Sha256Of(zip), zip.Length)], + "Serverless test variant"); + + var transport = new FakeHttpTransport(); + transport.Add(variant.Assets[0].DownloadUrl, zip); + + var ex = await Assert.ThrowsAsync(() => + NewManager(temp, transport).EnsureInstalledAsync(variant)); + + Assert.Contains(LlamaBackendCatalog.ServerExecutableName, ex.Message, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public async Task FindsAServerExecutableNestedInsideTheArchiveLayout() + { + // Upstream has moved between a flat layout and build\bin across releases. + using var temp = new TempDirectory(); + var zip = MakeZip([($"build/bin/{LlamaBackendCatalog.ServerExecutableName}", "server")]); + var variant = new LlamaBackendVariant( + LlamaBackend.Cpu, + Arch.X64, + [new LlamaBackendAsset("llama-bin.zip", FakeHttpTransport.Sha256Of(zip), zip.Length)], + "Nested test variant"); + + var transport = new FakeHttpTransport(); + transport.Add(variant.Assets[0].DownloadUrl, zip); + + var runtime = await NewManager(temp, transport).EnsureInstalledAsync(variant); + + Assert.True(File.Exists(runtime.ServerExecutablePath)); + Assert.Contains("bin", runtime.ServerExecutablePath, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public async Task ReportsProgressEndingAtTheTotalSize() + { + using var temp = new TempDirectory(); + var (transport, variant) = SetUpTwoArchiveVariant(); + + (long downloaded, long total) last = (0, 0); + var progress = new InlineProgress<(long downloaded, long total)>(p => last = p); + + await NewManager(temp, transport).EnsureInstalledAsync(variant, progress); + + Assert.Equal(variant.ApproximateSizeBytes, last.total); + Assert.Equal(variant.ApproximateSizeBytes, last.downloaded); + } + + [Fact] + public void ResolvesACustomBuildFromADirectoryAndFlagsItUnverified() + { + using var temp = new TempDirectory(); + var buildDir = temp.Combine("my-build"); + Directory.CreateDirectory(buildDir); + var exe = Path.Combine(buildDir, LlamaBackendCatalog.ServerExecutableName); + File.WriteAllText(exe, "custom"); + + var runtime = NewManager(temp, new FakeHttpTransport()).ResolveCustomBuild(buildDir); + + Assert.Equal(LlamaRuntimeSource.CustomBuild, runtime.Source); + Assert.True(runtime.IsUnverified); + Assert.Null(runtime.Variant); + Assert.Equal(Path.GetFullPath(exe), runtime.ServerExecutablePath); + } + + [Fact] + public void ResolvesACustomBuildFromAnExecutablePath() + { + using var temp = new TempDirectory(); + var exe = temp.Combine(LlamaBackendCatalog.ServerExecutableName); + File.WriteAllText(exe, "custom"); + + var runtime = NewManager(temp, new FakeHttpTransport()).ResolveCustomBuild(exe); + + Assert.True(runtime.IsUnverified); + } + + [Fact] + public void RejectsACustomPathThatIsNotTheServerExecutable() + { + using var temp = new TempDirectory(); + var wrong = temp.Combine("llama-cli.exe"); + File.WriteAllText(wrong, "not the server"); + + Assert.Throws(() => + NewManager(temp, new FakeHttpTransport()).ResolveCustomBuild(wrong)); + } + + [Fact] + public void RejectsAMissingCustomPath() + { + using var temp = new TempDirectory(); + + Assert.Throws(() => + NewManager(temp, new FakeHttpTransport()).ResolveCustomBuild(temp.Combine("nope"))); + } + + [Fact] + public async Task UninstallRemovesTheRuntime() + { + using var temp = new TempDirectory(); + var (transport, variant) = SetUpTwoArchiveVariant(); + var manager = NewManager(temp, transport); + await manager.EnsureInstalledAsync(variant); + + Assert.True(manager.Uninstall(variant)); + Assert.False(manager.IsInstalled(variant)); + Assert.False(manager.Uninstall(variant)); + } + + private static (FakeHttpTransport Transport, LlamaBackendVariant Variant) SetUpTwoArchiveVariant() + { + var binaries = MakeZip([(LlamaBackendCatalog.ServerExecutableName, "server"), ("ggml.dll", "ggml")]); + var cudart = MakeZip([("cudart64_12.dll", "cudart")]); + + var transport = new FakeHttpTransport(); + + // Asset URLs are derived from the pinned release base, so the fake is + // registered against the variant's own DownloadUrl values. + var variant = new LlamaBackendVariant( + LlamaBackend.Cuda12, + Arch.X64, + [ + new LlamaBackendAsset("llama-bin.zip", FakeHttpTransport.Sha256Of(binaries), binaries.Length), + new LlamaBackendAsset("cudart.zip", FakeHttpTransport.Sha256Of(cudart), cudart.Length), + ], + "Test CUDA variant"); + + transport.Add(variant.Assets[0].DownloadUrl, binaries); + transport.Add(variant.Assets[1].DownloadUrl, cudart); + + return (transport, variant); + } + + private static LlamaRuntimeManager NewManager(TempDirectory temp, FakeHttpTransport transport) => + new(temp.Path, NullLogger.Instance, new VerifiedFileDownloader(NullLogger.Instance, transport.ClientFactory)); + + private static byte[] MakeZip((string Name, string Content)[] entries) + { + using var buffer = new MemoryStream(); + using (var archive = new ZipArchive(buffer, ZipArchiveMode.Create, leaveOpen: true)) + { + foreach (var (name, content) in entries) + { + var entry = archive.CreateEntry(name); + using var entryStream = entry.Open(); + var bytes = Encoding.UTF8.GetBytes(content); + entryStream.Write(bytes, 0, bytes.Length); + } + } + return buffer.ToArray(); + } + + /// Reports on the calling thread so assertions see the final value. + private sealed class InlineProgress(Action handler) : IProgress + { + public void Report(T value) => handler(value); + } +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/SafeZipExtractorTests.cs b/tests/OpenClaw.Shared.Tests/Inference/SafeZipExtractorTests.cs new file mode 100644 index 000000000..e3ba65ec2 --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/SafeZipExtractorTests.cs @@ -0,0 +1,109 @@ +using System; +using System.IO; +using System.IO.Compression; +using System.Linq; +using System.Text; +using OpenClaw.Shared.Inference; +using OpenClaw.TestSupport; +using Xunit; + +namespace OpenClaw.Shared.Tests.Inference; + +/// +/// These archives carry native executables that are then launched, so an entry +/// that escapes the destination is a code-execution primitive, not a tidiness +/// issue. The guard is asserted directly rather than trusted to the framework. +/// +public class SafeZipExtractorTests +{ + [Fact] + public void ExtractsFilesAndNestedDirectories() + { + using var temp = new TempDirectory(); + var archive = temp.Combine("bundle.zip"); + WriteZip(archive, [("llama-server.exe", "server"), ("lib/ggml.dll", "lib")]); + + var destination = temp.Combine("out"); + var written = SafeZipExtractor.ExtractTo(archive, destination); + + Assert.Equal(2, written.Count); + Assert.Equal("server", File.ReadAllText(Path.Combine(destination, "llama-server.exe"))); + Assert.Equal("lib", File.ReadAllText(Path.Combine(destination, "lib", "ggml.dll"))); + } + + [Fact] + public void RejectsAnEntryThatEscapesTheDestination() + { + using var temp = new TempDirectory(); + var archive = temp.Combine("evil.zip"); + WriteZip(archive, [("../../evil.exe", "pwned")]); + + var destination = temp.Combine("out"); + + var ex = Assert.Throws(() => SafeZipExtractor.ExtractTo(archive, destination)); + + Assert.Contains("outside the destination", ex.Message, StringComparison.OrdinalIgnoreCase); + Assert.False(File.Exists(temp.Combine("evil.exe"))); + Assert.False(File.Exists(Path.Combine(temp.Path, "..", "evil.exe"))); + } + + [Fact] + public void RejectsASiblingDirectoryPrefixAttack() + { + // "out-evil" shares a string prefix with "out"; only a separator-aware + // containment check rejects it. + using var temp = new TempDirectory(); + var archive = temp.Combine("evil.zip"); + WriteZip(archive, [("../out-evil/payload.exe", "pwned")]); + + Assert.Throws(() => SafeZipExtractor.ExtractTo(archive, temp.Combine("out"))); + Assert.False(File.Exists(temp.Combine("out-evil", "payload.exe"))); + } + + [Fact] + public void OverwritesAnExistingFile() + { + // Two archives (llama.cpp binaries plus cudart) extract into one + // directory, so a second pass must not fail on an existing name. + using var temp = new TempDirectory(); + var destination = temp.Combine("out"); + Directory.CreateDirectory(destination); + File.WriteAllText(Path.Combine(destination, "shared.dll"), "old"); + + var archive = temp.Combine("bundle.zip"); + WriteZip(archive, [("shared.dll", "new")]); + + SafeZipExtractor.ExtractTo(archive, destination); + + Assert.Equal("new", File.ReadAllText(Path.Combine(destination, "shared.dll"))); + } + + [Fact] + public void CreatesTheDestinationWhenItDoesNotExist() + { + using var temp = new TempDirectory(); + var archive = temp.Combine("bundle.zip"); + WriteZip(archive, [("a.txt", "a")]); + + var destination = temp.Combine("deep", "nested", "out"); + SafeZipExtractor.ExtractTo(archive, destination); + + Assert.True(File.Exists(Path.Combine(destination, "a.txt"))); + } + + private static void WriteZip(string path, (string Name, string Content)[] entries) + { + using var stream = new FileStream(path, FileMode.Create, FileAccess.Write); + using var archive = new ZipArchive(stream, ZipArchiveMode.Create); + + foreach (var (name, content) in entries) + { + // CreateEntry is used directly (not CreateEntryFromFile) so the + // traversal names survive into the archive verbatim. + var entry = archive.CreateEntry(name); + using var entryStream = entry.Open(); + var bytes = Encoding.UTF8.GetBytes(content); + entryStream.Write(bytes, 0, bytes.Length); + } + } +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/VerifiedFileDownloaderTests.cs b/tests/OpenClaw.Shared.Tests/Inference/VerifiedFileDownloaderTests.cs new file mode 100644 index 000000000..35c4a01e7 --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/VerifiedFileDownloaderTests.cs @@ -0,0 +1,273 @@ +using System; +using System.IO; +using System.Security; +using System.Threading; +using System.Threading.Tasks; +using OpenClaw.Shared; +using OpenClaw.Shared.Inference; +using OpenClaw.TestSupport; +using Xunit; + +namespace OpenClaw.Shared.Tests.Inference; + +/// +/// This downloader is the single gate between the catalogs and the filesystem, +/// and everything it fetches is either launched as a process or loaded into one. +/// Its fail-closed behavior is therefore pinned in detail. +/// +public class VerifiedFileDownloaderTests +{ + private const string Url = "https://example.test/asset.bin"; + + [Fact] + public async Task DownloadsAndVerifiesAGoodFile() + { + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(4096, seed: 1); + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + + var destination = temp.Combine("asset.bin"); + await NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, hash, body.Length)); + + Assert.Equal(body, await File.ReadAllBytesAsync(destination)); + Assert.False(File.Exists(destination + ".part")); + } + + [Fact] + public async Task RefusesToDownloadWithoutAPinnedHash() + { + using var temp = new TempDirectory(); + var transport = new FakeHttpTransport(); + var destination = temp.Combine("asset.bin"); + + var ex = await Assert.ThrowsAsync(() => + NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, Sha256: null))); + + Assert.Contains("no pinned SHA-256", ex.Message, StringComparison.OrdinalIgnoreCase); + // The gate must close before any network traffic, not after. + Assert.Empty(transport.Requests); + Assert.False(File.Exists(destination)); + } + + [Fact] + public async Task RejectsATamperedBodyAndLeavesNothingBehind() + { + using var temp = new TempDirectory(); + var (body, _) = FakeHttpTransport.MakeBody(4096, seed: 2); + var (otherBody, otherHash) = FakeHttpTransport.MakeBody(4096, seed: 3); + Assert.NotEqual(body, otherBody); + + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + var destination = temp.Combine("asset.bin"); + + // Pin the hash of a different body: the served bytes must be refused. + await Assert.ThrowsAsync(() => + NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, otherHash, otherBody.Length))); + + Assert.False(File.Exists(destination)); + Assert.False(File.Exists(destination + ".part")); + } + + [Fact] + public async Task HashMismatchErrorDoesNotEchoTheComputedHash() + { + // Echoing the actual hash would give an attacker a confirmation oracle. + using var temp = new TempDirectory(); + var (body, actualHash) = FakeHttpTransport.MakeBody(1024, seed: 4); + var (_, wrongHash) = FakeHttpTransport.MakeBody(1024, seed: 5); + + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + + var ex = await Assert.ThrowsAsync(() => + NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, temp.Combine("asset.bin"), wrongHash, body.Length))); + + Assert.DoesNotContain(actualHash, ex.Message, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public async Task RejectsABodyWhoseLengthDisagreesWithTheCatalog() + { + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(4096, seed: 6); + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + var destination = temp.Combine("asset.bin"); + + var ex = await Assert.ThrowsAsync(() => + NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, hash, ExpectedSizeBytes: body.Length + 1))); + + Assert.Contains("expects", ex.Message, StringComparison.OrdinalIgnoreCase); + Assert.False(File.Exists(destination)); + } + + [Fact] + public async Task SkipsAFileThatIsAlreadyPresent() + { + using var temp = new TempDirectory(); + var destination = temp.Combine("asset.bin"); + await File.WriteAllBytesAsync(destination, [1, 2, 3]); + + var transport = new FakeHttpTransport(); + await NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, new string('a', 64), 3)); + + Assert.Empty(transport.Requests); + Assert.Equal(new byte[] { 1, 2, 3 }, await File.ReadAllBytesAsync(destination)); + } + + [Fact] + public async Task ResumesAPartialFileWithARangeRequest() + { + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(8192, seed: 7); + var destination = temp.Combine("asset.bin"); + + // Simulate an earlier attempt that died after 3000 bytes. + await File.WriteAllBytesAsync(destination + ".part", body[..3000]); + + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + + await NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, hash, body.Length, AllowResume: true)); + + Assert.Equal(body, await File.ReadAllBytesAsync(destination)); + Assert.Equal("bytes=3000-", Assert.Single(transport.RangeHeaders)); + } + + [Fact] + public async Task RestartsCleanlyWhenTheServerIgnoresTheRangeHeader() + { + // Appending a full 200 body onto an existing prefix would silently + // corrupt the file; a restart is the only safe response. + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(8192, seed: 8); + var destination = temp.Combine("asset.bin"); + await File.WriteAllBytesAsync(destination + ".part", body[..3000]); + + var transport = new FakeHttpTransport(); + transport.Add(Url, body, supportsRange: false); + + await NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, hash, body.Length, AllowResume: true)); + + Assert.Equal(body, await File.ReadAllBytesAsync(destination)); + } + + [Fact] + public async Task DiscardsAPartialFileWhenResumeIsNotAllowed() + { + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(4096, seed: 9); + var destination = temp.Combine("asset.bin"); + await File.WriteAllBytesAsync(destination + ".part", body[..1000]); + + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + + await NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, hash, body.Length, AllowResume: false)); + + Assert.Equal(body, await File.ReadAllBytesAsync(destination)); + Assert.Null(Assert.Single(transport.RangeHeaders)); + } + + [Fact] + public async Task DiscardsAPartialFileThatIsAlreadyAtOrOverTheExpectedSize() + { + // Such a file is the residue of an attempt that already failed + // verification. Resuming from its end would download zero bytes and + // fail the same way forever. + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(4096, seed: 10); + var destination = temp.Combine("asset.bin"); + await File.WriteAllBytesAsync(destination + ".part", new byte[body.Length + 50]); + + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + + await NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, hash, body.Length, AllowResume: true)); + + Assert.Equal(body, await File.ReadAllBytesAsync(destination)); + Assert.Null(Assert.Single(transport.RangeHeaders)); + } + + [Fact] + public async Task ATruncatedResponseFailsAndTheRetrySucceeds() + { + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(8192, seed: 11); + var destination = temp.Combine("asset.bin"); + + var transport = new FakeHttpTransport(); + transport.Add(Url, body, truncateAfterBytes: 2000); + var downloader = NewDownloader(transport); + var request = new VerifiedDownloadRequest(Url, destination, hash, body.Length, AllowResume: true); + + await Assert.ThrowsAsync(() => downloader.DownloadAsync(request)); + Assert.False(File.Exists(destination)); + + transport.HealTruncation(Url); + await downloader.DownloadAsync(request); + + Assert.Equal(body, await File.ReadAllBytesAsync(destination)); + } + + [Fact] + public async Task ReportsMonotonicProgressEndingAtTheFullSize() + { + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(300_000, seed: 12); + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + + var reports = new System.Collections.Generic.List(); + var progress = new SynchronousProgress(reports.Add); + + await NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, temp.Combine("asset.bin"), hash, body.Length), + progress); + + Assert.NotEmpty(reports); + Assert.Equal(body.Length, reports[^1]); + for (var i = 1; i < reports.Count; i++) + Assert.True(reports[i] >= reports[i - 1], "Progress went backwards."); + } + + [Fact] + public async Task AnHttpErrorLeavesNoPartialFile() + { + using var temp = new TempDirectory(); + var transport = new FakeHttpTransport(); + var destination = temp.Combine("asset.bin"); + + await Assert.ThrowsAsync(() => + NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest("https://example.test/missing.bin", destination, new string('a', 64), 10))); + + Assert.False(File.Exists(destination)); + Assert.False(File.Exists(destination + ".part")); + } + + private static VerifiedFileDownloader NewDownloader(FakeHttpTransport transport) => + new(NullLogger.Instance, transport.ClientFactory); + + /// + /// posts to the synchronization context, so reports + /// can arrive after the awaited call returns. These tests need the callbacks + /// to have run by then. + /// + private sealed class SynchronousProgress(Action handler) : IProgress + { + public void Report(T value) => handler(value); + } +} From 75fc7334e0211e284c33f885f9a7943a8c874d58 Mon Sep 17 00:00:00 2001 From: Pedro Larroy Date: Tue, 18 Aug 2026 17:07:02 -0700 Subject: [PATCH 4/7] feat(inference): add llama-server process host and settings page Third phase of local inference: launch and supervise the server, and give it a settings surface. Local inference is now usable end to end except for gateway registration. New in src/OpenClaw.Shared/Inference/: - LlamaServerArguments builds the command line and the health/base URLs. - ProcessJobObject wraps a kill-on-close Win32 job object. - LlamaServerProcess launches, health-polls, tails stderr, and stops. - LocalInferenceService sequences probe, selector, runtime, model, and server, and is the seam the UI talks to. New in the tray: Pages/LocalInferencePage (hardware, model, server, and advanced sections), Services/DisplayAdapterEnumerator (registry adapter fallback injected into the probe), eight LocalInference* settings with SettingsManager passthroughs, HubPageRegistry and HubWindow wiring, a lazily-built service on App, and a shutdown step that stops the server. Notable decisions: - The child runs inside a kill-on-close job object. Without it a tray crash leaves llama-server holding tens of gigabytes of VRAM with no UI left to stop it, and the next launch fails on a port conflict or an allocation error. A test proves an assigned process really dies when the job handle closes. - The health poll checks for child exit on each tick. A rejected recipe flag makes llama-server exit at once; without that check the user would wait out the full ready timeout for an error known in a second. - stderr is tailed for diagnostics; stdout is drained but discarded, because it is request logging and would put prompt content into our diagnostics. - Loopback unless explicitly widened, with its own opt-in and warning. - Start never begins a download. - Confirmation is an inline bar rather than a ContentDialog. REACTOR_DIALOG_001 keeps new surfaces off imperative dialogs and the per-file suppressions in .editorconfig are deliberately not extended. - Progress repaints are throttled to 150 ms; a 22 GB model produces roughly 280,000 progress callbacks. Also fixes a real BackendSelector gap found while testing on an ARM64 host with an NVIDIA GPU. The pinned release ships no CUDA 12 ARM64 build, so such a host lands on the CUDA 13 runtime, which may not load with a CUDA 12 driver. The plan now says so instead of reporting a confident choice the driver may reject. Strings are seeded English-only across all five locales using the repo's deferred-translation pattern and registered in LocalizationValidationTests. Validation: ./build.ps1 (all projects, 49 docs), OpenClaw.Shared.Tests 3848 passed across three consecutive runs, OpenClaw.Tray.Tests 2469 passed. Readiness against a real llama-server remains manual proof. Co-Authored-By: Claude Opus 5 (1M context) --- docs/LOCAL_INFERENCE_PLAN.md | 87 ++- .../Inference/BackendSelector.cs | 20 +- .../Inference/LlamaServerArguments.cs | 91 +++ .../Inference/LlamaServerProcess.cs | 410 +++++++++++++ .../Inference/LocalInferenceService.cs | 192 ++++++ .../Inference/ProcessJobObject.cs | 151 +++++ src/OpenClaw.Shared/SettingsData.cs | 42 ++ .../App.AppShutdownCoordinator.cs | 20 + src/OpenClaw.Tray.WinUI/App.xaml.cs | 28 + .../Pages/LocalInferencePage.xaml | 260 ++++++++ .../Pages/LocalInferencePage.xaml.cs | 564 ++++++++++++++++++ .../Presentation/HubPageRegistry.cs | 3 + .../Services/DisplayAdapterEnumerator.cs | 69 +++ .../Services/SettingsManager.cs | 19 + .../Strings/en-us/Resources.resw | 156 +++++ .../Strings/fr-fr/Resources.resw | 156 +++++ .../Strings/nl-nl/Resources.resw | 156 +++++ .../Strings/zh-cn/Resources.resw | 156 +++++ .../Strings/zh-tw/Resources.resw | 156 +++++ .../Windows/HubWindow.xaml.cs | 1 + .../Inference/BackendSelectorTests.cs | 22 + .../Inference/LlamaServerArgumentsTests.cs | 97 +++ .../Inference/LlamaServerProcessTests.cs | 173 ++++++ .../Inference/LocalInferenceServiceTests.cs | 213 +++++++ .../LocalizationValidationTests.cs | 57 ++ 25 files changed, 3263 insertions(+), 36 deletions(-) create mode 100644 src/OpenClaw.Shared/Inference/LlamaServerArguments.cs create mode 100644 src/OpenClaw.Shared/Inference/LlamaServerProcess.cs create mode 100644 src/OpenClaw.Shared/Inference/LocalInferenceService.cs create mode 100644 src/OpenClaw.Shared/Inference/ProcessJobObject.cs create mode 100644 src/OpenClaw.Tray.WinUI/Pages/LocalInferencePage.xaml create mode 100644 src/OpenClaw.Tray.WinUI/Pages/LocalInferencePage.xaml.cs create mode 100644 src/OpenClaw.Tray.WinUI/Services/DisplayAdapterEnumerator.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/LlamaServerArgumentsTests.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/LlamaServerProcessTests.cs create mode 100644 tests/OpenClaw.Shared.Tests/Inference/LocalInferenceServiceTests.cs diff --git a/docs/LOCAL_INFERENCE_PLAN.md b/docs/LOCAL_INFERENCE_PLAN.md index b53d3e8ca..42faa7297 100644 --- a/docs/LOCAL_INFERENCE_PLAN.md +++ b/docs/LOCAL_INFERENCE_PLAN.md @@ -7,7 +7,7 @@ inline; update it as phases land. | --- | --- | --- | | 1 | Hardware probe, backend selection, model recommender | Landed | | 2 | Runtime and GGUF download managers | Landed | -| 3 | Server process and settings UI | Not started | +| 3 | Server process and settings UI | Landed | | 4 | Gateway provider registration | Not started (blocked on live schema) | | 5 | Optional `localinference.status` node capability | Not started | @@ -177,37 +177,60 @@ API, and hashed on 2026-08-17; those values are pinned in `LlamaBackendCatalog` and guarded by `AssetHashPinningTests`. See `LOCAL_INFERENCE_ASSETS.md` for the provenance and its limits. -## Phase 3: server process and UI - -`OpenClawTray.Services.LlamaServerProcess` spawns `llama-server.exe` with -`--port

--host 127.0.0.1 -m ` plus the recipe args. - -- Port comes from a free-port scan; reuse `PortDiagnosticsService` and - `WindowsTcpListenerSnapshot` for conflict reporting. -- Bind to `127.0.0.1` by default. Binding beyond loopback exposes an - unauthenticated inference endpoint to the LAN and must be an explicit, warned - opt-in. -- Health: poll `GET /health` until ready or timeout, and surface the stderr tail - on failure. A recipe flag an older build does not know fails here, and the - user needs to see why. -- Assign the child to a Win32 job object with - `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE` so a tray crash cannot orphan a process - holding tens of gigabytes of VRAM. Graceful stop via `AppShutdownCoordinator`. - -`Pages/LocalInferencePage.xaml{,.cs}` follows `VoiceSettingsPage`, the closest -analogue: catalog combo, download button, progress bar driven by -`IProgress<(long downloaded, long total)>`, page-held `CancellationTokenSource`, -status text from resources. Register it in `Presentation/HubPageRegistry.cs` -(enum value, tag string, type map) and wire `Initialize` in -`Windows/HubWindow.xaml.cs`. All strings go through `LocalizationHelper` and -`Strings/en-us/Resources.resw`, with no em dashes per `AGENTS.md`. - -Settings added to `SettingsData`: `LocalInferenceEnabled`, -`LocalInferenceModelId`, `LocalInferenceBackendOverride`, -`LocalInferenceCustomRuntimePath`, `LocalInferencePort`, -`LocalInferenceAutoStart`, `LocalInferenceRegisterWithGateway`, -`LocalInferenceBindBeyondLoopback`. Change effects route through -`SettingsChangeCoordinator` and `SettingsChangeEffects`. +## Phase 3: server process and UI (landed) + +| File | Role | +| --- | --- | +| `Inference/LlamaServerArguments.cs` | Pure argument and URL construction. | +| `Inference/ProcessJobObject.cs` | Kill-on-close Win32 job object. | +| `Inference/LlamaServerProcess.cs` | Launch, health poll, stderr tail, stop. | +| `Inference/LocalInferenceService.cs` | Sequences probe, selector, runtime, model, server. | +| `Pages/LocalInferencePage.xaml{,.cs}` | Settings surface. | +| `Services/DisplayAdapterEnumerator.cs` | Registry adapter fallback injected into the probe. | + +The server host lives in `OpenClaw.Shared` rather than the tray project because +it has no WinUI dependency, which keeps the whole flow unit testable. + +Decisions worth preserving: + +- **The child runs inside a kill-on-close job object.** Without it a tray crash + leaves llama-server holding tens of gigabytes of VRAM with no UI left to stop + it, and the next launch fails on a port conflict or an allocation error. A test + proves an assigned process really dies when the job handle closes. Job creation + failure is logged and tolerated: losing the safety net beats refusing to run. +- **The health poll checks for child exit each tick.** A rejected recipe flag + makes llama-server exit immediately; without that check the user would wait out + the full ten-minute ready timeout for an error that was known in a second. +- **stderr is tailed, stdout is drained but discarded.** The tail is the only + place a bad flag or a CUDA initialization failure is explained. stdout is + llama-server's request logging, which would put prompt content into our + diagnostics, so it is read only to keep the pipe from blocking the child. +- **`-m`, `--host`, and `--port` are launcher-owned.** A recipe that sets one is + rejected rather than silently duplicated or overridden. +- **Loopback unless explicitly widened.** Binding all interfaces exposes an + unauthenticated inference endpoint to the LAN, so it is a separate opt-in with + its own warning, and the health poll still targets loopback. +- **Start never begins a download.** Kicking off a multi-hour transfer from a + Start button would be a surprising amount of work to trigger by accident. +- **Confirmation is inline, not a `ContentDialog`.** `REACTOR_DIALOG_001` keeps + new surfaces off imperative dialogs; the per-file suppressions in + `.editorconfig` are deliberately not extended. An in-page bar also keeps the + size being confirmed on screen. +- **Progress repaints are throttled to 150 ms.** A 22 GB model produces roughly + 280,000 progress callbacks; repainting on each saturates the dispatcher. + +Settings live on `SettingsData` with `SettingsManager` passthroughs: +`LocalInferenceEnabled`, `LocalInferenceModelId`, +`LocalInferenceBackendOverride`, `LocalInferenceCustomRuntimePath`, +`LocalInferencePort`, `LocalInferenceAutoStart`, +`LocalInferenceRegisterWithGateway`, `LocalInferenceBindBeyondLoopback`. + +The page is registered in `Presentation/HubPageRegistry.cs` and initialized from +`Windows/HubWindow.xaml.cs`. `App` builds the service lazily and the shutdown +coordinator stops the server so the GPU is released before exit; the job object +remains the crash backstop, not a substitute for an orderly stop. Strings are +seeded English-only across all five locales using the repo's +deferred-translation pattern and registered in `LocalizationValidationTests`. ## Phase 4: gateway registration (blocked on live schema) diff --git a/src/OpenClaw.Shared/Inference/BackendSelector.cs b/src/OpenClaw.Shared/Inference/BackendSelector.cs index 7c50ab8b6..a00d55eb8 100644 --- a/src/OpenClaw.Shared/Inference/BackendSelector.cs +++ b/src/OpenClaw.Shared/Inference/BackendSelector.cs @@ -100,10 +100,22 @@ private static BackendPlan SelectAutomatic(HostHardwareInfo hardware, Architectu if (cpu is not null) chain.Add(cpu); var cudaLabel = hardware.MaxCudaMajorVersion is { } major ? $"CUDA {major}.x" : "CUDA version unknown"; - return new BackendPlan( - chosen, - chain, - $"NVIDIA GPU detected ({cudaLabel}); using {chosen.DisplayName}."); + var reason = $"NVIDIA GPU detected ({cudaLabel}); using {chosen.DisplayName}."; + + // The preferred build can be unavailable for this architecture: + // the pinned release ships no CUDA 12 ARM64 build, so an ARM64 + // host with a CUDA 12 driver lands on the CUDA 13 build. That may + // fail to load, and the fallback chain will handle it, but the + // user is told rather than left to read a driver error. + if (!wantsCuda13 + && chosen.Backend == LlamaBackend.Cuda13 + && hardware.MaxCudaMajorVersion is { } reported) + { + reason += $" This release has no CUDA {reported} build for {arch}, " + + "so a newer CUDA runtime is used and may not load with the installed driver."; + } + + return new BackendPlan(chosen, chain, reason); } // NVIDIA hardware but no CUDA build for this architecture. diff --git a/src/OpenClaw.Shared/Inference/LlamaServerArguments.cs b/src/OpenClaw.Shared/Inference/LlamaServerArguments.cs new file mode 100644 index 000000000..fa63c7036 --- /dev/null +++ b/src/OpenClaw.Shared/Inference/LlamaServerArguments.cs @@ -0,0 +1,91 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace OpenClaw.Shared.Inference; + +///

+/// Builds the llama-server command line. +/// +/// Split from the process host so the argument contract is unit testable +/// without launching anything. The division of ownership matters: the launcher +/// owns -m, --host, and --port because they come from +/// runtime state, while everything else is the checkpoint's tuned recipe from +/// . A recipe that also set a launcher-owned flag +/// would either duplicate it or silently win, so that separation is enforced +/// here and asserted in tests. +/// +public static class LlamaServerArguments +{ + /// Loopback bind address. The default and the safe choice. + public const string LoopbackHost = "127.0.0.1"; + + /// + /// Bind address that accepts connections from outside the machine. Required + /// when a NAT-mode WSL gateway has to reach the server, and never selected + /// implicitly: it exposes an unauthenticated inference endpoint to the LAN. + /// + public const string AllInterfacesHost = "0.0.0.0"; + + /// + /// Flags the launcher owns. A catalog recipe must not contain any of these. + /// + public static readonly string[] LauncherOwnedFlags = ["-m", "--model", "--host", "--port"]; + + /// + /// Build the full argument list for a model run. + /// + /// Path to the checkpoint, or its first shard. + /// TCP port to listen on. + /// The checkpoint's tuned arguments. + /// + /// When true, bind instead of loopback. The + /// caller is responsible for having obtained explicit consent. + /// + /// + /// The recipe contains a flag the launcher owns. + /// + public static IReadOnlyList Build( + string modelPath, + int port, + IReadOnlyList recipeArgs, + bool bindBeyondLoopback = false) + { + ArgumentException.ThrowIfNullOrWhiteSpace(modelPath); + ArgumentNullException.ThrowIfNull(recipeArgs); + ArgumentOutOfRangeException.ThrowIfLessThan(port, 1); + ArgumentOutOfRangeException.ThrowIfGreaterThan(port, 65535); + + var conflict = recipeArgs.FirstOrDefault( + a => LauncherOwnedFlags.Contains(a, StringComparer.Ordinal)); + if (conflict is not null) + { + throw new ArgumentException( + $"Run recipe sets '{conflict}', which the launcher owns. Remove it from the catalog entry.", + nameof(recipeArgs)); + } + + var args = new List(recipeArgs.Count + 6) + { + "-m", modelPath, + "--host", bindBeyondLoopback ? AllInterfacesHost : LoopbackHost, + "--port", port.ToString(System.Globalization.CultureInfo.InvariantCulture), + }; + args.AddRange(recipeArgs); + return args; + } + + /// Health endpoint for a server on . + public static string BuildHealthUrl(int port) => + $"http://{LoopbackHost}:{port}/health"; + + /// + /// OpenAI-compatible base URL a client should use. + /// + /// + /// Host a client can actually reach the server on. Loopback for a local + /// client; the Windows host's name or address for a NAT-mode WSL gateway. + /// + public static string BuildBaseUrl(string host, int port) => + $"http://{host}:{port}/v1"; +} diff --git a/src/OpenClaw.Shared/Inference/LlamaServerProcess.cs b/src/OpenClaw.Shared/Inference/LlamaServerProcess.cs new file mode 100644 index 000000000..160251090 --- /dev/null +++ b/src/OpenClaw.Shared/Inference/LlamaServerProcess.cs @@ -0,0 +1,410 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Net; +using System.Net.Http; +using System.Net.Sockets; +using System.Threading; +using System.Threading.Tasks; + +namespace OpenClaw.Shared.Inference; + +/// Lifecycle state of the local inference server. +public enum LlamaServerState +{ + Stopped = 0, + /// Process launched; waiting for the health endpoint to answer. + Starting = 1, + /// Health endpoint is answering; the server can take requests. + Ready = 2, + /// Startup or the process itself failed. says why. + Failed = 3, +} + +/// +/// Observable status of the server. +/// +/// Current lifecycle state. +/// Port it is listening on, or null when stopped. +/// +/// PII-free explanation for the current state. On failure this carries the tail +/// of the server's stderr, which is the only place a bad recipe flag or a CUDA +/// initialization failure is explained. +/// +public sealed record LlamaServerStatus( + LlamaServerState State, + int? Port = null, + string? Detail = null) +{ + public static LlamaServerStatus Stopped { get; } = new(LlamaServerState.Stopped); + + /// Base URL for a client on this machine, or null when not ready. + public string? LoopbackBaseUrl => State == LlamaServerState.Ready && Port is { } port + ? LlamaServerArguments.BuildBaseUrl(LlamaServerArguments.LoopbackHost, port) + : null; +} + +/// +/// Launches and supervises a single llama-server process. +/// +/// One instance owns at most one running server. Starting while a server is +/// already running stops the old one first: two servers with the same weights +/// would each claim the GPU and the second would fail on allocation. +/// +/// The child is placed in a so it cannot +/// outlive the app even on an abrupt termination, and its stderr is tailed so a +/// startup failure can be explained rather than reported as a bare timeout. +/// +public sealed class LlamaServerProcess : IAsyncDisposable +{ + /// + /// Lines of stderr retained for diagnostics. Enough to carry a CUDA error and + /// its context; small enough that a chatty server cannot grow memory. + /// + private const int StderrTailLines = 40; + + /// Grace period for a polite shutdown before the process is killed. + private static readonly TimeSpan GracefulStopTimeout = TimeSpan.FromSeconds(10); + + private readonly IOpenClawLogger _logger; + private readonly Func _httpClientFactory; + private readonly SemaphoreSlim _lifecycleGate = new(1, 1); + + private Process? _process; + private ProcessJobObject? _job; + private ConcurrentQueue _stderrTail = new(); + private LlamaServerStatus _status = LlamaServerStatus.Stopped; + + public LlamaServerProcess(IOpenClawLogger logger, Func? httpClientFactory = null) + { + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _httpClientFactory = httpClientFactory ?? (static () => new HttpClient { Timeout = TimeSpan.FromSeconds(5) }); + } + + /// Raised whenever changes. + public event EventHandler? StatusChanged; + + /// Current status. Safe to read from any thread. + public LlamaServerStatus Status => _status; + + /// True when a child process is currently alive. + public bool IsRunning => _process is { HasExited: false }; + + /// + /// Start a server for the given runtime and model, waiting until its health + /// endpoint answers. + /// + /// Resolved runtime from . + /// Checkpoint path, or the first shard of a sharded model. + /// The checkpoint's tuned arguments. + /// Port to listen on, or null to allocate a free one. + /// + /// Bind all interfaces instead of loopback. Exposes an unauthenticated + /// endpoint to the network, so the caller must have obtained explicit consent. + /// + /// + /// How long to wait for the health endpoint. Loading tens of gigabytes of + /// weights from a cold page cache genuinely takes minutes, so this is + /// generous by default. + /// + public async Task StartAsync( + LlamaRuntime runtime, + string modelPath, + IReadOnlyList recipeArgs, + int? port = null, + bool bindBeyondLoopback = false, + TimeSpan? readyTimeout = null, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(runtime); + ArgumentException.ThrowIfNullOrWhiteSpace(modelPath); + + await _lifecycleGate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + await StopCoreAsync().ConfigureAwait(false); + + if (!File.Exists(runtime.ServerExecutablePath)) + return Fail($"Server executable not found at '{runtime.ServerExecutablePath}'."); + + if (!File.Exists(modelPath)) + return Fail("The selected model is not downloaded."); + + var resolvedPort = port ?? AllocateFreePort(); + var argv = LlamaServerArguments.Build(modelPath, resolvedPort, recipeArgs, bindBeyondLoopback); + + if (bindBeyondLoopback) + { + _logger.Warn( + "[Inference] Binding the local inference server beyond loopback. " + + "The endpoint is unauthenticated and reachable from the network."); + } + + SetStatus(new LlamaServerStatus(LlamaServerState.Starting, resolvedPort)); + + if (!TryLaunch(runtime, argv, out var launchError)) + return Fail(launchError!); + + var ready = await WaitForHealthAsync( + resolvedPort, + readyTimeout ?? TimeSpan.FromMinutes(10), + cancellationToken).ConfigureAwait(false); + + if (!ready) + { + var detail = _process is { HasExited: true } + ? $"The server exited during startup. {DescribeStderrTail()}" + : $"The server did not become ready in time. {DescribeStderrTail()}"; + await StopCoreAsync().ConfigureAwait(false); + return Fail(detail); + } + + _logger.Info($"[Inference] Server ready on port {resolvedPort}"); + SetStatus(new LlamaServerStatus(LlamaServerState.Ready, resolvedPort)); + return _status; + } + finally + { + _lifecycleGate.Release(); + } + } + + /// Stop the running server, if any. Safe to call when stopped. + public async Task StopAsync(CancellationToken cancellationToken = default) + { + await _lifecycleGate.WaitAsync(cancellationToken).ConfigureAwait(false); + try + { + await StopCoreAsync().ConfigureAwait(false); + SetStatus(LlamaServerStatus.Stopped); + } + finally + { + _lifecycleGate.Release(); + } + } + + private bool TryLaunch(LlamaRuntime runtime, IReadOnlyList argv, out string? error) + { + error = null; + var startInfo = new ProcessStartInfo + { + FileName = runtime.ServerExecutablePath, + // The runtime directory holds the native dependencies (ggml, CUDA), + // so the process has to start there for the loader to find them. + WorkingDirectory = runtime.Directory, + UseShellExecute = false, + CreateNoWindow = true, + RedirectStandardError = true, + RedirectStandardOutput = true, + }; + + foreach (var arg in argv) startInfo.ArgumentList.Add(arg); + + // The command line embeds the model and runtime paths, which contain the + // user's profile directory. Log the shape, not the values. + _logger.Info($"[Inference] Launching llama-server with {argv.Count} arguments"); + + _stderrTail = new ConcurrentQueue(); + + try + { + var process = new Process { StartInfo = startInfo, EnableRaisingEvents = true }; + process.ErrorDataReceived += OnStderr; + process.OutputDataReceived += OnStdout; + process.Exited += OnProcessExited; + + if (!process.Start()) + { + error = "The server process could not be started."; + return false; + } + + process.BeginErrorReadLine(); + process.BeginOutputReadLine(); + _process = process; + + AttachJobObject(process); + return true; + } + catch (Exception ex) + { + error = $"Failed to launch the server process: {ex.Message}"; + _logger.Error("[Inference] llama-server launch failed", ex); + return false; + } + } + + /// + /// Put the child in a kill-on-close job. A failure here is logged and + /// tolerated: losing the crash-cleanup guarantee is worse than nothing, but + /// refusing to run at all would be worse still. + /// + private void AttachJobObject(Process process) + { + try + { + _job = new ProcessJobObject(); + if (!_job.TryAssign(process.Handle)) + { + _logger.Warn("[Inference] Could not assign llama-server to a job object; " + + "it may survive an abrupt shutdown of this app."); + } + } + catch (Exception ex) + { + _logger.Warn($"[Inference] Job object unavailable: {ex.Message}. " + + "llama-server may survive an abrupt shutdown of this app."); + _job = null; + } + } + + private async Task WaitForHealthAsync(int port, TimeSpan timeout, CancellationToken cancellationToken) + { + var url = LlamaServerArguments.BuildHealthUrl(port); + var deadline = DateTimeOffset.UtcNow + timeout; + using var httpClient = _httpClientFactory(); + + while (DateTimeOffset.UtcNow < deadline) + { + cancellationToken.ThrowIfCancellationRequested(); + + // A process that already exited will never become healthy; failing + // fast here is what turns a ten-minute timeout into a prompt, + // explainable error. + if (_process is { HasExited: true }) return false; + + try + { + using var response = await httpClient.GetAsync(url, cancellationToken).ConfigureAwait(false); + if (response.StatusCode == HttpStatusCode.OK) return true; + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + throw; + } + catch (Exception) + { + // Connection refused while the server is still loading weights is + // the expected case, not an error worth logging on every poll. + } + + await Task.Delay(TimeSpan.FromMilliseconds(500), cancellationToken).ConfigureAwait(false); + } + + return false; + } + + private async Task StopCoreAsync() + { + var process = _process; + _process = null; + + if (process is not null) + { + try + { + if (!process.HasExited) + { + _logger.Info("[Inference] Stopping llama-server"); + process.Kill(entireProcessTree: true); + await process.WaitForExitAsync(new CancellationTokenSource(GracefulStopTimeout).Token) + .ConfigureAwait(false); + } + } + catch (Exception ex) + { + _logger.Warn($"[Inference] Could not stop llama-server cleanly: {ex.Message}"); + } + finally + { + process.ErrorDataReceived -= OnStderr; + process.OutputDataReceived -= OnStdout; + process.Exited -= OnProcessExited; + process.Dispose(); + } + } + + // Disposing the job terminates anything still in it, which covers a child + // that ignored the kill or spawned its own descendants. + _job?.Dispose(); + _job = null; + } + + private void OnStderr(object sender, DataReceivedEventArgs e) + { + if (e.Data is null) return; + + _stderrTail.Enqueue(e.Data); + while (_stderrTail.Count > StderrTailLines) _stderrTail.TryDequeue(out _); + } + + private void OnStdout(object sender, DataReceivedEventArgs e) + { + // Drained so the pipe buffer cannot fill and block the child, but not + // retained: llama-server's stdout is request logging, which would put + // prompt content into our diagnostics. + } + + private void OnProcessExited(object? sender, EventArgs e) + { + // An exit while we believed the server was serving is a crash, not a + // stop; surface it so the UI does not keep claiming Ready. + if (_status.State is LlamaServerState.Ready) + { + SetStatus(new LlamaServerStatus( + LlamaServerState.Failed, + _status.Port, + $"The server stopped unexpectedly. {DescribeStderrTail()}")); + } + } + + private string DescribeStderrTail() + { + var lines = _stderrTail.ToArray(); + return lines.Length == 0 + ? "No error output was captured." + : "Last output: " + string.Join(" | ", lines.TakeLast(5)); + } + + private LlamaServerStatus Fail(string detail) + { + _logger.Warn($"[Inference] {detail}"); + SetStatus(new LlamaServerStatus(LlamaServerState.Failed, _status.Port, detail)); + return _status; + } + + private void SetStatus(LlamaServerStatus status) + { + _status = status; + StatusChanged?.Invoke(this, status); + } + + /// + /// Ask the OS for an unused loopback port. Inherently racy, so a launch that + /// still hits a conflict fails with the server's own bind error rather than + /// this method pretending to guarantee availability. + /// + private static int AllocateFreePort() + { + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + try + { + return ((IPEndPoint)listener.LocalEndpoint).Port; + } + finally + { + listener.Stop(); + } + } + + public async ValueTask DisposeAsync() + { + await StopAsync().ConfigureAwait(false); + _lifecycleGate.Dispose(); + } +} diff --git a/src/OpenClaw.Shared/Inference/LocalInferenceService.cs b/src/OpenClaw.Shared/Inference/LocalInferenceService.cs new file mode 100644 index 000000000..c60c8e97f --- /dev/null +++ b/src/OpenClaw.Shared/Inference/LocalInferenceService.cs @@ -0,0 +1,192 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; + +namespace OpenClaw.Shared.Inference; + +/// +/// Everything the local-inference settings page needs to render. +/// +/// Detected hardware, or null before the first probe. +/// Selected backend plan for that hardware. +/// Model fit assessment for that hardware. +/// Model the user chose, or the recommended one. +/// On-disk state of the selected model. +/// Whether the selected backend is installed. +/// Whether a user-supplied build is configured. +/// Current server lifecycle status. +public sealed record LocalInferenceSnapshot( + HostHardwareInfo? Hardware, + BackendPlan? BackendPlan, + LocalModelRecommendation? Recommendation, + LocalModelInfo? SelectedModel, + LocalModelDownloadState? ModelState, + bool RuntimeInstalled, + bool UsingCustomRuntime, + LlamaServerStatus ServerStatus); + +/// +/// Orchestrates the local inference pipeline: probe hardware, select a backend, +/// resolve a runtime and model, and run the server. +/// +/// This is the seam the UI talks to. It holds no WinUI types so the whole +/// flow can be exercised in tests, and it deliberately owns no download or +/// process logic of its own beyond sequencing the components that do. +/// +public sealed class LocalInferenceService : IAsyncDisposable +{ + private readonly HardwareProbe _hardwareProbe; + private readonly LlamaRuntimeManager _runtimeManager; + private readonly GgufModelManager _modelManager; + private readonly LlamaServerProcess _server; + private readonly IOpenClawLogger _logger; + private readonly Func _settingsAccessor; + + public LocalInferenceService( + HardwareProbe hardwareProbe, + LlamaRuntimeManager runtimeManager, + GgufModelManager modelManager, + LlamaServerProcess server, + Func settingsAccessor, + IOpenClawLogger logger) + { + _hardwareProbe = hardwareProbe ?? throw new ArgumentNullException(nameof(hardwareProbe)); + _runtimeManager = runtimeManager ?? throw new ArgumentNullException(nameof(runtimeManager)); + _modelManager = modelManager ?? throw new ArgumentNullException(nameof(modelManager)); + _server = server ?? throw new ArgumentNullException(nameof(server)); + _settingsAccessor = settingsAccessor ?? throw new ArgumentNullException(nameof(settingsAccessor)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + + _server.StatusChanged += (_, status) => ServerStatusChanged?.Invoke(this, status); + } + + /// Raised when the server's lifecycle status changes. + public event EventHandler? ServerStatusChanged; + + /// Current server status. + public LlamaServerStatus ServerStatus => _server.Status; + + /// + /// Build a full snapshot for the UI, probing hardware on first use. + /// + /// Re-run detection instead of using the cache. + public async Task GetSnapshotAsync( + bool refreshHardware = false, + CancellationToken cancellationToken = default) + { + var settings = _settingsAccessor(); + + var hardware = refreshHardware + ? await _hardwareProbe.RefreshAsync(cancellationToken).ConfigureAwait(false) + : await _hardwareProbe.GetAsync(cancellationToken).ConfigureAwait(false); + + var plan = BackendSelector.Select(hardware, ParseBackendOverride(settings.LocalInferenceBackendOverride)); + var recommendation = ModelRecommender.Recommend(hardware); + var model = ResolveSelectedModel(settings, recommendation); + + var usingCustom = !string.IsNullOrWhiteSpace(settings.LocalInferenceCustomRuntimePath); + var runtimeInstalled = usingCustom + || (plan.Preferred is { } variant && _runtimeManager.IsInstalled(variant)); + + return new LocalInferenceSnapshot( + hardware, + plan, + recommendation, + model, + model is null ? null : _modelManager.GetState(model), + runtimeInstalled, + usingCustom, + _server.Status); + } + + /// + /// The model the user selected, falling back to the recommendation. Returns + /// null when neither is available, which the UI renders as "nothing to run". + /// + public LocalModelInfo? ResolveSelectedModel(SettingsData settings, LocalModelRecommendation? recommendation) => + LocalModelCatalog.Find(settings.LocalInferenceModelId) ?? recommendation?.Recommended; + + /// + /// Install the backend runtime for the current hardware, unless a custom + /// build is configured. + /// + public async Task EnsureRuntimeAsync( + IProgress<(long downloaded, long total)>? progress = null, + CancellationToken cancellationToken = default) + { + var settings = _settingsAccessor(); + + if (!string.IsNullOrWhiteSpace(settings.LocalInferenceCustomRuntimePath)) + return _runtimeManager.ResolveCustomBuild(settings.LocalInferenceCustomRuntimePath!); + + var hardware = await _hardwareProbe.GetAsync(cancellationToken).ConfigureAwait(false); + var plan = BackendSelector.Select(hardware, ParseBackendOverride(settings.LocalInferenceBackendOverride)); + + if (plan.Preferred is null) + throw new InvalidOperationException($"No llama.cpp build is available for this host. {plan.Reason}"); + + return await _runtimeManager + .EnsureInstalledAsync(plan.Preferred, progress, cancellationToken) + .ConfigureAwait(false); + } + + /// Download the selected model's weights. + public async Task EnsureModelAsync( + LocalModelInfo model, + IProgress<(long downloaded, long total)>? progress = null, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(model); + await _modelManager.DownloadAsync(model, progress, cancellationToken).ConfigureAwait(false); + } + + /// + /// Start the server for the selected model, installing the runtime first if + /// needed. The model must already be downloaded: starting a multi-hour + /// download from a Start button would be a surprising amount of work to + /// trigger by accident. + /// + public async Task StartAsync(CancellationToken cancellationToken = default) + { + var settings = _settingsAccessor(); + var hardware = await _hardwareProbe.GetAsync(cancellationToken).ConfigureAwait(false); + var model = ResolveSelectedModel(settings, ModelRecommender.Recommend(hardware)); + + if (model is null) + throw new InvalidOperationException("No local model is selected and none is recommended for this host."); + + if (!_modelManager.IsDownloaded(model)) + throw new InvalidOperationException($"Model '{model.DisplayName}' is not downloaded yet."); + + var modelPath = _modelManager.GetPrimaryShardPath(model) + ?? throw new InvalidOperationException($"Model '{model.DisplayName}' has no checkpoint file."); + + var runtime = await EnsureRuntimeAsync(cancellationToken: cancellationToken).ConfigureAwait(false); + + return await _server.StartAsync( + runtime, + modelPath, + model.RecipeArgs, + settings.LocalInferencePort, + settings.LocalInferenceBindBeyondLoopback, + cancellationToken: cancellationToken).ConfigureAwait(false); + } + + /// Stop the server if it is running. + public Task StopAsync(CancellationToken cancellationToken = default) => _server.StopAsync(cancellationToken); + + /// + /// Parse the persisted backend override. An unrecognized value is treated as + /// "no override" rather than an error: a settings file edited by hand should + /// degrade to automatic selection, not break the page. + /// + public static LlamaBackend? ParseBackendOverride(string? value) + { + if (string.IsNullOrWhiteSpace(value)) return null; + return Enum.TryParse(value.Trim(), ignoreCase: true, out var parsed) ? parsed : null; + } + + public async ValueTask DisposeAsync() => await _server.DisposeAsync().ConfigureAwait(false); +} diff --git a/src/OpenClaw.Shared/Inference/ProcessJobObject.cs b/src/OpenClaw.Shared/Inference/ProcessJobObject.cs new file mode 100644 index 000000000..dfd09bb70 --- /dev/null +++ b/src/OpenClaw.Shared/Inference/ProcessJobObject.cs @@ -0,0 +1,151 @@ +using System; +using System.Runtime.InteropServices; + +namespace OpenClaw.Shared.Inference; + +/// +/// A Win32 job object configured to kill its members when the handle closes. +/// +/// Without this, a tray crash or a force-kill leaves llama-server.exe +/// running and holding tens of gigabytes of VRAM, with no UI left to stop it. The +/// user's only recourse is Task Manager, and the next launch fails on a port +/// conflict or an out-of-memory allocation. Assigning the child to a job with +/// JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE makes the OS clean up even when the +/// process that created it dies without running any code. +/// +/// Windows-only. Construction failures are surfaced to the caller, which +/// treats the job as unavailable and falls back to best-effort shutdown rather +/// than refusing to launch. +/// +public sealed class ProcessJobObject : IDisposable +{ + private IntPtr _handle; + private bool _disposed; + + /// True when a usable job handle was created. + public bool IsValid => _handle != IntPtr.Zero; + + /// + /// Create a job whose members are terminated when this instance is disposed + /// or the owning process exits. + /// + /// The job could not be created or configured. + public ProcessJobObject() + { + _handle = CreateJobObject(IntPtr.Zero, null); + if (_handle == IntPtr.Zero) + { + throw new InvalidOperationException( + $"CreateJobObject failed (Win32 error {Marshal.GetLastWin32Error()})."); + } + + var limits = new JOBOBJECT_EXTENDED_LIMIT_INFORMATION + { + BasicLimitInformation = new JOBOBJECT_BASIC_LIMIT_INFORMATION + { + LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE, + }, + }; + + var size = Marshal.SizeOf(); + var buffer = Marshal.AllocHGlobal(size); + try + { + Marshal.StructureToPtr(limits, buffer, fDeleteOld: false); + if (!SetInformationJobObject(_handle, JobObjectExtendedLimitInformation, buffer, (uint)size)) + { + var error = Marshal.GetLastWin32Error(); + Dispose(); + throw new InvalidOperationException( + $"SetInformationJobObject failed (Win32 error {error})."); + } + } + finally + { + Marshal.FreeHGlobal(buffer); + } + } + + /// + /// Add a process to the job. Returns false when the assignment failed, which + /// the caller should log rather than treat as fatal: losing the safety net is + /// worse than nothing but better than refusing to run. + /// + public bool TryAssign(IntPtr processHandle) + { + if (!IsValid || processHandle == IntPtr.Zero) return false; + return AssignProcessToJobObject(_handle, processHandle); + } + + public void Dispose() + { + if (_disposed) return; + _disposed = true; + + if (_handle != IntPtr.Zero) + { + // Closing the last handle to a KILL_ON_JOB_CLOSE job terminates + // everything still in it. That is the point. + CloseHandle(_handle); + _handle = IntPtr.Zero; + } + } + + private const int JobObjectExtendedLimitInformation = 9; + private const uint JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE = 0x2000; + + [StructLayout(LayoutKind.Sequential)] + private struct JOBOBJECT_BASIC_LIMIT_INFORMATION + { + public long PerProcessUserTimeLimit; + public long PerJobUserTimeLimit; + public uint LimitFlags; + public UIntPtr MinimumWorkingSetSize; + public UIntPtr MaximumWorkingSetSize; + public uint ActiveProcessLimit; + public UIntPtr Affinity; + public uint PriorityClass; + public uint SchedulingClass; + } + + [StructLayout(LayoutKind.Sequential)] + private struct IO_COUNTERS + { + public ulong ReadOperationCount; + public ulong WriteOperationCount; + public ulong OtherOperationCount; + public ulong ReadTransferCount; + public ulong WriteTransferCount; + public ulong OtherTransferCount; + } + + [StructLayout(LayoutKind.Sequential)] + private struct JOBOBJECT_EXTENDED_LIMIT_INFORMATION + { + public JOBOBJECT_BASIC_LIMIT_INFORMATION BasicLimitInformation; + public IO_COUNTERS IoInfo; + public UIntPtr ProcessMemoryLimit; + public UIntPtr JobMemoryLimit; + public UIntPtr PeakProcessMemoryUsed; + public UIntPtr PeakJobMemoryUsed; + } + + [DllImport("kernel32.dll", CharSet = CharSet.Unicode, SetLastError = true)] + private static extern IntPtr CreateJobObject(IntPtr lpJobAttributes, string? lpName); + + [DllImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool SetInformationJobObject( + IntPtr hJob, + int jobObjectInfoClass, + IntPtr lpJobObjectInfo, + uint cbJobObjectInfoLength); + + [DllImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool AssignProcessToJobObject(IntPtr hJob, IntPtr hProcess); + + [DllImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool CloseHandle(IntPtr hObject); +} diff --git a/src/OpenClaw.Shared/SettingsData.cs b/src/OpenClaw.Shared/SettingsData.cs index 090e5cfcc..48999fb1d 100644 --- a/src/OpenClaw.Shared/SettingsData.cs +++ b/src/OpenClaw.Shared/SettingsData.cs @@ -221,6 +221,48 @@ public record class SettingsData // ── (Voice / STT settings consolidated into the block above.) ── + // ── Local inference (llama.cpp) ── + + /// + /// Master switch for running a model locally. When false no runtime or model + /// is downloaded and no server is started. + /// + public bool LocalInferenceEnabled { get; set; } = false; + + /// Catalog id of the selected model, or null to use the recommendation. + public string? LocalInferenceModelId { get; set; } + + /// + /// Explicit backend override (Cpu, Cuda12, Cuda13, + /// Vulkan), or null to select automatically from detected hardware. + /// An override naming a build this release does not ship for the host + /// architecture is ignored rather than honored into a dead end. + /// + public string? LocalInferenceBackendOverride { get; set; } + + /// + /// Path to a user-built llama-server.exe, or the directory holding it. + /// When set it wins over the pinned catalog and skips download and hash + /// verification, so the UI must show the unverified state explicitly. + /// + public string? LocalInferenceCustomRuntimePath { get; set; } + + /// Fixed port for the local server, or null to allocate a free one. + public int? LocalInferencePort { get; set; } + + /// Start the local server automatically when the app starts. + public bool LocalInferenceAutoStart { get; set; } = false; + + /// Register the local endpoint with the gateway once it is healthy. + public bool LocalInferenceRegisterWithGateway { get; set; } = true; + + /// + /// Bind the local server to all interfaces instead of loopback. Required for + /// a NAT-mode WSL gateway to reach it, and off by default because it exposes + /// an unauthenticated inference endpoint to the network. + /// + public bool LocalInferenceBindBeyondLoopback { get; set; } = false; + private static readonly JsonSerializerOptions s_options = new() { WriteIndented = true, diff --git a/src/OpenClaw.Tray.WinUI/App.AppShutdownCoordinator.cs b/src/OpenClaw.Tray.WinUI/App.AppShutdownCoordinator.cs index 9f35e1fc6..dc1f6d8de 100644 --- a/src/OpenClaw.Tray.WinUI/App.AppShutdownCoordinator.cs +++ b/src/OpenClaw.Tray.WinUI/App.AppShutdownCoordinator.cs @@ -138,6 +138,26 @@ private AppShutdownPlan BuildShutdownPlan() })); } + var localInference = _localInference; + if (localInference is not null) + { + // The job object kills llama-server if we die abruptly, but an + // orderly shutdown should release the GPU before the app exits so a + // quick restart does not hit an out-of-memory allocation. + steps.Add(new AppShutdownStep("local inference server", async () => + { + try + { + await localInference.DisposeAsync(); + } + finally + { + if (ReferenceEquals(_localInference, localInference)) + _localInference = null; + } + })); + } + steps.Add(new AppShutdownStep("ssh tunnel service", () => { _sshTunnelService?.Dispose(); diff --git a/src/OpenClaw.Tray.WinUI/App.xaml.cs b/src/OpenClaw.Tray.WinUI/App.xaml.cs index a57b2c8c1..6696295a6 100644 --- a/src/OpenClaw.Tray.WinUI/App.xaml.cs +++ b/src/OpenClaw.Tray.WinUI/App.xaml.cs @@ -117,6 +117,33 @@ public partial class App : Application, OpenClawTray.Services.IAppCommands, IPer AppIdentity.ResolveRoamingDataDirectory(), new AppLogger()); + /// + /// Local llama.cpp inference. Constructed on first use because the hardware + /// probe shells out to nvidia-smi and nothing needs it until the user opens + /// the page or auto-start runs. + /// + internal OpenClaw.Shared.Inference.LocalInferenceService LocalInference => + _localInference ??= BuildLocalInferenceService(); + + internal OpenClaw.Shared.Inference.LocalInferenceService? LocalInferenceOrNull => _localInference; + + private OpenClaw.Shared.Inference.LocalInferenceService BuildLocalInferenceService() + { + var logger = new AppLogger(); + var dataDirectory = SettingsManager.SettingsDirectoryPath; + + return new OpenClaw.Shared.Inference.LocalInferenceService( + new OpenClaw.Shared.Inference.HardwareProbe( + new LocalCommandRunner(logger), + logger, + fallbackGpuEnumerator: DisplayAdapterEnumerator.Enumerate), + new OpenClaw.Shared.Inference.LlamaRuntimeManager(dataDirectory, logger), + new OpenClaw.Shared.Inference.GgufModelManager(dataDirectory, logger), + new OpenClaw.Shared.Inference.LlamaServerProcess(logger), + () => Settings.ToSettingsData(), + logger); + } + /// /// Session key that the chat surface should select on its next mount. /// Used when the user clicks a session from SessionsPage or a notification @@ -256,6 +283,7 @@ public IntPtr GetHubWindowHandle() => // Node service (optional, enabled in settings) private NodeService? _nodeService; private ExecApprovalsStore? _execApprovalsStore; + private OpenClaw.Shared.Inference.LocalInferenceService? _localInference; private string[]? _startupArgs; private string? _pendingProtocolUri; private bool _isPostSetupRestart; diff --git a/src/OpenClaw.Tray.WinUI/Pages/LocalInferencePage.xaml b/src/OpenClaw.Tray.WinUI/Pages/LocalInferencePage.xaml new file mode 100644 index 000000000..4fa1aacdd --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Pages/LocalInferencePage.xaml @@ -0,0 +1,260 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/OpenClaw.Tray.WinUI/Pages/LocalInferencePage.xaml.cs b/src/OpenClaw.Tray.WinUI/Pages/LocalInferencePage.xaml.cs new file mode 100644 index 000000000..5fc8293cc --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Pages/LocalInferencePage.xaml.cs @@ -0,0 +1,564 @@ +using Microsoft.UI.Xaml; +using Microsoft.UI.Xaml.Controls; +using OpenClaw.Shared; +using OpenClaw.Shared.Inference; +using OpenClawTray.Helpers; +using OpenClawTray.Services; +using System; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; + +namespace OpenClawTray.Pages; + +/// +/// Settings surface for running a model locally with llama.cpp: detected +/// hardware, model download, server lifecycle, and the advanced overrides. +/// +/// All orchestration lives in ; this page +/// only projects a snapshot onto controls and forwards user intent. +/// +public sealed partial class LocalInferencePage : Page +{ + /// + /// Minimum gap between progress repaints. The downloader streams in 80 KB + /// chunks, so a 22 GB model produces roughly 280,000 callbacks. Repainting + /// on each one saturates the dispatcher and the window stops responding. + /// + private static readonly TimeSpan ProgressRepaintInterval = TimeSpan.FromMilliseconds(150); + + private static App CurrentApp => (App)Application.Current!; + private static string L(string key) => LocalizationHelper.GetString(key); + private static string Lf(string key, params object?[] args) => + string.Format(CultureInfo.CurrentCulture, LocalizationHelper.GetString(key), args); + + private LocalInferenceService? _service; + private LocalInferenceSnapshot? _snapshot; + private CancellationTokenSource? _downloadCts; + private bool _suppressEvents = true; + private bool _serverBusy; + private Action? _pendingConfirmation; + + public LocalInferencePage() + { + InitializeComponent(); + Unloaded += (_, _) => + { + if (_service is not null) _service.ServerStatusChanged -= OnServerStatusChanged; + _downloadCts?.Cancel(); + }; + } + + public void Initialize(LocalInferenceService? service) + { + if (_service is not null) _service.ServerStatusChanged -= OnServerStatusChanged; + + _service = service; + if (_service is not null) _service.ServerStatusChanged += OnServerStatusChanged; + + PopulateStaticChoices(); + LoadSettings(); + RefreshSnapshot(refreshHardware: false); + } + + // ─── Population ─── + + private void PopulateStaticChoices() + { + _suppressEvents = true; + try + { + ModelCombo.Items.Clear(); + foreach (var model in LocalModelCatalog.Models) + { + ModelCombo.Items.Add(new ComboBoxItem + { + Content = model.IsDownloadable + ? $"{model.DisplayName} ({FormatBytes(model.TotalSizeBytes)})" + : Lf("LocalInferencePage_ModelUnavailableSuffix", model.DisplayName), + Tag = model.Id, + IsEnabled = model.IsDownloadable, + }); + } + + BackendCombo.Items.Clear(); + BackendCombo.Items.Add(new ComboBoxItem + { + Content = L("LocalInferencePage_BackendAutomatic"), + Tag = string.Empty, + }); + foreach (var backend in Enum.GetValues()) + { + BackendCombo.Items.Add(new ComboBoxItem + { + Content = backend.ToString(), + Tag = backend.ToString(), + }); + } + } + finally + { + _suppressEvents = false; + } + } + + private void LoadSettings() + { + var settings = CurrentApp.Settings; + _suppressEvents = true; + try + { + EnabledToggle.IsOn = settings.LocalInferenceEnabled; + AutoStartCheck.IsChecked = settings.LocalInferenceAutoStart; + RegisterCheck.IsChecked = settings.LocalInferenceRegisterWithGateway; + BindBeyondLoopbackCheck.IsChecked = settings.LocalInferenceBindBeyondLoopback; + CustomRuntimeBox.Text = settings.LocalInferenceCustomRuntimePath ?? string.Empty; + + SelectByTag(ModelCombo, settings.LocalInferenceModelId); + SelectByTag(BackendCombo, settings.LocalInferenceBackendOverride ?? string.Empty); + } + finally + { + _suppressEvents = false; + } + } + + private static void SelectByTag(ComboBox combo, string? tag) + { + var match = combo.Items + .OfType() + .FirstOrDefault(i => string.Equals(i.Tag as string, tag, StringComparison.OrdinalIgnoreCase)); + combo.SelectedItem = match; + } + + // ─── Snapshot ─── + + private void RefreshSnapshot(bool refreshHardware) => + AsyncEventHandlerGuard.Run( + () => RefreshSnapshotAsync(refreshHardware), + new AppLogger(), + nameof(RefreshSnapshot)); + + private async Task RefreshSnapshotAsync(bool refreshHardware) + { + if (_service is null) + { + HardwareSummaryText.Text = L("LocalInferencePage_HardwareUnavailable"); + return; + } + + if (refreshHardware) + { + RedetectButton.IsEnabled = false; + HardwareSummaryText.Text = L("LocalInferencePage_HardwareDetecting"); + } + + try + { + _snapshot = await _service.GetSnapshotAsync(refreshHardware); + ApplySnapshot(_snapshot); + } + catch (Exception ex) + { + // Privacy: the exception can carry paths and URLs. Log it, show a + // neutral message. + Logger.Error($"[LocalInferencePage] Snapshot failed: {ex}"); + HardwareSummaryText.Text = L("LocalInferencePage_HardwareUnavailable"); + } + finally + { + RedetectButton.IsEnabled = true; + } + } + + private void ApplySnapshot(LocalInferenceSnapshot snapshot) + { + HardwareSummaryText.Text = DescribeHardware(snapshot.Hardware); + BackendSummaryText.Text = snapshot.BackendPlan?.Reason ?? string.Empty; + + CustomRuntimeNotice.Visibility = snapshot.UsingCustomRuntime ? Visibility.Visible : Visibility.Collapsed; + + var model = snapshot.SelectedModel; + if (model is null) + { + ModelFitText.Text = snapshot.Recommendation?.Summary ?? string.Empty; + ModelStatusText.Text = string.Empty; + LargeDownloadNotice.Visibility = Visibility.Collapsed; + } + else + { + if (ModelCombo.SelectedItem is null) SelectByTag(ModelCombo, model.Id); + + var assessment = snapshot.Recommendation?.Assessments + .FirstOrDefault(a => a.Model.Id == model.Id); + ModelFitText.Text = assessment?.Reason ?? string.Empty; + + // A download measured in hundreds of gigabytes must never start + // without the size stated first. + LargeDownloadNotice.Visibility = model.RequiresConfirmation ? Visibility.Visible : Visibility.Collapsed; + LargeDownloadNoticeText.Text = Lf( + "LocalInferencePage_LargeDownloadNotice", + FormatBytes(model.TotalSizeBytes)); + + ModelStatusText.Text = snapshot.ModelState is { IsComplete: true } + ? L("LocalInferencePage_ModelReady") + : snapshot.ModelState is { ShardsPresent: > 0 } partial + ? Lf("LocalInferencePage_ModelPartial", partial.ShardsPresent, partial.ShardCount) + : L("LocalInferencePage_ModelNotDownloaded"); + } + + UpdateActionStates(snapshot); + ApplyServerStatus(snapshot.ServerStatus); + } + + private void UpdateActionStates(LocalInferenceSnapshot snapshot) + { + var downloading = _downloadCts is not null; + var model = snapshot.SelectedModel; + var modelReady = snapshot.ModelState is { IsComplete: true }; + + DownloadButton.IsEnabled = !downloading && model is { IsDownloadable: true }; + DeleteModelButton.IsEnabled = !downloading && snapshot.ModelState is { ShardsPresent: > 0 }; + CancelDownloadButton.Visibility = downloading ? Visibility.Visible : Visibility.Collapsed; + + var running = snapshot.ServerStatus.State is LlamaServerState.Ready or LlamaServerState.Starting; + StartButton.IsEnabled = !_serverBusy && !running && modelReady && EnabledToggle.IsOn; + StopButton.IsEnabled = !_serverBusy && running; + } + + private void ApplyServerStatus(LlamaServerStatus status) + { + ServerStatusText.Text = status.State switch + { + LlamaServerState.Ready => Lf("LocalInferencePage_ServerReady", status.Port), + LlamaServerState.Starting => L("LocalInferencePage_ServerStarting"), + LlamaServerState.Failed => L("LocalInferencePage_ServerFailed"), + _ => L("LocalInferencePage_ServerStopped"), + }; + + ServerBusyRing.IsActive = status.State == LlamaServerState.Starting || _serverBusy; + + var endpoint = status.LoopbackBaseUrl; + EndpointPanel.Visibility = endpoint is null ? Visibility.Collapsed : Visibility.Visible; + EndpointText.Text = endpoint ?? string.Empty; + + // Deliberate exception to the "no raw error text in the UI" rule: this is + // captured child-process stderr, shown on the user's own machine, and it + // is the only place a rejected recipe flag or a CUDA initialization + // failure is ever explained. It is not logged at Info level and is not + // part of any diagnostics export. + var hasDetail = status.State == LlamaServerState.Failed && !string.IsNullOrWhiteSpace(status.Detail); + ServerErrorNotice.Visibility = hasDetail ? Visibility.Visible : Visibility.Collapsed; + ServerErrorText.Text = hasDetail ? status.Detail! : string.Empty; + } + + private void OnServerStatusChanged(object? sender, LlamaServerStatus status) => + DispatcherQueue.TryEnqueue(() => + { + ApplyServerStatus(status); + if (_snapshot is not null) UpdateActionStates(_snapshot with { ServerStatus = status }); + }); + + // ─── Handlers ─── + + private void OnRedetectClick(object sender, RoutedEventArgs e) => RefreshSnapshot(refreshHardware: true); + + private void OnModelChanged(object sender, SelectionChangedEventArgs e) + { + if (_suppressEvents) return; + CurrentApp.Settings.LocalInferenceModelId = SelectedTag(ModelCombo); + CurrentApp.Settings.Save(); + RefreshSnapshot(refreshHardware: false); + } + + private void OnBackendChanged(object sender, SelectionChangedEventArgs e) + { + if (_suppressEvents) return; + var tag = SelectedTag(BackendCombo); + CurrentApp.Settings.LocalInferenceBackendOverride = string.IsNullOrEmpty(tag) ? null : tag; + CurrentApp.Settings.Save(); + RefreshSnapshot(refreshHardware: false); + } + + private void OnCustomRuntimeChanged(object sender, RoutedEventArgs e) + { + if (_suppressEvents) return; + var path = CustomRuntimeBox.Text?.Trim(); + CurrentApp.Settings.LocalInferenceCustomRuntimePath = string.IsNullOrEmpty(path) ? null : path; + CurrentApp.Settings.Save(); + RefreshSnapshot(refreshHardware: false); + } + + private void OnEnabledToggled(object sender, RoutedEventArgs e) + { + if (_suppressEvents) return; + CurrentApp.Settings.LocalInferenceEnabled = EnabledToggle.IsOn; + CurrentApp.Settings.Save(); + if (_snapshot is not null) UpdateActionStates(_snapshot); + } + + private void OnAutoStartChanged(object sender, RoutedEventArgs e) + { + if (_suppressEvents) return; + CurrentApp.Settings.LocalInferenceAutoStart = AutoStartCheck.IsChecked == true; + CurrentApp.Settings.Save(); + } + + private void OnRegisterChanged(object sender, RoutedEventArgs e) + { + if (_suppressEvents) return; + CurrentApp.Settings.LocalInferenceRegisterWithGateway = RegisterCheck.IsChecked == true; + CurrentApp.Settings.Save(); + } + + private void OnBindBeyondLoopbackChanged(object sender, RoutedEventArgs e) + { + if (_suppressEvents) return; + CurrentApp.Settings.LocalInferenceBindBeyondLoopback = BindBeyondLoopbackCheck.IsChecked == true; + CurrentApp.Settings.Save(); + } + + private void OnCopyEndpointClick(object sender, RoutedEventArgs e) + { + if (!string.IsNullOrEmpty(EndpointText.Text)) + ClipboardHelper.CopyText(EndpointText.Text); + } + + private void OnCancelDownloadClick(object sender, RoutedEventArgs e) => _downloadCts?.Cancel(); + + private void OnDownloadClick(object sender, RoutedEventArgs e) + { + if (_service is null || _snapshot?.SelectedModel is not { } model) return; + + // A download measured in hundreds of gigabytes is never started by a + // single click. + if (model.RequiresConfirmation) + { + ShowConfirmation( + Lf("LocalInferencePage_ConfirmDownloadBody", model.DisplayName, FormatBytes(model.TotalSizeBytes)), + L("LocalInferencePage_ConfirmDownloadPrimary"), + () => StartDownload(model)); + return; + } + + StartDownload(model); + } + + private void StartDownload(LocalModelInfo model) => + AsyncEventHandlerGuard.Run(() => DownloadAsync(model), new AppLogger(), nameof(OnDownloadClick)); + + private async Task DownloadAsync(LocalModelInfo model) + { + if (_service is null) return; + + _downloadCts?.Cancel(); + _downloadCts = new CancellationTokenSource(); + + DownloadProgress.Visibility = Visibility.Visible; + DownloadProgress.Value = 0; + ModelStatusText.Text = L("LocalInferencePage_StatusPreparing"); + if (_snapshot is not null) UpdateActionStates(_snapshot); + + try + { + // The runtime is fetched first: without it a completed model download + // still cannot start a server, and the runtime is the smaller of the + // two, so a failure surfaces in seconds rather than hours. + await _service.EnsureRuntimeAsync(MakeProgress("LocalInferencePage_StatusRuntimePct"), _downloadCts.Token); + await _service.EnsureModelAsync(model, MakeProgress("LocalInferencePage_StatusModelPct"), _downloadCts.Token); + + ModelStatusText.Text = L("LocalInferencePage_ModelReady"); + } + catch (OperationCanceledException) + { + ModelStatusText.Text = L("LocalInferencePage_StatusDownloadCanceled"); + } + catch (Exception ex) + { + // Privacy: the message can carry URLs, paths, and hash digests. + Logger.Error($"[LocalInferencePage] Download failed: {ex}"); + ModelStatusText.Text = L("LocalInferencePage_StatusDownloadError"); + } + finally + { + _downloadCts?.Dispose(); + _downloadCts = null; + DownloadProgress.Visibility = Visibility.Collapsed; + RefreshSnapshot(refreshHardware: false); + } + } + + /// + /// Progress reporter that repaints at most once per + /// , always painting the final 100 so + /// the bar never sticks just short of complete. + /// + private IProgress<(long downloaded, long total)> MakeProgress(string statusKey) + { + var lastRepaint = DateTime.MinValue; + return new Progress<(long downloaded, long total)>(p => + { + var isFinal = p.total > 0 && p.downloaded >= p.total; + var now = DateTime.UtcNow; + if (!isFinal && now - lastRepaint < ProgressRepaintInterval) return; + lastRepaint = now; + + if (p.total <= 0) return; + var percent = (double)p.downloaded / p.total * 100; + DownloadProgress.Value = percent; + ModelStatusText.Text = Lf(statusKey, $"{percent:F0}"); + }); + } + + // ─── Inline confirmation ─── + + /// + /// Show the in-page confirmation bar and run if + /// the user accepts. Used instead of a ContentDialog so the size being + /// confirmed stays visible behind the prompt. + /// + private void ShowConfirmation(string message, string primaryLabel, Action onConfirm) + { + ConfirmText.Text = message; + ConfirmPrimaryButtonText.Text = primaryLabel; + _pendingConfirmation = onConfirm; + ConfirmBar.Visibility = Visibility.Visible; + } + + private void HideConfirmation() + { + _pendingConfirmation = null; + ConfirmBar.Visibility = Visibility.Collapsed; + } + + private void OnConfirmPrimaryClick(object sender, RoutedEventArgs e) + { + var action = _pendingConfirmation; + HideConfirmation(); + action?.Invoke(); + } + + private void OnConfirmCancelClick(object sender, RoutedEventArgs e) => HideConfirmation(); + + private void OnDeleteModelClick(object sender, RoutedEventArgs e) + { + if (_snapshot?.SelectedModel is not { } model) return; + + ShowConfirmation( + Lf("LocalInferencePage_ConfirmDeleteBody", model.DisplayName, FormatBytes(model.TotalSizeBytes)), + L("LocalInferencePage_ConfirmDeletePrimary"), + () => AsyncEventHandlerGuard.Run( + () => DeleteModelAsync(model), new AppLogger(), nameof(OnDeleteModelClick))); + } + + private Task DeleteModelAsync(LocalModelInfo model) + { + try + { + new GgufModelManager(SettingsManager.SettingsDirectoryPath, new AppLogger()).Delete(model); + } + catch (Exception ex) + { + // Privacy: the message carries on-disk paths. Log it, keep the UI neutral. + Logger.Error($"[LocalInferencePage] Model delete failed: {ex}"); + ModelStatusText.Text = L("LocalInferencePage_StatusDeleteError"); + } + + RefreshSnapshot(refreshHardware: false); + return Task.CompletedTask; + } + + private void OnStartClick(object sender, RoutedEventArgs e) => + AsyncEventHandlerGuard.Run(OnStartClickAsync, new AppLogger(), nameof(OnStartClick)); + + private async Task OnStartClickAsync() + { + if (_service is null) return; + + _serverBusy = true; + if (_snapshot is not null) UpdateActionStates(_snapshot); + ServerBusyRing.IsActive = true; + + try + { + await _service.StartAsync(); + } + catch (Exception ex) + { + Logger.Error($"[LocalInferencePage] Server start failed: {ex}"); + ServerStatusText.Text = L("LocalInferencePage_ServerFailed"); + } + finally + { + _serverBusy = false; + RefreshSnapshot(refreshHardware: false); + } + } + + private void OnStopClick(object sender, RoutedEventArgs e) => + AsyncEventHandlerGuard.Run(OnStopClickAsync, new AppLogger(), nameof(OnStopClick)); + + private async Task OnStopClickAsync() + { + if (_service is null) return; + + _serverBusy = true; + if (_snapshot is not null) UpdateActionStates(_snapshot); + + try + { + await _service.StopAsync(); + } + catch (Exception ex) + { + Logger.Error($"[LocalInferencePage] Server stop failed: {ex}"); + } + finally + { + _serverBusy = false; + RefreshSnapshot(refreshHardware: false); + } + } + + // ─── Formatting ─── + + private static string? SelectedTag(ComboBox combo) => + (combo.SelectedItem as ComboBoxItem)?.Tag as string; + + private string DescribeHardware(HostHardwareInfo? hardware) + { + if (hardware is null) return L("LocalInferencePage_HardwareUnavailable"); + + var parts = new System.Collections.Generic.List + { + hardware.CpuArchitecture.ToString(), + }; + + if (hardware.TotalPhysicalMemoryBytes is { } ram) + parts.Add(Lf("LocalInferencePage_HardwareRam", FormatBytes(ram))); + + var gpu = hardware.Gpus.FirstOrDefault(); + if (gpu is null) + { + parts.Add(L("LocalInferencePage_HardwareNoGpu")); + } + else + { + parts.Add(hardware.TotalNvidiaVramBytes is { } vram + ? $"{gpu.Name} ({FormatBytes(vram)})" + : gpu.Name); + } + + return string.Join(" · ", parts); + } + + private static string FormatBytes(long bytes) + { + if (bytes <= 0) return "0 GB"; + var gib = bytes / (1024.0 * 1024 * 1024); + return gib >= 1 + ? string.Create(CultureInfo.CurrentCulture, $"{gib:F1} GB") + : string.Create(CultureInfo.CurrentCulture, $"{bytes / (1024.0 * 1024):F0} MB"); + } +} diff --git a/src/OpenClaw.Tray.WinUI/Presentation/HubPageRegistry.cs b/src/OpenClaw.Tray.WinUI/Presentation/HubPageRegistry.cs index b1c1d57ad..353734293 100644 --- a/src/OpenClaw.Tray.WinUI/Presentation/HubPageRegistry.cs +++ b/src/OpenClaw.Tray.WinUI/Presentation/HubPageRegistry.cs @@ -17,6 +17,7 @@ internal enum HubPageKind Bindings, Permissions, Voice, + LocalInference, Sandbox, Settings, Notifications, @@ -152,6 +153,7 @@ internal static class HubPageRegistry "bindings" => HubPageKind.Bindings, "capabilities" or "permissions" => HubPageKind.Permissions, "voice" => HubPageKind.Voice, + "localinference" or "local-inference" => HubPageKind.LocalInference, "sandbox" => HubPageKind.Sandbox, "activity" => HubPageKind.Channels, "settings" or "info" or "about" => HubPageKind.Settings, @@ -227,6 +229,7 @@ public static bool ShouldKeepCurrentPageVisibleDuringDisconnect(string? currentT HubPageKind.Bindings => typeof(BindingsPage), HubPageKind.Permissions => typeof(PermissionsPage), HubPageKind.Voice => typeof(VoiceSettingsPage), + HubPageKind.LocalInference => typeof(LocalInferencePage), HubPageKind.Sandbox => typeof(SandboxPage), HubPageKind.Settings => typeof(SettingsPage), HubPageKind.Notifications => typeof(NotificationsPage), diff --git a/src/OpenClaw.Tray.WinUI/Services/DisplayAdapterEnumerator.cs b/src/OpenClaw.Tray.WinUI/Services/DisplayAdapterEnumerator.cs new file mode 100644 index 000000000..fb6e0baf6 --- /dev/null +++ b/src/OpenClaw.Tray.WinUI/Services/DisplayAdapterEnumerator.cs @@ -0,0 +1,69 @@ +using Microsoft.Win32; +using OpenClaw.Shared.Inference; +using System; +using System.Collections.Generic; + +namespace OpenClawTray.Services; + +/// +/// Enumerates installed display adapters from the driver class registry key. +/// +/// This is the fallback the hardware probe uses when nvidia-smi is +/// absent, which is exactly the AMD/Intel case where the choice is between the +/// Vulkan build and the CPU build. It lives here rather than in +/// OpenClaw.Shared because registry access needs a Windows-targeted +/// framework, and the probe takes it as an injected delegate for that reason. +/// +/// Vendor and name only. Adapter memory is deliberately not reported: the +/// value that is easy to read here is the same one WMI exposes as a 32-bit field +/// that wraps above 4 GB, and a wrong VRAM number is worse than no number, since +/// the model recommender would size a download against it. +/// +internal static class DisplayAdapterEnumerator +{ + /// Device class GUID for display adapters. + private const string DisplayClassKey = + @"SYSTEM\CurrentControlSet\Control\Class\{4d36e968-e325-11ce-bfc1-08002be10318}"; + + /// + /// Read the installed adapters. Never throws: an unreadable registry yields + /// an empty list, which the probe treats as "no GPU detected". + /// + public static IReadOnlyList Enumerate() + { + var adapters = new List(); + + try + { + using var classKey = Registry.LocalMachine.OpenSubKey(DisplayClassKey); + if (classKey is null) return adapters; + + foreach (var subKeyName in classKey.GetSubKeyNames()) + { + // Instance subkeys are four digits ("0000"). Anything else is + // configuration state, not an adapter. + if (subKeyName.Length != 4 || !uint.TryParse(subKeyName, out _)) continue; + + try + { + using var instance = classKey.OpenSubKey(subKeyName); + if (instance?.GetValue("DriverDesc") is not string name || name.Length == 0) continue; + + adapters.Add(new GpuInfo(NvidiaSmiParser.ClassifyVendor(name), name)); + } + catch (Exception) + { + // A single unreadable instance must not hide the others. + } + } + } + catch (Exception) + { + // Restricted registry access degrades to "no adapters", matching the + // probe's never-throw contract. + return adapters; + } + + return adapters; + } +} diff --git a/src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs b/src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs index 1dcdd306f..591550413 100644 --- a/src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs +++ b/src/OpenClaw.Tray.WinUI/Services/SettingsManager.cs @@ -128,6 +128,25 @@ public List UserRules /// Play audio feedback chimes on listen start/stop. public bool VoiceAudioFeedback { get => _data.VoiceAudioFeedback; set => _data = _data with { VoiceAudioFeedback = value }; } public bool NodeTtsEnabled { get => _data.NodeTtsEnabled; set => _data = _data with { NodeTtsEnabled = value }; } + + // ── Local inference (llama.cpp) ── + + /// Master switch for running a model locally. + public bool LocalInferenceEnabled { get => _data.LocalInferenceEnabled; set => _data = _data with { LocalInferenceEnabled = value }; } + /// Catalog id of the chosen model, or null to follow the recommendation. + public string? LocalInferenceModelId { get => _data.LocalInferenceModelId; set => _data = _data with { LocalInferenceModelId = value }; } + /// Backend override name, or null for automatic selection. + public string? LocalInferenceBackendOverride { get => _data.LocalInferenceBackendOverride; set => _data = _data with { LocalInferenceBackendOverride = value }; } + /// Path to a user-built llama-server, which skips hash verification. + public string? LocalInferenceCustomRuntimePath { get => _data.LocalInferenceCustomRuntimePath; set => _data = _data with { LocalInferenceCustomRuntimePath = value }; } + /// Fixed server port, or null to allocate a free one. + public int? LocalInferencePort { get => _data.LocalInferencePort; set => _data = _data with { LocalInferencePort = value }; } + /// Start the local server on app start. + public bool LocalInferenceAutoStart { get => _data.LocalInferenceAutoStart; set => _data = _data with { LocalInferenceAutoStart = value }; } + /// Register the healthy local endpoint with the gateway. + public bool LocalInferenceRegisterWithGateway { get => _data.LocalInferenceRegisterWithGateway; set => _data = _data with { LocalInferenceRegisterWithGateway = value }; } + /// Bind beyond loopback, exposing an unauthenticated endpoint to the network. + public bool LocalInferenceBindBeyondLoopback { get => _data.LocalInferenceBindBeyondLoopback; set => _data = _data with { LocalInferenceBindBeyondLoopback = value }; } public string TtsProvider { get => string.IsNullOrWhiteSpace(_data.TtsProvider) ? TtsCapability.PiperProvider : _data.TtsProvider; set => _data = _data with { TtsProvider = value }; } public string TtsElevenLabsApiKey { get => _data.TtsElevenLabsApiKey ?? ""; set => _data = _data with { TtsElevenLabsApiKey = value }; } public string TtsElevenLabsModel { get => _data.TtsElevenLabsModel ?? ""; set => _data = _data with { TtsElevenLabsModel = value }; } diff --git a/src/OpenClaw.Tray.WinUI/Strings/en-us/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/en-us/Resources.resw index 02c292072..d6c2e661c 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/en-us/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/en-us/Resources.resw @@ -6988,4 +6988,160 @@ Make sure the gateway is running. Last run stopped + + Local Inference + + + Run a model on this machine with llama.cpp. Weights and the runtime are downloaded once and stay on your device. + + + Detected hardware + + + Re-detect hardware + + + Model + + + Model + + + Download model + + + Cancel + + + Delete + + + Server + + + Enable local inference + + + Start + + + Stop + + + OpenAI-compatible endpoint + + + Copy + + + Advanced + + + Compute backend + + + Custom llama-server build (optional) + + + Path to llama-server.exe or its folder + + + A custom build is in use. It is not integrity verified, so only point this at a binary you built or trust. + + + Start the server when the app starts + + + Register this endpoint with the gateway + + + Accept connections from other machines + + + Needed when the gateway runs in a WSL distro that cannot reach this machine on localhost. The endpoint has no authentication, so anyone who can reach this port can use the model. + + + Hardware could not be detected. + + + Detecting hardware... + + + {0} RAM + + + No supported GPU detected + + + Automatic (recommended) + + + {0} (not available yet) + + + Model ready + + + Not downloaded + + + {0} of {1} files downloaded + + + This model is about {0} and will take a long time to download. + + + Preparing... + + + Downloading runtime: {0}% + + + Downloading model: {0}% + + + Download canceled + + + Download failed. See the log for details. + + + Stopped + + + Starting, loading weights... + + + Ready on port {0} + + + The server could not start + + + Download this model? + + + {0} is about {1}. The download can take hours and cannot be paused across app restarts without re-checking each file. + + + Download + + + Cancel + + + Delete this model? + + + This frees about {1} of disk space. Downloading {0} again will take the same time as the first download. + + + Delete + + + Cancel + + + Could not delete the model. See the log for details. + diff --git a/src/OpenClaw.Tray.WinUI/Strings/fr-fr/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/fr-fr/Resources.resw index 60e556866..714732933 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/fr-fr/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/fr-fr/Resources.resw @@ -6948,4 +6948,160 @@ Le binaire wxc-exec est introuvable. {1} S'il s'agit d'une build développeur, c Dernière exécution arrêtée + + Local Inference + + + Run a model on this machine with llama.cpp. Weights and the runtime are downloaded once and stay on your device. + + + Detected hardware + + + Re-detect hardware + + + Model + + + Model + + + Download model + + + Cancel + + + Delete + + + Server + + + Enable local inference + + + Start + + + Stop + + + OpenAI-compatible endpoint + + + Copy + + + Advanced + + + Compute backend + + + Custom llama-server build (optional) + + + Path to llama-server.exe or its folder + + + A custom build is in use. It is not integrity verified, so only point this at a binary you built or trust. + + + Start the server when the app starts + + + Register this endpoint with the gateway + + + Accept connections from other machines + + + Needed when the gateway runs in a WSL distro that cannot reach this machine on localhost. The endpoint has no authentication, so anyone who can reach this port can use the model. + + + Hardware could not be detected. + + + Detecting hardware... + + + {0} RAM + + + No supported GPU detected + + + Automatic (recommended) + + + {0} (not available yet) + + + Model ready + + + Not downloaded + + + {0} of {1} files downloaded + + + This model is about {0} and will take a long time to download. + + + Preparing... + + + Downloading runtime: {0}% + + + Downloading model: {0}% + + + Download canceled + + + Download failed. See the log for details. + + + Stopped + + + Starting, loading weights... + + + Ready on port {0} + + + The server could not start + + + Download this model? + + + {0} is about {1}. The download can take hours and cannot be paused across app restarts without re-checking each file. + + + Download + + + Cancel + + + Delete this model? + + + This frees about {1} of disk space. Downloading {0} again will take the same time as the first download. + + + Delete + + + Cancel + + + Could not delete the model. See the log for details. + diff --git a/src/OpenClaw.Tray.WinUI/Strings/nl-nl/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/nl-nl/Resources.resw index cbb9ea5f2..84e0b0ec8 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/nl-nl/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/nl-nl/Resources.resw @@ -6949,4 +6949,160 @@ Het binaire bestand wxc-exec is niet gevonden. {1} Als dit een ontwikkelaarsbuil Laatste uitvoering gestopt + + Local Inference + + + Run a model on this machine with llama.cpp. Weights and the runtime are downloaded once and stay on your device. + + + Detected hardware + + + Re-detect hardware + + + Model + + + Model + + + Download model + + + Cancel + + + Delete + + + Server + + + Enable local inference + + + Start + + + Stop + + + OpenAI-compatible endpoint + + + Copy + + + Advanced + + + Compute backend + + + Custom llama-server build (optional) + + + Path to llama-server.exe or its folder + + + A custom build is in use. It is not integrity verified, so only point this at a binary you built or trust. + + + Start the server when the app starts + + + Register this endpoint with the gateway + + + Accept connections from other machines + + + Needed when the gateway runs in a WSL distro that cannot reach this machine on localhost. The endpoint has no authentication, so anyone who can reach this port can use the model. + + + Hardware could not be detected. + + + Detecting hardware... + + + {0} RAM + + + No supported GPU detected + + + Automatic (recommended) + + + {0} (not available yet) + + + Model ready + + + Not downloaded + + + {0} of {1} files downloaded + + + This model is about {0} and will take a long time to download. + + + Preparing... + + + Downloading runtime: {0}% + + + Downloading model: {0}% + + + Download canceled + + + Download failed. See the log for details. + + + Stopped + + + Starting, loading weights... + + + Ready on port {0} + + + The server could not start + + + Download this model? + + + {0} is about {1}. The download can take hours and cannot be paused across app restarts without re-checking each file. + + + Download + + + Cancel + + + Delete this model? + + + This frees about {1} of disk space. Downloading {0} again will take the same time as the first download. + + + Delete + + + Cancel + + + Could not delete the model. See the log for details. + diff --git a/src/OpenClaw.Tray.WinUI/Strings/zh-cn/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/zh-cn/Resources.resw index bf824cb7a..5c1b06a9e 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/zh-cn/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/zh-cn/Resources.resw @@ -6948,4 +6948,160 @@ 上次运行已停止 + + Local Inference + + + Run a model on this machine with llama.cpp. Weights and the runtime are downloaded once and stay on your device. + + + Detected hardware + + + Re-detect hardware + + + Model + + + Model + + + Download model + + + Cancel + + + Delete + + + Server + + + Enable local inference + + + Start + + + Stop + + + OpenAI-compatible endpoint + + + Copy + + + Advanced + + + Compute backend + + + Custom llama-server build (optional) + + + Path to llama-server.exe or its folder + + + A custom build is in use. It is not integrity verified, so only point this at a binary you built or trust. + + + Start the server when the app starts + + + Register this endpoint with the gateway + + + Accept connections from other machines + + + Needed when the gateway runs in a WSL distro that cannot reach this machine on localhost. The endpoint has no authentication, so anyone who can reach this port can use the model. + + + Hardware could not be detected. + + + Detecting hardware... + + + {0} RAM + + + No supported GPU detected + + + Automatic (recommended) + + + {0} (not available yet) + + + Model ready + + + Not downloaded + + + {0} of {1} files downloaded + + + This model is about {0} and will take a long time to download. + + + Preparing... + + + Downloading runtime: {0}% + + + Downloading model: {0}% + + + Download canceled + + + Download failed. See the log for details. + + + Stopped + + + Starting, loading weights... + + + Ready on port {0} + + + The server could not start + + + Download this model? + + + {0} is about {1}. The download can take hours and cannot be paused across app restarts without re-checking each file. + + + Download + + + Cancel + + + Delete this model? + + + This frees about {1} of disk space. Downloading {0} again will take the same time as the first download. + + + Delete + + + Cancel + + + Could not delete the model. See the log for details. + diff --git a/src/OpenClaw.Tray.WinUI/Strings/zh-tw/Resources.resw b/src/OpenClaw.Tray.WinUI/Strings/zh-tw/Resources.resw index 58acfbc95..f5e06bb3a 100644 --- a/src/OpenClaw.Tray.WinUI/Strings/zh-tw/Resources.resw +++ b/src/OpenClaw.Tray.WinUI/Strings/zh-tw/Resources.resw @@ -6948,4 +6948,160 @@ 上次執行已停止 + + Local Inference + + + Run a model on this machine with llama.cpp. Weights and the runtime are downloaded once and stay on your device. + + + Detected hardware + + + Re-detect hardware + + + Model + + + Model + + + Download model + + + Cancel + + + Delete + + + Server + + + Enable local inference + + + Start + + + Stop + + + OpenAI-compatible endpoint + + + Copy + + + Advanced + + + Compute backend + + + Custom llama-server build (optional) + + + Path to llama-server.exe or its folder + + + A custom build is in use. It is not integrity verified, so only point this at a binary you built or trust. + + + Start the server when the app starts + + + Register this endpoint with the gateway + + + Accept connections from other machines + + + Needed when the gateway runs in a WSL distro that cannot reach this machine on localhost. The endpoint has no authentication, so anyone who can reach this port can use the model. + + + Hardware could not be detected. + + + Detecting hardware... + + + {0} RAM + + + No supported GPU detected + + + Automatic (recommended) + + + {0} (not available yet) + + + Model ready + + + Not downloaded + + + {0} of {1} files downloaded + + + This model is about {0} and will take a long time to download. + + + Preparing... + + + Downloading runtime: {0}% + + + Downloading model: {0}% + + + Download canceled + + + Download failed. See the log for details. + + + Stopped + + + Starting, loading weights... + + + Ready on port {0} + + + The server could not start + + + Download this model? + + + {0} is about {1}. The download can take hours and cannot be paused across app restarts without re-checking each file. + + + Download + + + Cancel + + + Delete this model? + + + This frees about {1} of disk space. Downloading {0} again will take the same time as the first download. + + + Delete + + + Cancel + + + Could not delete the model. See the log for details. + diff --git a/src/OpenClaw.Tray.WinUI/Windows/HubWindow.xaml.cs b/src/OpenClaw.Tray.WinUI/Windows/HubWindow.xaml.cs index ec54148f0..657b167d5 100644 --- a/src/OpenClaw.Tray.WinUI/Windows/HubWindow.xaml.cs +++ b/src/OpenClaw.Tray.WinUI/Windows/HubWindow.xaml.cs @@ -1089,6 +1089,7 @@ private void InitializeCurrentPage() case PermissionsPage: break; case SandboxPage sandbox: sandbox.Initialize(); break; case VoiceSettingsPage voice: voice.Initialize(CurrentApp.VoiceService); break; + case LocalInferencePage localInference: localInference.Initialize(CurrentApp.LocalInference); break; case AgentEventsPage agentEvents: agentEvents.Initialize(this); agentEvents.ClearCentralCache = () => AppModel?.ClearAgentEvents(); diff --git a/tests/OpenClaw.Shared.Tests/Inference/BackendSelectorTests.cs b/tests/OpenClaw.Shared.Tests/Inference/BackendSelectorTests.cs index e11892587..df7ddde2c 100644 --- a/tests/OpenClaw.Shared.Tests/Inference/BackendSelectorTests.cs +++ b/tests/OpenClaw.Shared.Tests/Inference/BackendSelectorTests.cs @@ -90,6 +90,28 @@ public void NvidiaPlan_FallsBackThroughTheOtherCudaBuildThenCpu() order); } + [Fact] + public void Arm64WithACuda12Driver_UsesTheCuda13BuildAndSaysWhy() + { + // The pinned release ships no CUDA 12 ARM64 build, so an ARM64 host with + // a CUDA 12 driver has to take the CUDA 13 runtime. That may not load, + // and the user should read why rather than a bare driver error. + var plan = BackendSelector.Select(Host(Arch.Arm64, gpus: [Nvidia(cudaMajor: 12)])); + + Assert.Equal(LlamaBackend.Cuda13, plan.Preferred!.Backend); + Assert.Equal(Arch.Arm64, plan.Preferred.Architecture); + Assert.Contains("may not load", plan.Reason, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public void MatchingCudaVersion_DoesNotWarnAboutTheRuntime() + { + var plan = BackendSelector.Select(Host(gpus: [Nvidia(cudaMajor: 12)])); + + Assert.Equal(LlamaBackend.Cuda12, plan.Preferred!.Backend); + Assert.DoesNotContain("may not load", plan.Reason, StringComparison.OrdinalIgnoreCase); + } + [Fact] public void AmdGpuWithVulkanLoader_SelectsVulkan() { diff --git a/tests/OpenClaw.Shared.Tests/Inference/LlamaServerArgumentsTests.cs b/tests/OpenClaw.Shared.Tests/Inference/LlamaServerArgumentsTests.cs new file mode 100644 index 000000000..9e78deca1 --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/LlamaServerArgumentsTests.cs @@ -0,0 +1,97 @@ +using System; +using System.Linq; +using OpenClaw.Shared.Inference; +using Xunit; + +namespace OpenClaw.Shared.Tests.Inference; + +public class LlamaServerArgumentsTests +{ + private static readonly string[] Recipe = ["--temp", "1.0", "-dio"]; + + [Fact] + public void PutsLauncherOwnedFlagsFirstAndAppendsTheRecipe() + { + var args = LlamaServerArguments.Build(@"C:\models\m.gguf", 8080, Recipe); + + Assert.Equal( + ["-m", @"C:\models\m.gguf", "--host", "127.0.0.1", "--port", "8080", "--temp", "1.0", "-dio"], + args); + } + + [Fact] + public void BindsLoopbackByDefault() + { + var args = LlamaServerArguments.Build(@"C:\m.gguf", 1234, Recipe); + + var host = args[args.ToList().IndexOf("--host") + 1]; + Assert.Equal("127.0.0.1", host); + } + + [Fact] + public void BindsAllInterfacesOnlyWhenExplicitlyRequested() + { + // Beyond-loopback exposes an unauthenticated endpoint to the network, so + // it must never be reachable by accident. + var args = LlamaServerArguments.Build(@"C:\m.gguf", 1234, Recipe, bindBeyondLoopback: true); + + var host = args[args.ToList().IndexOf("--host") + 1]; + Assert.Equal("0.0.0.0", host); + } + + [Theory] + [InlineData("-m")] + [InlineData("--model")] + [InlineData("--host")] + [InlineData("--port")] + public void RejectsARecipeThatSetsALauncherOwnedFlag(string flag) + { + // Such a recipe would either duplicate the flag or silently win. + var ex = Assert.Throws(() => + LlamaServerArguments.Build(@"C:\m.gguf", 8080, ["--temp", "1.0", flag, "x"])); + + Assert.Contains(flag, ex.Message, StringComparison.Ordinal); + } + + [Fact] + public void EveryCatalogRecipeIsAcceptedByTheBuilder() + { + // Guards the catalog against a recipe that cannot actually be launched. + foreach (var model in LocalModelCatalog.Models) + { + var args = LlamaServerArguments.Build(@"C:\m.gguf", 8080, model.RecipeArgs); + Assert.All(model.RecipeArgs, a => Assert.Contains(a, args)); + } + } + + [Theory] + [InlineData(0)] + [InlineData(-1)] + [InlineData(65536)] + public void RejectsAnOutOfRangePort(int port) + { + Assert.Throws(() => + LlamaServerArguments.Build(@"C:\m.gguf", port, Recipe)); + } + + [Fact] + public void RejectsAnEmptyModelPath() + { + Assert.Throws(() => LlamaServerArguments.Build(" ", 8080, Recipe)); + } + + [Fact] + public void HealthUrlAlwaysTargetsLoopback() + { + // The health poll runs on this machine even when the server binds wider. + Assert.Equal("http://127.0.0.1:9000/health", LlamaServerArguments.BuildHealthUrl(9000)); + } + + [Fact] + public void BaseUrlCarriesTheOpenAiCompatibleSuffix() + { + Assert.Equal("http://127.0.0.1:9000/v1", LlamaServerArguments.BuildBaseUrl("127.0.0.1", 9000)); + Assert.Equal("http://host.docker.internal:9000/v1", + LlamaServerArguments.BuildBaseUrl("host.docker.internal", 9000)); + } +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/LlamaServerProcessTests.cs b/tests/OpenClaw.Shared.Tests/Inference/LlamaServerProcessTests.cs new file mode 100644 index 000000000..587d8ffc2 --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/LlamaServerProcessTests.cs @@ -0,0 +1,173 @@ +using System; +using System.Diagnostics; +using System.IO; +using System.Threading.Tasks; +using OpenClaw.Shared; +using OpenClaw.Shared.Inference; +using OpenClaw.TestSupport; +using Xunit; + +namespace OpenClaw.Shared.Tests.Inference; + +/// +/// Covers the guard paths that must not require a real llama-server: bad inputs, +/// and a child that exits during startup. Readiness against a genuine server is +/// manual proof (see docs/LOCAL_INFERENCE_PLAN.md); what is pinned here is that +/// every failure is prompt and explained rather than a silent ten-minute hang. +/// +public class LlamaServerProcessTests +{ + [Fact] + public void StartsOutStopped() + { + using var temp = new TempDirectory(); + var server = new LlamaServerProcess(NullLogger.Instance); + + Assert.Equal(LlamaServerState.Stopped, server.Status.State); + Assert.False(server.IsRunning); + Assert.Null(server.Status.LoopbackBaseUrl); + } + + [Fact] + public async Task FailsWhenTheServerExecutableIsMissing() + { + using var temp = new TempDirectory(); + var modelPath = temp.Combine("model.gguf"); + await File.WriteAllTextAsync(modelPath, "weights"); + + await using var server = new LlamaServerProcess(NullLogger.Instance); + var runtime = new LlamaRuntime(temp.Combine("does-not-exist.exe"), LlamaRuntimeSource.Catalog, null); + + var status = await server.StartAsync(runtime, modelPath, []); + + Assert.Equal(LlamaServerState.Failed, status.State); + Assert.Contains("not found", status.Detail!, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public async Task FailsWhenTheModelIsNotDownloaded() + { + using var temp = new TempDirectory(); + var exe = temp.Combine("llama-server.exe"); + await File.WriteAllTextAsync(exe, "stub"); + + await using var server = new LlamaServerProcess(NullLogger.Instance); + var runtime = new LlamaRuntime(exe, LlamaRuntimeSource.Catalog, null); + + var status = await server.StartAsync(runtime, temp.Combine("missing.gguf"), []); + + Assert.Equal(LlamaServerState.Failed, status.State); + Assert.Contains("not downloaded", status.Detail!, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public async Task ReportsPromptlyWhenTheChildExitsDuringStartup() + { + // A recipe flag an older build rejects makes llama-server exit at once. + // The health poll must notice the exit instead of waiting out the full + // ready timeout, and the failure must carry the captured output. + var stub = Path.Combine(Environment.SystemDirectory, "ping.exe"); + Assert.True(File.Exists(stub), "This test needs ping.exe from System32 as a fast-exiting stub."); + + using var temp = new TempDirectory(); + var modelPath = temp.Combine("model.gguf"); + await File.WriteAllTextAsync(modelPath, "weights"); + + await using var server = new LlamaServerProcess(NullLogger.Instance); + var runtime = new LlamaRuntime(stub, LlamaRuntimeSource.Catalog, null); + + var stopwatch = Stopwatch.StartNew(); + var status = await server.StartAsync( + runtime, modelPath, [], readyTimeout: TimeSpan.FromMinutes(5)); + stopwatch.Stop(); + + Assert.Equal(LlamaServerState.Failed, status.State); + Assert.Contains("exited", status.Detail!, StringComparison.OrdinalIgnoreCase); + Assert.True( + stopwatch.Elapsed < TimeSpan.FromSeconds(30), + $"Startup failure took {stopwatch.Elapsed}, so the exited-child check did not short-circuit the poll."); + Assert.False(server.IsRunning); + } + + [Fact] + public async Task RaisesStatusChangedForEachTransition() + { + using var temp = new TempDirectory(); + var modelPath = temp.Combine("model.gguf"); + await File.WriteAllTextAsync(modelPath, "weights"); + + await using var server = new LlamaServerProcess(NullLogger.Instance); + var states = new System.Collections.Generic.List(); + server.StatusChanged += (_, s) => states.Add(s.State); + + await server.StartAsync( + new LlamaRuntime(temp.Combine("missing.exe"), LlamaRuntimeSource.Catalog, null), + modelPath, + []); + + Assert.Contains(LlamaServerState.Failed, states); + } + + [Fact] + public async Task StopIsSafeWhenNothingIsRunning() + { + await using var server = new LlamaServerProcess(NullLogger.Instance); + + await server.StopAsync(); + await server.StopAsync(); + + Assert.Equal(LlamaServerState.Stopped, server.Status.State); + } + + [Fact] + public void LoopbackBaseUrlIsOnlyExposedWhenReady() + { + Assert.Null(new LlamaServerStatus(LlamaServerState.Starting, 8080).LoopbackBaseUrl); + Assert.Null(new LlamaServerStatus(LlamaServerState.Failed, 8080).LoopbackBaseUrl); + Assert.Equal( + "http://127.0.0.1:8080/v1", + new LlamaServerStatus(LlamaServerState.Ready, 8080).LoopbackBaseUrl); + } + + [Fact] + public void JobObjectIsCreatableOnThisHost() + { + // If this ever fails, llama-server can outlive an abrupt app shutdown + // while holding tens of gigabytes of VRAM. + using var job = new ProcessJobObject(); + + Assert.True(job.IsValid); + } + + [Fact] + public async Task JobObjectTerminatesAnAssignedProcessWhenDisposed() + { + var stub = Path.Combine(Environment.SystemDirectory, "ping.exe"); + Assert.True(File.Exists(stub), "This test needs ping.exe from System32."); + + // -t pings forever, so the process only ends because the job ends. + using var process = Process.Start(new ProcessStartInfo(stub) + { + ArgumentList = { "-t", "127.0.0.1" }, + UseShellExecute = false, + CreateNoWindow = true, + RedirectStandardOutput = true, + })!; + + try + { + var job = new ProcessJobObject(); + Assert.True(job.TryAssign(process.Handle)); + + job.Dispose(); + + await process.WaitForExitAsync(new System.Threading.CancellationTokenSource(TimeSpan.FromSeconds(15)).Token); + Assert.True(process.HasExited); + } + finally + { + // slopwatch-ignore: SW003 Test cleanup is best-effort and must not mask the assertion above. + try { if (!process.HasExited) process.Kill(entireProcessTree: true); } catch { } + } + } +} diff --git a/tests/OpenClaw.Shared.Tests/Inference/LocalInferenceServiceTests.cs b/tests/OpenClaw.Shared.Tests/Inference/LocalInferenceServiceTests.cs new file mode 100644 index 000000000..60c1f6a65 --- /dev/null +++ b/tests/OpenClaw.Shared.Tests/Inference/LocalInferenceServiceTests.cs @@ -0,0 +1,213 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using OpenClaw.Shared; +using OpenClaw.Shared.Inference; +using OpenClaw.TestSupport; +using Xunit; + +namespace OpenClaw.Shared.Tests.Inference; + +public class LocalInferenceServiceTests +{ + [Theory] + [InlineData("Cuda12", LlamaBackend.Cuda12)] + [InlineData("cuda13", LlamaBackend.Cuda13)] + [InlineData(" Vulkan ", LlamaBackend.Vulkan)] + [InlineData("Cpu", LlamaBackend.Cpu)] + public void ParsesAKnownBackendOverride(string value, LlamaBackend expected) + { + Assert.Equal(expected, LocalInferenceService.ParseBackendOverride(value)); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData(" ")] + [InlineData("Rocm")] + [InlineData("nonsense")] + public void TreatsAnUnknownBackendOverrideAsAutomatic(string? value) + { + // A hand-edited settings file must degrade to automatic selection rather + // than breaking the page. + Assert.Null(LocalInferenceService.ParseBackendOverride(value)); + } + + [Fact] + public async Task SnapshotReportsHardwareBackendAndRecommendation() + { + using var temp = new TempDirectory(); + var settings = new SettingsData(); + await using var service = NewService(temp, () => settings, NvidiaHost()); + + var snapshot = await service.GetSnapshotAsync(); + + Assert.NotNull(snapshot.Hardware); + Assert.True(snapshot.Hardware!.HasNvidiaGpu); + + // Architecture-dependent on purpose: this feature's dev host is ARM64 + // with an NVIDIA GPU, where the pinned release ships only a CUDA 13 + // build, so asserting a specific CUDA major here would fail on one of + // the two architectures we support. + var preferred = snapshot.BackendPlan!.Preferred!; + Assert.Contains(preferred.Backend, new[] { LlamaBackend.Cuda12, LlamaBackend.Cuda13 }); + Assert.Equal(snapshot.Hardware.CpuArchitecture, preferred.Architecture); + + Assert.False(snapshot.RuntimeInstalled); + Assert.False(snapshot.UsingCustomRuntime); + Assert.Equal(LlamaServerState.Stopped, snapshot.ServerStatus.State); + } + + [Fact] + public async Task AnExplicitModelChoiceOverridesTheRecommendation() + { + using var temp = new TempDirectory(); + var settings = new SettingsData { LocalInferenceModelId = LocalModelCatalog.DeepSeekV4FlashId }; + await using var service = NewService(temp, () => settings, NvidiaHost()); + + var snapshot = await service.GetSnapshotAsync(); + + Assert.Equal(LocalModelCatalog.DeepSeekV4FlashId, snapshot.SelectedModel!.Id); + } + + [Fact] + public async Task ABackendOverrideIsHonoredInTheSnapshot() + { + using var temp = new TempDirectory(); + var settings = new SettingsData { LocalInferenceBackendOverride = "Cpu" }; + await using var service = NewService(temp, () => settings, NvidiaHost()); + + var snapshot = await service.GetSnapshotAsync(); + + Assert.Equal(LlamaBackend.Cpu, snapshot.BackendPlan!.Preferred!.Backend); + } + + [Fact] + public async Task AConfiguredCustomRuntimeIsReportedInTheSnapshot() + { + // The page relies on this to show the "not integrity verified" notice. + using var temp = new TempDirectory(); + var settings = new SettingsData { LocalInferenceCustomRuntimePath = temp.Combine("build") }; + await using var service = NewService(temp, () => settings, NvidiaHost()); + + var snapshot = await service.GetSnapshotAsync(); + + Assert.True(snapshot.UsingCustomRuntime); + Assert.True(snapshot.RuntimeInstalled); + } + + [Fact] + public async Task StartRefusesWhenTheModelIsNotDownloaded() + { + // Start must not silently kick off a multi-hour download. + using var temp = new TempDirectory(); + var settings = new SettingsData(); + await using var service = NewService(temp, () => settings, NvidiaHost()); + + var ex = await Assert.ThrowsAsync(() => service.StartAsync()); + + Assert.Contains("not downloaded", ex.Message, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public async Task StartRefusesAnUnpublishedCheckpoint() + { + // Deterministic regardless of host memory: this checkpoint has no shards + // at all, so there is nothing on disk to launch against. + using var temp = new TempDirectory(); + var settings = new SettingsData { LocalInferenceModelId = LocalModelCatalog.Qwen27BId }; + await using var service = NewService(temp, () => settings, NvidiaHost()); + + var ex = await Assert.ThrowsAsync(() => service.StartAsync()); + + Assert.Contains("not downloaded", ex.Message, StringComparison.OrdinalIgnoreCase); + } + + [Fact] + public async Task EnsureRuntimeResolvesTheCustomBuildWithoutDownloading() + { + using var temp = new TempDirectory(); + var buildDir = temp.Combine("mybuild"); + System.IO.Directory.CreateDirectory(buildDir); + var exe = System.IO.Path.Combine(buildDir, LlamaBackendCatalog.ServerExecutableName); + await System.IO.File.WriteAllTextAsync(exe, "custom"); + + var settings = new SettingsData { LocalInferenceCustomRuntimePath = buildDir }; + var transport = new FakeHttpTransport(); + await using var service = NewService(temp, () => settings, NvidiaHost(), transport); + + var runtime = await service.EnsureRuntimeAsync(); + + Assert.Equal(LlamaRuntimeSource.CustomBuild, runtime.Source); + Assert.True(runtime.IsUnverified); + Assert.Empty(transport.Requests); + } + + [Fact] + public async Task ResolveSelectedModelFallsBackToTheRecommendation() + { + using var temp = new TempDirectory(); + var settings = new SettingsData(); + await using var service = NewService(temp, () => settings, NvidiaHost()); + var snapshot = await service.GetSnapshotAsync(); + + var resolved = service.ResolveSelectedModel(settings, snapshot.Recommendation); + + Assert.Equal(snapshot.Recommendation!.Recommended!.Id, resolved!.Id); + } + + [Fact] + public async Task ServerStatusChangesAreForwardedToSubscribers() + { + // The page renders from this event rather than polling. + using var temp = new TempDirectory(); + var settings = new SettingsData(); + await using var service = NewService(temp, () => settings, NvidiaHost()); + + var seen = new List(); + service.ServerStatusChanged += (_, s) => seen.Add(s.State); + + await service.StopAsync(); + + Assert.Contains(LlamaServerState.Stopped, seen); + } + + private static LocalInferenceService NewService( + TempDirectory temp, + Func settings, + Func, CommandResult> nvidiaSmi, + FakeHttpTransport? transport = null) + { + transport ??= new FakeHttpTransport(); + var logger = NullLogger.Instance; + var downloader = new VerifiedFileDownloader(logger, transport.ClientFactory); + + return new LocalInferenceService( + new HardwareProbe(new ScriptedRunner(nvidiaSmi), logger, vulkanLoaderProbe: () => false), + new LlamaRuntimeManager(temp.Path, logger, downloader), + new GgufModelManager(temp.Path, logger, downloader, freeSpaceProbe: _ => 512L * 1024 * 1024 * 1024), + new LlamaServerProcess(logger), + settings, + logger); + } + + /// A workstation with a large NVIDIA GPU and plenty of RAM. + private static Func, CommandResult> NvidiaHost() => argv => + argv.Any(a => a.StartsWith("--query-gpu", StringComparison.Ordinal)) + ? new CommandResult { ExitCode = 0, Stdout = "NVIDIA RTX 6000 Ada Generation, 49140, 570.86.10" } + : new CommandResult { ExitCode = 0, Stdout = "CUDA Version: 12.8" }; + + /// A machine with no GPU, where nothing in the catalog fits. + private static Func, CommandResult> TinyHost() => + _ => new CommandResult { ExitCode = 1, Stdout = "" }; + + private sealed class ScriptedRunner(Func, CommandResult> respond) : ICommandRunner + { + public string Name => "scripted"; + + public Task RunAsync(CommandRequest request, CancellationToken ct = default) => + Task.FromResult(respond(request.Argv ?? [])); + } +} diff --git a/tests/OpenClaw.Tray.Tests/LocalizationValidationTests.cs b/tests/OpenClaw.Tray.Tests/LocalizationValidationTests.cs index 0cdec5a5f..0722bd1c4 100644 --- a/tests/OpenClaw.Tray.Tests/LocalizationValidationTests.cs +++ b/tests/OpenClaw.Tray.Tests/LocalizationValidationTests.cs @@ -239,6 +239,63 @@ public class LocalizationValidationTests // / CronPage entries above. The Description_Format key takes the WSL // distro name as {0} and is formatted in ConnectionPage.xaml.cs. "ConnectionPage_GatewayHostControlsDescription_Format", + // LocalInferencePage (local llama.cpp inference) — seeded English-only + // across all 5 locales using the deferred-translation pattern, same + // precedent as the ConfigPage / SessionsPage / CronPage runtime keys + // above. Several take format placeholders: {0} is a size, a model name, + // or a port depending on the key. + "LocalInferencePage_PageTitle.Text", + "LocalInferencePage_PageDescription.Text", + "LocalInferencePage_HardwareHeader.Text", + "LocalInferencePage_RedetectButtonText.Text", + "LocalInferencePage_ModelHeader.Text", + "LocalInferencePage_ModelCombo.Header", + "LocalInferencePage_DownloadButtonText.Text", + "LocalInferencePage_CancelDownloadButtonText.Text", + "LocalInferencePage_DeleteModelButtonText.Text", + "LocalInferencePage_ServerHeader.Text", + "LocalInferencePage_EnabledToggle.Header", + "LocalInferencePage_StartButtonText.Text", + "LocalInferencePage_StopButtonText.Text", + "LocalInferencePage_EndpointLabel.Text", + "LocalInferencePage_CopyEndpointButtonText.Text", + "LocalInferencePage_AdvancedHeader.Text", + "LocalInferencePage_BackendCombo.Header", + "LocalInferencePage_CustomRuntimeBox.Header", + "LocalInferencePage_CustomRuntimeBox.PlaceholderText", + "LocalInferencePage_CustomRuntimeNoticeText.Text", + "LocalInferencePage_AutoStartCheck.Content", + "LocalInferencePage_RegisterCheck.Content", + "LocalInferencePage_BindBeyondLoopbackCheck.Content", + "LocalInferencePage_BindBeyondLoopbackHelp.Text", + "LocalInferencePage_HardwareUnavailable", + "LocalInferencePage_HardwareDetecting", + "LocalInferencePage_HardwareRam", + "LocalInferencePage_HardwareNoGpu", + "LocalInferencePage_BackendAutomatic", + "LocalInferencePage_ModelUnavailableSuffix", + "LocalInferencePage_ModelReady", + "LocalInferencePage_ModelNotDownloaded", + "LocalInferencePage_ModelPartial", + "LocalInferencePage_LargeDownloadNotice", + "LocalInferencePage_StatusPreparing", + "LocalInferencePage_StatusRuntimePct", + "LocalInferencePage_StatusModelPct", + "LocalInferencePage_StatusDownloadCanceled", + "LocalInferencePage_StatusDownloadError", + "LocalInferencePage_ServerStopped", + "LocalInferencePage_ServerStarting", + "LocalInferencePage_ServerReady", + "LocalInferencePage_ServerFailed", + "LocalInferencePage_ConfirmDownloadTitle", + "LocalInferencePage_ConfirmDownloadBody", + "LocalInferencePage_ConfirmDownloadPrimary", + "LocalInferencePage_ConfirmDownloadCancel", + "LocalInferencePage_ConfirmDeleteTitle", + "LocalInferencePage_ConfirmDeleteBody", + "LocalInferencePage_ConfirmDeletePrimary", + "LocalInferencePage_ConfirmCancelButtonText.Text", + "LocalInferencePage_StatusDeleteError", // GatewayHostAccess plan strings (terminal label / tooltip / disabled // reasons). Resolved in the classifier via LocalizationHelper so the // OpenTerminal button and any consumers of DisabledReason show From 28628f49f073a776e0cff1016ce0fa0a97b954aa Mon Sep 17 00:00:00 2001 From: Pedro Larroy Date: Tue, 18 Aug 2026 21:37:48 -0700 Subject: [PATCH 5/7] fix(inference): correct nvidia-smi parsing for current NVIDIA drivers Probing the real dev host (a GB10 DGX Spark: ARM64 Windows, RTX Spark N1X, driver 616.29) surfaced two parser bugs that made the hardware probe wrong on that exact hardware. 1. The CUDA version banner. Older drivers print "CUDA Version: 12.8"; 616.29 prints "CUDA UMD Version: 13.4", which does not contain the older marker as a substring. The probe therefore reported an unknown CUDA version and, per the deliberate degrade-downward rule, preferred the CUDA 12 build. Both spellings are now matched. 2. The NVIDIA NPU. --query-gpu lists it alongside the GPU because it shares the driver, but it is not a CUDA device. It was being counted as a second adapter with unknown VRAM, which would show a phantom entry in the UI and imply an accelerator llama.cpp cannot use. Rows whose name matches NPU as a whole word are skipped. Both are pinned by tests using output captured verbatim from that host. Verified against real hardware after the fix: architecture Arm64, CUDA 13 detected, one GPU at 25,702,694,912 bytes VRAM, backend b10472-cuda13-arm64 with both the llama.cpp and cudart archives. Installing that runtime for real downloaded and SHA-256 verified both pinned archives in 15 seconds, extracted 43 files, and llama-server.exe --version reported "version: 0.1.1-dev (build 10472, commit 60eeeb608)", confirming the pinned tag and that the CUDA dependencies resolve. Known limitation, not addressed here: the recommender describes a model that exceeds VRAM as running "from system RAM" and being slow. On a unified-memory host like this one that framing is misleading, because the GPU carve-out and system RAM come from the same physical pool. There is no reliable way to detect unified memory from the probe's current sources, so the wording is left accurate for discrete GPUs and flagged rather than guessed at. Validation: ./build.ps1 (all projects, 49 docs), OpenClaw.Shared.Tests 3854 passed across six consecutive full runs plus ten runs of the process-spawning subset, OpenClaw.Tray.Tests 2469 passed. Co-Authored-By: Claude Opus 5 (1M context) --- .../Inference/NvidiaSmiParser.cs | 47 +++++++++++++++++-- .../Inference/NvidiaSmiParserTests.cs | 46 ++++++++++++++++++ 2 files changed, 88 insertions(+), 5 deletions(-) diff --git a/src/OpenClaw.Shared/Inference/NvidiaSmiParser.cs b/src/OpenClaw.Shared/Inference/NvidiaSmiParser.cs index c2141384c..df07441e4 100644 --- a/src/OpenClaw.Shared/Inference/NvidiaSmiParser.cs +++ b/src/OpenClaw.Shared/Inference/NvidiaSmiParser.cs @@ -52,6 +52,11 @@ public static IReadOnlyList ParseQueryGpu(string? stdout, int? cudaMajo var name = fields[0].Trim(); if (name.Length == 0) continue; + // A GB10 host lists an "NVIDIA NPU" alongside the GPU. It shares the + // driver but is not a CUDA device, so counting it would put a phantom + // adapter in the UI and imply an accelerator llama.cpp cannot use. + if (IsNonCudaAccelerator(name)) continue; + var memoryBytes = fields.Length > 1 ? ParseMebibytes(fields[1]) : null; var driver = fields.Length > 2 ? NullIfBlank(fields[2]) : null; @@ -66,21 +71,42 @@ public static IReadOnlyList ParseQueryGpu(string? stdout, int? cudaMajo return results; } + /// + /// Banner labels that precede the CUDA version, most specific first. + /// + /// + /// Older drivers print CUDA Version: 12.8. Newer ones (seen on a GB10 + /// host with driver 616.29) print CUDA UMD Version: 13.4 instead, which + /// does not contain the older marker as a substring. Missing the newer form + /// makes the probe report "unknown" on current hardware and silently degrade + /// to the older CUDA build, so both spellings are matched. + /// + private static readonly string[] CudaVersionMarkers = ["CUDA UMD Version:", "CUDA Version:"]; + /// /// Extract the CUDA major version from plain nvidia-smi output, whose /// header line reads e.g. - /// | NVIDIA-SMI 570.86.10 Driver Version: 570.86.10 CUDA Version: 12.8 |. - /// Returns null when the marker is missing or unparseable. + /// | NVIDIA-SMI 570.86.10 Driver Version: 570.86.10 CUDA Version: 12.8 | + /// or | NVIDIA-SMI 616.29 KMD Version: 616.29 CUDA UMD Version: 13.4 |. + /// Returns null when no marker is present or the value is unparseable. /// public static int? TryParseCudaMajorVersion(string? stdout) { if (string.IsNullOrWhiteSpace(stdout)) return null; - const string marker = "CUDA Version:"; - var index = stdout.IndexOf(marker, StringComparison.OrdinalIgnoreCase); + var index = -1; + var markerLength = 0; + foreach (var marker in CudaVersionMarkers) + { + index = stdout.IndexOf(marker, StringComparison.OrdinalIgnoreCase); + if (index < 0) continue; + markerLength = marker.Length; + break; + } + if (index < 0) return null; - var rest = stdout.AsSpan(index + marker.Length).TrimStart(); + var rest = stdout.AsSpan(index + markerLength).TrimStart(); // Take the leading digit run; "12.8" and "13" both yield the major part. var end = 0; @@ -116,6 +142,17 @@ static bool Contains(string haystack, string needle) => haystack.Contains(needle, StringComparison.OrdinalIgnoreCase); } + /// + /// True for devices the NVIDIA driver enumerates that are not CUDA compute + /// devices. Matched as a whole word so an adapter whose name merely contains + /// these letters is not dropped. + /// + internal static bool IsNonCudaAccelerator(string name) => + System.Text.RegularExpressions.Regex.IsMatch( + name, + @"\bNPU\b", + System.Text.RegularExpressions.RegexOptions.IgnoreCase); + private static long? ParseMebibytes(string field) { var text = field.Trim(); diff --git a/tests/OpenClaw.Shared.Tests/Inference/NvidiaSmiParserTests.cs b/tests/OpenClaw.Shared.Tests/Inference/NvidiaSmiParserTests.cs index dd074028a..4864b8fe0 100644 --- a/tests/OpenClaw.Shared.Tests/Inference/NvidiaSmiParserTests.cs +++ b/tests/OpenClaw.Shared.Tests/Inference/NvidiaSmiParserTests.cs @@ -81,6 +81,52 @@ public void TryParseCudaMajorVersion_HandlesAMajorOnlyVersion() Assert.Equal(13, NvidiaSmiParser.TryParseCudaMajorVersion("CUDA Version: 13")); } + [Fact] + public void TryParseCudaMajorVersion_ReadsTheNewerUmdBannerLabel() + { + // Captured from a GB10 host on driver 616.29. This label does not contain + // the older "CUDA Version:" marker as a substring, so missing it made the + // probe report "unknown" on current hardware and silently degrade to the + // older CUDA build. + const string banner = + """ + Tue Aug 18 21:06:44 2026 + +-----------------------------------------------------------------------------------------+ + | NVIDIA-SMI 616.29 KMD Version: 616.29 CUDA UMD Version: 13.4 | + """; + + Assert.Equal(13, NvidiaSmiParser.TryParseCudaMajorVersion(banner)); + } + + [Fact] + public void ParseQueryGpu_SkipsTheNpuOnAGb10Host() + { + // The NVIDIA NPU shares the driver and appears in --query-gpu output, but + // it is not a CUDA device. Counting it would put a phantom adapter in the + // UI and imply an accelerator llama.cpp cannot use. + const string stdout = + """ + NVIDIA RTX Spark N1X (5120-core Blackwell RTX GPU), 24512, 616.29 + NVIDIA NPU, [N/A], 616.29 + """; + + var gpus = NvidiaSmiParser.ParseQueryGpu(stdout, cudaMajorVersion: 13); + + var gpu = Assert.Single(gpus); + Assert.StartsWith("NVIDIA RTX Spark N1X", gpu.Name, System.StringComparison.Ordinal); + Assert.Equal(24512L * 1024 * 1024, gpu.DedicatedMemoryBytes); + } + + [Theory] + [InlineData("NVIDIA NPU", true)] + [InlineData("nvidia npu", true)] + [InlineData("NVIDIA RTX Spark N1X (5120-core Blackwell RTX GPU)", false)] + [InlineData("NVIDIA GeForce RTX 4090", false)] + public void IsNonCudaAccelerator_MatchesWholeWordsOnly(string name, bool expected) + { + Assert.Equal(expected, NvidiaSmiParser.IsNonCudaAccelerator(name)); + } + [Theory] [InlineData(null)] [InlineData("")] From af41a25258038bcdda4581849bc5ba5e034aea67 Mon Sep 17 00:00:00 2001 From: Pedro Larroy Date: Wed, 19 Aug 2026 09:30:14 -0700 Subject: [PATCH 6/7] fix(inference): survive dropped connections and stop the server cleanly A real end-to-end run on the GB10 host found two defects that no unit test had reached. 1. A dropped connection destroyed the download. The 21 GB model transfer died at 50% on a socket read, and the catch-all cleanup deleted the .part file, discarding 10.6 GB of perfectly good bytes. That is precisely the case resume was built for. Cleanup is now scoped to verification failures, where the bytes really are known bad, and a transport failure keeps the partial file and retries with a resumed range request (bounded, with backoff). Cancellation also keeps the file so the user can resume later. The retry delay is injectable so tests exercise the path without paying wall-clock time. 2. A deliberate stop was reported as a crash. Killing the child raised Exited while the status was still Ready, so every normal Stop emitted "the server stopped unexpectedly" before Stopped, flashing a failure in the UI. The exit handler is now detached before the kill, guarded by an explicit stop-requested flag, and the decision is a pure testable policy. Tests reproduce the real failure: a fake transport that delivers part of a body and then throws, asserting the transfer resumes, that the partial file survives an exhausted retry budget, and that retries stay bounded. End-to-end proof on this host, now recorded in the plan doc: hardware probe, backend selection to b10472-cuda13-arm64, runtime install with both archives SHA-256 verified, 22,663,387,424 bytes of Qwen3.6-35B-A3B downloaded and verified in 11.9 min, server ready in 78 s, HTTP 200 from /v1/chat/completions answering "What is 2+2?" with "4", speculative decoding active at 0.88 draft acceptance, 40.4 tokens/second, and a clean Starting -> Ready -> Stopped with no spurious failure. A second run skipped both downloads. Also corrects a stale deferral: the arm64 CUDA path is now the proven one, so x64 is what remains unverified. Validation: ./build.ps1 (all projects, 49 docs), OpenClaw.Shared.Tests 3862 passed across three consecutive runs, OpenClaw.Tray.Tests 2469 passed. Co-Authored-By: Claude Opus 5 (1M context) --- docs/LOCAL_INFERENCE_PLAN.md | 118 ++++++++++++++++-- .../Inference/LlamaServerProcess.cs | 38 ++++-- .../Inference/VerifiedFileDownloader.cs | 92 +++++++++++++- .../Inference/FakeHttpTransport.cs | 69 +++++++++- .../Inference/LlamaServerProcessTests.cs | 15 +++ .../Inference/VerifiedFileDownloaderTests.cs | 68 +++++++++- 6 files changed, 371 insertions(+), 29 deletions(-) diff --git a/docs/LOCAL_INFERENCE_PLAN.md b/docs/LOCAL_INFERENCE_PLAN.md index 42faa7297..d802e0f55 100644 --- a/docs/LOCAL_INFERENCE_PLAN.md +++ b/docs/LOCAL_INFERENCE_PLAN.md @@ -8,7 +8,7 @@ inline; update it as phases land. | 1 | Hardware probe, backend selection, model recommender | Landed | | 2 | Runtime and GGUF download managers | Landed | | 3 | Server process and settings UI | Landed | -| 4 | Gateway provider registration | Not started (blocked on live schema) | +| 4 | Gateway provider registration | Not started | | 5 | Optional `localinference.status` node capability | Not started | ## Context @@ -232,7 +232,7 @@ remains the crash backstop, not a substitute for an orderly stop. Strings are seeded English-only across all five locales using the repo's deferred-translation pattern and registered in `LocalizationValidationTests`. -## Phase 4: gateway registration (blocked on live schema) +## Phase 4: gateway registration Once the server is healthy, patch the gateway config to add an OpenAI-compatible provider pointing at it, via @@ -246,12 +246,78 @@ route the user to the Config page. Silently clobbering real API keys with redaction placeholders while enabling local inference would be a severe regression. -**Blocker.** This repo does not contain the gateway's config schema; -`ConfigPage` fetches it at runtime from `config.get`. The dot-path and shape for -registering an OpenAI-compatible provider is therefore not verifiable from this -checkout. First step of this phase: connect to a real gateway, inspect the -schema the Config page renders, and write the builder against the real shape. Do -not guess it. +**Schema, confirmed against the gateway source (`../openclaw`).** This repo +does not vendor the gateway's config schema, but it lives in the gateway +checkout and is no longer a guess: + +- `src/config/zod-schema.core.ts` defines `ModelProviderSchema` (the shape of + one entry under `models.providers.`) and `ModelProvidersSchema` + (the `Record` map), plus a `superRefine` + that requires `baseUrl` and a non-empty `models[]` array for any provider id + outside `BUILT_IN_MODEL_PROVIDER_OVERLAY_IDS` (`openai`, `ollama`, + `lmstudio`, `vllm`, etc.). Our registered id will not be a built-in, so both + fields are mandatory. +- `docs/gateway/local-models.md` and `docs/gateway/local-model-services.md` in + the gateway repo document this exact scenario (a local OpenAI-compatible + server such as llama-server) with worked examples. + +Patch target is `models.providers.`, merged (`mode: "merge"`) via +`config.patch`: + +```json5 +{ + "models": { + "mode": "merge", + "providers": { + "llama-local": { + "baseUrl": "http://127.0.0.1:8080/v1", + "apiKey": "sk-local", + "api": "openai-completions", + "timeoutSeconds": 300, + "models": [ + { + "id": "my-local-model", + "name": "My Local Model", + "reasoning": false, + "input": ["text"], + "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }, + "contextWindow": 32768, + "maxTokens": 4096 + } + ] + } + } + } +} +``` + +Notes for the builder: + +- `api: "openai-completions"` is the right value for a plain + OpenAI-compatible `/v1/chat/completions` server like llama-server; it is + also the default when `api` is omitted on a custom provider with a + `baseUrl`. +- `apiKey` accepts any non-empty string for a loopback/LAN `baseUrl`; there is + no real secret to protect for a local server, but the field is still + registered as sensitive (`SecretInputSchema.optional().register(sensitive)`), + so it round-trips through `config.get` as the sentinel + `__OPENCLAW_REDACTED__`. This confirms the redaction-sentinel safety rail + already planned above: resending the config unchanged is safe because + `restoreRedactedValues(...)` on the gateway swaps the sentinel back before + merge/validate, but our own outgoing patch must not write a sentinel into a + field it did not read one from. +- `models[].id` is provider-local; a model is addressed elsewhere in config + (e.g. `agents.defaults.model.primary`) as `/`. The + provider id and the model id here should be generated deterministically from + our runtime/model identity so re-registration is idempotent. +- `config.patch` requires a `baseHash` from a prior `config.get` (compare-and- + swap), matching `PatchConfigDetailedAsync(fullConfig, baseHash)`. +- Optional and not required for the first cut: `localService` (`command`, + `args`, `healthUrl`, `readyTimeoutMs`, `idleStopMs`) lets the gateway itself + spawn and health-check a local server. We already own process lifecycle via + `LlamaServerProcess`, so registration should populate `baseUrl` + + `models[]` only and leave `localService` unset, to avoid a double process + owner. **WSL reachability.** When the gateway runs in WSL, `127.0.0.1` inside the distro is not the Windows host. `WINDOWS_NODE_ARCHITECTURE.md` records this: @@ -320,6 +386,36 @@ Real behavior proof, which CI cannot establish: picker and that a turn reaches the local server. 7. Cover both mirrored and NAT WSL networking, or record which was not covered. -Not verifiable in the current environment and explicitly deferred: the gateway -provider config schema, the Vulkan and arm64 CUDA paths, and the DeepSeek path. -State these as blockers in the PR rather than implying coverage. +### Real-behavior proof captured 2026-08-19 + +Run on the development host (GB10 DGX Spark: ARM64 Windows, RTX Spark N1X, +24512 MiB VRAM, driver 616.29): + +| Step | Result | +| --- | --- | +| Hardware probe | `Arm64`, CUDA 13, 25,702,694,912 bytes VRAM, one GPU (NPU correctly excluded) | +| Backend selected | `b10472-cuda13-arm64`, both llama.cpp and cudart archives | +| Runtime install | 293 MB downloaded, both SHA-256 verified, 43 files extracted, 15 s | +| `llama-server --version` | `version: 0.1.1-dev (build 10472, commit 60eeeb608)` | +| Model download | Qwen3.6-35B-A3B UD-Q4_K_M, 22,663,387,424 bytes, SHA-256 verified, 11.9 min at 35 MB/s | +| Server start | Ready in 78 s | +| Completion | `POST /v1/chat/completions` returned HTTP 200 in 4.0 s; "What is 2+2?" answered `4` | +| Speculative decoding | `--spec-type draft-mtp` active: draft acceptance 0.88 (45/51), mean length 3.65 | +| Throughput | 40.4 tokens/second eval | +| Stop | `Starting -> Ready -> Stopped`, no spurious failure | +| Idempotence | A second run skipped both downloads and started in 78 s | + +Two defects were found only by this run and are fixed: the downloader deleted +the partial file on a dropped connection (losing 10.6 GB of good bytes), and a +deliberate stop was reported as an unexpected crash. + +Still not covered: the Vulkan and x64 CUDA paths (need that hardware), the +DeepSeek path (needs ~160 GB), and gateway registration (Phase 4). + +Not verifiable in the current environment and explicitly deferred: exercising +the registration patch against a real running gateway (the schema itself is +now confirmed from the gateway source, see Phase 4, but end-to-end proof +still needs a live instance), the Vulkan and x64 CUDA paths, and the +DeepSeek path. The arm64 CUDA path is no longer deferred: it is the one proven +end to end above. State the rest as blockers in the PR rather than implying +coverage. diff --git a/src/OpenClaw.Shared/Inference/LlamaServerProcess.cs b/src/OpenClaw.Shared/Inference/LlamaServerProcess.cs index 160251090..5a927ceff 100644 --- a/src/OpenClaw.Shared/Inference/LlamaServerProcess.cs +++ b/src/OpenClaw.Shared/Inference/LlamaServerProcess.cs @@ -78,6 +78,9 @@ public sealed class LlamaServerProcess : IAsyncDisposable private ConcurrentQueue _stderrTail = new(); private LlamaServerStatus _status = LlamaServerStatus.Stopped; + /// Set while a deliberate stop is in progress, so the resulting child exit is not reported as a crash. + private volatile bool _stopRequested; + public LlamaServerProcess(IOpenClawLogger logger, Func? httpClientFactory = null) { _logger = logger ?? throw new ArgumentNullException(nameof(logger)); @@ -143,6 +146,7 @@ public async Task StartAsync( "The endpoint is unauthenticated and reachable from the network."); } + _stopRequested = false; SetStatus(new LlamaServerStatus(LlamaServerState.Starting, resolvedPort)); if (!TryLaunch(runtime, argv, out var launchError)) @@ -305,6 +309,13 @@ private async Task StopCoreAsync() if (process is not null) { + // Detach the exit handler BEFORE killing. Otherwise a deliberate stop + // raises Exited while the status is still Ready and gets reported as + // "the server stopped unexpectedly", so the UI flashes a failure on + // every normal Stop. Observed on a real run before this was fixed. + process.Exited -= OnProcessExited; + _stopRequested = true; + try { if (!process.HasExited) @@ -323,7 +334,6 @@ await process.WaitForExitAsync(new CancellationTokenSource(GracefulStopTimeout). { process.ErrorDataReceived -= OnStderr; process.OutputDataReceived -= OnStdout; - process.Exited -= OnProcessExited; process.Dispose(); } } @@ -351,17 +361,25 @@ private void OnStdout(object sender, DataReceivedEventArgs e) private void OnProcessExited(object? sender, EventArgs e) { - // An exit while we believed the server was serving is a crash, not a - // stop; surface it so the UI does not keep claiming Ready. - if (_status.State is LlamaServerState.Ready) - { - SetStatus(new LlamaServerStatus( - LlamaServerState.Failed, - _status.Port, - $"The server stopped unexpectedly. {DescribeStderrTail()}")); - } + if (!ShouldReportUnexpectedExit(_status.State, _stopRequested)) return; + + SetStatus(new LlamaServerStatus( + LlamaServerState.Failed, + _status.Port, + $"The server stopped unexpectedly. {DescribeStderrTail()}")); } + /// + /// Whether a child exit should be surfaced as a crash. + /// + /// + /// Only an exit we did not ask for, while we believed the server was serving, + /// is a crash. An exit during a deliberate stop is the expected outcome, and + /// reporting it as a failure makes every normal Stop look like an error. + /// + internal static bool ShouldReportUnexpectedExit(LlamaServerState state, bool stopRequested) => + !stopRequested && state is LlamaServerState.Ready; + private string DescribeStderrTail() { var lines = _stderrTail.ToArray(); diff --git a/src/OpenClaw.Shared/Inference/VerifiedFileDownloader.cs b/src/OpenClaw.Shared/Inference/VerifiedFileDownloader.cs index 8f0ed5061..a36b6cd69 100644 --- a/src/OpenClaw.Shared/Inference/VerifiedFileDownloader.cs +++ b/src/OpenClaw.Shared/Inference/VerifiedFileDownloader.cs @@ -59,20 +59,38 @@ public sealed class VerifiedFileDownloader { private const int BufferSize = 81920; + /// + /// How many times a transfer is attempted before giving up. Each retry + /// resumes from the bytes already on disk when the request allows it. + /// + private const int MaxTransportAttempts = 5; + private readonly IOpenClawLogger _logger; private readonly Func _httpClientFactory; + private readonly Func _retryDelay; /// Diagnostics sink. /// /// Optional override so tests can inject a fake handler. Each call gets a /// client that the downloader disposes. /// - public VerifiedFileDownloader(IOpenClawLogger logger, Func? httpClientFactory = null) + /// + /// Backoff before retry attempt N. Injectable so tests exercise the retry + /// path without paying real wall-clock delays. + /// + public VerifiedFileDownloader( + IOpenClawLogger logger, + Func? httpClientFactory = null, + Func? retryDelay = null) { _logger = logger ?? throw new ArgumentNullException(nameof(logger)); _httpClientFactory = httpClientFactory ?? CreateDefaultClient; + _retryDelay = retryDelay ?? DefaultRetryDelay; } + private static TimeSpan DefaultRetryDelay(int attempt) => + TimeSpan.FromSeconds(Math.Min(30, 2 * attempt)); + private static HttpClient CreateDefaultClient() => // Large weights over a slow link legitimately take hours. The timeout // that matters is enforced by the caller's cancellation token, not here. @@ -116,10 +134,10 @@ public async Task DownloadAsync( var partPath = request.DestinationPath + ".part"; + await FetchWithRetryAsync(request, partPath, bytesCompleted, cancellationToken).ConfigureAwait(false); + try { - await FetchToPartFileAsync(request, partPath, bytesCompleted, cancellationToken).ConfigureAwait(false); - if (request.ExpectedSizeBytes > 0) { var actualSize = new FileInfo(partPath).Length; @@ -138,14 +156,76 @@ public async Task DownloadAsync( } catch { - // Any failure discards the partial file. Keeping a mismatched or - // truncated .part would make the next resume attempt repeat the same - // failure forever. + // A verification failure means the bytes on disk are known bad, so the + // partial file is discarded. Keeping it would make every later resume + // attempt repeat the same failure forever. + // + // Note this is deliberately narrower than "any failure": a dropped + // connection is handled in FetchWithRetryAsync, which keeps the + // partial file precisely so the transfer can resume. TryDelete(partPath); throw; } } + /// + /// Run the transfer, retrying a dropped connection with a resumed range + /// request. + /// + /// + /// A multi-gigabyte transfer over tens of minutes will meet a transient + /// network failure sooner or later; a real 21 GB run died at 50% on a socket + /// read. Failing the whole download there, or worse deleting the partial + /// file, throws away everything already fetched. Transport errors therefore + /// keep the partial file and retry from where it stopped, while cancellation + /// keeps the file but stops immediately so the user can resume later. + /// + private async Task FetchWithRetryAsync( + VerifiedDownloadRequest request, + string partPath, + IProgress? bytesCompleted, + CancellationToken cancellationToken) + { + for (var attempt = 1; ; attempt++) + { + try + { + await FetchToPartFileAsync(request, partPath, bytesCompleted, cancellationToken).ConfigureAwait(false); + return; + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + // The user asked to stop. Leave the partial file so a later call + // can resume instead of starting over. + throw; + } + catch (Exception ex) when (IsTransientTransportFailure(ex) && attempt < MaxTransportAttempts) + { + var delay = _retryDelay(attempt); + _logger.Warn( + $"[Inference] Transfer of '{request.Label}' failed on attempt {attempt} " + + $"({ex.GetType().Name}); retrying in {delay.TotalSeconds:F0}s"); + await Task.Delay(delay, cancellationToken).ConfigureAwait(false); + } + } + } + + /// + /// True for failures that are worth retrying with a resumed range request: + /// the connection dropped or the server hiccuped, rather than the content + /// being wrong. + /// + private static bool IsTransientTransportFailure(Exception ex) => ex switch + { + HttpRequestException => true, + System.Net.Sockets.SocketException => true, + // Surfaces as an inner exception of an IOException on a mid-body drop, + // and directly when HttpClient times out an idle read. + IOException => true, + TaskCanceledException => true, + _ => ex.InnerException is not null && IsTransientTransportFailure(ex.InnerException), + }; + private async Task FetchToPartFileAsync( VerifiedDownloadRequest request, string partPath, diff --git a/tests/OpenClaw.Shared.Tests/Inference/FakeHttpTransport.cs b/tests/OpenClaw.Shared.Tests/Inference/FakeHttpTransport.cs index bd7866d5e..6dcc828cb 100644 --- a/tests/OpenClaw.Shared.Tests/Inference/FakeHttpTransport.cs +++ b/tests/OpenClaw.Shared.Tests/Inference/FakeHttpTransport.cs @@ -1,5 +1,6 @@ using System; using System.Collections.Generic; +using System.IO; using System.Net; using System.Net.Http; using System.Security.Cryptography; @@ -74,9 +75,75 @@ protected override Task SendAsync( if (entry.TruncateAfterBytes is { } limit && body.Length > limit) body = body[..limit]; + // Simulate a connection that drops partway through the body, which is how + // a long real transfer actually fails. + if (entry.DropAfterBytes is { } dropAfter && _dropsRemaining > 0) + { + _dropsRemaining--; + var delivered = Math.Min(dropAfter, body.Length); + return Task.FromResult(new HttpResponseMessage(status) + { + Content = new StreamContent(new DroppingStream(body[..delivered])), + }); + } + return Task.FromResult(new HttpResponseMessage(status) { Content = new ByteArrayContent(body) }); } + /// + /// Make the next responses for + /// deliver bytes and then fail the stream, the + /// way a dropped TCP connection does mid-transfer. + /// + public void DropConnectionAfter(string url, int afterBytes, int times = 1) + { + _entries[url] = _entries[url] with { DropAfterBytes = afterBytes }; + _dropsRemaining = times; + } + + private int _dropsRemaining; + + /// Yields its buffer, then throws as a broken connection would. + private sealed class DroppingStream(byte[] payload) : Stream + { + private int _position; + + public override int Read(byte[] buffer, int offset, int count) + { + if (_position >= payload.Length) + throw new IOException("The connection was closed unexpectedly."); + + var n = Math.Min(count, payload.Length - _position); + Array.Copy(payload, _position, buffer, offset, n); + _position += n; + return n; + } + + public override int Read(Span buffer) + { + if (_position >= payload.Length) + throw new IOException("The connection was closed unexpectedly."); + + var n = Math.Min(buffer.Length, payload.Length - _position); + payload.AsSpan(_position, n).CopyTo(buffer); + _position += n; + return n; + } + + public override ValueTask ReadAsync(Memory buffer, CancellationToken cancellationToken = default) => + ValueTask.FromResult(Read(buffer.Span)); + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + public override long Length => payload.Length; + public override long Position { get => _position; set => throw new NotSupportedException(); } + public override void Flush() { } + public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException(); + public override void SetLength(long value) => throw new NotSupportedException(); + public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException(); + } + /// Factory to hand to the downloader under test. public Func ClientFactory => () => new HttpClient(this, disposeHandler: false); @@ -91,5 +158,5 @@ public static (byte[] Body, string Sha256) MakeBody(int length, int seed) public static string Sha256Of(byte[] body) => Convert.ToHexString(SHA256.HashData(body)).ToLowerInvariant(); - private sealed record Entry(byte[] Body, bool SupportsRange, int? TruncateAfterBytes); + private sealed record Entry(byte[] Body, bool SupportsRange, int? TruncateAfterBytes, int? DropAfterBytes = null); } diff --git a/tests/OpenClaw.Shared.Tests/Inference/LlamaServerProcessTests.cs b/tests/OpenClaw.Shared.Tests/Inference/LlamaServerProcessTests.cs index 587d8ffc2..f0e8e5535 100644 --- a/tests/OpenClaw.Shared.Tests/Inference/LlamaServerProcessTests.cs +++ b/tests/OpenClaw.Shared.Tests/Inference/LlamaServerProcessTests.cs @@ -129,6 +129,21 @@ public void LoopbackBaseUrlIsOnlyExposedWhenReady() new LlamaServerStatus(LlamaServerState.Ready, 8080).LoopbackBaseUrl); } + [Theory] + // A crash while serving is the only case worth reporting as a failure. + [InlineData(LlamaServerState.Ready, false, true)] + // A deliberate stop kills the child; reporting that as "stopped unexpectedly" + // made the UI flash a failure on every normal Stop. Observed on a real run. + [InlineData(LlamaServerState.Ready, true, false)] + [InlineData(LlamaServerState.Starting, false, false)] + [InlineData(LlamaServerState.Stopped, false, false)] + [InlineData(LlamaServerState.Failed, false, false)] + public void ShouldReportUnexpectedExit_OnlyForAnUnrequestedExitWhileServing( + LlamaServerState state, bool stopRequested, bool expected) + { + Assert.Equal(expected, LlamaServerProcess.ShouldReportUnexpectedExit(state, stopRequested)); + } + [Fact] public void JobObjectIsCreatableOnThisHost() { diff --git a/tests/OpenClaw.Shared.Tests/Inference/VerifiedFileDownloaderTests.cs b/tests/OpenClaw.Shared.Tests/Inference/VerifiedFileDownloaderTests.cs index 35c4a01e7..29269fb05 100644 --- a/tests/OpenClaw.Shared.Tests/Inference/VerifiedFileDownloaderTests.cs +++ b/tests/OpenClaw.Shared.Tests/Inference/VerifiedFileDownloaderTests.cs @@ -222,6 +222,68 @@ public async Task ATruncatedResponseFailsAndTheRetrySucceeds() Assert.Equal(body, await File.ReadAllBytesAsync(destination)); } + [Fact] + public async Task ResumesAfterTheConnectionDropsMidTransfer() + { + // Reproduces a real failure: a 21 GB download died at 50% on a socket + // read. The transfer must continue from the bytes already on disk rather + // than failing or starting over. + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(60_000, seed: 20); + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + transport.DropConnectionAfter(Url, afterBytes: 25_000); + + var destination = temp.Combine("asset.bin"); + await NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, hash, body.Length, AllowResume: true)); + + Assert.Equal(body, await File.ReadAllBytesAsync(destination)); + Assert.Equal(2, transport.Requests.Count); + // The retry resumed rather than restarting from zero. + Assert.Null(transport.RangeHeaders[0]); + Assert.NotNull(transport.RangeHeaders[1]); + } + + [Fact] + public async Task KeepsThePartialFileWhenEveryAttemptIsExhausted() + { + // Deleting it would discard gigabytes of good bytes for a network fault + // that says nothing about their validity. + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(60_000, seed: 21); + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + transport.DropConnectionAfter(Url, afterBytes: 10_000, times: 99); + + var destination = temp.Combine("asset.bin"); + + await Assert.ThrowsAnyAsync(() => + NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, destination, hash, body.Length, AllowResume: true))); + + Assert.False(File.Exists(destination)); + Assert.True(File.Exists(destination + ".part"), "The partial file must survive a transport failure."); + Assert.True(new FileInfo(destination + ".part").Length > 0); + } + + [Fact] + public async Task StopsRetryingAfterABoundedNumberOfAttempts() + { + using var temp = new TempDirectory(); + var (body, hash) = FakeHttpTransport.MakeBody(60_000, seed: 22); + var transport = new FakeHttpTransport(); + transport.Add(Url, body); + transport.DropConnectionAfter(Url, afterBytes: 10_000, times: 99); + + await Assert.ThrowsAnyAsync(() => + NewDownloader(transport).DownloadAsync( + new VerifiedDownloadRequest(Url, temp.Combine("asset.bin"), hash, body.Length, AllowResume: true))); + + // Bounded, so a permanently broken source fails instead of looping. + Assert.InRange(transport.Requests.Count, 2, 10); + } + [Fact] public async Task ReportsMonotonicProgressEndingAtTheFullSize() { @@ -258,8 +320,12 @@ public async Task AnHttpErrorLeavesNoPartialFile() Assert.False(File.Exists(destination + ".part")); } + /// + /// Retry backoff is zeroed so the retry paths cost no wall-clock time; the + /// production default is a real escalating delay. + /// private static VerifiedFileDownloader NewDownloader(FakeHttpTransport transport) => - new(NullLogger.Instance, transport.ClientFactory); + new(NullLogger.Instance, transport.ClientFactory, retryDelay: _ => TimeSpan.Zero); /// /// posts to the synchronization context, so reports From 7a4fa51bc2b68c8713657e24d39a2c46b11e0348 Mon Sep 17 00:00:00 2001 From: Pedro Larroy Date: Wed, 19 Aug 2026 09:33:04 -0700 Subject: [PATCH 7/7] fix(connection): trust MSIX-packaged wslrelay.exe when Authenticode check fails Modern WSL distributes as an MSIX package (Program Files\WSL / WindowsApps), which doesn't embed per-file Authenticode/catalog signatures on its EXEs -- trust is established at the package level instead. This caused genuine Microsoft wslrelay.exe binaries to be rejected during setup's loopback listener provenance check with "WSL relay Authenticode verification failed (Unsigned)". Add a fallback in WindowsAuthenticodeVerifier: when the classic Authenticode check fails and the file is wslrelay.exe, corroborate trust via the installed AppX package instead. Requires an exact match on the known WSL package family name (MicrosoftCorporationII.WindowsSubsystemForLinux_8wekyb3d8bbwe, whose suffix is derived from the publisher's signing cert), a real SignatureKind, and a Microsoft publisher (reusing the existing HasMicrosoftPublisherIdentity check). Looked up via a PowerShell Get-AppxPackage shell-out, matching this file's existing pattern for invoking schtasks.exe/wsl.exe. The primary Authenticode/catalog check is unchanged and always consulted first; the fallback narrows to wslrelay.exe specifically and never weakens the trust bar for any other binary. --- .../WindowsAuthenticodeVerifier.cs | 198 +++++++++++++++++- ...dLocalGatewayPortProvenanceServiceTests.cs | 88 ++++++++ 2 files changed, 285 insertions(+), 1 deletion(-) diff --git a/src/OpenClaw.Connection/WindowsAuthenticodeVerifier.cs b/src/OpenClaw.Connection/WindowsAuthenticodeVerifier.cs index f2e190d3d..d251a249e 100644 --- a/src/OpenClaw.Connection/WindowsAuthenticodeVerifier.cs +++ b/src/OpenClaw.Connection/WindowsAuthenticodeVerifier.cs @@ -1,5 +1,9 @@ using System; +using System.Diagnostics; using System.IO; +using System.Linq; +using System.Text.Json; +using System.Threading; using Microsoft.Security.Extensions; namespace OpenClaw.Connection; @@ -11,9 +15,51 @@ internal readonly record struct AuthenticodeTrustResult(bool IsTrusted, string? public static AuthenticodeTrustResult Rejected(string detail) => new(false, detail); } +/// +/// Minimal projection of a Windows AppX/MSIX package as reported by +/// Get-AppxPackage, used to corroborate genuine Microsoft WSL binaries that +/// ship without a per-file Authenticode/catalog signature (MSIX packages are +/// trusted at the package level, not per-file). +/// +internal readonly record struct AppxPackageInfo( + string? PackageFamilyName, + string? Publisher, + string? SignatureKind); + internal static class WindowsAuthenticodeVerifier { - public static AuthenticodeTrustResult VerifyMicrosoftSignedFile(string path) + // Modern WSL ships as an MSIX package. The trailing suffix is a hash derived + // from the publisher's signing certificate/public key, so an impostor package + // re-signed with a different certificate would get a different family name. + private const string WslPackageFamilyName = + "MicrosoftCorporationII.WindowsSubsystemForLinux_8wekyb3d8bbwe"; + + public static AuthenticodeTrustResult VerifyMicrosoftSignedFile(string path) => + VerifyMicrosoftSignedFile(path, LookupWslAppxPackageViaPowerShell); + + /// + /// Test seam: allows callers to substitute a fake WSL AppX package lookup + /// instead of shelling out to PowerShell. + /// + internal static AuthenticodeTrustResult VerifyMicrosoftSignedFile( + string path, + Func lookupWslPackage) + { + var primary = VerifyAuthenticodeSignature(path); + if (primary.IsTrusted) + return primary; + + // MSIX packages (modern WSL) don't carry a per-file Authenticode/catalog + // signature, so a failed classic check isn't conclusive for wslrelay.exe. + // Only consult the package-level fallback for the WSL binaries this + // codebase actually cares about; never widen trust for arbitrary files. + if (!string.Equals(Path.GetFileName(path), "wslrelay.exe", StringComparison.OrdinalIgnoreCase)) + return primary; + + return VerifyWslPackageFallback(primary, lookupWslPackage); + } + + private static AuthenticodeTrustResult VerifyAuthenticodeSignature(string path) { try { @@ -45,6 +91,50 @@ public static AuthenticodeTrustResult VerifyMicrosoftSignedFile(string path) } } + private static AuthenticodeTrustResult VerifyWslPackageFallback( + AuthenticodeTrustResult primaryFailure, + Func lookupWslPackage) + { + AppxPackageInfo? package; + try + { + package = lookupWslPackage(); + } + catch + { + package = null; + } + + if (package is not { } info) + { + // No genuine WSL AppX package installed to corroborate; surface the + // original Authenticode diagnostic rather than a new, less useful one. + return primaryFailure; + } + + if (!string.Equals(info.PackageFamilyName, WslPackageFamilyName, StringComparison.Ordinal)) + { + // Doesn't match the well-known family name (a different/impostor + // package can't corroborate this binary); surface the original detail. + return primaryFailure; + } + + if (string.IsNullOrEmpty(info.SignatureKind) || + string.Equals(info.SignatureKind, "None", StringComparison.OrdinalIgnoreCase)) + { + return AuthenticodeTrustResult.Rejected( + "WSL package signature verification failed: the installed WSL package is unsigned."); + } + + if (info.Publisher is null || !HasMicrosoftPublisherIdentity(info.Publisher)) + { + return AuthenticodeTrustResult.Rejected( + "WSL package signature verification failed: the installed WSL package's publisher is not Microsoft Corporation."); + } + + return AuthenticodeTrustResult.Trusted(); + } + internal static bool HasMicrosoftPublisherIdentity(string subject) => subject.Split(',') .Select(part => part.Trim()) @@ -53,4 +143,110 @@ internal static bool HasMicrosoftPublisherIdentity(string subject) => part, "O=Microsoft Corporation", StringComparison.OrdinalIgnoreCase)); + + private static AppxPackageInfo? LookupWslAppxPackageViaPowerShell() + { + Process? process = null; + try + { + var psi = new ProcessStartInfo + { + FileName = ResolvePowerShellPath(), + UseShellExecute = false, + CreateNoWindow = true, + RedirectStandardOutput = true, + RedirectStandardError = true, + }; + psi.ArgumentList.Add("-NoProfile"); + psi.ArgumentList.Add("-NonInteractive"); + psi.ArgumentList.Add("-Command"); + psi.ArgumentList.Add( + "Get-AppxPackage -Name 'MicrosoftCorporationII.WindowsSubsystemForLinux' | " + + "Select-Object -First 1 PackageFamilyName, Publisher, SignatureKind | " + + "ConvertTo-Json -Compress"); + + process = Process.Start(psi); + if (process is null) + return null; + + using var timeout = new CancellationTokenSource(TimeSpan.FromSeconds(8)); + var stdoutTask = process.StandardOutput.ReadToEndAsync(timeout.Token); + var stderrTask = process.StandardError.ReadToEndAsync(timeout.Token); + + if (!process.WaitForExit(8_000)) + { + try { process.Kill(entireProcessTree: true); } catch { } + return null; + } + + string output; + try + { + output = stdoutTask.GetAwaiter().GetResult(); + _ = stderrTask.GetAwaiter().GetResult(); + } + catch + { + return null; + } + + if (process.ExitCode != 0 || string.IsNullOrWhiteSpace(output)) + return null; + + return ParseAppxPackageJson(output); + } + catch + { + return null; + } + finally + { + process?.Dispose(); + } + } + + private static string ResolvePowerShellPath() + { + var systemRoot = Environment.GetFolderPath(Environment.SpecialFolder.Windows); + if (string.IsNullOrWhiteSpace(systemRoot)) + systemRoot = Environment.GetEnvironmentVariable("SystemRoot"); + if (string.IsNullOrWhiteSpace(systemRoot)) + systemRoot = @"C:\Windows"; + return Path.Combine(systemRoot, "System32", "WindowsPowerShell", "v1.0", "powershell.exe"); + } + + private static AppxPackageInfo? ParseAppxPackageJson(string json) + { + try + { + using var document = JsonDocument.Parse(json); + var root = document.RootElement; + if (root.ValueKind == JsonValueKind.Array) + { + if (root.GetArrayLength() == 0) + return null; + root = root[0]; + } + if (root.ValueKind != JsonValueKind.Object) + return null; + + var familyName = GetStringProperty(root, "PackageFamilyName"); + if (familyName is null) + return null; + + return new AppxPackageInfo( + familyName, + GetStringProperty(root, "Publisher"), + GetStringProperty(root, "SignatureKind")); + } + catch + { + return null; + } + } + + private static string? GetStringProperty(JsonElement element, string propertyName) => + element.TryGetProperty(propertyName, out var value) && value.ValueKind == JsonValueKind.String + ? value.GetString() + : null; } diff --git a/tests/OpenClaw.Connection.Tests/ManagedLocalGatewayPortProvenanceServiceTests.cs b/tests/OpenClaw.Connection.Tests/ManagedLocalGatewayPortProvenanceServiceTests.cs index b5856a970..c2ab454f4 100644 --- a/tests/OpenClaw.Connection.Tests/ManagedLocalGatewayPortProvenanceServiceTests.cs +++ b/tests/OpenClaw.Connection.Tests/ManagedLocalGatewayPortProvenanceServiceTests.cs @@ -53,6 +53,94 @@ public void VerifyMicrosoftSignedFile_RejectsUnsignedAssembly() Assert.Contains("Authenticode verification failed", result.Detail); } + private const string WslPackageFamilyName = + "MicrosoftCorporationII.WindowsSubsystemForLinux_8wekyb3d8bbwe"; + private const string NonCanonicalWslRelayPath = @"C:\Program Files\WSL\wslrelay.exe"; + + [Fact] + public void VerifyMicrosoftSignedFile_AuthenticodeFailsButWslPackageCorroborates_IsTrusted() + { + var result = WindowsAuthenticodeVerifier.VerifyMicrosoftSignedFile( + NonCanonicalWslRelayPath, + () => new AppxPackageInfo( + WslPackageFamilyName, + "CN=Microsoft Windows, O=Microsoft Corporation, C=US", + "Developer")); + + Assert.True(result.IsTrusted, result.Detail); + } + + [Fact] + public void VerifyMicrosoftSignedFile_AuthenticodeFailsAndNoWslPackageFound_IsRejectedWithOriginalDetail() + { + var result = WindowsAuthenticodeVerifier.VerifyMicrosoftSignedFile( + NonCanonicalWslRelayPath, + () => null); + + Assert.False(result.IsTrusted); + Assert.Contains("Authenticode", result.Detail, StringComparison.Ordinal); + } + + [Fact] + public void VerifyMicrosoftSignedFile_AuthenticodeFailsAndWslPackageFamilyNameMismatches_IsRejected() + { + var result = WindowsAuthenticodeVerifier.VerifyMicrosoftSignedFile( + NonCanonicalWslRelayPath, + () => new AppxPackageInfo( + "SomeImpostor.WindowsSubsystemForLinux_deadbeefcafe", + "CN=Microsoft Windows, O=Microsoft Corporation, C=US", + "Developer")); + + Assert.False(result.IsTrusted); + } + + [Fact] + public void VerifyMicrosoftSignedFile_AuthenticodeFailsAndWslPackageUnsigned_IsRejected() + { + var result = WindowsAuthenticodeVerifier.VerifyMicrosoftSignedFile( + NonCanonicalWslRelayPath, + () => new AppxPackageInfo( + WslPackageFamilyName, + "CN=Microsoft Windows, O=Microsoft Corporation, C=US", + "None")); + + Assert.False(result.IsTrusted); + Assert.Contains("WSL package", result.Detail, StringComparison.Ordinal); + } + + [Fact] + public void VerifyMicrosoftSignedFile_AuthenticodeFailsAndWslPackagePublisherNotMicrosoft_IsRejected() + { + var result = WindowsAuthenticodeVerifier.VerifyMicrosoftSignedFile( + NonCanonicalWslRelayPath, + () => new AppxPackageInfo( + WslPackageFamilyName, + "CN=Evil Corp, O=Evil Corp, C=US", + "Developer")); + + Assert.False(result.IsTrusted); + Assert.Contains("WSL package", result.Detail, StringComparison.Ordinal); + } + + [Fact] + public void VerifyMicrosoftSignedFile_AuthenticodeSucceeds_NeverConsultsWslPackageFallback() + { + var windowsDir = Environment.GetFolderPath(Environment.SpecialFolder.Windows); + var wslPath = Path.Combine(windowsDir, "System32", "wsl.exe"); + var fallbackInvoked = false; + + var result = WindowsAuthenticodeVerifier.VerifyMicrosoftSignedFile( + wslPath, + () => + { + fallbackInvoked = true; + throw new InvalidOperationException("Fallback should not be consulted."); + }); + + Assert.True(result.IsTrusted, result.Detail); + Assert.False(fallbackInvoked); + } + [Theory] [InlineData("CN=Microsoft Windows, O=Microsoft Corporation, C=US", true)] [InlineData("CN=Microsoft Corporation Test Certificate, O=Example Corp, C=US", false)]