Extension: first-class llama.cpp local model experience

[alejandro-ao]

Goal

Ship llama.cpp support as a separately maintained Tau extension, not as a built-in provider or a permanent llama.cpp-specific code path in Tau core.

The extension should make a running local llama-server feel like a first-class Tau model source: discover it, configure it without a fake API key, select its loaded model, diagnose coding-agent compatibility, and use it from the normal Tau model picker, TUI, and print mode.

PR #352 explored this experience as a built-in feature. Treat that PR as a prototype/reference for behavior, but implement the product surface as an extension against current main.

Target user experience

Install/load the extension

The extension should live in its own repository and support Tau's normal extension loading conventions:

git clone <llama-cpp-extension-repo>
tau -x ./<llama-cpp-extension-repo>

It should also work when installed under ~/.tau/extensions/ and should declare its entry point through [tool.tau] if it uses a src/ layout.

No project extension should be enabled implicitly.

First run

The user starts llama.cpp independently:

llama-server -hf <tool-capable-gguf>

Then, inside Tau:

/llama-cpp setup

The setup flow should:

1. Default to http://127.0.0.1:8080.
2. Accept a URL with or without /v1 and normalize it.
3. Check the server and distinguish unreachable, loading, authentication, and malformed-response failures.
4. Query /v1/models and use the real model IDs returned by the server.
5. If one model is available, select it automatically.
6. If several models are available, show context.ui.select(...) in the TUI; allow an explicit model argument in headless/scripted use.
7. Save only non-secret settings: endpoint, discovered models, and selected default model.
8. Register/refresh the provider so it immediately appears in Tau's provider/model surfaces without restarting.
9. Offer to switch the current session to the configured llama.cpp model.
10. Never require users to invent LLAMA_API_KEY=local or a fake model ID such as local.

Expected completion feedback:

✓ Found llama.cpp at http://127.0.0.1:8080/v1
✓ Discovered model: qwen3-coder.gguf
✓ Saved llama-cpp:qwen3-coder.gguf

After setup, users should be able to:

/model

and select the discovered llama-cpp:<model-id> entry, or start Tau explicitly:

tau -x ./<extension-repo> --provider llama-cpp --model <model-id>
tau -x ./<extension-repo> --provider llama-cpp --model <model-id> -p "summarize this project"

Extension commands

Register one command namespace with subcommands:

/llama-cpp

Alias for /llama-cpp status. Show:

• configured endpoint
• reachable/unreachable/loading state
• selected model
• all currently discovered models
• authentication source (none, environment, or stored credential if Tau exposes a safe credential API)
• last diagnostic result when available

/llama-cpp setup [URL] [MODEL]

Discover and configure a server. In the TUI, missing values should use host dialogs. In headless contexts, require explicit values when a choice cannot be made safely.

Examples:

/llama-cpp setup
/llama-cpp setup http://127.0.0.1:9090
/llama-cpp setup http://127.0.0.1:9090 coder.gguf

Because Tau extension command handlers are synchronous, the handler may spawn an event-loop task for discovery/dialog work and return immediately, following the documented extension pattern.

/llama-cpp models

Refresh /v1/models, show the available models, and allow selection in the TUI.

/llama-cpp use [MODEL]

Switch the active session to the configured llama.cpp provider/model. If MODEL is omitted and multiple models exist, open a picker.

/llama-cpp doctor [MODEL]

Run a deterministic compatibility check and report each stage independently:

1. server reachable/healthy
2. model discovery
3. unauthenticated or authenticated request accepted
4. streaming chat completion accepted
5. tool schema accepted
6. model emits a tool call

Example:

✓ Server reachable
✓ Model discovered: qwen3-coder.gguf
✓ Streaming chat completions
✓ Tool calls supported

The tool-call check should be a warning, not a fatal connectivity failure, when the server streams successfully but the model does not call the probe tool. Explain likely causes:

• model is not a tool-capable instruct model
• GGUF/chat template does not support tool calls
• llama.cpp needs updating or different server flags

/llama-cpp reset

Remove extension-owned non-secret configuration and unregister/disable the configured provider after confirmation. Do not stop an externally managed llama-server process.

Authentication behavior

llama.cpp does not require a bearer token unless launched with --api-key.

Required behavior:

• omit the Authorization header when no key is configured
• read LLAMA_API_KEY when present
• on HTTP 401/403, explain how to set LLAMA_API_KEY
• never write API keys to the extension's plain configuration file
• do not accept a command-line key by default because it leaks into shell history
• use Tau's credential store only if a supported public extension API exists; otherwise environment-only authentication is acceptable for v1

Provider behavior

The extension should reuse Tau's existing OpenAI-compatible transport rather than implement a separate model streaming stack.

The registered provider must support:

• base URL supplied by extension configuration
• optional authentication
• /chat/completions streaming
• dynamically discovered model IDs
• model-picker integration
• current-session provider/model switching
• session resume with the same provider/model
• print mode after setup
• no provider-specific assumptions in tau_agent

llama.cpp-specific discovery, diagnostics, configuration, and UX belong in the extension. Generic extension-provider registration seams belong in tau_coding. The OpenAI-compatible wire adapter remains in tau_ai.

Core extension API prerequisites

Current Tau extensions can register tools, slash commands, hooks, renderers, dialogs, and components, but the documented API does not support custom providers, CLI flag registration, or provider/model switching. Do not solve this by importing private tau_coding session/TUI internals from the extension.

Add the smallest generic public seams needed by this extension. The exact API can be refined during implementation, but it must cover:

1. Provider registration during setup(tau) or session_start.
   • register a provider name/display name
   • provide a runtime factory or an OpenAI-compatible provider descriptor
   • declare optional/no authentication
   • supply and later refresh a dynamic model list
2. Provider lifecycle.
   • unregister or replace an extension-owned provider on reload/reset
   • clear registrations automatically when an extension generation is invalidated
   • isolate provider-factory/discovery failures as extension diagnostics
3. Provider/model selection.
   • read the active provider/model
   • switch the current session through a public action
   • optionally persist the user's default selection
4. Startup ordering.
   • extension providers must be registered before CLI --provider/--model selection is resolved in TUI and print mode
5. Durable extension settings.
   • either expose extension-scoped non-secret storage or document a stable location/API
   • settings must be user-level by default so a cloned project cannot silently redirect a local provider endpoint

A possible shape is illustrative only:

def setup(tau):
    tau.register_provider(
        name="llama-cpp",
        display_name="llama.cpp",
        factory=create_provider,
        models=get_models,
    )

    # Later, after discovery:
    tau.update_provider_models("llama-cpp", models, default_model=selected)
    tau.select_model("llama-cpp", selected, persist_default=True)

Any final API must be generic enough for another local/OpenAI-compatible provider extension; do not add llama.cpp names to core.

UI behavior

Use host-provided extension APIs before custom widgets:

• context.ui.input for endpoint entry
• context.ui.select for model selection
• context.ui.confirm for switching/reset
• context.ui.notify for short success/failure feedback

A small prompt-adjacent status component is optional, not required for v1. The extension must remain fully functional without component support and in print mode.

All errors should be actionable. Avoid raw httpx tracebacks in normal UX.

Persistence

Persist only:

• normalized base URL
• discovered model IDs
• selected default model
• optional timestamp/summary of the last successful probe

Do not persist:

• API keys in plaintext extension settings
• server PIDs for processes the extension did not start
• guessed model context windows or capabilities

A /reload must preserve settings, re-register the provider, and avoid duplicate commands/providers/background tasks.

Testing

Deterministic tests

Use httpx.MockTransport or an equivalent fake server to cover:

• URL normalization with and without /v1
• successful /v1/models discovery
• one-model automatic selection
• multi-model picker/explicit selection
• empty/malformed model responses
• connection refused and timeout messages
• loading/unavailable status
• 401/403 guidance
• omission and inclusion of the bearer header
• streaming completion probe
• successful and inconclusive tool-call probes
• persistence and reload
• provider registration before CLI selection
• TUI and headless command behavior
• extension failure isolation
• reset/unregister behavior

Load the extension through the real ExtensionRuntime; do not only import its modules directly.

Manual validation

# 1. Start a tool-capable model
llama-server -hf <tool-capable-gguf>

# 2. Load the extension
tau -x ./<extension-repo>

# 3. In Tau
/llama-cpp setup
/llama-cpp status
/llama-cpp doctor
/model

# 4. Validate a real coding tool turn
Use the bash tool to run pwd and report the result.

# 5. Validate print mode after setup
tau -x ./<extension-repo> --provider llama-cpp -p "summarize this project"

Also validate:

• a server on a non-default port
• multiple models returned by /v1/models
• a server protected by --api-key with LLAMA_API_KEY
• a non-tool-capable model, which should produce a useful warning rather than a generic failure
• /reload and session resume

Non-goals for v1

• installing or compiling llama.cpp
• downloading or quantizing models inside Tau
• starting/stopping/managing llama-server
• CUDA/Metal/backend configuration
• benchmarking model quality or tokens per second
• guessing context windows from model names
• adding llama.cpp to Tau's built-in provider catalog
• adding llama.cpp-specific commands, settings, or UI to Tau core
• top-level tau llama-cpp ... CLI subcommands; extension slash commands are the v1 interface

Process management such as /llama-cpp serve can be considered later, after connecting to an independently managed server is reliable.

Acceptance criteria

• llama.cpp integration is delivered as a loadable extension in a separate repository.
• Tau core contains only generic extension-provider seams required to support it.
• /llama-cpp setup discovers real model IDs and requires no fake API key.
• The configured model appears in Tau's normal provider/model selection surfaces.
• The extension works in TUI and print mode after setup.
• Optional LLAMA_API_KEY authentication works without persisting plaintext secrets.
• /llama-cpp doctor distinguishes connectivity, streaming, and tool compatibility.
• /reload and session resume preserve correct behavior without duplicate registrations.
• Deterministic extension-runtime tests and a documented live llama.cpp smoke test exist.
• Published Tau extension docs describe the new generic provider-registration API.