The Big Brain Update

[giveen]

llama.cpp "Think" Effort Levels — Integration Design for LATE

Status: Research complete. Architecture revised for production stability.
Target models: Gemma 4 and Qwen3.6
Context: Replace the current --gemma-thinking string-concatenation hack with clean, API-level thinking control using chat_template_kwargs.enable_thinking + thinking_budget_tokens backed by context-aware clamping.

---

TLDR

Replace --gemma-thinking (string hack) with --thinking <mode> (API-level control). The llama.cpp server accepts two cooperating fields: chat_template_kwargs.enable_thinking (master on/off) and thinking_budget_tokens (token ceiling). Five modes: off, auto (ctx-dependent, default), low (1K), medium (4K), high (16K), xhigh (32K).

The core system (P0): ThinkingConfig struct with *int Budget → buildExtraBody() constructs the payload with context-aware budget clamping. Dynamic MaxTokens expansion prevents truncation. Config via --thinking flag or config.json (thinking_budget, late_subagent_thinking_budget).

The upgrades (P1–P4):

Priority	What	Why
P1	NDJSON append + async save	xhigh budgets make synchronous full-rewrite saves painfully slow
P2	Parallel file walk + sqz stderr fix	~8x search speedup, clean terminal
P3	Tool result cache + parallel tool execution	Skip redundant reads, concurrent independent calls
P4	Symbol index (go-tree-sitter, CGO)	O(1) structural queries, lazy-build

Verified safe (no action): slot isolation between concurrent subagents, no attention drift at 16K reasoning, budget cutoff is exact (±2 tokens for UTF-8), tag injection guarded by token-level matching.

Summary

LATE targets Gemma 4 and Qwen3.6. Verified against the llama.cpp server source at /mnt/storage/llama.cpp/tools/server/server-common.cpp.

The API uses two cooperating fields in the /v1/chat/completions request body:

Field	Location	Effect	Source
chat_template_kwargs.enable_thinking	Request body	Master on/off — tells the Jinja template whether to render thinking tags. false = no [REASONING] / <think> markers.	server-common.cpp:1071-1086
thinking_budget_tokens	Request body	Token cap on the reasoning block. 0 = immediate end, N = N tokens max, -1/omit = server default (unbounded).	server-common.cpp:1119-1129
--reasoning on|off	Server startup	Prerequisite. Without it, neither field has effect.	server flag
--reasoning-format	Server startup	Controls output parsing: none (raw), deepseek (reasoning_content), deepseek-legacy (both). Default: auto.	common/arg.cpp:3258-3267, common/chat.cpp:837-851

Key finding from source: Both fields work cooperatively. chat_template_kwargs is NOT silently ignored — it's explicitly parsed and forwarded to the Jinja engine. enable_thinking is the master gate; thinking_budget_tokens is the ceiling.

CLI: --thinking off|auto|low|medium|high|xhigh replaces --gemma-thinking. Default is auto — proportional to the model's context window.

---

The Problem

Legacy Hack

The current implementation forcibly prepends <|think|> to the system prompt.

1. It is a fragile hack that ignores the API contract.
2. It breaks entirely on Qwen3.6 (which expects different tags).
3. It lacks token budgeting, leading to runaway reasoning that starves the actual response of context.

The Upstream Reality

The llama.cpp server exposes two cooperating controls:

• chat_template_kwargs.enable_thinking — parsed from body at server-common.cpp:1079-1086. Passed to the Jinja template to control whether <think> / [REASONING] tags are rendered at all. Works for both Gemma 4 and Qwen3.6.
• thinking_budget_tokens — parsed from body at server-common.cpp:1119. Sets a hard token ceiling on reasoning output. Only activates when the template produced a thinking_end_tag.

You must set both: enable_thinking toggles the template rendering; thinking_budget_tokens caps the reasoning block length.

---

The Design

1. The Payload

Every thinking-enabled request sends both fields via extraBody (hoisted to root by marshalFlattened):

{
  "model": "...",
  "messages": [...],
  "chat_template_kwargs": {
    "enable_thinking": true
  },
  "thinking_budget_tokens": 4096
}

2. Go Implementation — ThinkingConfig

To avoid Go's zero-value trap (0 cannot be distinguished from "omitted"), the configuration uses a pointer (*int). nil means auto (ctx-dependent). 0 means disabled. >0 means that budget.

// internal/session/session.go
type ThinkingConfig struct {
    Budget           *int // nil = auto (ctx-dependent), 0 = disabled, >0 = cap
    ExpectedOutput   int  // tokens reserved for the visible answer (default: 512)
    SafetyMargin     int  // tokens reserved from context window (default: 1024)
}

func (s *Session) buildExtraBody() map[string]any {
    extra := make(map[string]any)

    if s.thinkingConfig == nil {
        return extra
    }

    // Resolve auto mode: nil budget → ctx-dependent value
    budget := s.thinkingConfig.Budget
    if budget == nil {
        b := autoBudget(s.client.ContextSize())
        budget = &b
    }

    // Master switch — enable_thinking false only when budget is exactly 0
    enableThinking := *budget > 0
    extra["chat_template_kwargs"] = map[string]bool{
        "enable_thinking": enableThinking,
    }

    // Budget — always send when config is set (even 0)
    actualBudget := *budget

    // Context-aware clamp: never exceed remaining context
    if actualBudget > 0 && s.client.IsLlamaCPP() {
        ctxSize := s.client.ContextSize()
        if ctxSize > 0 {
            usedTokens := common.CalculateHistoryTokens(s.History, s.systemPrompt, s.GetToolDefinitions())
            remaining := ctxSize - usedTokens - s.thinkingConfig.SafetyMargin
            if remaining < 0 {
                remaining = 0
            }
            if actualBudget > remaining {
                // Floor: if clamping would starve thinking, disable it entirely
                if remaining < 256 {
                    actualBudget = 0
                    extra["chat_template_kwargs"] = map[string]bool{"enable_thinking": false}
                } else {
                    actualBudget = remaining
                }
            }
        }
    }

    extra["thinking_budget_tokens"] = actualBudget
    return extra
}

func autoBudget(ctxSize int) int {
    switch {
    case ctxSize <= 16384:
        return 1024   // low — small models
    case ctxSize <= 32768:
        return 4096   // medium
    case ctxSize <= 65536:
        return 8192   // between medium and high
    case ctxSize <= 131072:
        return 16384  // high
    default:
        return 32768  // xhigh — 256K models
    }
}

3. Dynamic MaxTokens Adjustment

Reasoning tokens consume the overall max_tokens (aliased to n_predict in llama.cpp). If LATE requests 4,096 thinking tokens but max_tokens defaults to 2,048, the server truncates mid-thought. LATE must expand the ceiling.

const ReasoningEndOverhead = 16 // generous bound for forced </think> + newline

if s.thinkingConfig != nil && budget != nil && *budget > 0 {
    needed := *budget + s.thinkingConfig.ExpectedOutput + ReasoningEndOverhead
    if req.MaxTokens < needed {
        req.ExtraBody["max_tokens"] = needed
    }
}

MaxTokens is passed through extraBody["max_tokens"] since ChatCompletionRequest doesn't have a MaxTokens field. marshalFlattened hoists it to root.

Key parameters:

Parameter	Default	Description
ExpectedOutput	512	Min tokens reserved for the visible answer
SafetyMargin	1024	Padding to prevent n_ctx overflow
ReasoningEndOverhead	16	Covers forced \n + end_tag + \n (2–5 tokens real, 16 safe)

4. Configuration & CLI Mapping

CLI flag: --thinking <mode> replaces --gemma-thinking. Default is auto.

Mode	Budget Value	enable_thinking	thinking_budget_tokens	Use Case
off	0	false	0	Direct tool execution, no thinking overhead
auto	ctx-dependent	true	dynamic	Default — proportional to context size
low	1024	true	1024	Quick tasks, minimal reasoning
medium	4096	true	4096	Balanced reasoning
high	16384	true	16384	Complex multi-step planning
xhigh	32768	true	32768	Deep research, long-form reasoning

Config config.json schema (values override auto mode):

{
  "thinking_budget": null,
  "late_subagent_thinking_budget": null
}

These integers map to ThinkingConfig.Budget:

• null or omitted → use auto mode (ctx-dependent budget)
• 0 → thinking disabled (off)
• >0 → that exact budget (maps to low/medium/high/xhigh tiers)

5. Subagent Propagation

Subagents receive their own ThinkingConfig via the runner closure, allowing independent budgets (e.g., primary agent at high, subagents at low to preserve context).

// Replaces gemmaThinking bool param in NewSubagentOrchestrator
func NewSubagentOrchestrator(
    c *client.Client,
    goal string,
    ctxFiles []string,
    agentType string,
    enabledTools map[string]bool,
    injectCWD bool,
    thinkingCfg *ThinkingConfig, // was gemmaThinking bool
    maxTurns int,
    parent common.Orchestrator,
    messenger tui.Messenger,
) (common.Orchestrator, error) {
    // ... (same logic, but build extraBody with chat_template_kwargs + thinking_budget_tokens)
}

6. Verification / Observability

The stream response signals whether the budget is respected:

Signal	Where	What it tells you
delta.reasoning_content	Stream chunks	Non-empty = thinking is active. Empty = disabled or budget exhausted.
usage.completion_tokens	Final chunk	Total generated tokens (thinking + output). Compare vs budget.
Estimated reasoning tokens	len(accumulated.Reasoning)/3.5	Approximate check: if estimated >> budget, the server may be ignoring it.

Pre-flight detection: if the first N chunks contain no reasoning_content despite enable_thinking: true, emit a warning that the server may not support reasoning.

---

Implementation Plan

Priority	Step	Component	Action
P0	1	session	Add ThinkingConfig struct with *int Budget, ExpectedOutput, SafetyMargin. Add buildExtraBody() method with auto mode and context-aware clamp.
P0	2	session	Implement dynamic MaxTokens expansion with ReasoningEndOverhead.
P0	3	cmd/late	Implement --thinking <mode> CLI parser. Remove --gemma-thinking. Default auto.
P0	4	config	Add thinking_budget and late_subagent_thinking_budget ints to Config (default null).
P0	5	agent	Replace gemmaThinking bool param with *ThinkingConfig in NewSubagentOrchestrator.
P1	6	session/persistence	NDJSON append — replace full-rewrite SaveHistory with AppendHistory. O(1) per save.
P1	7	session/persistence	Async save — fire-and-forget goroutine in saveAndNotify(). Zero-latency mutations.
P2	8	tool/search.go	Parallel walk — replace sequential filepath.WalkDir with goroutine worker pool. ~8x speedup.
P2	9	tool/utils.go	sqz stderr fix — capture cmd.Stderr instead of leaking to terminal.
P3	10	executor/cache.go	Tool result cache — skip redundant reads via (tool, args_hash) key with 30s TTL.
P3	11	executor/executor.go	Parallel tool execution — dependency-graph wave scheduling for independent tool calls.
P3	12	session/persistence	Gzip reasoning_content — compress the bulk field, decompress on load. 5:1 reduction.
P4	13	search/index.go	Symbol index — go-tree-sitter lazy-build index for O(1) structural queries. CGO.

Net Impact: Complete eradication of prompt hacking, correct API contract (two cooperating fields), guaranteed context safety via auto-clamping, O(1) session persistence, parallelized I/O throughout the tool stack, and strict alignment with the llama.cpp /v1/chat/completions standard.

---

Operational Considerations

P1: Session Persistence — Rewrite Tax at High Thinking Budgets

Every message mutation calls saveAndNotify() synchronously — the full []ChatMessage JSON is marshaled and atomically written to disk before execution continues. With xhigh (32K) thinking, this becomes a measurable bottleneck:

Per turn with xhigh (32K reasoning):
  1 assistant message with 32K reasoning + tool calls  → save 32K+  (growing)
  N tool result messages                                → save N times
  Next assistant response                               → save again

50-turn session: ~500 saves × ~6.4MB each = O(n²) total write

History size	Save time (NVMe)	Save time (SATA SSD)	Cumulative over 50 turns
100K tokens (~400KB)	~0.8ms	~4ms	~200ms
500K tokens (~2MB)	~4ms	~20ms	~1s
1.6M tokens (~6.4MB)	~10ms	~60ms	~3s
5M tokens (~20MB)	~30ms	~200ms	~10s

For reference: a 50-turn session with high (16K) thinking and average 3 tool calls per turn produces ~1M tokens of history. Each save rewrites the entire file.

Fix 1: Append-only NDJSON

Replace the full JSON array with newline-delimited JSON. Each message is appended as a single line — O(1) per save regardless of history size:

// internal/session/persistence.go

// AppendHistory adds one message to the history file (O(1)).
func AppendHistory(path string, msg client.ChatMessage) error {
    f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600)
    if err != nil {
        return err
    }
    defer f.Close()
    return json.NewEncoder(f).Encode(msg) // one JSON line
}

// LoadHistory streams all lines from the history file.
func LoadHistory(path string) ([]client.ChatMessage, error) {
    f, err := os.Open(path)
    if err != nil {
        if os.IsNotExist(err) {
            return []client.ChatMessage{}, nil
        }
        return nil, err
    }
    defer f.Close()

    var history []client.ChatMessage
    scanner := bufio.NewScanner(f)
    buf := make([]byte, 0, 1024*64) // 64KB scan buffer for long reasoning fields
    scanner.Buffer(buf, len(buf))

    for scanner.Scan() {
        var msg client.ChatMessage
        if err := json.Unmarshal(scanner.Bytes(), &msg); err != nil {
            // Skip corrupt lines — append-only means the last entry may be partial
            // if the process was killed mid-write
            continue
        }
        history = append(history, msg)
    }
    return history, scanner.Err()
}

Compaction: Periodically rewrite the NDJSON file as a single compact JSON array to reclaim space from deleted/corrupt lines. Trigger on session load or when the file exceeds a threshold (e.g., 100MB).

Fix 2: Async Save (non-blocking)

Don't block the execution loop waiting for disk:

// internal/session/session.go
func (s *Session) saveAndNotify() error {
    if s.HistoryPath == "" || len(s.History) == 0 {
        return nil
    }
    // Deep-copy the last message to avoid race with ongoing mutations
    go func(path string, msg client.ChatMessage) {
        if err := AppendHistory(path, msg); err != nil {
            log.Printf("failed to append history: %v", err)
        }
    }(s.HistoryPath, s.History[len(s.History)-1])
    return nil
}

The goroutine captures the last message by value (no pointer aliasing). If the write fails, it's logged but doesn't halt the agent. On session load, trailing corrupt lines are skipped.

Fix 3 (P3): Gzip reasoning_content

The reasoning_content field is the bulk of the history file (80%+ at high thinking budgets). Gzip it independently:

import "compress/gzip"

func compressReasoning(content string) []byte {
    var buf bytes.Buffer
    w := gzip.NewWriter(&buf)
    w.Write([]byte(content))
    w.Close()
    return buf.Bytes()
}

func decompressReasoning(data []byte) (string, error) {
    r, err := gzip.NewReader(bytes.NewReader(data))
    if err != nil {
        return "", err
    }
    defer r.Close()
    out, err := io.ReadAll(r)
    return string(out), err
}

Store as a []byte (base64-encoded by JSON marshaling). At 5:1 compression for prose reasoning, this cuts the 6.4MB example to ~1.3MB.

Fix 4 (P4): Subagent Sidecar Files

Subagents currently set HistoryPath = "" — their reasoning is invisible on session reload. Without capturing the full conversation (which would bloat the primary session), write a lightweight sidecar with the subagent's structure:

// internal/agent/agent.go
type SubagentTrace struct {
    ID        string   `json:"id"`
    Goal      string   `json:"goal"`
    AgentType string   `json:"agent_type"`
    ToolsUsed []string `json:"tools_used"`
    TurnCount int      `json:"turn_count"`
    Result    string   `json:"result"`        // final output (summary, not full reasoning)
    Error     string   `json:"error,omitempty"`
}

Saved to <session-path>-sidecar-{subagent-id}.json. On load, displayed as a collapsible trace in the TUI.

---

P2: Search Tool Parallelization — Pure Go Speedup

The agent's SearchTool (internal/tool/search.go) currently walks directories sequentially with filepath.WalkDir, reading files one by one. For large codebases this is a bottleneck during reasoning — the model burns thinking budget waiting for search results.

Fix: Replace the sequential walk with a parallel worker pool. Pure Go, no system tools, no CGO:

import (
    "runtime"
    "sync"
)

// In Execute(), replace:
//   err = filepath.WalkDir(searchPath, walkFn)
//
// With a two-phase approach:

// Phase 1: Fast collect — walk dir tree collecting candidate file paths,
// applying .gitignore and extension filters but NOT reading files.
paths := collectFiles(searchPath) // sequential, but just stat + filter

// Phase 2: Parallel scan — fan out reads across worker goroutines.
numWorkers := runtime.GOMAXPROCS(0) // typically 8–16 on modern hardware
work := make(chan string, len(paths))
results := make(chan matchResult, len(paths))

var wg sync.WaitGroup
for w := 0; w < numWorkers; w++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        for path := range work {
            // readFileContentWithCount is the existing scanner — thread-safe
            // since each worker gets its own file handle
            content, count, err := readFileContentWithCount(path, matchFunc, ...)
            if count > 0 {
                results <- matchResult{path, content, count}
            }
        }
    }()
}

// Send paths to workers
for _, path := range paths {
    work <- path
}
close(work)
wg.Wait()
close(results)

Speedup: On an 8-core machine, this gives ~8x faster search for file-read-bound workloads. The sequential Phase 1 (collectFiles) is fast because it only stats directories and checks names — no file reads.

Memory: Peak memory is bounded by the result channel buffer and one open file per worker. The existing maxSearchChars (32KB) limit on output prevents unbounded growth.

This is independent of the thinking system but directly impacts the agent's responsiveness during reasoning: a search that takes 8s sequentially completes in ~1s with parallel workers, saving the thinking budget for actual reasoning instead of blocking on I/O.

---

P2: sqz Compression — Stderr Leak Fix

LATE uses sqz (internal/tool/utils.go) to compress shell command output via CompressWithSqz(). The current implementation leaks sqz's per-compression stats to the terminal:

// Current — stderr goes to parent process stderr (terminal noise)
cmd := exec.CommandContext(ctx, "sqz", "compress", "--cmd", command)
cmd.Stdin = strings.NewReader(string(input))
return cmd.Output() // stderr inherited from os.Stderr

Fix: Capture stderr explicitly — log it for debugging instead of leaking to the user's terminal:

func CompressWithSqz(ctx context.Context, input []byte, command string) ([]byte, error) {
    if !IsSqzAvailable() {
        return input, nil
    }

    cmd := exec.CommandContext(ctx, "sqz", "compress", "--cmd", command)
    cmd.Stdin = strings.NewReader(string(input))

    var stderrBuf strings.Builder
    cmd.Stderr = &stderrBuf

    out, err := cmd.Output()
    if err != nil {
        // Log stderr for debugging, but fall back to raw input
        log.Printf("sqz compress failed: %v, stderr: %s", err, stderrBuf.String())
        return input, nil
    }

    // Optional: parse stderr for token savings stats and feed back into
    // the thinking budget accounting (e.g., log saved tokens)
    return out, nil
}

The dedup cache is persistent (SHA-256 content hash in ~/.sqz/sessions.db), so it works correctly across independent sqz compress invocations — no MCP server needed.

---

P3: Tool Result Cache

The same file is often read multiple times across different reasoning steps (e.g., "read config → edit config → verify config"). Currently each read walks the filesystem, reads, and compresses via sqz — even when the content hasn't changed.

A result cache keyed by (tool_name, args_hash) skips redundant execution entirely:

// internal/executor/cache.go
var resultCache sync.Map

type cachedResult struct {
    data      string
    expiresAt time.Time
}

func cacheKey(tool string, args json.RawMessage) string {
    return fmt.Sprintf("%s:%x", tool, sha256.Sum256(args))
}

func cachedExecute(tool string, args json.RawMessage) (string, bool) {
    key := cacheKey(tool, args)
    if cached, ok := resultCache.Load(key); ok {
        cr := cached.(cachedResult)
        if time.Now().Before(cr.expiresAt) {
            return cr.data, true
        }
        resultCache.Delete(key)
    }
    return "", false
}

func storeCache(tool string, args json.RawMessage, data string) {
    key := cacheKey(tool, args)
    resultCache.Store(key, cachedResult{
        data:      data,
        expiresAt: time.Now().Add(30 * time.Second), // TTL for freshness
    })
}

Where it wins:

Pattern	Without cache	With cache	Saved
Read file → edit → re-read	3 reads, 3 walks, 3 compressions	1 read + 2 cache hits	~66% of I/O
git status → git diff → commit	3 git calls	1 git call + 2 cache hits	~66% of I/O
Search for symbol → navigate → search again	2 walks	1 walk + 1 cache hit	~50% of I/O

TTL of 30s prevents stale results while catching repeated reads within the same reasoning block. On cache miss, the result is still stored for the next call in the sequence.

---

P3: Parallel Tool Execution

When the model emits multiple tool calls in one response, they're currently executed serially. If the calls are independent (no data dependency), execute them concurrently:

// internal/executor/executor.go
func ExecuteToolCalls(ctx context.Context, sess *session.Session,
    toolCalls []client.ToolCall, middlewares []common.ToolMiddleware) error {

    // Group calls into dependency chains
    type callGroup struct {
        calls []client.ToolCall
        mu    sync.Mutex
    }

    // First pass: detect dependencies between calls
    // (e.g., write_file depends on read_file returning first)
    // Independent calls → parallel group
    // Dependent calls → serial chain
    depGraph := buildDependencyGraph(toolCalls)
    waves := depGraph.TopologicalWaves()

    for _, wave := range waves {
        if len(wave) == 1 {
            // Single call, execute normally
            if err := executeOne(ctx, sess, wave[0], middlewares); err != nil {
                return err
            }
        } else {
            // Independent calls, execute in parallel
            var wg sync.WaitGroup
            errs := make([]error, len(wave))

            for i, tc := range wave {
                wg.Add(1)
                go func(idx int, call client.ToolCall) {
                    defer wg.Done()
                    errs[idx] = executeOne(ctx, sess, call, middlewares)
                }(i, tc)
            }
            wg.Wait()

            for _, err := range errs {
                if err != nil {
                    return err
                }
            }
        }
    }
    return nil
}

Dependency detection is conservative: file reads are independent, file writes are ordered, and the same variable being set and then read is serialized. For the common case of "read A, read B, read C, then edit both", reads A/B/C run in parallel while the edit waits.

Wall-clock impact: A model response with 3 independent tool calls completes in the time of the slowest call, not the sum. Over a 50-turn session where ~40% of turns emit multiple independent calls, this cuts session time by ~25%.

---

P4: Symbol Index (go-tree-sitter)

The agent frequently asks structural questions: "where is this function defined?", "find all callers of X", "what imports does this file use?" Currently each query walks the filesystem and regex-scans files — O(n) per query.

A pre-built symbol index answers in O(1):

// internal/search/index.go — build once, incremental updates
type SymbolIndex struct {
    Functions map[string][]Location  // function name → file:line
    Types     map[string][]Location  // type name → file:line
    Imports   map[string][]string    // package → importing files
}

func (idx *SymbolIndex) Lookup(symbol string) []Location {
    return idx.Functions[symbol]
}

func BuildIndex(root string) (*SymbolIndex, error) {
    idx := &SymbolIndex{
        Functions: make(map[string][]Location),
        Types:     make(map[string][]Location),
        Imports:   make(map[string][]string),
    }

    // Walk files in parallel (same pattern as Search Tool)
    // For each recognized language, parse with go-tree-sitter
    // Extract symbol definitions at the AST level
    // Populate maps
    return idx, nil
}

Speedup: A sequential directory walk for "find function X" takes 200ms–2s depending on repo size. The index lookup is <1µs. Over a 50-turn agent session, this saves 10–100s of wall-clock time — directly converting I/O wait into thinking budget.

Tradeoff: Requires go-tree-sitter (CGO). Only pays off when the agent asks multiple structural questions per session. For the common "fix one file" pattern, the index build time (~1s for a moderate repo) is a net loss. Only build the index lazily on first structural query, not at session start.

var globalIndex *SymbolIndex
var indexOnce sync.Once

func GetIndex(root string) *SymbolIndex {
    indexOnce.Do(func() {
        idx, err := BuildIndex(root)
        if err == nil {
            globalIndex = idx
        }
    })
    return globalIndex
}

---

Deep Analysis (No Action Required)

These sections document verified behavior that is safe as-is. No code changes needed.

Context Sizing & Auto Mode

The auto mode reads the model's context size via client.ContextSize() and picks a proportional budget. The mapping reserves ~10–12.5% of context for thinking at each tier:

Context	Auto budget	% of ctx	Tier
≤16K	1024	6–12%	low
≤32K	4096	12–25%	medium
≤64K	8192	12–25%	mid-high
≤128K	16384	12–25%	high
>128K	32768	≤25%	xhigh

At 64K context, even xhigh (32K) is not clamped until conversation history exceeds ~31K tokens (~100 tool call rounds). The clamp only matters for very long sessions.

Recommendation:

• 64K minimum — all modes work, clamping only affects very long sessions with xhigh
• 256K recommended for primary agent — accommodates deep conversation histories
• Subagents with short lifespans (<30 turns) work fine at 64K

Slot Isolation — Concurrent Requests Are Safe

LATE spawns multiple subagents concurrently, sometimes with different thinking budgets. This works because the llama.cpp server strictly isolates thinking configuration per request:

Layer	What happens	Isolation boundary	Source
Template kwargs	chat_template_kwargs.enable_thinking is consumed during common_chat_templates_apply() at request parse time. Not stored in slot state.	Per-request — rendered prompt is static by the time it hits the slot.	server-common.cpp:1071-1091
Thinking budget	thinking_budget_tokens → task_params.sampling.reasoning_budget_tokens. Each server_task carries its own copy.	Per-task — eval_llama_cmpl_schema() creates a local task_params per task.	server-schema.cpp:378-380, server-context.cpp:4132
Sampler	Reasoning-budget sampler is re-initialized from scratch per task: slot.smpl.reset(common_sampler_init(model_tgt, task.params.sampling)).	Per-task — destroyed and recreated every time a slot picks up a new task.	server-context.cpp:1817
Prompt	Rendered prompt is a plain string with thinking tags already baked in (or not). Different requests = different prompt strings.	Per-request — just tokenized text fed to the model.	server-common.cpp:1091-1094

Example: Slot 0 gets enable_thinking: true, budget: 4096. Slot 1 gets enable_thinking: false, budget: 0. No shared state.

Long Thinking & Attention Drift

After 16K tokens of reasoning, will the model "forget" the tool schema? No. Three layers of defense:

1. Grammar enforces syntax — lazy grammar is compiled from tool definitions, pre-filled with the generation prompt, and constrains token-level output regardless of attention state.
2. YaRN scaling handles 16K comfortably — both models use YaRN for 256K context. 16K = 6% of the trained window.
3. Wrong tool choices are recoverable — grammar prevents structurally invalid JSON, tool execution failures feed back as error responses.

Model	Max ctx	16K thinking	% used	Attention risk
Qwen3.6	262,144	16,384	6%	Negligible
Gemma 4	262,144	16,384	6%	Negligible

Budget Cutoff Precision

The budget sampler counts down per-token and transitions to FORCING when remaining <= 0. The next generated token is forced to be the end-of-reasoning sequence.

Condition	Overshoot	Source
Normal (UTF-8 complete at exhaustion)	0 tokens — exact cutoff	remaining <= 0 → immediate FORCING at reasoning-budget.cpp:103-107
UTF-8 split (multi-byte char straddles boundary)	1–2 tokens — waits for completion	WAITING_UTF8 state at reasoning-budget.cpp:108-112
Re-arm on new <think>	Fresh budget tokens — restarts	DONE → re-arm at reasoning-budget.cpp:127-137

The formula budget + ExpectedOutput + ReasoningEndOverhead(16) provides 23 tokens of slack — comfortable for any tokenizer.

Security: Tag Injection & Reasoning Leakage

LATE ingests untrusted content (file reads, curl output, logs). The server protects at three layers:

1. Tool calls are a separate JSON field — delta.tool_calls is independent of reasoning_content. (server-chat.cpp:552-553, server-task.cpp:768-1210)
2. The budget sampler operates on tokens, not text — token_matcher matches exact token sequences, not substrings. (reasoning-budget.cpp:42-43)
3. The PEG parser is derived from the template — reasoning markers detected at template analysis time; input context tags don't affect delimiter matching. (chat-diff-analyzer.cpp)

Clamped budget behavior: When context pressure clamps to a very low value (e.g., 20), the consequence is quality degradation, not agent deadlock. The lazy grammar only activates after the thinking block closes. Floor the clamped budget at 256; below that, disable thinking entirely.

---

Server Requirements

LATE relies on the user to boot the llama.cpp server correctly.

Gemma 4:

llama-server -m gemma4.gguf --jinja --reasoning on -c 262144

Qwen3.6:

llama-server -m qwen3.6.gguf --jinja --reasoning on -c 262144

Both require --jinja (template engine) and --reasoning on (reasoning support). --reasoning-format defaults to auto (resolves to deepseek) — no explicit flag needed for either model.

[mlhher]

Need some time to understand this fully. While I agree that having flags only for specific models isn't nice I think for gemma the exception is fine as it is likely the one with Qwen that is going to be used everywhere (e.g. --mistral-thinking would be rejected random example).

[giveen]

Trust me, I had to have AI explain it over and over to me, but basically, a lot of coding agents have the option to increase their reasoning budget and basically this is an implementation of that, but trying to make it more generic.

[mlhher]

Doesn't llama-server just do it with --reasoning-budget and --reasoning-budget-message? If a user explicitly sets them I don't think we should interfere with that. Maybe I am misunderstanding the purpose here.

[giveen]

Doesn't llama-server just do it with --reasoning-budget and --reasoning-budget-message? If a user explicitly sets them I don't think we should interfere with that. Maybe I am misunderstanding the purpose here.

I think I was looking to do on the fly adjustments , would be my end goal, sometimes I need a model to think longer and consider all options first before implementing something.