Skip to content

fix: remove grammar constraint to fix node-llama-cpp 3.18.1 crash#567

Draft
LevinFaber wants to merge 1 commit intotobi:mainfrom
LevinFaber:fix/grammar-crash
Draft

fix: remove grammar constraint to fix node-llama-cpp 3.18.1 crash#567
LevinFaber wants to merge 1 commit intotobi:mainfrom
LevinFaber:fix/grammar-crash

Conversation

@LevinFaber
Copy link
Copy Markdown

Summary

Fixes crash during qmd query expansion phase caused by node-llama-cpp 3.18.1 regression in grammar-constrained generation.

Error: Failed to accept token in sampler: Unexpected empty grammar stack after accepting piece: 0 (15)

Root Cause

node-llama-cpp 3.18.1 (bundles llama.cpp b8390) introduced a regression where grammar-constrained generation crashes deterministically with the GBNF grammar used for query expansion. This affects both Bun and Node.js — not Bun-specific as initially assumed.

Changes

  • src/llm.ts: Remove grammar parameter from expandQuery() generation call
  • src/db.ts: Add missing transaction() method to Database interface (build fix)

Why Remove Grammar Instead of Downgrade?

Approach Pros Cons
Remove grammar (chosen) • Stays on latest node-llama-cpp (3.18.1)
• Includes CVE-2025-52566 & CVE-2026-2069 fixes
• No version pinning technical debt
• Slightly less strict output format enforcement
Downgrade to 3.17.1 • Grammar constraints work • Misses security fixes
• Pins to old version indefinitely
• llama.cpp b8390 had no grammar fixes planned

The grammar was redundant — manual parsing already correctly extracts lex:, vec:, hyde: lines from model output and validates types. The GBNF grammar provided no additional safety that wasn't already handled by the parser.

Test Results

Runtime node-llama-cpp Grammar Result
Bun v1.3.10 3.18.1 Removed ✓ Works (18.4s)
Node.js v24.13.1 3.18.1 Removed ✓ Works
Bun v1.3.10 3.17.1 Enabled ✓ Works (verified)
Node.js v24.13.1 3.18.1 Enabled ✗ Crash confirmed

Verification

  • Builds successfully (npm run build)
  • Tested with Bun — query expansion works
  • Tested with Node.js — query expansion works

Related

  • CVE-2025-52566: Fixed in 3.18.1 (tokenizer)
  • CVE-2026-2069: Fixed in 3.18.1 (grammar buffer overflow)

node-llama-cpp 3.18.1 crashes during grammar-constrained generation with:
'Failed to accept token in sampler: Unexpected empty grammar stack after accepting piece'

This affects BOTH Bun and Node.js - not Bun-specific as initially assumed.

Changes:
- Remove grammar constraint from expandQuery() - manual parsing handles output
- Add missing transaction() method to Database interface (build fix)

The manual parsing already correctly extracts lex:/vec:/hyde: lines from
the model output, making the grammar constraint redundant anyway.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant