Skip to content

nb__search matches literal prefixes, not natural-language tool queries #33

@mgoldsborough

Description

@mgoldsborough

Observation

From conversation conv_30f049cdb75d464f on ws_mat: an agent needed to find the todo-board tools and went through three searches before hitting any:

Query Result
"todo task create" 0 results
"todo board" 0 results
"synapse-todo" 26 results

Every attempt after the first was a wasted tool-call round-trip. The app name is synapse-todo-board; the tools are create_board_task, batch_archive, etc. Natural-language terms ("todo", "task", "create") appear in tool names and descriptions but nb__search didn't surface them.

Behavior

nb__search appears to match on exact token or simple substring against tool names only, not against tool descriptions. Hyphenated app slugs (synapse-todo-board) aren't tokenized on -, so "todo" doesn't match.

Why this matters

  • Visible cost: each failed search is a tool-call round-trip — LLM output tokens + latency + context growth. Three searches ≈ 300+ output tokens on the example above.
  • Invisible cost: agents that can't find a tool in 2-3 tries typically give up or fall back to a worse approach. Users see "the agent doesn't know how to do X" when really the agent didn't know which tool to call.

Suggested

  1. Tokenize on - and _: synapse-todo-board matches on todo, board, synapse.
  2. Match against descriptions, not just names: the tool description for create_board_task starts with "Create a task on a specific board" — the query "create task" should hit it.
  3. Rank by query-term coverage: a match on all query terms beats a match on one.
  4. Fuzzy / edit-distance fallback (optional): handle "todo boards" → matches "todo-board".

A simple BM25 or tf-idf over name + description per tool would cover (2) and (3) with ~20 LOC and no new deps.

Severity

Medium. Every agent-driven workflow involving tool discovery pays this cost. Affects UX more than correctness.

Source

conv_30f049cdb75d464f, ws_mat workspace, lines 9-22 in the JSONL log.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions