fix(ai): ground the bot's claims, drop the hedge copy and the GitHub ticket comment

[NathanTarbert]

What this fixes

Three things about what the support bot says on public surfaces.

1. The bot can't confirm what it hasn't verified. The generator is one Claude call over a docs search with no tools attached — it can't read our source, reproduce a problem, or run a test. The prompt never said so, and asked for confident, polished answers, so the bot asserted root causes and named identifiers that don't exist in our codebase. ai/generator.ts now carries a grounding block that overrides the personality and formatting rules:

• The bot has not read our source, reproduced the problem, or run a test — and may never write or imply otherwise
• It may not confirm a bug: no "bug confirmed", "this is a real bug", "known issue", "root cause is", "the fix is". It acknowledges the report and says engineering will verify
• It may only name a file, class, prop, hook, or version that appears word for word in the docs it was given — otherwise it describes the idea in prose instead of inventing a name
• A causal explanation is marked as a guess once, and can't be restated as fact later in the same response
• It doesn't tell maintainers what to change in core. Workarounds in the reporter's own code are fine
• When it doesn't have enough to answer, escalating beats a good-sounding guess

2. The disclaimer no longer hedges about the answer. "and may be incomplete" is gone from every variant; all copy opens with This is an AI-generated response. If we're not confident the response escalates to a human — that's the mechanism, not a warning label on an answer we chose to post. The copy now lives in three constants in formatter.ts so GitHub, Discord, Slack, Teams and the web widget can't drift apart. That collapses the old three-way branch to two, since the LOW and MEDIUM copy differed by nothing but the hedge. The #115 gate is intact: we only promise a follow-up when the worker actually queues one.

3. No ticket-created comment on GitHub. 🎫 Ticket TKT-… created… is gone from issues-opened.ts and discussion-created.ts. The ticket id is internal and the real answer lands in the same thread moments later.

Discord, Slack and Teams keep their acknowledgment — in a chat thread it reads as the bot picking the thread up, rather than a permanent comment on a public issue. Easy to pull those too if uniform is better.

Prompted by CopilotKit/CopilotKit#6167.

Tests

• generator.test.ts — each grounding rule is present and wired into the system prompt
• pipeline.test.ts — no hedge copy at any confidence band from 0.1 to 0.95; existing fix(ai): reword low-confidence disclaimer (escalated to engineering) #115 escalation-gate tests still hold
• formatter.test.ts — no hedge copy on any platform; the three disclaimer constants stay consistent
• both GitHub webhook tests assert no acknowledgment comment is posted

npm test green across all 10 packages · npm run typecheck clean · npm run build clean.

npm run lint fails the same way on clean main — an eslint flat-config migration that predates this branch. Prettier flags these files both before and after the change (335 files repo-wide are unformatted), so formatting is left alone rather than burying a copy change under a reformat.

Follow-up, not in this PR

assessConfidence in generator.ts never reads the response it scores — the parameter is _response, unused. The score is docs relevance plus a source-count bonus, so a guess scores as high as a cited answer. Being fixed next: a groundedness penalty that reads the response against its sources, plus a rubric update to the LLM scorer.

[jerelvelarde]

Approving. All three changes are the right call and the reasoning in the description holds up: the generator is a single stateless call over docs search, so "confident, polished answers" was an instruction it had no way to satisfy honestly. Moving the disclaimer copy into three constants is the fix that keeps GitHub/Discord/Slack/Teams/web from drifting, and collapsing the LOW/MEDIUM branch is correct — they really did differ by nothing but the hedge. The #115 escalation gate survives intact.

Verified locally: CI green, postSystemMessage still has live callers in slack-bot and discord-bot so nothing goes dead, and adapter is still needed for parseInboundEvent in both webhook handlers.

Merging this first since #143 is stacked on this branch.

Notes below — none of them block, and the first one I'd rather see folded into #143 than bounce this PR.

1. The empty-sources fallback contradicts the grounding rules

generator.ts:208 — when sources is empty, the Documentation Context section ends with:

(No relevant documentation found — answer from general CopilotKit knowledge if possible, otherwise say you need to escalate)

That's the exact opposite of the last grounding rule:

Prefer "I don't have enough to answer this — escalating to the team" over a plausible-sounding answer assembled from general framework knowledge.

And it lands in the precise scenario that produced CopilotKit/CopilotKit#6167 — a report the docs don't cover, so search comes back empty or near-empty. The grounding block says it overrides "the personality and formatting rules above," but this line sits below it and is specific to the empty-sources case, so it reads as the more targeted instruction. Worse, with zero sources the "appear verbatim in the Documentation Context" allow-list is empty, so any answer at all violates rule 3.

This line predates the branch, so it's not a regression — but it does mean the guardrail has a hole in the one case it most needs to hold. It's a one-line change and #143 is already in this territory, so that seems like the natural home.

2. Literal … / — in the new comments

issues-opened.ts:60,62 and the matching comment in discussion-created.ts carry … and — as literal text. Those are valid escapes inside the template literal they replaced on main, but in a // comment they don't decode — they render as backslash-u-2-0-2-6. Should just be … and —.

3. index.ts exports with no consumers

GROUNDING_RULES and SYSTEM_PROMPT_PREFIX are added to the package's public exports, but nothing imports them from there — generator.test.ts pulls them from ./generator.js directly. That puts internal prompt copy on the public surface of @copilotkit/outpost/ai, where rewording it becomes an API change. I'd drop both from index.ts and leave them exported from the module for tests. (The three AI_DISCLAIMER* constants have no external consumer either, but those are at least plausibly useful to a web widget rendering its own disclaimer, so no objection to keeping them.)

4. The GROUNDING_RULES assertions are close to tautological

Six of the seven assert that a constant contains substrings of itself, imported from the same module. They'll break on any rewording of the prompt without ever catching a behavioral regression. The one that earns its keep is expect(SYSTEM_PROMPT_PREFIX).toContain(GROUNDING_RULES) — that catches real un-wiring. Not worth churn now, but once #143 lands its scorer, an LLMock-driven test that a bug-confirming response gets withheld would cover the intent far better than substring matching does.

5. Rule 3 may be stricter than intended

Only name identifiers … that appear verbatim in the Documentation Context.

The reporter's own identifiers — their component name, their prop, the version in their lockfile — aren't in the docs context either, so read literally the bot can't echo back the user's own code when asking a clarifying question. Suggest widening to "appear verbatim in the Documentation Context or in the user's own message."

One question

Dropping the GitHub ack means there's now no signal in the thread until the AI response arrives. The degraded path in generate() still returns text so a model failure still posts something — but if the worker job itself dies, the reporter now sees nothing at all, where before they at least knew a ticket existed. Is that covered by internal alerting, or is a hard worker failure silent to the reporter on GitHub now?