AI bot UA overrides explicit `Accept: text/html` (spec section 5 violation)

[abhay-codes07]

Summary

Every edge and framework adapter serves the markdown twin to any known AI bot User-Agent, even when that request carries an explicit Accept: text/html. This contradicts the AEO spec, which says UA-based markdown negotiation must not override an explicit Accept.

From spec/content-negotiation.md section 5 (Implicit Markdown for AI Agents):

MUST still respect explicit Accept: text/html from a bot UA (i.e. UA detection MUST NOT override an explicit Accept)

Where it happens

All adapters compute the decision as bot.isBot || fmt === "markdown":

• packages/cloudflare/src/worker.ts
• packages/deno/src/handler.ts
• packages/fastly/src/handler.ts
• packages/netlify/src/worker.ts
• packages/nextjs/src/middleware.ts
• packages/sveltekit/src/handle.ts
• packages/vercel/src/middleware.ts
• packages/nuxt/src/module.ts and packages/nuxt/src/runtime/server/endpoints/middleware.ts

Because bot.isBot short-circuits, the Accept header is ignored entirely for bot UAs.

Reproduction

A request with a bot UA and an explicit HTML preference:

GET /blog/post HTTP/1.1
User-Agent: GPTBot/1.0
Accept: text/html

Actual: 200 with Content-Type: text/markdown; charset=utf-8 and X-Robots-Tag: noindex on the canonical URL.

Expected (per spec section 5): text/html, because the client explicitly asked for HTML.

Confirmed against the core negotiation logic directly:

// current adapter rule
const serveMarkdown = bot.isBot || negotiateFormat("text/html") === "markdown";
// => true  (serves markdown, wrong)

Why it matters

This is the same class of problem raised in #67: a crawler that sends a bot UA together with Accept: text/html is handed a noindex markdown body on its canonical URL. The existing conformance checks do not catch it because negotiation.botUa only tests a bot UA with Accept: */*, never a bot UA with an explicit Accept: text/html.

Proposed fix

Keep the UA-based markdown extension, but defer to RFC 7231 negotiation when the client states a concrete format preference. A bot still gets markdown for Accept: */*, no Accept, or Accept: text/markdown, and stays on HTML when it explicitly sends Accept: text/html (or any Accept where HTML strictly outranks markdown).

I have a fix ready that adds a shouldServeMarkdown(accept, isBot) helper to @dualmark/core and wires it through all adapters, with tests. PR incoming.