Add guardrail to ensure Googlebot receives indexable HTML on canonical URLs

[Noctivoro]

Problem

Dualmark's Markdown twins should be noindex, but canonical HTML URLs must remain indexable for search crawlers.

A downstream Cloudflare Workers integration drifted into treating Googlebot as an AI bot for runtime negotiation. The result was that a normal canonical URL returned:

Content-Type: text/markdown; charset=utf-8
X-Robots-Tag: noindex

to Googlebot. Search Console then reported the homepage as Crawled - currently not indexed even though browser requests received HTML. This is easy to miss because the Markdown twin behavior is correct for explicit .md / Accept: text/markdown requests, but dangerous when applied to Googlebot or other search/index crawlers on the canonical URL.

Why this can happen

Dualmark integrations and templates encourage runtime negotiation for bot user agents. When AI-assisted edits expand or copy bot lists, a search crawler such as Googlebot can accidentally be included in the Markdown/noindex path. Existing checks can pass if they only verify that bot UAs receive valid Markdown, rather than also verifying that search crawlers receive indexable canonical HTML.

Suggested guardrail

Add a built-in verification check that fetches representative canonical routes as Googlebot and asserts:

• status is 200
• Content-Type includes text/html
• response does not include X-Robots-Tag: noindex
• response still includes Link: <...>; rel="alternate"; type="text/markdown"

And separately verify:

• direct .md twins return text/markdown + X-Robots-Tag: noindex
• explicit Accept: text/markdown negotiation returns text/markdown + X-Robots-Tag: noindex

Possible API/design improvement

Consider separating crawler classes instead of a single broad “AI bot” runtime path:

• canonical/search crawlers: Googlebot, Bingbot, etc. should always get HTML
• explicit Markdown clients: Accept: text/markdown or direct .md twins
• optional AI training/user-action bots: only receive Markdown if explicitly configured

At minimum, Googlebot should be denied from any automatic Markdown/noindex canonical negotiation path, and templates/tests should fail if it receives Markdown on canonical URLs.

Environment where seen

Downstream integration using @dualmark/cloudflare/Dualmark-style Cloudflare Worker negotiation with static Markdown twins. The issue was caused by integration drift, not necessarily the current upstream default bot list. The ask is for upstream verification/templates to prevent this class of regression.