Add support for robots.txt Content-Signals

[lajawi]

Goals

1. Add support for robots.txt Content-Signals Cloudflare already deploys: https://contentsignals.org/

Non-Goals

1. An overhaul or change of how the dynamic robots.txt can be generated

Background

Current alternative to my knowledge is not using the dynamic robots handler, and instead including a handmade robots.txt file.

Proposal

Extend the type MetadataRoute.Robots/RobotsFile to include another string with the name contentSignal that in turn has three booleans: search, aiInput, and aiTrain.

type RobotsFile = {
    rules: {
        userAgent?: string | string[] | undefined;
        contentSignal?: {
            search?: boolean | undefined;
            aiInput?: boolean | undefined;
            aiTrain?: boolean | undefined;
        };
        allow?: string | string[] | undefined;
        disallow?: string | string[] | undefined;
        crawlDelay?: number | undefined;
    } | Array<{
        userAgent: string | string[];
        contentSignal?: {
            search?: boolean | undefined;
            aiInput?: boolean | undefined;
            aiTrain?: boolean | undefined;
        };
        allow?: string | string[] | undefined;
        disallow?: string | string[] | undefined;
        crawlDelay?: number | undefined;
    }>;
    sitemap?: string | string[] | undefined;
    host?: string | undefined;
};

[mahirliy55]

That’s an interesting idea — Content-Signals could help SEO teams test indexing behavior.
Maybe it could be added as a plugin or middleware for the next-sitemap package?

[lajawi]

I have to add, you can (but don't need to) specify specific pages for each content-signal too.
That would just be an additional variable path?: string | undefined;, however I do not know how this would be handled for multiple.

[lajawi]

I've had a chat with Cloudflare about it to clear up some things.

In short

• Content Signals may vary between directives for one (set of) User Agent(s)
• Content Signals may be missing, depends on what you to set
• You can set the Content Signals for disallowed paths, but it's quite redundant

Below a little example result that includes all of the above rules.

User-Agent: *
Content-Signal: /blog search=yes, ai-train=no, ai-input=yes
Content-Signal: /about search=yes
Content-Signal: /projects search=yes, ai-train=no, ai-input=no
Allow: /
Disallow: /projects

It would make sense that multiple non-root paths of a site may receive the same Content Signals, so as to not have to write duplicate code, I propose the following structure for the Type:

type RobotsFile = {
    rules: {
        userAgent?: string | string[] | undefined;
        contentSignal?: {
            path: string | string [];
            search?: boolean | undefined;
            aiInput?: boolean | undefined;
            aiTrain?: boolean | undefined;
        } | Array<{
            path: string | string [];
            search?: boolean | undefined;
            aiInput?: boolean | undefined;
            aiTrain?: boolean | undefined;
        };
        allow?: string | string[] | undefined;
        disallow?: string | string[] | undefined;
        crawlDelay?: number | undefined;
    } | Array<{
        userAgent: string | string[];
        contentSignal?: {
            path: string | string [];
            search?: boolean | undefined;
            aiInput?: boolean | undefined;
            aiTrain?: boolean | undefined;
        } | Array<{
            path: string | string [];
            search?: boolean | undefined;
            aiInput?: boolean | undefined;
            aiTrain?: boolean | undefined;
        };
        allow?: string | string[] | undefined;
        disallow?: string | string[] | undefined;
        crawlDelay?: number | undefined;
    }>;
    sitemap?: string | string[] | undefined;
    host?: string | undefined;
};