Add meta tags "noindex, nofollow" for hidden version

[aanm]

What's the problem this feature will solve?

We are trying to prevent search engines from indexing older pages of our documentation. Although we want them to be reachable, by marking them as "active" and "hidden", we don't want them to be indexed.

Describe the solution you'd like

For each "hidden" version, readthedocs should add a <meta name="robots" content="noindex, nofollow" />

Alternative solutions

We have followed https://docs.readthedocs.io/en/latest/faq.html#how-can-i-avoid-search-results-having-a-deprecated-version-of-my-docs but this is not enough as it's not possible to use robots.txt to prevent indexing.

Additional context

We could add robots meta tags by adding to our Documentation/conf.py:

html_context = {
    ...
    'meta_robots': '<meta name="robots" content="noindex, nofollow" />',
}

but we would like to avoid doing this as it requires us from pushing commits to all branches that we have currently marked as hidden.

[humitos]

this is not enough as it's not possible to use robots.txt to prevent indexing.

Why this is not enough?

Read the Docs generates a robots.txt automatically with the following:

Read the Docs automatically generates one for you with a configuration that works for most projects. By default, the automatically created robots.txt:

• Hides versions which are set to Hidden from being indexed.
• Allows indexing of all other versions.

(from https://docs.readthedocs.io/en/latest/reference/robots.html)

For each "hidden" version, readthedocs should add a <meta name="robots" content="noindex, nofollow" />

We are moving away from automatically add things on behalf of the user for Sphinx projects and we are making our platform more generic, so more doctools can be used. You could use the API to query the active versions at build time via https://readthedocs.org/api/v3/projects/<slug>/versions/?active=true and inject the meta HTML tag as you mentioned if the version is hidden: true.

[aanm]

this is not enough as it's not possible to use robots.txt to prevent indexing.

Why this is not enough?

Read the Docs generates a robots.txt automatically with the following:

Read the Docs automatically generates one for you with a configuration that works for most projects. By default, the automatically created robots.txt:

• Hides versions which are set to Hidden from being indexed.
• Allows indexing of all other versions.

(from https://docs.readthedocs.io/en/latest/reference/robots.html)

For each "hidden" version, readthedocs should add a <meta name="robots" content="noindex, nofollow" />

We are moving away from automatically add things on behalf of the user for Sphinx projects and we are making our platform more generic, so more doctools can be used. You could use the API to query the active versions at build time via https://readthedocs.org/api/v3/projects/<slug>/versions/?active=true and inject the meta HTML tag as you mentioned if the version is hidden: true.

@humitos thank you for the reply.

You could use the API to query the active versions

Can you tell me from where would this API query be made? On which step of the docs building process?

[humitos]

Can you tell me from where would this API query be made? On which step of the docs building process?

I think you can call this API from your Sphinx configuration file (conf.py), parse its response and then use something like you already mentioned before with the html_context

[astrojuanlu]

this is not enough as it's not possible to use robots.txt to prevent indexing.

Why this is not enough?

Learned this the hard way:

https://support.google.com/webmasters/answer/7440203#indexed_though_blocked_by_robots_txt

Indexed, though blocked by robots.txt

The page was indexed despite being blocked by your website's robots.txt file. Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page. Google won't request and crawl the page, but we can still index it, using the information from the page that links to your blocked page. Because of the robots.txt rule, any snippet shown in Google Search results for the page will probably be very limited.

Next steps:

• If you do want to block this page from Google Search, robots.txt is not the correct mechanism to avoid being indexed. To avoid being indexed, remove the robots.txt block and use 'noindex'.

[astrojuanlu]

We'll soon have to figure out how to do this ourselves so if anybody has any other pointers beyond #10648 (comment), happy to get some advice...

[humitos]

@astrojuanlu you can use https://docs.readthedocs.io/en/stable/api/v3.html#version-detail to get the actual version and check for the hidden property and append the meta HTML tag in that case:

# Pseudo code
#

# https://docs.readthedocs.io/en/stable/reference/environment-variables.html
project_slug = os.environ.get("READTHEDOCS_PROJECT")
version_slug = os.environ.get("READTHEDOCS_VERSION")

# https://docs.readthedocs.io/en/stable/api/v3.html#version-detail
url = f"/api/v3/projects/{project_slug}/versions/{version_slug}/"
version = requests.get(url).json()  # Need to use an API token here, probably
if version["hidden"]:
  html_context.update({
    'hidden_version': True,
})

Then in your theme template:

{% block extrahead %}

{{ super() }}
{% if hidden_version %}
<meta name="robots" content="noindex, nofollow" />
{% endif %}

{% endblock %}

Note this code is just an example and you may need to adapt it to your needs. However, it should be pretty close to what you need. Let me know if this works for you.

[humitos]

We are moving away from automatically add things on behalf of the user for Sphinx projects and we are making our platform more generic, so more doctools can be used

Thinking a little more about this. It could be implemented at a CF Worker level, in case we really want to build this feature in a doctool agnostic way, tho.

[noklam]

https://www.stevenhicks.me/blog/2023/11/how-to-deindex-your-docs-from-google/

I stumble on this post. I don't really know enough about SEO to comment. The problem that the articles describe is almost identical to ours, it suggest to use canonical instead of noindex

[astrojuanlu]

What about an HTTP header? https://developers.google.com/search/docs/crawling-indexing/block-indexing#http-response-header, https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag#xrobotstag

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)

At least this doesn't require retrofitting changes to HTML that has already been generated with a static theme.