Generate general sitemap.xml for projects #5122

humitos · 2019-01-16T21:29:31Z

This PR makes Read the Docs to generate a general (non specific per project) sitemap.xml served at the root of the project /sitemap.xml based on discussions from #557

I think it would be good to split this in two phases:

Generate a general sitemap.xml for all the project without option to customize it (this PR as is)
Check if the project is already generating a sitemap.xml and instead of generating a general one, generate a specific one for this project using sitemapindex

Example with a toy project locally,

$ http http://fastcgi-for-net.dev.readthedocs.io:8000/sitemap.xml
HTTP/1.0 200 OK
Content-Language: en
Content-Length: 1713
Content-Type: application/xml
Date: Wed, 16 Jan 2019 21:15:44 GMT
Server: WSGIServer/0.2 CPython/3.6.6
Vary: Accept-Language, Cookie

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:xhtml="http://www.w3.org/1999/xhtml">
  
  <url>
    <loc>http://fastcgi-for-net.dev.readthedocs.io:8000/bn/latest/</loc>
    
    <xhtml:link
        rel="alternate"
        hreflang="en"
        href="http://fastcgi-for-net.dev.readthedocs.io:8000/en/latest/"/>
    
    <xhtml:link
        rel="alternate"
        hreflang="bn"
        href="http://fastcgi-for-net.dev.readthedocs.io:8000/bn/latest/"/>
    
    <lastmod>2019-01-16T21:13:53.325602+00:00</lastmod>
    <changefreq>daily</changefreq>
    <priority>1</priority>
  </url>
  
  <url>
    <loc>http://fastcgi-for-net.dev.readthedocs.io:8000/bn/stable/</loc>
    
    <xhtml:link
        rel="alternate"
        hreflang="en"
        href="http://fastcgi-for-net.dev.readthedocs.io:8000/en/stable/"/>
    
    <xhtml:link
        rel="alternate"
        hreflang="bn"
        href="http://fastcgi-for-net.dev.readthedocs.io:8000/bn/stable/"/>
    
    <lastmod>2019-01-16T10:54:16.792524+00:00</lastmod>
    <changefreq>weekly</changefreq>
    <priority>0.9</priority>
  </url>
  
  <url>
    <loc>http://fastcgi-for-net.dev.readthedocs.io:8000/bn/make-docs-dir-a-list/</loc>
    
    <xhtml:link
        rel="alternate"
        hreflang="en"
        href="http://fastcgi-for-net.dev.readthedocs.io:8000/en/make-docs-dir-a-list/"/>
    
    <xhtml:link
        rel="alternate"
        hreflang="bn"
        href="http://fastcgi-for-net.dev.readthedocs.io:8000/bn/make-docs-dir-a-list/"/>
    
    <lastmod>2019-01-16T11:03:10.324661+00:00</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
  
</urlset>
$

humitos · 2019-01-16T22:09:46Z

readthedocs/core/views/serve.py

+            yield c
+
+        while True:
+            yield 'monthly'


I'm using monthly because I think never is too aggressive. If the tag is removed and a branch is created with the same name, we will want bots to revisit this.

NOTE: maybe this should be a comment in the code itself.

humitos · 2019-01-16T22:11:20Z

readthedocs/core/views/serve.py

+    for version, priority, changefreq in zip(
+            sorted_versions, priorities_generator(), changefreqs_generator()):
+        element = {
+            'loc': version.get_subdomain_url(),


The URL should be properly escaped: https://www.sitemaps.org/protocol.html#escaping

I'm not sure if there is something we need to do here, actually.

humitos · 2019-01-17T14:05:47Z

readthedocs/core/views/serve.py

+        iteration. After 0.1 is reached, it will keep returning 0.1.
+        """
+        priorities = [1, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2]
+        yield from itertools.chain(priorities, itertools.repeat(0.1))


yield from is not Python2 syntax compatible :(

Typo on line 288: change change
and line 293: this one i not

humitos · 2019-01-17T14:33:42Z

This is ready to be merged. Although, it uses yield from which is syntax not compatible with python2. I think we don't want to re-write it to make it compatible with it since we are deprecating it: #4543

(we should probably already remove it from travis and tox)

davidfischer · 2019-01-31T20:28:14Z

readthedocs/core/views/serve.py

+
+
+@map_project_slug
+# TODO: make this cache dependent on the project's slug


The docs for cache_page say it is "is keyed off of the URL". Looking at the code, it does look to me like they mean the fully qualified URL including the host so I think we're ok.

When I read that I wasn't sure if by URL it meant the path or the full URL.

Digging a little into the code I found that this line is the one that generates the cache key:

https://github.com/django/django/blob/b9cf764be62e77b4777b3a75ec256f6209a57671/django/utils/cache.py#L314

Which returns the absolute URL as you said. Thanks. We are good!

https://docs.djangoproject.com/en/1.11/ref/request-response/#django.http.HttpRequest.build_absolute_uri

davidfischer · 2019-01-31T20:30:47Z

readthedocs/core/views/serve.py

+    context = {
+        'versions': versions,
+    }
+    return render(request, 'sitemap.xml', context, content_type='application/xml')


Any reason you went with a template instead of Django's builtin sitemap framework? https://docs.djangoproject.com/en/1.11/ref/contrib/sitemaps/

The framework might have some advantages in case we ever get so large we need to break up sitemaps into multiple files.

Ignorance.

That said and after reading the documentation, I'm not sure how to:

get the project from inside the Sitemap class (I think it doesn't have access to the request object --where we could check for .slug property, for example)

generate different locations for translations that are not based on LANGUAGE variable but on project.translations

Sitemap objects look pretty clean and clear but I'd need some help on those two things to be able to make it work using the framework.

davidfischer · 2019-01-31T20:36:36Z

readthedocs/core/views/serve.py

+
+    versions = []
+    for version, priority, changefreq in zip(
+            sorted_versions, priorities_generator(), changefreqs_generator()):


If I'm reading this right:

latest will always have priority=1 and changefreq=daily

stable will always have priority=0.9 and changefreq=weekly

other versions will have decreasing priorities and changefreq=monthly

Is that right?

Wouldn't it be better to just guess at a priority and changefrequency based on the last build date? If there is no last build date (version was never built), we don't include the version.

You are reading it right, yes.

Wouldn't it be better to just guess at a priority and changefrequency based on the last build date?

Our idea behind this was that you (as author of the project) want to point your readers to the latest version published first. That's why the priority works like that: latest, stable, v1.5, v1.4, etc in that order... I think that build date is not associated with priority.

Regarding changefreq, I have a similar opinion: I expect last versions to change more frequently than v0.1.

I wouldn't complicate the logic for this.

If there is no last build date (version was never built), we don't include the version.

I'm including only active versions, as we do on the flyout also. This could change once more states be implemented: #4001 (comment)

humitos · 2019-02-04T11:55:33Z

I added small docs for sitemap to at least communicate that we are doing this automatically, without deep too much into its implementation. Once sitemap index is implemented, we can use this page to extend for that feature.

davidfischer

This change looks good to me.

I will monitor our organic traffic over the next month and make sure there isn't a dip from this change. Based on my understanding of sitemaps, I this should be positive but SEO is a bit of the dark arts.

I know this is a first step but here are some improvements we can do in future iterations:

Have a sub-sitemap per version. Our root sitemap (slug.readthedocs.io/sitemap.xml) could point to version specific sitemaps (slug.readthedocs.io/en/1.x/sitemap.xml) that have entries for each HTML file.
We can support user submitted sitemaps. If the user has a sitemap.xml file in their build output, our root sitemap could point to it or maybe it replaces the root sitemap.
Use the Django sitemap features. I think this is slightly better than using an XML template if possible because there are a few intricacies for sitemaps (max 50k entries per file, etc.) that we won't hit with this implementation but we might if we expand it.

jdillard · 2019-02-09T18:55:43Z

I was looking through the changes and it doesn't seem the sitemapindex is added to the robots.txt file, which would likely make it easier for the search engines to find.

If you think that is worth implementing here is an example of what I am talking about: https://github.com/jdillard/sphinx-sitemap#getting-the-most-out-of-the-sitemap

humitos · 2019-02-09T20:15:39Z

doesn't seem the sitemapindex is added to the robots.txt file, which would likely make it easier for the search engines to find.

I think this is a good addition and should be easy to add it to the default robots.txt returned by Read the Docs at https://github.com/rtfd/readthedocs.org/blob/799480827f20f04ded7239a6307853c721de39fa/readthedocs/core/views/serve.py#L312

See https://github.com/django/django/blob/b9cf764be62e77b4777b3a75ec256f6209a57671/django/utils/cache.py#L314 and https://docs.djangoproject.com/en/1.11/ref/request-response/#django.http.HttpRequest.build_absolute_uri

ericholscher

Looks simple enough. I could see needing to play with the priorities or something over time, but this is definitely better than no sitemap (hopefully :)

ericholscher · 2019-02-14T13:39:04Z

docs/sitemaps.rst

@@ -0,0 +1,19 @@
+Sitemaps


Is this linked from anywhere? Should be in an toctree somewhere.

Yes. I wanted to link it under Feature Documentation but I forgot to do it.

I will move this file under docs/features/sitemaps.rst and will be linked automatically on that section.

ericholscher · 2019-02-14T13:42:57Z

readthedocs/core/views/serve.py

+        raise Http404
+
+    sorted_versions = sort_version_aware(
+        project.versions.filter(


any reason not to use public( on the queryset here?

I believe it also defaults to only active projects, but can pass only_active=True also

No specific reason. I just changed to Version.objects.public(project=project, only_active=True)

humitos · 2019-02-18T11:36:04Z

I just pushed the changes suggested on feedback. I will merge this PR once tests pass.

humitos · 2019-02-18T11:36:51Z

Mmm... It seems that I can't merge because as I sent new changes I need a new approval now:

At least 1 approving review is required by reviewers with write access.

humitos added Needed: tests Tests are required PR: work in progress Pull request is not ready for full review labels Jan 16, 2019

humitos requested a review from a team January 16, 2019 21:29

humitos force-pushed the humitos/sitemap-xml branch from 5de800d to 41b6c16 Compare January 16, 2019 21:41

humitos mentioned this pull request Jan 16, 2019

Generate a sitemap.xml #557

Closed

humitos force-pushed the humitos/sitemap-xml branch from 41b6c16 to 0ed9952 Compare January 16, 2019 21:55

humitos mentioned this pull request Jan 16, 2019

WIP: Add multilingual support for issue #9 jdillard/sphinx-sitemap#15

Merged

humitos commented Jan 16, 2019

View reviewed changes

humitos force-pushed the humitos/sitemap-xml branch from 88beea8 to 9c05020 Compare January 17, 2019 11:21

humitos commented Jan 17, 2019

View reviewed changes

humitos removed Needed: tests Tests are required PR: work in progress Pull request is not ready for full review labels Jan 17, 2019

humitos added Needed: documentation Documentation is required PR: work in progress Pull request is not ready for full review labels Jan 20, 2019

agjohnson added this to the 3.1 milestone Jan 22, 2019

agjohnson added the Feature New feature label Jan 25, 2019

davidfischer reviewed Jan 31, 2019

View reviewed changes

humitos force-pushed the humitos/sitemap-xml branch from 07f0867 to 9670c85 Compare February 4, 2019 11:29

humitos removed PR: work in progress Pull request is not ready for full review Needed: documentation Documentation is required labels Feb 4, 2019

humitos force-pushed the humitos/sitemap-xml branch from 03043d6 to 3de35d0 Compare February 4, 2019 12:29

humitos requested a review from a team February 4, 2019 12:33

davidfischer previously approved these changes Feb 6, 2019

View reviewed changes

humitos dismissed davidfischer’s stale review via ab64878 February 12, 2019 09:05

Generate general sitemap.xml for projects

7edeeeb

humitos added 10 commits February 12, 2019 10:07

Handle no yet built versions (no lastmod attribute)

f460539

Test case for public projects fixed

fdb2509

Linting

9f2233e

Typo

51be556

@cache_page does consider the full URL (including subdomain)

b69f4ea

See https://github.com/django/django/blob/b9cf764be62e77b4777b3a75ec256f6209a57671/django/utils/cache.py#L314 and https://docs.djangoproject.com/en/1.11/ref/request-response/#django.http.HttpRequest.build_absolute_uri

Fix regex for URLs

fd5ddbf

Documentation for sitemaps

8cb488e

Fix merge conflict

8a42529

Return sitemap on robots.txt file

a40b493

Remove invalid test case

6b3cf9f

humitos force-pushed the humitos/sitemap-xml branch from ab64878 to 6b3cf9f Compare February 12, 2019 09:10

humitos requested a review from a team February 12, 2019 09:12

Lint

21b0015

humitos force-pushed the humitos/sitemap-xml branch from 1bb65e3 to 21b0015 Compare February 12, 2019 09:38

humitos added 2 commits February 12, 2019 10:40

Fix robots test case

2f41620

More fixes on tests

2a7f6bd

ericholscher previously approved these changes Feb 14, 2019

View reviewed changes

davidfischer previously approved these changes Feb 14, 2019

View reviewed changes

humitos added 3 commits February 18, 2019 12:27

Update common submodule

acfbfc4

Move sitemap docs under features section

a0efd4a

Use Version.objects.public helper

6482bac

humitos dismissed stale reviews from davidfischer and ericholscher via 6482bac February 18, 2019 11:35

humitos requested a review from ericholscher February 18, 2019 11:53

ericholscher approved these changes Feb 19, 2019

View reviewed changes

humitos merged commit f1c15d4 into master Feb 19, 2019

delete-merged-branch bot deleted the humitos/sitemap-xml branch February 19, 2019 21:01

jarrodmillman mentioned this pull request Nov 6, 2019

improve Search Engine results for documentation networkx/networkx#2730

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate general sitemap.xml for projects #5122

Generate general sitemap.xml for projects #5122

humitos commented Jan 16, 2019

humitos Jan 16, 2019

humitos Jan 16, 2019

humitos Feb 4, 2019

humitos Jan 17, 2019

jdillard Jan 18, 2019

humitos Jan 20, 2019

humitos commented Jan 17, 2019

davidfischer Jan 31, 2019

humitos Feb 4, 2019 •

edited

Loading

davidfischer Jan 31, 2019

humitos Feb 4, 2019

davidfischer Jan 31, 2019

humitos Feb 4, 2019

humitos commented Feb 4, 2019

davidfischer left a comment

jdillard commented Feb 9, 2019

humitos commented Feb 9, 2019

ericholscher left a comment

ericholscher Feb 14, 2019

humitos Feb 18, 2019

ericholscher Feb 14, 2019

ericholscher Feb 14, 2019

humitos Feb 18, 2019

humitos commented Feb 18, 2019

humitos commented Feb 18, 2019



		@map_project_slug
		# TODO: make this cache dependent on the project's slug

Generate general sitemap.xml for projects #5122

Generate general sitemap.xml for projects #5122

Conversation

humitos commented Jan 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Jan 17, 2019

Choose a reason for hiding this comment

humitos Feb 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Feb 4, 2019

davidfischer left a comment

Choose a reason for hiding this comment

jdillard commented Feb 9, 2019

humitos commented Feb 9, 2019

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Feb 18, 2019

humitos commented Feb 18, 2019

humitos Feb 4, 2019 •

edited

Loading