Implement hidden state for versions

[stsewd]

Ref #6194 and #5321

This doesn't remove the protected privacy level yet (we can do that after this change is released).

Changes:

• Add hidden field
• Change search and footer to respect the hidden state
• Add the hidden field to API v3
• Migrate all protected versions to be hidden

[stsewd]

Should we filter these from the sitemap as well?

[ericholscher]

This looks like a good start. I feel like we need a bit more explanation for users before we really ship this, since it can be a bit confusing I think.

[stsewd]

I feel like we need a bit more explanation for users before we really ship this, since it can be a bit confusing I think.

Do you mean like improving the UX or communicating the users with a blog post/email?

[ericholscher]

Do you mean like improving the UX or communicating the users with a blog post/email?

Improving the on-site copy and docs. Perhaps some general uses cases and examples, and a Guide (eg. how do I hide a version and keep docs online?).

I think the main thing is "I'm a user and I see this "hidden" option, how do I understand what it actually does".

[stsewd]

@ericholscher I expanded the docs and wrote a small guide https://docs--6792.org.readthedocs.build/en/6792/guides/hiding-a-version.html

And used crispy forms to group the elements on the UI. Let me know what you think.

[stsewd]

Oh, and one thing. This migration is adding a non-nullable field, so when deploying users aren't going to be able to create new versions during the deploy.

We can avoid the downtime by setting the new field as nullable and:

• run the migration that adds the new field
• deploy the code
• run the data migration to migrate all protected versions to hidden=True and update all versions with hidden=None to hidden=False
• Make the field non-nullable in the next deploy

I also saw this package that solves the problem without doing all those steps https://github.com/3YOURMIND/django-add-default-value but I haven't tested it

[ericholscher]

We can avoid the downtime by setting the new field as nullable and:

I think this is a good idea 👍

[ericholscher]

Do we actually want this? It doesn't make hidden very hidden, I think.

[humitos]

Yeah. However, if we hide the versions from there, you can't un-hide them. Although, we could show hidden versions in that list if you are an admin of the project.

[stsewd]

we could show hidden versions in that list if you are an admin of the project.

Hmm, yeah. Not sure, I mean, users will only be able to get there by search results on google or old links, not from the project. I'll create an issue to discuss that.

[stsewd]

#6935

[ericholscher]

I'd like to not break versions during deploy, so merge after we edit the migration

[humitos]

Looks good.

I left some important questions about privacy levels on search and the migration file.

[stsewd]

Would the appear in the sitemap.xml, robots.txt? I may worth to mention these things here as well.

I think there was a discussion on slack around listing this in robots.txt, currently we only put this as default

readthedocs.org/readthedocs/proxito/views/serve.py

Lines 380 to 383 in bc30c5e

	return HttpResponse(
	'User-agent: *\nAllow: /\nSitemap: {}\n'.format(sitemap_url),
	content_type='text/plain',
	)

About the sitemap, that still needs an answer #6792 (comment). Currently, we are listing those in the sitemap. /cc @ericholscher

[humitos]

I think they should not appear in the sitemap.xml and should be Disallow in robots.txt. IMO that's aligned with the meaning of hidden: you don't want people to find these versions.

[ericholscher]

Yea, definitely they be Disallowed in the robots.txt. I don't feel strongly about sitemap. Is there a use for the sitemap.xml that isn't search indexing? Hidden isn't private, so I think showing them in the sitemap and just asking the crawlers not to index them seems fine.

[stsewd]

Active versions that are hidden are listed in the robots.txt file now.

[ericholscher]

This looks 💯 to me. Excited to ship it!

[humitos]

👍