Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove indexes that aren't used and that are large #11623

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Remove indexes that aren't used and that are large
I ran some queries in prod to see what indexes weren't being used sorted by size.
These are the biggest ones without many queries hitting them.

Required updating django-simple-history to get the no_db_index.
ericholscher committed Sep 26, 2024
commit 7820e88b45fef7493f8755fee40f2a0d46e95f71
2 changes: 1 addition & 1 deletion readthedocs/builds/models.py
Original file line number Diff line number Diff line change
@@ -101,6 +101,7 @@ class Version(TimeStampedModel):
max_length=20,
choices=VERSION_TYPES,
default="unknown",
db_index=True,
)
# used by the vcs backend

@@ -858,7 +859,6 @@ class Meta:
# Query: ``version.builds.filter(success=True, state=BUILD_STATE_FINISHED)``
["version", "state", "date", "success"],
["version", "state", "type"],
["date", "id"],
]
indexes = [
models.Index(fields=["project", "date"]),
4 changes: 1 addition & 3 deletions readthedocs/core/history.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import structlog
from functools import partial

import structlog
from django import forms
from django.db import models
from django.utils.translation import gettext_lazy as _
@@ -66,13 +66,11 @@ class ExtraFieldsHistoricalModel(models.Model):
_("ID"),
blank=True,
null=True,
db_index=True,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can use the help from @stsewd here. I don't know where these extra fields are used, and if the usage is intense.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think we use this models directly in our application.

)
extra_history_user_username = models.CharField(
_("username"),
max_length=150,
null=True,
db_index=True,
)
extra_history_ip = models.CharField(
_("IP address"),
2 changes: 1 addition & 1 deletion readthedocs/oauth/models.py
Original file line number Diff line number Diff line change
@@ -169,7 +169,7 @@ class RemoteRepository(TimeStampedModel):
blank=True,
)
# VCS provider repository id
remote_id = models.CharField(db_index=True, max_length=128)
remote_id = models.CharField(max_length=128)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm, this called my attention. I think we perform a few queries based on remote_id in .com because of SSO. Have you checked this index there?

vcs_provider = models.CharField(
_("VCS provider"), choices=VCS_PROVIDER_CHOICES, max_length=32
)
5 changes: 3 additions & 2 deletions readthedocs/projects/models.py
Original file line number Diff line number Diff line change
@@ -258,7 +258,6 @@ class Project(models.Model):
max_length=255,
validators=[validate_repository_url],
help_text=_("Git repository URL"),
db_index=True,
)

# NOTE: this field is going to be completely removed soon.
@@ -535,7 +534,9 @@ class Project(models.Model):
)

tags = TaggableManager(blank=True, ordering=["name"])
history = ExtraHistoricalRecords()
history = ExtraHistoricalRecords(
no_db_index=["repo", "slug", "remote_repository_id", "main_language_project_id"]
)
objects = ProjectQuerySet.as_manager()

remote_repository = models.ForeignKey(
4 changes: 1 addition & 3 deletions requirements/pip.in
Original file line number Diff line number Diff line change
@@ -10,9 +10,7 @@ django-autoslug
# https://www.psycopg.org/psycopg3/docs/basic/install.html
psycopg[binary,pool]

# 3.1.0 includes changes that require running `makemigrations`
# https://django-simple-history.readthedocs.io/en/latest/#changes
django-simple-history==3.0.0
django-simple-history

djangorestframework
djangorestframework-api-key
13 changes: 7 additions & 6 deletions requirements/pip.txt
Original file line number Diff line number Diff line change
@@ -14,9 +14,9 @@ async-timeout==4.0.3
# via redis
billiard==3.6.4.0
# via celery
boto3==1.35.24
boto3==1.35.26
# via django-storages
botocore==1.35.24
botocore==1.35.26
# via
# boto3
# s3transfer
@@ -79,13 +79,14 @@ django==4.2.16
# django-formtools
# django-polymorphic
# django-safemigrate
# django-simple-history
# django-storages
# django-structlog
# django-taggit
# django-timezone-field
# djangorestframework
# jsonfield
django-allauth[mfa,saml,socialaccount]==64.2.1
django-allauth[mfa,saml,socialaccount]==65.0.1
# via -r requirements/pip.in
django-annoying==0.10.7
# via -r requirements/pip.in
@@ -121,7 +122,7 @@ django-polymorphic==3.1.0
# via -r requirements/pip.in
django-safemigrate==4.3
# via -r requirements/pip.in
django-simple-history==3.0.0
django-simple-history==3.7.0
# via -r requirements/pip.in
django-storages[boto3]==1.14.3
# via -r requirements/pip.in
@@ -208,7 +209,7 @@ packaging==24.1
# gunicorn
platformdirs==4.3.6
# via virtualenv
prompt-toolkit==3.0.47
prompt-toolkit==3.0.48
# via click-repl
psycopg[binary,pool]==3.2.2
# via -r requirements/pip.in
@@ -303,7 +304,7 @@ typing-extensions==4.12.2
# psycopg
# psycopg-pool
# qrcode
tzdata==2024.1
tzdata==2024.2
# via
# -r requirements/pip.in
# django-celery-beat