Skip to content

INTPYTHON-527 Add queryable encryption support #319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions django_mongodb_backend/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Check Django compatibility before other imports which may fail if the
# wrong version of Django is installed.
from .utils import check_django_compatability, parse_uri
from .utils import check_django_compatability, get_auto_encryption_options, parse_uri

check_django_compatability()

Expand All @@ -15,7 +15,7 @@
from .lookups import register_lookups # noqa: E402
from .query import register_nodes # noqa: E402

__all__ = ["parse_uri"]
__all__ = ["get_auto_encryption_options", "parse_uri"]

register_aggregates()
register_checks()
Expand Down
18 changes: 18 additions & 0 deletions django_mongodb_backend/features.py
Original file line number Diff line number Diff line change
Expand Up @@ -577,3 +577,21 @@ def supports_atlas_search(self):
return False
else:
return True

@cached_property
def supports_queryable_encryption(self):
"""
Queryable Encryption is supported if the server is Atlas or Enterprise
and if pymongocrypt is installed.
"""
self.connection.ensure_connection()
client = self.connection.connection.admin
build_info = client.command("buildInfo")
is_enterprise = "enterprise" in build_info.get("modules")
try:
import pymongocrypt # noqa: F401

has_pymongocrypt = True
except ImportError:
has_pymongocrypt = False
Comment on lines +591 to +596
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's inappropriate to check for this package here. Instead the ImportError should be surfaced to the user if they try to use a feature that requires it. I can imagine a couple of ways to do so, but I think this concern should be deferred until the implementation is ironed out.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll move it to get_auto_encryption_options

return is_enterprise and has_pymongocrypt
5 changes: 5 additions & 0 deletions django_mongodb_backend/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,8 @@ def delete(self, *args, **kwargs):

def save(self, *args, **kwargs):
raise NotSupportedError("EmbeddedModels cannot be saved.")


class EncryptedModel(models.Model):
class Meta:
abstract = True
21 changes: 19 additions & 2 deletions django_mongodb_backend/utils.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import copy
import os
import time

import django
Expand All @@ -8,6 +9,7 @@
from django.utils.functional import SimpleLazyObject
from django.utils.text import format_lazy
from django.utils.version import get_version_tuple
from pymongo.encryption_options import AutoEncryptionOpts
from pymongo.uri_parser import parse_uri as pymongo_parse_uri


Expand All @@ -28,7 +30,19 @@ def check_django_compatability():
)


def parse_uri(uri, *, db_name=None, test=None):
def get_auto_encryption_options(crypt_shared_lib_path=None):
key_vault_database_name = "encryption"
key_vault_collection_name = "__keyVault"
key_vault_namespace = f"{key_vault_database_name}.{key_vault_collection_name}"
kms_providers = {"local": {"key": os.urandom(96)}}
return AutoEncryptionOpts(
key_vault_namespace=key_vault_namespace,
kms_providers=kms_providers,
crypt_shared_lib_path=crypt_shared_lib_path,
)
Comment on lines +33 to +42
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's still obvious to me from the design doc to what extent it's appropriate for this library to provide helpers to generate AutoEncryptionOpts.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From the design doc

Take less steps than manual configuration of QE

We can expand that bullet into a section about configuration and it does not have to be a helper like get_auto_encryption_options but the tutorial takes this approach and I like it so far.



def parse_uri(uri, *, auto_encryption_options=None, db_name=None, test=None):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather than options=None than auto_encryption_options=None so that we're not blowing up the signature of parse_uri() with every kwarg of MongoClient. (If it's even in scope to continually expand parse_uri() rather than to guide more advanced users toward a dictionary DATABASES). I think all the changes in this file could be discussed/implemented in a follow up.

Copy link
Collaborator Author

@aclark4life aclark4life Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To enable this feature in PyMongo and start development in Django, an auto_encryption_options object has to be passed to MongoClient in base.py. As someone who opposed db_name in parse_uri I can tell you this is not a casual addition and this implementation is based on your suggestion to merge the options. We could rename, but I'm not convinced yet. I'll shorten it to auto_encryption_opts to match PyMongo for now.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm suggesting:

def parse_uri(uri, *, ... options=None)`:
    ...
    if options:
          options = {**uri.get("options"), options}

Usage:

parse_uri(uri, options={"auto_encryption_options": ...})

Otherwise, we're headed down the road of adding a new keyword to parse_uri() for each MongoClient keyword that the user wants to customize.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! Got it, thanks

"""
Convert the given uri into a dictionary suitable for Django's DATABASES
setting.
Expand All @@ -48,14 +62,17 @@ def parse_uri(uri, *, db_name=None, test=None):
db_name = db_name or uri["database"]
if not db_name:
raise ImproperlyConfigured("You must provide the db_name parameter.")
options = uri.get("options")
if auto_encryption_options:
options = {**uri.get("options"), "auto_encryption_options": auto_encryption_options}
settings_dict = {
"ENGINE": "django_mongodb_backend",
"NAME": db_name,
"HOST": host,
"PORT": port,
"USER": uri.get("username"),
"PASSWORD": uri.get("password"),
"OPTIONS": uri.get("options"),
"OPTIONS": options,
}
if "authSource" not in settings_dict["OPTIONS"] and uri["database"]:
settings_dict["OPTIONS"]["authSource"] = uri["database"]
Expand Down
12 changes: 10 additions & 2 deletions tests/backend_/utils/test_parse_uri.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

import pymongo
from django.core.exceptions import ImproperlyConfigured
from django.test import SimpleTestCase
from django.test import SimpleTestCase, TestCase, skipUnlessDBFeature

from django_mongodb_backend import parse_uri
from django_mongodb_backend import get_auto_encryption_options, parse_uri


class ParseURITests(SimpleTestCase):
Expand Down Expand Up @@ -94,3 +94,11 @@ def test_invalid_credentials(self):
def test_no_scheme(self):
with self.assertRaisesMessage(pymongo.errors.InvalidURI, "Invalid URI scheme"):
parse_uri("cluster0.example.mongodb.net")


# TODO: This can be moved to `test_features` once transaction support is merged.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_features.py would only be for testing the logic of DatabaseFeatures.supports_queryable_encryption.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed

class ParseUriOptionsTests(TestCase):
@skipUnlessDBFeature("supports_queryable_encryption")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we just make installing pymongo[encryption] a required part of running our test suite. In that case, this test wouldn't need a skip since it doesn't interact with the server.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure, but as you said above we shouldn't check imports in features and so this check is for enterprise_or_atlas and we'd skip it on anything that is not that (similar to transaction support checks) and that has nothing to do with encryption libraries.

def test_auto_encryption_options(self):
auto_encryption_options = get_auto_encryption_options(crypt_shared_lib_path="/path/to/lib")
parse_uri("mongodb://localhost/db", auto_encryption_options=auto_encryption_options)
6 changes: 5 additions & 1 deletion tests/model_fields_/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
EmbeddedModelField,
ObjectIdField,
)
from django_mongodb_backend.models import EmbeddedModel
from django_mongodb_backend.models import EmbeddedModel, EncryptedModel


# ObjectIdField
Expand Down Expand Up @@ -136,6 +136,10 @@ class Author(EmbeddedModel):
skills = ArrayField(models.CharField(max_length=100), null=True, blank=True)


class EncryptedData(EncryptedModel):
pass


class Book(models.Model):
name = models.CharField(max_length=100)
author = EmbeddedModelField(Author)
Expand Down
8 changes: 8 additions & 0 deletions tests/model_fields_/test_encrypted_model.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
from django.test import TestCase

from .models import EncryptedData


class ModelTests(TestCase):
def test_save_load(self):
EncryptedData.objects.create()