INTPYTHON-527 Add queryable encryption support

[aclark4life]

Another WIP

• Completed most setup in qe.py
• Completed some setup in Django
  • Refactored get_auto_encryption_options to support ephemeral local KMS and allow crypt_shared_lib_path to be passed in
  • Add check for presence of libmongocrypt to supports_queryable_encryption feature check
• Started thinking about models, fields, migrations and planning an issubclassof check for EncryptedModel to call create_encrypted_collection instead of create_collection but need to pass encrypted_client to ClientEncryption somewhere in Django.

TODO

• Decide on a configurable way to support local KMS so we can add other KMS later using the same configuration framework.
• Decide on how to configure crypt_shared_lib_path. I don't like setting an environment variable for this configuration task so I settled on passing to get_auto_encryption_options in the short term but maybe a Django setting or something else in long term.
• Decide on EncryptedModel and EncryptedField (not EncryptedModelField or EncryptedEmbeddedModelField yet!) implementation to support this:

from django_mongodb_backend.models import EncryptedModel
from django_mongodb_backend.models import EncryptedField
from django import models

class Person(EncryptedModel):
  ssn = EncryptedField(models.CharField)

Question

Lastly, I'm not sure why PyMongo is trying to do explicit here or at least that is what it appears to be doing despite the auto config:

% j q
python qe.py
Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 124, in _wrap_encryption_errors
    yield
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 861, in create_data_key
    self._encryption.create_data_key(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        kms_provider,
        ^^^^^^^^^^^^^
    ...<2 lines>...
        key_material=key_material,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    ),
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/synchronous/explicit_encrypter.py", line 66, in create_data_key
    with self.mongocrypt.data_key_context(kms_provider, opts) as ctx:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/mongocrypt.py", line 284, in data_key_context
    return DataKeyContext(
        self._create_context(),
    ...<3 lines>...
        self.__callback,
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/.venv/lib/python3.13/site-packages/pymongocrypt/mongocrypt.py", line 560, in __init__
    raise ValueError(f"unknown kms_provider: {kms_provider}")
ValueError: unknown kms_provider: None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 762, in create_encrypted_collection
    encrypted_fields["fields"][i]["keyId"] = self.create_data_key(
                                             ~~~~~~~~~~~~~~~~~~~~^
        kms_provider=kms_provider,  # type:ignore[arg-type]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        master_key=master_key,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 858, in create_data_key
    with _wrap_encryption_errors():
         ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/alexclark/.pyenv/versions/3.13.3/lib/python3.13/contextlib.py", line 162, in __exit__
    self.gen.throw(value)
    ~~~~~~~~~~~~~~^^^^^^^
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 130, in _wrap_encryption_errors
    raise EncryptionError(exc) from exc
pymongo.errors.EncryptionError: unknown kms_provider: None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alexclark/Developer/django-mongodb-cli/qe.py", line 34, in <module>
    encrypted_collection = client_encryption.create_encrypted_collection(
        encrypted_database, "encrypted_collection", encrypted_fields
    )
  File "/Users/alexclark/Developer/django-mongodb-cli/src/mongo-python-driver/pymongo/synchronous/encryption.py", line 767, in create_encrypted_collection
    raise EncryptedCollectionError(exc, encrypted_fields) from exc
pymongo.errors.EncryptedCollectionError: unknown kms_provider: None

Seems like I may have missed passing kms_providers somewhere.

I ruled out crypted_shared_lib_path being the cause. If you configure an invalid /path/to/libmongocrypt.so you get:

pymongocrypt.errors.MongoCryptError: A crypt_shared override path was specified [/path/to/libmongocrypt.so], but we failed to open a dynamic library at that location. Load error: [Error while opening candidate for crypt_shared dynamic library [/path/to/libmongocrypt.so]: dlopen(/path/to/libmongocrypt.so, 0x0005): tried: '/path/to/libmongocrypt.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/path/to/libmongocrypt.so' (no such file), '/path/to/libmongocrypt.so' (no such file)]

So… yeah.

[timgraham]

test_features.py would only be for testing the logic of DatabaseFeatures.supports_queryable_encryption.

[aclark4life]

Agreed

[timgraham]

It's inappropriate to check for this package here. Instead the ImportError should be surfaced to the user if they try to use a feature that requires it. I can imagine a couple of ways to do so, but I think this concern should be deferred until the implementation is ironed out.

[aclark4life]

I'll move it to get_auto_encryption_options

[timgraham]

It's still obvious to me from the design doc to what extent it's appropriate for this library to provide helpers to generate AutoEncryptionOpts.

[aclark4life]

From the design doc

Take less steps than manual configuration of QE

We can expand that bullet into a section about configuration and it does not have to be a helper like get_auto_encryption_options but the tutorial takes this approach and I like it so far.

[timgraham]

I'd rather than options=None than auto_encryption_options=None so that we're not blowing up the signature of parse_uri() with every kwarg of MongoClient. (If it's even in scope to continually expand parse_uri() rather than to guide more advanced users toward a dictionary DATABASES). I think all the changes in this file could be discussed/implemented in a follow up.

[aclark4life]

To enable this feature in PyMongo and start development in Django, an auto_encryption_options object has to be passed to MongoClient in base.py. As someone who opposed db_name in parse_uri I can tell you this is not a casual addition and this implementation is based on your suggestion to merge the options. We could rename, but I'm not convinced yet. I'll shorten it to auto_encryption_opts to match PyMongo for now.

[timgraham]

I'm suggesting:

def parse_uri(uri, *, ... options=None)`:
    ...
    if options:
          options = {**uri.get("options"), options}

Usage:

parse_uri(uri, options={"auto_encryption_options": ...})

Otherwise, we're headed down the road of adding a new keyword to parse_uri() for each MongoClient keyword that the user wants to customize.

[aclark4life]

Ah! Got it, thanks

[timgraham]

Probably we just make installing pymongo[encryption] a required part of running our test suite. In that case, this test wouldn't need a skip since it doesn't interact with the server.

[aclark4life]

Not sure, but as you said above we shouldn't check imports in features and so this check is for enterprise_or_atlas and we'd skip it on anything that is not that (similar to transaction support checks) and that has nothing to do with encryption libraries.