Binary parameter handling for GeoDjango

[eduzen]

Description

hey folks, this is my very first PR in the django-commons. I did this because I faced the issue myself in one project that I'm working on. I don't know how I can test this in that project. So I will wait for some of you to tell me how I can do that... but in the meantime this is the fix for the SQL Explain functionality for GeoDjango queries with binary parameters.

Fixes #423

SQL Explain feature throws a 500 error when used with GeoDjango queries containing binary parameters. This affects any query using PostGIS functions like ST_GeomFromEWKB(), ST_Distance_Sphere(), etc.
I'm trying to create a robust binary parameter handling system:

1. Enhanced Parameter Encoding (tracking.py)

• Detects binary data (bytes, bytearray) during parameter logging
• Encodes binary data as base64 with a special marker: {"djdt_binary": "base64_data"}
• Preserves exact binary integrity through JSON serialization

2. Parameter Reconstruction (forms.py)

• Added _reconstruct_params() function to decode stored parameters
• Recursively handles nested structures (lists, dicts) containing binary data
• Restores original binary data from base64 encoding

3. Updated SQL Operations

• Modified select(), explain(), and profile() methods to use parameter reconstruction
• Added helper method _get_query_params() to eliminate code duplication

Checklist:

• I have added the relevant tests for this change.
• I have added an item to the Pending section of docs/changes.rst.

[eduzen]

@tim-schilling I added some commits following what you were pointing out, let me know what do you think!

[tim-schilling]

Rather than the approach you've taken here, you may want to consider going with something like:

import base64
import json

def object_hook(obj):
    for key, value in obj.items():
        if isinstance(value, str) and value.startswith("__djdt_binary__"):
            _, encoded = value.split("__djdt_binary__", maxsplit=1)
            obj[key] = base64.b64decode(encoded)
    return obj


class DebugToolbarJSONDecoder(json.JSONDecoder):
    """Custom JSON decoder that reconstructs binary data during parsing."""
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        if self.object_hook is None:
            self.object_hook = object_hook

This will require you to change the tests. The decoding will only work if there's an actual object being passed in. Most of the tests are passing in a list of values. Using an object/dictionary is more representative of what will actually be used since this is for parameters for a SQL query which is a dictionary.

[eduzen]

Oh, I got it, I did not know about this object_hook possibility.

[tim-schilling]

We should move this logic into a shared utility that way we don't have the magic value "__djdt_binary__" floating around the code-base. Plus it would be clearer how the logic gets paired together.

I'm also curious if we should be using this with the Store logic. If we do, then we could potentially ignore this and handle it within DebugToolbarJSONEncoder and then have DebugToolbarJSONDecoder as above. What do you think?

[eduzen]

Thank you for the feedback @tim-schilling ! I think an encoders.py or moving even everything into utils.py is a more consistent way.
About your second question, what do you mean with using the store logic?

[tim-schilling]

The Store logic converts all the panels data into JSON. So this binary parameter would eventually be converted to JSON to be added to the Store. So if the Store handles binary data properly, it may remove the need for the SQL panel to deal with it itself. Does that make more sense?

[tim-schilling]

I don't think there's a benefit to being this verbose. Try to be more concise such as:

Suggested change

	self.assertEqual(len(reconstructed), 1)
	self.assertEqual(reconstructed[0], binary_data)
	self.assertIsInstance(reconstructed[0], bytes)
	self.assertEqual(reconstructed, [binary_data])

[tim-schilling]

Thank you for continuing to work on this! I have a few directional changes. If you want more specific feedback, I can provide it. I didn't want to be overly prescriptive.

[eduzen]

Thank you for continuing to work on this! I have a few directional changes. If you want more specific feedback, I can provide it. I didn't want to be overly prescriptive.

No please! Tell me more, because I see that you are pointing to a deeper idea that I did not catch yet.

[tim-schilling]

@eduzen oh, I meant I was writing out the solution when I did the last round of review. It seemed like you may be interested in working through the issues to arrive at your own implementation and I didn't want to deprive you of the opportunity. So I should have said was, "if you'd like me to make these edits myself, please let me know."