Content-Encoding: gzip

[aminroosta]

Is it possible to SET and GET gzip or brotli encoded JSON?

[nicolasff]

Hello,

It is technically possible to set and retrieve gzip-encoded JSON, but this does not mean that Webdis will compress a regular JSON output. So to be clear, it does not mean that you can have Webdis respond with a compressed version of its regular responses.
Instead, the only way to do this is really independent of JSON, and involves storing binary data into Redis with Webdis fetching it as it was stored. The advantage is that you use less memory in Redis, but the downside is that Webdis is then used to simply fetch binary data instead of giving you JSON.

There is an example in the README showing how to store a PNG file in Redis via Webdis. You do get a standard {"SET":[true,"OK"]} when you upload the file, but then you'd need to fetch its binary contents rather than get a JSON response like {"GET":"(some data here)"}.

Here's a demo showing a gzip'd file stored in Redis via Webdis, similar to the PNG example. Let's say we have a JSON file containing some record (really, it could be any kind of file):

$ wc -c data.json 
     408 data.json

$ head -4 data.json
{
    "id": 834719,
    "username": "johndoe",
    "email": "[email protected]",
...

It's 408 bytes raw, so let's gzip it and store it under the key user:johndoe in Redis, via Webdis:

$ curl -v --upload-file <(cat data.json | gzip -) http://127.0.0.1:7379/SET/user:johndoe
> PUT /SET/user:johndoe HTTP/1.1
...
{"SET":[true,"OK"]}

This command uses process substitution, so <(cat data.json | gzip -) is a file (a pipe, really) containing the compressed data resulting from the command. You can always do this in two steps by compressing the file first and then sending the compressed file.

The original file was 408 bytes, but let's see how many bytes are now stored at that key:

$ curl -s http://127.0.0.1:7379/STRLEN/user:johndoe              
{"STRLEN":256} 

So we are storing compressed data. Let's now need to fetch this binary data alone, meaning not wrapped in a {"GET": ...} response. You can choose the exact content-type that Webdis will return, but for this demo we'll just add .bin at the end of the key to get it as binary/octet-stream.

First, let's see how many bytes that returns, and what this data looks like:

$ curl -s http://127.0.0.1:7379/GET/user:johndoe.bin | wc -c
     256
$ curl -s http://127.0.0.1:7379/GET/user:johndoe.bin | xxd | head -3
00000000: 1f8b 0800 a84d d467 0003 4d90 bb4e c430  .....M.g..M..N.0
00000010: 1045 fb7c 4594 1a23 3f12 2d76 4541 45cb  .E.|E..#?.-vEAE.
00000020: d280 9035 1b4f 1ee0 d8c8 7104 d26a ff1d  ...5.O....q..j..
...

Let's now decode this gzip data to get back our original data file:

$ curl -s http://127.0.0.1:7379/GET/user:johndoe.bin | gunzip -     
{
    "id": 834719,
    "username": "johndoe",
    "email": "[email protected]",
...

We can make sure the data is the same by hashing the original data.json file, and comparing that to a hash of this decoded binary data:

$ shasum -a 256 data.json 
cfd2da2f82b1de6e2703efbc21d8a5c145fc7cf7d9939458042e81cc4c0ad383  data.json

$ curl -s http://127.0.0.1:7379/GET/user:johndoe.bin | gunzip - | shasum -a 256
cfd2da2f82b1de6e2703efbc21d8a5c145fc7cf7d9939458042e81cc4c0ad383  -

We did get back our full original file, although we had to (1) send it compressed to Redis via Webdis, and (2) fetch it compressed from Redis via Webdis before decompressing it.

So that's pretty much the only way to do it with Webdis alone at this time, given that it does not support compressing its responses. An alternative is to have a reverse proxy in front of it like nginx, that would forward all requests to Webdis and compress its own responses back to the client if it accepts compressed data. In that setup, you could send regular JSON and get regular JSON back, with compression being only a property of the transport, an attribute of the HTTP request sent via nginx to Webdis.

(Click here for the full contents of `data.json` used in this test)

{
    "id": 834719,
    "username": "johndoe",
    "email": "[email protected]",
    "full_name": "John Doe",
    "date_of_birth": "1992-07-15",
    "created_at": "2023-09-10T14:23:45Z",
    "last_login": "2024-03-12T09:15:33Z",
    "settings": {
        "theme": "dark",
        "language": "en-US",
        "two_factor_enabled": true
    },
    "roles": [
        "user",
        "moderator"
    ]
}

I realize this might not be exactly what you were looking for, but I'm afraid this is the only option for Webdis at this time: sending Webdis the data already compressed, which it will store as provided into Redis, and retrieving it compressed.

Best,
Nicolas

[aminroosta]

Hey @nicolasff given that modern browsers all send the Accept-Encoding: gzip, br, zstd header, I wanted a way to leverage that by storing compressed data in Redis.

I came up with a proof of concept commit: aminroosta@5038b12 to also send the Content-Encoding header.

And it seems to work just fine.

curl -v -s http://127.0.0.1:7379/GET/user:johndoe.json_gzip | gunzip -
*   Trying 127.0.0.1:7379...
* Connected to 127.0.0.1 (127.0.0.1) port 7379
> GET /GET/user:johndoe.json_gzip HTTP/1.1
> Host: 127.0.0.1:7379
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Webdis
< Allow: GET,POST,PUT,OPTIONS
< Access-Control-Allow-Methods: GET,POST,PUT,OPTIONS
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: X-Requested-With, Content-Type, Authorization
< Content-Type: application/json
< ETag: "e905059b26cfdf0862495103f63287e5"
< Content-Encoding: gzip
< Connection: Keep-Alive
< Content-Length: 256
<
{ [256 bytes data]
* Connection #0 to host 127.0.0.1 left intact
{
    "id": 834719,
    "username": "johndoe",
    "email": "[email protected]",
    "full_name": "John Doe",
    "date_of_birth": "1992-07-15",
    "created_at": "2023-09-10T14:23:45Z",
    "last_login": "2024-03-12T09:15:33Z",
    "settings": {
        "theme": "dark",
        "language": "en-US",
        "two_factor_enabled": true
    },
    "roles": [
        "user",
        "moderator"
    ]
}