Skip to content

Conversation

sauravkumarrr
Copy link
Contributor

Problem

When Redis keys have very low TTL values, they may expire after keyspace notifications are sent but before the source connector can process them. In such cases, the connector receives an event indicating that something happened with the key, but when it attempts to retrieve the key data, it returns null since the key has already expired. This caused the processing flow to break because the type field was null, leading to NullPointerException or unexpected behavior while schema conversion since its of required string type field.

Solution

To handle expired keys gracefully, I've made the a schema of type field OPTIONAL_STRING_SCHEMA. When the type is null (indicating an expired key), the code now not throw any error and will set the type as null. This allows the connector to continue processing these records as expired/non-existent keys rather than failing.

Examples

Normal ingested record (key exists and has data):

{
  "key": "redis_keys:User_2",
  "ttl": {
    "long": 1756401363994
  },
  "type": "ReJSON-RL",
  "hash": null,
  "string": null,
  "json": {
    "string": "{\"registertime\":1500765952457,\"userid\":\"User_2\",\"regionid\":\"Region_8\",\"gender\":\"FEMALE\"}"
  },
  "list": null,
  "set": null,
  "zset": null
}

Expired key ingested record (key expired, type set to null):

{
  "key": "redis_keys:User_5",
  "ttl": {
    "long": -2
  },
  "type": null,
  "hash": null,
  "string": null,
  "json": null,
  "list": null,
  "set": null,
  "zset": null
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant