fix(): fixing exception in ingestion of expired keys in keys source #55
Conversation
|
hello there! is there a chance this fix could be released? I face similar issue regarding keys which are either expired or evicted. I believe this optional setting on string, especially on redis type would prevent connector from such error :
thanks a lot! |
|
Hello, do we know when this will be fixed I am also having problems with this issue, and my connector always goes offline, and I need to restart the whole kafka service to fix it, because only restarting the task of the connector does not solve the problem. |
3 participants
Problem
When Redis keys have very low TTL values, they may expire after keyspace notifications are sent but before the source connector can process them. In such cases, the connector receives an event indicating that something happened with the key, but when it attempts to retrieve the key data, it returns null since the key has already expired. This caused the processing flow to break because the
typefield was null, leading toNullPointerExceptionor unexpected behavior while schema conversion since its of required string type field.Solution
To handle expired keys gracefully, I've made the a schema of
typefieldOPTIONAL_STRING_SCHEMA. When the type is null (indicating an expired key), the code now not throw any error and will set the type asnull. This allows the connector to continue processing these records as expired/non-existent keys rather than failing.Examples
Normal ingested record (key exists and has data):
{ "key": "redis_keys:User_2", "ttl": { "long": 1756401363994 }, "type": "ReJSON-RL", "hash": null, "string": null, "json": { "string": "{\"registertime\":1500765952457,\"userid\":\"User_2\",\"regionid\":\"Region_8\",\"gender\":\"FEMALE\"}" }, "list": null, "set": null, "zset": null }Expired key ingested record (key expired, type set to
null):{ "key": "redis_keys:User_5", "ttl": { "long": -2 }, "type": null, "hash": null, "string": null, "json": null, "list": null, "set": null, "zset": null }