fix(): fixing exception in ingestion of expired keys in keys source #55
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
When Redis keys have very low TTL values, they may expire after keyspace notifications are sent but before the source connector can process them. In such cases, the connector receives an event indicating that something happened with the key, but when it attempts to retrieve the key data, it returns null since the key has already expired. This caused the processing flow to break because the
type
field was null, leading toNullPointerException
or unexpected behavior while schema conversion since its of required string type field.Solution
To handle expired keys gracefully, I've made the a schema of
type
fieldOPTIONAL_STRING_SCHEMA
. When the type is null (indicating an expired key), the code now not throw any error and will set the type asnull
. This allows the connector to continue processing these records as expired/non-existent keys rather than failing.Examples
Normal ingested record (key exists and has data):
Expired key ingested record (key expired, type set to
null
):