Skip to content

[IDEA] buffer filter should set an expiration on redis lists #212

@frikilax

Description

@frikilax

What is your feature about ?
the buffer filter currently has a "flaw" in data management:
if buffer sources are modified after some time, the old sources (aka redis lists) will probably not be empty, this create a "memory leak" of sorts, as those lists won't probably ever be deleted from Redis, stacking up and using unnecessary memory

If your request is about an existing filter/functionality
This Feature Request is about the buffer filter

Describe what you'd like
To prevent that from happening, a minimal change needs to be done:
when the buffer starts or triggers to get all cached data from Redis, it should set/reset an expiry on the key

Thanks to that, 2 problems are solved:

  • if sources names change inside the buffer filter, old sources will eventually be cleared out of Redis -> no "memory leak"
  • if Darwin happens to have some downtime, this technique can remove obsoleted data from Redis

Additional context
The expiry value should be coherent with data longevity AND the frequency at which the buffer needs to fetch cached data, a sensible option could be to take the maximum value between twice the fetch interval, and a fixed value corresponding to data "freshness" (time before considering data is obsolete).
In all cases, the expiry value should be strictly more than the configured fetch interval, to avoid losing data before fetching it

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions