OPT_ALLOW_PATTERNS not impacting CLIENT TRACKING?

[dorrogeray]

Hi,

I have a question regarding OPT_ALLOW_PATTERNS. When I specify them using the Relay\Relay class, I would expect to see them have an impact on the CLIENT TRACKING ON command, ie: CLIENT TRACKING ON BCAST PREFIX myprefix:, but it seems that they are getting ignored on the subscription level? I am using relay 0.8.1 and just want to understand if this is expected behavior or not.

I also noticed that calling $relay->setOption(Relay::OPT_USE_CACHE, false); does not prevent the CLIENT TRACKING ON which I was thinking it perhaps should. Instead, the presence of CLIENT TRACKING ON seems to be driven only by relay.maxmemory in relay.ini (command CLIENT TRACKING ON is not emitted when its set to 0)

Thanks for your help!

[michael-grunder]

We don't currently use broadcast mode client tracking but rather the non-broadcasting mode where Redis will only send us an invalidation when a key the connection has cached is invalidated. It's been some time since I played around with BCAST mode but that sends invalidations regardless of whether its a key that the client has read.

Also, we have both OPT_ALLOW_PATTERN and OPT_IGNORE_PATTERN.

Not sending CLIENT TRACKING ON for cache-disabled objects is probably a good idea and should be simple enough to implement. I''m not sure whether it's a good idea to toggle it off and on with an already-connected object but certainly should be possible if the cache is disabled before any commands are delivered.

[dorrogeray]

Thanks for all the answers, I really appreciate it!

The workload pattern is indeed the key. Since relay is using persistent connections (I know they can be disabled but having them is better to reduce unnecesary TSL handshakes, etc) when combined with php-fpm with lifecycle of 5000 requests, it means that each connection will, over the course of these 5000 requests, touch all kinds of application flows (and thus various redis keys).

My worry is that this will cause massive amounts of invalidation messages for each php-fpm process.

The BCAST seems like a good way to control that only keys matching all of the bellow criteria are cached and tracked:

• high frequency of reads
• low frequency of writes
• reasonable value size (avoid overflowing relay memory)

I am currently testing relay without the BCAST at scale & depending on the results it is possible that I will reach out regarding possibility of adding BCAST capability into relay.

[michael-grunder]

My worry is that this will cause massive amounts of invalidation messages for each php-fpm process.

This is a good insight. If each php-fpm worker maintains its own CLIENT TRACKING ON connection and there are hundreds, invalidations can become pathological and make caching a net negative.

That said, this can be mitigated with relay.max_endpoint_dbs.

Relay will only enable client tracking on connections until this limit is reached. So you end up having a small number of cache writers with every worker able to read from the cache. Some workload patterns benefit from setting this value as low as 1. For example, if there are a variety of read-heavy keys.

We're also working on a feature allowing Relay to share these tracking-enabled connections between fpm workers, using a new-ish syscall pidfd_getfd.

[dorrogeray]

So does this mean that if I have pool with 40 static php-fpm workers, and each of them connects to 3 master nodes of redis cluster and I set relay.max_endpoint_dbs=1, I will get a total of 1 connection with CLIENT TRACKING ON out of the total of 120 connections (40 x 3)?

Could syscall pidfd_getfd also work for cli processes (ie cli workers consuming jobs from SQS)?

[michael-grunder]

So does this mean that if I have pool with 40 static php-fpm workers, and each of them connects to 3 master nodes of redis cluster and I set relay.max_endpoint_dbs=1, I will get a total of 1 connection with CLIENT TRACKING ON out of the total of 120 connections (40 x 3)?

yes, one in total which is probably not what you would want for cluster. We'd need to add some logic to allow the semantics of caching to one specific "cluster".

Could syscall pidfd_getfd also work for cli processes (ie cli workers consuming jobs from SQS)?

Yes you can grab an fd from any arbitrary process on the same machine. To share between arbitrary cli connections would require some mechanism in Relay to let it know that the cli process belongs to some specific group. That too could be placed in the shared memory segment.