RedisCommandException: Multi-key operations must involve a single slot while storing PersistedGrant

[vadimdr]

Hi,

I am getting the following exception while trying to perform the Authorization code flow with IdentityServer4 configured to use the Redis as an operational store:

RedisCommandException: Multi-key operations must involve a single slot; keys can use 'hash tags' to help this, i.e. '{/users/12345}/account' and '{/users/12345}/contacts' will always be in the same slot
IdentityServer4.Contrib.RedisStore.Stores.PersistedGrantStore.StoreAsync(PersistedGrant grant)
IdentityServer4.Stores.DefaultGrantStore<T>.StoreItemAsync(string key, T item, string clientId, string subjectId, DateTime created, Nullable<DateTime> expiration)
IdentityServer4.Stores.DefaultGrantStore<T>.CreateItemAsync(T item, string clientId, string subjectId, DateTime created, int lifetime)
IdentityServer4.ResponseHandling.AuthorizeResponseGenerator.CreateCodeFlowResponseAsync(ValidatedAuthorizeRequest request)
...

My instance of IdentityServer4 based server is configured to use Redis as an operational store:

.AddOperationalStore(options =>
                {
                    options.KeyPrefix = IdSrvConstants.IdentityServerSolution;
                    options.RedisConnectionMultiplexer = scope.ServiceProvider.GetRequiredService<IConnectionMultiplexer>();
                });

and the Redis is AWS ElasticCache clustered Redis instance.
The same code works fine on a single-node Redis.

Based on my understanding, the problem is related to the way the PersistedGrantStore generates the key's names - it does not use the keys hash tags (see https://redis.io/topics/cluster-spec#keys-hash-tags) in order to make sure all the keys will involve a single slot while performing a single Redis transaction.

[AliBazzi]

Can you elaborate more on the mode of Elasticache instance, is Cluster mode On or Off ?

[AliBazzi]

Another question, what is the value of IdSrvConstants.IdentityServerSolution ?

[vadimdr]

Elasticache instance is in clustered mode. (Cluster mode on). There are 3 nodes and 1 shard.
The constant value is IdM.
We have switched the ElastiCache to non-clustered mode to test (Cluster mode off) and now it's working fine ...

I can certainly live with that in non-production environments, but for production, a cluster is a mandatory requirement ...

[AliBazzi]

I'm trying to figure out why it's not working in Cluster mode on.
I will keep digging, I though IdSrvConstants.IdentityServerSolution may have { } as a value, but looks it's not.

[vadimdr]

I was actually thinking that putting the KeyPrefix with { } might resolve the issue because then all the keys will be hashed based on KeyPrefix only and will fall in the same slot. I will try to change the code and see if it has any impact...

[AliBazzi]

is it possible that the SubjectId format on your side is formatted like {...} ? maybe it's a GUID that you convert into a string by calling .ToString() which will result in default formatting as {....} ?
To clarify SubjectId is sub claim.

[vadimdr]

This is how the Redis store looks like when I run it locally:

As you can see, all sub are GUIDs, but they are formated as a GUID without any {}. The name of the client also doesn't contain the {}

[vadimdr]

I have tried to wrap the PrefixKey with {}, so it will be {IdM} and it's actually working with clustered ElastiCache

[AliBazzi]

but what will happens is that hashing algo will result in storing all the key/values in one shard only of your cluster, so I don't think this is the outcome you want.

I'm looking at the code, it will be complicated to do this as a fix, rather it will need big investment from my side to implement that here.

I will try to dedicate a time for this, but I can't promise to be anytime soon, sorry for that, feel free to PR if you want to drill into that.

[vadimdr]

Understood, thank you for your efforts.
In the meanwhile, since I have only 1 shard in my ElastiCache cluster anyway, this resolution will work fine in this use case.

[simonpinn]

I've also just run into this issue, similar AWS config - has there been any update to this @AliBazzi ?

[simonpinn]

Can confirm - works when prefix is wrapped in curlies

[AliBazzi]

Hi @simonpinn wrapping the prefix will solve the problem but that will leave your Redis cluster unbalanced and not taking advantage of the clustered Redis

maybe you can use non-clustered instance so you don't waste money/resources on unused nodes, I found out that "fixing" this will leave current data in Redis unreachable, was thinking of wrapping the sub with curlies last year but found out that will leave current users in "logged-out" state cause their old data are no longer reachable.

[ShoSashko]

@AliBazzi Any updates on the issue?