Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deadlock with redis #185

Closed
thorstenfleischmann opened this issue Feb 11, 2025 · 2 comments
Closed

Deadlock with redis #185

thorstenfleischmann opened this issue Feb 11, 2025 · 2 comments

Comments

@thorstenfleischmann
Copy link
Contributor

thorstenfleischmann commented Feb 11, 2025

I experience deadlocks when having a lot of parallel requests.

With set $session_redis_pool_size 1 I can reproduce it always with just two requests.

The code is basically:

session:open()
dosomething()
session:close()

With some debug logs I think I figured out the reason.

  1. request 1 opens/locks session
  2. request 1 "open" puts connect back to pool ("set_keepalive")
  3. requests 1 does something
  4. requests 2 comes in and opens connection to redis pool. session is locked so it trys to aquire lock which takes time to time out
  5. request 1 did something and wants to close session. session:close cannot get a connection to unlock because requests 2 blocks the session waiting for the lock

Is there a good way to prevent this?

I am using lua resty session 3.

With a bigger pool size there are factors which makes the deadlock more or less likely. It looks like http2 is a factor since there can be more parallel client connections.

@thorstenfleischmann
Copy link
Contributor Author

I found a possible solution, but it requires code change in storage/redis.lua at function storage.lock(key)

        -- put connection back to pool
        self:set_keepalive()

        sleep(waittime)

        -- reconnect
        local ok, err = self:connect()
        if not ok then
            return nil, err
        end

This puts the connection during spin lock wait back to the connection pool. This might introduce some overhead during waiting for a lock but it eliminates the deadlocks.

@bungle
Copy link
Owner

bungle commented Mar 18, 2025

@thorstenfleischmann,

The 4.x uses only lockless mode. With 3.x you can use regenerate strategy:
https://github.com/bungle/lua-resty-session/tree/v3.10?tab=readme-ov-file#pluggable-session-strategies

@bungle bungle closed this as completed Mar 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants