Skip to content

BZPOPMIN will hang for ever when a short disconnection happens and the port is different from 6379 #1888

@manast

Description

@manast

This one was pretty weird but quite serious. If you issue a BZPOPMIN command, and there is a small disconnection while the command is blocking, then it will stay blocking forever, effectively hanging the app that uses the command.

Now, one of the insane things about this issue is that to reproduce it you must use a port different than the standard 6379, and also use docker network disconnect, it is not enough to just stop the docker container running Redis:

Try this code:
bug.mjs:

import Redis from "ioredis";

const connection = new Redis({
    host: "localhost",
    port: 6380,
});

connection.on("connect", () => console.log("Redis connected!"));
connection.on("ready", () => console.log("Redis ready!"));
connection.on("error", (err) => console.error("Redis error:", err));
connection.on("end", () => console.log("Redis connection ended"));
connection.on("reconnecting", () => console.log("Redis reconnecting..."));

async function test() {
    console.log("Gonna issue a BZPOPMIN command...")
    const result = await connection.bzpopmin("key", 0)
    console.log("Issued BZPOPMIN command!", result)
}

test().catch(console.error);

docker-compose.yml:

version: "3.7"
services:
  redis:
    image: redis:alpine
    ports:
      - "6380:6379"

Start the docker container with:

docker-compose up

Run the test with:

node bug.mjs

Results in:

Gonna issue a BZPOPMIN command...
Redis connected!
Redis ready!

Now in a different terminal, and depending on where you run the docker (I had my docker compose on directory ioredis-econnreset:

docker network disconnect ioredis-econnreset_default ioredis-econnreset-redis-1
sleep 3
docker network connect ioredis-econnreset_default ioredis-econnreset-redis-1

The Redis network was disconnected and connected again after 3 seconds. Open a Redis cli:

redis-cli -p 6380
> 127.0.0.1:6380> zadd key 10 test
(integer) 1
127.0.0.1:6380>

The terminal with the node app will stay the same, with no error or nothing. If you restart the app and without disconnecting and reconnecting you try again to ZADD to the key you will get instead:

Gonna issue a BZPOPMIN command...
Redis connected!
Redis ready!
Issued BZPOPMIN command! [ 'key', 'test', '10' ]

Let me know if you need more information.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions