Inaccurate Delayed Message Processing and Future Plans for Resolution

[parkaran]

Describe the bug

Hello,

I would like to report an issue regarding the rabbitmq-delayed-message-exchange plugin in our production cluster. Currently, we have a cluster consisting of three nodes, and we are managing over 100,000 delayed messages and growing. These delayed messages have a wide range of delay intervals, from 2 weeks to 6 or 7 months, with a significant majority of messages approaching the upper limit mentioned.

We have noticed that certain messages, which should be processed in the future (e.g., 6 months from now), are being processed months earlier. This behavior is unexpected and inconsistent with the intended delay interval.

After reviewing the limitations of the plugin and an older issue (#72) discussing similar challenges with performance and scalability under high volumes of delayed messages, we are eager to know if any progress has been made to address this issue or if there are any plans for future enhancements.

Accurate delayed message processing is not only crucial for our system but also for many other users who rely on the rabbitmq-delayed-message-exchange plugin. It would significantly expand the potential use cases and value of this plugin across various domains.

Reproduction steps

n/a

Expected behavior

n/a

Additional context

RabbitMQ Version: 3.8.14

[michaelklishin]

Consider using Discussions for questions, they are no longer a new and unknown GitHub feature.

The plugin was never designed to handle delays of 6 months. You want a data store, not a RabbitMQ plugin for delayed publishing.

We have some ideas for what a different version of the plugin (targeting RabbitMQ 3.12) might look like but there are no plans to actively work on it this year.

[michaelklishin]

We are not aware of any Erlang runtime timer inaccuracies, although I have never seen a test case that would run for six months. We cannot suggest anything else without having at least some specific steps to reproduce (we will not run a 6 month long test, sorry).

This plugin is reasonable (if you understand the limitations) for delaying publishing for a few minutes or several hours. Maybe a few days at most. For anything beyond that, use a data store and an external (to RabbitMQ) scheduling tool of some kind.

It is a particularly bad idea to try to delay messages by weeks or months given the limitations of the current design of the plugin. They are pretty well documented in the README, so I won't repeat them.

[michaelklishin]

Someone has pointed it out to me that 6 months in milliseconds is 15,778,476,000, which is greater than 2^32 (2,147,483,647), so this is the longest delay period that the current design can support.

This is mentioned in the README:

For each message that crosses an "x-delayed-message" exchange, the plugin will try to determine
if the message has to be expired by making sure the delay is within range,
ie: Delay > 0, Delay =< ?ERL_MAX_T

(In Erlang a timer can be set up to (2^32)-1 milliseconds in the future).

[ikavgo]

It seems erts internally uses 64 bit ints for the timer. So 6 months in milliseconds shoudln't be a problem. @parkaran would you mind posting a code snippet that sends your messages, I would like to make sure that the correct data type is used and nothing gets truncated/wrapped along the wire?