Redis queue needs to implement job priority

[farmani]

mysql driver implement job priority but it doesnt support by redis yet. can you please implement it for redis and may be other driver please?

[farmani]

I hope this will be doable before 2.0.14

[kushalhalder]

You can create multiple queues, for example: high priority, medium priority and low priority. Have different redis databases for the queues. Then add 3 workers for the high priority queue, 2 for medium priority and 1 for low priority.

[samdark]

I hope this will be doable before 2.0.14

doesn't matter, releases of extensions aren't too tied to framework releases

[farmani]

@kushalhalder I know but not a good idea when other driver support priority.
@samdark yeah you right. it seems I do not used to extensions yet

[zhuravljov]

Do you have any ideas to implement this lossless?
Now redis driver works using redis lists. Publisher pushes new message into start of list using LPUSH command (complexity: O(1)), and subscriber waits and receives the message from end of list using BRPOP (complexity: O(1)). Subscriber should be without changes, because this is fastest way to receive messages. Therefore publisher should be modified to insert new message into list according its priority O(n). The changing will complicate publishing, and redis driver will not be so fast.

[Insolita]

You can create multiple queues, for example: high priority, medium priority and low priority. Have different redis databases for the queues. Then add 3 workers for the high priority queue, 2 for medium priority and 1 for low priority.

It's good, but it require computing power for parallel execution 6 tasks, it may be sensitive for intensive tasks.

If only high and default priority needs, for high priority possible to make RPUSH
For high/default/low priority - only O(n) complexity is possible.
So i think, that it may be realized as custom, non-default driver, only for cases when queue speed has less significance than number of workers

[Insolita]

Also is the way - to make 3 Redis lists for high, default and low priorities, it can be faster on redis level, but more complex on queue manager level

[ldkafka]

How about we implement non-transparent manipulation of the queue channel name appending a unique priority suffix for the drivers that do not support priority. So, if the channel name is 'my-queue', the ->priority(2)->push() would push onto 'my_queue:_pri_2'. Then we just sort in the subscription listener to the channel?

[samdark]

Sounds hacky but might be the best available way to do it.