Consumer occur Bad frame read

[linbo]

I use librabbitmq for consumer data from rabbitmq, but got Bad frame read exception

Traceback (most recent call last):
  File "consumer.py", line 119, in consume
    connection.drain_events()
  File "/home/push/python/lib/python2.6/site-packages/librabbitmq/__init__.py", line 198, in drain_events
    self._basic_recv(timeout)
ChannelError: Bad frame read

Python: 2.6
RabbitMQ: 3.1.3, Erlang R14B04
librabbitmq==1.0.1

[ericworkman]

I'm also seeing this exception.

[2014-03-26 20:40:31,317: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/home/staffknex/.virtualenvs/staffknex/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
    blueprint.start(self)
  File "/home/staffknex/.virtualenvs/staffknex/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/home/staffknex/.virtualenvs/staffknex/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 796, in start
    c.loop(*c.loop_args())
  File "/home/staffknex/.virtualenvs/staffknex/local/lib/python2.7/site-packages/celery/worker/loops.py", line 72, in asynloop
    next(loop)
  File "/home/staffknex/.virtualenvs/staffknex/local/lib/python2.7/site-packages/kombu/async/hub.py", line 272, in create_loop
    item()
  File "/home/staffknex/.virtualenvs/staffknex/local/lib/python2.7/site-packages/amqp/utils.py", line 42, in __call__
    self.set_error_state(exc)
  File "/home/staffknex/.virtualenvs/staffknex/local/lib/python2.7/site-packages/amqp/utils.py", line 39, in __call__
    **dict(self.kwargs, **kwargs) if self.kwargs else kwargs
  File "/home/staffknex/.virtualenvs/staffknex/local/lib/python2.7/site-packages/kombu/transport/base.py", line 144, in _read
    drain_events(timeout=0)
  File "/home/staffknex/.virtualenvs/staffknex/local/lib/python2.7/site-packages/librabbitmq/__init__.py", line 205, in drain_events
    self._basic_recv(timeout)
ConnectionError: Bad frame read

Python: 2.7.4
RabbitMQ: 3.2.3, Erlang R14B04
librabbitmq==1.0.3
kombu==3.0.12
celery==3.1.9
amqp==1.4.3
billiard==3.3.0.16

I'm running three queues called high, medium, and low, two RabbitMQ servers in a cluster, and two servers called jobs0 and jobs1, each running a worker with 8 processes for each queue. I am unable to figure out the exactly what causes this issue. This only affects my high queue, which is the busiest of the queues.

Example behavior of the issue:
One of the processes on, for example, p0 on jobs0 for the high queue starts to throw this exception. The other processes appear to work correctly, and p0 sometimes work correctly. However the other processes start to throw the error, rarely at first and increasing in frequency over a few hours. Eventually, all processes for this queue on both jobs0 and jobs1 constantly throw the exception, and the cluster reports that there are no consumers on the queue. The other queues and their consumers are fine.

The only way I've managed to restore this system to working order is by:

1. Stopping all consumers on both jobs0 and jobs1
2. Uninstalling librabbitmq
3. Starting the consumers
4. Stopping the consumers
5. Installing librabbitmq
6. Starting the consumers

I'm unable to find any event in any of my logging that would indicate an problem. The cluster reports as healthy with plenty of available file descriptors, socket descriptors, Elrang processes, memory, and disk space. I do not see any network issues or events that would correlate to this.

Any ideas with what's going on? Thank you.

[ericworkman]

I went through the logs and found an interesting message. This log message was logged when I restarted the consumers after uninstalling librabbitmq, so it is using pyamqp.

[2014-03-31 08:06:44,609: WARNING/MainProcess] celery@celery-high.jobs1 ready.
[2014-03-31 08:06:44,611: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?

The full contents of the message body was: body: '' (0b)
{content_type:u'application/json' content_encoding:None
  delivery_info:{'consumer_tag': u'4', 'redelivered': True, 'routing_key': u'high', 'delivery_tag': 3, 'exchange': u'high'} headers={}}


[2014-03-31 08:06:44,631: WARNING/MainProcess] Received and deleted unknown message. Wrong destination?!?

The full contents of the message body was: body: '' (0b)
{content_type:u'application/json' content_encoding:None
  delivery_info:{'consumer_tag': u'4', 'redelivered': True, 'routing_key': u'high', 'delivery_tag': 39, 'exchange': u'high'} headers={}}

[GregUK]

I am having this issues aswell.

When a message is placed onto the queue with a payload of empty string the drain_events throws ConnectionError Bad frame read

[armyofevilrobots]

See #59