Skip to content

feat: implement Dead Letter Queue (DLQ) #111

@LeonardoVieira1630

Description

@LeonardoVieira1630

📘 Context

Currently, when the system encounters a message it cannot process (e.g., unknown trigger), it enters an infinite reprocessing loop due to NACK with requeue. This causes:

  • High redelivery rate (200+ messages/second)
  • Blocking of other messages in the queue (prefetch=1)
  • Unnecessary resource consumption
  • Inability to process valid messages

🐞 Problem Identified

During an accidental deployment of development code to production, voting-power-changed messages were generated. When production code was restored, these messages got stuck in the queue causing an infinite loop, as the dispatcher doesn't have a handler for this trigger type.

🔧 Tasks

  1. Update RabbitMQConsumer to automatically create DLQ

    • Modify consumer.ts to configure DLQ for all queues
    • Configure queue with x-dead-letter-exchange and x-dead-letter-routing-key
    • Automatically create DLQ with .dlq suffix
    • Set 7-day TTL for messages in DLQ
  2. Implement retry logic with counter

    • Add x-retry-count header to messages
    • After 3 attempts, use channel.reject(msg, false) to send to DLQ
    • Include error information in message header
  3. Add default handler for unknown triggers

    • Warning log for unregistered trigger
    • Direct ACK after logging (don't send to DLQ)
  4. Create monitoring script

    • Script to check messages in DLQs
    • Prepare structure for n8n integration
  5. Tests

    • Unit test for retry behavior
    • Integration test with unknown trigger message
    • Verify messages go to DLQ after 3 attempts

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions