Skip to content

missing deadletter? #16

@nicolasburtey

Description

@nicolasburtey
pg=# \dT+ inboxeventstatus 
                                             List of data types
 Schema |       Name       |  Internal name   | Size |  Elements  | Owner | Access privileges | Description 
--------+------------------+------------------+------+------------+-------+-------------------+-------------
 public | inboxeventstatus | inboxeventstatus | 4    | pending   +| user  |                   | 
        |                  |                  |      | processing+|       |                   | 
        |                  |                  |      | completed +|       |                   | 
        |                  |                  |      | failed     |       |                   | 

conversation with claude:

                                                                                
 ▐▛███▜▌   Claude Code v2.1.9                                                   
▝▜█████▛▘  Opus 4.5 · Claude Max                                                
  ▘▘ ▝▝    ~/Code/lana-bank-2                                                   
                                                                                
❯ tell me, if I were to use an inbox for an event source project + CQRS,        
  where inbox would be used for commands, what would be the feature of the      
  inbox. what would be its fields/struct?                                       
                                                                                
⏺ This is a great architectural question. Let me explain the inbox pattern for  
  event sourcing + CQRS.                                                        
                                                                                
  Inbox Pattern for Commands                                                    
                                                                                
  The inbox ensures exactly-once processing of commands through idempotency and 
  reliable delivery. Here's a typical structure:                                
                                                                                
  struct InboxEntry {                                                           
      // Identity & Idempotency                                                 
      id: Uuid,                          // Primary key                         
      message_id: String,                // Idempotency key (dedup commands)    
                                                                                
      // Command Data                                                           
      command_type: String,              // e.g., "CreateAccount",              
  "TransferFunds"                                                               
      payload: serde_json::Value,        // Serialized command data             
                                                                                
      // Processing State                                                       
      status: InboxStatus,               // Pending, Processing, Completed,     
  Failed                                                                        
      processed_at: Option<DateTime>,    // When successfully processed         
                                                                                
      // Reliability & Retry                                                    
      received_at: DateTime,             // When command arrived                
      retry_count: u32,                  // Number of attempts                  
      next_retry_at: Option<DateTime>,   // For exponential backoff             
      error: Option<String>,             // Last error message                  
                                                                                
      // Tracing & Context                                                      
      correlation_id: Option<String>,    // Trace across services               
      causation_id: Option<String>,      // What triggered this command         
      metadata: Option<serde_json::Value>, // Headers, user context, etc.       
  }                                                                             
                                                                                
  enum InboxStatus {                                                            
      Pending,                                                                  
      Processing,                                                               
      Completed,                                                                
      Failed,                                                                   
      DeadLetter,                                                               
  }                                                                             
                                                                                
  Key Features                                                                  
  ┌────────────────┬─────────────────────────────────────────────────────┐      
  │    Feature     │                       Purpose                       │      
  ├────────────────┼─────────────────────────────────────────────────────┤      
  │ Idempotency    │ message_id ensures duplicate commands are ignored   │      
  ├────────────────┼─────────────────────────────────────────────────────┤      
  │ Atomicity      │ Command insert + business logic in same transaction │      
  ├────────────────┼─────────────────────────────────────────────────────┤      
  │ Reliability    │ Retry with backoff on transient failures            │      
  ├────────────────┼─────────────────────────────────────────────────────┤      
  │ Observability  │ correlation_id for distributed tracing              │      
  ├────────────────┼─────────────────────────────────────────────────────┤      
  │ Dead-lettering │ Move poison messages after max retries              │      
  └────────────────┴─────────────────────────────────────────────────────┘      
  Processing Flow                                                               
                                                                                
  1. Receive command → Insert into inbox (if message_id not exists)             
  2. Worker picks up Pending entries, marks Processing                          
  3. Execute command handler                                                    
  4. On success → mark Completed, emit events to outbox                         
  5. On failure → increment retry_count, schedule next_retry_at                 
                                                                                
  Would you like me to explore your codebase to see if there are existing       
  patterns you're using that we could build upon?                               
                                                                                
❯ what is the difference between dead-lettering and failed?                     
                                                                                
⏺ Good question. The distinction is about recoverability:                       
  Status: Failed                                                                
  Meaning: Transient failure, will retry                                        
  What happens next: Worker retries after backoff (e.g., network timeout, DB    
    lock, temporary unavailability)                                             
  ────────────────────────────────────────                                      
  Status: DeadLetter                                                            
  Meaning: Permanent failure, won't retry                                       
  What happens next: Requires manual intervention (e.g., invalid payload,       
    business rule violation, max retries exceeded)                              
  Examples                                                                      
                                                                                
  Failed (transient):                                                           
  - Database connection dropped mid-transaction                                 
  - External service returned 503                                               
  - Lock contention / timeout                                                   
                                                                                
  DeadLetter (permanent):                                                       
  - Command schema validation failed                                            
  - Referenced entity doesn't exist and never will                              
  - Retry count exceeded threshold (e.g., 5 attempts)                           
  - Business invariant violated (e.g., insufficient funds with no pending       
  deposits)                                                                     
                                                                                
  Why separate them?                                                            
                                                                                
  Pending → Processing → Completed                                              
                ↓                                                               
             Failed (retry_count < max)                                         
                ↓                                                               
           DeadLetter (retry_count >= max OR unrecoverable)                     
                                                                                
  Dead-lettered messages typically:                                             
  1. Trigger alerts for ops/dev teams                                           
  2. Get moved to a separate table/queue for analysis                           
  3. May need code fixes before manual replay                                   
                                                                                
  This separation keeps your retry loop clean—workers only poll Pending/Failed, 
  while DeadLetter entries sit untouched until a human investigates.            

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions