Skip to content

[Reliability] Ack/Nack errors unhandled — messages double-processed or permanently stuck #3

Description

@YHQZ1

Both builder and deployer workers ignore the error return from msg.Ack() and msg.Nack().

If Ack fails after processing: the message is redelivered and the job runs twice — a user's deployment spins up duplicate ECS services.
If Nack fails after a bad parse: the message stays unacknowledged indefinitely and blocks the channel.

Affected:

apps/builder/internal/queue/worker.go:89 (Nack), 94 (Ack)
apps/deployer/internal/queue/worker.go:115, 130 (Nack), 120, 139 (Ack)

Fix:
go// BEFORE
m.Ack(false)

// AFTER
if err := m.Ack(false); err != nil {
log.Printf("failed to ack message %s: %v", m.MessageId, err)
}

Handle Ack errors in builder worker
Handle Nack errors in builder worker
Handle Ack errors in deployer worker
Handle Nack errors in deployer worker
Add structured logging on failures with message ID

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Critical — fix immediatelyReliabilityCorrectness / uptime issuebugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions