-
Notifications
You must be signed in to change notification settings - Fork 84
Description
Current Limitation
The logstash-output-http plugin does not expose HTTP response codes to the pipeline, which prevents conditional routing based on success or failure.
Use Case
We are using the logstash-input-s3-sns-sqs plugin to ingest logs from AWS S3 buckets via SQS notifications in a high-volume environment. These logs are then sent downstream using the logstash-output-http plugin.
Even with retry_failed and retry_non_idempotent enabled, we still experience data loss when the downstream system is under heavy load. To mitigate this, we want to route failed HTTP requests (based on response codes) to an SQS dead-letter queue (DLQ) so they can be retried later.
Additional Concern
The logstash-output-http plugin does not acknowledge events, which sometimes results in duplicate events during retries.
Feature Request
Expose HTTP response codes (or success/failure status) to the pipeline for conditional routing.
Consider adding an acknowledgment mechanism to reduce duplicate events.