Skip to content

Expose HTTP response codes to the pipeline for conditional routing #153

@nanjum88

Description

@nanjum88

Current Limitation
The logstash-output-http plugin does not expose HTTP response codes to the pipeline, which prevents conditional routing based on success or failure.

Use Case
We are using the logstash-input-s3-sns-sqs plugin to ingest logs from AWS S3 buckets via SQS notifications in a high-volume environment. These logs are then sent downstream using the logstash-output-http plugin.
Even with retry_failed and retry_non_idempotent enabled, we still experience data loss when the downstream system is under heavy load. To mitigate this, we want to route failed HTTP requests (based on response codes) to an SQS dead-letter queue (DLQ) so they can be retried later.

Additional Concern
The logstash-output-http plugin does not acknowledge events, which sometimes results in duplicate events during retries.

Feature Request

Expose HTTP response codes (or success/failure status) to the pipeline for conditional routing.
Consider adding an acknowledgment mechanism to reduce duplicate events.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions