Acknowledgement for splunk_hec_logs sink does not behave as expected when indexers are behind an LB #19417
Description
A note for the community
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Problem
We are using the splunk_hec_logs sink with acknowledgements.indexer_acknowledgements_enabled set to true to test sending logs to our Splunk indexer cluster fronted by an AWS ALB, which has sticky sessions enabled. We have noticed that even though all the logs were ingested into Splunk successfully, the ack status would frequently return false, causing Vector to retry and create duplicate logs. The expected behavior is for Vector to work with LB sticky sessions so that requests to query ack statuses would be routed to the same indexers that processed the events.
Configuration
sinks:
splunk_indexer_cluster:
type: splunk_hec_logs
inputs:
- vector_source
acknowledgements:
indexer_acknowledgements_enabled: true
enabled: true
healthcheck:
enabled: true
endpoint: <SPLUNK_LB_ENDPOINT>
auto_extract_timestamp: true
encoding:
codec: json
default_token: abc1234
Version
v0.34.0
Debug Output
The following DEBUG output shows that Vector made a request to Splunk LB:
2023-12-16T07:56:14.385194Z DEBUG sink{component_kind="sink" component_id=splunk_indexer_cluster component_type=splunk_hec_logs}:request{request_id=1}:http: vector::internal_events::http_client: Sending HTTP request. uri=<SPLUNK_LB_ENDPOINT>/services/collector/event?auto%5Fextract%5Ftimestamp=true method=POST version=HTTP/1.1 headers={"content-type": "application/json", "authorization": Sensitive, "x-splunk-request-channel": "e7d5d8da-3a6a-495f-a3b8-b11ddcff65b0", "accept-encoding": "gzip", "user-agent": "Vector/0.34.0 (x86_64-unknown-linux-gnu c909b66 2023-11-07 15:07:26.748571656)"} body=[2070 bytes]
And got back a response with the AWSALBCORS cookie:
2023-12-16T07:56:14.411855Z DEBUG sink{component_kind="sink" component_id=splunk_indexer_cluster component_type=splunk_hec_logs}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"date": "Sat, 16 Dec 2023 07:56:14 GMT", "content-type": "application/json; charset=UTF-8", "content-length": "37", "connection": "keep-alive", "set-cookie": Sensitive, "set-cookie": "AWSALBCORS=prVL+uA5uOQRvB/tpnpNrAoK8DCSh4ml/dSGbbPkl4/NNI22uzNVPNdYSq1zt190/ywFX5TvRbBOCjWIVWeMrkwLalUTCyGM4Fss3xw5eGXt4/pXzyeOQewIxG+N; Expires=Sat, 23 Dec 2023 07:56:14 GMT; Path=/; SameSite=None; Secure", "x-content-type-options": "nosniff", "vary": "Authorization", "x-frame-options": "SAMEORIGIN", "server": "Splunkd"} body=[37 bytes]
Vector made a subsequent request to query the ack status, but did not include the AWS ALB cookie:
2023-12-16T07:57:12.249892Z DEBUG http: vector::internal_events::http_client: Sending HTTP request. uri=<SPLUNK_LB_ENDPOINT>/services/collector/ack method=POST version=HTTP/1.1 headers={"content-type": "application/json", "authorization": Sensitive, "x-splunk-request-channel": "e7d5d8da-3a6a-495f-a3b8-b11ddcff65b0", "user-agent": "Vector/0.34.0 (x86_64-unknown-linux-gnu c909b66 2023-11-07 15:07:26.748571656)", "accept-encoding": "identity"} body=[12 bytes]
And got back { acks: {0: false} } even though the test log was indexed successfully by Splunk
2023-12-16T07:57:12.277107Z DEBUG http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"date": "Sat, 16 Dec 2023 07:57:12 GMT", "content-type": "application/json; charset=UTF-8", "content-length": "20", "connection": "keep-alive", "set-cookie": Sensitive, "set-cookie": "AWSALBCORS=05ikGEEoAESzmQ3ObpUQP/FCkLo1erqyKWg1jUCRFNAYCXabl3AHosFvyTJ7izTtI3MCgKNJH0BRyPrhmiza1BGtLKGA5UsbWp2mnRxXnFRHsiCO1hQOgW41hqVc; Expires=Sat, 23 Dec 2023 07:57:12 GMT; Path=/; SameSite=None; Secure", "x-content-type-options": "nosniff", "vary": "Authorization", "x-frame-options": "SAMEORIGIN", "server": "Splunkd"} body=[20 bytes]
2023-12-16T07:57:12.277146Z DEBUG vector::sinks::splunk_hec::common::acknowledgements: Received ack statuses. ack_query_response=HecAckStatusResponse { acks: {0: false} }
And finally the request was marked failed:
ERROR sink{component_kind="sink" component_id=splunk_indexer_cluster component_type=splunk_hec_logs}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
### Example Data
_No response_
### Additional Context
_No response_
### References
_No response_