PingAsync arbitrarily fails

[fluffynuts]

Elastic.Clients.Elasticsearch version: 9.2.1

Disclaimer: I don't definitely know this is a bug so far - more of a question. But the bug template provides good places to include information.

Elasticsearch version: ES Cloud, hosted, 9.2.1

.NET runtime version:
net8.0

Operating system version:
Linux

Description of the problem including expected versus actual behavior:
Sometimes, .PingAsync on a client fails with a socket error. I've tried increasing the ping timeout to 10s, but I'm still seeing the issue.

Steps to reproduce:
I have a process that is indexing some of our data, going back in time. What I've seen is that about 20k items are queued at our end to index in our staging environment, but the process appears to stall at about 600 or so. I see a few errors like this (and then our own mechanisms back off, so indexing stops for a while):

2025-12-02 13:01:54,961 country::STG-2-ZA thread::.NET TP Worker level::ERROR logger::BLL.Infrastructure.SagaLogger - LogForQueue - message::Queue log meta::{"LogType":"MessageError","Info":null,"MessageGuid":"c22bab97-5139-4bce-8633-6ba95d0a9570","QueueName":"staging-2-yumbi-app:order.index","Meta":"{
  \"Type\": \"BLL.Providers.CallCenters.CustomerIndexing.ElasticsearchConnectionError, BLL, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null\",
  \"PingResponse\": {
    \"$type\": \"Elastic.Clients.Elasticsearch.PingResponse, Elastic.Clients.Elasticsearch\",
    \"ElasticsearchWarnings\": {
      \"$type\": \"Elastic.Transport.Products.Elasticsearch.ElasticsearchResponse+<get_ElasticsearchWarnings>d__1, Elastic.Transport\",
      \"$values\": []
    },
    \"DebugInformation\": \"Invalid Elasticsearch response built from a unsuccessful () low level call on HEAD: /?pretty=true&error_trace=true\
 Exception: Unknown socket error (3ebdb6c772c14c61830ed5a3032fe082.eu-west-1.aws.found.io:443)\
\
# Audit trail of this API call:\
 - [1] BadRequest: Node: https://elastic:redacted@3ebdb6c772c14c61830ed5a3032fe082.eu-west-1.aws.found.io/ Took: 00:00:00.0038886\
# OriginalException: System.Net.Http.HttpRequestException: Unknown socket error (3ebdb6c772c14c61830ed5a3032fe082.eu-west-1.aws.found.io:443)\
 ---> System.Net.Sockets.SocketException (0xFFFDFFFE): Unknown socket error\
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)\
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\
   --- End of inner exception stack trace ---\
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)\
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, CancellationToken cancellationToken)\
# Request:\
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>\
# Response:\
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>\
# TCP states:\
  Established: 531\
  LastAck: 1\
  TimeWait: 7\
\
# ThreadPool statistics:\
  Worker: \
    Busy: 1\
    Free: 32766\
    Min: 2200\
    Max: 32767\
  IOCP: \
    Busy: 0\
    Free: 1000\
    Min: 32\
    Max: 1000\
\
\",
    \"IsValidResponse\": false,
    \"ElasticsearchServerError\": null,
    \"ApiCallDetails\": {
      \"AuditTrail\": {
        \"$type\": \"Elastic.Transport.Diagnostics.Auditing.Auditor, Elastic.Transport\",
        \"$values\": [
          {
            \"Event\": 13,
            \"Node\": {
              \"Features\": [],
              \"Settings\": {},
              \"Id\": null,
              \"Name\": null,
              \"Uri\": \"https://elastic:Ho3IcNH4NEqw3przpKmuBKAF@3ebdb6c772c14c61830ed5a3032fe082.eu-west-1.aws.found.io:443/\",
              \"IsAlive\": true,
              \"DeadUntil\": \"0001-01-01T00:00:00+00:00\",
              \"FailedAttempts\": 0,
              \"IsResurrected\": true
            },
            \"Ended\": \"2025-12-02T13:01:54.9524037+00:00\",
            \"Started\": \"2025-12-02T13:01:54.9485151+00:00\",
            \"Exception\": null
          }
        ]
      },
      \"DebugInformation\": \"Unsuccessful () low level call on HEAD: /?pretty=true&error_trace=true\
 Exception: Unknown socket error (3ebdb6c772c14c61830ed5a3032fe082.eu-west-1.aws.found.io:443)\
\
# Audit trail of this API call:\
 - [1] BadRequest: Node: https://elastic:redacted@3ebdb6c772c14c61830ed5a3032fe082.eu-west-1.aws.found.io/ Took: 00:00:00.0038886\
# OriginalException: System.Net.Http.HttpRequestException: Unknown socket error (3ebdb6c772c14c61830ed5a3032fe082.eu-west-1.aws.found.io:443)\
 ---> System.Net.Sockets.SocketException (0xFFFDFFFE): Unknown socket error\
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)\
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\
   --- End of inner exception stack trace ---\
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)\
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, CancellationToken cancellationToken)\
# Request:\
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>\
# Response:\
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>\
# TCP states:\
  Established: 531\
  LastAck: 1\
  TimeWait: 7\
\
# ThreadPool statistics:\
  Worker: \
    Busy: 1\
    Free: 32766\
    Min: 2200\
    Max: 32767\
  IOCP: \
    Busy: 0\
    Free: 1000\
    Min: 32\
    Max: 1000\
\
\",
      \"HttpMethod\": 4,
      \"HttpStatusCode\": null,
      \"OriginalException\": {
        \"$type\": \"System.Net.Http.HttpRequestException, System.Net.Http\",
        \"HttpRequestError\": 2,
        \"StatusCode\": null,
        \"Message\": \"Unknown socket error (3ebdb6c772c14c61830ed5a3032fe082.eu-west-1.aws.found.io:443)\",
        \"Data\": {
          \"$type\": \"System.Collections.ListDictionaryInternal, System.Private.CoreLib\"
        },
        \"InnerException\": {
          \"$type\": \"System.Net.Sockets.SocketException, System.Net.Primitives\",
          \"ClassName\": \"System.Net.Sockets.SocketException\",
          \"Message\": \"Unknown socket error\",
          \"Data\": null,
          \"InnerException\": null,
          \"HelpURL\": null,
          \"StackTraceString\": \"   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)\
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\",
          \"RemoteStackTraceString\": null,
          \"RemoteStackIndex\": 0,
          \"ExceptionMethod\": null,
          \"HResult\": -2147467259,
          \"Source\": \"System.Net.Sockets\",
          \"WatsonBuckets\": null,
          \"NativeErrorCode\": -131074
        },
        \"HelpLink\": null,
        \"Source\": \"System.Net.Http\",
        \"HResult\": -2147467259,
        \"StackTrace\": \"   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)\
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)\
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, CancellationToken cancellationToken)\"
      },
      \"RequestBodyInBytes\": null,
      \"ResponseBodyInBytes\": null,
      \"ResponseMimeType\": null,
      \"ResponseContentType\": null,
      \"HasSuccessfulStatusCode\": false,
      \"HasExpectedContentType\": false,
      \"Uri\": \"https://elastic:Ho3IcNH4NEqw3przpKmuBKAF@3ebdb6c772c14c61830ed5a3032fe082.eu-west-1.aws.found.io/?pretty=true&error_trace=true\"
    }
  },
  \"Message\": \"Unable to connect to Elasticsearch provider\",
  \"Data\": {
    \"$type\": \"System.Collections.ListDictionaryInternal, System.Private.CoreLib\"
  },
  \"InnerException\": null,
  \"HelpLink\": null,
  \"Source\": \"BLL\",
  \"HResult\": -2146233088,
  \"StackTrace\": \"   at BLL.Providers.CallCenters.CustomerIndexing.ElasticSearchConnectionVerifier.VerifyCanConnect(ElasticsearchClient client) in /home/runner/work/yumbi-web/yumbi-web/BLL/Providers/CallCenters/CustomerIndexing/ElasticSearchConnectionVerifier.cs:line 22\
   at BLL.Providers.CallCenters.CustomerIndexing.YumbiElasticsearchClientFactory.CreateAsync() in /home/runner/work/yumbi-web/yumbi-web/BLL/Providers/CallCenters/CustomerIndexing/YumbiElasticSearchClientFactory.cs:line 47\
   at BLL.Providers.CallCenters.CustomerIndexing.CustomerOrderIndex.<>c__DisplayClass36_0.<<Process>b__0>d.MoveNext() in /home/runner/work/yumbi-web/yumbi-web/BLL/Providers/CallCenters/CustomerIndexing/CustomerOrderIndex.cs:line 474\
--- End of stack trace from previous location ---\
   at PeanutButter.Utils.Async.<>c__DisplayClass0_0.<<RunSync>b__0>d.MoveNext()\
--- End of stack trace from previous location ---\
   at PeanutButter.Utils.Async.RunSync(Func`1 task)\
   at BLL.Providers.CallCenters.CustomerIndexing.CustomerOrderIndex.Process(DeliveryArgs args, IndexOrderMessage message) in /home/runner/work/yumbi-web/yumbi-web/BLL/Providers/CallCenters/CustomerIndexing/CustomerOrderIndex.cs:line 471\
   at Clyral.RabbitBus.MessageHandlers.DurableMessageWrapper`1.TryProcess(DeliveryArgs args, T message)\",
  \"EnvironmentLogInfo\": {
    \"$type\": \"Codeo.Core.Logging.EnvironmentLogInfo, Codeo.Core\",
    \"ProcessId\": 1,
    \"ProcessName\": \"redacted\",
    \"EntryAssembly\": \"redacted, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null\",
    \"EntryAssemblyVersion\": \"1.0.0.0\",
    \"MachineName\": \"31a8765e2dd7\",
    \"IPAddress\": \"172.17.0.7\",
    \"Is64BitProcess\": true,
    \"ThreadCulture\": \"\",
    \"HttpRequestStartDatetimeUTC\": null,
    \"DateTimeUTC\": \"2025-12-02T13:01:54.953635Z\",
    \"DateTimeZone\": \"South Africa Standard Time\",
    \"BrowserVersion\": null,
    \"IsBot\": false,
    \"UserAgent\": null,
    \"Referer\": null,
    \"Url\": null,
    \"QueryString\": null,
    \"RequestMethod\": null,
    \"IsAuthenticated\": false,
    \"UserName\": \"app\",
    \"Headers\": null,
    \"Form\": null,
    \"RawPostData\": null
  }
}","TimeSinceMessageStart":26.0} exception::

I have called EnableDebugMode on connection settings when creating this connection - according to the docs, this should have disabled streaming (but the error message still suggests it, and I see no request/response info)

the flow is:

1. a connection factory creates the connection
2. that connection is passed into a connection verifier to issue a .PingAsync - there's no point giving back a client if it definitely can't reach the service. This is where the exception above is raised
3. do the indexing (we don't seem to always get here)

Expected behavior
I would expect a ping to either completely work, or come back with an error about being unable to connect. But this is complaining about a socket disconnect, which I guess could happen if the request is cancelled, either via timeout or with a cancellation token. I've upped the ping timeout to 10s, but I still see the above cropping up after some processing.

Perhaps I'm just doing something wrong. I'm not sure. I'm unable to reproduce this in a test, spawning 2000 connections in about a minute (I'm running a Parallel.ForAsync to reduce time and give concurrency, and the ping time has to be factored into the equation). In the code, I should only be indexing one item at a time, so the parallel test was just to see if there was something like a rate limiter causing issues. Perhaps there's something else causing issues? Any assistance appreciated.