Add configurable single node retry

[ban-xiu]

Add configurable single node retry mechanism to improve reliability in single-node deployments (e.g., gateway forwarding architectures).

Changes:

• Add enableSingleNodeRetry configuration (default: false)
• Add maxSingleNodeRetryAttempts configuration (default: 1)
• Add new test methods

This feature is particularly useful in gateway forwarding architectures where the client can only connect to a single gateway node, improving reliability by automatically retrying transient failures.

Backward compatible: disabled by default, no breaking changes.

I hope this feature will bring tangible improvements to the project's stability. The code is ready, and I welcome any suggestions for improvement. Looking forward to merging it soon!

[ban-xiu]

Enabling single node retry

To enable single node retry, use the setEnableSingleNodeRetry method when building the client:

Rest5Client client = Rest5Client.builder(HttpHost.create("http://gateway.example.com:9200"))
    .setEnableSingleNodeRetry(true)
    .build();

Setting maximum retry attempts

You can configure the maximum number of retry attempts using setMaxSingleNodeRetryAttempts:

Rest5Client client = Rest5Client.builder(HttpHost.create("http://gateway.example.com:9200"))
    .setEnableSingleNodeRetry(true)
    .setMaxSingleNodeRetryAttempts(3)
    .build();

The default value is 1, meaning the client will retry the request once after the initial failure.

[ban-xiu]

@l-trotta hi, could you please help me take a look at this?

[l-trotta]

hey @ban-xiu, thanks for contributing! I took a quick look, and I'm not sure this is how we want this implemented: we already had plans for a generic retry mechanism (see this PR) which would work regardless of the number of nodes, also the idea would be to make the feature independent from the rest client implementation (since the rest5client is only the default implementation we provide), and to reuse the BackoffPolicy logic that is currently only used in the bulk ingester. I'll give this some thought and update soon.

[ban-xiu]

@l-trotta The focus of these two PRs is different: the logic for node retries already exists by default, but there is a lack of retry logic specifically for gateway nodes.

[ban-xiu]

Architecturally, rest5client and the planned new client are independent of each other. There is no subordinate relationship between them, and they do not conflict with one another. They represent two separate sets of solutions that can be used either independently or concurrently.

[l-trotta]

Hello @ban-xiu, after a lot of thinking I'm confirming my initial decision not to accept these changes.

It's not a refusal to implement a retry mechanism, I agree that the client is missing one, since currently it only works if there are multiple nodes, and there's no way to configure the dead node marking behavior, but my goal is to have something flexible that will work regardless of the number of nodes, and regardless of the http client implementation.

To clarify on the architecture: RestClient and the new Rest5Client are only a possible implementation of the networking layer of the client, the ones that we provide by default, because the modular architecture of the client allows users to create their own implementation of TransportHttpClient and ElasticsearchTransportBase to handle networking.
You can see an example of this in the examples-transport folder where you can find an example alternative implementation of the transport layer that uses Netty instead of Apache Http.

This is the reason why I'd like the retry mechanism not to be strictly tied to Rest5Client, and instead be implemented at a higher level so that it can be a wrapper to whatever transport implementation the user provided... meaning it will be a bit harder to write, but I'll try to work on it ASAP since it's a missing feature.

As for what you need right now, I'd suggest reusing the code you wrote to create a custom transport that suits your case: without having to fork the client you can simply copy your implementation of Rest5Client and call it something else, create a new YourRest5ClientTransport (which can probably be just a copy paste of Rest5ClientTransport replacing Rest5Client with YourRest5Client), and then create the client like this:

YourRest5Client restClient = YourRest5Client
    .builder(...).build();
    
ElasticsearchTransport transport = new YourRest5ClientTransport(
    restClient, new JacksonJsonpMapper(), options);
    
ElasticsearchClient esClient = new ElasticsearchClient(transport);

If you'd like to follow this route I'd be happy to help, we could also have this implementation added to example-transport, or you could publish it as a separate github repo since it could be useful for others.

Thanks again for your time contributing to the client, I'm closing this for now, but feel free to reply below or open a new issue if you need further clarification.