Skip to content

Pass JOIN_TIME_OUT value to keepalive #3826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 24 commits into
base: main
Choose a base branch
from

Conversation

ahkcs
Copy link
Contributor

@ahkcs ahkcs commented Jun 26, 2025

Description

Pass JOIN_TIME_OUT value to keepalive
issue link

#3820

Result

When running this:

curl -X POST "localhost:9200/_plugins/_sql" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT /*! JOIN_TIME_OUT(120) */ l.id, l.name, r.value FROM test_left AS l LEFT JOIN test_right AS r ON l.epoch = r.epoch LIMIT 5"
  }'
Request SQLQueryRequest(jsonContent={"query":"SELECT /*! JOIN_TIME_OUT(120) */ l.id, l.name, r.value FROM test_left AS l LEFT JOIN test_right AS r ON l.epoch = r.epoch LIMIT 5"}, query=SELECT /*! JOIN_TIME_OUT(120) */ l.id, l.name, r.value FROM test_left AS l LEFT JOIN test_right AS r ON l.epoch = r.epoch LIMIT 5, path=/_plugins/_sql, format=jdbc, params={}, sanitize=true, pretty=false, cursor=Optional.empty) is not supported and falling back to old SQL engine
[2025-07-01T11:42:37,617][INFO ][o.o.s.l.p.RestSqlAction  ] [integTest-0] Request Query: ( SELECT /*! JOIN_TIME_OUT(12number) */ identifier, identifier, identifier FROM table l LEFT JOIN table r ON identifier = identifier LIMIT number )
[2025-07-01T11:42:37,649][INFO ][o.o.s.l.q.p.HashJoinQueryPlanRequestBuilder] [integTest-0] HashJoinQueryPlanRequestBuilder: Found JOIN_TIME_OUT hint: 120 seconds, configuring PIT keepalive
[2025-07-01T11:42:37,650][INFO ][o.o.s.l.q.p.c.Config     ] [integTest-0] Config: Set custom PIT keepalive to: 2m (120000ms)
[2025-07-01T11:42:37,657][INFO ][o.o.s.l.e.j.QueryPlanElasticExecutor] [integTest-0] QueryPlanElasticExecutor: Using custom PIT keepalive from JOIN_TIME_OUT hint: 120 seconds
[2025-07-01T11:42:37,666][INFO ][o.o.s.l.q.p.p.n.p.PointInTime] [integTest-0] PointInTime: Creating PIT with JOIN_TIME_OUT hint support: 120 seconds
[2025-07-01T11:42:37,666][INFO ][o.o.s.l.p.PointInTimeHandlerImpl] [integTest-0] Using custom PIT keepalive from config: 2m (120000ms)
[2025-07-01T11:42:37,666][INFO ][o.o.s.l.p.PointInTimeHandlerImpl] [integTest-0] Creating PIT with keepalive: 2m (120000ms)
[2025-07-01T11:42:37,680][INFO ][o.o.s.l.p.PointInTimeHandlerImpl] [integTest-0] Created Point In Time s9y3QQEJdGVzdF9sZWZ0 with keepalive 2m successfully.
[2025-07-01T11:42:37,680][INFO ][o.o.s.l.q.p.p.n.p.PointInTime] [integTest-0] Loading first batch of response using Point In Time
[2025-07-01T11:42:37,728][INFO ][o.o.s.l.q.p.p.n.p.PointInTime] [integTest-0] Loading next batch of response using Point In Time. - s9y3QQEJdGVzdF9sZWZ0
[2025-07-01T11:42:37,733][INFO ][o.o.s.l.q.p.p.n.p.PointInTime] [integTest-0] PointInTime: Creating PIT with JOIN_TIME_OUT hint support: 120 seconds
[2025-07-01T11:42:37,733][INFO ][o.o.s.l.p.PointInTimeHandlerImpl] [integTest-0] Using custom PIT keepalive from config: 2m (120000ms)
[2025-07-01T11:42:37,733][INFO ][o.o.s.l.p.PointInTimeHandlerImpl] [integTest-0] Creating PIT with keepalive: 2m (120000ms)
[2025-07-01T11:42:37,734][INFO ][o.o.s.l.p.PointInTimeHandlerImpl] [integTest-0] Created Point In Time s9y3QQEKdGVzdF9yaWdo with keepalive 2m successfully.
[2025-07-01T11:42:37,734][INFO ][o.o.s.l.q.p.p.n.p.PointInTime] [integTest-0] Loading first batch of response using Point In Time
[2025-07-01T11:42:37,736][INFO ][o.o.s.l.q.p.p.n.p.PointInTime] [integTest-0] Loading next batch of response using Point In Time. - s9y3QQEKdGVzdF9yaWdo
[2025-07-01T11:42:37,742][INFO ][o.o.s.l.p.PointInTimeHandlerImpl] [integTest-0] Delete Point In Time s9y3QQEJdGVzdF9sZWZ0 status: 200
[2025-07-01T11:42:37,742][INFO ][o.o.s.l.p.PointInTimeHandlerImpl] [integTest-0] Delete Point In Time s9y3QQEKdGVzdF9yaWdo status: 200



@ahkcs ahkcs changed the title Fix JOIN_TIME_OUT_HINT Pass JOIN_TIME_OUT value to keepalive Jun 27, 2025
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
ahkcs added 2 commits June 27, 2025 01:37
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Copy link
Collaborator

@Swiddis Swiddis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Misclicked approve 😔

Signed-off-by: Kai Huang <[email protected]>
@ahkcs ahkcs requested a review from RyanL1997 June 30, 2025 22:15
@ahkcs ahkcs requested a review from dai-chen July 1, 2025 18:53
Swiddis
Swiddis previously approved these changes Jul 1, 2025
Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes! Just one last comment is UT / IT are both missing? If so, I think we should add both if possible.

Copy link
Collaborator

@noCharger noCharger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove internal link in PR description.

@noCharger
Copy link
Collaborator

Thanks for the changes! Just one last comment is UT / IT are both missing? If so, I think we should add both if possible.

+1 on test coverage

Comment on lines 157 to 171
public void setCustomPitKeepAlive(TimeValue timeout) {
this.customPitKeepAlive = Optional.ofNullable(timeout);
LOG.info(
"Config: Set custom PIT keepalive to: {}",
customPitKeepAlive.map(t -> t + " (" + t.getMillis() + "ms)").orElse("default"));
}

/**
* Check if custom PIT keepalive is set
*
* @return true if custom timeout is configured
*/
public boolean hasCustomPitKeepAlive() {
return customPitKeepAlive.isPresent();
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. can we remove log on getter/setter?
  2. can we name methods consistently with the rest in this class?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed methods

Comment on lines 102 to 105
Pattern pattern =
Pattern.compile(
"/\\*!\\s*JOIN_TIME_OUT\\s*\\(\\s*(\\d+)\\s*\\)\\s*\\*/", Pattern.CASE_INSENSITIVE);
Matcher matcher = pattern.matcher(sql);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using regex is concerning on correctness and performance; can we reuse HintFactory / SqlParser?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the use of regex and reused HintFactory/SqlParser

@@ -17,6 +20,7 @@ public class TableInJoinRequestBuilder {
private List<Field> returnedFields;
private Select originalSelect;
private Integer hintLimit;
private Config hintConfig;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config has more fields than just hint. At this level this should be the same as hintLimit

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed hintConfig, using TimeValue hintJoinTimeout

import org.opensearch.transport.client.Client;

/** Handler for Point In Time */
public class PointInTimeHandlerImpl implements PointInTimeHandler {
private final Client client;
private String[] indices;
private final Optional<Config> config;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config has more fields than just hint. We should reduce the scope

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reduced scope to only TimeValue

Comment on lines 37 to 41
request
.getConfig()
.getCustomPitKeepAlive()
.map(timeout -> String.valueOf(timeout.getSeconds()))
.orElse("unknown"));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why need this transformation here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed transformation

this.queryPlanner = request.plan();
}

@Override
public void run() throws IOException, SqlParseException {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why need to override the execution here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the run() method

ahkcs added 5 commits July 2, 2025 16:20
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
@ahkcs ahkcs requested review from dai-chen, noCharger and Swiddis July 3, 2025 04:18
ahkcs added 2 commits July 3, 2025 12:13
Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
@ahkcs ahkcs requested a review from dai-chen July 3, 2025 20:42
dai-chen
dai-chen previously approved these changes Jul 3, 2025
Copy link
Collaborator

@dai-chen dai-chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!


/** Consistency test: Verify repeated queries with JOIN_TIME_OUT produce consistent results */
@Test
public void testJoinTimeoutHintConsistency() throws IOException {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may become flaky because no deterministic guarantee either in index scan or limit operator?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed test case, trimmed further

Signed-off-by: Kai Huang <[email protected]>
Signed-off-by: Kai Huang <[email protected]>
@ahkcs ahkcs requested a review from dai-chen July 7, 2025 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants