[CORE-8848] rptest: adjust compaction settings in datalake/compaction_test
#25075
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Retry command for Build#61801please wait until all jobs are finished before running the slash command |
CI test resultstest results on build#61801
test results on build#61807
test results on build#61813
test results on build#61823
|
Because we now schedule adjacent segment compaction after sliding window compaction, this test was having trouble trying to reach the desired number of segments while producing. Increase the timeout as well as the `log_compaction_interval_ms` to allow the test to reach the desired number of segments.
WillemKauf
force-pushed
the
datalake_compaction_timeout_fix
branch
from
57d7ac8 to
a2fe061
Compare
|
/ci-repeat 1 |
bharathv
previously approved these changes
Feb 12, 2025
|
@bharathv the timeout issues in Separate issue? |
|
I think compute_storage.py should look like try:
return [f for f in p.iterdir()]
except FileNotFoundError:
return []
except NotADirectoryError:
return [] but I'm not sure yet, quite tired |
|
Force push to:
|
WillemKauf
commented
Feb 12, 2025
| @@ -69,7 +69,7 @@ def __iter__(self) -> Iterator[Header]: | |||
|
|
|||
| def safe_isdir(p: Path) -> bool: | |||
| """ | |||
| It's valid for files to be deleted at any time, | |||
| It's valid for files to be deleted at any time, | |||
There was a problem hiding this comment.
trailing whitespace removal
WillemKauf
commented
Feb 12, 2025
| @@ -81,10 +81,13 @@ def safe_isdir(p: Path) -> bool: | |||
|
|
|||
| def safe_listdir(p: Path) -> list[Path]: | |||
| """ | |||
| It's valid for directories to be deleted at any time, | |||
| It's valid for directories to be deleted at any time, | |||
There was a problem hiding this comment.
trailing whitespace removal
|
I think we should also add "datalake_staging" to this list here, it was recently moved under the data directory. |
The `datalake_staging` folder was recently moved under the redpanda data directory. It should not be considered as a namespace in `compute_size()`.
Awesome, added. |
bharathv
approved these changes
Feb 12, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Because we now schedule adjacent segment compaction after sliding window compaction, this test was having trouble trying to reach the desired number of segments while producing.
Increase the timeout as well as the
log_compaction_interval_msto allow the test to reach the desired number of segments.Backports Required
Release Notes