Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix iceberg data migration test timeout #25069

Draft
wants to merge 4 commits into
base: dev
Choose a base branch
from
Draft

Fix iceberg data migration test timeout #25069

wants to merge 4 commits into from

Conversation

bashtanov
Copy link
Contributor

The test is broken: it swallows errors, timeout ones in particular.
Make it propagate errors. To avoid timeouts:

  • make sure verifier offline mode only waits for consumption not for anything in the querying thread
  • reduce production rate
  • allow more time to complete

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

@bashtanov
tests/datalake/verifier: do not swallow exceptions when waiting
b49381e
@bashtanov
tests/datalake/verifier: offline mode improvements
bea3869 
- wait for consuming all messages regardless translation state
- avoid race conditions when stopping consumer
@bashtanov
tests/util/rpcn/counter_stream_config: make pace configurable
fd1851e 
sometimes maximum throughput is not desired
@bashtanov
tests/datalake/mount_unmount: remove sleep, reduce pace, give time
162ed94 
Pace reduced as otherwise RPCN produces too much data while unmounting,
so it takes unreasonable time to complete.

Sleep removed as verifier is more robust.

Time limit increased as sometimes it still takes time for Verifier to go
offline:
- offline mode waits for consuming till migrations blocking offset
- consume thread waits for query thread (limited comparison buffer)
- query thread may lag because translation lags
- on unmount translation is forced to only eventually translate all
data, it still may arrive later than the topic is removed
@vbotbuildovich
Copy link
Collaborator

CI test results

test results on build#61767
test_id test_kind job_url test_status passed
rptest.tests.compaction_recovery_test.CompactionRecoveryTest.test_index_recovery ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed78-5b46-4f61-a4d1-c0b9f80c8fe8 FLAKY 1/3
rptest.tests.compaction_recovery_test.CompactionRecoveryUpgradeTest.test_index_recovery_after_upgrade ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed78-5b46-4a7a-b743-4765ba61ffe9 FLAKY 1/2
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_HADOOP ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed65-07a0-4a9d-b85f-052e159a33fa FLAKY 1/2
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3.catalog_type=CatalogType.REST_JDBC ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed65-079e-47f9-8066-e78270290712 FLAKY 1/2
rptest.tests.datalake.mount_unmount_test.MountUnmountIcebergTest.test_simple_unmount.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed65-079f-4549-a7c5-cf74a8afb220 FLAKY 1/2
rptest.tests.e2e_shadow_indexing_test.ShadowIndexingWhileBusyTest.test_create_or_delete_topics_while_busy.short_retention=True.cloud_storage_type=CloudStorageType.ABS ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed65-079f-43ff-85ad-4a407350be3a FLAKY 1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=10 ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed65-079f-43ff-85ad-4a407350be3a FLAKY 1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.delete.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3 ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed65-079e-47f9-8066-e78270290712 FLAKY 1/2
rptest.tests.log_compaction_test.LogCompactionTest.compaction_stress_test.cleanup_policy=compact.key_set_cardinality=1000.storage_compaction_key_map_memory_kb=3 ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed65-079f-43ff-85ad-4a407350be3a FLAKY 1/2
rptest.tests.scaling_up_test.ScalingUpTest.test_scaling_up_with_recovered_topic ducktape https://buildkite.com/redpanda/redpanda/builds/61767#0194ed65-079f-43ff-85ad-4a407350be3a FLAKY 1/3

@bashtanov bashtanov marked this pull request as draft February 10, 2025 08:52
@bashtanov
Copy link
Contributor Author

Meh. It worked locally. Will debug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bharathv bharathv Awaiting requested review from bharathv

@andrwng andrwng Awaiting requested review from andrwng

@mmaslankaprv mmaslankaprv Awaiting requested review from mmaslankaprv

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bashtanov @vbotbuildovich