[WIP] Iceberg custom partitioning #24838

ztlpn · 2025-01-16T14:48:59Z

Backports Required

Release Notes

none

ztlpn · 2025-01-16T14:57:36Z

/ci-repeat

ztlpn · 2025-01-16T15:51:21Z

/ci-repeat

vbotbuildovich · 2025-01-16T19:06:50Z

Retry command for Build#60852

please wait until all jobs are finished before running the slash command



/ci-repeat 1
tests/rptest/tests/alter_topic_configuration_test.py::AlterTopicConfiguration.test_set_config_from_describe
tests/rptest/tests/describe_topics_test.py::DescribeTopicsTest.test_describe_topics_with_documentation_and_types
tests/rptest/tests/topic_creation_test.py::CreateTopicsResponseTest.test_create_topic_response_configs

vbotbuildovich · 2025-01-16T19:59:33Z

CI test results

test results on build#60852

test_id	test_kind	job_url	test_status	passed
rptest.tests.alter_topic_configuration_test.AlterTopicConfiguration.test_set_config_from_describe	ducktape	https://buildkite.com/redpanda/redpanda/builds/60852#0194702a-72da-42c4-b38b-45d1193c0422	FAIL	0/1
rptest.tests.alter_topic_configuration_test.AlterTopicConfiguration.test_set_config_from_describe	ducktape	https://buildkite.com/redpanda/redpanda/builds/60852#0194702e-388f-45f2-bb77-f4604a467470	FAIL	0/1
rptest.tests.describe_topics_test.DescribeTopicsTest.test_describe_topics_with_documentation_and_types	ducktape	https://buildkite.com/redpanda/redpanda/builds/60852#0194702a-72da-42c4-b38b-45d1193c0422	FAIL	0/6
rptest.tests.describe_topics_test.DescribeTopicsTest.test_describe_topics_with_documentation_and_types	ducktape	https://buildkite.com/redpanda/redpanda/builds/60852#0194702e-388f-45f2-bb77-f4604a467470	FAIL	0/6
rptest.tests.partition_reassignments_test.PartitionReassignmentsTest.test_reassignments_kafka_cli	ducktape	https://buildkite.com/redpanda/redpanda/builds/60852#0194702a-72d8-44e2-93c0-5264addcf6df	FLAKY	3/6
rptest.tests.topic_creation_test.CreateTopicsResponseTest.test_create_topic_response_configs	ducktape	https://buildkite.com/redpanda/redpanda/builds/60852#0194702a-72d9-42e9-be03-ee4075bcecc4	FAIL	0/6
rptest.tests.topic_creation_test.CreateTopicsResponseTest.test_create_topic_response_configs	ducktape	https://buildkite.com/redpanda/redpanda/builds/60852#0194702e-388f-4755-b0e2-a14faa5035f8	FAIL	0/6

test results on build#60992

test_id	test_kind	job_url	test_status	passed
coordinator_rpunit.coordinator_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/60992#019488fe-5c35-48c0-aea5-4b2181d65e3b	FAIL	0/2

test results on build#61014

test_id	test_kind	job_url	test_status	passed
gtest_raft_rpunit.gtest_raft_rpunit	unit	https://buildkite.com/redpanda/redpanda/builds/61014#01948af3-92fa-47fb-baa2-437c1e358a48	FLAKY	1/2

test results on build#61513

test_id	test_kind	job_url	test_status	passed
rptest.tests.datalake.compaction_test.CompactionGapsTest.test_translation_no_gaps.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca5-0a81-4314-81a5-a2c99ed61cfe	FLAKY	1/5
rptest.tests.datalake.custom_partitioning_test.DatalakeCustomPartitioningTest.test_basic.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9aff-4ad1-adfe-4718bdd456c1	FAIL	0/20
rptest.tests.datalake.custom_partitioning_test.DatalakeCustomPartitioningTest.test_basic.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca5-0a81-4314-81a5-a2c99ed61cfe	FAIL	0/20
rptest.tests.datalake.datalake_dlq_test.DatalakeDLQTest.test_dlq_table_for_invalid_records.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9aff-4ad1-adfe-4718bdd456c1	FAIL	0/20
rptest.tests.datalake.datalake_dlq_test.DatalakeDLQTest.test_dlq_table_for_invalid_records.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca5-0a81-4314-81a5-a2c99ed61cfe	FAIL	0/20
rptest.tests.datalake.datalake_dlq_test.DatalakeDLQTest.test_dlq_table_for_invalid_records.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9afd-4147-9652-887cab6dcb3e	FAIL	0/20
rptest.tests.datalake.datalake_dlq_test.DatalakeDLQTest.test_dlq_table_for_mixed_records.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9aff-4ad1-adfe-4718bdd456c1	FAIL	0/20
rptest.tests.datalake.datalake_dlq_test.DatalakeDLQTest.test_dlq_table_for_mixed_records.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca5-0a81-4314-81a5-a2c99ed61cfe	FAIL	0/20
rptest.tests.datalake.datalake_dlq_test.DatalakeDLQTest.test_dlq_table_for_mixed_records.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9afd-4147-9652-887cab6dcb3e	FAIL	0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.SPARK.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9afd-4147-9652-887cab6dcb3e	FAIL	0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9aff-4ad1-adfe-4718bdd456c1	FAIL	0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_e2e_basic.cloud_storage_type=CloudStorageType.S3.query_engine=QueryEngineType.TRINO.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca5-0a81-4314-81a5-a2c99ed61cfe	FAIL	0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_iceberg_files_location.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9afd-4147-9652-887cab6dcb3e	FAIL	0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeMetricsTest.test_lag_metrics.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9aff-4ad1-adfe-4718bdd456c1	FAIL	0/20
rptest.tests.datalake.datalake_e2e_test.DatalakeMetricsTest.test_lag_metrics.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca5-0a81-4314-81a5-a2c99ed61cfe	FAIL	0/20
rptest.tests.datalake.datalake_verifier_test.DatalakeVerifierTest.test_detecting_gap_in_offset_sequence.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9afd-4147-9652-887cab6dcb3e	FAIL	0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_disabled_partitions.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9afd-4147-9652-887cab6dcb3e	FAIL	0/20
rptest.tests.datalake.recovery_mode_test.DatalakeRecoveryModeTest.test_recovery_mode.cloud_storage_type=CloudStorageType.S3.filesystem_catalog_mode=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9aff-4ad1-adfe-4718bdd456c1	FAIL	0/20
rptest.tests.datalake.rest_catalog_connection_test.RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog.cloud_storage_type=CloudStorageType.S3	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9afd-4147-9652-887cab6dcb3e	FAIL	0/20
rptest.tests.datalake.simple_connect_test.RedpandaConnectIcebergTest.test_translating_avro_serialized_records.cloud_storage_type=CloudStorageType.S3.scenario=remount	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9afe-4062-ba7d-3e595f98c61a	FAIL	0/20
rptest.tests.datalake.simple_connect_test.RedpandaConnectIcebergTest.test_translating_avro_serialized_records.cloud_storage_type=CloudStorageType.S3.scenario=remount	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca5-0a81-4314-81a5-a2c99ed61cfe	FAIL	0/20
rptest.tests.datalake.simple_connect_test.RedpandaConnectIcebergTest.test_translating_avro_serialized_records.cloud_storage_type=CloudStorageType.S3.scenario=simple	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9aff-4ad1-adfe-4718bdd456c1	FAIL	0/20
rptest.tests.datalake.transactions_test.DatalakeTransactionTests.test_with_transactions.cloud_storage_type=CloudStorageType.S3.compaction=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca0-9afe-4062-ba7d-3e595f98c61a	FAIL	0/20
rptest.tests.datalake.transactions_test.DatalakeTransactionTests.test_with_transactions.cloud_storage_type=CloudStorageType.S3.compaction=False	ducktape	https://buildkite.com/redpanda/redpanda/builds/61513#0194cca5-0a81-4314-81a5-a2c99ed61cfe	FAIL	0/20

ztlpn · 2025-01-20T12:15:56Z

/ci-repeat

ztlpn · 2025-01-20T20:22:36Z

/ci-repeat

ztlpn · 2025-01-21T13:10:54Z

/ci-repeat

ztlpn · 2025-01-21T22:18:05Z

/ci-repeat

ztlpn · 2025-01-22T18:03:01Z

/ci-repeat

vbotbuildovich · 2025-01-22T21:04:17Z

Retry command for Build#61061

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/cluster_config_test.py::ClusterConfigTest.test_valid_settings

ztlpn · 2025-01-27T19:58:01Z

/ci-repeat

ztlpn · 2025-01-27T22:31:24Z

/ci-repeat

ztlpn · 2025-02-03T15:00:35Z

/ci-repeat

vbotbuildovich · 2025-02-03T18:24:55Z

Retry command for Build#61513

please wait until all jobs are finished before running the slash command



/ci-repeat 1
tests/rptest/tests/datalake/simple_connect_test.py::RedpandaConnectIcebergTest.test_translating_avro_serialized_records@{"cloud_storage_type":1,"scenario":"remount"}
tests/rptest/tests/datalake/transactions_test.py::DatalakeTransactionTests.test_with_transactions@{"cloud_storage_type":1,"compaction":false}
tests/rptest/tests/datalake/rest_catalog_connection_test.py::RestCatalogConnectionTest.test_redpanda_connection_to_rest_catalog@{"cloud_storage_type":1}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_disabled_partitions@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"spark"}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_iceberg_files_location@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/datalake/datalake_dlq_test.py::DatalakeDLQTest.test_dlq_table_for_invalid_records@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"trino"}
tests/rptest/tests/datalake/datalake_verifier_test.py::DatalakeVerifierTest.test_detecting_gap_in_offset_sequence@{"cloud_storage_type":1}
tests/rptest/tests/datalake/datalake_dlq_test.py::DatalakeDLQTest.test_dlq_table_for_mixed_records@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"trino"}
tests/rptest/tests/datalake/custom_partitioning_test.py::DatalakeCustomPartitioningTest.test_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/datalake/simple_connect_test.py::RedpandaConnectIcebergTest.test_translating_avro_serialized_records@{"cloud_storage_type":1,"scenario":"simple"}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeMetricsTest.test_lag_metrics@{"cloud_storage_type":1}
tests/rptest/tests/datalake/recovery_mode_test.py::DatalakeRecoveryModeTest.test_recovery_mode@{"cloud_storage_type":1,"filesystem_catalog_mode":false}
tests/rptest/tests/datalake/datalake_dlq_test.py::DatalakeDLQTest.test_dlq_table_for_invalid_records@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"spark"}
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_e2e_basic@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"trino"}
tests/rptest/tests/datalake/datalake_dlq_test.py::DatalakeDLQTest.test_dlq_table_for_mixed_records@{"cloud_storage_type":1,"filesystem_catalog_mode":false,"query_engine":"spark"}

To correctly append files in case of partition spec changes, merge_append_action has to know the spec id used when preparing each file. Pass this info from the coordinator, but don't use it yet. No functional changes.

github-actions bot added area/build area/redpanda labels Jan 16, 2025

ztlpn force-pushed the iceberg-custom-partitioning-dev branch 2 times, most recently from 8a6670c to 005bb9b Compare January 16, 2025 14:55

ztlpn force-pushed the iceberg-custom-partitioning-dev branch from 005bb9b to fe9fa9b Compare January 16, 2025 15:31

ztlpn force-pushed the iceberg-custom-partitioning-dev branch from 9c7d055 to fcf181a Compare January 20, 2025 12:13

ztlpn force-pushed the iceberg-custom-partitioning-dev branch from fcf181a to 7d0f9c5 Compare January 20, 2025 20:19

ztlpn force-pushed the iceberg-custom-partitioning-dev branch from 7d0f9c5 to 1586e3c Compare January 21, 2025 13:07

ztlpn force-pushed the iceberg-custom-partitioning-dev branch from 1586e3c to 8af772e Compare January 21, 2025 22:13

ztlpn force-pushed the iceberg-custom-partitioning-dev branch from 8af772e to 9947ddd Compare January 22, 2025 17:58

ztlpn force-pushed the iceberg-custom-partitioning-dev branch from 9947ddd to 6511bed Compare January 27, 2025 19:45

ztlpn force-pushed the iceberg-custom-partitioning-dev branch from 6511bed to b928a76 Compare January 27, 2025 22:21

ztlpn force-pushed the iceberg-custom-partitioning-dev branch 5 times, most recently from 664869a to e07f2af Compare February 3, 2025 14:58

ztlpn added 8 commits February 11, 2025 02:16

TO REMOVE: print create_table catalog request json

4e1ebd6

WIP add days transform impl

35b5dc8

iceberg: add assert_default_spec_id table requirement

4ac8172

iceberg: add set_default_spec table update

09a86d9

WIP update_spec action

a5a2ea5

WIP adapt merge_append_action

47c92bb

WIP datalake: update partition spec for existing tables in ensure_schema

51d7aef

iceberg: introduce file_to_append and use it in merge_append_action

f5ce4bb

To correctly append files in case of partition spec changes, merge_append_action has to know the spec id used when preparing each file. Pass this info from the coordinator, but don't use it yet. No functional changes.

ztlpn force-pushed the iceberg-custom-partitioning-dev branch from e07f2af to f5ce4bb Compare February 11, 2025 01:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Iceberg custom partitioning #24838

[WIP] Iceberg custom partitioning #24838

ztlpn commented Jan 16, 2025

ztlpn commented Jan 16, 2025

ztlpn commented Jan 16, 2025

vbotbuildovich commented Jan 16, 2025 •

edited

Loading

vbotbuildovich commented Jan 16, 2025 •

edited

Loading

ztlpn commented Jan 20, 2025

ztlpn commented Jan 20, 2025

ztlpn commented Jan 21, 2025

ztlpn commented Jan 21, 2025

ztlpn commented Jan 22, 2025

vbotbuildovich commented Jan 22, 2025 •

edited

Loading

ztlpn commented Jan 27, 2025

ztlpn commented Jan 27, 2025

ztlpn commented Feb 3, 2025

vbotbuildovich commented Feb 3, 2025 •

edited

Loading

[WIP] Iceberg custom partitioning #24838

Are you sure you want to change the base?

[WIP] Iceberg custom partitioning #24838

Conversation

ztlpn commented Jan 16, 2025

Backports Required

Release Notes

ztlpn commented Jan 16, 2025

ztlpn commented Jan 16, 2025

vbotbuildovich commented Jan 16, 2025 • edited Loading

Retry command for Build#60852

vbotbuildovich commented Jan 16, 2025 • edited Loading

CI test results

ztlpn commented Jan 20, 2025

ztlpn commented Jan 20, 2025

ztlpn commented Jan 21, 2025

ztlpn commented Jan 21, 2025

ztlpn commented Jan 22, 2025

vbotbuildovich commented Jan 22, 2025 • edited Loading

Retry command for Build#61061

ztlpn commented Jan 27, 2025

ztlpn commented Jan 27, 2025

ztlpn commented Feb 3, 2025

vbotbuildovich commented Feb 3, 2025 • edited Loading

Retry command for Build#61513

vbotbuildovich commented Jan 16, 2025 •

edited

Loading

vbotbuildovich commented Jan 16, 2025 •

edited

Loading

vbotbuildovich commented Jan 22, 2025 •

edited

Loading

vbotbuildovich commented Feb 3, 2025 •

edited

Loading