MINOR: Cleanups in storage module #20087

mimaison · 2025-07-02T08:34:45Z

Cleanups including:

Java 17 syntax, record and switch
assertEquals() order
javadoc

Reviewers: Andrew Schofield [email protected], Jhen-Yung Hsu
[email protected], Ken Huang [email protected], Chia-Ping Tsai
[email protected]

m1a2st

Thanks for this PR, left some comments

m1a2st · 2025-07-02T15:01:10Z

storage/src/main/java/org/apache/kafka/storage/internals/log/LoadedLogOffsets.java

-    public final long recoveryPoint;
-    public final LogOffsetMetadata nextOffsetMetadata;
-
+public record LoadedLogOffsets(long logStartOffset, long recoveryPoint, LogOffsetMetadata nextOffsetMetadata) {
    public LoadedLogOffsets(final long logStartOffset,
                            final long recoveryPoint,
                            final LogOffsetMetadata nextOffsetMetadata) {


Should we remain equals(), hashCode(), toString() for this record?

Removed equals() and hashCode()

m1a2st · 2025-07-02T15:15:07Z

storage/src/test/java/org/apache/kafka/storage/internals/log/LogValidatorTest.java


        int i = 0;
        for (RecordBatch batch : validatedRecords.batches()) {
            assertTrue(batch.isValid());
-            assertEquals(batch.timestampType(), TimestampType.CREATE_TIME);
+            assertEquals(TimestampType.CREATE_TIME, batch.timestampType());
            maybeCheckBaseTimestamp(timestampSeq.get(0), batch);
            assertEquals(batch.maxTimestamp(), batch.maxTimestamp());


This assertion also can remove

I think there are the same issue at L365, L1811, We could update the actual value to TestUtils.toList(batch).stream().map(Record::timestamp).max(Long::compare).get(), the root cause is #16167

I also found this issue while reviewing this PR and have already filed a MINOR to fix it before the comments. #20093

Yunyung · 2025-07-02T17:23:51Z

storage/src/main/java/org/apache/kafka/storage/internals/log/LastRecord.java

-    public final short producerEpoch;
-
-    public LastRecord(OptionalLong lastDataOffset, short producerEpoch) {
+public record LastRecord(OptionalLong lastDataOffset, short producerEpoch) {


Let's remove equals and hashCode.

wernerdv · 2025-07-02T18:45:47Z

@mimaison I think you meant the 'storage' module in the PR title.

AndrewJSchofield · 2025-07-03T12:40:53Z

storage/src/test/java/org/apache/kafka/tiered/storage/utils/RecordsKeyValueMatcher.java

@@ -34,6 +34,12 @@
 import java.util.Collection;
 import java.util.Iterator;

+/**
+ * @param <R1> The type of records used to formulate the expectations.


The formatting of the type parameters in this javadoc comment is a little odd.

I blame IntelliJ :) Fixed

chia7712

@mimaison thanks for this cleanup

chia7712 · 2025-07-07T18:53:18Z

storage/src/main/java/org/apache/kafka/server/log/remote/quota/RLMQuotaManagerConfig.java

-    public int quotaWindowSizeSeconds() {
-        return quotaWindowSizeSeconds;
-    }
-
    @Override
    public String toString() {


the generated toString should be good enough I think

chia7712 · 2025-07-07T18:56:57Z

...ge/src/main/java/org/apache/kafka/storage/internals/checkpoint/CleanShutdownFileHandler.java

@@ -94,7 +94,7 @@ public OptionalLong read() {
            Content content = Json.parseStringAs(text, Content.class);
            return OptionalLong.of(content.brokerEpoch);
        } catch (Exception e) {
-            logger.debug("Fail to read the clean shutdown file in " + cleanShutdownFile.toPath() + ":" + e);
+            logger.debug("Fail to read the clean shutdown file in {}:{}", cleanShutdownFile.toPath(), e);


the last parameter could be handled well if it is an exception object. Perhaps, we could use {} instead of {}:{}?

chia7712 · 2025-07-07T18:58:49Z

storage/src/main/java/org/apache/kafka/storage/internals/log/BatchMetadata.java

-        result = 31 * result + Long.hashCode(timestamp);
-        return result;
-    }
-
    @Override
    public String toString() {


Not here as this also prints firstSeq and firstOffset which are not fields.

chia7712 · 2025-07-07T18:59:06Z

storage/src/main/java/org/apache/kafka/storage/internals/log/CompletedTxn.java

-        result = 31 * result + Boolean.hashCode(isAborted);
-        return result;
-    }
+public record CompletedTxn(long producerId, long firstOffset, long lastOffset, boolean isAborted) {

    @Override
    public String toString() {


chia7712 · 2025-07-07T19:13:50Z

storage/src/main/java/org/apache/kafka/storage/internals/log/LazyIndex.java

-        IndexValue(T index) {
-            this.index = index;
-        }
+    private record IndexValue<T extends AbstractIndex>(T index) implements IndexWrapper {


There was an argument before for the "suitable" variables in a record class. I prefer to use record class only if all variables are immutable. That can ensure the hashCode and equals are consistent. What do you think? That is a kind of personal taste, so any feedback is welcomed

Yeah it's a bit of a gray area. I'm happy to keep it as a regular class. I undid that change

chia7712 · 2025-07-12T16:38:25Z

@mimaison could you please fix the conflicts?

mimaison · 2025-07-18T16:47:01Z

Rebased on trunk. The test failure, HandlingSourceTopicDeletionIntegrationTest, seems unrelated, it's in streams. There's a Jira for it: https://issues.apache.org/jira/browse/KAFKA-19511

chia7712

@mimaison thanks for this cleanup. overall LGTM

chia7712 · 2025-07-20T16:08:49Z

storage/src/test/java/org/apache/kafka/server/log/remote/storage/RemoteLogSegmentFileset.java

            if (!file.exists()) {
                return true;
            }
            return file.delete();
        } catch (final Exception e) {
-            LOGGER.error(format("Encountered error while deleting %s", file.getAbsolutePath()));
+            LOGGER.error("Encountered error while deleting {}", file.getAbsolutePath());


Perhaps we should add the e to the logger

LOGGER.error("Encountered error while deleting {}", file.getAbsolutePath(), e);

chia7712 · 2025-07-20T16:12:59Z

storage/src/test/java/org/apache/kafka/tiered/storage/specs/ConsumableSpec.java

-    public Integer getExpectedFromSecondTierCount() {
-        return expectedFromSecondTierCount;
-    }
+public record ConsumableSpec(Long fetchOffset, Integer expectedTotalCount, Integer expectedFromSecondTierCount) {


I checked the usage of this class, and it appears using primitive types should be fine.

Yep, I switched to primitive types in a bunch of classes

chia7712 · 2025-07-20T16:13:55Z

storage/src/test/java/org/apache/kafka/tiered/storage/specs/DeletableSpec.java

-    public Integer getEventCount() {
-        return eventCount;
-    }
+public record DeletableSpec(Integer sourceBrokerId, LocalTieredStorageEvent.EventType eventType, Integer eventCount) {


chia7712 · 2025-07-20T16:14:45Z

storage/src/test/java/org/apache/kafka/tiered/storage/specs/FetchableSpec.java

-    public RemoteFetchCount getFetchCount() {
-        return fetchCount;
-    }
+public record FetchableSpec(Integer sourceBrokerId, RemoteFetchCount fetchCount) {


chia7712 · 2025-07-20T16:15:21Z

storage/src/test/java/org/apache/kafka/tiered/storage/specs/OffloadableSpec.java

-    public List<ProducerRecord<String, String>> getRecords() {
-        return records;
-    }
+public record OffloadableSpec(Integer sourceBrokerId, Integer baseOffset,


chia7712

LGTM

I will file a separate minor PR, so we can focus on merging this large one first.

chia7712 · 2025-07-30T06:49:42Z

storage/src/test/java/org/apache/kafka/tiered/storage/actions/ConsumeAction.java


            List<LocalTieredStorageEvent> events = history.getEvents(eventType, topicPartition);
            List<LocalTieredStorageEvent> eventsInScope = latestEvent
                    .map(e -> events.stream().filter(event -> event.isAfter(e)).toList())
                    .orElse(events);

-            RemoteFetchCount remoteFetchCount = remoteFetchSpec.getRemoteFetchCount();
-            RemoteFetchCount.FetchCountAndOp expectedCountAndOp;


FetchCountAndOp could be rewritten as a record class

chia7712 · 2025-07-30T07:57:13Z

storage/src/test/java/org/apache/kafka/tiered/storage/utils/LocalTieredStorageOutput.java

-            this.t1 = t1;
-            this.t2 = t2;
-        }
+    private record Tuple2<T1, T2>(T1 t1, T2 t2) {


this could be replaced by Map.Entry

chia7712 · 2025-07-30T08:01:51Z

https://issues.apache.org/jira/browse/KAFKA-19513 will handle the flaky

mimaison · 2025-07-30T08:16:28Z

Thanks!

github-actions bot added core Kafka Broker storage Pull requests that target the storage module tiered-storage Related to the Tiered Storage feature KIP-932 Queues for Kafka labels Jul 2, 2025

m1a2st reviewed Jul 2, 2025

View reviewed changes

Yunyung reviewed Jul 2, 2025

View reviewed changes

AndrewJSchofield reviewed Jul 3, 2025

View reviewed changes

mimaison force-pushed the cleanups-server branch from 7a8da3b to a5c49ae Compare July 3, 2025 13:32

mimaison changed the title ~~MINOR: Cleanups in server module~~ MINOR: Cleanups in storage module Jul 3, 2025

chia7712 reviewed Jul 7, 2025

View reviewed changes

mimaison added 3 commits July 18, 2025 15:20

MINOR: Cleanups in storage module

a91ec3b

Remove hashCode/equals + Javadoc fix

06105ba

Address feedback from Chia-Ping

ffdc1a1

mimaison force-pushed the cleanups-server branch from 79eb911 to ffdc1a1 Compare July 18, 2025 13:21

chia7712 reviewed Jul 20, 2025

View reviewed changes

Use primitive types

94e0b2f

chia7712 approved these changes Jul 30, 2025

View reviewed changes

chia7712 merged commit 6973dea into apache:trunk Jul 30, 2025
24 of 26 checks passed

mimaison deleted the cleanups-server branch July 30, 2025 08:16

MINOR: Cleanups in storage module #20087

MINOR: Cleanups in storage module #20087

Conversation

mimaison commented Jul 2, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

m1a2st left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wernerdv commented Jul 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 commented Jul 12, 2025

Uh oh!

mimaison commented Jul 18, 2025

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chia7712 commented Jul 30, 2025

Uh oh!

Uh oh!

mimaison commented Jul 30, 2025

mimaison commented Jul 2, 2025 •

edited by github-actions bot

Loading