KAFKA-19820: remove the unnecessary copy from AbstractFetch#fetchablePartitions #20745
+4
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
triage
m1a2st
reviewed
Oct 22, 2025
Comment on lines
338
to
344
| * Return the set of <em>fetchable</em> partitions, which are the set of partitions to which we are subscribed, | ||
| * but <em>excluding</em> any partitions for which we still have buffered data. The idea is that since the user | ||
| * has yet to process the data for the partition that has already been fetched, we should not go send for more data | ||
| * until the previously-fetched data has been processed. | ||
| * | ||
| * @param buffered The set of partitions we have in our buffer | ||
| * @return {@link Set} of {@link TopicPartition topic partitions} for which we should fetch data |
There was a problem hiding this comment.
Please also update the Javadoc accordingly
chia7712
reviewed
Oct 23, 2025
| // Return all partitions that are in an otherwise fetchable state *and* for which we don't already have some | ||
| // messages sitting in our buffer. | ||
| return new HashSet<>(subscriptions.fetchablePartitions(isNotBuffered)); | ||
| return subscriptions.fetchablePartitions(isNotBuffered); |
There was a problem hiding this comment.
@kirktrue do you have free cycle to take a look at this change? thanks!
kirktrue
reviewed
Oct 23, 2025
There was a problem hiding this comment.
Thanks for the PR.
Is there any reason not to make this change at SubscriptionState.fetchablePartitions()? That is...
@@ -482,9 +482,9 @@ public class SubscriptionState {
}
// Visible for testing
- public synchronized List<TopicPartition> fetchablePartitions(Predicate<TopicPartition> isAvailable) {
+ public synchronized Set<TopicPartition> fetchablePartitions(Predicate<TopicPartition> isAvailable) {
// Since this is in the hot-path for fetching, we do this instead of using java.util.stream API
- List<TopicPartition> result = new ArrayList<>();
+ Set<TopicPartition> result = new HashSet<>();
assignment.forEach((topicPartition, topicPartitionState) -> {
// Cheap check is first to avoid evaluating the predicate if possible
if ((subscriptionType.equals(SubscriptionType.AUTO_TOPICS_SHARE) || isFetchableAndSubscribed(topicPartition, topicPartitionState))As @chia7712 mentioned here, it's already a distinct set, so we could reduce ambiguity by addressing the problem at "the source."
That would still require the change to remove the duplicate set in AbstractFetch as well as a minor change to ShareConsumeRequestManager and SubscriptionStateTest.
Thoughts?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Remove the redundant copy and change the return type to List. This
change also align
ShareConsumer