Skip to content

HBASE-15625 Make minimum free heap memory percentage configurable #7076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

jinhyukify
Copy link
Contributor

@jinhyukify jinhyukify commented Jun 7, 2025

Jira https://issues.apache.org/jira/browse/HBASE-15625


In modern deployments with large heap sizes, there are cases where allocating a very large memstore aggressively can be beneficial for performance. (e.g., memstore=0.6, blockcache=0.3)
In such scenarios, keeping 20% of the heap free is unnecessarily conservative. To better accommodate these cases, I reduced the default minimum free heap threshold to 10% and made it configurable.

  • Two new configuration options added to make the minimum free heap memory requirement configurable:
    • hbase.regionserver.free.heap.min.size
    • hbase.regionserver.free.heap.memory.min.size
  • Default minimum free heap threshold lowered from 20% to 10%. <-- Reverted. Please see the comment.
  • Improved error messages to reflect that the minimum free heap threshold is now configurable.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

*/
public static final float HBASE_CLUSTER_MINIMUM_MEMORY_THRESHOLD = 0.2f;
public static final String HBASE_REGION_SERVER_FREE_HEAP_MEMORY_MIN_SIZE_KEY =
"hbase.regionserver.free.heap.memory.min.size";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not immediately clear to me that heap.min.size expects a fraction, whereas heap.memory.min.size expects an actual size. But I don't really have a better idea, and I can imagine you had a hard time coming up with those names anyway, given that some existing .size parameters expect an actual size, and some other .size parameters expect a fraction.

That said, do you think we need both parameters? Wouldn't the actual size (which takes precedence over the fraction value) be enough in most cases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I realize I didn't provide enough context earlier, sorry about that.

The naming of this configuration follows the convention used by the existing block cache size configurations. There are currently two settings that determine the size of the block cache: (HBASE-28600)

  • hfile.block.cache.size
  • hfile.block.cache.memory.size

While users are generally familiar with the fraction-based approach, I thought it would be helpful to introduce an alternative that accepts human-readable memory sizes (e.g., 512m, 4g). So both options are now available.

At least for memory-related configurations, such as free heap, memstore, and block cache, I believe supporting both styles (fraction and absolute size) makes sense for the near future. Would love to hear your thoughts!

(For now, this dual option is only available for block cache, but I’m planning to extend it to memstore configuration soon as well.)

Copy link
Member

@junegunn junegunn Jun 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No problem. Thanks for clarification.

  • hfile.block.cache.size
  • hfile.block.cache.memory.size

Maybe it's just me, but when I see this pair, I get the impression that size expects a fraction, whereas memory.size expects an actual value. And that made me wonder whether the absolute version of hbase.regionserver.free.heap.min.size should be named hbase.regionserver.free.heap.min.memory.size to follow the same size vs memory.size convention. But maybe there was no such convention in the first place?

I believe supporting both styles (fraction and absolute size) makes sense for the near future. Would love to hear your thoughts!

I'm not the one to make the call here, but personally, I'm not a fan of adding options just for completeness' sake, because increasing number of options tends to increase cognitive burden of the users, and as the number of possible combinations of options increases, it gets harder to reason about their combined effects. So I wanted to make sure if we definitely needed both. However, we have to admit that the number of configuration parameters of HBase is already out of control, so my point here is probably moot, because not adding a single parameter wouldn't make much difference to the project.

Anyway, the reason you and the initial reporter of the issue wanted this to be configurable was to reserve as much memory as possible for memstores and block cache when using large heap sizes. And I imagine you are more likely to use the actual size version than the fraction one, to better deal with the systems with different heap sizes and keep the safety net.

For example, let's say you have these:

  • memstore = 0.6
  • blockcache = 0.3
  • minimum unreserved size = 10GB

You can use these values across systems with different heap sizes and still be sure that at least 10GB of the heap is unreserved. However, if you use the fraction version instead and set it to 0.9, there is no guarantee of the actual unreserved size, and that is not very different from just removing the check.

So could you explain how you are going to use these parameters on your clusters?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the detailed insights 😄

And that made me wonder whether the absolute version of hbase.regionserver.free.heap.min.size should be named hbase.regionserver.free.heap.min.memory.size to follow the same size vs memory.size convention. But maybe there was no such convention in the first place?

You're right. There wasn't a clear convention initially, but following the existing blockCache configuration and using the memory.size suffix seems like a better approach.

So could you explain how you are going to use these parameters on your clusters?

I was planning to use an absolute value for the heap size as well.
The reason I first added the fraction-based version was because the original config was already managed as a fraction.

But I agree with your point. Using an absolute size is not only more explicit for users, but also provides more stability.
Unlike memStore or blockCache, where we're allocating memory, this one is about preserving a minimum free heap, so an absolute value feels more appropriate.

If we go with absolute size only, we still need to think about the default value.
I typically use at least 31GB heap per RegionServer, but for low-spec VMs (e.g. 4GB heap), something like 512m might make more sense? To avoid startup issues in smaller clusters, I’m leaning toward a conservative default for now.
I'm also considering allowing this value to be disabled (by setting it to 0) or overridden to something like 128m in conf/hbase-site.xml, especially for test environments.
Do you have any thoughts on this?

Copy link
Member

@junegunn junegunn Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using an absolute size is not only more explicit for users, but also provides more stability.

Thanks for giving my suggestion some thought. Considering the use cases and requirements, I think it's enough to offer a single parameter for setting the minimum amount of heap that should be left unreserved. Anyone who wants to bypass the check can set it to 0, and those who want to have some safety guarantee can set it to some realistic value. I believe it covers all scenarios, and having another option might distract/confuse the users.

But that's just one perspective, and I hope to hear what others think as well.

If we go with absolute size only, we still need to think about the default value.

Yeah, it's important to provide sensible defaults, but we should be extra careful when changing the existing ones or introducing one that might impact the existing use cases. Following "the principal of least surprise", I would just leave the value undefined, which means that 20% of the heap is used as the threshold as before.

I'm also considering allowing this value to be disabled (by setting it to 0)

We should still allow 0 as a valid value for the option, for users who want to bypass the check entirely. An empty value or a negative value can be used instead to indicate an undefined state.

overridden to something like 128m in conf/hbase-site.xml, especially for test environments.

While I think it's reasonable, I'm not confident that it won't impact existing user setups. In pseudo-distributed environments, users often run many processes locally, each with a relatively small heap (e.g., 512MB). In such cases, even 128m could cause their setup to stop working.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agreed with your suggestion and updated the code accordingly.
Now the configuration uses an absolute amount for setting the free heap instead of the previous fraction-based approach.
If no configuration is provided, it falls back to the default value of 0.2.

The commit is 3d2b641

Not sure what others will think, but feel free to take a look and share your thoughts.
Thanks again for your helpful input!

@Apache-HBase

This comment has been minimized.

@jinhyukify jinhyukify changed the title HBASE-15625 Make minimum free heap memory percentage configurable and lower default to 0.1 HBASE-15625 Make minimum free heap memory percentage configurable Jun 8, 2025
@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@junegunn junegunn requested a review from Copilot June 18, 2025 01:49
Copilot

This comment was marked as outdated.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
_ master Compile Tests _
+0 🆗 mvndep 0m 37s Maven dependency ordering for branch
+1 💚 mvninstall 3m 31s master passed
+1 💚 compile 4m 10s master passed
+1 💚 checkstyle 0m 53s master passed
+1 💚 spotbugs 2m 11s master passed
+0 🆗 refguide 2m 36s branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 spotless 0m 48s branch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 11s Maven dependency ordering for patch
+1 💚 mvninstall 3m 7s the patch passed
+1 💚 compile 4m 1s the patch passed
+1 💚 javac 4m 1s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 51s the patch passed
+1 💚 xmllint 0m 1s No new issues.
+1 💚 spotbugs 2m 22s the patch passed
+0 🆗 refguide 2m 11s patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚 hadoopcheck 12m 20s Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚 spotless 0m 46s patch has no errors when running spotless:check.
_ Other Tests _
+1 💚 asflicense 0m 18s The patch does not generate ASF License warnings.
49m 14s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #7076
Optional Tests dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless xmllint refguide
uname Linux 14a8b6d2c807 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e583c2b
Default Java Eclipse Adoptium-17.0.11+9
refguide https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-7076/14/yetus-general-check/output/branch-site/book.html
refguide https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-7076/14/yetus-general-check/output/patch-site/book.html
Max. process+thread count 85 (vs. ulimit of 30000)
modules C: hbase-common hbase-server U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/console
versions git=2.34.1 maven=3.9.8 spotbugs=4.7.3 xmllint=20913
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 35s Docker mode activated.
-0 ⚠️ yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗 mvndep 0m 37s Maven dependency ordering for branch
+1 💚 mvninstall 3m 34s master passed
+1 💚 compile 1m 17s master passed
+1 💚 javadoc 0m 44s master passed
+1 💚 shadedjars 6m 7s branch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 14s Maven dependency ordering for patch
+1 💚 mvninstall 3m 8s the patch passed
+1 💚 compile 1m 17s the patch passed
+1 💚 javac 1m 17s the patch passed
+1 💚 javadoc 0m 43s the patch passed
+1 💚 shadedjars 6m 5s patch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚 unit 2m 40s hbase-common in the patch passed.
+1 💚 unit 215m 12s hbase-server in the patch passed.
246m 51s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR #7076
Optional Tests javac javadoc unit compile shadedjars
uname Linux 13b27f73861b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e583c2b
Default Java Eclipse Adoptium-17.0.11+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/testReport/
Max. process+thread count 4414 (vs. ulimit of 30000)
modules C: hbase-common hbase-server U: .
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/console
versions git=2.34.1 maven=3.9.8
Powered by Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR makes the minimum free heap memory percentage configurable for HBase RegionServers, allowing deployments with large heap sizes to allocate more memory to memstore and block cache while maintaining operational stability.

Key changes:

  • Adds new configuration hbase.regionserver.free.heap.min.memory.size to make minimum free heap configurable
  • Refactors heap memory validation logic to use the configurable threshold instead of hardcoded 20%
  • Improves error messages to reflect the configurable nature of the minimum free heap requirement

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
hbase-default.xml Adds new configuration property for minimum free heap memory size
HConstants.java Updates comment to clarify the default threshold is only used when not explicitly configured
MemorySizeUtil.java Replaces hardcoded validation with configurable threshold and adds new utility methods
HeapMemoryManager.java Updates heap memory validation to use new configurable threshold
HRegionServer.java Updates method call to use renamed validation function
TestMemorySizeUtil.java Adds comprehensive test coverage for new configurable heap validation logic
Comments suppressed due to low confidence (1)

hbase-server/src/test/java/org/apache/hadoop/hbase/io/util/TestMemorySizeUtil.java:80

  • This test sets the configuration to an empty string, but there's no test coverage for when the configuration key is completely missing (null). This is an important edge case that should be tested.
    conf.set(MemorySizeUtil.HBASE_REGION_SERVER_FREE_HEAP_MIN_MEMORY_SIZE_KEY, "");

Copy link
Member

@junegunn junegunn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks for the patience.

I'll leave a comment on the Jira issue to gather any remaining feedback. It's a very old issue, so chances are slim anyone's still watching, but let's give it a couple of days before merging.

@jinhyukify
Copy link
Contributor Author

Thank you so much for your thorough review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants