HBASE-15625 Make minimum free heap memory percentage configurable

[jinhyukify]

Jira https://issues.apache.org/jira/browse/HBASE-15625

---

In modern deployments with large heap sizes, there are cases where allocating a very large memstore aggressively can be beneficial for performance. (e.g., memstore=0.6, blockcache=0.3)
In such scenarios, keeping 20% of the heap free is unnecessarily conservative. To better accommodate these cases, I reduced the default minimum free heap threshold to 10% and made it configurable.

• Two new configuration options added to make the minimum free heap memory requirement configurable:
  • hbase.regionserver.free.heap.min.size
  • hbase.regionserver.free.heap.memory.min.size
• Default minimum free heap threshold lowered from 20% to 10%. <-- Reverted. Please see the comment.
• Improved error messages to reflect that the minimum free heap threshold is now configurable.

[junegunn]

It's not immediately clear to me that heap.min.size expects a fraction, whereas heap.memory.min.size expects an actual size. But I don't really have a better idea, and I can imagine you had a hard time coming up with those names anyway, given that some existing .size parameters expect an actual size, and some other .size parameters expect a fraction.

That said, do you think we need both parameters? Wouldn't the actual size (which takes precedence over the fraction value) be enough in most cases?

[jinhyukify]

I realize I didn't provide enough context earlier, sorry about that.

The naming of this configuration follows the convention used by the existing block cache size configurations. There are currently two settings that determine the size of the block cache: (HBASE-28600)

• hfile.block.cache.size
• hfile.block.cache.memory.size

While users are generally familiar with the fraction-based approach, I thought it would be helpful to introduce an alternative that accepts human-readable memory sizes (e.g., 512m, 4g). So both options are now available.

At least for memory-related configurations, such as free heap, memstore, and block cache, I believe supporting both styles (fraction and absolute size) makes sense for the near future. Would love to hear your thoughts!

(For now, this dual option is only available for block cache, but I’m planning to extend it to memstore configuration soon as well.)

[junegunn]

No problem. Thanks for clarification.

• hfile.block.cache.size
• hfile.block.cache.memory.size

Maybe it's just me, but when I see this pair, I get the impression that size expects a fraction, whereas memory.size expects an actual value. And that made me wonder whether the absolute version of hbase.regionserver.free.heap.min.size should be named hbase.regionserver.free.heap.min.memory.size to follow the same size vs memory.size convention. But maybe there was no such convention in the first place?

I believe supporting both styles (fraction and absolute size) makes sense for the near future. Would love to hear your thoughts!

I'm not the one to make the call here, but personally, I'm not a fan of adding options just for completeness' sake, because increasing number of options tends to increase cognitive burden of the users, and as the number of possible combinations of options increases, it gets harder to reason about their combined effects. So I wanted to make sure if we definitely needed both. However, we have to admit that the number of configuration parameters of HBase is already out of control, so my point here is probably moot, because not adding a single parameter wouldn't make much difference to the project.

Anyway, the reason you and the initial reporter of the issue wanted this to be configurable was to reserve as much memory as possible for memstores and block cache when using large heap sizes. And I imagine you are more likely to use the actual size version than the fraction one, to better deal with the systems with different heap sizes and keep the safety net.

For example, let's say you have these:

• memstore = 0.6
• blockcache = 0.3
• minimum unreserved size = 10GB

You can use these values across systems with different heap sizes and still be sure that at least 10GB of the heap is unreserved. However, if you use the fraction version instead and set it to 0.9, there is no guarantee of the actual unreserved size, and that is not very different from just removing the check.

So could you explain how you are going to use these parameters on your clusters?

[jinhyukify]

Thanks for the detailed insights 😄

And that made me wonder whether the absolute version of hbase.regionserver.free.heap.min.size should be named hbase.regionserver.free.heap.min.memory.size to follow the same size vs memory.size convention. But maybe there was no such convention in the first place?

You're right. There wasn't a clear convention initially, but following the existing blockCache configuration and using the memory.size suffix seems like a better approach.

So could you explain how you are going to use these parameters on your clusters?

I was planning to use an absolute value for the heap size as well.
The reason I first added the fraction-based version was because the original config was already managed as a fraction.

But I agree with your point. Using an absolute size is not only more explicit for users, but also provides more stability.
Unlike memStore or blockCache, where we're allocating memory, this one is about preserving a minimum free heap, so an absolute value feels more appropriate.

If we go with absolute size only, we still need to think about the default value.
I typically use at least 31GB heap per RegionServer, but for low-spec VMs (e.g. 4GB heap), something like 512m might make more sense? To avoid startup issues in smaller clusters, I’m leaning toward a conservative default for now.
I'm also considering allowing this value to be disabled (by setting it to 0) or overridden to something like 128m in conf/hbase-site.xml, especially for test environments.
Do you have any thoughts on this?

[junegunn]

Using an absolute size is not only more explicit for users, but also provides more stability.

Thanks for giving my suggestion some thought. Considering the use cases and requirements, I think it's enough to offer a single parameter for setting the minimum amount of heap that should be left unreserved. Anyone who wants to bypass the check can set it to 0, and those who want to have some safety guarantee can set it to some realistic value. I believe it covers all scenarios, and having another option might distract/confuse the users.

But that's just one perspective, and I hope to hear what others think as well.

If we go with absolute size only, we still need to think about the default value.

Yeah, it's important to provide sensible defaults, but we should be extra careful when changing the existing ones or introducing one that might impact the existing use cases. Following "the principal of least surprise", I would just leave the value undefined, which means that 20% of the heap is used as the threshold as before.

I'm also considering allowing this value to be disabled (by setting it to 0)

We should still allow 0 as a valid value for the option, for users who want to bypass the check entirely. An empty value or a negative value can be used instead to indicate an undefined state.

overridden to something like 128m in conf/hbase-site.xml, especially for test environments.

While I think it's reasonable, I'm not confident that it won't impact existing user setups. In pseudo-distributed environments, users often run many processes locally, each with a relatively small heap (e.g., 512MB). In such cases, even 128m could cause their setup to stop working.

[jinhyukify]

I agreed with your suggestion and updated the code accordingly.
Now the configuration uses an absolute amount for setting the free heap instead of the previous fraction-based approach.
If no configuration is provided, it falls back to the default value of 0.2.

The commit is 3d2b641

Not sure what others will think, but feel free to take a look and share your thoughts.
Thanks again for your helpful input!

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 28s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 37s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 31s		master passed
+1 💚	compile	4m 10s		master passed
+1 💚	checkstyle	0m 53s		master passed
+1 💚	spotbugs	2m 11s		master passed
+0 🆗	refguide	2m 36s		branch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	spotless	0m 48s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 7s		the patch passed
+1 💚	compile	4m 1s		the patch passed
+1 💚	javac	4m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 51s		the patch passed
+1 💚	xmllint	0m 1s		No new issues.
+1 💚	spotbugs	2m 22s		the patch passed
+0 🆗	refguide	2m 11s		patch has no errors when building the reference guide. See footer for rendered docs, which you should manually inspect.
+1 💚	hadoopcheck	12m 20s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 46s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 18s		The patch does not generate ASF License warnings.
		49m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7076
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless xmllint refguide
uname	Linux 14a8b6d2c807 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / e583c2b
Default Java	Eclipse Adoptium-17.0.11+9
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-7076/14/yetus-general-check/output/branch-site/book.html
refguide	https://nightlies.apache.org/hbase/HBase-PreCommit-GitHub-PR/PR-7076/14/yetus-general-check/output/patch-site/book.html
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 37s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 34s		master passed
+1 💚	compile	1m 17s		master passed
+1 💚	javadoc	0m 44s		master passed
+1 💚	shadedjars	6m 7s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 14s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 8s		the patch passed
+1 💚	compile	1m 17s		the patch passed
+1 💚	javac	1m 17s		the patch passed
+1 💚	javadoc	0m 43s		the patch passed
+1 💚	shadedjars	6m 5s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	2m 40s		hbase-common in the patch passed.
+1 💚	unit	215m 12s		hbase-server in the patch passed.
		246m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7076
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 13b27f73861b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / e583c2b
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/testReport/
Max. process+thread count	4414 (vs. ulimit of 30000)
modules	C: hbase-common hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7076/14/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Copilot]

Pull Request Overview

This PR makes the minimum free heap memory percentage configurable for HBase RegionServers, allowing deployments with large heap sizes to allocate more memory to memstore and block cache while maintaining operational stability.

Key changes:

• Adds new configuration hbase.regionserver.free.heap.min.memory.size to make minimum free heap configurable
• Refactors heap memory validation logic to use the configurable threshold instead of hardcoded 20%
• Improves error messages to reflect the configurable nature of the minimum free heap requirement

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
hbase-default.xml	Adds new configuration property for minimum free heap memory size
HConstants.java	Updates comment to clarify the default threshold is only used when not explicitly configured
MemorySizeUtil.java	Replaces hardcoded validation with configurable threshold and adds new utility methods
HeapMemoryManager.java	Updates heap memory validation to use new configurable threshold
HRegionServer.java	Updates method call to use renamed validation function
TestMemorySizeUtil.java	Adds comprehensive test coverage for new configurable heap validation logic

Comments suppressed due to low confidence (1)

hbase-server/src/test/java/org/apache/hadoop/hbase/io/util/TestMemorySizeUtil.java:80

• This test sets the configuration to an empty string, but there's no test coverage for when the configuration key is completely missing (null). This is an important edge case that should be tested.

    conf.set(MemorySizeUtil.HBASE_REGION_SERVER_FREE_HEAP_MIN_MEMORY_SIZE_KEY, "");

[junegunn]

Looks good to me, thanks for the patience.

I'll leave a comment on the Jira issue to gather any remaining feedback. It's a very old issue, so chances are slim anyone's still watching, but let's give it a couple of days before merging.

[jinhyukify]

Thank you so much for your thorough review.