HBASE-28589 ServerCall.setResponse swallows IOException and leaves client without response

[JHSUYU]

Description

ServerCall.setResponse() method currently swallows IOException when creating response,
leaving client without any indication of server-side failure. This violates the RPC
protocol requirement that server should always send a response (either success or error).

This issue has existed since HBASE-14598 and continues to affect the latest branches
(3.0 and 2.6).

Background

The original fix for HBASE-14598 attempted to prevent OOM errors by checking array size
before allocation and throwing DoNotRetryIOException to stop client retries. However,
the DoNotRetryIOException never reaches the client due to a flaw in ServerCall.setResponse().

Root Cause

When IOException occurs during response creation (e.g., during cell block building):

1. The exception is caught in ServerCall.setResponse()
2. Only a warning is logged: "Exception while creating response"
3. The response is set to null
4. Client receives no response, causing connection closure and endless retries

This particularly affects scenarios where:

• hbase.ipc.server.reservoir.enabled is set to false
• BufferOverflowException occurs in ByteBufferOutputStream.checkSizeAndGrow()
• DoNotRetryIOException is thrown but never propagated to client

Changes

• Modified ServerCall.setResponse() to create a proper error response when IOException occurs
• Ensures DoNotRetryIOException and other exceptions are properly communicated to clients
• Added unit tests in TestServerCall

Impact

This fix ensures that:

• Clients receive proper error responses and take action based on the error(e.g. stop retry)

JIRA

HBASE-28589

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 38s		master passed
+1 💚	compile	3m 29s		master passed
+1 💚	checkstyle	0m 39s		master passed
+1 💚	spotbugs	1m 41s		master passed
+1 💚	spotless	0m 53s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	3m 26s		the patch passed
+1 💚	compile	3m 27s		the patch passed
+1 💚	javac	3m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 41s		the patch passed
+1 💚	spotbugs	1m 51s		the patch passed
+1 💚	hadoopcheck	12m 22s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 48s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 11s		The patch does not generate ASF License warnings.
		41m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7156/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7156
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 4c84bd35fade 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 4528557
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	84 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7156/2/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[Apache-HBase]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	4m 29s		master passed
+1 💚	compile	1m 29s		master passed
+1 💚	javadoc	0m 35s		master passed
+1 💚	shadedjars	8m 23s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 28s		the patch passed
+1 💚	compile	1m 25s		the patch passed
+1 💚	javac	1m 25s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed
+1 💚	shadedjars	8m 39s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	351m 36s		hbase-server in the patch passed.
		388m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7156/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7156
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux e44c784ae5e3 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / 4528557
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7156/2/testReport/
Max. process+thread count	4457 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7156/2/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

[JHSUYU]

cc @Apache9 @charlesconnell @rmdmattingly

[charlesconnell]

Makes sense to me. Approving but giving time for others to comment.

[JHSUYU]

Makes sense to me. Approving but giving time for others to comment.

Thanks charles! I will continue following up this issue until it is resolved.