HBASE-29291: Add a command to refresh/sync hbase:meta table #7058

Kota-SH · 2025-06-02T20:04:40Z

Jira: https://issues.apache.org/jira/browse/HBASE-29291

taklwu

LGTM and have few minor comments, let me try to use the copilot to have another round of verification

hbase-server/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

Copilot

Pull Request Overview

This pull request implements a new command to refresh/sync the hbase:meta table, addressing Jira ticket HBASE-29291. The changes include adding a refreshMeta method across various Admin interfaces and RPC endpoints, integration and unit tests for the new procedure, and corresponding protocol buffer definitions and updates.

Added refreshMeta methods in HBase admin and client classes.
Introduced new tests and integration procedures for the meta table refresh workflow.
Updated RPC and protobuf definitions to support the refreshMeta command.

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/client/ThriftAdmin.java	Added a refreshMeta stub throwing NotImplementedException
hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdmin.java	Delegated refreshMeta call to underlying admin
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestRefreshMetaProcedureIntegration.java	Integrated end-to-end test for meta refresh behavior
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestRefreshMetaProcedure.java	Added various unit tests for RefreshMetaProcedure functionality
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java	Allowed deletes on system tables during read-only mode
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java	Implemented RPC endpoint for refreshMeta
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java	Added refreshMeta implementation using RefreshMetaProcedure
hbase-server/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java	Changed deleteFromMetaTable method visibility to public
hbase-protocol-shaded/src/main/protobuf/server/master/MasterProcedure.proto	Added RefreshMeta state definitions
hbase-protocol-shaded/src/main/protobuf/server/master/Master.proto	Added RefreshMeta request/response messages
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java	Added async refreshMeta call
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncHBaseAdmin.java	Exposed async refreshMeta in client API
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncAdmin.java	Added refreshMeta to AsyncAdmin interface
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java	Exposed refreshMeta over AsyncAdmin
hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java	Added refreshMeta method in the Admin API

Comments suppressed due to low confidence (2)

hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestRefreshMetaProcedureIntegration.java:168

[nitpick] The fixed wait time of 3000 ms might be insufficient on slower environments, potentially leading to flaky tests. Consider increasing the timeout or making it configurable.

TEST_UTIL.waitFor(3000, () -> {

hbase-server/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java:745

The visibility of deleteFromMetaTable has been changed from private to public. Please confirm that exposing this method aligns with the overall API design and does not introduce unintended access.

public static void deleteFromMetaTable(final Connection connection, final List<Delete> deletes)

taklwu

nit: one more thought, we don't need to address in this PR, but assuming one cluster with 100k regions, would this refresh_meta for all tables still work?

Kota-SH · 2025-06-03T22:04:40Z

@taklwu - Thanks for the review!

nit: one more thought, we don't need to address in this PR, but assuming one cluster with 100k regions, would this refresh_meta for all tables still work?

Well, it becomes a heavy operation, and that's something we should test for.
There is another idea about periodically persisting a region to hfiles mapping in the storage, and being able to apply edits on demand based on that mapping. For now, it is just an idea that was discussed a long time ago upstream.

I'm thinking this refresh_meta implementation will be a fallback operation, in case we need to refresh_meta manually without depending on the active cluster persisting updates.

kgeisz

This looks good overall! I just have some minor comments, and I think the changes to ReadOnlyController.java can be cleaned up a bit.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java

...-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestRefreshMetaProcedure.java

kgeisz

The changes based on my feedback look good to me.

anmolnar

Please find my initial review comments below.

anmolnar · 2025-06-09T22:41:07Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+    Set<RegionInfo> latestSet = new HashSet<>(latest);
+
+    // Find regions to add (present in latest but not in current)
+    for (RegionInfo ri : latest) {


Does the loop work if "latest" is null? You checked that both cannot be the null at the same time, but one of them still can.

anmolnar · 2025-06-09T22:49:47Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+          .findFirst()
+          .orElse(null);
+
+        if (currentRegion != null && hasBoundaryChanged(currentRegion, latestRegion)) {


Does this logic actually work? You say that currentSet contains the latestRegion which - based on the COMPARATOR - means that tableName and boundaries are the same. How could they be different here?

Why don't you just rely on region IDs? You can build two hashmaps where key is RegionId and value is RegionInfo. Missing / extra regions are handled as above, comparison would go on boundaries.

anmolnar · 2025-06-09T23:01:24Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+    for (int i = 0; i < puts.size(); i += CHUNK_SIZE) {
+      int end = Math.min(puts.size(), i + CHUNK_SIZE);


You can partition a List easily with Guava or Apache Commons libraries.

(Guava)

List<List<Put>> subSets = Lists.partition(puts, CHUNK_SIZE);

https://www.baeldung.com/java-list-split

anmolnar · 2025-06-09T23:05:02Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+      for (int attempt = 1; attempt <= 3; attempt++) {
+        try {
+          MetaTableAccessor.deleteFromMetaTable(connection, chunk);
+          LOG.debug("Successfully processed delete batch {}-{}", i, end);
+          break;
+        } catch (IOException e) {
+          LOG.warn("Delete batch {}-{} failed on attempt {}/3", i, end, attempt, e);
+          if (attempt == 3) {
+            throw e;
+          }
+          try {
+            Thread.sleep(100);
+          } catch (InterruptedException ie) {
+            Thread.currentThread().interrupt();
+            throw new IOException("Interrupted during retry", ie);
+          }
+        }


The retry logic is redundant with the Put case. I'm not sure if it's already something like this available in the codebase that you can reuse, but at least you could move it to a common method.

Shall we make the number of retries and the wait time configurable?

anmolnar · 2025-06-09T23:07:27Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+    if (current == null || latest == null) {
+      LOG.warn("Cannot compare null region lists - current: {}, latest: {}",
+        current != null, latest != null);
+      return false;
+    }


You've already done this check in the calling method. I think you should either do it here or there only.

anmolnar · 2025-06-09T23:10:39Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+  /**
+   * Determines if an update is needed by comparing current and latest regions.
+   */
+  boolean needsUpdate(List<RegionInfo> current, List<RegionInfo> latest) {


I don't think you need this method at all. Scanning the two lists you will end up having a number of Puts and Deletes and if both are empty no update is needed (as you already log in the Puts.isEmpty() and Deletes.isEmpty() if branch). No need to do these scans upfront. Without using the results of the first scan, it will just doubles processing time.

hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java

anmolnar · 2025-06-10T00:46:29Z

...-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestRefreshMetaProcedure.java

+  }
+
+  @Test
+  public void testDetectBoundaryChangesInRegions() throws Exception {


I don't think this test validates the code path that you've intended for. From the logs:

2025-06-09T19:35:02,351 INFO [Time-limited test {}] procedure.RefreshMetaProcedure(284): Region mismatch: current=2, latest=2

Indicates that the HashSet has already detected the difference between the regions and no need for your further boundary check.

This is because the hash code of MutableRegionInfo is calculated based on everything in the object including region boundaries.

Change-Id: Ia04bb12cdaf580f26cb14d9a34b5963105065faa

Apache-HBase · 2025-06-24T00:33:08Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-29081 Compile Tests _
+0 🆗	mvndep	0m 44s		Maven dependency ordering for branch
+1 💚	mvninstall	4m 33s		HBASE-29081 passed
+1 💚	compile	7m 11s		HBASE-29081 passed
+1 💚	checkstyle	1m 48s		HBASE-29081 passed
+1 💚	spotbugs	8m 20s		HBASE-29081 passed
+1 💚	spotless	1m 0s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 36s		the patch passed
+1 💚	compile	6m 56s		the patch passed
+1 💚	cc	6m 56s		the patch passed
+1 💚	javac	6m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 16s	/buildtool-patch-checkstyle-hbase-server.txt	The patch fails to run checkstyle in hbase-server
-0 ⚠️	rubocop	0m 30s	/results-rubocop.txt	The patch generated 5 new + 444 unchanged - 0 fixed = 449 total (was 444)
-1 ❌	spotbugs	1m 43s	/new-spotbugs-hbase-server.html	hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	hadoopcheck	11m 59s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	hbaseprotoc	2m 11s		the patch passed
-1 ❌	spotless	0m 21s		patch has 30 errors when running spotless:check, run spotless:apply to fix.
			_ Other Tests _
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		67m 12s

Reason	Tests
SpotBugs	module:hbase-server
	org.apache.hadoop.hbase.master.procedure.RefreshMetaProcedure.compareAndUpdateRegions(Map, Map, Connection, MasterProcedureEnv) makes inefficient use of keySet iterator instead of entrySet iterator At RefreshMetaProcedure.java:inefficient use of keySet iterator instead of entrySet iterator At RefreshMetaProcedure.java:[line 175]

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7058/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7058
JIRA Issue	HBASE-29291
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless cc buflint bufcompat hbaseprotoc rubocop
uname	Linux 1ca46493f99e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29081 / `ee81220`
Default Java	Eclipse Adoptium-17.0.11+9
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7058/5/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7058/5/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 rubocop=1.37.1
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

anmolnar · 2025-06-24T01:37:37Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+
+    List<Mutation> mutations = new ArrayList<>();
+
+    for (String regionId : latestMap.keySet()) {


Iterate this way:

for (Map.Entry<String, RegionInfo> entry : latestMap.entrySet()) { String regionId = entry.getKey(); RegionInfo latestRegion = entry.getValue(); ... }

anmolnar · 2025-06-24T01:53:27Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+  private List<RegionInfo> scanRegionsInTable(FileSystem fs, List<Path> regionDirs) throws IOException {
+    List<RegionInfo> regions = new ArrayList<>();
+
+    for (Path regionDir : regionDirs) {


What's the reason for using standard single-threaded for loop here and stream parallel one level up?

I parallelized at the table level because a directory listing call might have higher latency than reading individual .regioninfo files. So I just used a loop to avoid the overhead of thread coordination.

Gotcha. Makes sense to me.

anmolnar · 2025-06-24T02:10:06Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+  private boolean isValidTableDirectory(Path path) {
+    return !(path.getName().matches("^[._-].*")) &&
+      !(path.getName().startsWith(TableName.META_TABLE_NAME.getNameAsString()));
+  }


Is there a similar function already in HBase which is the standard way of this check?

anmolnar · 2025-06-24T02:10:34Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+      !(path.getName().startsWith(TableName.META_TABLE_NAME.getNameAsString()));
+  }
+
+  private boolean isValidRegionInfo(RegionInfo regionInfo, String expectedEncodedName) {


Same here. No HBase standard tool for that?

anmolnar · 2025-06-24T02:10:56Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+    return true;
+  }
+
+  private boolean isRelevantDirectory(Path path) {


Apache-HBase · 2025-06-24T04:27:13Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 37s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ HBASE-29081 Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 49s		HBASE-29081 passed
+1 💚	compile	2m 59s		HBASE-29081 passed
+1 💚	javadoc	2m 4s		HBASE-29081 passed
+1 💚	shadedjars	6m 57s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s		Maven dependency ordering for patch
+1 💚	mvninstall	4m 12s		the patch passed
+1 💚	compile	3m 17s		the patch passed
+1 💚	javac	3m 17s		the patch passed
+1 💚	javadoc	1m 47s		the patch passed
+1 💚	shadedjars	6m 58s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	0m 42s		hbase-protocol-shaded in the patch passed.
+1 💚	unit	1m 54s		hbase-client in the patch passed.
-1 ❌	unit	244m 31s	/patch-unit-hbase-server.txt	hbase-server in the patch failed.
+1 💚	unit	6m 46s		hbase-thrift in the patch passed.
-1 ❌	unit	7m 46s	/patch-unit-hbase-shell.txt	hbase-shell in the patch failed.
		300m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7058/5/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7058
JIRA Issue	HBASE-29291
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 97f3045de941 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29081 / `ee81220`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7058/5/testReport/
Max. process+thread count	5528 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7058/5/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

ragarkar · 2025-06-25T13:05:47Z

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RefreshMetaProcedure.java

+    if (currentRegions == null || latestRegions == null) {
+      LOG.error("Can not execute update on null lists. "
+        + "Meta Table Regions - {}, Storage Regions - {}", currentRegions, latestRegions);
+      throw new IOException((currentRegions == null ? "current regions" : "latest regions") + "list is null");


nit: Add a space before "list is null". Otherwise, it will be concatenated with "current regions" or "latest regions" without a space between.

Kota-SH marked this pull request as draft June 2, 2025 20:05

Kota-SH marked this pull request as ready for review June 2, 2025 20:08

This comment has been minimized.

Sign in to view

taklwu reviewed Jun 3, 2025

View reviewed changes

taklwu requested a review from Copilot June 3, 2025 19:10

Copilot AI reviewed Jun 3, 2025

View reviewed changes

taklwu reviewed Jun 3, 2025

View reviewed changes

kgeisz suggested changes Jun 6, 2025

View reviewed changes

Kota-SH force-pushed the HBASE-29291-ref-meta branch from 6e01319 to f861d08 Compare June 9, 2025 16:06

Kota-SH requested a review from kgeisz June 9, 2025 16:37

kgeisz approved these changes Jun 9, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

anmolnar requested changes Jun 9, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

anmolnar requested changes Jun 10, 2025

View reviewed changes

Kota-SH force-pushed the HBASE-29291-ref-meta branch from f861d08 to 2fa72c4 Compare June 16, 2025 21:11

This comment has been minimized.

Sign in to view

Kota-SH force-pushed the HBASE-29291-ref-meta branch from 2fa72c4 to b9469a3 Compare June 17, 2025 15:43

This comment has been minimized.

Sign in to view

HBASE-29291: Add a command to refresh/sync hbase:meta table

ee81220

Change-Id: Ia04bb12cdaf580f26cb14d9a34b5963105065faa

Kota-SH force-pushed the HBASE-29291-ref-meta branch from b9469a3 to ee81220 Compare June 23, 2025 21:50

anmolnar reviewed Jun 24, 2025

View reviewed changes

ragarkar reviewed Jun 25, 2025

View reviewed changes

		for (int i = 0; i < puts.size(); i += CHUNK_SIZE) {
		int end = Math.min(puts.size(), i + CHUNK_SIZE);


		List<Mutation> mutations = new ArrayList<>();

		for (String regionId : latestMap.keySet()) {

HBASE-29291: Add a command to refresh/sync hbase:meta table #7058

Are you sure you want to change the base?

HBASE-29291: Add a command to refresh/sync hbase:meta table #7058

Conversation

Kota-SH commented Jun 2, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

taklwu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

taklwu left a comment

Choose a reason for hiding this comment

Uh oh!

Kota-SH commented Jun 3, 2025

Uh oh!

kgeisz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kgeisz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

anmolnar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

anmolnar Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Jun 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kgeisz left a comment •

edited

Loading

anmolnar Jun 10, 2025 •

edited

Loading