HIVE-29231: Fix flaky TestEmbeddedHiveMetaStore.testDatabase by enforcing deterministic database lookup

[KangweiZhu]

What changes were proposed in this pull request?

HIVE-29231
This PR fixes a nondeterministic behavior in TestEmbeddedHiveMetaStore.testDatabase
Previously, the test used getDatabases(".*") to verify database existence. However, this method relies on cached lists which may become stale when test execution order is randomized by NonDex, which is a tool for detecting hidden assumptions in code by exploring non-deterministic behaviors of specifications that allow multiple valid implementations.

The fix is to replace getDatabases(".*") with direct getDatabase(name) calls in the test. This ensures consistent behavior regardless of test shuffling and avoids relying on potentially stale cached results.

Why are the changes needed?

The test was failing nondeterministically under NonDex with errors like:

java.lang.AssertionError: first database is not testdb1 in list: []

Root cause: Cached database lists may not reflect the actual state when randomized execution occurs.

Does this PR introduce any user-facing change?

No. I believe this improve the harness of tests without altering and damaging the core database operations being tested(add, find, drop).

How was this patch tested?

Failure observed with running NonDex:

mvn -pl standalone-metastore/metastore-server \
    edu.illinois:nondex-maven-plugin:2.2.1:nondex \
    -Dtest=org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore#testDatabase

Under environment:

Maven home: /home/anicaazhu/.sdkman/candidates/maven/current
Java version: 21.0.7, vendor: Oracle Corporation, runtime: /home/anicaazhu/.sdkman/candidates/java/21.0.7-oracle
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.16.8-arch3-1", arch: "amd64", family: "unix"

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[deniskuzZ]

i don't follow why this change is necessary. client.getDatabases(".*") should return list containing both TEST_DB1_NAME & TEST_DB2_NAME

[InvisibleProgrammer]

I was thinking about the same. The ticket description says:

The test asserted database existence using getDatabases(".*") which relies on cached database lists that may return stale results when nondex randomizes test execution. This led to nondeterministic test outcomes where databases existed but were not found in the cached list.

It is interesting. As I see, getDatabase uses cache as well:

  @Override public List<String> getDatabases(String catName, String pattern) throws MetaException {
    if (!sharedCache.isDatabaseCachePrewarmed() || (canUseEvents && rawStore.isActiveTransaction())) {
      return rawStore.getDatabases(catName, pattern);
    }
    return sharedCache.listCachedDatabases(catName, pattern);
  }

  @Override public Database getDatabase(String catName, String dbName) throws NoSuchObjectException {
    // in case of  event based cache update, cache will be updated during commit. So within active transaction, read
    // directly from rawStore to avoid reading stale data as the data updated during same transaction will not be
    // updated in the cache.
    if (!sharedCache.isDatabaseCachePrewarmed() || (canUseEvents && rawStore.isActiveTransaction())) {
      return rawStore.getDatabase(catName, dbName);
    }
    dbName = dbName.toLowerCase();
    Database db = sharedCache
        .getDatabaseFromCache(StringUtils.normalizeIdentifier(catName), StringUtils.normalizeIdentifier(dbName));
    if (db == null) {
      throw new NoSuchObjectException();
    }
    return db;
  }

[KangweiZhu]

Yes, seems cache is not the cause. Nondex appears to be changing the pattern matching behavior. We are still investigating why Nondex breaks this test.