rs_monitor_check reports snapshot_gc_ts delay for a long time

[cms-cms]

Environment

• Version: OceanBase 4.3....
  • (From bin/observer -V output)
  • Welcome to the OceanBase. Commands end with ; or \g.
  • Your OceanBase connection id is 3221561286
  • Server version: OceanBase 4.3....

Description

An rs_monitor_check alert indicates that the snapshot_gc_ts has been delayed for a long time. The reported snapshot garbage collection SCN is {val:1776401181617336000, v:0}.

[hnwyllmm]

Now the complete cause-and-effect chain is very clear.

Root cause: TxShare memory exhaustion led to continuous failures of renew_snapshot_gc_scn.

Phase 1 (11:32 ~ 12:35): HNSW vector index construction exhausted memory.

A large HNSW vector index construction transaction (txid: 17304, table: idx_ob_hnsw_bench_index_id_table) used 21+ parallel write branches (CSWorker00 ~ CSWorker210), heavily consuming the sys tenant's TxShare memory. This transaction eventually timed out and aborted at 12:33:38, but the memory was not released immediately.

During this period, the internal table transaction for renew_snapshot_gc_scn continuously failed due to row lock conflicts and transaction timeouts.

Phase 2 (12:46:21): A single, barely successful attempt.

The renew operation barely squeezed through under TxShare memory throttling, successfully updating snapshot_gc_scn once. The logs clearly show the FrzInfoDet thread being throttled.

Unit Name=TxShare
Resource Limit(MB)=464
cur_resource_hold=456868502 (~435MB, 93.7% used)
avaliable_resource=0 ← Available memory is 0!
Throttle Percentage=60
Max Duration(us)=7200000000 ← Max throttle duration of 2 hours

Phase 3 (12:50 ~ present): OB_NO_MEMORY leads to complete inability to update.

Starting from 12:50:03, every attempt directly reports OB_NO_MEMORY (-4013), unable to allocate even a 7936-byte array.

[errcode=-4013] no memory(ret=-4013, new_size=7936, block_size_=7936)

This means the FrzInfoDet thread cannot even obtain the basic memory needed to execute internal SQL, making renew_snapshot_gc_scn completely impossible to execute. Since then, only the EDIAG warning for check_snapshot_gc_scn is printed every minute, with the delay continuously increasing.

Complete Cause-and-Effect Chain

HNSW index construction (21+ parallel writes) → TxShare memory exhaustion (435/464MB)
→ Transaction timed out and aborted, but memory not released
→ TxShare throttling
→ One barely successful attempt at 12:46:21
→ OB_NO_MEMORY (-4013) from 12:50 onwards
→ renew_snapshot_gc_scn completely unable to execute
→ snapshot_gc_scn permanently stuck
→ EDIAG warnings continuously output

Solutions

1. Short-term: Need to release the sys tenant's TxShare memory. If the aborted HNSW index construction transaction has been fully cleaned up (memtable flush/compaction completed), the memory should be released automatically. Otherwise, manual triggering of minor compaction or a restart may be necessary.

2. Long-term:

   • HNSW vector index construction should not be performed on the sys tenant (tenant_id=1); user tenants should be used instead.
   • Evaluate whether the memory configuration for the sys tenant needs to be increased.
   • Consider whether renew_snapshot_gc_scn should use an independent, protected memory pool to avoid being starved by business workloads.