Skip to content

Fix/event pagination tx watch ttl#431

Merged
hman38705 merged 2 commits intosolutions-plug:mainfrom
joel-metal:fix/event-pagination-tx-watch-ttl
Mar 30, 2026
Merged

Fix/event pagination tx watch ttl#431
hman38705 merged 2 commits intosolutions-plug:mainfrom
joel-metal:fix/event-pagination-tx-watch-ttl

Conversation

@joel-metal
Copy link
Copy Markdown
Contributor

closes #387
closes #388

fix: paginate event fetch and bound watched_txs with TTL/cap

Problem 1 — Event sync misses events on busy ledgers

fetch_events_since used a single RPC call with limit:100. On ledgers
with >100 contract events the excess events were silently dropped,
causing the sync cursor to advance past un-processed data.

Fix:

  • Introduce EVENTS_PAGE_SIZE = 100 constant.
  • Replace the single call with a pagination loop:
    · First iteration uses startLedger = from_ledger.
    · Subsequent iterations use the opaque cursor returned by the RPC.
    · Loop exits when page length < EVENTS_PAGE_SIZE (last page reached)
    or when the RPC returns no cursor.
  • RPC errors on any page still fall back to empty + rpc_errors_total
    increment, preserving the existing resilience behaviour.

Problem 2 — watched_txs grows without bound for never-finalized hashes

watched_txs was a HashSet with no eviction policy. Hashes for
transactions that never reach a terminal status (network drop, reorg,
stuck mempool) accumulated indefinitely, causing unbounded memory growth.

Fix:

  • Replace HashSet with HashMap<String, Instant> to record the
    insertion timestamp of each hash.
  • Add MAX_WATCHED_TXS = 1_000 hard cap:
    · watch_transaction checks the cap before inserting.
    · When at capacity the entry with the oldest Instant is evicted first.
  • Add WATCHED_TX_TTL = 1 hour soft TTL:
    · run_transaction_monitor calls retain() each tick.
    · Any entry whose age exceeds WATCHED_TX_TTL is removed.
  • Both eviction paths:
    · Emit tracing::warn! with the evicted hash and reason.
    · Increment cache_invalidations_total{scope="tx_watch_eviction"} so
    operators can alert on unexpected eviction rates.

Tests added

Pagination:

  • test_pagination_accumulates_more_than_100_events — 250 events parsed
  • test_pagination_stops_on_partial_page — last-page guard
  • test_pagination_cursor_advances_without_duplicates — no cross-page dups

TTL eviction:

  • test_ttl_eviction_removes_expired_entries — expired entry gone
  • test_ttl_eviction_keeps_all_fresh_entries — fresh entries kept

Cap / memory growth:

  • test_cap_evicts_oldest_when_full — size stays at MAX
  • test_memory_growth_bounded_under_burst — 3×MAX insertions
    never exceed cap

Existing tx-monitor tests updated to use the new HashMap<String, Instant>
API (contains_key / insert with Instant instead of HashSet::contains /
insert).

Event sync:
- Replace fixed limit:100 with a paginating loop in fetch_events_since;
  continues fetching while page == EVENTS_PAGE_SIZE and cursor is present
- Add EVENTS_PAGE_SIZE = 100 constant

watched_txs:
- Replace HashSet<String> with HashMap<String, Instant> to track insertion time
- Add MAX_WATCHED_TXS = 1000 cap; oldest entry evicted on overflow
- Add WATCHED_TX_TTL = 1h; stale entries purged each monitor tick
- Evictions increment cache_invalidations_total{scope=tx_watch_eviction}
  and emit a warn! log

Tests:
- Update existing tx-monitor tests for HashMap API
- Add TTL eviction unit tests
- Add cap/memory-growth tests (>MAX_WATCHED_TXS insertions stay bounded)
- Add pagination accumulation tests (>100 events, partial page, no duplicates)
@drips-wave
Copy link
Copy Markdown

drips-wave bot commented Mar 30, 2026

@joel-metal Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@hman38705 hman38705 merged commit 551d73b into solutions-plug:main Mar 30, 2026
1 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bound watched transaction set growth Paginate blockchain events fetch beyond 100-item limit

2 participants