Skip to content

Enrichment process restarts from minID=0 after container restart, causing reprocessing of already enriched actions #1

@astrostake

Description

@astrostake

Environment:

  • Lumescope version: latest (built from master branch)
  • Deployment: Docker Compose with network_mode: host
  • Data source endpoint: self-hosted node at https://lumera-api.linknode.org
  • Database: Embedded PostgreSQL 14-alpine (via volume)
  • OS/Host: Ubuntu 22.04

Issue Description:
After the initial sync/enrichment runs successfully and reaches high action IDs (e.g., actionID ~12571 in batch 232), performing a simple docker compose down && docker compose up -d causes the action tx enricher to restart from the beginning:

  • Before restart: Processing batch 232, actionID around 12569–12571
  • After restart: Starts again from batch 1, minID=0, actionID=2,3,4,... and begins re-enriching actions

The database data itself is preserved (volume not deleted), but GetUnenrichedActions returns 50 actions again, leading to re-querying and potentially re-persisting tx data.

Expected behavior:
The enricher should have a persistent checkpoint (e.g., last processed minID or fully enriched status per action) loaded from the DB on startup, so it can resume from where it left off instead of restarting from minID=0.

Actual behavior:
Every container restart triggers full re-check/reprocess from the start, which causes unnecessary load on the endpoint and temporary DB growth spikes during re-enrichment.

Steps to Reproduce:

  1. Deploy Lumescope normally and let initial enrichment run until it reaches high action IDs (e.g., >10,000 actions).
  2. Observe logs showing high batch/actionID numbers.
  3. Run docker compose down && docker compose up -d (without removing volume).
  4. Check logs: enricher restarts from batch 1, minID=0, and begins processing low actionIDs again.

Logs Before Restart:

lumescope  | 2026/03/05 15:14:14 action tx enricher: processing action 38/50 in batch 232: actionID=12569 state=ACTION_STATE_DONE
lumescope  | 2026/03/05 15:14:14 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_registered.action_id%3D12569
lumescope  | 2026/03/05 15:14:14 searchTxsByEvent: got 1 tx_responses for event action_registered.action_id=12569
lumescope  | 2026/03/05 15:14:14 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_finalized.action_id%3D12569
lumescope  | 2026/03/05 15:14:15 searchTxsByEvent: got 1 tx_responses for event action_finalized.action_id=12569
lumescope  | 2026/03/05 15:14:15 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_approved.action_id%3D12569
lumescope  | 2026/03/05 15:14:15 action tx enricher: GetActionTransactions returned 2 txs for action 12569
lumescope  | 2026/03/05 15:14:15 action tx enricher: persisted tx for action 12569 type register
lumescope  | 2026/03/05 15:14:15 action tx enricher: persisted tx for action 12569 type finalize
lumescope  | 2026/03/05 15:14:15 action tx enricher: processing action 39/50 in batch 232: actionID=12570 state=ACTION_STATE_DONE
lumescope  | 2026/03/05 15:14:15 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_registered.action_id%3D12570
lumescope  | 2026/03/05 15:14:15 searchTxsByEvent: got 1 tx_responses for event action_registered.action_id=12570
lumescope  | 2026/03/05 15:14:15 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_finalized.action_id%3D12570
lumescope  | 2026/03/05 15:14:15 searchTxsByEvent: got 1 tx_responses for event action_finalized.action_id=12570
lumescope  | 2026/03/05 15:14:15 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_approved.action_id%3D12570
lumescope  | 2026/03/05 15:14:15 action tx enricher: GetActionTransactions returned 2 txs for action 12570
lumescope  | 2026/03/05 15:14:15 action tx enricher: persisted tx for action 12570 type register
lumescope  | 2026/03/05 15:14:15 action tx enricher: persisted tx for action 12570 type finalize

Logs After Restart:

lumescope  | server started
lumescope  | Checking for database 'lumescope'...
lumescope  | Database 'lumescope' already exists.
lumescope  | Starting LumeScope on port 18080...
lumescope  | 2026/03/05 15:58:06 LumeScope API starting on :18080
lumescope  | 2026/03/05 15:58:36 action tx enricher: starting run (minID=0)
lumescope  | 2026/03/05 15:58:36 action tx enricher: fetching batch 1 with minID=0, batchSize=50
lumescope  | 2026/03/05 15:58:36 action tx enricher: GetUnenrichedActions returned 50 actions needing enrichment
lumescope  | 2026/03/05 15:58:36 action tx enricher: processing action 1/50 in batch 1: actionID=2 state=ACTION_STATE_EXPIRED
lumescope  | 2026/03/05 15:58:36 Fetched and cached action module address: lumera1hkfcc6y0fxmhcl79xlrtjg3w9jt7hhwk96zj4h
lumescope  | 2026/03/05 15:58:36 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_registered.action_id%3D2
lumescope  | 2026/03/05 15:58:36 searchTxsByEvent: got 1 tx_responses for event action_registered.action_id=2
lumescope  | 2026/03/05 15:58:36 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_finalized.action_id%3D2
lumescope  | 2026/03/05 15:58:36 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_approved.action_id%3D2
lumescope  | 2026/03/05 15:58:36 action tx enricher: GetActionTransactions returned 1 txs for action 2
lumescope  | 2026/03/05 15:58:36 action tx enricher: persisted tx for action 2 type register
lumescope  | 2026/03/05 15:58:36 action tx enricher: processing action 2/50 in batch 1: actionID=3 state=ACTION_STATE_DONE
lumescope  | 2026/03/05 15:58:36 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_registered.action_id%3D3
lumescope  | 2026/03/05 15:58:36 searchTxsByEvent: got 1 tx_responses for event action_registered.action_id=3
lumescope  | 2026/03/05 15:58:36 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_finalized.action_id%3D3
lumescope  | 2026/03/05 15:58:36 searchTxsByEvent: got 1 tx_responses for event action_finalized.action_id=3
lumescope  | 2026/03/05 15:58:36 searchTxsByEvent: querying https://lumera-api.linknode.org/cosmos/tx/v1beta1/txs?pagination.limit=10&query=action_approved.action_id%3D3
lumescope  | 2026/03/05 15:58:36 action tx enricher: GetActionTransactions returned 2 txs for action 3
lumescope  | 2026/03/05 15:58:36 action tx enricher: persisted tx for action 3 type register
lumescope  | 2026/03/05 15:58:36 action tx enricher: persisted tx for action 3 type finalize
lumescope  | 2026/03/05 15:58:36 action tx enricher: processing action 3/50 in batch 1: actionID=4 state=ACTION_STATE_DONE

Additional Context:

  • DB size before restart: ~3.3 GB
  • DB size after restart: ~3.6 GB
  • This happens even though data is persisted (no volume removal).
  • No custom changes to code; using repo as-is.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions