feat: add tracing support for StateSyncTx by milando12 · Pull Request #2062 · 0xPolygon/bor

milando12 · 2026-02-18T12:08:35Z

Description

Adding tracing support for StateSyncTx.

Changes

Bugfix (non-breaking change that solves an issue)
Hotfix (change that solves an urgent issue, and requires immediate attention)
New feature (non-breaking change that adds functionality)
Breaking change (change that is not backwards-compatible and/or changes current functionality)
Changes only for a subset of nodes

Checklist

I have added at least 2 reviewer or the whole pos-v1 team
I have added sufficient documentation in code
I will be resolving comments - if any - by pushing each fix in a separate commit and linking the commit hash in the comment reply
Created a task in Jira and informed the team for implementation in Erigon client (if applicable)
Includes RPC methods changes, and the Notion documentation has been updated

Cross repository changes

This PR requires changes to heimdall
- In case link the PR here:
This PR requires changes to matic-cli
- In case link the PR here:

Testing

I have added unit tests
I have added tests to CI
I have tested this code manually on local environment
I have tested this code manually on remote devnet using express-cli
I have tested this code manually on amoy
I have created new e2e tests into express-cli

Manual tests

Ran the Node with this code for few days - no problems showed up and I verified that StateSyncTx trace that we get is correct.

consensus/bor/bor.go

maoueh · 2026-02-18T17:13:10Z

consensus/bor/bor.go

 		start := time.Now()
 		cx := statefull.ChainContext{Chain: chain, Bor: c}
+
+		// Start tracing StateSyncTx (if present in the block body)


I think we might have integration tests locally (we maintain our own Polygon Live Tracing branch for now).

I'll check if I could add that on top of your PR.

That sounds great. Feel free to add them on top of this PR whenever you're ready.

- Introduced `vm.Config` to multiple functions for customizable VM tracing. - Added tracer configuration flags (`vmtrace` and `vmtrace.jsonconfig`) to streamline monitoring during blockchain synchronization. - Updated GenesisContract and spanner interfaces for trace integration. - Integrated tracing hooks for StateSyncTx events.

Remove the 4 [DEBUG] log lines that were added during development to diagnose StateSyncTx tracing issues.

When Finalize is called, the chain parameter can be a *core.HeaderChain (not just *core.BlockChain). Without this change, the tracer in vmCfg is nil during finalization, so StateSyncTx tracing hooks never fire. - Add vmConfig field and GetVMConfig() method to HeaderChain - Set hc.vmConfig in NewBlockChain after creating the HeaderChain - Add else-if branch in Finalize and FinalizeAndAssemble to extract VM config from HeaderChain

The Process function creates a tracingStateDB wrapper for tracing but was passing the unwrapped statedb to Finalize, causing the tracer to miss StateSyncTx state changes.

Expand the `CommitState` method signature to include the `vm.Config` parameter, enabling proper VM tracing integration and configuration.

…enabled

EVM struct has a comment saying never reuse

Add support for preconf and private transaction APIs - Introduce new flags and APIs for preconf and private transactions. - Add methods for transaction preconfirmation and private transaction handling in backends and API layers. - Update transaction submission logic to support preconf/private paths. - Include tests for all new functionalities and edge cases.

- Update method signatures to include `vm.Config`, enabling better VM tracing and configuration integration in testing and state processing logic.

…lper function - Consolidate duplicate code for retrieving `vm.Config` from `ChainHeaderReader`. - Simplify `Finalize` method by utilizing the new helper.

The state processor now passes the traced (hooked) StateDB to the consensus engine's Finalize method, so live tracers can observe balance changes from block reward distribution. This is the correct behavior — previously, Finalize received the raw StateDB, making block rewards invisible to live tracers like the supply tracer. Update TestSupplyRewards and TestSupplyEip1559Burn to include the expected ethash block reward (2 ETH) in their assertions, matching the new traced output.

pratikspatil024 · 2026-03-02T09:13:39Z

@claude - review this PR.

claude · 2026-03-02T09:13:59Z

Claude encountered an error —— View job

Command failed: git fetch origin --depth=20 md/feat-tracer

I'll analyze this and get back to you.

kamuikatsurgi · 2026-03-02T09:15:16Z

@claude - review this PR.

It won't be able to review this PR is from a fork, and hence it doesn't have perms to fetch the branch.

claude · 2026-03-02T09:15:39Z

Claude encountered an error —— View job

Command failed: git fetch origin --depth=20 md/feat-tracer

I'll analyze this and get back to you.

marcello33 · 2026-03-09T11:03:04Z

consensus/bor/bor.go

 		cx := statefull.ChainContext{Chain: chain, Bor: c}
+
+		// Start tracing StateSyncTx (if present in the block body)
+		if hooks := vmCfg.Tracer; hooks != nil && hooks.OnTxStart != nil {


it seems like this starts a single tx trace before checkAndCommitSpan and the CommitStates loop, but statefull.ApplyMessage does not emit per-call OnTxStart/OnTxEnd hooks. Can this cause any issue on sprint blocks (span commits and multiple state-syncs) breaking tracers that keep single tx state? If that's the case, maybe the hooks need to wrap each system call, not the whole finalize.

marcello33 · 2026-03-09T11:03:55Z

consensus/bor/bor.go

 		})
 		body.Transactions = append(body.Transactions, stateSyncTx)
-		receipts = insertStateSyncTransactionAndCalculateReceipt(stateSyncTx, header, body, state, receipts)
+		receipts = insertStateSyncTransactionAndCalculateReceipt(stateSyncTx, header, body, state, receipts, vmCfg)


Can we make the tracer lifecycle symmetric between Finalize and FinalizeAndAssemble? I think there's a difference there wrt insertStateSyncTransactionAndCalculateReceipt calls

marcello33 · 2026-03-09T11:05:34Z

cmd/cli/main.go

 	"github.com/ethereum/go-ethereum/params"
+
+	// Force-load the tracer engines to trigger registration
+	_ "github.com/ethereum/go-ethereum/eth/tracers/js"


Do we need to register js and native tracers here? IIUC, you're only leveraging LiveDirectory in this PR.

marcello33 · 2026-03-09T11:06:53Z

docs/cli/example_config.toml

+backtrace = ""                             # Request a stack trace at a specific logging statement (e.g. "block.go:271")
+debug = true                               # Prepends log messages with call-site location (file and line number)
+enable-block-tracking = false              # Enables additional logging of information collected while tracking block lifecycle
+vmtrace = "supply"                         # Name of a tracer to record internal VM operations during blockchain synchronization (costly)


Shouldn't VMTrace and VMTraceJsonConfig be top level fields? I see they’re under [log] here

marcello33 · 2026-03-09T11:08:25Z

consensus/bor/statefull/processor.go

 	return Callmsg{
 		ethereum.CallMsg{
-			From:     systemAddress,
+			From:     SystemAddress,


Can we use a getter here instead of exposing?

marcello33 · 2026-03-09T11:10:08Z

Can we ad integration tests that verify the trace output for a block containing a StateSyncTx?
Also, I think you need to solve the conflicts with base branch

…able

sonarqubecloud · 2026-03-12T08:15:50Z

Quality Gate failed

Failed conditions
5.2% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

milando12 marked this pull request as draft February 18, 2026 12:08

milando12 changed the base branch from develop to master February 18, 2026 12:14

maoueh reviewed Feb 18, 2026

View reviewed changes

milando12 added 5 commits February 19, 2026 13:17

generate mocks

7b13a88

Remove debug trace logs from StateSyncTx processing

12e21b6

Remove the 4 [DEBUG] log lines that were added during development to diagnose StateSyncTx tracing issues.

Pass tracingStateDB to Finalize in state processor

2b43baf

The Process function creates a tracingStateDB wrapper for tracing but was passing the unwrapped statedb to Finalize, causing the tracer to miss StateSyncTx state changes.

milando12 force-pushed the md/feat-tracer branch from da6e677 to 2b43baf Compare February 19, 2026 12:21

milando12 marked this pull request as ready for review February 23, 2026 15:01

milando12 added 5 commits February 23, 2026 16:19

Add vm.Config parameter to failingGenesisContract's CommitState method

afa51c2

Expand the `CommitState` method signature to include the `vm.Config` parameter, enabling proper VM tracing integration and configuration.

Refactor StateSyncTx tracing logic to instantiate only if tracing is …

ee7c122

…enabled

Remove unused comment in StateSyncTx tracing initialization logic

f1e065e

EVM struct has a comment saying never reuse

trigger PR update

9e48c5e

trigger PR update

8ddbaad

pratikspatil024 requested a review from a team February 25, 2026 11:05

pratikspatil024 changed the base branch from master to develop February 25, 2026 11:08

milando12 added 7 commits February 25, 2026 13:21

Add vm.Config parameter to methods for improved VM tracing support

fbbc506

- Update method signatures to include `vm.Config`, enabling better VM tracing and configuration integration in testing and state processing logic.

Merge remote-tracking branch 'upstream/develop' into md/feat-tracer

61ffef5

trigger PR update

8c27ab3

Refactor: Extract VM config retrieval logic into extractVMConfig he…

52a90b4

…lper function - Consolidate duplicate code for retrieving `vm.Config` from `ChainHeaderReader`. - Simplify `Finalize` method by utilizing the new helper.

Merge remote-tracking branch 'upstream/develop' into md/feat-tracer

d7d8dca

marcello33 reviewed Mar 9, 2026

View reviewed changes

milando12 added 2 commits March 11, 2026 17:04

merge upstream

ce855ca

consensus/bor: use getter for systemAddress instead of exporting vari…

bd255d4

…able

Conversation

milando12 commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Checklist

Cross repository changes

Testing

Manual tests

Uh oh!

Uh oh!

maoueh Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

milando12 Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

pratikspatil024 commented Mar 2, 2026

Uh oh!

claude bot commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kamuikatsurgi commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude bot commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcello33 Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

marcello33 Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

marcello33 Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

marcello33 Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

marcello33 Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

marcello33 commented Mar 9, 2026

Uh oh!

sonarqubecloud bot commented Mar 12, 2026

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

milando12 commented Feb 18, 2026 •

edited

Loading

claude bot commented Mar 2, 2026 •

edited

Loading

kamuikatsurgi commented Mar 2, 2026 •

edited

Loading

claude bot commented Mar 2, 2026 •

edited

Loading