feat(benchmark): create new `benchmark_test` test type

[LouisTsai-Csie]

Background

Following discussions around general benchmark refactoring and the transaction gas limit cap issue, the ZkEVM team, Nethermind team (for gas benchmarking purpose),along with other teams, agreed that using the blockchain_test format is sufficient. This aligns with zkEVM’s goal of proving entire blocks rather than individual transactions (as in state_test).

However, most existing test cases are currently in the state_test format, which requires conversion using the reth deserialization tools. Additionally, EIP-7825 has been introduced in Fusaka, which enforces a maximum transaction gas limit of 2^24. This requires us to rewrite a lot of benchmark tests. Since the test must be split into multiple parts and use the blockchain_test format when the transaction gas exceeds the cap.

Proposed Solution

Introduce a new test type: benchmark_test, designed specifically for benchmark test cases. This format would support:

• Act as a wrapper and only generates the blockchain_test format as output
• High compatibility with both blockchain_test and state_test formats
• Support EIP-7825 for transaction limit cap

Usage Scenarios

• For existing blockchain_test cases: If the blocks field is present, benchmark_test will execute using the blockchain_test format:

blockchain_test(
    genesis_environment=Environment(),
    pre=pre,
    post=post_state,
    blocks=[Block(txs=txs)],
    exclude_full_post_state_in_output=True,
)

->

benchmark_test(
    genesis_environment=Environment(),
    pre=pre,
    post=post_state,
    blocks=[Block(txs=txs)],
    exclude_full_post_state_in_output=True,
)

• For existing state_test input: If only a single transaction is provided, benchmark_test will convert it into a blockchain_test:

state_test(
    env=Environment(),
    pre=pre,
    post={},
    tx=tx,
)

->

benchmark_test(
    env=Environment(),
    pre=pre,
    post={},
    tx=tx,
)

In both scenarios, benchmark_test will:

• Verify the gas limit specified in the transaction.
• If EIP-7825 is active, compare the transaction gas limit against the 2**24 cap:
• If the limit exceeds the cap, split the test into multiple parts accordingly.
• If EIP-7825 is not active, proceed with the original gas limit.

[spencer-tb]

Thanks for adding the issue. Dan briefly spoke to the geth team and I think they wanted to keep state_test. We could still remove it nonetheless. We should still add the following where bencmark_test is simply a wrapper around state_test/blockchain_test.

For opcodes/precompiles added in a fork <Osaka, the benchmark test wrapper would select a state test. For those >= Osaka the test would select the blockchain test variant. I think we could also write in such way where most of the logic is handled by the wrapper, so if we wanted to remove state test completely in the future it would be simple isolated change!

[marioevz]

I like the idea!

One caveat is that we would need to define benchmark_test and benchmark_state_test as different specs because the fixture formats they produce are statically defined, and have to be defined before the test function is executed. See example here.

Another comment is that, taking into account this #1895 (comment), we have to modify the test function to receive the env from the fixture, and also pass along the gas_benchmark_value from #1895 (maybe just so that, internally in benchmark_test, as a sanity check the last block is verified to not use more gas than expected?)

def test_some_benchmark(benchmark_test: BenchmarkTestFiller, gas_benchmark_value: int, env: Environment):
    ... # Test logic
    benchmark_test(
        env=env,
        pre=pre,
        post=post_state,
        gas_benchmark_value=gas_benchmark_value,
        setup_blocks=[Block(txs=setup_txs)],
        benchmark_block=Block(txs=attack_txs),
        exclude_full_post_state_in_output=True,
    )

For the state test, we can simplify and use the code_loop_precompile_call(setup_bytecode, attack_bytecode, fork) template it's normally used in benchmark state tests so benchmark_state_test generates a lot of the boilerplate code internally, more automatically and more consistently:

def test_some_state_benchmark(benchmark_state_test: BenchmarkStateTestFiller, gas_benchmark_value: int, env: Environment):
    ... # Test logic
    benchmark_state_test(
        env=env,
        pre=pre,
        post=post_state,
        gas_benchmark_value=gas_benchmark_value,
        setup_bytecode=setup_bytecode,
        attack_bytecode=attack_bytecode,
    )

Just an idea, there might be something else that works better.

[marioevz]

One the last comment on benchmark_state_test I just thought about: taking into account EIP-7825, the test format, when generating a blockchain test, should cap the tx limit and fill the block with multiple of these txs automatically.

[LouisTsai-Csie]

After reviewing the state_test again, based on my current understanding, it also generates a blockchain test fixture. Does that mean we might not need to remove state_test after all?

[spencer-tb]

After reviewing the state_test again, based on my current understanding, it also generates a blockchain test fixture. Does that mean we might not need to remove state_test after all?

Exactly so for context with fill currently the following gets generated from the defined test:

• state_test() -> state test fixture, blockchain test fixture (just the tx defined in state test wrapped in a block), blockchain engine test fixture (the block but in the engine new payload format).
• blockchain_test() -> blockchain test fixture, blockchain engine test fixture (block or blocks in the engine new payload format).

Regarding removing state test, for me the idea came to mind for simplicity. If no one was using it for benchmarking then it made sense to remove it and just use blockchain test. As some teams including geth/evmone like the state test format its worth keeping for now imo. Maybe there is also added benefit to help clients optimize other code paths for opcodes/precompiles < Osaka.

[danceratopz]

@LouisTsai-Csie @marioevz This new test type should additionally create a "benchmark contract code fixture" (or something with a better name :) ) that is required by spamoor, I create an issue here so we don't forget it:

• feat(fill,benchmark): add new contract-only fixture format for spamoor #1913