Skip to content

cmd/evm, execution: add enginetest and the shared test runner flags - #26178

Open
fselmo wants to merge 15 commits into
erigontech:mainfrom
fselmo:feature/evm-test-runners
Open

fselmo wants to merge 15 commits into
erigontech:mainfrom
fselmo:feature/evm-test-runners

Conversation

@fselmo

@fselmo fselmo commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

This makes evm blocktest and a new evm enginetest follow the client test runner interface in ethereum/execution-specs#3846. The idea is that EEST fixtures and the spec fuzzer can drive every client's runners the same way and read the same results.

I've been running this against a private fuzzer for about a month now and would be very nicee to upstream. I am puting up PRs for all clients aligning on this spec. Thoughts welcome... this is a lot of code on Erigon. I tried to clean it up as much as I could but would appreciate any thoughts.

What it adds:

  • evm enginetest, which runs blockchain_test_engine fixtures through erigon's own Engine API server in-process. It fails a fixture that is still SYNCING after --syncing-timeout.
  • --workers, --jsonout and any mix of fixture files and directories in one run.
  • Block access lists from the fixture are delivered to the block, so the parallel executor runs on the committed list as it does on a node. --bal-sequential, --bal-withhold and --bal-report pick and report the execution path, and stagedsync reports how each block was executed.
  • Each rejected block is reported with erigon's own error, so EEST's exception mapper can check the reason. Exit 0 when everything passes, 1 on a failing fixture, 2 when not everything could run.

To try it:

go build ./cmd/evm
./evm blocktest --jsonout --workers 4 --bal-report fixtures/blockchain_tests/for_amsterdam
./evm enginetest --jsonout --workers 4 --bal-sequential fixtures/blockchain_tests_engine/for_amsterdam

@awskii awskii left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking on one inline point: an expected-invalid payload answered SYNCING or ACCEPTED counts as rejected, so the fixture passes without being validated.

Comment thread tools/run-eest-spec-test.sh Resolved
Comment thread execution/stagedsync/exec3.go Resolved
@yperbasis yperbasis added this to the 3.9.0 milestone Nov 12, 2026
@yperbasis
yperbasis requested a balanced review from Copilot November 12, 2026 09:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Null fixtures and missing Engine API payloads can succeed without executing tests, while unsupported forks receive the wrong exit code and queued blocks are reported before execution.

4 open issues
What changed in this PR

This PR adds an in-process Engine API test runner and aligns Erigon’s fixture runners with the shared client interface, allowing EEST and fuzzing tools to use consistent arguments and results.

Changes:

  • Adds evm enginetest, mixed file/directory inputs, worker support, and distinct failure and incomplete-run exit codes.
  • Delivers fixture block access lists, adds execution controls, and reports execution paths and raw rejection errors.
  • Adds regression tests and fixtures, keeps startup logs off stdout, and updates the EEST script’s exit handling.
File Description
tools/​run-eest-spec-test.sh Fails shards when the runner exits with incomplete input.
node/​ethconfig/​config.go Defines the execution-path reporting callback and data.
execution/​tests/​testutil/​block_test_util.go Adds access-list delivery, rejection tracking, and shared validation helpers.
execution/​tests/​testdata/​engine/​bal_invalid_balance_value.json Adds an invalid access-list balance fixture.
execution/​tests/​testdata/​engine/​bal_cross_block_ripemd160_state_leak.json Adds a cross-block RIPEMD-160 state-leak fixture.
execution/​tests/​execution_path_report_test.go Tests executor-path reporting.
execution/​tests/​engine_test_runner_test.go Tests Engine API fixture execution and rejection handling.
execution/​tests/​delivered_access_list_test.go Tests matching, mismatched, and malformed access lists.
execution/​stagedsync/​exec3.go Adds execution-path reports to block dispatch.
execution/​stagedsync/​exec3_serial.go Reports serial execution.
execution/​execmodule/​execmoduletester/​exec_module_tester.go Wires reporting into the execution test harness.
execution/​engineapi/​engineapitester/​engine_test_runner.go Implements in-process Engine API fixture execution and SYNCING retries.
execution/​engineapi/​engineapitester/​engine_test_runner_test.go Tests retry behavior and RPC error formatting.
execution/​engineapi/​engine_server.go Exports the Engine API registration descriptor.
execution/​engineapi/​engine_api_registration_test.go Uses the exported registration method.
cmd/​evm/​staterunner_test.go Checks that environment warnings stay off stdout.
cmd/​evm/​runner_args_test.go Tests path handling, partial input failures, and exit codes.
cmd/​evm/​reporter.go Adds rejection details to JSON results.
cmd/​evm/​rejections_test.go Tests raw rejection output from both runners.
cmd/​evm/​main.go Registers enginetest and execution-control flags.
cmd/​evm/​internal/​stderrlog/​stderrlog.go Routes initialization logs to stderr.
cmd/​evm/​execpath.go Formats execution-path reports as JSON lines.
cmd/​evm/​execpath_test.go Tests reporting and access-list control flags.
cmd/​evm/​enginerunner.go Adds CLI orchestration for engine fixtures.
cmd/​evm/​blockrunner.go Adds multi-path inputs, execution controls, and incomplete-run handling.

Review effort: Balanced

Comment thread cmd/evm/blockrunner.go Resolved
Comment thread cmd/evm/blockrunner.go Resolved
Comment thread execution/stagedsync/exec3.go Resolved
@AskAlexSharov

Copy link
Copy Markdown
Collaborator
  1. cmd/evm/blockrunner.go L264: delete the os.Open/Close loop in collectArgFiles. It stops the whole run with exit 2 on one unreadable file. wrap already reports that file, runs the rest and exits 2.
  2. cmd/evm/blockrunner.go L306: reuse. jsonFiles is a second copy of collectFiles (L275). Add the error return and EvalSymlinks to collectFiles and keep one walker for all runners.
  3. cmd/evm/blockrunner.go L216: shrink. incompleteRun.err() has one caller. Inline it into status.
  4. cmd/evm/blockrunner.go L123: shrink. runBlockTestsParallel is a one-line wrapper; enginetest already inlines the same call (enginerunner.go L79). Inline it. The if err != nil { return err } after it (L90, enginerunner.go L82) is dead, because wrap never returns an error.
  5. cmd/evm/enginerunner.go L91: shrink. runEngineTest repeats runBlockTest (blockrunner.go L333) line by line. Only the decoded type and SyncingTimeout differ. Use one runFixtures(fname, filter, newTest) where newTest returns the fixture with its *testutil.BlockTest.
  6. cmd/evm/execpath.go L62: delete the rename of single-worker to disabled. Emit the final name in exec3.go L791. Now two layers rewrite the reason: this one and block_test_util.go L292.
  7. execution/engineapi/engineapitester/engine_test_runner.go L172: idiom. forkchoiceUpdated switches on the version and calls the srv methods directly, but newPayload goes through the in-process RPC client. Use client.CallContext(ctx, &r, "engine_forkchoiceUpdatedV"+version, state). FCU then gets the same param checks as from a CL, and the switch goes away.
  8. engine_test_runner.go L201: shrink. rpc.NewJsonErrorFromErr(err).(interface{...}) wraps the client's error a second time. Use rpcErr from errors.As at L142, plus errors.As to rpc.DataError. Drop the %v fallback: the data came from JSON, so json.Marshal cannot fail.
  9. engine_test_runner.go L103: reuse. The reject closure is a copy of BlockTest.reject (block_test_util.go L79). Export Reject and call it.
  10. execution/tests/testdata/delivered_access_list/mismatched_list.json L1345: bloat. This file is a 1406-line copy of valid.json with only the fixture name and one postNonce changed. Load valid.json in delivered_access_list_test.go, change that value in code, and delete the file.

@fselmo

fselmo commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @AskAlexSharov. Took everything except the first one.

  1. I'd keep the upfront read. The runner interface asks for a path that can't be read to fail with exit 2 before any fixture runs, so a long run doesn't find out about a bad path at the very end. Happy to revisit if you feel strongly though.
  2. collectFiles is the one walker now. It returns its error and follows a symlinked dir, and the other runners handle the error.
  3. Inlined into status.
  4. Inlined, and the dead error check is gone. incompleteRun.run does the parallel run since wrap never errors.
  5. Both go through runFixtures now.
  6. exec3 emits disabled itself and the rename in execpath.go is gone. The one in block_test_util.go stays since only the runner knows a list was withheld or dropped.
  7. FCU goes through the RPC client too and the switch is gone.
  8. Done, it takes the rpc.Error and only looks for rpc.DataError. The %v fallback is gone but I kept an err == nil check on the marshal since the linter won't let it be ignored.
  9. Exported Reject and sendPayload calls it.
  10. The test makes that one change to valid.json itself and the file is gone.

@fselmo
fselmo force-pushed the feature/evm-test-runners branch from 1cb2fe5 to 2377d15 Compare November 12, 2026 15:58
@fselmo

fselmo commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Applied updates from recent comments and rebased as it got stale with upstream 👍🏼

@fselmo
fselmo force-pushed the feature/evm-test-runners branch from 2377d15 to ae224a3 Compare November 12, 2026 18:53
@fselmo
fselmo requested a review from awskii November 12, 2026 18:53
@yperbasis
yperbasis requested a balanced review from Copilot November 13, 2026 08:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the Engine fixture runner and shared test-runner updates.

No open issues 🎉

4 resolved since last review

Review effort: Balanced

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking on the inline point: malformed optional BAL data can stop a runnable block fixture with exit 2 before the block is validated.

Comment on lines +106 to +108
RlpDecoded *struct {
BlockAccessList btBlockAccessList `json:"blockAccessList"`
} `json:"rlp_decoded"`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Decode the optional BAL separately from the fixture

This field decodes the delivered list directly into bounded Go types. An unrepresentable value therefore fails the whole fixture before accessList() can drop the sidecar.

Reproduced with this PR's execution/tests/testdata/delivered_access_list/invalid_list.json: change a balance change's blockAccessIndex under blocks[0].rlp_decoded.blockAccessList to 0x100000000. The runner exits 2 with pass: false and empty rejections. Removing only that sidecar gives the expected block rejection and exits 0.

Decode the optional sidecar separately and drop it when conversion fails, so the block is still judged against its header as required by ethereum/execution-specs#3846. Add a regression test for this case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch ty. The list stays raw JSON now until accessList decodes it, and one that can't be decoded gets dropped like a mismatched one. So the block is still judged against the header. Added your repro to TestRunnersReportRejections and the same edit on valid.json to TestBlockTestDropsMismatchedBlockAccessList 👍🏼

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Malformed payloads with params: [null, ...] produce rejection responses containing an invented all-zero block hash instead of omitting the absent hash.

1 open issue

Review effort: Balanced

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rebase needs one change that Git will not flag; see inline.

Comment thread cmd/evm/enginerunner.go Resolved
@fselmo
fselmo force-pushed the feature/evm-test-runners branch from 84ffc02 to f344277 Compare November 16, 2026 22:41
Add an optional ethconfig.Sync.ExecutionPathReporter, called for every block executeBlocks runs with the path it runs on (parallel, or sequential on a single worker) and whether its access list seeds the scheduler. A node leaves it nil; test runners set it, and the exec module tester gets an option for it.
enginetest runs each blockchain_tests_engine fixture on a fresh execution module, the one blocktest uses, with an EngineServer on it, and calls NewPayloadV<n> and ForkchoiceUpdatedV<n> directly at the fixture's versions; no node, RPC server or port is started. EngineTest lives in engineapitester because testutil cannot import engineapi.

blocktest and enginetest now run every path argument, decode each fixture on its own, and list every block or payload erigon rejected in each result's rejections, with erigon's own error. They exit 2 when they cannot run all of their input: a path that cannot be read, before any fixture runs, or a fixture file or fixture that cannot be decoded, reported on stderr while the rest still run. When every fixture ran, they exit 1 if one failed and 0 otherwise. tools/run-eest-spec-test.sh judges a shard from the results, not the exit status, so it is unaffected. evm also points the root logger at stderr before erigon's packages log their env overrides, so --jsonout output stays parseable.
A block test attaches a block's access list only when its hash matches the header's blockAccessListHash, as validateBALResponse checks a peer's list; otherwise the block runs without one and is judged on its header.

--bal-sequential runs every block on a single exec worker. --bal-withhold (blocktest) sets IGNORE_BAL, so blocks execute without their access lists. --bal-report writes one JSON line per executed block to stderr with the path it ran on and why. blocktest names withheld and dropped lists itself, since only it knows which blocks carried one, so a block delivered without a list keeps the reason the executor gives it.
enginetest retried a newPayload or forkchoiceUpdated answering SYNCING with no limit, so a stuck server hung the run. It now fails the fixture once a call has answered SYNCING for --syncing-timeout, one minute by default, well above the slowest tests@v21.0.2 engine fixture (about 2 seconds), on which erigon never answered SYNCING.
newPayload now goes through the engine RPC handler in process, so a param it cannot decode, or one too many for the version, is answered -32602 as a CL would see it. An expected-invalid payload passes only when answered INVALID or with an RPC error, so SYNCING or ACCEPTED fails the fixture.
Exit 2 means evm could not run some input, so its tests are missing from the JSON and the shard could still pass.
Every runner now collects files with collectFiles, which returns its error and follows a symlinked directory, and blocktest and enginetest run each file through runFixtures. The one-caller helpers are inlined.
null decodes to a nil map, so the runners ran nothing and exited 0. It is now a file the run cannot read, with exit 2.
The parallel executor reports each block once, when it starts executing it, instead of when the block is queued, and names the final reason itself, so cmd/evm no longer renames single-worker to disabled.
A missing or null engineNewPayloads ran as an empty chain and could pass. It now fails to decode, and an explicit empty list still runs.
enginetest sends forkchoiceUpdated through the in-process RPC client like newPayload, records rejections with BlockTest.Reject and renders an RPC error from the error itself.
The mismatched_list fixture was valid.json with one postNonce changed, so the test now makes that change itself and the file is gone.
The fixture's list now stays raw JSON until accessList decodes it, so a value its Go types cannot hold drops the list instead of failing the fixture, and the block is still judged against its header.
A null payload or one without blockHash decoded to a zero hash, which the rejection then reported. The hash is now optional, so such a rejection carries no hash.
--workers 0, a non-positive --syncing-timeout, a bad --run or --exclude regex and a missing path exited 1, which the runner interface keeps for a run with failed fixtures. Nothing runs in these cases, so they now exit 2.
@fselmo
fselmo force-pushed the feature/evm-test-runners branch from f344277 to 97049b6 Compare November 16, 2026 23:03

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants