Repository navigation
Conversation
awskii
left a comment
There was a problem hiding this comment.
Blocking on one inline point: an expected-invalid payload answered SYNCING or ACCEPTED counts as rejected, so the fixture passes without being validated.
There was a problem hiding this comment.
🟡 Changes recommended
Null fixtures and missing Engine API payloads can succeed without executing tests, while unsupported forks receive the wrong exit code and queued blocks are reported before execution.
4 open issues
What changed in this PR
This PR adds an in-process Engine API test runner and aligns Erigon’s fixture runners with the shared client interface, allowing EEST and fuzzing tools to use consistent arguments and results.
Changes:
- Adds
evm enginetest, mixed file/directory inputs, worker support, and distinct failure and incomplete-run exit codes. - Delivers fixture block access lists, adds execution controls, and reports execution paths and raw rejection errors.
- Adds regression tests and fixtures, keeps startup logs off stdout, and updates the EEST script’s exit handling.
| File | Description |
|---|---|
tools/run-eest-spec-test.sh |
Fails shards when the runner exits with incomplete input. |
node/ethconfig/config.go |
Defines the execution-path reporting callback and data. |
execution/tests/testutil/block_test_util.go |
Adds access-list delivery, rejection tracking, and shared validation helpers. |
execution/tests/testdata/engine/bal_invalid_balance_value.json |
Adds an invalid access-list balance fixture. |
execution/tests/testdata/engine/bal_cross_block_ripemd160_state_leak.json |
Adds a cross-block RIPEMD-160 state-leak fixture. |
execution/tests/execution_path_report_test.go |
Tests executor-path reporting. |
execution/tests/engine_test_runner_test.go |
Tests Engine API fixture execution and rejection handling. |
execution/tests/delivered_access_list_test.go |
Tests matching, mismatched, and malformed access lists. |
execution/stagedsync/exec3.go |
Adds execution-path reports to block dispatch. |
execution/stagedsync/exec3_serial.go |
Reports serial execution. |
execution/execmodule/execmoduletester/exec_module_tester.go |
Wires reporting into the execution test harness. |
execution/engineapi/engineapitester/engine_test_runner.go |
Implements in-process Engine API fixture execution and SYNCING retries. |
execution/engineapi/engineapitester/engine_test_runner_test.go |
Tests retry behavior and RPC error formatting. |
execution/engineapi/engine_server.go |
Exports the Engine API registration descriptor. |
execution/engineapi/engine_api_registration_test.go |
Uses the exported registration method. |
cmd/evm/staterunner_test.go |
Checks that environment warnings stay off stdout. |
cmd/evm/runner_args_test.go |
Tests path handling, partial input failures, and exit codes. |
cmd/evm/reporter.go |
Adds rejection details to JSON results. |
cmd/evm/rejections_test.go |
Tests raw rejection output from both runners. |
cmd/evm/main.go |
Registers enginetest and execution-control flags. |
cmd/evm/internal/stderrlog/stderrlog.go |
Routes initialization logs to stderr. |
cmd/evm/execpath.go |
Formats execution-path reports as JSON lines. |
cmd/evm/execpath_test.go |
Tests reporting and access-list control flags. |
cmd/evm/enginerunner.go |
Adds CLI orchestration for engine fixtures. |
cmd/evm/blockrunner.go |
Adds multi-path inputs, execution controls, and incomplete-run handling. |
Review effort: Balanced
|
|
Thanks @AskAlexSharov. Took everything except the first one.
|
1cb2fe5 to
2377d15
Compare
|
Applied updates from recent comments and rebased as it got stale with upstream 👍🏼 |
2377d15 to
ae224a3
Compare
yperbasis
left a comment
There was a problem hiding this comment.
Blocking on the inline point: malformed optional BAL data can stop a runnable block fixture with exit 2 before the block is validated.
| RlpDecoded *struct { | ||
| BlockAccessList btBlockAccessList `json:"blockAccessList"` | ||
| } `json:"rlp_decoded"` |
There was a problem hiding this comment.
[P2] Decode the optional BAL separately from the fixture
This field decodes the delivered list directly into bounded Go types. An unrepresentable value therefore fails the whole fixture before accessList() can drop the sidecar.
Reproduced with this PR's execution/tests/testdata/delivered_access_list/invalid_list.json: change a balance change's blockAccessIndex under blocks[0].rlp_decoded.blockAccessList to 0x100000000. The runner exits 2 with pass: false and empty rejections. Removing only that sidecar gives the expected block rejection and exits 0.
Decode the optional sidecar separately and drop it when conversion fails, so the block is still judged against its header as required by ethereum/execution-specs#3846. Add a regression test for this case.
There was a problem hiding this comment.
Good catch ty. The list stays raw JSON now until accessList decodes it, and one that can't be decoded gets dropped like a mismatched one. So the block is still judged against the header. Added your repro to TestRunnersReportRejections and the same edit on valid.json to TestBlockTestDropsMismatchedBlockAccessList 👍🏼
ae224a3 to
84ffc02
Compare
yperbasis
left a comment
There was a problem hiding this comment.
The rebase needs one change that Git will not flag; see inline.
84ffc02 to
f344277
Compare
Add an optional ethconfig.Sync.ExecutionPathReporter, called for every block executeBlocks runs with the path it runs on (parallel, or sequential on a single worker) and whether its access list seeds the scheduler. A node leaves it nil; test runners set it, and the exec module tester gets an option for it.
enginetest runs each blockchain_tests_engine fixture on a fresh execution module, the one blocktest uses, with an EngineServer on it, and calls NewPayloadV<n> and ForkchoiceUpdatedV<n> directly at the fixture's versions; no node, RPC server or port is started. EngineTest lives in engineapitester because testutil cannot import engineapi. blocktest and enginetest now run every path argument, decode each fixture on its own, and list every block or payload erigon rejected in each result's rejections, with erigon's own error. They exit 2 when they cannot run all of their input: a path that cannot be read, before any fixture runs, or a fixture file or fixture that cannot be decoded, reported on stderr while the rest still run. When every fixture ran, they exit 1 if one failed and 0 otherwise. tools/run-eest-spec-test.sh judges a shard from the results, not the exit status, so it is unaffected. evm also points the root logger at stderr before erigon's packages log their env overrides, so --jsonout output stays parseable.
A block test attaches a block's access list only when its hash matches the header's blockAccessListHash, as validateBALResponse checks a peer's list; otherwise the block runs without one and is judged on its header. --bal-sequential runs every block on a single exec worker. --bal-withhold (blocktest) sets IGNORE_BAL, so blocks execute without their access lists. --bal-report writes one JSON line per executed block to stderr with the path it ran on and why. blocktest names withheld and dropped lists itself, since only it knows which blocks carried one, so a block delivered without a list keeps the reason the executor gives it.
enginetest retried a newPayload or forkchoiceUpdated answering SYNCING with no limit, so a stuck server hung the run. It now fails the fixture once a call has answered SYNCING for --syncing-timeout, one minute by default, well above the slowest tests@v21.0.2 engine fixture (about 2 seconds), on which erigon never answered SYNCING.
newPayload now goes through the engine RPC handler in process, so a param it cannot decode, or one too many for the version, is answered -32602 as a CL would see it. An expected-invalid payload passes only when answered INVALID or with an RPC error, so SYNCING or ACCEPTED fails the fixture.
Exit 2 means evm could not run some input, so its tests are missing from the JSON and the shard could still pass.
Every runner now collects files with collectFiles, which returns its error and follows a symlinked directory, and blocktest and enginetest run each file through runFixtures. The one-caller helpers are inlined.
null decodes to a nil map, so the runners ran nothing and exited 0. It is now a file the run cannot read, with exit 2.
The parallel executor reports each block once, when it starts executing it, instead of when the block is queued, and names the final reason itself, so cmd/evm no longer renames single-worker to disabled.
A missing or null engineNewPayloads ran as an empty chain and could pass. It now fails to decode, and an explicit empty list still runs.
enginetest sends forkchoiceUpdated through the in-process RPC client like newPayload, records rejections with BlockTest.Reject and renders an RPC error from the error itself.
The mismatched_list fixture was valid.json with one postNonce changed, so the test now makes that change itself and the file is gone.
The fixture's list now stays raw JSON until accessList decodes it, so a value its Go types cannot hold drops the list instead of failing the fixture, and the block is still judged against its header.
A null payload or one without blockHash decoded to a zero hash, which the rejection then reported. The hash is now optional, so such a rejection carries no hash.
--workers 0, a non-positive --syncing-timeout, a bad --run or --exclude regex and a missing path exited 1, which the runner interface keeps for a run with failed fixtures. Nothing runs in these cases, so they now exit 2.
f344277 to
97049b6
Compare


This makes
evm blocktestand a newevm enginetestfollow the client test runner interface in ethereum/execution-specs#3846. The idea is that EEST fixtures and the spec fuzzer can drive every client's runners the same way and read the same results.I've been running this against a private fuzzer for about a month now and would be very nicee to upstream. I am puting up PRs for all clients aligning on this spec. Thoughts welcome... this is a lot of code on Erigon. I tried to clean it up as much as I could but would appreciate any thoughts.
What it adds:
evm enginetest, which runsblockchain_test_enginefixtures through erigon's own Engine API server in-process. It fails a fixture that is still SYNCING after--syncing-timeout.--workers,--jsonoutand any mix of fixture files and directories in one run.--bal-sequential,--bal-withholdand--bal-reportpick and report the execution path, and stagedsync reports how each block was executed.To try it: