Spec-to-client evidence for Ethereum execution clients. PRSpec checks Besu, go-ethereum, Nethermind and Reth source code against the EIP text they implement. It uses deterministic witnesses where a rule can be checked mechanically, and LLM analysis with adversarial verification where it cannot. Each result records the exact spec revision and client commit it was produced from.
Status of 2.0.0 (2026-11-10). Evidence Pilot is a preliminary build, not a tagged production release. Everything under Verified results was produced by commands you can rerun (VERIFICATION.md). The live Gemini experiment runs in
gemini-live.yml; its results are published indocs/evidence/only after the run has actually happened. As of 2026-11-10 the full scoreboard has run once on Meta Muse Spark; the optional Gemini scoreboard is still pending (free-tier quota, and GitHub Actions is blocked by an account billing lock). See PROJECT_STATUS.md for what is done, partial, and planned. On 2026-11-12 the Besu pilot ran end to end on a Besu fork and was recorded live (details).
Run the focused demo: PRSpec analyzed the real Besu PR #5786 before and after its published fix. The targeted Meta run flagged the known parent-vs-current blob gas issue before the fix and returned no findings on the selected requirement after it. This is one retrospective case, not a new discovery, a proven CI capability, or evidence of overall accuracy. Method and raw outputs.
The pilot workflows ran on the fork Hardisky/besu, and a fork-only
workflow recorded a live PRSpec session on a GitHub-hosted runner:
video (5 min 10 s, real-time screen capture, no cuts or overlays) ·
terminal transcript · all evidence ·
summary.
| Run | Besu code | Result (Meta muse-wonder-1.0, one joint request) |
|---|---|---|
| Fork PR #1 → pilot run | fork main + a comment-only change; 8 mapped EIP-4844 files |
FULL_MATCH, 0 findings; job green in 2 min 15 s, never blocks (input 17,953, output 8,895 tokens) |
| Replay before the #5786 fix | 19c989a6, 3 files |
5 findings. The HIGH one is the bug #5786 fixed: the header excessBlobGas is computed from the current block's blobs and would be double-counted on the next block (input 11,578, output 11,405 tokens, 131.8 s) |
| Replay after the fix | 7605351c, same 3 files |
Excess-blob-gas formula judged to match the spec; the 7 remaining items are mostly requirements that live outside the 3 supplied files (input 11,577, output 8,090 tokens, 78.6 s) |
Limits: Gemini (gemini-3.8-flash, then gemini-3.6-flash) was tried first and returned
401 API key not valid for the fork's secret, so the recorded Meta fallback produced every result. Each
case ran once, model output is not deterministic, the replay is a known public bug rather than a new
discovery, and the pilot has not run inside besu-eth/besu.
- Spec churn is real. EIP text keeps changing after clients start implementing it. In June 2023 alone, one EIP-4844 commit changed three fee-market constants. More recently, Nethermind #14030 was an implementation that kept an older EIP-8272 design after the spec moved on. A client can be correct against last month's spec and wrong against today's.
- Client diversity multiplies the work. Every spec change has to be found in four codebases in four languages. PRSpec maps each rule to the file and line in each client, so the question "which of our files does this spec PR touch?" has a reproducible answer.
- It complements, not replaces, existing testing. EEST and hive test behaviour. PRSpec reads the spec and the code before a test exists, shows reviewers where to look, and produces candidates that a human can turn into tests.
- It can run without code leaving your machine. The deterministic layer needs no model at all. The LLM
layer can use a local OpenAI-compatible server, and
--no-cloudenforces that.
| Layer | What it checks | Needs an LLM? | Status |
|---|---|---|---|
Witnesses (prspec witness) |
EIP constants and simple derived values (for example TARGET_BLOB_GAS_PER_BLOCK = target_blobs × GAS_PER_BLOB) against anchors in client code |
No | Implemented and verified for EIP-1559 (3 rules) and EIP-4844 (9 rules) on Besu, go-ethereum and Nethermind |
Fork Watch (prspec fork-watch) |
Parameter-level diff between two EIP revisions, mapped to rules and to the client files to review | No | Implemented and verified on real EIP-4844 history. Prose and algorithm changes are not detected yet |
Semantic analysis (prspec analyze, prspec diff) |
Logic and missing checks that cannot be reduced to a constant. Spec and code go to an LLM, and every finding is re-examined adversarially and grounded in the quoted spec text | Yes (Gemini, OpenAI, Meta Muse, local, Azure/Claude) | Implemented. Gemini 2.0 integration is covered by unit tests. The live run is in gemini-live.yml |
Executable witnesses (EEST-format tests checked against the EELS reference before they touch a client) are planned, not built. See ROADMAP.md.
All of the following are deterministic and key-free. Raw outputs are in
docs/evidence/2026-11-10/.
1. Cross-client witness matrix. Spec ethereum/EIPs@542e62eb. Clients: Besu 8f01c7a3,
go-ethereum c863eecb, Nethermind 8ba0b6d0. There were 34 checks: 34 matched, 0 mismatched,
0 incomplete. Two rule/client pairs (FIELD_ELEMENTS_PER_BLOB for Besu and Nethermind) have no
anchor yet and are shown as ⚪, not as passes.
EIP-1559 · EIP-4844
2. Historical spec-drift replay. The three clients' current code was checked against the EIP-4844
text before commit 2511e0bc. Exactly the three constants that the commit changed fail in all three
clients (BLOB_BASE_FEE_UPDATE_FRACTION, TARGET_BLOB_GAS_PER_BLOCK and MAX_BLOB_GAS_PER_BLOCK),
and nothing else does. Fork Watch's impact card for that commit names the same three rules and the
files to review in each client.
Replay ·
Impact card
This replay runs in one direction: current clients against an old spec. It shows that the mechanism pinpoints a spec change. It is not a claim that PRSpec would have caught a historical client bug.
3. Besu mutation scoreboard, deterministic column. 10 realistic EIP-1559/4844 deviations were
pre-registered in pilot/besu/mutations.yaml and injected in memory into
Besu 8f01c7a3. Witnesses catch 4 of 10 (M1, M2, M7 and M8, all constant changes) with file and
line. The other 6 are logic changes (a removed max(1, …), rounding, an off-by-one in
fake_exponential, a flipped sign, the fork-block base fee and the excess-blob-gas formula). No
constant check can see those, so they are what the Gemini layer is measured on.
Output
4. Engineering checks. There are 210 passing tests (2 skipped because they need a key) and ruff is
clean. A clean install outside the checkout gives prspec --version → 2.0.0. A real HTTP round trip
to a local OpenAI-compatible server passes, and so do the --no-cloud guard and the Gemini fail-closed
handling for truncated, blocked, empty and transient-error responses.
5. Besu mutation scoreboard, LLM column, Meta Muse Spark (one run, not Gemini). All 10 pre-registered
mutations, muse-spark-1.3, verification 2 rounds, 1 run, executed on the maintainer's agent machine with
the same script and pinned commits as the workflow. Pre-registered outcome: 6 confirmed, 1 flagged,
3 inconclusive, 0 missed, 0 errors. Inconclusive (M4, M8, M10) means the clean baseline already raised a
keyword-matching issue, so the rule does not count the hit. On clean Besu the model raised 33 issues in
12 files and verification refuted all 33, so there are 0 confirmed baseline false-positive candidates.
Primary-pass tokens: 226,847 in / 82,430 out (verification calls are not metered by the script yet).
Wall time 916 s. Human note, separate from the tool result: in each of the 10 mutated files, a matching
issue describes the injected deviation correctly. n = 1 run on one commit, so this is not a detection rate.
Scoreboard
6. Gemini, so far. The key works (gemini-3.6-flash and gemini-3.5-flash answer). gemini-3.8-flash
returned 503 "high demand" on every probe on 2026-11-10. A one-mutation smoke run (M3, gemini-3.6-flash,
verify 2 rounds) gave confirmed, with 2 baseline issue candidates that still need hand triage
(evidence). The full run cannot finish on the
current key: it is on the free tier, with 20 requests per day per model, and one scoreboard needs about
22 primary calls plus about 2 verification calls per raised issue.
Not verified yet: the Gemini scoreboard; CI on GitHub (every job of the 2026-11-10 push was refused before starting: "account is locked due to a billing issue"); the Docker build, which no Docker daemon was available to test; a run inside Besu's own CI; and inference against a real local model.
git clone https://github.com/Fosurero/PRSpec && cd PRSpec
python -m venv .venv && . .venv/bin/activate
pip install .
# Deterministic, no key: fetch pinned client sources and run the witness matrix
scripts/fetch_clients.sh ../clients 8f01c7a3200780b00163ab76d544e0ef6646da41 "" "" 542e62eb129517130b7facc52bbb9a1f333c711e
prspec witness --eip 4844 -t besu=../clients/besu -t go-ethereum=../clients/go-ethereum \
-t nethermind=../clients/nethermind --eips-dir ../clients/EIPs
# Impact card for a real EIP-4844 spec change
prspec fork-watch --eip 4844 --from 2511e0bc^ --to 2511e0bc --eips-dir ../clients/EIPs \
--check besu=../clients/besu
# Semantic analysis with Gemini (key from https://aistudio.google.com/apikey)
export GEMINI_API_KEY=... # never commit it
prspec models --probe gemini-3.8-flash
prspec analyze --eip 1559 --client besu --source-dir ../clients/besu --output json,markdown,htmldocker build -t prspec:2.0.0 .
docker run --rm -v "$PWD/../clients:/src:ro" prspec:2.0.0 \
witness --eip 1559 -t besu=/src/besu --eips-dir /src/EIPs
docker run --rm -e GEMINI_API_KEY -v "$PWD/../clients/besu:/src/besu:ro" prspec:2.0.0 \
analyze --eip 1559 --client besu --source-dir /src/besuThe image runs as a non-root user (uid 10001) and contains no secrets. The key is passed at run time.
Set MODEL_API_KEY securely in your environment and run:
prspec analyze --provider meta --eip 1559 --client besu --source-dir ../clients/besuThe default model is muse-spark-1.3; override it using META_MODEL or the
meta section of config.yaml. Spec and source code are sent to Meta and
usage may be billed. Never commit API keys. --no-cloud blocks this provider.
Meta supports the OpenAI-compatible Chat Completions endpoint and JSON Schema
responses: https://dev.meta.ai/docs/structured-output.
ollama serve & ollama pull qwen3-coder:30b # or vLLM / llama.cpp / LM Studio
export PRSPEC_LOCAL_BASE_URL=http://127.0.0.1:11434/v1 PRSPEC_LOCAL_MODEL=qwen3-coder:30b
prspec --no-cloud analyze --provider local --eip 1559 --client besu --source-dir ../clients/besuWith --no-cloud, PRSpec refuses Gemini, OpenAI, Meta and Azure. It also refuses a "local" URL that is not a
loopback or private address. Whether a given open-weights model is good enough has not been measured
yet. The transport is tested; the quality is not.
| Command | Purpose | Exit codes |
|---|---|---|
prspec witness --eip N -t CLIENT=DIR … |
Deterministic witness matrix | 0 match · 10 mismatch · 20 incomplete |
prspec fork-watch --eip N --from A --to B [--check CLIENT=DIR] |
Spec-change impact card | 0 · 10 a checked client disagrees · 20 incomplete |
prspec analyze --eip N --client C [--source-dir D | --ref SHA] |
LLM analysis + verification. --output json,markdown,html writes all three from one request; reports carry code commit, spec SHA-256, model version, tokens and latency |
0 · 10 findings (--fail-on findings) · 20 incomplete · 1 tool error |
prspec diff --eip N --clients a,b,c |
Cross-client comparison of LLM results | as analyze |
prspec models [--probe MODEL] |
Models visible to GEMINI_API_KEY, with a one-request probe |
|
prspec --no-cloud … |
Privacy guard for any command | |
prspec list-eips, list-files, fetch-spec, check-config, clear-cache |
Registry and config helpers |
An incomplete run always takes precedence over a findings or clean exit code: a partial analysis can never look clean.
config.yaml (also shipped inside the package) selects the provider and model. The environment
overrides it.
| Variable | Meaning |
|---|---|
GEMINI_API_KEY / GEMINI_MODEL |
Gemini key and model id (default gemini-3.8-flash) |
OPENAI_API_KEY / OPENAI_BASE_URL |
OpenAI, or an OpenAI-compatible endpoint |
PRSPEC_LOCAL_BASE_URL / PRSPEC_LOCAL_MODEL / PRSPEC_LOCAL_API_KEY |
Local provider |
PRSPEC_NO_CLOUD=1 |
Same as --no-cloud |
AZURE_AI_API_KEY / AZURE_AI_ENDPOINT / AZURE_AI_DEPLOYMENT |
Claude via Azure AI Foundry |
MODEL_API_KEY / META_MODEL |
Meta Model API key and model id (default muse-spark-1.3) |
LLM_PROVIDER |
gemini · openai · meta · local · azure |
GITHUB_TOKEN |
Optional, for GitHub rate limits when fetching remote code |
Gemini model choice. On 2026-11-10, Google's models page
listed gemini-3.8-flash and gemini-3.5-flash-lite as the recommended models for new projects. It
limited 2.5 to existing users and listed 2.0 Flash as shut down. PRSpec therefore defaults to
gemini-3.8-flash. Run prspec models to see what your key can call. The free tier has quotas, and
Google's pricing page says free-tier content may be used to improve its products. Do not send
unreleased client code through the free tier; use the paid tier or --provider local.
- This repository (
ci.yml) runs lint, tests on Python 3.10, 3.12 and 3.13, an install outside the checkout, a Docker build and smoke test, and the key-free witness matrix against Besu, go-ethereum and Nethermind at their default branches. If a client moves a constant or changes a value, the job fails with the file and line. - Besu pilot kit (
pilot/besu/) is a read-only, non-blocking workflow pair for a client repository. The PR job has no secrets; the LLM job runs fromworkflow_run. It also has a baseline config pinned to model and commit, a policy-vs-consensus suppression list, and the mutation scoreboard. It triesgemini-3.8-flash, thengemini-3.6-flash, then Metamuse-wonder-1.0, and keeps every failed attempt in the artifact and job summary. It ran end to end on a Besu fork on 2026-11-12 (above); it has not been run insidebesu-eth/besu. - Live demo recorder (
pilot/besu/live-demo/) is a fork-only workflow that screen-records a real PRSpec session on a GitHub-hosted runner and publishes the video, terminal transcript, reports and a secret-scan manifest to an evidence branch of the fork. - Gemini live experiment (
gemini-live.yml) uses the repository secretGEMINI_API_KEY. It is triggered by editingpilot/live/REQUESTonevidence-pilot, or by manual dispatch once the file is onmain.
| go-ethereum | Nethermind | Besu | Reth | |
|---|---|---|---|---|
| LLM file registry (10 EIPs) | ✅ | ✅ | ✅ | ✅ (7 EIPs) |
| Witness anchors EIP-1559 | 3/3 | 3/3 | 3/3 | — |
| Witness anchors EIP-4844 | 9/9 | 8/9 | 8/9 | — |
Limits, stated plainly:
- Witnesses check constants, not behaviour. A ✅ does not prove compliance.
- Anchors are regular expressions reviewed at the commits listed in each rulebook. When code moves, the
result becomes
⚠️ incomplete, never ✅. - EIP-4844 witnesses check Cancun values. Later forks change blob parameters (EIP-7691, BPO forks), and those are not modelled yet.
- LLM findings are candidates for human review. The verification score is an internal vote, not a probability.
- No detection rate, false-positive rate or cost is claimed. One Meta Muse run (n = 1) is published as evidence, and the Gemini scoreboard has not run yet.
These are links to public threads, with what the teams actually said:
- Nethermind #10522: PRSpec flagged
FeeCollectoragainst EIP-1559's burn. The team explained it is an optional extension for other chains, not a mainnet deviation. A contributor suggested focusing on type sizes, limits and MUST/SHOULD/MAY semantics. That suggestion shaped the witness layer. - Ethereum Foundation execution-specs #2212: maintainers advised using fork-to-fork diffs, which PRSpec now does.
- Besu #9850: a maintainer asked for a minimal GitHub Actions workflow and a reproducible baseline config. The pilot kit answers that request. Besu also clarified #10714 and #10715 as pool policy or validation done in another layer, and PRSpec's suppression list encodes that.
- go-ethereum #35263: a latent divergence for a
missing checked-request predeploy, with no mainnet impact because predeploys exist at activation.
It is fixed upstream in
26d0b2171, and EEST covers it. Write-up.
None of these is an endorsement or an adoption decision.
The EF Protocol Security wishlist lists "Differential LLMing" and "LLM Test Generation". EPF Cohort 6 has a Differential LLMing project. SpecTrum reports consensus-layer divergences, and LLM-driven differential testing was presented at Black Hat Asia 2026. PRSpec's focus is narrower and complementary: pinned, reproducible spec-to-code evidence for execution clients, with a deterministic layer that does not depend on a model.
See demo/README.md. It contains the exact commands and the video storyboard, and
explains how each frame maps to a raw artifact.
ROADMAP.md separates what is implemented, what is partial, and what is proposed and worth funding. The proposed work includes executable EEST witnesses verified against EELS, a spec-PR watcher that posts impact cards, fork-aware blob schedules, a policy atlas, and Reth anchors.
PRSpec is MIT-licensed and maintained by one developer. The milestone-based TheDAO Security Fund initiative and the Giveth project fund the roadmap. A pledge is collected only under the fund's stated conditions, and backing is not an endorsement of results. Technical evaluation and any security disclosure stay independent of funding. Suspected live vulnerabilities go through each client's private disclosure channel (SECURITY.md).
The most useful contribution from a client team is technical: run the witness matrix or the pilot on one PR, and tell us whether it saved review time or added noise.
pip install -e ".[dev]"
ruff check . && python -m pytest -qLayout: src/ holds the CLI, analyzers, fetchers, witness engine and rulebooks (src/rules/).
scripts/ holds the scoreboard and client fetch scripts, pilot/ the Besu kit and live request,
docs/evidence/ dated raw results, and demo/ the video sources. The full history is in
CHANGELOG.md.
MIT. Fosurero/PRSpec