Skip to content
FosureroPublic

About

AI-powered CI checks for Ethereum spec compliance. Catches spec↔code mismatches before they hit mainnet. Built by Safi El-Hassanine | AI-powered CI guardian for Ethereum protocol specs.

Resources

Contributing

Security policy

Stars

16 stars

Watchers

0 watching

Forks

Latest commit

 

History

94 Commits

Folders and files

Repository files navigation

PRSpec

Spec-to-client evidence for Ethereum execution clients. PRSpec checks Besu, go-ethereum, Nethermind and Reth source code against the EIP text they implement. It uses deterministic witnesses where a rule can be checked mechanically, and LLM analysis with adversarial verification where it cannot. Each result records the exact spec revision and client commit it was produced from.

Python 3.10+ MIT 2.0.0

Status of 2.0.0 (2026-11-10). Evidence Pilot is a preliminary build, not a tagged production release. Everything under Verified results was produced by commands you can rerun (VERIFICATION.md). The live Gemini experiment runs in gemini-live.yml; its results are published in docs/evidence/ only after the run has actually happened. As of 2026-11-10 the full scoreboard has run once on Meta Muse Spark; the optional Gemini scoreboard is still pending (free-tier quota, and GitHub Actions is blocked by an account billing lock). See PROJECT_STATUS.md for what is done, partial, and planned. On 2026-11-12 the Besu pilot ran end to end on a Besu fork and was recorded live (details).


Besu: real historical PR replay

Run the focused demo: PRSpec analyzed the real Besu PR #5786 before and after its published fix. The targeted Meta run flagged the known parent-vs-current blob gas issue before the fix and returned no findings on the selected requirement after it. This is one retrospective case, not a new discovery, a proven CI capability, or evidence of overall accuracy. Method and raw outputs.

End-to-end run on a Besu fork (2026-11-12)

The pilot workflows ran on the fork Hardisky/besu, and a fork-only workflow recorded a live PRSpec session on a GitHub-hosted runner: video (5 min 10 s, real-time screen capture, no cuts or overlays) · terminal transcript · all evidence · summary.

Run Besu code Result (Meta muse-wonder-1.0, one joint request)
Fork PR #1 → pilot run fork main + a comment-only change; 8 mapped EIP-4844 files FULL_MATCH, 0 findings; job green in 2 min 15 s, never blocks (input 17,953, output 8,895 tokens)
Replay before the #5786 fix 19c989a6, 3 files 5 findings. The HIGH one is the bug #5786 fixed: the header excessBlobGas is computed from the current block's blobs and would be double-counted on the next block (input 11,578, output 11,405 tokens, 131.8 s)
Replay after the fix 7605351c, same 3 files Excess-blob-gas formula judged to match the spec; the 7 remaining items are mostly requirements that live outside the 3 supplied files (input 11,577, output 8,090 tokens, 78.6 s)

Limits: Gemini (gemini-3.8-flash, then gemini-3.6-flash) was tried first and returned 401 API key not valid for the fork's secret, so the recorded Meta fallback produced every result. Each case ran once, model output is not deterministic, the replay is a known public bug rather than a new discovery, and the pilot has not run inside besu-eth/besu.


Why a client team would want this

  • Spec churn is real. EIP text keeps changing after clients start implementing it. In June 2023 alone, one EIP-4844 commit changed three fee-market constants. More recently, Nethermind #14030 was an implementation that kept an older EIP-8272 design after the spec moved on. A client can be correct against last month's spec and wrong against today's.
  • Client diversity multiplies the work. Every spec change has to be found in four codebases in four languages. PRSpec maps each rule to the file and line in each client, so the question "which of our files does this spec PR touch?" has a reproducible answer.
  • It complements, not replaces, existing testing. EEST and hive test behaviour. PRSpec reads the spec and the code before a test exists, shows reviewers where to look, and produces candidates that a human can turn into tests.
  • It can run without code leaving your machine. The deterministic layer needs no model at all. The LLM layer can use a local OpenAI-compatible server, and --no-cloud enforces that.

What PRSpec does: three layers

Layer What it checks Needs an LLM? Status
Witnesses (prspec witness) EIP constants and simple derived values (for example TARGET_BLOB_GAS_PER_BLOCK = target_blobs × GAS_PER_BLOB) against anchors in client code No Implemented and verified for EIP-1559 (3 rules) and EIP-4844 (9 rules) on Besu, go-ethereum and Nethermind
Fork Watch (prspec fork-watch) Parameter-level diff between two EIP revisions, mapped to rules and to the client files to review No Implemented and verified on real EIP-4844 history. Prose and algorithm changes are not detected yet
Semantic analysis (prspec analyze, prspec diff) Logic and missing checks that cannot be reduced to a constant. Spec and code go to an LLM, and every finding is re-examined adversarially and grounded in the quoted spec text Yes (Gemini, OpenAI, Meta Muse, local, Azure/Claude) Implemented. Gemini 2.0 integration is covered by unit tests. The live run is in gemini-live.yml

Executable witnesses (EEST-format tests checked against the EELS reference before they touch a client) are planned, not built. See ROADMAP.md.


Verified results (2026-11-10)

All of the following are deterministic and key-free. Raw outputs are in docs/evidence/2026-11-10/.

1. Cross-client witness matrix. Spec ethereum/EIPs@542e62eb. Clients: Besu 8f01c7a3, go-ethereum c863eecb, Nethermind 8ba0b6d0. There were 34 checks: 34 matched, 0 mismatched, 0 incomplete. Two rule/client pairs (FIELD_ELEMENTS_PER_BLOB for Besu and Nethermind) have no anchor yet and are shown as ⚪, not as passes. EIP-1559 · EIP-4844

2. Historical spec-drift replay. The three clients' current code was checked against the EIP-4844 text before commit 2511e0bc. Exactly the three constants that the commit changed fail in all three clients (BLOB_BASE_FEE_UPDATE_FRACTION, TARGET_BLOB_GAS_PER_BLOCK and MAX_BLOB_GAS_PER_BLOCK), and nothing else does. Fork Watch's impact card for that commit names the same three rules and the files to review in each client. Replay · Impact card

This replay runs in one direction: current clients against an old spec. It shows that the mechanism pinpoints a spec change. It is not a claim that PRSpec would have caught a historical client bug.

3. Besu mutation scoreboard, deterministic column. 10 realistic EIP-1559/4844 deviations were pre-registered in pilot/besu/mutations.yaml and injected in memory into Besu 8f01c7a3. Witnesses catch 4 of 10 (M1, M2, M7 and M8, all constant changes) with file and line. The other 6 are logic changes (a removed max(1, …), rounding, an off-by-one in fake_exponential, a flipped sign, the fork-block base fee and the excess-blob-gas formula). No constant check can see those, so they are what the Gemini layer is measured on. Output

4. Engineering checks. There are 210 passing tests (2 skipped because they need a key) and ruff is clean. A clean install outside the checkout gives prspec --version → 2.0.0. A real HTTP round trip to a local OpenAI-compatible server passes, and so do the --no-cloud guard and the Gemini fail-closed handling for truncated, blocked, empty and transient-error responses.

5. Besu mutation scoreboard, LLM column, Meta Muse Spark (one run, not Gemini). All 10 pre-registered mutations, muse-spark-1.3, verification 2 rounds, 1 run, executed on the maintainer's agent machine with the same script and pinned commits as the workflow. Pre-registered outcome: 6 confirmed, 1 flagged, 3 inconclusive, 0 missed, 0 errors. Inconclusive (M4, M8, M10) means the clean baseline already raised a keyword-matching issue, so the rule does not count the hit. On clean Besu the model raised 33 issues in 12 files and verification refuted all 33, so there are 0 confirmed baseline false-positive candidates. Primary-pass tokens: 226,847 in / 82,430 out (verification calls are not metered by the script yet). Wall time 916 s. Human note, separate from the tool result: in each of the 10 mutated files, a matching issue describes the injected deviation correctly. n = 1 run on one commit, so this is not a detection rate. Scoreboard

6. Gemini, so far. The key works (gemini-3.6-flash and gemini-3.5-flash answer). gemini-3.8-flash returned 503 "high demand" on every probe on 2026-11-10. A one-mutation smoke run (M3, gemini-3.6-flash, verify 2 rounds) gave confirmed, with 2 baseline issue candidates that still need hand triage (evidence). The full run cannot finish on the current key: it is on the free tier, with 20 requests per day per model, and one scoreboard needs about 22 primary calls plus about 2 verification calls per raised issue.

Not verified yet: the Gemini scoreboard; CI on GitHub (every job of the 2026-11-10 push was refused before starting: "account is locked due to a billing issue"); the Docker build, which no Docker daemon was available to test; a run inside Besu's own CI; and inference against a real local model.


Quick start

git clone https://github.com/Fosurero/PRSpec && cd PRSpec
python -m venv .venv && . .venv/bin/activate
pip install .

# Deterministic, no key: fetch pinned client sources and run the witness matrix
scripts/fetch_clients.sh ../clients 8f01c7a3200780b00163ab76d544e0ef6646da41 "" "" 542e62eb129517130b7facc52bbb9a1f333c711e
prspec witness --eip 4844 -t besu=../clients/besu -t go-ethereum=../clients/go-ethereum \
  -t nethermind=../clients/nethermind --eips-dir ../clients/EIPs

# Impact card for a real EIP-4844 spec change
prspec fork-watch --eip 4844 --from 2511e0bc^ --to 2511e0bc --eips-dir ../clients/EIPs \
  --check besu=../clients/besu

# Semantic analysis with Gemini (key from https://aistudio.google.com/apikey)
export GEMINI_API_KEY=...            # never commit it
prspec models --probe gemini-3.8-flash
prspec analyze --eip 1559 --client besu --source-dir ../clients/besu --output json,markdown,html

Docker

docker build -t prspec:2.0.0 .
docker run --rm -v "$PWD/../clients:/src:ro" prspec:2.0.0 \
  witness --eip 1559 -t besu=/src/besu --eips-dir /src/EIPs
docker run --rm -e GEMINI_API_KEY -v "$PWD/../clients/besu:/src/besu:ro" prspec:2.0.0 \
  analyze --eip 1559 --client besu --source-dir /src/besu

The image runs as a non-root user (uid 10001) and contains no secrets. The key is passed at run time.

Meta Muse (optional cloud provider)

Set MODEL_API_KEY securely in your environment and run:

prspec analyze --provider meta --eip 1559 --client besu --source-dir ../clients/besu

The default model is muse-spark-1.3; override it using META_MODEL or the meta section of config.yaml. Spec and source code are sent to Meta and usage may be billed. Never commit API keys. --no-cloud blocks this provider. Meta supports the OpenAI-compatible Chat Completions endpoint and JSON Schema responses: https://dev.meta.ai/docs/structured-output.

Fully local: no code leaves your machine

ollama serve & ollama pull qwen3-coder:30b        # or vLLM / llama.cpp / LM Studio
export PRSPEC_LOCAL_BASE_URL=http://127.0.0.1:11434/v1 PRSPEC_LOCAL_MODEL=qwen3-coder:30b
prspec --no-cloud analyze --provider local --eip 1559 --client besu --source-dir ../clients/besu

With --no-cloud, PRSpec refuses Gemini, OpenAI, Meta and Azure. It also refuses a "local" URL that is not a loopback or private address. Whether a given open-weights model is good enough has not been measured yet. The transport is tested; the quality is not.


CLI

Command Purpose Exit codes
prspec witness --eip N -t CLIENT=DIR … Deterministic witness matrix 0 match · 10 mismatch · 20 incomplete
prspec fork-watch --eip N --from A --to B [--check CLIENT=DIR] Spec-change impact card 0 · 10 a checked client disagrees · 20 incomplete
prspec analyze --eip N --client C [--source-dir D | --ref SHA] LLM analysis + verification. --output json,markdown,html writes all three from one request; reports carry code commit, spec SHA-256, model version, tokens and latency 0 · 10 findings (--fail-on findings) · 20 incomplete · 1 tool error
prspec diff --eip N --clients a,b,c Cross-client comparison of LLM results as analyze
prspec models [--probe MODEL] Models visible to GEMINI_API_KEY, with a one-request probe
prspec --no-cloud … Privacy guard for any command
prspec list-eips, list-files, fetch-spec, check-config, clear-cache Registry and config helpers

An incomplete run always takes precedence over a findings or clean exit code: a partial analysis can never look clean.

Configuration

config.yaml (also shipped inside the package) selects the provider and model. The environment overrides it.

Variable Meaning
GEMINI_API_KEY / GEMINI_MODEL Gemini key and model id (default gemini-3.8-flash)
OPENAI_API_KEY / OPENAI_BASE_URL OpenAI, or an OpenAI-compatible endpoint
PRSPEC_LOCAL_BASE_URL / PRSPEC_LOCAL_MODEL / PRSPEC_LOCAL_API_KEY Local provider
PRSPEC_NO_CLOUD=1 Same as --no-cloud
AZURE_AI_API_KEY / AZURE_AI_ENDPOINT / AZURE_AI_DEPLOYMENT Claude via Azure AI Foundry
MODEL_API_KEY / META_MODEL Meta Model API key and model id (default muse-spark-1.3)
LLM_PROVIDER gemini · openai · meta · local · azure
GITHUB_TOKEN Optional, for GitHub rate limits when fetching remote code

Gemini model choice. On 2026-11-10, Google's models page listed gemini-3.8-flash and gemini-3.5-flash-lite as the recommended models for new projects. It limited 2.5 to existing users and listed 2.0 Flash as shut down. PRSpec therefore defaults to gemini-3.8-flash. Run prspec models to see what your key can call. The free tier has quotas, and Google's pricing page says free-tier content may be used to improve its products. Do not send unreleased client code through the free tier; use the paid tier or --provider local.


CI integration

  • This repository (ci.yml) runs lint, tests on Python 3.10, 3.12 and 3.13, an install outside the checkout, a Docker build and smoke test, and the key-free witness matrix against Besu, go-ethereum and Nethermind at their default branches. If a client moves a constant or changes a value, the job fails with the file and line.
  • Besu pilot kit (pilot/besu/) is a read-only, non-blocking workflow pair for a client repository. The PR job has no secrets; the LLM job runs from workflow_run. It also has a baseline config pinned to model and commit, a policy-vs-consensus suppression list, and the mutation scoreboard. It tries gemini-3.8-flash, then gemini-3.6-flash, then Meta muse-wonder-1.0, and keeps every failed attempt in the artifact and job summary. It ran end to end on a Besu fork on 2026-11-12 (above); it has not been run inside besu-eth/besu.
  • Live demo recorder (pilot/besu/live-demo/) is a fork-only workflow that screen-records a real PRSpec session on a GitHub-hosted runner and publishes the video, terminal transcript, reports and a secret-scan manifest to an evidence branch of the fork.
  • Gemini live experiment (gemini-live.yml) uses the repository secret GEMINI_API_KEY. It is triggered by editing pilot/live/REQUEST on evidence-pilot, or by manual dispatch once the file is on main.

Supported clients and limits

go-ethereum Nethermind Besu Reth
LLM file registry (10 EIPs) ✅ ✅ ✅ ✅ (7 EIPs)
Witness anchors EIP-1559 3/3 3/3 3/3 —
Witness anchors EIP-4844 9/9 8/9 8/9 —

Limits, stated plainly:

  • Witnesses check constants, not behaviour. A ✅ does not prove compliance.
  • Anchors are regular expressions reviewed at the commits listed in each rulebook. When code moves, the result becomes ⚠️ incomplete, never ✅.
  • EIP-4844 witnesses check Cancun values. Later forks change blob parameters (EIP-7691, BPO forks), and those are not modelled yet.
  • LLM findings are candidates for human review. The verification score is an internal vote, not a probability.
  • No detection rate, false-positive rate or cost is claimed. One Meta Muse run (n = 1) is published as evidence, and the Gemini scoreboard has not run yet.

Public interactions with client teams

These are links to public threads, with what the teams actually said:

  • Nethermind #10522: PRSpec flagged FeeCollector against EIP-1559's burn. The team explained it is an optional extension for other chains, not a mainnet deviation. A contributor suggested focusing on type sizes, limits and MUST/SHOULD/MAY semantics. That suggestion shaped the witness layer.
  • Ethereum Foundation execution-specs #2212: maintainers advised using fork-to-fork diffs, which PRSpec now does.
  • Besu #9850: a maintainer asked for a minimal GitHub Actions workflow and a reproducible baseline config. The pilot kit answers that request. Besu also clarified #10714 and #10715 as pool policy or validation done in another layer, and PRSpec's suppression list encodes that.
  • go-ethereum #35263: a latent divergence for a missing checked-request predeploy, with no mainnet impact because predeploys exist at activation. It is fixed upstream in 26d0b2171, and EEST covers it. Write-up.

None of these is an endorsement or an adoption decision.

Related work

The EF Protocol Security wishlist lists "Differential LLMing" and "LLM Test Generation". EPF Cohort 6 has a Differential LLMing project. SpecTrum reports consensus-layer divergences, and LLM-driven differential testing was presented at Black Hat Asia 2026. PRSpec's focus is narrower and complementary: pinned, reproducible spec-to-code evidence for execution clients, with a deterministic layer that does not depend on a model.

Reproducing the demo

See demo/README.md. It contains the exact commands and the video storyboard, and explains how each frame maps to a raw artifact.

Roadmap and support

ROADMAP.md separates what is implemented, what is partial, and what is proposed and worth funding. The proposed work includes executable EEST witnesses verified against EELS, a spec-PR watcher that posts impact cards, fork-aware blob schedules, a policy atlas, and Reth anchors.

PRSpec is MIT-licensed and maintained by one developer. The milestone-based TheDAO Security Fund initiative and the Giveth project fund the roadmap. A pledge is collected only under the fund's stated conditions, and backing is not an endorsement of results. Technical evaluation and any security disclosure stay independent of funding. Suspected live vulnerabilities go through each client's private disclosure channel (SECURITY.md).

The most useful contribution from a client team is technical: run the witness matrix or the pilot on one PR, and tell us whether it saved review time or added noise.

Development

pip install -e ".[dev]"
ruff check . && python -m pytest -q

Layout: src/ holds the CLI, analyzers, fetchers, witness engine and rulebooks (src/rules/). scripts/ holds the scoreboard and client fetch scripts, pilot/ the Besu kit and live request, docs/evidence/ dated raw results, and demo/ the video sources. The full history is in CHANGELOG.md.

License

MIT. Fosurero/PRSpec

About

AI-powered CI checks for Ethereum spec compliance. Catches spec↔code mismatches before they hit mainnet. Built by Safi El-Hassanine | AI-powered CI guardian for Ethereum protocol specs.

Resources

Contributing

Security policy

Stars

16 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages