Skip to content

Add native FStar.BV normalization - #4743

Open
amosr-msft wants to merge 1 commit into
FStarLang:masterfrom
amosr-msft:users/amos/20261016-fstar-bv-eval
Open

amosr-msft wants to merge 1 commit into
FStarLang:masterfrom
amosr-msft:users/amos/20261016-fstar-bv-eval

Conversation

@amosr-msft

Copy link
Copy Markdown
Contributor

Evaluate concrete bit-vector operations through the extracted FStar.BV definitions in both the simple normalizer and NBE. Use canonical int2bv terms for bit-vector values and cover the complete FStar.BV operation family with friend-free model agreement tests.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

@amosr-msft
amosr-msft marked this pull request as ready for review October 21, 2026 01:40
@amosr-msft

Copy link
Copy Markdown
Contributor Author

!bench

@gebner

gebner commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

@amosr-msft The benchmarking workflow didn't recognize you because you're not a member yet. Sent you an invite!

!bench

@amosr-msft

Copy link
Copy Markdown
Contributor Author

awesome, thanks!
I also overestimated how baked this was - it causes huge proofs regression, so not worth benchmarking yet

@amosr-msft
amosr-msft force-pushed the users/amos/20261016-fstar-bv-eval branch from 5f4e042 to 65f72ee Compare October 21, 2026 04:56
@amosr-msft

Copy link
Copy Markdown
Contributor Author

!bench

@github-actions

Copy link
Copy Markdown
Contributor

🔬 F* Performance Comparison

Baseline: Base (master @ 5275509e26b2e4c3df3ade7df1e01b3e38f9553a) | Patched: PR #4743 (users/amos/20261016-fstar-bv-eval @ 65f72eed0c407097dfc496b58623d342d78d8f17) | Matched: 3288 tests

Summary

Metric Median Mean Geo Mean Std Dev P5 → P95 Min → Max
Memory +0.0% +0.1% 1.001× 0.7% -0.1% → +0.6% -26.1% → +17.5%
Time -3.6% -3.6% 0.960× 7.7% -15.1% → +8.3% -86.6% → +50.0%

Totals: Memory: 40.4 MiB | Time: -425.4s

Change Distribution

Metric 🟢 Improved (>5%) ⚪ Unchanged (±5%) 🔴 Regressed (>5%)
Memory 2 (0%) 3283 (100%) 3 (0%)
Time 1270 (41%) 1508 (49%) 307 (10%)

Heavy Tests (baseline > 100 MiB, n=743)

Metric Median Mean Geo Mean Std Dev P5 → P95
Memory +0.0% +0.1% 1.001× 1.1% -0.4% → +0.8%
Time -5.0% -5.0% 0.947× 7.4% -15.1% → +6.2%
📉 Top 20 Memory Improvements
File Mem (base) Mem (patch) Mem Δ Time (base) Time (patch) Time Δ
…sts/micro-benchmarks/_cache/BvModelAgreesWithZ3.fst.checked 717.3 MiB 530.5 MiB -26.1% 313.52s 42.07s -86.6%
tests/custard/_output/NormBudget.rejected 5.4 GiB 5.3 GiB -1.8% 145.19s 147.17s +1.4%
tests/custard/plugin/_output/CustardPlugin.ml 386.1 MiB 381.2 MiB -1.3% 1.94s 1.98s +1.9%
tests/semiring/_cache/CanonCommSemiring.fst.checked 349.6 MiB 344.8 MiB -1.4% 8.96s 8.65s -3.5%
examples/algorithms/_output/IntervalIntersect.cherry.c 108.5 MiB 104.6 MiB -3.6% 0.50s 0.46s -7.0%
…ples/dsls/bool_refinement/_cache/BoolRefinement.fst.checked 436.7 MiB 433.7 MiB -0.7% 79.92s 73.10s -8.5%
tests/micro-benchmarks/_cache/Test.QuickCode.fst.checked 38.4 MiB 36.2 MiB -5.7% 0.20s 0.19s -1.5%
doc/book/code/_cache/UInt32BV.fst.checked 99.1 MiB 97.4 MiB -1.7% 0.73s 0.69s -5.4%
doc/book/code/_cache/Part2.FreeFunExt.fst.checked 65.3 MiB 63.6 MiB -2.6% 0.81s 0.81s +0.4%
doc/book/code/_cache/Part2.Free.fst.checked 61.6 MiB 60.3 MiB -2.2% 0.70s 0.71s +1.3%
examples/algorithms/_cache/MergeSort2.fst.checked 70.8 MiB 69.5 MiB -1.9% 2.35s 2.02s -14.2%
tests/vale/_cache/X64.Poly1305.Math_i.fst.checked 141.7 MiB 140.4 MiB -0.9% 2.15s 2.12s -1.1%
examples/algorithms/_output/IntervalIntersect.ml 126.1 MiB 124.8 MiB -1.0% 0.75s 0.54s -27.8%
tests/micro-benchmarks/_cache/BvBinaryOps.fst.checked 74.6 MiB 73.4 MiB -1.5% 5.93s 4.71s -20.7%
examples/metatheory/_cache/LambdaOmega.fst.checked 241.9 MiB 240.9 MiB -0.4% 6.45s 6.30s -2.3%
tests/extraction/backends/_output/custard/ExtUIntUnsigned.c 84.5 MiB 83.5 MiB -1.2% 0.42s 0.42s -0.5%
…lse/share/pulse/examples/_cache/EqualOrDisjoint.fst.checked 222.0 MiB 221.0 MiB -0.5% 1.99s 2.10s +5.1%
…lse/share/pulse/examples/_cache/MSort.SeqLemmas.fst.checked 133.0 MiB 132.0 MiB -0.8% 1.33s 1.31s -1.6%
pulse/share/pulse/examples/_cache/PledgeArith.fst.checked 163.0 MiB 162.0 MiB -0.6% 0.54s 0.55s +2.6%
pulse/share/pulse/examples/_cache/PulseLambdas.fst.checked 139.0 MiB 138.0 MiB -0.7% 0.54s 0.58s +6.1%
📈 Top 20 Memory Regressions
File Mem (base) Mem (patch) Mem Δ Time (base) Time (patch) Time Δ
tests/custard/_cache/ArrowUnfoldLimit.fst.checked 2.9 GiB 2.9 GiB +0.9% 29.17s 27.60s -5.4%
tests/tactics/Hacl_Meta.ml 474.6 MiB 495.2 MiB +4.3% 2.72s 2.91s +7.1%
tests/semiring/_output/CanonCommSemiring.ml 510.9 MiB 530.2 MiB +3.8% 3.12s 3.09s -1.1%
tests/tactics/_cache/Prettify.fst.checked 444.6 MiB 461.3 MiB +3.8% 10.31s 10.08s -2.2%
tests/calc/_cache/Long.fst.checked 1019.3 MiB 1.0 GiB +1.0% 27.97s 25.66s -8.2%
examples/metatheory/_cache/MiniValeSemantics.fst.checked 802.3 MiB 812.0 MiB +1.2% 14.40s 14.47s +0.5%
tests/vale/_cache/X64.Poly1305.Bitvectors_i.fst.checked 112.3 MiB 119.2 MiB +6.2% 2.19s 2.28s +3.9%
…s/data_structures/_cache/BinaryTreesEnumeration.fst.checked 134.9 MiB 139.8 MiB +3.6% 1.78s 1.48s -17.0%
tests/tactics/_cache/CheckMatchComplete.fst.checked 170.1 MiB 174.8 MiB +2.8% 1.98s 2.12s +7.1%
tests/ide/emacs/_output/OvlIde.lookup-unchecked.ideout 19.5 MiB 22.9 MiB +17.5% 0.08s 0.08s -3.7%
tests/ide/emacs/_output/Minimal.unset-current-module.ideout 19.5 MiB 22.9 MiB +17.5% 0.08s 0.08s -5.0%
…/share/pulse/examples/dice/_cache/CBOR.Spec.Map.fst.checked 482.0 MiB 485.0 MiB +0.6% 8.16s 7.05s -13.6%
…ples/by-example/_cache/PulseTutorial.Algorithms.fst.checked 295.0 MiB 298.0 MiB +1.0% 5.63s 5.19s -7.9%
examples/algorithms/_cache/BinarySearch.fst.checked 82.7 MiB 84.7 MiB +2.5% 1.13s 1.09s -3.3%
tests/custard/pulse/_output/cherry/PulseBlit.chc 191.0 MiB 193.0 MiB +1.0% 0.88s 0.92s +4.4%
examples/oplss2021/_cache/OPLSS2021.IFC.fst.checked 129.1 MiB 130.6 MiB +1.2% 3.22s 3.12s -3.0%
examples/algorithms/_cache/QuickSort.Seq.fst.checked 68.8 MiB 70.0 MiB +1.7% 0.86s 0.70s -18.7%
…lse/examples/dice/_cache/Spec.Hash.Definitions.fsti.checked 65.5 MiB 66.6 MiB +1.7% 0.15s 0.14s -7.4%
tests/custard/.depend 44.4 MiB 45.5 MiB +2.5% 1.33s 1.21s -9.4%
tests/extraction/_cache/Div.fst.checked 25.0 MiB 26.0 MiB +4.3% 0.09s 0.09s -2.1%
⬇️ Top 20 Time Improvements
File Mem (base) Mem (patch) Mem Δ Time (base) Time (patch) Time Δ
…sts/micro-benchmarks/_cache/BvModelAgreesWithZ3.fst.checked 717.3 MiB 530.5 MiB -26.1% 313.52s 42.07s -86.6%
tests/extraction/backends/_cache/ExtUIntMask.fst.checked 88.5 MiB 89.1 MiB +0.7% 255.77s 224.18s -12.4%
tests/extraction/backends/_cache/ExtUIntUnsigned.fst.checked 107.5 MiB 106.8 MiB -0.6% 110.62s 93.97s -15.1%
…re/pulse/examples/dice/_cache/DPE.Messages.Spec.fst.checked 289.0 MiB 289.0 MiB +0.0% 96.14s 85.89s -10.7%
…ples/dsls/bool_refinement/_cache/BoolRefinement.fst.checked 436.7 MiB 433.7 MiB -0.7% 79.92s 73.10s -8.5%
tests/extraction/backends/_cache/ExtUInt8Lognot.fst.checked 74.5 MiB 73.8 MiB -1.0% 23.95s 18.53s -22.6%
tests/custard/pulse/_cache/CborBoundarySlice.fst.checked 514.0 MiB 514.0 MiB +0.0% 32.79s 28.62s -12.7%
…sts/extraction/backends/_cache/ExtIntShiftArith.fst.checked 69.7 MiB 69.9 MiB +0.3% 23.86s 20.13s -15.6%
tests/vale/_cache/X64.Vale.Decls.fst.checked 180.1 MiB 179.8 MiB -0.2% 83.86s 80.21s -4.4%
examples/data_structures/_cache/RBTreeIntrinsic.fst.checked 122.8 MiB 122.5 MiB -0.3% 25.83s 22.22s -14.0%
…lse/share/pulse/examples/dice/_cache/CBOR.Pulse.fst.checked 725.0 MiB 725.0 MiB +0.0% 22.83s 19.84s -13.1%
tests/calc/_cache/Long.fst.checked 1019.3 MiB 1.0 GiB +1.0% 27.97s 25.66s -8.2%
tests/extraction/backends/_cache/ExtUIntRotate.fst.checked 74.7 MiB 74.8 MiB +0.1% 13.75s 12.05s -12.4%
…reports/closed/_cache/SquashSubtypingDivergence.fst.checked 63.2 MiB 63.2 MiB +0.0% 7.82s 6.18s -20.9%
…se/examples/by-example/_cache/ParallelIncrement.fst.checked 501.0 MiB 500.0 MiB -0.2% 14.13s 12.53s -11.3%
tests/custard/_cache/ArrowUnfoldLimit.fst.checked 2.9 GiB 2.9 GiB +0.9% 29.17s 27.60s -5.4%
…-example/_cache/PulseTutorial.ParallelIncrement.fst.checked 416.0 MiB 416.0 MiB +0.0% 12.28s 11.00s -10.4%
tests/extraction/backends/_cache/ExtIntDivRem.fst.checked 69.5 MiB 69.5 MiB +0.0% 11.65s 10.39s -10.8%
tests/micro-benchmarks/_cache/BvBinaryOps.fst.checked 74.6 MiB 73.4 MiB -1.5% 5.93s 4.71s -20.7%
…/share/pulse/examples/dice/_cache/CBOR.Spec.Map.fst.checked 482.0 MiB 485.0 MiB +0.6% 8.16s 7.05s -13.6%
⬆️ Top 20 Time Regressions
File Mem (base) Mem (patch) Mem Δ Time (base) Time (patch) Time Δ
tests/custard/_output/NormBudget.rejected 5.4 GiB 5.3 GiB -1.8% 145.19s 147.17s +1.4%
tests/micro-benchmarks/_cache/Test.IrrationalPow.fst.checked 40.1 MiB 40.1 MiB +0.1% 0.53s 0.74s +39.9%
tests/tactics/Hacl_Meta.ml 474.6 MiB 495.2 MiB +4.3% 2.72s 2.91s +7.1%
doc/book/code/_cache/Part3.DataTypesALaCarte.fst.checked 180.3 MiB 180.0 MiB -0.2% 4.60s 4.76s +3.4%
tests/custard/pulse/_output/PulseBasic.dc 196.0 MiB 196.0 MiB +0.0% 1.00s 1.14s +14.1%
tests/tactics/_cache/CheckMatchComplete.fst.checked 170.1 MiB 174.8 MiB +2.8% 1.98s 2.12s +7.1%
pulse/share/pulse/examples/_cache/Quicksort.Base.fst.checked 325.0 MiB 325.0 MiB +0.0% 5.75s 5.89s +2.4%
tests/custard/pulse/_output/ArrowAb.custard.rust.krml 189.0 MiB 189.0 MiB +0.0% 0.89s 1.00s +13.0%
pulse/share/pulse/examples/_cache/Quicksort.Task.fst.checked 227.0 MiB 228.0 MiB +0.4% 1.62s 1.74s +7.0%
examples/tactics/_cache/Antiquote.fst.checked 102.8 MiB 102.8 MiB -0.1% 0.59s 0.70s +18.7%
tests/vale/_cache/X64.Semantics_s.fst.checked 95.2 MiB 95.2 MiB -0.1% 1.08s 1.19s +10.0%
tests/custard/_output/MachineInts.fsout 97.3 MiB 96.8 MiB -0.5% 0.47s 0.58s +22.4%
tests/bug-reports/closed/_cache/Bug3207.fst.checked 212.5 MiB 212.1 MiB -0.2% 4.71s 4.81s +2.2%
…lse/share/pulse/examples/_cache/EqualOrDisjoint.fst.checked 222.0 MiB 221.0 MiB -0.5% 1.99s 2.10s +5.1%
pulse/test/bug-reports/_cache/Bug94.fst.checked 164.0 MiB 164.0 MiB +0.0% 0.55s 0.66s +18.2%
tests/custard/pulse/_output/ExtFlag.dc 193.0 MiB 193.0 MiB +0.0% 0.88s 0.98s +11.0%
tests/tactics/_cache/CallSubtac.fst.checked 117.0 MiB 117.0 MiB +0.0% 0.66s 0.75s +13.9%
tests/custard/pulse/_output/PulseFnPtrRet.dc 188.0 MiB 189.0 MiB +0.5% 0.89s 0.98s +10.2%
tests/vale/_cache/X64.Poly1305.Bitvectors_i.fst.checked 112.3 MiB 119.2 MiB +6.2% 2.19s 2.28s +3.9%
tests/micro-benchmarks/_cache/TestSealed.fst.checked 121.7 MiB 121.7 MiB +0.0% 0.82s 0.91s +10.0%

ℹ️ The full comparison table (3288 tests) was omitted to keep this report within GitHub's comment size limit. See the attached HTML report for all tests.

Note: Memory values report the peak OCaml heap of the F* process (excluding Z3 subprocesses). Time measurements are from single runs and may be noisy, especially for fast tests. Tests with baseline time < 0.1s are excluded from time statistics. The geometric mean (Geo Mean) of the patched/baseline ratio is the most robust summary statistic for performance comparisons (1.0× = no change, <1× = improvement).


📎 Download HTML report and raw .ramon files

Evaluate concrete bit-vector operations through the extracted FStar.BV definitions in both the simple normalizer and NBE. Use canonical int2bv terms for bit-vector values and cover the complete FStar.BV operation family with friend-free model agreement tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 27a5bd47-e213-4805-84c2-84d36719df3e
@amosr-msft
amosr-msft force-pushed the users/amos/20261016-fstar-bv-eval branch from 65f72ee to f16b574 Compare October 22, 2026 04:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants