Skip to content

docs: record the CPU TinyLlama HumanEval pair - #15

Merged
Mattral merged 1 commit into
mainfrom
c3-cpu-remeasure
Oct 16, 2026
Merged

Mattral merged 1 commit into
mainfrom
c3-cpu-remeasure

Conversation

@Mattral

@Mattral Mattral commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Records the CPU float32 TinyLlama pair measured in this session at git f59b5b4.

  • Target TinyLlama/TinyLlama-1.1B-Chat-v1.0, drafter Felladrin/Llama-160M-Chat-v1, HumanEval/23, 16 new tokens, n_warmup=0, n_samples=1, gamma 4, one thread.
  • Command: python benchmarks/run_all.py --config <yaml outside benchmarks/configs/> --output-dir <dir>, with OMP, MKL, OPENBLAS, and NUMEXPR threads set to 1.
  • FlashSpec 1.7507264092133707 tok/s, vanilla autoregressive 4.2977579080394 tok/s, speedup_vs_ar_mean 0.40735807988124606, acceptance 0.11363636363636363.
  • The README opening says the engine is currently slower than plain decoding, and the tagline no longer says the verification kernel is Triton-optimised.
  • The Colab Tesla T4 JSON files were already in the tree (4e48c55). The status box now cites benchmarks/results/tinyllama_gpu_flashspec_ucb_cb02ca1.json and benchmarks/results/tinyllama_gpu_vanilla_ar_cb02ca1.json. That run is a recorded measurement.
  • The H100 tok/s table removed earlier was an unmeasured design-target table. Those figures were never in a result JSON. paper/flashspec.tex still labels the same table as unmeasured. This PR does not restore those numbers and does not call that table a deleted measurement.

flashspec/sampling/rejection.py is unchanged, so merging should not queue asm-residual.yml.

FlashSpec at git f59b5b4 measured 1.7507264092133707 tok/s against vanilla autoregressive decoding at 4.2977579080394 tok/s on HumanEval/23. The status box cites the Colab T4 JSON files that are already in the tree. The removed H100 README rows stay described as unmeasured design targets.

Co-authored-by: DavidMacha <minmattral@gmail.com>
@Mattral
Mattral merged commit 38a57f0 into main Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant