Repository navigation
entail 1.0.2: the measured case, and what it does not do #1
wwoosshh
announced in
Announcements
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Your model files say how they must be run. Your engine doesn't always listen. Of the 300 most-downloaded text-generation models on Hugging Face, 180 take vLLM's launch-time
rope_scalingoverride; it silently changes the RoPE base of 64 of them (Llama-3.2-3B-Instruct: GSM8K 379 → 273). Reported upstream: vllm-project/vllm#58675, sgl-project/sglang#41227.The write-up, with everything measured (38 popular models × 3 engines, 102 runs, 0 false alarms; and the 8 real bugs it did not catch): https://huggingface.co/blog/wwoosshh/entail
Scripts, protocol and raw results: https://github.com/wwoosshh/entail-research
Try it on your model:
pip install entail-ai, thenentail preflight --model <dir> --engine vllm --list(no GPU), orENTAIL=loadin front of your server. Wrong reports and misses: open an issue with the line fromentail_logs/record-<date>.jsonl(see CONTRIBUTING.md).All reactions