Kuang-Da Wang1,†, Kai-Jie Lin2,†, Guo-Xun Ko3, Yu-Heng Lin2, Yu Heng Hung1, Travis M. Bartley2, Shuoyang Ding2, Yu-Chiang Frank Wang2,3, Chao-Han Huck Yang2, Guan-Ting Liu2, Wen-Chih Peng1, Ping-Chun Hsieh1
1National Yang Ming Chiao Tung University 2NVIDIA 3National Taiwan University †Equal contribution
AskR learns what to ask so that a frozen reasoner's own self-verification becomes a reliable intrinsic reward. Instead of training an external process reward model (PRM), AskR trains a sub-question policy that structures reasoning into sub-questions and sub-answers (SubQAs). The reasoner checks every SubQA, and the mean of these checks guides test-time optimization.
- Better than PRMs: with the same optimizer, AskR beats PRM-guided reasoning by +7.12 points on average, and by up to +9.18.
- Data-efficient: AskR beats RLVR on average while using only 5% (text) and 17% (vision) of RLVR's training data.
- Reusable: one 4B AskR steers Qwen3 reasoners from 1.7B to 32B.
conda create -n askr python=3.11 -y
conda activate askr
pip install -r requirements.txtThe versions in requirements.txt are pinned together: unsloth, trl, vllm and torchao break each other
across versions. All experiments ran on a single NVIDIA RTX PRO 6000 Blackwell (96 GB).
# Text
python trainers/askr_trainer.py --model Qwen/Qwen2.5-7B-Instruct --dataset math --output-dir askr
# Vision
python trainers/askr_trainer.py --model Qwen/Qwen2.5-VL-7B-Instruct --dataset lpt30k --visual-mode --output-dir askr_vlpython AskR_Solver.py \
--model Qwen/Qwen2.5-7B-Instruct \
--lora-path ./outputs/train/askr_<timestamp>/final_model \
--mode tmpc \
--dataset lab-bench-
Each run produces one answer per question and writes
overall_accuracy(unparseable answers count as wrong) tooutputs/eval/. The paper reports the mean over 8 independent runs:for t in $(seq 1 8); do python AskR_Solver.py --lora-path <adapter> --dataset lab-bench --output outputs/eval/lab-bench_trial$t.json done
-
Vision: add
--model Qwen/Qwen2.5-VL-7B-Instruct --visual-modeand use--dataset vstar,mmstarorcvbench. -
Separate asker and reasoner: one 4B asker can guide a larger Qwen3 reasoner:
python AskR_Solver.py --asker-model Qwen/Qwen3-4B --lora-path <adapter> --model Qwen/Qwen3-32B \ --gpu-memory-utilization 0.65 --asker-gpu-memory-utilization 0.2 --dataset mmlu
-
--mode askrskips TMPC. It samples--n-samplesSubQA trajectories per question and reports their mean accuracy. -
Omit
--lora-pathto run the untrained asker.
The evaluation benchmarks are downloaded from the Hugging Face Hub.
mixed_math_stem_dataset We use half math and half STEM for training. It holds math and STEM problems from SYNTHETIC-1.:
type |
Source | answer |
Graded by |
|---|---|---|---|
math |
PrimeIntellect/verifiable-math-problems |
verification_info["ground_truth"] (LaTeX) |
math-verify |
stem |
PrimeIntellect/stackexchange-question-answering |
gold_standard_solution |
LLM judge (score 0–1; > 0.5 counts as correct) |
--dataset math and --dataset stem load these two sources directly from the Hub.
LongPerceptualThoughts-30k/ is built from
LongPerceptualThoughts-30k and the
DOCCI images. Build the rest from the project root:
cd data/LongPerceptualThoughts-30k
huggingface-cli download andrewliao11/LongPerceptualThoughts-30k --repo-type dataset \
--revision 5e8d7ab7a447e5b2d2d991b2320aafdd423c249d --local-dir .
mkdir -p docci && cd docci
wget https://storage.googleapis.com/docci/data/docci_descriptions.jsonlines
wget https://storage.googleapis.com/docci/data/docci_images.tar.gz
tar -xzf docci_images.tar.gz && rm docci_images.tar.gz # -> docci/images/
cd .. && python merge_with_docci_and_convert_to_sharegpt.py # -> LongPerceptualThoughts_SFT.jsonThe ground-truth letter of each question comes from the <answer> tag of its assistant message.
LLM judge (STEM rows only). The judge is called through an OpenAI-compatible API:
export DEEPINFRA_API_KEY=... # default provider: DeepInfra, model deepseek-ai/DeepSeek-V3.2
# or
export LLM_JUDGE_PROVIDER=openai OPENAI_API_KEY=... # model gpt-4o
export LLM_JUDGE_MODEL=... # optional: override the judge modelBenchmarks:
--dataset |
Benchmark | Split | Modality |
|---|---|---|---|
mmlu |
MMLU-Pro (physics, chemistry, CS, math, biology, health, engineering) | test | text |
lab-bench |
LAB-Bench (CloningScenarios, ProtocolQA, SeqQA) | train* | text |
scieval |
SciEval | test | text |
vstar |
V* Bench | test | image |
mmstar |
MMStar (coarse, fine-grained perception, instance reasoning) | val | image |
cvbench |
CV-Bench (2D) | test | image |
* LAB-Bench is an evaluation-only benchmark; its Hugging Face release puts every question in a split named train.
The LLM judge builds on Genesys. Training uses Unsloth and TRL, and inference uses vLLM.
