[5/11] Make the ONNX tests cheaper: CPU-only ones to the unit lane, small models

[shengliangxu]

What does this PR do?

Type of change: CI (test runtime, tests only)

Four changes to the ONNX tests of the gpu job:

• Three CPU-only files move to the unit lane (tests/gpu/onnx/quantization/ to tests/unit/onnx/quantization/, content unchanged): test_dla_qdq_rules.py, test_dla_scalar_constants.py and test_quantization_initializer_names.py (added with [6860073][6863176][6824697] Fix ONNX quantization parameters and DLA scalar constants #2678, 68 cases). They quantize with calibration_eps=["cpu"] and run the result on the CPUExecutionProvider, so they need no GPU, but they cost 26.8 s per gpu job (1.5 + 5.3 + 20.0 s in the CI logs); the unit job runs them in a few seconds on its xdist workers.
• test_concat_elim calibrates a 32-channel 16x16 model on the CPU instead of the (1, 128, 240, 64) three-conv model on [cpu, cuda:0, trt]: the two tests took about 22 s each on the base (15 s together in the series' CI runs with the sysmon core of [2/11] Use the sysmon coverage core in CI #2846), and their assertions only look at where the pass places Q/DQ nodes. The convs stay above the 16-channel limit of graph_selection.py, and both modes produce the same 8 Q and 8 DQ nodes as with the default size. build_conv_concat_model takes optional channels, height and width (defaults: the old values; with a seeded RNG the default model serializes to the same bytes).
• The autotune benchmark tests build a 1x32x32x32 model instead of the 64x32x224x224 one (103 M random floats and about 1.2 GB copied per iteration; the five tests took 31 s, and none looks at the latency beyond "a positive float"). The two trtexec tests also pass --duration=0, because trtexec measures for at least 3 s whatever the iteration count. _create_simple_conv_onnx_model takes optional batch_size and spatial_size with the old defaults, so the unit autotuner tests that share it build the same model (their 161 tests pass).
• The autotune workflow test passes --duration=0 to trtexec: test_export_quantized_model[True] took 44 s, mostly inside trtexec's 3 s minimum measurement per scheme. It still runs the 100 requested iterations per measurement; the model and timing_runs keep their size, because the test asserts the best scheme carries Q/DQ nodes.

Time improvement (measured in CI)

gpu lane, time per file from the per-test timestamps; before = the runs of #2846-#2848, after = this PR's run and the 4 runs above it (median):

	before	after	change
three CPU-only files (68 cases), moved to the unit lane	7.4 s	0	-7.4 s
test_concat_elim.py	15.2 s	1.2 s	-14.0 s
autotune/test_benchmark.py (36 cases)	37.1 s	21.0 s	-16.0 s
autotune/test_workflow.py	44.4 s	28.5 s	-15.9 s
sum	104 s	51 s	-53 s per gpu job (-59 s after speed scaling)

The whole tests/gpu/onnx directory takes 110-120 s before and 58 s in the three later runs on normal-speed runners (81 s and 74 s in #2849 and #2850, which ran on 1.3x slower runners). The 68 moved cases cost the unit job 7-13 s of CPU, spread over four xdist workers (below the unit job's run-to-run spread). My first estimate of the gpu saving (97-107 s) came from base runs without sysmon, where these files were slower.

Method: every PR of the series has its own CI run, so "before" is the run of the PR(s) below that lack the change and "after" the runs that carry it. Times come from the job logs (--durations, or the gap between consecutive result lines). Where runner speed matters, the tests that no PR of the series touches scale each run, and the raw and the scaled medians are both shown. One test varies by 15-30% from run to run on its own, so smaller differences are not evidence.

Usage

Nothing user-facing.

Testing

• The three moved files: 68 passed in 7.1 s at the old location and in 13.3 s from the new one (this includes the unit conftest's session setup) on Python 3.12 with onnx 1.21.0 and onnxruntime 1.24.4, and 68 passed in 15.3 s on Python 3.10.19 with onnxruntime 1.22.0, the oldest runtime pyproject allows. The whole tests/unit/onnx directory passes (756 passed, 12 xfailed).
• test_concat_elim: the two tests pass in 1.3 s on CPU; quantizing the default and the small model in both modes gives Q=8, DQ=8.
• The shared helper changes: the unit autotune tests (161) pass.
• Not run: the TensorRT tests, because there is no TensorRT here; this PR's gpu job is the check. Pre-commit passes.

Before your PR is "Ready for review"

Make sure you read and follow Contributor guidelines and your commits are signed (git commit -s -S).

• Is this change backward compatible?: N/A (CI and tests)
• If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: no new dependency
• Did you write any new necessary tests?: N/A (existing tests moved or made smaller; the 68 moved cases are unchanged)
• Did you update Changelog?: N/A (CI and tests)
• Did you get Claude approval on this PR?: N/A (draft)

Additional Information

Slice 5 of 11 of the GPU-CI speed-up work, split out of the reference draft #2730 (not for merge; it holds the whole series with its measurements). Merge in order; each PR stands on its own and passes CI without the later ones.

1. [1/11] Run the unit tests in parallel with pytest-xdist #2845 [1/11] Run the unit tests in parallel with pytest-xdist
2. [2/11] Use the sysmon coverage core in CI #2846 [2/11] Use the sysmon coverage core in CI
3. [3/11] Run release-marked tests on nightly and on-demand GPU runs #2847 [3/11] Run release-marked tests on nightly and on-demand GPU runs
4. [4/11] Give gpu_megatron's slow cold starts more time #2848 [4/11] Give gpu_megatron's slow cold starts more time
5. [5/11] Make the ONNX tests cheaper: CPU-only ones to the unit lane, small models #2849 [5/11] Make the ONNX tests cheaper: CPU-only ones to the unit lane, small models (this PR)
6. [6/11] Use local tokenizers and calibration text in GPU tests, shrink puzzletron data #2850 [6/11] Use local tokenizers and calibration text in GPU tests, shrink puzzletron data
7. [7/11] Trim the llm_eval, llm_qat, EAGLE and megatron_bridge example tests #2851 [7/11] Trim the llm_eval, llm_qat, EAGLE and megatron_bridge example tests
8. [8/11] Trim the DFlash regression test #2852 [8/11] Trim the DFlash regression test
9. [9/11] Trim the torch_onnx example lane #2853 [9/11] Trim the torch_onnx example lane
10. [10/11] Make the distributed worker pool fail fast when a worker dies #2854 [10/11] Make the distributed worker pool fail fast when a worker dies
11. [11/11] Share one Megatron worker pool across the gpu_megatron modules #2855 [11/11] Share one Megatron worker pool across the gpu_megatron modules

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.90%. Comparing base (be2bdd8) to head (68a297f).

Additional details and impacted files

@@                           Coverage Diff                            @@
##           shengliangx/ci-4-megatron-cold-start    #2849      +/-   ##
========================================================================
+ Coverage                                 72.54%   78.90%   +6.35%     
========================================================================
  Files                                       658      658              
  Lines                                     73649    73649              
========================================================================
+ Hits                                      53430    58114    +4684     
+ Misses                                    20219    15535    -4684     

Flag	Coverage Δ
examples-diffusers	20.09% <ø> (ø)
examples-gpt-oss	13.53% <ø> (ø)
examples-hf_ptq	23.24% <ø> (ø)
examples-llm_distill	13.61% <ø> (ø)
examples-llm_eval	17.27% <ø> (ø)
examples-llm_qat	17.52% <ø> (ø)
examples-llm_sparsity	15.79% <ø> (ø)
examples-megatron_bridge	26.47% <ø> (ø)
examples-specdec_bench	13.31% <ø> (ø)
examples-speculative_decoding	17.51% <ø> (ø)
examples-torch_onnx	21.46% <ø> (ø)
examples-torch_trt	15.26% <ø> (ø)
examples-vllm_serve	15.01% <ø> (ø)
gpu	58.96% <ø> (+24.94%)	⬆️
regression	15.12% <ø> (ø)
unit	60.40% <ø> (+0.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[shengliangxu]

/ok to test 7b998c7

[shengliangxu]

/ok to test 68a297f