[3/11] Run release-marked tests on nightly and on-demand GPU runs

[shengliangxu]

What does this PR do?

Type of change: CI (test runtime)

Tests marked release collect only when pytest gets --run-release, and no workflow passes it, so a test marked release never runs in CI. This sets PYTEST_ADDOPTS=--run-release in the "Run gpu tests" step for every run that is not a pull-request run (the nightly schedule and manual dispatches), which lets slow or network-heavy checks leave the pull-request jobs without leaving CI. Pull-request runs are unchanged by the flag itself. The expression is !startsWith(github.ref, 'refs/heads/pull-request/') && '--run-release' || ''. The release example suites (hf_ptq/test_deploy.py, gpt-oss QAT) belong to example_tests.yml, which does not get the flag, so the docs sentence that said no workflow passes it now says no example workflow does.

Two tests use it:

• test_megatron_generate_and_mmlu[tp] builds a 28-layer Qwen3-0.6B Megatron model, downloads 1.2-1.5 GB of real weights into a fresh container cache and scores about 1,400 MMLU prompts: a 21-39 s call in every PR gpu_megatron job (21 s with the sysmon core of [2/11] Use the sysmon coverage core in CI #2846, 39 s without it), and the only test in the lane that downloads real weights.
• test_get_dataset_dataloader_nemotron_v3_chat_template streams the seven gated Nemotron post-training v3 datasets from the Hub (a 17-24 s call that needs an HF token and the network, and a source of Hub flakes). The combo expansion and sample distribution it goes through are covered by CPU unit tests that record the per-dataset calls.

Trade-off: these two checks no longer run on pull requests. The real-weight accuracy check (0.36 < MMLU < 0.39) and the gated-data smoke test run every night and on demand, a day after a change instead of on the PR.

Time improvement (measured in CI)

Time of the module that holds the skipped test, from the per-test timestamps; before = the run of #2846, after = this PR's run and the runs above it that carry the change (median):

lane	skipped test (call time before)	module before	module after	change
gpu_megatron	test_megatron_generate_and_mmlu[tp] (21 s)	64 s	41 s (39 s in this PR's run)	-22 s (-35% after speed scaling)
gpu	test_get_dataset_dataloader_nemotron_v3_chat_template (17 s)	32 s	13.5 s (13 s in this PR's run)	-19 s (-62%)

So 22 s off the gpu_megatron job and 19 s off the gpu job of every PR run. The lane totals cannot show a change this small (gpu_megatron 1,212 -> 1,206 s, gpu 1,289 -> 1,351 s: their run-to-run spread is several times larger). The price is that the two checks run only on the nightly and on demand.

Method: every PR of the series has its own CI run, so "before" is the run of the PR(s) below that lack the change and "after" the runs that carry it. Times come from the job logs (--durations, or the gap between consecutive result lines). Where runner speed matters, the tests that no PR of the series touches scale each run, and the raw and the scaled medians are both shown. One test varies by 15-30% from run to run on its own, so smaller differences are not evidence.

Usage

pytest --run-release runs the release-marked tests locally.

Testing

A stub test marked release, collected under the repo's tests/conftest.py, is skipped with "Only run when --run-release is given" when PYTEST_ADDOPTS is empty and runs with --run-release; the expression gives '' for a pull-request ref and --run-release otherwise. Pre-commit and actionlint pass. Not run on a GPU here: the two tests themselves, and the flag on a nightly (this PR's GPU run shows both skipped on a pull request; a nightly, where they run, is what verifies the flag end to end).

Before your PR is "Ready for review"

Make sure you read and follow Contributor guidelines and your commits are signed (git commit -s -S).

• Is this change backward compatible?: N/A (CI and tests)
• If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: no new dependency
• Did you write any new necessary tests?: N/A (marks two existing tests; no new tests)
• Did you update Changelog?: N/A (CI and tests)
• Did you get Claude approval on this PR?: N/A (draft)

Additional Information

Slice 3 of 11 of the GPU-CI speed-up work, split out of the reference draft #2730 (not for merge; it holds the whole series with its measurements). Merge in order; each PR stands on its own and passes CI without the later ones.

1. [1/11] Run the unit tests in parallel with pytest-xdist #2845 [1/11] Run the unit tests in parallel with pytest-xdist
2. [2/11] Use the sysmon coverage core in CI #2846 [2/11] Use the sysmon coverage core in CI
3. [3/11] Run release-marked tests on nightly and on-demand GPU runs #2847 [3/11] Run release-marked tests on nightly and on-demand GPU runs (this PR)
4. [4/11] Give gpu_megatron's slow cold starts more time #2848 [4/11] Give gpu_megatron's slow cold starts more time
5. [5/11] Make the ONNX tests cheaper: CPU-only ones to the unit lane, small models #2849 [5/11] Make the ONNX tests cheaper: CPU-only ones to the unit lane, small models
6. [6/11] Use local tokenizers and calibration text in GPU tests, shrink puzzletron data #2850 [6/11] Use local tokenizers and calibration text in GPU tests, shrink puzzletron data
7. [7/11] Trim the llm_eval, llm_qat, EAGLE and megatron_bridge example tests #2851 [7/11] Trim the llm_eval, llm_qat, EAGLE and megatron_bridge example tests
8. [8/11] Trim the DFlash regression test #2852 [8/11] Trim the DFlash regression test
9. [9/11] Trim the torch_onnx example lane #2853 [9/11] Trim the torch_onnx example lane
10. [10/11] Make the distributed worker pool fail fast when a worker dies #2854 [10/11] Make the distributed worker pool fail fast when a worker dies
11. [11/11] Share one Megatron worker pool across the gpu_megatron modules #2855 [11/11] Share one Megatron worker pool across the gpu_megatron modules

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.90%. Comparing base (73d351b) to head (dddb321).

Additional details and impacted files

@@                      Coverage Diff                      @@
##           shengliangx/ci-2-coverage    #2847      +/-   ##
=============================================================
- Coverage                      78.91%   78.90%   -0.02%     
=============================================================
  Files                            658      658              
  Lines                          73649    73649              
=============================================================
- Hits                           58122    58114       -8     
- Misses                         15527    15535       +8     

Flag	Coverage Δ
gpu	59.17% <ø> (-0.16%)	⬇️
unit	60.32% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[shengliangxu]

/ok to test 9c2eebb

[shengliangxu]

/ok to test dddb321