Repository navigation
refactor(reporting): build the timeout limit advice in one helper (#708) - #944
Conversation
…kelane#708) The capped-limit advice was copied in three places: the downgraded-timeout message and the pre-scan warning in plugin.py, and the score warning in reporting/score.py. limit_advice and mutant_timeout_advice now build it. A limit configured at the 86400s maximum still told users to raise it, which is impossible. Those messages now say the limit is already at the maximum. The downgraded-timeout messages leave speeding up the tests as the remedy. Only the wording of these messages changes, and TIMEOUT_NOT_CONFIRMED_PREFIX, which the score uses to count downgrades, stays the same. No verdict changes, so neither cache marker moves. Digests re-recorded.
|
Thanks for this, @a1h8. It's a clean change and the tests pin the wording well. The review approved it with no blocking findings, and I've turned on auto-merge, so it will land once CI is green. Nothing below needs a change before merge:
I'll track both optional items in our rolling cleanup issue, so a follow-up PR is welcome but not expected. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #944 +/- ##
==========================================
- Coverage 97.20% 97.19% -0.01%
==========================================
Files 60 60
Lines 5643 5640 -3
Branches 827 827
==========================================
- Hits 5485 5482 -3
Misses 98 98
Partials 60 60
🚀 New features to boost your workflow:
|
mikelane
left a comment
There was a problem hiding this comment.
Thanks for this. #942 just merged and moved the verdict code out of plugin.py, so this PR now conflicts with main in src/pytest_gremlins/plugin.py and tests/cache/test_cache_version_guard.py. The fix:
git fetch upstream && git merge upstream/main(a merge, not a rebase, so no force-push is needed)._timeout_not_confirmed_messagenow lives insrc/pytest_gremlins/verdicts.py. Apply yourmutant_timeout_advice(...)change there, and update thereporting.resultsimport inverdicts.py(addmutant_timeout_advice; dropCAPPED_LIMIT_ADVICEandlimit_is_cappedif nothing else there uses them).- In
plugin.py, keep main's version of everything except your_warn_prescan_timeoutchange, and adjust itsreporting.resultsimport to whatplugin.pystill uses. - In
tests/cache/test_cache_version_guard.py, take main'sRECORDED_DIGESTS, then runuv run pytest tests/cache/test_cache_version_guard.py -q. It prints the new digest for each changed file (plugin.py,verdicts.py,reporting/results.py, ...). Paste those in. - Run
uv run pytest tests/plugin tests/reporting -q -n 4, then commit (signed, as before) and push.
If you'd rather not, say so and I'll push the merge commit for you with credit.
Head branch was pushed to by a user without write access
|
Merged |
|
Note: my push turned auto-merge off ( |
|
Thanks, @a1h8. The merge looks right, and the commit is signed. Main moved again after your push, so the PR conflicts once more, only in #952 is about to merge and also touches that file. To save you a second round, please hold off until it lands. I'll comment here when it does. Then the fix is the same as before: merge After that I'll approve the workflow runs, run our review on the new head, and turn auto-merge back on. |
|
@a1h8 #952 has landed, so main is settled for now. Whenever you are ready: merge |
|
@mikelane I merged |
# Conflicts: # tests/cache/test_cache_version_guard.py
|
@mikelane I merged |
|
@a1h8 Thank you for merging main again. I'm sorry this one keeps getting caught in the churn. #974's fix (#985) landed this afternoon and touched Please hold off for now. Once #975 and #976 have merged, I'll ping you here, and that should be the last merge needed. As soon as you push after that, I'll approve the workflow runs and review the new head right away. Thanks for your patience. |
# Conflicts: # tests/cache/test_cache_version_guard.py
|
@mikelane #975 and #976 have both merged (#999 and #987), so I merged |
|
Merged. Thank you, @a1h8! Having the timeout advice in one helper makes the pre-scan and verdict messages much easier to keep consistent. This will ship in the next release. |
Summary
Builds the timeout limit advice in one helper, and stops telling users to raise a limit that is already at the 86400s maximum.
Changes
limit_adviceandmutant_timeout_adviceinreporting/results.pyreplace three copies of the logic:_timeout_not_confirmed_messageinverdicts.py,_warn_prescan_timeoutinplugin.py, andMutationScore.timeout_warninginreporting/score.py(the third copy the issue asked to check for).CAPPED_LIMIT_ADVICE(lower--rerunsor--reruns-delay) would not help there:The limit is already at the 86400s maximum, so speed up these tests.(those testsin the score warning)(the limit is already at the 86400s maximum)RECORDED_DIGESTSre-recorded forplugin.py,verdicts.py,reporting/results.pyandreporting/score.py. No cache marker moves: only message wording changes, andTIMEOUT_NOT_CONFIRMED_PREFIX, which the score uses to count downgrades, stays the same.Not addressed: the issue's related LOW that the capped pre-scan advice does not mention the number of flaky tests also inflates the limit. It is not in the acceptance criteria.
Type of Change
Docs (required)
docs/guide/configuration.md,docs/guide/reports.mdRelated Issues
Fixes #708
Testing
The new call-site tests failed before the change, on the old "raise the limit" advice, and pass now. The helpers have their own tests in
tests/reporting/test_results.py, including 86399 vs 86400 and an unknown limit.scripts/check_verdict_change.pyagainstmain: 26 sample gremlins, no verdict changed.Small + medium on Linux, before the final rebase: 4803 passed, 1 failed. The failure is
tests/plugin/test_rerun_blocked_in_session.py::DescribeGremlinRunsUnderACommandLineBlock::it_does_not_let_the_ini_reruns_delay_outlast_the_timeout, which uses a 3s limit and ran at a load average near 20. It passed 3 out of 3 when run alone. I did not capture the failure's cause or run it onmainunder the same load.In an earlier run stopped at the first failure,
tests/benchmark/test_sources_json_payload.pyfailed on its timing ratio under the same load. It passed 3 out of 3 alone, with and without this change, and passed in the full run.After the rebase: the affected test files, guard test, ruff, mypy and
mkdocs build --strict. Large tests were left to CI.After merging
main(with feat: shard the coverage pre-scan across parallel processes #952):tests/plugin,tests/reportingandtests/coverage_module(2140 passed, 2 skipped, 2 xfailed), guard test, ruff, mypy andmkdocs build --strict.All existing tests pass (see the note above)
Added new tests for this change
Manually tested
Checklist