Skip to content

refactor(reporting): build the timeout limit advice in one helper (#708) - #944

Merged
mikelane merged 6 commits into
mikelane:mainfrom
a1h8:refactor/708-capped-timeout-advice
Nov 16, 2026
Merged

mikelane merged 6 commits into
mikelane:mainfrom
a1h8:refactor/708-capped-timeout-advice

Conversation

@a1h8

@a1h8 a1h8 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Builds the timeout limit advice in one helper, and stops telling users to raise a limit that is already at the 86400s maximum.

Changes

  • limit_advice and mutant_timeout_advice in reporting/results.py replace three copies of the logic: _timeout_not_confirmed_message in verdicts.py, _warn_prescan_timeout in plugin.py, and MutationScore.timeout_warning in reporting/score.py (the third copy the issue asked to check for).
  • A limit configured at the maximum now gets its own wording, because CAPPED_LIMIT_ADVICE (lower --reruns or --reruns-delay) would not help there:
    • Downgraded timeout: The limit is already at the 86400s maximum, so speed up these tests. (those tests in the score warning)
    • Pre-scan: (the limit is already at the 86400s maximum)
  • All other wording is unchanged, including the case where rerun scaling reaches the cap.
  • RECORDED_DIGESTS re-recorded for plugin.py, verdicts.py, reporting/results.py and reporting/score.py. No cache marker moves: only message wording changes, and TIMEOUT_NOT_CONFIRMED_PREFIX, which the score uses to count downgrades, stays the same.

Not addressed: the issue's related LOW that the capped pre-scan advice does not mention the number of flaky tests also inflates the limit. It is not in the acceptance criteria.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test coverage improvement

Docs (required)

  • User-facing behavior changed; docs updated in: docs/guide/configuration.md, docs/guide/reports.md
  • No user-facing change

Related Issues

Fixes #708

Testing

  • The new call-site tests failed before the change, on the old "raise the limit" advice, and pass now. The helpers have their own tests in tests/reporting/test_results.py, including 86399 vs 86400 and an unknown limit.

  • scripts/check_verdict_change.py against main: 26 sample gremlins, no verdict changed.

  • Small + medium on Linux, before the final rebase: 4803 passed, 1 failed. The failure is tests/plugin/test_rerun_blocked_in_session.py::DescribeGremlinRunsUnderACommandLineBlock::it_does_not_let_the_ini_reruns_delay_outlast_the_timeout, which uses a 3s limit and ran at a load average near 20. It passed 3 out of 3 when run alone. I did not capture the failure's cause or run it on main under the same load.

  • In an earlier run stopped at the first failure, tests/benchmark/test_sources_json_payload.py failed on its timing ratio under the same load. It passed 3 out of 3 alone, with and without this change, and passed in the full run.

  • After the rebase: the affected test files, guard test, ruff, mypy and mkdocs build --strict. Large tests were left to CI.

  • After merging main (with feat: shard the coverage pre-scan across parallel processes #952): tests/plugin, tests/reporting and tests/coverage_module (2140 passed, 2 skipped, 2 xfailed), guard test, ruff, mypy and mkdocs build --strict.

  • All existing tests pass (see the note above)

  • Added new tests for this change

  • Manually tested

Checklist

  • Code follows project style guidelines (ruff, mypy pass)
  • Self-reviewed my own code
  • Tests written FIRST (TDD)
  • Documentation updated if needed
  • No new warnings introduced

…kelane#708)

The capped-limit advice was copied in three places: the downgraded-timeout
message and the pre-scan warning in plugin.py, and the score warning in
reporting/score.py. limit_advice and mutant_timeout_advice now build it.

A limit configured at the 86400s maximum still told users to raise it, which
is impossible. Those messages now say the limit is already at the maximum.
The downgraded-timeout messages leave speeding up the tests as the remedy.

Only the wording of these messages changes, and TIMEOUT_NOT_CONFIRMED_PREFIX,
which the score uses to count downgrades, stays the same. No verdict changes,
so neither cache marker moves. Digests re-recorded.
@mikelane
mikelane enabled auto-merge (squash) November 1, 2026 16:43
@mikelane

mikelane commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Thanks for this, @a1h8. It's a clean change and the tests pin the wording well. The review approved it with no blocking findings, and I've turned on auto-merge, so it will land once CI is green.

Nothing below needs a change before merge:

  • Cache note (I'll handle it): the PR changes guarded files (plugin.py, reporting/results.py, reporting/score.py). Those normally get a (cache kept: ...) line. I'll put this line in the release changelog for you: (cache kept: only timeout-advice wording changed; downgraded-timeout results are never cached and TIMEOUT_NOT_CONFIRMED_PREFIX is unchanged).
  • Optional: at the 86400s maximum, the pre-scan notice says the limit is already at the maximum but gives no next step. Adding something like "so speed up the test suite" would help.
  • Optional: mutant_timeout_advice picks "so" or "or" with advice == MAXIMUM_LIMIT_ADVICE. A small tuple or enum from limit_advice (for example (advice, is_at_maximum)) would remove the string comparison.

I'll track both optional items in our rolling cleanup issue, so a follow-up PR is welcome but not expected.

@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.19%. Comparing base (246bcee) to head (9ff808a).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #944      +/-   ##
==========================================
- Coverage   97.20%   97.19%   -0.01%     
==========================================
  Files          60       60              
  Lines        5643     5640       -3     
  Branches      827      827              
==========================================
- Hits         5485     5482       -3     
  Misses         98       98              
  Partials       60       60              
Files with missing lines Coverage Δ
src/pytest_gremlins/plugin.py 94.50% <100.00%> (-0.01%) ⬇️
src/pytest_gremlins/verdicts.py 98.30% <100.00%> (-0.02%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mikelane mikelane left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. #942 just merged and moved the verdict code out of plugin.py, so this PR now conflicts with main in src/pytest_gremlins/plugin.py and tests/cache/test_cache_version_guard.py. The fix:

  1. git fetch upstream && git merge upstream/main (a merge, not a rebase, so no force-push is needed).
  2. _timeout_not_confirmed_message now lives in src/pytest_gremlins/verdicts.py. Apply your mutant_timeout_advice(...) change there, and update the reporting.results import in verdicts.py (add mutant_timeout_advice; drop CAPPED_LIMIT_ADVICE and limit_is_capped if nothing else there uses them).
  3. In plugin.py, keep main's version of everything except your _warn_prescan_timeout change, and adjust its reporting.results import to what plugin.py still uses.
  4. In tests/cache/test_cache_version_guard.py, take main's RECORDED_DIGESTS, then run uv run pytest tests/cache/test_cache_version_guard.py -q. It prints the new digest for each changed file (plugin.py, verdicts.py, reporting/results.py, ...). Paste those in.
  5. Run uv run pytest tests/plugin tests/reporting -q -n 4, then commit (signed, as before) and push.

If you'd rather not, say so and I'll push the merge commit for you with credit.

auto-merge was automatically disabled November 2, 2026 03:59

Head branch was pushed to by a user without write access

@a1h8

a1h8 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Merged upstream/main as you described: the advice change now lives in verdicts.py, and the digests are re-recorded. tests/plugin and tests/reporting pass locally. Thanks for the detailed steps.

@a1h8

a1h8 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Note: my push turned auto-merge off (push_from_non_writer), and the CI and Documentation workflows are waiting for approval. Could you approve them and re-enable auto-merge when you get a chance?

@mikelane

mikelane commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Thanks, @a1h8. The merge looks right, and the commit is signed.

Main moved again after your push, so the PR conflicts once more, only in tests/cache/test_cache_version_guard.py. That file holds a digest for each guarded source file, so it conflicts whenever another PR that touches a guarded file merges first. GitHub won't run CI on a conflicted PR, so there's nothing for me to approve yet.

#952 is about to merge and also touches that file. To save you a second round, please hold off until it lands. I'll comment here when it does. Then the fix is the same as before: merge upstream/main, keep main's lines for the other files in the conflict, and re-record the digests for plugin.py, verdicts.py, reporting/results.py and reporting/score.py.

After that I'll approve the workflow runs, run our review on the new head, and turn auto-merge back on.

@mikelane

mikelane commented Nov 2, 2026

Copy link
Copy Markdown
Owner

@a1h8 #952 has landed, so main is settled for now. Whenever you are ready: merge upstream/main, keep main's lines for the other files in the conflict in tests/cache/test_cache_version_guard.py, and re-record the digests for plugin.py, verdicts.py, reporting/results.py and reporting/score.py. Once you push, I'll approve the workflow runs, review the new head and turn auto-merge back on. Thanks for your patience.

@a1h8

a1h8 commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

@mikelane I merged upstream/main (with #952) and pushed f8a2a82. The PR no longer conflicts, and the digests are re-recorded (only plugin.py and verdicts.py changed after the merge). tests/plugin, tests/reporting, tests/coverage_module, the guard test, ruff, mypy and mkdocs build --strict pass locally. CI and Documentation are waiting for your approval. Thanks!

# Conflicts:
#	tests/cache/test_cache_version_guard.py
@a1h8

a1h8 commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

@mikelane I merged upstream/main again and pushed 88b32b6. The PR no longer conflicts. The only conflict was in tests/cache/test_cache_version_guard.py: I kept main's lines and re-recorded the digests for plugin.py, verdicts.py, reporting/results.py and reporting/score.py. Ruff, mypy and the guard test pass locally. Could you approve the workflow runs and turn auto-merge back on? Thanks!

@mikelane

mikelane commented Nov 6, 2026

Copy link
Copy Markdown
Owner

@a1h8 Thank you for merging main again. I'm sorry this one keeps getting caught in the churn.

#974's fix (#985) landed this afternoon and touched plugin.py, verdicts.py and the cache-guard digests, so the PR conflicts again. Two more critical fixes, #975 and #976, are about to land in those same files. If you merge main now, you'd only have to do it again after each of them.

Please hold off for now. Once #975 and #976 have merged, I'll ping you here, and that should be the last merge needed. As soon as you push after that, I'll approve the workflow runs and review the new head right away. Thanks for your patience.

# Conflicts:
#	tests/cache/test_cache_version_guard.py
@a1h8

a1h8 commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

@mikelane #975 and #976 have both merged (#999 and #987), so I merged upstream/main and pushed 0931d77. The PR no longer conflicts. As before, the only conflict was in tests/cache/test_cache_version_guard.py: I kept main's lines and re-recorded the digests for plugin.py, verdicts.py, reporting/results.py and reporting/score.py. tests/plugin, tests/reporting, tests/coverage_module, tests/cache, ruff, mypy and mkdocs build --strict pass locally. Could you approve the workflow runs and turn auto-merge back on? Thanks!

@mikelane mikelane left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. The consolidated timeout advice addresses the earlier review, and the re-review at 9ff808a came back clean. A few minor notes went to the rolling follow-up issues (#868, #870, #871); none of them block this.

@mikelane
mikelane merged commit d10c96c into mikelane:main Nov 16, 2026
32 checks passed
@mikelane

Copy link
Copy Markdown
Owner

Merged. Thank you, @a1h8! Having the timeout advice in one helper makes the pre-scan and verdict messages much easier to keep consistent. This will ship in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor: share the capped-timeout advice between the timeout and score messages

2 participants