test: run tests in a random order
Metadata
Current evaluation
Adds pytest-randomly so craft-parts tests run in random order to surface order dependence. Maintainer-authored, 380 days old, zero review comments, no updates, and 11 CI jobs failing (most test-common/test-other slow+fast runs).
Suggested action: close stale
Reason: The branch has been inactive for 378 days with0 reviewers and 0 comments, and 11 checks are red (test-common fast+slow on most platforms, test-other slow) — the diff (+17/-2 across pyproject config) is what exposes those order-dependent failures, and no follow-up fixing them was ever pushed, so the branch has drifted a year behind the suite. Not technically superseded: main's [tool.pytest.ini_options] still has only addopts = "--ignore=tests/spread" with no random ordering, and the equivalent change was merged in the sibling repo (craft-platforms#136), so if this is wanted it should be re-raised as a fresh PR. Confidence is moderate because no review feedback was ever given and 'abandoned' here is inferred purely from CI red plus age.
Evaluated by: mr-cal-bot
Issue body
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha |
Impact:
28
Quick Win:
3
Complexity:
34
Confidence:
58
Actionability:
14
|
close stale | Adds pytest-randomly so craft-parts tests run in random order to surface order dependence. Maintainer-authored, 380 days old, zero review comments, no updates, and 11 CI jobs failing (most test-common/test-other slow+fast runs). | |
| qwen/qwen3.6-35b-a3b |
Impact:
50
Quick Win:
0
Complexity:
20
Confidence:
60
Actionability:
0
|
close stale | Enables random test execution order to catch interdependence issues. PR is 370 days old, inactive for 368 days, with failing CI on multiple test jobs and no reviewer engagement. | |
| qwen/qwen3.8-27b |
Impact:
35
Quick Win:
21
Staleness:
78
Complexity:
40
Confidence:
78
|
needs review | Adds pytest-randomly to run the test suite in a random order, catching hidden interdependencies between tests. Open ~11.5 months with no review or comments; 11 CI jobs fail because randomization exposes test coupling. Not yet merged. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
92
Complexity:
10
Confidence:
85
|
close stale | Enables random test execution order to detect hidden interdependencies. Currently pending review with zero comments, though multiple CI checks are failing across various platforms and Python versions. | |
| qwen3.6-35b-a3b-mtp-q6 |
Staleness:
92
Complexity:
10
Confidence:
88
|
close stale | Introduces random test execution to catch interdependencies via --randomseed. Inactive for ~10 months with no CI runs, maintainer feedback, or completed checklist. |
Update history
No update history recorded yet.
Related work
-
Related To:
craft-platforms#136
(confidence 90%)
Merged sibling implementation 'test: randomize the order of tests' (inspired by rockcraft#895), showing the same approach was accepted elsewhere in the same family.
-
Related To:
craft-parts#1308
(confidence 55%)
Merged fix marking network-sensitive chisel integration tests flaky; adjacent test-stability work around the same period but does not implement random ordering.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #136 test: randomize the order of tests | craft-platforms | merged | PR #136 merged as commit 1cf8225 ("test: randomize the order of tests"), adding pytest-randomly>=3.15.0 to test dependencies to catch order-dependent test failures; inspired by rockcraft#895. |