← Back to issue list

test: run tests in a random order

View original Github issue

Metadata

Project
craft-parts
Number
#1309
Type
pull request
State
open
Author
lengau
Labels
—
Created
Updated
Closed
—

Current evaluation

Adds pytest-randomly to run the test suite in a random order, catching hidden interdependencies between tests. Open ~11.5 months with no review or comments; 11 CI jobs fail because randomization exposes test coupling. Not yet merged.

Suggested action: needs review

Reason: The PR is by maintainer lengau but has sat 347 days with 0 comments, 0 reviewers, and an unchecked 'make lint && make test' box. 11 CI jobs (mostly test-common slow/fast) fail because randomizing order surfaces real test interdependencies that were never fixed, so it is not mergeable as-is. The goal is still valid and not implemented elsewhere (no pytest-randomly in pyproject.toml), so a maintainer must decide whether to pick it up and resolve the coupling or close it.

Impact: 35 Quick Win: 21.0 Staleness: 78 Complexity: 40 Confidence: 78

Issue body

This should help catch surprise interdependence issues between tests. One can reproduce a specific order using the `--randomseed` provided at the top of the test output. - [x] Have you followed the guidelines for contributing? - [x] Have you signed the [CLA](http://www.ubuntu.com/legal/contributors/)? - [ ] Have you successfully run `make lint && make test`? - [ ] Have you added an entry to the changelog (`docs/reference/changelog.rst`)? ---

Evaluation history

Date Model Scores Action Summary
qwen/qwen3.8-27b
Impact: 35
Quick Win: 21.0
Staleness: 78
Complexity: 40
Confidence: 78
needs review Adds pytest-randomly to run the test suite in a random order, catching hidden interdependencies between tests. Open ~11.5 months with no review or comments; 11 CI jobs fail because randomization exposes test coupling. Not yet merged.
qwen/qwen3.6-35b-a3b
Staleness: 92
Complexity: 10
Confidence: 85
close stale Enables random test execution order to detect hidden interdependencies. Currently pending review with zero comments, though multiple CI checks are failing across various platforms and Python versions.
qwen3.6-35b-a3b-mtp-q6
Staleness: 92
Complexity: 10
Confidence: 88
close stale Introduces random test execution to catch interdependencies via --randomseed. Inactive for ~10 months with no CI runs, maintainer feedback, or completed checklist.
qwen3.6-35b-a3b-mtp-q6
Staleness: 75
Complexity: 15
Confidence: 65
needs review Runs tests in random order to detect hidden interdependencies, with --randomseed support for reproducibility. Currently open, awaiting review, and pending lint/test execution and changelog entry.

Update history

No update history recorded yet.

Related issues

No related issues found above the similarity threshold.