test: randomize the order of tests
Metadata
Current evaluation
Merged canonical/rockcraft#895, adding randomized test ordering to uncover inter-test dependencies; landed as commit 2676b48e with all CI checks passing.
Resolution: closed resolved
Reason: PR #895 received two approvals and was merged into main as commit 2676b48e. It adopted pytest-randomly to randomize test execution.
Evaluated by: mr-cal-bot
No scores available.
Issue body
- [ ] Have you followed the guidelines for contributing?
- [ ] Have you signed the [CLA](http://www.ubuntu.com/legal/contributors/)?
- [ ] Have you successfully run `make lint && make test`?
---
In order to avoid inter-test dependencies, tests should be randomized. This is a pilot PR to see how easy this would be to adopt into all craft repositories.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Merged canonical/rockcraft#895, adding randomized test ordering to uncover inter-test dependencies; landed as commit 2676b48e with all CI checks passing. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged in canonical/rockcraft#895 as a pilot randomizing test execution to reduce order-dependent failures; all CI checks passed and two reviewers approved it. | |
| qwen/qwen3.6-35b-a3b | — | — | Merged a pilot implementation to randomize test execution order, preventing inter-test dependencies. Approved by two reviewers with all CI checks passing. Diff adds 16 lines and removes 1 across two files. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged a pilot change to randomize test execution order, eliminating inter-test dependencies. The update establishes a reusable testing pattern for broader adoption across craft repositories. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #289 test: randomize the order of tests | craft-store | merged | Merged PR #289 in commit e0e4f4f, randomizing test execution order to prevent order-dependent failures; all CI checks passed. | |
| #136 test: randomize the order of tests | craft-platforms | merged | PR #136 merged as commit 1cf8225 ("test: randomize the order of tests"), adding pytest-randomly>=3.15.0 to test dependencies to catch order-dependent test failures; inspired by rockcraft#895. | |
| #1304 test(spread): fix cross-test contamination | rockcraft | merged | Merged PR #1304: spread variants shared the python-from-chisel-bare dir, causing cross-test contamination; each test now copies a clean rockcraft.yaml template in first (+6 lines, 2 files). |