test: randomize the order of tests
Metadata
Current evaluation
PR #136 merged as commit 1cf8225 ("test: randomize the order of tests"), adding pytest-randomly>=3.15.0 to test dependencies to catch order-dependent test failures; inspired by rockcraft#895.
Resolution: closed resolved
Reason: The PR itself was merged by maintainer lengau with 2 approvals and no unresolved review comments; commit 1cf8225 on main introduced the pytest-randomly dependency (pyproject.toml line 41, uv.lock updated), so test order is now randomized.
Evaluated by: mr-cal-bot
No scores available.
Issue body
Inspired by https://github.com/canonical/rockcraft/pull/895
- [x] Have you followed the guidelines for contributing?
- [x] Have you signed the [CLA](http://www.ubuntu.com/legal/contributors/)?
- [x] Have you successfully run `make lint && make test`?
---
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | PR #136 merged as commit 1cf8225 ("test: randomize the order of tests"), adding pytest-randomly>=3.15.0 to test dependencies to catch order-dependent test failures; inspired by rockcraft#895. | |
| qwen/qwen3.6-35b-a3b | — | closed resolved | Merged to randomize test execution order, preventing order-dependent test failures. Inspired by canonical/rockcraft#895. |
Update history
| Date | Change |
|---|---|
| created |
Related work
-
Related To:
canonical/rockcraft#895
(confidence 92%)
Cited in the PR body as the inspiration for randomizing test order
-
Related To:
canonical/craft-platforms#221
(confidence 70%)
Commit cbd51a8 later did lock file maintenance, updating pytest-randomly in uv.lock
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #895 test: randomize the order of tests | rockcraft | merged | Merged canonical/rockcraft#895, adding randomized test ordering to uncover inter-test dependencies; landed as commit 2676b48e with all CI checks passing. | |
| #289 test: randomize the order of tests | craft-store | merged | Merged PR #289 in commit e0e4f4f, randomizing test execution order to prevent order-dependent failures; all CI checks passed. | |
| #1309 test: run tests in a random order | craft-parts | open | Adds pytest-randomly so craft-parts tests run in random order to surface order dependence. Maintainer-authored, 380 days old, zero review comments, no updates, and 11 CI jobs failing (most test-common/test-other slow+fast runs). |