ci: separate flaky tests into their own run
Metadata
Current evaluation
Merged PR #1088 in commit 6d755f00, splitting flaky slow tests into dedicated test-flaky CI runs for quicker reruns during network outages.
Resolution: closed resolved
Reason: PR #1088 was approved and merged as commit 6d755f00. All listed regular and test-flaky CI jobs passed.
Evaluated by: mr-cal-bot
No scores available.
Issue body
This should make it quicker and easier to re-run flaky tests once the network issue breaking them is resolved.
- [x] Have you followed the guidelines for contributing?
- [x] Have you signed the [CLA](http://www.ubuntu.com/legal/contributors/)?
- [x] Have you successfully run `make lint && make test`?
- [ ] Have you added an entry to the changelog (`docs/reference/changelog.rst`)?
---
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Merged PR #1088 in commit 6d755f00, splitting flaky slow tests into dedicated test-flaky CI runs for quicker reruns during network outages. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged PR #1088 separated flaky tests into dedicated test-flaky runs, simplifying reruns during network-related failures; all listed CI checks passed. | |
| qwen/qwen3.6-35b-a3b | — | — | Merged changes that isolate flaky tests into a dedicated CI workflow, streamlining re-execution after network disruptions. Approved by two reviewers with all checks passing across multiple OS and Python versions. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged a maintainer's pull request isolating flaky tests into a dedicated CI run. This separation simplifies re-executing unstable tests after network issues resolve, streamlining debugging and improving CI workflow efficiency. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1583 ci: separate flaky tests into their own job | craft-parts | merged | Merged PR #1583 in commit ec6458af, separating @pytest.mark.flaky tests into a dedicated test-flaky CI job for quick, independent reruns. | |
| #5899 ci: improve the state of some flaky tests | snapcraft | merged | Merged PR #5899 as commit 9db127202, stabilizing several flaky CI tests; store, remote-build, and snapd-bases failures remained out of scope. | |
| #1090 ci: silence the fast flaky tests | craft-application | merged | PR #1090 merged a one-file CI adjustment to silence flaky fast-test runs; all checks passed. Commit: 51736da5. | |
| #391 fix(test): make endpoint tests less flaky | craft-store | merged | Merged PR #391 as commit c42cd69, updating two test files to reduce endpoint-test flakiness on slow CI machines. |