ci: separate flaky tests into their own job
Metadata
Current evaluation
Merged PR isolates flaky tests into a dedicated test-flaky CI job, enabling independent reruns and avoiding full suite re-executions. Approved by two reviewers and merged.
Suggested action: —
No scores available.
Issue body
Move tests marked with `@pytest.mark.flaky` into a dedicated `test-flaky` CI job so they can be rerun independently and quickly when they fail due to transient issues (e.g. Ubuntu archive syncs).
This avoids needing to re-run the full suite of slow tests if flaky tests fail.
Artifact upload failure is fixed in https://github.com/canonical/starflow/pull/139
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen/qwen3.6-35b-a3b | — | — | Merged PR isolates flaky tests into a dedicated test-flaky CI job, enabling independent reruns and avoiding full suite re-executions. Approved by two reviewers and merged. | |
| qwen3.6-35b-a3b-mtp-q6 |
Staleness:
35
Complexity:
15
Confidence:
75
|
needs review | Separates flaky pytest tests into a dedicated CI job for independent reruns, avoiding full suite re-runs. Open pull request with no comments or labels, last updated 37 days ago, awaiting review or merge. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1088 ci: separate flaky tests into their own run | craft-application | merged | Merged changes that isolate flaky tests into a dedicated CI workflow, streamlining re-execution after network disruptions. Approved by two reviewers with all checks passing across multiple OS and Python versions. | |
| #5899 ci: improve the state of some flaky tests | snapcraft | merged | Merged CI updates fixing flaky tests. Several unstable tests were resolved, while integration store, remote build, and snapd tests were excluded due to upstream issues. Approved by two reviewers and passed all CI checks before merge. | |
| #1090 ci: silence the fast flaky tests | craft-application | merged | Merged after two approvals and passing CI checks. The change updates CI configuration to silence fast flaky tests, reducing test noise with a single file modification. | |
| #107 ci: separate unit and integration tests | starbase | merged | Merged into main after approval and passing CI checks. The PR separated unit and integration tests in the CI pipeline, modifying two files (+46/-7 lines). Code coverage remained unchanged at 100%. | |
| #1316 fix(test): mark complex script piping test as flaky | craft-parts | merged | Merged to mark a complex script piping test as flaky, resolving intermittent failures on ppc64el and s390x runners caused by pipe ordering. Approved by reviewers, passed all CI checks, and successfully merged. |