ci: improve the state of some flaky tests
Metadata
Current evaluation
Merged PR #5899 as commit 9db127202, stabilizing several flaky CI tests; store, remote-build, and snapd-bases failures remained out of scope.
Resolution: closed resolved
Reason: PR #5899 was approved and merged into main as commit 9db127202. It fixed a subset of flaky tests while explicitly leaving integration-store, remote-build, and upstream snapd build failures unresolved.
Evaluated by: mr-cal-bot
No scores available.
Issue body
Improves/fixes a number of flaky tests. I recommend reviewing per-commit.
The integration store tests are currently broken and seem to need store-side fixes, so they will be left out of this PR. The remote build tests are failing for similar reasons. The snapd failures are upstream build errors, which I have alerted the snapd team of.
---
- [x] I've followed the [contribution guidelines](https://github.com/canonical/snapcraft/blob/main/CONTRIBUTING.md).
- [x] I've signed the [CLA](http://www.ubuntu.com/legal/contributors/).
- [x] I've successfully run `make lint && make test`.
- [x] I've added or updated any relevant documentation.
- [x] I've updated the relevant release notes.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Merged PR #5899 as commit 9db127202, stabilizing several flaky CI tests; store, remote-build, and snapd-bases failures remained out of scope. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged as commit 9db127202, stabilizing several flaky CI tests; integration-store, remote-build, and snapd-bases failures were excluded pending external fixes. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged as 9db127202: stabilized several flaky CI tests; store integration, remote-build, and snapd-bases failures remained explicitly out of scope. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged in PR #5899 via commit 9db127202, stabilizing several flaky CI tests; integration-store, remote-build, and snapd-bases failures remained outside scope. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged as commit 9db127202 (PR #5899), stabilizing several flaky CI tests; integration-store, remote-build, and snapd-bases failures remained outside its scope. | |
| qwen/qwen3.6-35b-a3b | — | — | Merged CI updates fixing flaky tests. Several unstable tests were resolved, while integration store, remote build, and snapd tests were excluded due to upstream issues. Approved by two reviewers and passed all CI checks before merge. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged CI improvements addressing flaky tests. Integration store tests, remote build tests, and snapd failures were excluded due to upstream issues. All included changes passed linting and testing checks. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1088 ci: separate flaky tests into their own run | craft-application | merged | Merged PR #1088 in commit 6d755f00, splitting flaky slow tests into dedicated test-flaky CI runs for quicker reruns during network outages. | |
| #1090 ci: silence the fast flaky tests | craft-application | merged | PR #1090 merged a one-file CI adjustment to silence flaky fast-test runs; all checks passed. Commit: 51736da5. | |
| #391 fix(test): make endpoint tests less flaky | craft-store | merged | Merged PR #391 as commit c42cd69, updating two test files to reduce endpoint-test flakiness on slow CI machines. | |
| #534 test(spread): workaround for flaky test | fetch-service | merged | Merged PR #534 (commit 4f89bb7) worked around #533 by skipping the flaky snap names-query assertion while retaining the other spread-test checks. |