ci: improve the state of some flaky tests
Metadata
Current evaluation
Merged CI updates fixing flaky tests. Several unstable tests were resolved, while integration store, remote build, and snapd tests were excluded due to upstream issues. Approved by two reviewers and passed all CI checks before merge.
Suggested action: —
No scores available.
Issue body
Improves/fixes a number of flaky tests. I recommend reviewing per-commit.
The integration store tests are currently broken and seem to need store-side fixes, so they will be left out of this PR. The remote build tests are failing for similar reasons. The snapd failures are upstream build errors, which I have alerted the snapd team of.
---
- [x] I've followed the [contribution guidelines](https://github.com/canonical/snapcraft/blob/main/CONTRIBUTING.md).
- [x] I've signed the [CLA](http://www.ubuntu.com/legal/contributors/).
- [x] I've successfully run `make lint && make test`.
- [x] I've added or updated any relevant documentation.
- [x] I've updated the relevant release notes.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen/qwen3.6-35b-a3b | — | — | Merged CI updates fixing flaky tests. Several unstable tests were resolved, while integration store, remote build, and snapd tests were excluded due to upstream issues. Approved by two reviewers and passed all CI checks before merge. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged CI improvements addressing flaky tests. Integration store tests, remote build tests, and snapd failures were excluded due to upstream issues. All included changes passed linting and testing checks. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged CI improvements addressing flaky tests. Integration store and remote build tests were excluded due to external store-side issues, while upstream snapd build failures were reported to the snapd team. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1583 ci: separate flaky tests into their own job | craft-parts | merged | Merged PR isolates flaky tests into a dedicated test-flaky CI job, enabling independent reruns and avoiding full suite re-executions. Approved by two reviewers and merged. | |
| #18 ci: ensure tests succeed in CI | craft-application | merged | Merged updates to CI configuration ensuring tests pass successfully. Approved by one reviewer, modifying five files with 23 additions and 13 deletions. | |
| #1088 ci: separate flaky tests into their own run | craft-application | merged | Merged changes that isolate flaky tests into a dedicated CI workflow, streamlining re-execution after network disruptions. Approved by two reviewers with all checks passing across multiple OS and Python versions. | |
| #82 test: improve integration testing | craft-store | merged | Merged improvements to integration testing across eight files. The update works around external fork credential limitations by running full tests only after merging to main. Approved by one reviewer and passed CI. | |
| #1090 ci: silence the fast flaky tests | craft-application | merged | Merged after two approvals and passing CI checks. The change updates CI configuration to silence fast flaky tests, reducing test noise with a single file modification. | |
| #1803 ci: correctly run from snap | snapcraft | merged | Merged a CI fix to correctly run tests from snap by separating development and runtime dependencies. Approved by one reviewer, passed CI, and resolved dependency isolation to expose hidden bugs for future fixes. | |
| #391 fix(test): make endpoint tests less flaky | craft-store | merged | Merged changes that stabilize endpoint tests in CI by reducing flakiness on slower machines. Approved with all CI checks passing. Minimal adjustments across two files resolve test instability. | |
| #46 ci: test on more platforms | debcraft | merged | Merged after two reviewer approvals and passing all CI checks. The change expands continuous integration testing to additional Ubuntu versions and architectures, enabling earlier issue detection with minimal modifications across two files. | |
| #1910 ci: fix tics tests | charmcraft | merged | Merged a single-file update to fix TICS CI tests. Approved by two reviewers with no unresolved comments. The change resolves failing continuous integration checks. | |
| #1600 test(spread): move flaky parallel-build test to weeklies | charmcraft | merged | Merged to address issue #1332 by relocating a flaky parallel-build test to the weekly suite. Approved by two reviewers, the change modifies two files (+19/-2) to stabilize CI runs. |