ci: speed up spread test suite
Metadata
Current evaluation
Merged PR #5418 (commit a5682c276, SNAPCRAFT-353) sped up the spread test suite from 1h40m to 40m: split slow tests into a manually triggered workflow, used wheels, added store retries, fixed the matter test. Fixes #5296.
Resolution: closed resolved
Reason: PR #5418 was approved and merged as commit a5682c276 ('ci: speed up spread test suite'), which closed the issue it declared as fixing (#5296, 'Change ROS spread testing strategy'). The slow-workflow split persisted and was later renamed/improved in #5885.
Evaluated by: mr-cal-bot
No scores available.
Issue body
- [x] Have you followed the [guidelines for contributing](https://github.com/canonical/snapcraft/blob/main/CONTRIBUTING.md)?
- [x] Have you signed the [CLA](http://www.ubuntu.com/legal/contributors/)?
- [x] Have you successfully run `make lint`?
- [x] Have you successfully run `make test`?
---
This PR makes many improvements to the spread test suite to speed up the CI workflow.
Previously, the spread tests took 1 hour and 40 minutes to complete and 3 hours of total compute time.
Now, it takes 40 minutes to complete and 1 hour and 40 minutes of total compute time.
* Move extension, colcon, and core20 plugin tests to the slow workflow
* Allow triggering the slow workflow with a label ([using rockcraft's design](https://github.com/canonical/rockcraft/blob/main/.github/workflows/spread-manual.yaml))
* Use wheels when building the snap for spread tests
* Retry calls to the store that often fail due to store-side issues
* Fix the matter plugin spread test
Fixes #5296
(SNAPCRAFT-353)
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Merged PR #5418 (commit a5682c276, SNAPCRAFT-353) sped up the spread test suite from 1h40m to 40m: split slow tests into a manually triggered workflow, used wheels, added store retries, fixed the matter test. Fixes #5296. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged PR #5418 (commit a5682c276) sped up spread tests from 1h40m to 40m, moved slow tests, used wheels, added store retries, and fixed the matter plugin test. | |
| qwen/qwen3.6-35b-a3b | — | — | Merged after approval, cutting CI spread test execution from 1h40m to 40m. Changes route slow tests to a dedicated workflow, use build wheels, add store retries, and fix the matter plugin test. Proceeded despite unrelated CI timeouts. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged. Reduced spread test CI runtime from 1h40m to 40m by routing tests to a slow workflow, adding manual triggers, using build wheels, adding store retries, and fixing the matter plugin test. Unrelated CI failures were confirmed as external. |
Update history
No update history recorded yet.
Related work
-
Likely Fixed By:
canonical/snapcraft#5296
(confidence 97%)
Issue #5296 was closed by this PR; its title and the SNAPCRAFT-353 reference appear in the merge commit.
-
Related To:
canonical/snapcraft#5885
(confidence 80%)
Follow-up commit 'feat: rename manual spread file, fix using wheels' builds directly on the manual spread workflow added here.
-
Related To:
canonical/snapcraft#5899
(confidence 65%)
Later CI work continuing spread test flakiness reduction (store retries, test fixes).
-
Related To:
canonical/snapcraft#6335
(confidence 55%)
Further manual spread workflow hardening: 'ci: always discard spread workers in manual tests'.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #432 ci: speed up spread testing workflow | fetch-service | merged | Merged PR #432 (commit 9eec284), removing snap-test's dependency on snap-build so spread tests start sooner while snap-build still produces downloadable artifacts. | |
| #944 test(spread): speed up the skip-retest spread test | craft-application | closed | PR #944 was closed unmerged and superseded by craft-application#964, which carried the spread-test speedup changes forward and merged the adjusted repack-test setup. | |
| #2774 test: speed up spread test runtime by removing redundant package setup | charmcraft | open | Removes redundant apt update calls and consolidates package installs in spread tests to reduce runtime. All CI passing, awaiting review of 1 unresolved comment. | |
| #2614 [WIP] travis-ci: split spread tests to reduce job times | snapcraft | closed | Closed unmerged in favor of snapcraft#2615, which increased Spread worker parallelism and merged as commit 5c91f9681 to reduce job times. | |
| #534 test(spread): workaround for flaky test | fetch-service | merged | Merged PR #534 (commit 4f89bb7) worked around #533 by skipping the flaky snap names-query assertion while retaining the other spread-test checks. | |
| #3090 ci: move staging store tests to spread | snapcraft | merged | Merged into master in commit 3a037fb11, moving staging store tests from the legacy suite to Spread and removing 1,434 lines across 27 files. | |
| #6408 Re-organize spread tests | snapcraft | open | Maintainer-filed, triaged enhancement to reorganize/dedupe Snapcraft's spread tests and drop low-value ones for faster, less flaky CI; no implementation plan or PR yet, tracked as SNAPCRAFT-1377. |