ci: enable and stabilize multipass tests in QA
Metadata
Current evaluation
Maintainer CI PR enabling multipass integration tests in qa.yaml on a large self-hosted runner (LXD disabled), wiring setup-tests to install-multipass, waiting on multipassd with diagnostics, and fixing bytes-vs-string and cache-mount test assertions. All listed checks pass; 5 review comments unresolved.
Suggested action: needs review
Reason: Author is a maintainer (lengau) and the PR advances open issue #754, with recent activity (3 days) so it is not stale. However, it has 0 reviewers and 5 unresolved review comments, and the surfaced CI list contains only policy/CLA/security/copilot checks with no test job results, so the multipass job this PR enables is itself unverified here. Confidence is limited because I could not see the diff: the body says 'Keep only test-multipass enabled in qa.yaml', which may mean test-basic and test-lxd are being disabled, and the new install-multipass dependency on setup-tests also affects the unrelated LXD job in tests-weekly.yaml (line 48, `make -j setup-tests`), which would now run `sudo snap install multipass` unnecessarily.
Evaluated by: mr-cal-bot
Impact:
40
Quick Win:
11
Complexity:
45
Confidence:
64
Actionability:
48
Issue body
Summary: Enable and stabilize Multipass integration testing in CI by focusing QA on test-multipass and fixing test-side issues exposed by this mode.
Changes:
- Keep only test-multipass enabled in qa.yaml for this branch.
- Run slow multipass tests on self-hosted-linux-amd64-resolute-large.
- Disable LXD setup for this job (use-lxd false).
- Make setup-tests depend on install-multipass via Makefile.
- Wait for multipassd readiness before running tests.
- Dump multipass diagnostics on readiness timeout.
- Fix multipass integration test bytes-vs-string assertions.
- Make cache-mount assertion resilient to the documented fallback path.
Goal: make multipass CI runs reliable and actionable.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha |
Impact:
40
Quick Win:
11
Complexity:
45
Confidence:
64
Actionability:
48
|
needs review | Maintainer CI PR enabling multipass integration tests in qa.yaml on a large self-hosted runner (LXD disabled), wiring setup-tests to install-multipass, waiting on multipassd with diagnostics, and fixing bytes-vs-string and cache-mount test assertions. All listed checks pass; 5 review comments unresolved. | |
| stealth/space-bunny-alpha |
Impact:
40
Quick Win:
14
Complexity:
35
Confidence:
72
Actionability:
55
|
needs review | Maintainer PR re-enabling the commented-out test-multipass job in qa.yaml on a self-hosted runner, disabling LXD, wiring setup-tests to install-multipass, waiting for multipassd, and fixing multipass test assertions. All CI checks green, awaiting review. | |
| stealth/space-bunny-alpha |
Impact:
42
Quick Win:
17
Complexity:
45
Confidence:
72
Actionability:
72
|
needs review | Maintainer CI PR enabling only test-multipass in qa.yaml on a large self-hosted runner, disabling LXD, making setup-tests depend on install-multipass, waiting for multipassd with diagnostics, and fixing test assertions. All checks green; 5 review comments unresolved. | |
| stealth/space-bunny-alpha |
Impact:
38
Quick Win:
13
Complexity:
42
Confidence:
93
Actionability:
58
|
needs review | Enables QA Multipass tests on a dedicated runner, adds multipassd readiness diagnostics, and fixes test assertions. CI passes, but review is unassigned with four unresolved comments. | |
| qwen3.6-35b-a3b-mtp-q6 |
Impact:
35
Quick Win:
26
Staleness:
45
Complexity:
25
Confidence:
65
|
needs review | Enables and stabilizes multipass integration tests in CI by updating qa.yaml, Makefile, and test assertions. CI passing, 4 unresolved review comments, 0 reviewers assigned. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
85
|
needs review | Enables and stabilizes Multipass integration testing in CI by configuring qa.yaml, adjusting runner targets, fixing test assertions, and adding readiness checks. Currently pending maintainer review with 4 unresolved comments and 2 failing CI jobs. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
90
|
needs review | Enables and stabilizes Multipass integration tests in CI by updating qa.yaml, adding readiness checks, and fixing test assertions. Currently pending maintainer review with four unresolved comments and two failing CI checks on slow multipass jobs. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
90
|
needs review | Enables and stabilizes Multipass integration tests in CI by configuring QA workflows, fixing test assertions, and improving readiness checks. Currently pending maintainer review with one unresolved comment and mostly passing CI checks. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
85
|
needs review | Enables and stabilizes Multipass integration testing in CI by focusing QA, fixing test assertions, and improving setup readiness. Currently passing initial checks with several slow tests and policy checks pending. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
90
|
needs review | Enables and stabilizes Multipass integration tests in CI by configuring qa.yaml, adding readiness checks, and fixing assertion failures. Currently pending maintainer review with unresolved comments and several CI checks still running. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
90
|
needs review | Enables and stabilizes Multipass integration testing in CI by configuring QA workflows, fixing test assertions, and improving readiness checks. Currently pending review with 4 unresolved comments and one failing CI check (zizmor). | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
30
Confidence:
90
|
needs review | Enables and stabilizes Multipass integration tests in CI by focusing QA on test-multipass, fixing test assertions, and adjusting Makefile dependencies. Currently pending reviewer approval with slow CI jobs running. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
90
|
needs review | Enables and stabilizes Multipass integration tests in CI by updating qa.yaml, adjusting Makefile dependencies, fixing test assertions, and configuring self-hosted runners. Currently pending review with one unresolved comment and several pending CI checks. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
25
Confidence:
90
|
needs review | Enables and stabilizes Multipass integration tests in CI by updating QA config, fixing test assertions, and adjusting Makefile dependencies. Currently pending maintainer review with one unresolved comment and multiple CI checks still running. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
15
Confidence:
90
|
keep open | Temporarily disables most QA CI jobs except test-multipass for debugging multipass runner behavior. Marked as draft, pending maintainer review, with failing security scans and unresolved comments. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
15
Confidence:
90
|
keep open | Temporarily disables most QA workflow jobs to isolate test-multipass for debugging. Marked as draft, pending review, and explicitly notes it should not be merged until disabled jobs are restored. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
15
Confidence:
90
|
keep open | Temporarily disables most QA workflow jobs to isolate multipass testing for debugging. Marked as draft, pending review, and explicitly notes it should not be merged until disabled jobs are restored. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
5
Confidence:
90
|
keep open | Temporarily disables most QA workflow jobs to isolate multipass testing for debugging. Marked as draft, not ready for merge, and requires restoring disabled jobs before landing. |
Update history
| Date | Change |
|---|---|
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| created |
Related work
-
Likely Fixed By:
craft-providers#754
(confidence 82%)
Open enhancement 'Run Multipass tests on Linux CI'; test-multipass is still commented out at qa.yaml:47, which this PR enables.
-
Superseded By:
craft-providers#932
(confidence 75%)
Prior WIP PR doing the same thing (un-commenting test-multipass in qa.yaml), idle 196 days with failing checks; this smaller, cleaner PR appears to replace it.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #754 Run Multipass tests on Linux CI | craft-providers | open | Enhancement to run Multipass integration tests on Linux CI (nested virt now available). Still open: qa.yaml's test-multipass job remains commented out and WIP PR #932 is inactive with failing checks. | |
| #932 Work/ci linux multipass | craft-providers | open | Maintainer WIP enabling Multipass integration tests on Linux CI (uncommenting test-multipass in qa.yaml); +293/-178 over 5 files. Idle 196 days, 0 reviews/comments; policy and macOS slow-integration checks failing. Tracks open issue #754. | |
| #44 integration tests: consolidate lxd/multipass fixtures for reuse | craft-providers | merged | Merged PR #44 in commit ee0f643, consolidating reusable LXD and Multipass integration-test fixtures and including a documentation fix. |