Improve integration tests
Metadata
Current evaluation
Enhancement proposal to restructure integration tests, add timing, and create new test suites for bases verification. Open, awaiting implementation.
Suggested action: keep open
Reason: This is a well-structured enhancement request from maintainer mr-cal with clear goals (moving lengthy tests to weekly jobs, adding timing, creating new integration test suites for bases). The issue has the 'Type: Enhancement' label and should remain on the backlog for future implementation.
Impact:
60
Quick Win:
27.0
Staleness:
50
Complexity:
55
Confidence:
70
Support Request:
0
Issue body
### What needs to get done
#### Weekly tests
Move lengthy tests to a weekly test job. This may require reworking parameterization (i.e. a PR tests on `ubuntu-latest` and the weekly tests
on all supported releases).
#### Timing
Time the integration tests to inform which test suites need attention.
#### New integration tests
Add a new set of integration tests, `tests/integration/bases/`, to test the contents and behaviors inside instances and verify the setup/warmup steps are operating as intended.
Here are a few examples of what we could verify from the setup/warmup calls:
- correct packages installed
- snapd installed and running
- timezone matches host
- snap refreshes held
The tests should run on all bases on all providers. (this may add a lot of time to the tests)
### Why it needs to get done
The test suits are slow, do not verify behavior inside instances, and do not verify consistent behavior across providers.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen3.6-35b-a3b-mtp-q6 |
Impact:
60
Quick Win:
27.0
Staleness:
50
Complexity:
55
Confidence:
70
Support Request:
0
|
keep open | Enhancement proposal to restructure integration tests, add timing, and create new test suites for bases verification. Open, awaiting implementation. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
75
Complexity:
45
Confidence:
80
Support Request:
0
|
keep open | Proposal to restructure and expand integration tests by moving lengthy suites to weekly jobs, adding timing metrics, and verifying instance contents across providers. Currently awaiting implementation. | |
| qwen3.6-35b-a3b-mtp-q6 |
Staleness:
80
Complexity:
65
Confidence:
85
Support Request:
5
|
needs triage | Enhancement to restructure integration tests, add timing, and verify instance contents across providers. Open for nearly three years with no maintainer comments or assignee, awaiting triage. | |
| qwen3.6-35b-a3b-mtp-q6 |
Staleness:
70
Complexity:
65
Confidence:
80
Support Request:
5
|
needs triage | Enhancement request to restructure CI integration tests, add timing instrumentation, and implement new tests verifying instance setup and provider consistency. Open for ~3 years with no discussion or assignee. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #430 Improve integration tests | craft-providers | open | Enhancement proposal to restructure integration tests, either by migrating to spread or reorganizing existing tests to address disk space exhaustion on GitHub runners. |