← Back to issue list

Improve integration tests

View original Github issue

Metadata

Project
craft-providers
Number
#381
Type
issue
State
open
Author
mr-cal
Labels
Type: Enhancement
Created
Updated
Closed
—

Current evaluation

Integration-test overhaul is partially implemented: weekly macOS fast/slow jobs and pytest timing exist, but base-instance behavior tests and full base/provider coverage remain absent.

Suggested action: keep open

Reason: The maintainer-authored enhancement remains valid: its explicit base-instance checks and all-base/all-provider matrix are not present in the current tree, which still lacks tests/integration/bases. Existing weekly macOS fast/slow jobs and pytest timing address only part of the scope, while open #430 substantially overlaps the remaining integration-test work.

Evaluated by: mr-cal-bot

Impact: 58 Quick Win: 10 Complexity: 76 Confidence: 91 Actionability: 74

Issue body

### What needs to get done #### Weekly tests Move lengthy tests to a weekly test job. This may require reworking parameterization (i.e. a PR tests on `ubuntu-latest` and the weekly tests on all supported releases). #### Timing Time the integration tests to inform which test suites need attention. #### New integration tests Add a new set of integration tests, `tests/integration/bases/`, to test the contents and behaviors inside instances and verify the setup/warmup steps are operating as intended. Here are a few examples of what we could verify from the setup/warmup calls: - correct packages installed - snapd installed and running - timezone matches host - snap refreshes held The tests should run on all bases on all providers. (this may add a lot of time to the tests) ### Why it needs to get done The test suits are slow, do not verify behavior inside instances, and do not verify consistent behavior across providers.

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha
Impact: 58
Quick Win: 10
Complexity: 76
Confidence: 91
Actionability: 74
keep open Integration-test overhaul is partially implemented: weekly macOS fast/slow jobs and pytest timing exist, but base-instance behavior tests and full base/provider coverage remain absent.
stealth/space-bunny-alpha
Impact: 47
Quick Win: 13
Complexity: 63
Confidence: 91
Actionability: 72
keep open Integration-test enhancement remains open. Weekly macOS fast/slow jobs and pytest timing exist, but no tests/integration/bases directory is present and full base/provider coverage is not demonstrated.
stealth/space-bunny-alpha
Impact: 48
Quick Win: 15
Complexity: 61
Confidence: 94
Actionability: 78
keep open Partially implemented: weekly macOS jobs and duration reporting exist, but no cross-provider tests/integration/bases suite or comprehensive all-base setup/warmup checks.
qwen3.6-35b-a3b-mtp-q6
Impact: 60
Quick Win: 27
Staleness: 50
Complexity: 55
Confidence: 70
Support Request: 0
keep open Enhancement proposal to restructure integration tests, add timing, and create new test suites for bases verification. Open, awaiting implementation.
qwen/qwen3.6-35b-a3b
Staleness: 75
Complexity: 45
Confidence: 80
Support Request: 0
keep open Proposal to restructure and expand integration tests by moving lengthy suites to weekly jobs, adding timing metrics, and verifying instance contents across providers. Currently awaiting implementation.
qwen3.6-35b-a3b-mtp-q6
Staleness: 80
Complexity: 65
Confidence: 85
Support Request: 5
needs triage Enhancement to restructure integration tests, add timing, and verify instance contents across providers. Open for nearly three years with no maintainer comments or assignee, awaiting triage.

Update history

No update history recorded yet.

Related work

  • Related To: craft-providers#430 (confidence 92%)

    Open issue with the same title and overlapping goals to overhaul integration-test infrastructure, parametrization, cleanup, and execution.

Related issues

No related issues found above the similarity threshold.