feat!(test): switch from the custom test command to upstream
Metadata
Current evaluation
Merged PR charmcraft#2385 (commit b7730efc) for CHARMCRAFT-631, replacing charmcraft's custom 'test' command with craft-application's upstream test infrastructure; it fixes and closes issue #2364. Some spread tests on 20.04/22.04 still fail.
Resolution: closed resolved
Reason: The PR was approved and merged as commit b7730efc with the message 'feat!(test): switch from the custom test command to upstream (#2385)', implementing CHARMCRAFT-631 and closing issue #2364; cmatsuoka commented 'Thanks!' on it.
Evaluated by: mr-cal-bot
No scores available.
Issue body
CHARMCRAFT-631
Fixes #2364
The following spread tests are known to be broken and are unrelated:
- google:ubuntu-20.04-64:tests/spread/smoketests/basic:focal_bases
- google:ubuntu-20.04-64:tests/spread/smoketests/different-dir
- google:ubuntu-20.04-64:tests/spread/smoketests/full-lifecycle:kubernetes
- google:ubuntu-20.04-64:tests/spread/smoketests/full-lifecycle:machine
- google:ubuntu-20.04-64:tests/spread/smoketests/lxd
- google:ubuntu-20.04-64:tests/spread/smoketests/pinned-dependencies
- google:ubuntu-20.04-64:tests/spread/smoketests/reactive:stable
- google:ubuntu-20.04-64:tests/spread/smoketests/reactive:two
- google:ubuntu-22.04-64:tests/spread/dependencies/charmlibs
- google:ubuntu-22.04-64:tests/spread/dependencies/setuptools-scm-python:charm
- google:ubuntu-22.04-64:tests/spread/dependencies/setuptools-scm-python:python
- google:ubuntu-22.04-64:tests/spread/smoketests/basic:focal_bases
- google:ubuntu-22.04-64:tests/spread/smoketests/different-dir
- google:ubuntu-22.04-64:tests/spread/smoketests/full-lifecycle:kubernetes
- google:ubuntu-22.04-64:tests/spread/smoketests/full-lifecycle:machine
- google:ubuntu-22.04-64:tests/spread/smoketests/pinned-dependencies
- google:ubuntu-22.04-64:tests/spread/smoketests/reactive:stable
- google:ubuntu-22.04-64:tests/spread/smoketests/reactive:two
- google:ubuntu-22.04-64:tests/spread/smoketests/help-slug
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Merged PR charmcraft#2385 (commit b7730efc) for CHARMCRAFT-631, replacing charmcraft's custom 'test' command with craft-application's upstream test infrastructure; it fixes and closes issue #2364. Some spread tests on 20.04/22.04 still fail. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged PR #2385 (commit b7730efc) replaced the custom test command with upstream testing infrastructure, resolving #2364. Unrelated spread-test failures were documented as known. | |
| qwen/qwen3.6-35b-a3b | — | — | Merged. Replaced the custom test command with the upstream version, resolving #2364. Approved after review with CI passing. Several unrelated spread tests remain known to be broken. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged the switch from the custom test command to the upstream version, resolving #2364. Several unrelated spread tests remain broken. Reviewers approved with a note to add init-test coverage once templates are available. |
Update history
No update history recorded yet.
Related work
-
Likely Fixed By:
charmcraft#2364
(confidence 96%)
The PR body states 'Fixes #2364'; that issue is now closed.
-
Related To:
canonical/craft-application#723
(confidence 72%)
Upstream craft-application testing improvements (merged, commit 6b319b77) that charmcraft's switch to the upstream test command builds on.
-
Related To:
canonical/craft-application#613
(confidence 65%)
Upstream tracking of spread testing support in craft-application that this migration adopts.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #2364 Update charmcraft test to use the new testing infrastructure | charmcraft | closed | Fixed by merged PR charmcraft#2385 (commit b7730efc), replacing Charmcraft’s custom test command with craft-application’s upstream test infrastructure. | |
| #2426 fix(test): fix the reactive charm spread test | charmcraft | merged | Merged in canonical/charmcraft#2426 via commit 7c33dd1b, applying a one-line fix to the reactive-charm spread test for CHARMCRAFT-641. | |
| #1988 feat: add --verbose flag to pass -vv to spread from charmcraft test | charmcraft | closed | Closed unmerged as superseded by PR #2385 (commit b7730efc), which switched charmcraft test to the upstream test command. | |
| #1198 tests: add charm library multiple versions update test | charmcraft | merged | Merged PR #1198 (commit 154bf5d6) added a 120-line regression test covering updates across multiple charm-library versions, linked to #1020/CRAFT-1916. | |
| #1660 feat: add test command and templates | charmcraft | merged | Merged as commit 097a4e9a, adding `charmcraft test` with templates for local or remote Spread suites using a Multipass backend. | |
| #49 ci: test the charmcraft actions in more places | craft-actions | merged | Merged PR #49 in commit a8e5310, expanding Charmcraft action CI coverage across more Ubuntu architectures, channels, and pinned-revision runs. | |
| #132 test: update charm integration test | craft-actions | merged | Merged in canonical/craft-actions#132 (54f7849), updating the charm pack integration test to use another simple charm after the previous Juju fixture was removed. | |
| #1556 refactor(tests/spread): test with explicit charmcraft.yaml files | charmcraft | merged | Merged PR #1556 in commit 4682b32b, replacing prepare-step manipulation of charmcraft.yaml with explicit spread-test configuration files. | |
| #1214 tests: add more charms to large spread tests | charmcraft | closed | Closed unmerged after 1,135 days; its large-spread test expansion landed through merged PR canonical/charmcraft#1217 in commit 0f4591a9. | |
| #1287 fix(tests): fixes bundle spread test | charmcraft | merged | Merged PR #1287 (commit f60186ce, CRAFT-2052) fixed the failing bundle spread test by validating YAML with Python and dropping a non-working charm path check, resolving #1262. |