tests: fix validation sets spread test
Metadata
Current evaluation
Merged fix for validation sets spread test switching to the hello-world snap and flipping the presence check instead of a specific revision. Approved by two reviewers with passing CI, resolving issue #5053.
Suggested action: —
No scores available.
Issue body
- [x] Have you followed the [guidelines for contributing](https://github.com/canonical/snapcraft/blob/main/CONTRIBUTING.md)?
- [x] Have you signed the [CLA](http://www.ubuntu.com/legal/contributors/)?
- [x] Have you successfully run `tox run -m lint`?
- [x] Have you successfully run `tox run -e test-py310`? (supported versions: `py39`, `py310`, `py311`, `py312`)
-----
[Behind the scenes](https://github.com/canonical/snapcraft/pull/5074/files#diff-610b821ececc50adda118ec968fe837957ad1941ad14c938e7e61830d9b0d929R7-R17), I updated the validation sets spread test to use the `hello-world` snap.
This PR flip-flops the `presence` instead of the `revision`, which should make it a little more robust because it won't depend on a specific revision of the snap being accessible.
Fixes #5053
(CRAFT-3418)
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen/qwen3.6-35b-a3b | — | — | Merged fix for validation sets spread test switching to the hello-world snap and flipping the presence check instead of a specific revision. Approved by two reviewers with passing CI, resolving issue #5053. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged a fix for the validation sets spread test. The update switches to the hello-world snap and flips the presence check instead of revision to improve robustness. Resolves #5053. Reviewer noted unrelated test errors. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged a fix for the validation sets spread test. Using hello-world and flipping the presence check instead of revision improves robustness. Resolves #5053. Unrelated errors were noted but did not block the merge. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #4924 tests: add spread tests for validation-sets | snapcraft | merged | Merged changes adding spread tests for edit-validation-sets and list-validation-sets. Approved by two reviewers with passing CI, resolving issue #4918. | |
| #534 test(spread): workaround for flaky test | fetch-service | merged | Merged a workaround for a flaky spread test (issue #533). The change adjusts one file (+2/-1 lines) to skip a no-op name query check while retaining other validations. Approved by two reviewers and passed all CI checks. | |
| #1366 tests(spread): make release test less flaky | charmcraft | merged | Merged changes that reduce flakiness in the spread release test, resolving issue #1362. Approved by two reviewers, the update modifies one file (+17/-7) after addressing feedback. | |
| #1029 fix(testing): error early if spread.yaml is invalid. | craft-application | merged | Merged after two approvals. Adds early validation to fail tests immediately if spread.yaml is invalid. CI passed except for an unrelated snap-test failure, which was overridden. | |
| #240 tests: check for submodules before running spread | rockcraft | merged | Merged. Adds a pre-check for the snapd-testing-tools submodule before running spread tests, providing a clear error if missing. Approved by two reviewers, passed CI, and merged successfully. | |
| #521 test(spread): add an improved snap inspector test | fetch-service | merged | Merged an improved snap inspector test that emulates system setup by stopping snapd, clearing device files, setting a global proxy, and restarting snapd to uncover network requests. Approved by two reviewers, passed all CI checks, and merged. | |
| #926 test: address a couple of non-flaky spread failures | rockcraft | merged | Merged after two approvals and passing all CI checks. Resolves non-flaky spread test failures by updating test configurations across three files, eliminating consistent failures. | |
| #456 fix(spread): tests must also use newer Chisel | rockcraft | merged | Merged. Resolves failing spread tests by switching Chisel installation from a hardcoded commit to the snap store. Addressed review feedback and re-triggered CI after a timeout. | |
| #1084 tests: check for submodules before running spread | charmcraft | merged | Merged changes verifying the snapd-testing-tools submodule before running Spread tests. Adds a clear error if missing. Approved by a reviewer, passed CI, and merged with minimal modifications. | |
| #4105 tests: check for submodules before running spread | snapcraft | merged | Merged. Adds a pre-run check for the snapd-testing-tools submodule in spread tests to provide a clear error if missing. Approved by two reviewers, passed CI, and introduced no coverage changes. |