Warn on validation set update without incrementing sequence number
Metadata
Current evaluation
Added the validation-set sequence warning in merged PR #5535 (commit f012b667e).
Resolution: closed resolved
Reason: PR #5535 resolved the issue in commit f012b667e. It warns when an edited validation set does not increment its sequence and allows the assertion to be amended before submission.
Evaluated by: mr-cal-bot
No scores available.
Issue body
### What needs to get done
Issue a warning when snapcraft edit-validation-sets <account-id> <set-name> <sequence> is used to modify an assertion without incrementing the sequence number.
**BEHAVIOR:**
The warning should highlight that:
- if anything goes wrong with an intended snap refresh, snap will not automatically revert to a valid state, the validation set state will become and remain invalid.
- if the validation set specifies e.g. a revision that exists in the store and is not available on the architecture it will still refresh the validation set and become invalid without warning.
Perhaps something like: `WARNING: not incrementing the sequence number means invalid changes or snap refresh failure will invalidate this assertion with no means to automatically revert to a valid state`
Provide the opportunity to edit again, as is already the case with store error
**APPROACH:**
I am aware this can in theory be solved by the store as well, leaving it up to snapcraft team to decide on preferred approach.
### Why it needs to get done
This request stems from an investigation into unexpected behavior observed by an early adopter of the validation-sets feature, as reported in SF. The outcome led us to re-evaluate the implications of making changes within incrementing assertion sequence number:
When a device is pinned to an enforced and valid validation set at sequence <n>, subsequent mutation of the SAME SEQUENCE via `snapcraft edit-validation-sets <account-id> <set-name> <n>`, on refresh, can cause transition to invalid state due to things like:
- refresh failure combined with inability to revert due to non-distinct sequence
- revision of the correct snap but for a different architecture which is not store checked during submission
The general idea/promise with enforced pinned validation sets are: `An assertion will move to the latest sequence if present in the store and if the installed snaps, including any newer revisions in the store, still satisfy their respective validation set assertions.` (https://snapcraft.io/docs/validation-sets) So at a sequence level it guards against actions that will cause transition to invalid. This is not the case with sub-sequence changes, which presents the real danger of accidentally invalidating a managed fleet.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Added the validation-set sequence warning in merged PR #5535 (commit f012b667e). | |
| stealth/space-bunny-alpha | — | closed resolved | Added an edit warning when a validation set is changed without incrementing its sequence; implemented by merged PR #5535 (commit f012b667e). | |
| qwen/qwen3.6-35b-a3b | — | — | Implemented via PR #5535. Added a client-side warning prompt when editing a validation set without incrementing the sequence number to prevent accidental fleet invalidation. Changes are available in the latest/edge channel. | |
| qwen/qwen3.6-35b-a3b | — | — | Implemented via PR #5535. Added a client-side warning prompt when editing validation sets without incrementing the sequence number to prevent accidental fleet invalidation. The fix is available in the latest/edge/pr-5535 channel. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Resolved via PR #5535. Added a client-side warning when updating validation sets without incrementing the sequence number, preventing accidental fleet invalidation. The fix is available on the latest/edge channel. |
Update history
No update history recorded yet.
Related work
-
Likely Fixed By:
canonical/snapcraft#5535
(confidence 100%)
Merged PR implementing the requested sequence validation and retryable warning.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #5545 fix(validation-sets): allow users to ignore sequence increment warning | snapcraft | merged | Merged PR #5545 (cb96b3533), allowing users to ignore validation-set sequence-increment warnings and still submit assertions, fixing the overlooked scenario in #5535. | |
| #5535 feat(validation-sets): warn when the sequence isn't incremented | snapcraft | merged | PR #5535 by mr-cal merged (commit f012b667e): adds a SnapcraftAssertionWarning when a validation set's sequence isn't incremented, plus shared _validate_assertion infrastructure. Fixes #5367. |