tests: update the store failure tests
Metadata
Current evaluation
Merged in #1538 via commit 0a9284cee, updating store-failure tests for autopkgtest reporting, exit-code-2 exceptions, and fake-store execution.
Resolution: closed resolved
Reason: PR #1538 was approved and merged as commit 0a9284cee, updating the affected store-failure integration tests.
Evaluated by: mr-cal-bot
No scores available.
Issue body
- [x] Have you followed the [guidelines for contributing](https://github.com/snapcore/snapcraft/blob/master/CONTRIBUTING.md)?
- [x] Have you signed the [CLA](http://www.ubuntu.com/legal/contributors/)?
- [x] If this is a bugfix. Have you checked that there is a bug report open for the issue you are trying to fix on [bug reports](https://bugs.launchpad.net/snapcraft)?
- [x] If this is a new feature. Have you discussed the design on the [forum](https://forum.snapcraft.io)?
- [x] Have you successfully run `./runtests.sh static`?
- [x] Have you successfully run `./runtests.sh unit`?
-----
There is a good bunch of things going on here.
First, the autopkgtests scripts are run without -e, so only the last error is reported.
Then, through a set of changes, when there is an exception that causes snapcraft to exit the return code is 2 ( https://github.com/snapcore/snapcraft/blob/67089e05002c5fcbfed0d64ac1b3f69363102e3a/snapcraft/internal/errors.py#L35 ).
Finally, this only happens when the integration tests are run against the fake servers. Otherwise, the tests are skipped. Now this is confusing, because the pull requests from forks should run against the fakes, and should have caught this. However, the tests are being skipped, which means that TEST_STORE env var is not set to `fake`. Probably, we are not passing the right var to the container, but I will take care of that tomorrow.
I'm sorry about this list of wrongs.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Merged in #1538 via commit 0a9284cee, updating store-failure tests for autopkgtest reporting, exit-code-2 exceptions, and fake-store execution. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged PR #1538 as commit 0a9284cee, updating store failure tests for autopkgtest error reporting, exception exit code 2, and fake-server runs. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged as #1538 in commit 0a9284cee, updating store failure tests for autopkgtest error reporting, exit code 2 on exceptions, and fake-server integration tests. | |
| qwen/qwen3.6-35b-a3b | — | — | Merged after approval and passing CI. Updates store failure tests by adjusting autopkgtest scripts to report final errors, sets exception exit codes to 2, and clarifies integration test execution against fake servers. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Merged test updates adjusting exception return codes and clarifying fake server execution. Resolves environment variable configuration for automated test runs. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1984 tests: update store tests user | snapcraft | merged | Merged PR #1984 (3c313dab1), updating the Store test account, removing two obsolete tests, and revising related test data, documentation, and warnings. | |
| #452 Fix store downloading | snapcraft | merged | Store downloading was fixed and merged in PR #452 via commit f5a1bf971 after a successful autopkgtest run and retest. | |
| #5954 tests: fix store tests | snapcraft | merged | PR #5954 merged as commit 0be1e384d: store spread tests switched to new staging credentials/key test-staging-key-2 and to wheel-based automated tests (as in #5885). | |
| #5370 test: fix store tests | snapcraft | closed | Store-test fixes were superseded by merged #5954, updating credentials and configuration after secret re-provisioning in commit 0be1e384d. | |
| #1460 fix(tests): make store spread tests less flaky | charmcraft | merged | Merged PR #1460 as fd5bd6ba, making store spread tests less timing-sensitive and fixing flaky upload/release checks in #1429. | |
| #1541 tests: fix the TEST_STORE environment variable | snapcraft | merged | Merged in PR #1541 as commit a36d3e053, fixing TEST_STORE handling so tests no longer skip before the empty value is translated to the fake store. |