Flaky spread tests
Metadata
Current evaluation
Flaky LXD spread-test timeouts were addressed by PR #4141 and commit 4b20c8435, which improved the resilience of the core18 lxd-type-base test.
Resolution: closed resolved
Reason: PR #4141, merged as 918ce0582, extended the core18 base-test runtime, and commit 4b20c8435 hardened lxd-type-base against intermittent LXD shutdown timeouts. The same failure was later tracked in #4236 and addressed by PR #4239.
Evaluated by: mr-cal-bot
No scores available.
Issue body
### Bug Description
Some of our spread tests are flaky. Example failure: https://github.com/snapcore/snapcraft/actions/runs/4961379589/jobs/8878193699?pr=4128
They seem to be sometimes timing out, but not reliably so
### To Reproduce
1. Create a PR with any change.
2. Run the spread tests
3. If they all succeed, try re-running
### Environment
CI
### snapcraft.yaml
```shell
N/A
```
### Relevant log output
```shell
Loading plugin module 'nil' with sys.path ['/root/project/snap/plugins', '/snap/snapcraft/x1/bin', '/snap/snapcraft/x1/usr/lib/python38.zip', '/snap/snapcraft/x1/usr/lib/python3.8', '/snap/snapcraft/x1/usr/lib/python3.8/lib-dynload', '/snap/snapcraft/x1/lib/python3.8/site-packages']
Setting up part 'base' with plugin 'nil' and properties {'plugin': 'nil', 'stage': [], 'prime': []}.
Requested build-packages: []
Requested build-packages already installed: []
snapcraft is running in a docker or podman (OCI) container
Preparing to pull base
Pulling base
snapcraft is running as a snap True, SNAP_NAME set to 'snapcraft'
+ snapcraftctl pull
2023-05-12 17:24:11.077 Starting Snapcraft 7.3.1.post80+gitbe1f2db6
2023-05-12 17:24:11.077 Logging execution to '/root/.local/state/snapcraft/log/snapcraft-20230512-172411.075965.log'
2023-05-12 17:24:11.079 lifecycle command: 'pull', arguments: Namespace(bind_ssh=False, build_for=None, debug=False, destructive_mode=False, enable_developer_debug=False, enable_experimental_extensions=False, enable_experimental_plugins=False, enable_experimental_target_arch=False, enable_experimental_ua_services=False, enable_manifest=True, http_proxy=None, https_proxy=None, manifest_image_information=None, parts=[], provider=None, shell=False, shell_after=False, target_arch=None, ua_token=None, use_lxd=True)
2023-05-12 17:24:11.079 command: pull, arguments: Namespace(bind_ssh=False, build_for=None, debug=False, destructive_mode=False, enable_developer_debug=False, enable_experimental_extensions=False, enable_experimental_plugins=False, enable_experimental_target_arch=False, enable_experimental_ua_services=False, enable_manifest=True, http_proxy=None, https_proxy=None, manifest_image_information=None, parts=[], provider=None, shell=False, shell_after=False, target_arch=None, ua_token=None, use_lxd=True)
+ /snap/bin/lxc start snapcraft-core18
+ MATCH VERSION_CODENAME=bionic
+ /snap/bin/lxc exec snapcraft-core18 cat /etc/os-release
+ /snap/bin/lxc stop snapcraft-core18
Error: Failed shutting down instance, status is "Running": context deadline exceeded
Try `lxc info --show-log snapcraft-core18` for more info
-----
.
2023-05-12 17:36:43 Restoring google:ubuntu-18.04-64:tests/spread/providers/lxd-type-base (may121721-738851)...
```
### Additional context
_No response_
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Flaky LXD spread-test timeouts were addressed by PR #4141 and commit 4b20c8435, which improved the resilience of the core18 lxd-type-base test. | |
| stealth/space-bunny-alpha | — | closed resolved | Flaky spread-test timeouts were fixed by merged PR #4141 (commit 918ce0582); commit 4b20c8435 further hardened lxd-type-base against intermittent LXD shutdown timeouts. | |
| stealth/space-bunny-alpha | — | closed resolved | Fixed by commit 4b20c8435, improving lxd-type-base resilience to intermittent LXD shutdown timeouts. | |
| stealth/space-bunny-alpha | — | closed resolved | Fixed by commit 4b20c8435, which improved the resilience of the lxd-type-base spread test and addressed intermittent LXD shutdown timeouts. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed without resolution. The flaky spread tests causing intermittent CI timeouts were never addressed, indicating the report was abandoned. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed without resolution. The flaky spread tests and LXD container timeout issues were not addressed, indicating the report was abandoned. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed without resolution. The flaky spread tests causing LXC container shutdown timeouts were abandoned due to inactivity, with no fix implemented. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Closed without resolution. Flaky spread tests timed out due to LXD container shutdown delays in CI. Zero comments and rapid closure indicate the issue was abandoned or resolved externally. |
Update history
No update history recorded yet.
Related work
-
Likely Fixed By:
snapcraft#4141
(confidence 98%)
Merged as 918ce0582 to let the core18 base test run longer.
-
Related To:
snapcraft#4236
(confidence 94%)
Follow-up issue documenting the same lxd-type-base LXC shutdown timeout.
-
Related To:
snapcraft#4239
(confidence 90%)
Later hardening of lxd-type-base, recorded as squash commit ab2c13576.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #4236 lxd-type-base spread test is flaky due to lxd timeouts | snapcraft | closed | Flaky lxd-type-base spread test (LXC stop 'context deadline exceeded') fixed by PR #4239, squash-merged as commit ab2c13576 'spread: improve resilience of lxd-type-base test'. | |
| #4239 spread: improve resilience of lxd-type-base test | snapcraft | merged | PR #4239 was squash-merged as ab2c13576, making the lxd-type-base spread test more resilient to frequent lxc stop timeouts on Google runners. | |
| #99 Broken spread tests on 20.04 and 22.04 | imagecraft | closed | Ubuntu 20.04/22.04 spread failures were fixed by merged PR #110 (commit 00f1348), correcting legacy LXD client handling in spread setup. | |
| #2983 tests: add LXD spread test | snapcraft | merged | Merged PR #2983 as 4b8522a14, adding spread coverage that builds the LXD package with LXD and destructive/host providers; installation checks were deferred. | |
| #5856 fix(spread): pack with lxd for core20 base tests | snapcraft | merged | Merged PR #5856 (commit 66b9ae9f4) updated Spread core20-base tests to pack with LXD. | |
| #467 test(spread): use LXD provider for spread tests | imagecraft | closed | PR closed without merge; spread LXD provider changes were implemented across follow-up PRs (#483, #485, #486, #487, #488, #495) | |
| #2631 spread tests: enable LXD build provider tests | snapcraft | merged | Merged PR #2631 enabled LXD build-provider spread tests and reused early-prepared LXD setup, reducing repeated setup and image-download overhead. | |
| #1133 test(spread): make LXD refresh in prepare more reliable | craft-application | closed | Closed unmerged; its LXD auto-refresh race fix was superseded by merged PR #1170 (commit fea3e835), which waits for active snap refreshes during spread prepare. | |
| #1407 Spread test failure: LXD forkstart failure in cross-compile test | rockcraft | open | Intermittent LXD forkstart failures (~52%) in cross-compile spread tests on OpenStack runners; related to known snapcraft#6381 | |
| #694 tests(spread): use LXD from `5.21/candidate` | rockcraft | merged | Merged PR #694 in commit 90bc9b34, pinning spread tests to LXD 5.21/candidate to restore main and strengthen LTS regression coverage. |