← Back to issue list

Flaky spread tests

View original Github issue

Metadata

Project
snapcraft
Number
#4140
Type
issue
State
closed
Author
lengau
Labels
—
Created
Updated
Closed

Current evaluation

Flaky LXD spread-test timeouts were addressed by PR #4141 and commit 4b20c8435, which improved the resilience of the core18 lxd-type-base test.

Resolution: closed resolved

Reason: PR #4141, merged as 918ce0582, extended the core18 base-test runtime, and commit 4b20c8435 hardened lxd-type-base against intermittent LXD shutdown timeouts. The same failure was later tracked in #4236 and addressed by PR #4239.

Evaluated by: mr-cal-bot

No scores available.

Issue body

### Bug Description Some of our spread tests are flaky. Example failure: https://github.com/snapcore/snapcraft/actions/runs/4961379589/jobs/8878193699?pr=4128 They seem to be sometimes timing out, but not reliably so ### To Reproduce 1. Create a PR with any change. 2. Run the spread tests 3. If they all succeed, try re-running ### Environment CI ### snapcraft.yaml ```shell N/A ``` ### Relevant log output ```shell Loading plugin module 'nil' with sys.path ['/root/project/snap/plugins', '/snap/snapcraft/x1/bin', '/snap/snapcraft/x1/usr/lib/python38.zip', '/snap/snapcraft/x1/usr/lib/python3.8', '/snap/snapcraft/x1/usr/lib/python3.8/lib-dynload', '/snap/snapcraft/x1/lib/python3.8/site-packages'] Setting up part 'base' with plugin 'nil' and properties {'plugin': 'nil', 'stage': [], 'prime': []}. Requested build-packages: [] Requested build-packages already installed: [] snapcraft is running in a docker or podman (OCI) container Preparing to pull base Pulling base snapcraft is running as a snap True, SNAP_NAME set to 'snapcraft' + snapcraftctl pull 2023-05-12 17:24:11.077 Starting Snapcraft 7.3.1.post80+gitbe1f2db6 2023-05-12 17:24:11.077 Logging execution to '/root/.local/state/snapcraft/log/snapcraft-20230512-172411.075965.log' 2023-05-12 17:24:11.079 lifecycle command: 'pull', arguments: Namespace(bind_ssh=False, build_for=None, debug=False, destructive_mode=False, enable_developer_debug=False, enable_experimental_extensions=False, enable_experimental_plugins=False, enable_experimental_target_arch=False, enable_experimental_ua_services=False, enable_manifest=True, http_proxy=None, https_proxy=None, manifest_image_information=None, parts=[], provider=None, shell=False, shell_after=False, target_arch=None, ua_token=None, use_lxd=True) 2023-05-12 17:24:11.079 command: pull, arguments: Namespace(bind_ssh=False, build_for=None, debug=False, destructive_mode=False, enable_developer_debug=False, enable_experimental_extensions=False, enable_experimental_plugins=False, enable_experimental_target_arch=False, enable_experimental_ua_services=False, enable_manifest=True, http_proxy=None, https_proxy=None, manifest_image_information=None, parts=[], provider=None, shell=False, shell_after=False, target_arch=None, ua_token=None, use_lxd=True) + /snap/bin/lxc start snapcraft-core18 + MATCH VERSION_CODENAME=bionic + /snap/bin/lxc exec snapcraft-core18 cat /etc/os-release + /snap/bin/lxc stop snapcraft-core18 Error: Failed shutting down instance, status is "Running": context deadline exceeded Try `lxc info --show-log snapcraft-core18` for more info ----- . 2023-05-12 17:36:43 Restoring google:ubuntu-18.04-64:tests/spread/providers/lxd-type-base (may121721-738851)... ``` ### Additional context _No response_

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha — closed resolved Flaky LXD spread-test timeouts were addressed by PR #4141 and commit 4b20c8435, which improved the resilience of the core18 lxd-type-base test.
stealth/space-bunny-alpha — closed resolved Flaky spread-test timeouts were fixed by merged PR #4141 (commit 918ce0582); commit 4b20c8435 further hardened lxd-type-base against intermittent LXD shutdown timeouts.
stealth/space-bunny-alpha — closed resolved Fixed by commit 4b20c8435, improving lxd-type-base resilience to intermittent LXD shutdown timeouts.
stealth/space-bunny-alpha — closed resolved Fixed by commit 4b20c8435, which improved the resilience of the lxd-type-base spread test and addressed intermittent LXD shutdown timeouts.
qwen/qwen3.6-35b-a3b — — Closed without resolution. The flaky spread tests causing intermittent CI timeouts were never addressed, indicating the report was abandoned.
qwen/qwen3.6-35b-a3b — — Closed without resolution. The flaky spread tests and LXD container timeout issues were not addressed, indicating the report was abandoned.
qwen/qwen3.6-35b-a3b — — Closed without resolution. The flaky spread tests causing LXC container shutdown timeouts were abandoned due to inactivity, with no fix implemented.
qwen3.6-35b-a3b-mtp-q6 — — Closed without resolution. Flaky spread tests timed out due to LXD container shutdown delays in CI. Zero comments and rapid closure indicate the issue was abandoned or resolved externally.

Update history

No update history recorded yet.

Related work

  • Likely Fixed By: snapcraft#4141 (confidence 98%)

    Merged as 918ce0582 to let the core18 base test run longer.

  • Related To: snapcraft#4236 (confidence 94%)

    Follow-up issue documenting the same lxd-type-base LXC shutdown timeout.

  • Related To: snapcraft#4239 (confidence 90%)

    Later hardening of lxd-type-base, recorded as squash commit ab2c13576.

Related issues

Issue Project State Summary Similarity
#4236 lxd-type-base spread test is flaky due to lxd timeouts snapcraft closed Flaky lxd-type-base spread test (LXC stop 'context deadline exceeded') fixed by PR #4239, squash-merged as commit ab2c13576 'spread: improve resilience of lxd-type-base test'.
87%
#4239 spread: improve resilience of lxd-type-base test snapcraft merged PR #4239 was squash-merged as ab2c13576, making the lxd-type-base spread test more resilient to frequent lxc stop timeouts on Google runners.
79%
#99 Broken spread tests on 20.04 and 22.04 imagecraft closed Ubuntu 20.04/22.04 spread failures were fixed by merged PR #110 (commit 00f1348), correcting legacy LXD client handling in spread setup.
78%
#2983 tests: add LXD spread test snapcraft merged Merged PR #2983 as 4b8522a14, adding spread coverage that builds the LXD package with LXD and destructive/host providers; installation checks were deferred.
77%
#5856 fix(spread): pack with lxd for core20 base tests snapcraft merged Merged PR #5856 (commit 66b9ae9f4) updated Spread core20-base tests to pack with LXD.
76%
#467 test(spread): use LXD provider for spread tests imagecraft closed PR closed without merge; spread LXD provider changes were implemented across follow-up PRs (#483, #485, #486, #487, #488, #495)
75%
#2631 spread tests: enable LXD build provider tests snapcraft merged Merged PR #2631 enabled LXD build-provider spread tests and reused early-prepared LXD setup, reducing repeated setup and image-download overhead.
75%
#1133 test(spread): make LXD refresh in prepare more reliable craft-application closed Closed unmerged; its LXD auto-refresh race fix was superseded by merged PR #1170 (commit fea3e835), which waits for active snap refreshes during spread prepare.
74%
#1407 Spread test failure: LXD forkstart failure in cross-compile test rockcraft open Intermittent LXD forkstart failures (~52%) in cross-compile spread tests on OpenStack runners; related to known snapcraft#6381
73%
#694 tests(spread): use LXD from `5.21/candidate` rockcraft merged Merged PR #694 in commit 90bc9b34, pinning spread tests to LXD 5.21/candidate to restore main and strengthen LTS regression coverage.
73%