tests: spread tests fail intermittently
Metadata
Current evaluation
Spread tests flake when the docker snap daemon never starts. Still present in-tree: spread.yaml does 'snap install docker' then a bare 'retry -n 10 --wait 2 docker run --rm hello-world' with no failure diagnostics. Fix proposed (log capture, longer retry) not yet implemented.
Suggested action: keep open
Reason: Confirmed unfixed in-tree: spread.yaml:150/161 still runs 'snap install docker' followed by 'retry -n 10 --wait 2 sh -c "docker run --rm hello-world"' with no diagnostics on failure, and a repo-wide grep for 'snap logs docker' returns zero matches, proving the PR #96 mitigation was never merged. The maintainer triage comment explicitly states 'this is still present in-tree (not obsolete)' and the Jira sync (ROCKCRAFT-296) shows it is tracked work, so it must stay open. Impact is moderate (CI flake wasting runs, not user-facing), and the fix is a small, well-understood diagnostics/retry tweak; actionability is slightly reduced because the original log is from the retired Google backend and ubuntu-18.04, so the exact original trigger cannot be reproduced against today's openstack config.
Evaluated by: mr-cal-bot
Impact:
42
Quick Win:
22
Complexity:
22
Confidence:
82
Actionability:
68
Issue body
### Overview
The spread tests are failing intermittently due to a docker error, `docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?`.
### Details
@tigarmo added the [`retry`](https://github.com/canonical/rockcraft/pull/94/files#diff-e1e5ecb5e7fe7aa228c64f680564673541f91452212c3c6c23cbbd27bb78dcedR47) command from [`snapd-testing-tools`](https://github.com/snapcore/snapd-testing-tools/blob/main/tools/retry), but we're still seeing the failure.
We have two more ideas:
1. Increase the `retry` timeout
2. [Run `snap logs docker`](https://github.com/canonical/rockcraft/pull/96/files) if docker never starts
### Logs
[source](https://github.com/canonical/rockcraft/actions/runs/3152544084/jobs/5128003514)
```
2022-09-29T15:30:03.8353470Z 2022-09-29 15:30:03 Preparing google:ubuntu-20.04-64 (sep291529-710396)...
2022-09-29T15:30:50.7936227Z 2022-09-29 15:30:50 Error preparing google:ubuntu-18.04-64 (sep291529-710434) :
2022-09-29T15:30:50.7937212Z -----
...
2022-09-29T15:30:50.8202707Z + snap install docker
2022-09-29T15:30:50.8203825Z docker 20.10.14 from Canonical** installed
2022-09-29T15:30:50.8204725Z + retry -n 10 --wait 2 sh -c 'docker run --rm hello-world'
2022-09-29T15:30:50.8205417Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8206155Z See 'docker run --help'.
2022-09-29T15:30:50.8206755Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8207558Z See 'docker run --help'.
2022-09-29T15:30:50.8208252Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8208953Z See 'docker run --help'.
2022-09-29T15:30:50.8209589Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8210394Z See 'docker run --help'.
2022-09-29T15:30:50.8211104Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8211805Z See 'docker run --help'.
2022-09-29T15:30:50.8212509Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8213310Z See 'docker run --help'.
2022-09-29T15:30:50.8213902Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8214654Z See 'docker run --help'.
2022-09-29T15:30:50.8215310Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8215947Z See 'docker run --help'.
2022-09-29T15:30:50.8216654Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8217364Z See 'docker run --help'.
2022-09-29T15:30:50.8217867Z docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
2022-09-29T15:30:50.8218561Z See 'docker run --help'.
2022-09-29T15:30:50.8219376Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8220001Z retry: next attempt in 2.0 second(s) (attempt 1 of 10)
2022-09-29T15:30:50.8220713Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8221349Z retry: next attempt in 2.0 second(s) (attempt 2 of 10)
2022-09-29T15:30:50.8222215Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8222833Z retry: next attempt in 2.0 second(s) (attempt 3 of 10)
2022-09-29T15:30:50.8223556Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8224459Z retry: next attempt in 2.0 second(s) (attempt 4 of 10)
2022-09-29T15:30:50.8225376Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8225965Z retry: next attempt in 2.0 second(s) (attempt 5 of 10)
2022-09-29T15:30:50.8227152Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8227788Z retry: next attempt in 2.0 second(s) (attempt 6 of 10)
2022-09-29T15:30:50.8228467Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8228917Z retry: next attempt in 2.0 second(s) (attempt 7 of 10)
2022-09-29T15:30:50.8229547Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8230011Z retry: next attempt in 2.0 second(s) (attempt 8 of 10)
2022-09-29T15:30:50.8230616Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8231159Z retry: next attempt in 2.0 second(s) (attempt 9 of 10)
2022-09-29T15:30:50.8231896Z retry: command sh -c docker run --rm hello-world failed with code 125
2022-09-29T15:30:50.8232682Z retry: command sh -c docker run --rm hello-world keeps failing after 10 attempts
2022-09-29T15:30:50.8233455Z -----
2022-09-29T15:30:50.8233874Z .
2022-09-29T15:30:50.8234466Z 2022-09-29 15:30:50 Discarding google:ubuntu-18.04-64 (sep291529-710434)...
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha |
Impact:
42
Quick Win:
22
Complexity:
22
Confidence:
82
Actionability:
68
|
keep open | Spread tests flake when the docker snap daemon never starts. Still present in-tree: spread.yaml does 'snap install docker' then a bare 'retry -n 10 --wait 2 docker run --rm hello-world' with no failure diagnostics. Fix proposed (log capture, longer retry) not yet implemented. | |
| qwen/qwen3.6-35b-a3b |
Impact:
30
Quick Win:
13
Complexity:
40
Confidence:
75
Actionability:
70
|
keep open | Intermittent spread test failures due to Docker daemon not starting after installation. Triaged, still present in current spread.yaml with unchanged retry logic. | |
| qwen/qwen3.8-27b |
Impact:
35
Quick Win:
21
Staleness:
50
Complexity:
40
Confidence:
85
Support Request:
5
|
keep open | Rockcraft spread tests intermittently fail because the docker snap daemon never starts in test VMs, and the retry loop is insufficient. Feb 2026 triage confirmed the same pattern is still in spread.yaml; a fix (diagnostics on failure) is suggested but no PR proposed. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
55
Complexity:
30
Confidence:
85
Support Request:
10
|
keep open | Intermittent spread test failures due to Docker daemon startup timeout. Triaged bug with recent maintainer suggestions for improved diagnostics, awaiting implementation. | |
| qwen3.6-35b-a3b-mtp-q6 |
Staleness:
55
Complexity:
40
Confidence:
75
Support Request:
10
|
keep open | Intermittent spread test failures due to Docker daemon not starting during CI setup. Triaged and open, with recent suggestions to add diagnostic logging and increase retry timeouts. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #534 test(spread): workaround for flaky test | fetch-service | merged | Merged PR #534 (commit 4f89bb7) worked around #533 by skipping the flaky snap names-query assertion while retaining the other spread-test checks. |