Openstack spread tests randomly fail with `Failed to run: /snap/lxd/current/sbin/lxd forkstart`
Metadata
Current evaluation
Intermittent LXD 'forkstart' failures (<0.5%) in the OpenStack spread environment across craft repos; reproduced under heavy host memory/CPU pressure and reported upstream to canonical/lxd#18857, no snapcraft-side fix yet
Suggested action: keep open
Reason: The report is well-evidenced: it includes LXD container logs showing 'Connection reset by peer - Sync wait failure' / 'Failed to spawn container' plus full Snapcraft tracebacks, and @lengau confirmed reproduction with 'the exact same error class' under combined disk I/O + CPU + ~85-90% memory pressure (disk I/O alone was not sufficient). Maintainer mr-cal already triaged it by filing upstream in canonical/lxd#18857, so it is a valid, acknowledged infrastructure bug that is blocked on an LXD-side fix rather than a snapcraft code change. Impact is capped because failure rate is under 0.5% and it only affects CI, but complexity/actionability remain low-mid since the fix lives in LXD or the shared OpenStack infrastructure, reproduction requires unrealistic 30+ VM / 50-minute stress rigs, and no local mitigation (retry logic, instance sizing) has been proposed in the thread.
Evaluated by: mr-cal-bot
Impact:
32
Quick Win:
4
Complexity:
72
Confidence:
70
Actionability:
48
Issue body
All craft repos that use the openstack backend for spread and launch LXD containers randomly get the following error:
`Error: Failed to run: /snap/lxd/current/sbin/lxd forkstart snapcraft_snapcraft-test-multi-keys-amd64-1575506 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/snapcraft_snapcraft-test-multi-keys-amd64-1575506/lxc.conf: exit status 1`
It happens less than 0.5% of the time in the `openstack` environment. It fails significantly less often in `openstack-dev` environment.
My theory is that this is due to some sort of resource contention on the shared Openstack infrastructure. https://github.com/canonical/lxd/issues/11312 is a similar issue on a server running 100+ containers. Their error message is slightly different, so I don't think the same fix will apply.
### LXD logs
([source](https://github.com/canonical/snapcraft/actions/runs/31530642372/job/94107635498?pr=6380#step:5:352))
```bash
> lxc --project snapcraft info --show-log snapcraft-test-multi-keys-amd64-1575506
Name: snapcraft-test-multi-keys-amd64-1575506
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2026/08/12 12:19 UTC
Last Used: 2026/08/12 12:19 UTC
Log:
[74](https://github.com/canonical/snapcraft/actions/runs/31530642372/job/94107635498?pr=6380#step:5:375)
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.129 ERROR sync - ../src/src/lxc/sync.c:sync_wait:25 - Connection reset by peer - Sync wait failure
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.130 ERROR start - ../src/src/lxc/start.c:__lxc_start:2118 - Failed to spawn container "snapcraft_snapcraft-test-multi-keys-amd64-1575506"
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.130 ERROR lxccontainer - ../src/src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING"
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.130 WARN start - ../src/src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 17 for process 27696
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.228 WARN cgfsng - ../src/src/lxc/cgroups/cgfsng.c:cgfsng_monitor_destroy:925 - Device or resource busy - Failed to destroy 10(lxc.monitor.snapcraft_snapcraft-test-multi-keys-amd64-1575506)
lxc 20260812121940.228 ERROR af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.228 ERROR af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20260812121940.228 ERROR commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.228 ERROR commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "add_state_client"
```
### Snapcraft logs
([source](https://github.com/canonical/snapcraft/actions/runs/31530642372/job/94107635498?pr=6380#step:5:269))
```
2026-08-12 12:19:39.587 Executing on host: lxc --project snapcraft info local:snapcraft-test-multi-keys-amd64-1575506
2026-08-12 12:19:39.704 Executing on host: lxc --project snapcraft start local:snapcraft-test-multi-keys-amd64-1575506
2026-08-12 12:19:40.604 Failed to start 'snapcraft-test-multi-keys-amd64-1575506'.
2026-08-12 12:19:40.604 Detailed information:
* Command that failed: 'lxc --project snapcraft start local:snapcraft-test-multi-keys-amd64-1575506'
* Command exit code: 1
* Command standard error output: b'Error: Failed to run: /snap/lxd/current/sbin/lxd forkstart snapcraft_snapcraft-test-multi-keys-amd64-1575506 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/snapcraft_snapcraft-test-multi-keys-amd64-1575506/lxc.conf: exit status 1\nTry `lxc info --show-log local:snapcraft-test-multi-keys-amd64-1575506` for more info\n'
2026-08-12 12:19:40.613 Traceback (most recent call last):
2026-08-12 12:19:40.613 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxc.py", line 1194, in start
2026-08-12 12:19:40.613 self._run_lxc(command, capture_output=True, project=project)
2026-08-12 12:19:40.613 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxc.py", line 213, in _run_lxc
2026-08-12 12:19:40.613 return subprocess.run(
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/current/usr/lib/python3.12/subprocess.py", line 571, in run
2026-08-12 12:19:40.614 raise CalledProcessError(retcode, process.args,
2026-08-12 12:19:40.614 subprocess.CalledProcessError: Command '['lxc', '--project', 'snapcraft', 'start', 'local:snapcraft-test-multi-keys-amd64-1575506']' returned non-zero exit status 1.
2026-08-12 12:19:40.614
2026-08-12 12:19:40.614 The above exception was the direct cause of the following exception:
2026-08-12 12:19:40.614 Traceback (most recent call last):
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/application.py", line 565, in run
2026-08-12 12:19:40.614 return_code = self._run_inner()
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/snapcraft/application.py", line 180, in _run_inner
2026-08-12 12:19:40.614 return_code = super()._run_inner()
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/application.py", line 550, in _run_inner
2026-08-12 12:19:40.614 dispatcher_return = dispatcher.run()
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_cli/dispatcher.py", line 576, in run
2026-08-12 12:19:40.614 return self._loaded_command.run(self._parsed_command_args)
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/commands/base.py", line 180, in run
2026-08-12 12:19:40.614 result = self._run(parsed_args, **kwargs) or result
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/snapcraft/commands/lifecycle.py", line 211, in _run
2026-08-12 12:19:40.614 super()._run(parsed_args)
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/commands/lifecycle.py", line 636, in _run
2026-08-12 12:19:40.614 return super()._run(parsed_args=parsed_args, step_name=step_name)
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/commands/lifecycle.py", line 268, in _run
2026-08-12 12:19:40.614 self._run_manager_for_build_plan(fetch_service_policy)
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/commands/lifecycle.py", line 117, in _run_manager_for_build_plan
2026-08-12 12:19:40.614 provider.run_managed(build, bool(fetch_service_policy))
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/services/provider.py", line 586, in run_managed
2026-08-12 12:19:40.614 with self.instance(
2026-08-12 12:19:40.614 File "/snap/snapcraft/current/usr/lib/python3.12/contextlib.py", line 137, in __enter__
2026-08-12 12:19:40.614 return next(self.gen)
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/services/provider.py", line 275, in instance
2026-08-12 12:19:40.614 with provider.launched_environment(
2026-08-12 12:19:40.614 File "/snap/snapcraft/current/usr/lib/python3.12/contextlib.py", line 137, in __enter__
2026-08-12 12:19:40.614 return next(self.gen)
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxd_provider.py", line 221, in launched_environment
2026-08-12 12:19:40.614 instance = launch(
2026-08-12 12:19:40.614 ^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/launcher.py", line 917, in launch
2026-08-12 12:19:40.614 instance.start()
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxd_instance.py", line 582, in start
2026-08-12 12:19:40.614 self.lxc.start(
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxc.py", line 1196, in start
2026-08-12 12:19:40.614 raise LXDError(
2026-08-12 12:19:40.614 craft_providers.lxd.errors.LXDError: Failed to start 'snapcraft-test-multi-keys-amd64-1575506'.
2026-08-12 12:19:40.614 * Command that failed: 'lxc --project snapcraft start local:snapcraft-test-multi-keys-amd64-1575506'
2026-08-12 12:19:40.614 * Command exit code: 1
2026-08-12 12:19:40.614 * Command standard error output: b'Error: Failed to run: /snap/lxd/current/sbin/lxd forkstart snapcraft_snapcraft-test-multi-keys-amd64-1575506 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/snapcraft_snapcraft-test-multi-keys-amd64-1575506/lxc.conf: exit status 1\nTry `lxc info --show-log local:snapcraft-test-multi-keys-amd64-1575506` for more info\n'
2026-08-12 12:19:40.614 Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20260812-121933.664475.log'
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha |
Impact:
32
Quick Win:
4
Complexity:
72
Confidence:
70
Actionability:
48
|
keep open | Intermittent LXD 'forkstart' failures (<0.5%) in the OpenStack spread environment across craft repos; reproduced under heavy host memory/CPU pressure and reported upstream to canonical/lxd#18857, no snapcraft-side fix yet | |
| stealth/space-bunny-alpha |
Impact:
22
Quick Win:
4
Complexity:
58
Confidence:
72
Actionability:
46
|
keep open | OpenStack-environment spread tests randomly fail (<0.5%) with LXD forkstart errors across all craft repos. Traced to host resource contention, reproduced under stress, and reported upstream to LXD; awaiting upstream resolution. | |
| stealth/space-bunny-alpha |
Impact:
38
Quick Win:
5
Complexity:
72
Confidence:
72
Actionability:
48
|
keep open | Flaky 'lxd forkstart' failures (<0.5% of runs) starting LXD containers in all craft repos' OpenStack spread tests, traced to host resource pressure; reproduced under stress and reported upstream in canonical/lxd#18857. | |
| stealth/space-bunny-alpha |
Impact:
55
Quick Win:
9
Complexity:
75
Confidence:
90
Actionability:
66
|
keep open | LXD forkstart intermittently fails under host resource pressure, disrupting all craft repos' OpenStack spread tests (<0.5%); reproduced with stressed multipass and tracked upstream in canonical/lxd#18857. | |
| qwen3.6-35b-a3b-mtp-q6 |
Impact:
45
Quick Win:
16
Staleness:
35
Complexity:
65
Confidence:
75
Support Request:
15
|
keep open | Intermittent LXD forkstart failures on shared Openstack infrastructure affecting all craft repos' spread tests (<0.5% failure rate); being tracked upstream at LXD #18857. | |
| qwen/qwen3.8-27b |
Impact:
40
Quick Win:
8
Staleness:
5
Complexity:
80
Confidence:
85
Support Request:
0
|
keep open | Flaky LXD forkstart failures in Openstack spread tests across craft repos (<0.5% of runs). Reproduced under combined disk I/O, CPU, and memory pressure. Reported upstream to LXD (#18857). Actively investigated by maintainers. | |
| deepseek/deepseek-v4-pro-0813 |
Impact:
40
Quick Win:
16
Staleness:
10
Complexity:
60
Confidence:
75
Support Request:
10
|
keep open | Intermittent LXD 'forkstart' failures on OpenStack spread tests in all craft repos; reproduced under combined disk/CPU/memory contention and reported upstream to canonical/lxd#18857. | |
| qwen/qwen3.8-27b |
Impact:
45
Quick Win:
11
Staleness:
10
Complexity:
75
Confidence:
80
Support Request:
5
|
keep open | Intermittent (<0.5%) LXD `forkstart` failures when launching spread test containers on the shared Openstack backend, suspected resource contention. Reproduced locally under combined disk I/O + CPU + memory pressure; reported upstream to LXD (canonical/lxd#18857). | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
Support Request:
5
|
keep open | Intermittent LXD container startup failures under resource contention on OpenStack/Multipass backends. Currently under investigation with a reproducer found; reported upstream to LXD. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
0
Complexity:
65
Confidence:
95
Support Request:
5
|
needs triage | Intermittent LXD container start failures in OpenStack spread tests caused by sync errors under resource contention. Currently unlabelled with active reproduction efforts. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
Support Request:
15
|
needs triage | Intermittent LXD container startup failures during Openstack spread tests, likely due to resource contention. Recently replicated under high disk I/O; awaiting maintainer triage. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
65
Confidence:
90
Support Request:
15
|
needs triage | Random LXD container startup failures in OpenStack spread tests due to forkstart errors. Reported by maintainer with detailed logs; currently unlabelled and awaiting triage. |
Update history
| Date | Change |
|---|---|
| updated | |
| updated | |
| updated | |
| created |
Related work
-
Blocked By:
lxd#18857
(confidence 90%)
mr-cal explicitly reported upstream: 'Reported upstream in https://github.com/canonical/lxd/issues/18857' — the real fix is in LXD.
-
Related To:
lxd#11312
(confidence 75%)
Cited in the body as a similar failure on a host running 100+ containers, though the author notes 'Their error message is slightly different, so I don't think the same fix will apply.'
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #6380 test: debug lxd forkstart failure | snapcraft | closed | Debug-only PR by maintainer mr-cal to diagnose random LXD forkstart failures in OpenStack spread tests was closed unmerged after ~24 days with no reviews, 5 unresolved comments and failing lint/CLA checks. | |
| #2515 `google:ubuntu-20.04-64:tests/spread/smoketests/reactive:stable` spread test is failing | charmcraft | closed | Flaky LXD error 'Failed to wait for snap refreshes to complete' (snapd race SNAPDENG-36387) traced to craft-providers; fixed by retrying `snap watch` in craft-providers PR #899, pulled in by charmcraft PR #2583 and released in charmcraft 4.2.0. | |
| #4236 lxd-type-base spread test is flaky due to lxd timeouts | snapcraft | closed | Flaky lxd-type-base spread test (LXC stop 'context deadline exceeded') fixed by PR #4239, squash-merged as commit ab2c13576 'spread: improve resilience of lxd-type-base test'. | |
| #4140 Flaky spread tests | snapcraft | closed | Flaky LXD spread-test timeouts were addressed by PR #4141 and commit 4b20c8435, which improved the resilience of the core18 lxd-type-base test. |