Openstack spread tests randomly fail with `Failed to run: /snap/lxd/current/sbin/lxd forkstart`
Metadata
Current evaluation
Intermittent LXD forkstart failures on shared Openstack infrastructure affecting all craft repos' spread tests (<0.5% failure rate); being tracked upstream at LXD #18857.
Suggested action: keep open
Reason: The issue is well-documented with reproduction steps and LXD logs, has active maintainer engagement (mr-cal opened, lengau replicated with multipass under disk I/O pressure), and is being tracked upstream at canonical/lxd#18857. The intermittent nature (<0.5% failure rate) and infrastructure-specific cause (resource contention on shared Openstack) require ongoing investigation.
Impact:
45
Quick Win:
15.75
Staleness:
35
Complexity:
65
Confidence:
75
Support Request:
15
Issue body
All craft repos that use the openstack backend for spread and launch LXD containers randomly get the following error:
`Error: Failed to run: /snap/lxd/current/sbin/lxd forkstart snapcraft_snapcraft-test-multi-keys-amd64-1575506 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/snapcraft_snapcraft-test-multi-keys-amd64-1575506/lxc.conf: exit status 1`
It happens less than 0.5% of the time in the `openstack` environment. It fails significantly less often in `openstack-dev` environment.
My theory is that this is due to some sort of resource contention on the shared Openstack infrastructure. https://github.com/canonical/lxd/issues/11312 is a similar issue on a server running 100+ containers. Their error message is slightly different, so I don't think the same fix will apply.
### LXD logs
([source](https://github.com/canonical/snapcraft/actions/runs/31530642372/job/94107635498?pr=6380#step:5:352))
```bash
> lxc --project snapcraft info --show-log snapcraft-test-multi-keys-amd64-1575506
Name: snapcraft-test-multi-keys-amd64-1575506
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2026/08/12 12:19 UTC
Last Used: 2026/08/12 12:19 UTC
Log:
[74](https://github.com/canonical/snapcraft/actions/runs/31530642372/job/94107635498?pr=6380#step:5:375)
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.129 ERROR sync - ../src/src/lxc/sync.c:sync_wait:25 - Connection reset by peer - Sync wait failure
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.130 ERROR start - ../src/src/lxc/start.c:__lxc_start:2118 - Failed to spawn container "snapcraft_snapcraft-test-multi-keys-amd64-1575506"
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.130 ERROR lxccontainer - ../src/src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING"
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.130 WARN start - ../src/src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 17 for process 27696
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.228 WARN cgfsng - ../src/src/lxc/cgroups/cgfsng.c:cgfsng_monitor_destroy:925 - Device or resource busy - Failed to destroy 10(lxc.monitor.snapcraft_snapcraft-test-multi-keys-amd64-1575506)
lxc 20260812121940.228 ERROR af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.228 ERROR af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20260812121940.228 ERROR commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"
lxc snapcraft_snapcraft-test-multi-keys-amd64-1575506 20260812121940.228 ERROR commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "add_state_client"
```
### Snapcraft logs
([source](https://github.com/canonical/snapcraft/actions/runs/31530642372/job/94107635498?pr=6380#step:5:269))
```
2026-08-12 12:19:39.587 Executing on host: lxc --project snapcraft info local:snapcraft-test-multi-keys-amd64-1575506
2026-08-12 12:19:39.704 Executing on host: lxc --project snapcraft start local:snapcraft-test-multi-keys-amd64-1575506
2026-08-12 12:19:40.604 Failed to start 'snapcraft-test-multi-keys-amd64-1575506'.
2026-08-12 12:19:40.604 Detailed information:
* Command that failed: 'lxc --project snapcraft start local:snapcraft-test-multi-keys-amd64-1575506'
* Command exit code: 1
* Command standard error output: b'Error: Failed to run: /snap/lxd/current/sbin/lxd forkstart snapcraft_snapcraft-test-multi-keys-amd64-1575506 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/snapcraft_snapcraft-test-multi-keys-amd64-1575506/lxc.conf: exit status 1\nTry `lxc info --show-log local:snapcraft-test-multi-keys-amd64-1575506` for more info\n'
2026-08-12 12:19:40.613 Traceback (most recent call last):
2026-08-12 12:19:40.613 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxc.py", line 1194, in start
2026-08-12 12:19:40.613 self._run_lxc(command, capture_output=True, project=project)
2026-08-12 12:19:40.613 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxc.py", line 213, in _run_lxc
2026-08-12 12:19:40.613 return subprocess.run(
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/current/usr/lib/python3.12/subprocess.py", line 571, in run
2026-08-12 12:19:40.614 raise CalledProcessError(retcode, process.args,
2026-08-12 12:19:40.614 subprocess.CalledProcessError: Command '['lxc', '--project', 'snapcraft', 'start', 'local:snapcraft-test-multi-keys-amd64-1575506']' returned non-zero exit status 1.
2026-08-12 12:19:40.614
2026-08-12 12:19:40.614 The above exception was the direct cause of the following exception:
2026-08-12 12:19:40.614 Traceback (most recent call last):
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/application.py", line 565, in run
2026-08-12 12:19:40.614 return_code = self._run_inner()
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/snapcraft/application.py", line 180, in _run_inner
2026-08-12 12:19:40.614 return_code = super()._run_inner()
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/application.py", line 550, in _run_inner
2026-08-12 12:19:40.614 dispatcher_return = dispatcher.run()
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_cli/dispatcher.py", line 576, in run
2026-08-12 12:19:40.614 return self._loaded_command.run(self._parsed_command_args)
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/commands/base.py", line 180, in run
2026-08-12 12:19:40.614 result = self._run(parsed_args, **kwargs) or result
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/snapcraft/commands/lifecycle.py", line 211, in _run
2026-08-12 12:19:40.614 super()._run(parsed_args)
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/commands/lifecycle.py", line 636, in _run
2026-08-12 12:19:40.614 return super()._run(parsed_args=parsed_args, step_name=step_name)
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/commands/lifecycle.py", line 268, in _run
2026-08-12 12:19:40.614 self._run_manager_for_build_plan(fetch_service_policy)
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/commands/lifecycle.py", line 117, in _run_manager_for_build_plan
2026-08-12 12:19:40.614 provider.run_managed(build, bool(fetch_service_policy))
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/services/provider.py", line 586, in run_managed
2026-08-12 12:19:40.614 with self.instance(
2026-08-12 12:19:40.614 File "/snap/snapcraft/current/usr/lib/python3.12/contextlib.py", line 137, in __enter__
2026-08-12 12:19:40.614 return next(self.gen)
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_application/services/provider.py", line 275, in instance
2026-08-12 12:19:40.614 with provider.launched_environment(
2026-08-12 12:19:40.614 File "/snap/snapcraft/current/usr/lib/python3.12/contextlib.py", line 137, in __enter__
2026-08-12 12:19:40.614 return next(self.gen)
2026-08-12 12:19:40.614 ^^^^^^^^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxd_provider.py", line 221, in launched_environment
2026-08-12 12:19:40.614 instance = launch(
2026-08-12 12:19:40.614 ^^^^^^^
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/launcher.py", line 917, in launch
2026-08-12 12:19:40.614 instance.start()
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxd_instance.py", line 582, in start
2026-08-12 12:19:40.614 self.lxc.start(
2026-08-12 12:19:40.614 File "/snap/snapcraft/x1/lib/python3.12/site-packages/craft_providers/lxd/lxc.py", line 1196, in start
2026-08-12 12:19:40.614 raise LXDError(
2026-08-12 12:19:40.614 craft_providers.lxd.errors.LXDError: Failed to start 'snapcraft-test-multi-keys-amd64-1575506'.
2026-08-12 12:19:40.614 * Command that failed: 'lxc --project snapcraft start local:snapcraft-test-multi-keys-amd64-1575506'
2026-08-12 12:19:40.614 * Command exit code: 1
2026-08-12 12:19:40.614 * Command standard error output: b'Error: Failed to run: /snap/lxd/current/sbin/lxd forkstart snapcraft_snapcraft-test-multi-keys-amd64-1575506 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/snapcraft_snapcraft-test-multi-keys-amd64-1575506/lxc.conf: exit status 1\nTry `lxc info --show-log local:snapcraft-test-multi-keys-amd64-1575506` for more info\n'
2026-08-12 12:19:40.614 Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20260812-121933.664475.log'
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen3.6-35b-a3b-mtp-q6 |
Impact:
45
Quick Win:
15.75
Staleness:
35
Complexity:
65
Confidence:
75
Support Request:
15
|
keep open | Intermittent LXD forkstart failures on shared Openstack infrastructure affecting all craft repos' spread tests (<0.5% failure rate); being tracked upstream at LXD #18857. | |
| qwen/qwen3.8-27b |
Impact:
40
Quick Win:
8.0
Staleness:
5
Complexity:
80
Confidence:
85
Support Request:
0
|
keep open | Flaky LXD forkstart failures in Openstack spread tests across craft repos (<0.5% of runs). Reproduced under combined disk I/O, CPU, and memory pressure. Reported upstream to LXD (#18857). Actively investigated by maintainers. | |
| deepseek/deepseek-v4-pro-0813 |
Impact:
40
Quick Win:
16.0
Staleness:
10
Complexity:
60
Confidence:
75
Support Request:
10
|
keep open | Intermittent LXD 'forkstart' failures on OpenStack spread tests in all craft repos; reproduced under combined disk/CPU/memory contention and reported upstream to canonical/lxd#18857. | |
| qwen/qwen3.8-27b |
Impact:
45
Quick Win:
11.25
Staleness:
10
Complexity:
75
Confidence:
80
Support Request:
5
|
keep open | Intermittent (<0.5%) LXD `forkstart` failures when launching spread test containers on the shared Openstack backend, suspected resource contention. Reproduced locally under combined disk I/O + CPU + memory pressure; reported upstream to LXD (canonical/lxd#18857). | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
Support Request:
5
|
keep open | Intermittent LXD container startup failures under resource contention on OpenStack/Multipass backends. Currently under investigation with a reproducer found; reported upstream to LXD. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
0
Complexity:
65
Confidence:
95
Support Request:
5
|
needs triage | Intermittent LXD container start failures in OpenStack spread tests caused by sync errors under resource contention. Currently unlabelled with active reproduction efforts. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
Support Request:
15
|
needs triage | Intermittent LXD container startup failures during Openstack spread tests, likely due to resource contention. Recently replicated under high disk I/O; awaiting maintainer triage. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
65
Confidence:
90
Support Request:
15
|
needs triage | Random LXD container startup failures in OpenStack spread tests due to forkstart errors. Reported by maintainer with detailed logs; currently unlabelled and awaiting triage. |
Update history
| Date | Change |
|---|---|
| updated | |
| updated | |
| updated | |
| created |
Related work
-
Related To:
canonical/lxd#18857
(confidence 90%)
Upstream report filed by mr-cal on 2026-08-13 to track this issue in the LXD project.
-
Related To:
canonical/lxd#11312
(confidence 80%)
Mentioned in the issue body as a similar LXD forkstart failure on a server running 100+ containers, though the error message differs.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #4236 lxd-type-base spread test is flaky due to lxd timeouts | snapcraft | closed | Resolved flaky spread test failures caused by LXD container stop timeouts. Closed via pull request #4239. | |
| #6380 test: debug lxd forkstart failure | snapcraft | closed | Closed without merging. Submitted as a test PR to debug a random LXD forkstart failure in OpenStack. Remained under review with unresolved comments and failing CI checks, indicating the debugging effort was abandoned. |