Spread test failure: LXD forkstart failure in cross-compile test
Metadata
Current evaluation
Intermittent LXD forkstart failures (~52%) in cross-compile spread tests on OpenStack runners; related to known snapcraft#6381
Suggested action: needs triage
Reason: This is a new, unlabelled issue with no maintainer response or comments (4 days old, 0 comments). While the issue is well-written with clear reproduction steps and the LXD forkstart failure is a real test infrastructure problem, it has not yet been assessed or prioritized by maintainers. Related issues exist in snapcraft (#6381 open, #6380 debug PR closed unmerged) showing this is a known intermittent problem with LXD on OpenStack spread runners.
Evaluated by: mr-cal-bot
Impact:
30
Quick Win:
10
Complexity:
50
Confidence:
70
Actionability:
70
Issue body
### Bug description
The `tests/spread/rockcraft/cross-compile` spread test fails regularly (~52% across workflow runs in October 2026).
When Rockcraft starts a managed LXD instance for a non-native target platform (e.g. `riscv64`), LXD fails during container initialization (`forkstart`) with exit code 1.
#### Affected Spread Tests:
- `tests/spread/rockcraft/cross-compile`
### Steps to reproduce
Run the spread cross-compile test:
```shell
spread openstack:tests/spread/rockcraft/cross-compile
```
Or run `rockcraft pack` for a project configured with cross-compilation platforms on an OpenStack spread runner.
### Environment
- Self-hosted OpenStack runner (`openstack:ubuntu-22.04-64`)
- LXD managed container environment attempting to start cross-compilation instances (e.g., `rockcraft-curl-consumer-riscv64-*`)
### Log output
```shell
Launching managed ubuntu 24.04 instance...
Creating instance from base instance
Starting instance
Failed to start 'rockcraft-curl-consumer-riscv64-1548588'.
Detailed information:
* Command that failed: 'lxc --project rockcraft start local:rockcraft-curl-consumer-riscv64-1548588'
* Command exit code: 1
* Command standard error output: b'Error: Failed to run: /snap/lxd/current/sbin/lxd forkstart rockcraft_rockcraft-curl-consumer-riscv64-1548588 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/rockcraft_rockcraft-curl-consumer-riscv64-1548588/lxc.conf: exit status 1\nTry `lxc info --show-log local:rockcraft-curl-consumer-riscv64-1548588` for more info\n'
Full execution log: '/root/.local/state/rockcraft/log/rockcraft-20261014-135022.777743.log'
```
### Suggested fix
Investigate LXD container startup logs and kernel/QEMU binfmt emulation (`qemu-user-static`) on the spread runner instances to ensure containers configured for foreign architectures can start reliably under LXD.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen/qwen3.6-35b-a3b |
Impact:
30
Quick Win:
10
Complexity:
50
Confidence:
70
Actionability:
70
|
needs triage | Intermittent LXD forkstart failures (~52%) in cross-compile spread tests on OpenStack runners; related to known snapcraft#6381 |
Update history
| Date | Change |
|---|---|
| created |
Related work
-
Related To:
snapcraft#6381
(confidence 85%)
Same type of intermittent LXD forkstart failure in OpenStack spread tests; reported upstream to canonical/lxd#18857
-
Related To:
snapcraft#6380
(confidence 80%)
Debug-only PR to diagnose the same LXD forkstart failure, closed unmerged after 24 days
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #6381 Openstack spread tests randomly fail with `Failed to run: /snap/lxd/current/sbin/lxd forkstart` | snapcraft | open | Intermittent LXD 'forkstart' failures (<0.5%) in the OpenStack spread environment across craft repos; reproduced under heavy host memory/CPU pressure and reported upstream to canonical/lxd#18857, no snapcraft-side fix yet | |
| #6380 test: debug lxd forkstart failure | snapcraft | closed | Debug-only PR by maintainer mr-cal to diagnose random LXD forkstart failures in OpenStack spread tests was closed unmerged after ~24 days with no reviews, 5 unresolved comments and failing lint/CLA checks. | |
| #4140 Flaky spread tests | snapcraft | closed | Flaky LXD spread-test timeouts were addressed by PR #4141 and commit 4b20c8435, which improved the resilience of the core18 lxd-type-base test. | |
| #2515 `google:ubuntu-20.04-64:tests/spread/smoketests/reactive:stable` spread test is failing | charmcraft | closed | Flaky LXD error 'Failed to wait for snap refreshes to complete' (snapd race SNAPDENG-36387) traced to craft-providers; fixed by retrying `snap watch` in craft-providers PR #899, pulled in by charmcraft PR #2583 and released in charmcraft 4.2.0. | |
| #4236 lxd-type-base spread test is flaky due to lxd timeouts | snapcraft | closed | Flaky lxd-type-base spread test (LXC stop 'context deadline exceeded') fixed by PR #4239, squash-merged as commit ab2c13576 'spread: improve resilience of lxd-type-base test'. | |
| #99 Broken spread tests on 20.04 and 22.04 | imagecraft | closed | Ubuntu 20.04/22.04 spread failures were fixed by merged PR #110 (commit 00f1348), correcting legacy LXD client handling in spread setup. | |
| #2937 spread tests: do not attempt to remove snapd snap | snapcraft | merged | Merged in commit b9290bb32; spread tests now skip removing the protected snapd snap, fixing LXD test failures. | |
| #6370 test: migrate to openstack | snapcraft | merged | Merged PR #6370 (commit eabad1564) migrated Snapcraft's spread integration tests from LXD-on-GitHub runners to OpenStack instances, modelled on rockcraft#1279 (SNAPCRAFT-1349). Remaining CI failures were attributed to unrelated flakiness tracked in #6381. | |
| #1200 Spread tests fail on openstack: "remote directory /root/proj ... is not empty" | craft-application | open | OpenStack-only nested testcraft failure: the inner spread run cannot populate /root/proj because a freshly allocated image already contains files; provenance and the fix require backend investigation. | |
| #694 tests(spread): use LXD from `5.21/candidate` | rockcraft | merged | Merged PR #694 in commit 90bc9b34, pinning spread tests to LXD 5.21/candidate to restore main and strengthen LTS regression coverage. |