Issues with concurrent builds
Metadata
Current evaluation
Concurrent LXD setup races were fixed by Craft Providers #158 (commit 3069492), and the maintainer confirmed the fix was included in Charmcraft.
Resolution: closed resolved
Reason: The issue was resolved through Craft Providers #158, with commit 3069492 identified as the fix. The maintainer confirmed on 2022-10-10 that the change was included in Charmcraft.
Evaluated by: mr-cal-bot
No scores available.
Issue body
I am working on a test in [metallb-operator](https://github.com/charmed-kubernetes/metallb-operator) and was trying to build the two charms for that in parallel. It worked fine for me locally, but on the GitHub runner (a fresh VM, with a fresh install of both lxd and charmcraft on the default channel / track), I ran into a few errors that appear to be due to race conditions in setting up LXD for the builder.
The first one I [ran into](https://github.com/charmed-kubernetes/metallb-operator/pull/18/checks?check_run_id=3693265761#step:6:58) (NB: these build log links will eventually expire) was:
```
Packing charm 'metallb-speaker_ubuntu-20.04-amd64.charm'...
charmcraft internal error! LXDError: Failed to add remote 'craft-com.ubuntu.cloud-buildd'.
* Command that failed: 'lxc remote add craft-com.ubuntu.cloud-buildd https://cloud-images.ubuntu.com/buildd/releases --protocol=simplestreams'
* Command exit code: 1
* Command standard error output: b'Error: Remote craft-com.ubuntu.cloud-buildd exists as <https://cloud-images.ubuntu.com/buildd/releases>\n' (full execution logs in /home/runner/snap/charmcraft/common/charmcraft-log-osa2otki)
```
I tried working around that by running that `lxc` command manually earlier in the Workflow, but then I [ran into](https://github.com/charmed-kubernetes/metallb-operator/pull/18/checks?check_run_id=3700739113#step:7:58):
```
Packing charm 'metallb-controller_ubuntu-20.04-amd64.charm'...
charmcraft internal error! LXDError: Failed to create project 'charmcraft'.
* Command that failed: 'lxc --project charmcraft project create local:charmcraft'
* Command exit code: 1
* Command standard error output: b'Error: Failed creating project "charmcraft": Failed adding database record: This project already exists\n' (full execution logs in /home/runner/snap/charmcraft/common/charmcraft-log-4l5wgu0x)
```
Once again, I added the `lxc` command to the Workflow before the build, and then I [ran into](https://github.com/charmed-kubernetes/metallb-operator/pull/18/checks?check_run_id=3700929568#step:7:59):
```
Packing charm 'metallb-controller_ubuntu-20.04-amd64.charm'...
charmcraft internal error! LXDError: Failed to launch instance 'charmcraft-metallb-controller-1058053-0-0-amd64'.
* Command that failed: "lxc --project charmcraft launch craft-com.ubuntu.cloud-buildd:20.04 local:charmcraft-metallb-controller-1058053-0-0-amd64 --config 'raw.idmap=both 1001 0' --config security.syscalls.intercept.mknod=true"
* Command exit code: 1
* Command output: b'Creating charmcraft-metallb-controller-1058053-0-0-amd64\n'
* Command standard error output: b'Error: Failed instance creation: Failed creating instance record: Failed initialising instance: Invalid devices: Failed detecting root disk device: No root device could be found\n' (full execution logs in /home/runner/snap/charmcraft/common/charmcraft-log-jsos__q1)
```
That one has me stumped so I ended up just de-parallelizing the builds and that works, but it is slower and the errors were confusing and not immediately obvious that they were due to the concurrent build conflict.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Concurrent LXD setup races were fixed by Craft Providers #158 (commit 3069492), and the maintainer confirmed the fix was included in Charmcraft. | |
| stealth/space-bunny-alpha | — | closed resolved | Concurrent-build LXD setup races were addressed by Craft Providers #158 (commit 3069492); the maintainer confirmed the fix was included in Charmcraft. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Concurrent charm builds triggered LXD race conditions and project conflicts. A fix resolving these concurrency issues is now included in Charmcraft, closing the issue. | |
| qwen/qwen3.6-35b-a3b | — | — | Concurrent charm builds triggered LXD race conditions in charmcraft. The issue was fixed and the resolution was included in Charmcraft, eliminating the need for workarounds like de-parallelizing builds or using destructive mode. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Concurrent charm builds triggered LXD race conditions causing remote and project creation failures. Resolved by a fix included in Charmcraft. Workarounds included de-parallelizing builds or using destructive mode. |
Update history
No update history recorded yet.
Related work
-
Likely Fixed By:
canonical/craft-providers#158
(confidence 100%)
The issue resolution directly identifies PR #158 and commit 3069492 as the fix.
-
Related To:
canonical/craft-providers#96
(confidence 95%)
The maintainer linked the problem to Craft Providers' concurrency-support request.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1132 LXD race condition when packing multiple charms simultaneously | charmcraft | closed | The concurrent LXD project-creation race was fixed in Charmcraft edge and 2.4.0; a follow-up instance-copy race was addressed by craft-providers PR #357. | |
| #1640 Incompatibility with latest LXD LTS | charmcraft | closed | Fixed in Charmcraft 2.6.0 by merged PR #1641 (commit 2248543f), shipping the craft-providers 1.23.1 fix for LXD versions with the LTS suffix; released to stable. | |
| #341 lxd: race condition when running multiple instances of craft-providers | craft-providers | closed | Fixed by merged PR #357, which rechecks LXC project existence after creation fails, preventing concurrent project-creation races (CRAFT-1849). | |
| #925 providers: set lxd project to 'charmcraft' | charmcraft | merged | Merged PR #925 in d82b9bb8, fixing the LXD provider regression by using the charmcraft project instead of default. | |
| #1138 Packing multiple charms (?) on CI fails with "Instance is busy" message | charmcraft | closed | Parallel charm packing’s LXD race was fixed in craft-providers PR #357 for #341; Charmcraft adopted the provider fix after #1132 proved insufficient. | |
| #1260 Charmcraft hangs or fails with LXD 5.18 | charmcraft | closed | LXD released a fix for the 5.18 image lookup failure; Charmcraft’s separate edge retry hang was fixed by craft-providers#415 (commit da95d03). | |
| #823 more transparency for container build in first charmcraft pack | charmcraft | closed | Firewall-blocked LXD setup hung `charmcraft pack` indefinitely on instance network setup. No charmcraft change needed; maintainer tracked the missing timeout in craft-providers and closed the issue. | |
| #9 providers: add build provider support (CRAFT-28) | rockcraft | merged | Merged build-provider support via commit be6dec45, integrating craft-providers to run builds on LXD. User messages and provider log retrieval were deferred to follow-up work. |