← Back to issue list

Issues with concurrent builds

View original Github issue

Metadata

Project
charmcraft
Number
#554
Type
issue
State
closed
Author
johnsca
Labels
—
Created
Updated
Closed

Current evaluation

Concurrent LXD setup races were fixed by Craft Providers #158 (commit 3069492), and the maintainer confirmed the fix was included in Charmcraft.

Resolution: closed resolved

Reason: The issue was resolved through Craft Providers #158, with commit 3069492 identified as the fix. The maintainer confirmed on 2022-10-10 that the change was included in Charmcraft.

Evaluated by: mr-cal-bot

No scores available.

Issue body

I am working on a test in [metallb-operator](https://github.com/charmed-kubernetes/metallb-operator) and was trying to build the two charms for that in parallel. It worked fine for me locally, but on the GitHub runner (a fresh VM, with a fresh install of both lxd and charmcraft on the default channel / track), I ran into a few errors that appear to be due to race conditions in setting up LXD for the builder. The first one I [ran into](https://github.com/charmed-kubernetes/metallb-operator/pull/18/checks?check_run_id=3693265761#step:6:58) (NB: these build log links will eventually expire) was: ``` Packing charm 'metallb-speaker_ubuntu-20.04-amd64.charm'... charmcraft internal error! LXDError: Failed to add remote 'craft-com.ubuntu.cloud-buildd'. * Command that failed: 'lxc remote add craft-com.ubuntu.cloud-buildd https://cloud-images.ubuntu.com/buildd/releases --protocol=simplestreams' * Command exit code: 1 * Command standard error output: b'Error: Remote craft-com.ubuntu.cloud-buildd exists as <https://cloud-images.ubuntu.com/buildd/releases>\n' (full execution logs in /home/runner/snap/charmcraft/common/charmcraft-log-osa2otki) ``` I tried working around that by running that `lxc` command manually earlier in the Workflow, but then I [ran into](https://github.com/charmed-kubernetes/metallb-operator/pull/18/checks?check_run_id=3700739113#step:7:58): ``` Packing charm 'metallb-controller_ubuntu-20.04-amd64.charm'... charmcraft internal error! LXDError: Failed to create project 'charmcraft'. * Command that failed: 'lxc --project charmcraft project create local:charmcraft' * Command exit code: 1 * Command standard error output: b'Error: Failed creating project "charmcraft": Failed adding database record: This project already exists\n' (full execution logs in /home/runner/snap/charmcraft/common/charmcraft-log-4l5wgu0x) ``` Once again, I added the `lxc` command to the Workflow before the build, and then I [ran into](https://github.com/charmed-kubernetes/metallb-operator/pull/18/checks?check_run_id=3700929568#step:7:59): ``` Packing charm 'metallb-controller_ubuntu-20.04-amd64.charm'... charmcraft internal error! LXDError: Failed to launch instance 'charmcraft-metallb-controller-1058053-0-0-amd64'. * Command that failed: "lxc --project charmcraft launch craft-com.ubuntu.cloud-buildd:20.04 local:charmcraft-metallb-controller-1058053-0-0-amd64 --config 'raw.idmap=both 1001 0' --config security.syscalls.intercept.mknod=true" * Command exit code: 1 * Command output: b'Creating charmcraft-metallb-controller-1058053-0-0-amd64\n' * Command standard error output: b'Error: Failed instance creation: Failed creating instance record: Failed initialising instance: Invalid devices: Failed detecting root disk device: No root device could be found\n' (full execution logs in /home/runner/snap/charmcraft/common/charmcraft-log-jsos__q1) ``` That one has me stumped so I ended up just de-parallelizing the builds and that works, but it is slower and the errors were confusing and not immediately obvious that they were due to the concurrent build conflict.

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha — closed resolved Concurrent LXD setup races were fixed by Craft Providers #158 (commit 3069492), and the maintainer confirmed the fix was included in Charmcraft.
stealth/space-bunny-alpha — closed resolved Concurrent-build LXD setup races were addressed by Craft Providers #158 (commit 3069492); the maintainer confirmed the fix was included in Charmcraft.
qwen3.6-35b-a3b-mtp-q6 — — Concurrent charm builds triggered LXD race conditions and project conflicts. A fix resolving these concurrency issues is now included in Charmcraft, closing the issue.
qwen/qwen3.6-35b-a3b — — Concurrent charm builds triggered LXD race conditions in charmcraft. The issue was fixed and the resolution was included in Charmcraft, eliminating the need for workarounds like de-parallelizing builds or using destructive mode.
qwen3.6-35b-a3b-mtp-q6 — — Concurrent charm builds triggered LXD race conditions causing remote and project creation failures. Resolved by a fix included in Charmcraft. Workarounds included de-parallelizing builds or using destructive mode.

Update history

No update history recorded yet.

Related work

  • Likely Fixed By: canonical/craft-providers#158 (confidence 100%)

    The issue resolution directly identifies PR #158 and commit 3069492 as the fix.

  • Related To: canonical/craft-providers#96 (confidence 95%)

    The maintainer linked the problem to Craft Providers' concurrency-support request.

Related issues

Issue Project State Summary Similarity
#1132 LXD race condition when packing multiple charms simultaneously charmcraft closed The concurrent LXD project-creation race was fixed in Charmcraft edge and 2.4.0; a follow-up instance-copy race was addressed by craft-providers PR #357.
80%
#1640 Incompatibility with latest LXD LTS charmcraft closed Fixed in Charmcraft 2.6.0 by merged PR #1641 (commit 2248543f), shipping the craft-providers 1.23.1 fix for LXD versions with the LTS suffix; released to stable.
76%
#341 lxd: race condition when running multiple instances of craft-providers craft-providers closed Fixed by merged PR #357, which rechecks LXC project existence after creation fails, preventing concurrent project-creation races (CRAFT-1849).
76%
#925 providers: set lxd project to 'charmcraft' charmcraft merged Merged PR #925 in d82b9bb8, fixing the LXD provider regression by using the charmcraft project instead of default.
75%
#1138 Packing multiple charms (?) on CI fails with "Instance is busy" message charmcraft closed Parallel charm packing’s LXD race was fixed in craft-providers PR #357 for #341; Charmcraft adopted the provider fix after #1132 proved insufficient.
72%
#1260 Charmcraft hangs or fails with LXD 5.18 charmcraft closed LXD released a fix for the 5.18 image lookup failure; Charmcraft’s separate edge retry hang was fixed by craft-providers#415 (commit da95d03).
71%
#823 more transparency for container build in first charmcraft pack charmcraft closed Firewall-blocked LXD setup hung `charmcraft pack` indefinitely on instance network setup. No charmcraft change needed; maintainer tracked the missing timeout in craft-providers and closed the issue.
71%
#9 providers: add build provider support (CRAFT-28) rockcraft merged Merged build-provider support via commit be6dec45, integrating craft-providers to run builds on LXD. User messages and provider log retrieval were deferred to follow-up work.
70%