LXD race condition when packing multiple charms simultaneously
Metadata
Current evaluation
The concurrent LXD project-creation race was fixed in Charmcraft edge and 2.4.0; a follow-up instance-copy race was addressed by craft-providers PR #357.
Resolution: closed resolved
Reason: Maintainer @lengau confirmed the reported lxc project create failure was resolved in edge and candidate 2.4.0. The remaining concurrent-copy race was tracked in #1138 and fixed through craft-providers PR #357, then confirmed in Charmcraft edge.
Evaluated by: mr-cal-bot
No scores available.
Issue body
### Bug Description
When we run our CI we get following error:
```
Traceback (most recent call last):
File "/home/runner/work/kfp-operators/kfp-operators/tests/integration/test_bundle.py", line 39, in test_build_and_deploy
built_charms = await ops_test.build_charms(*charms_to_build.values())
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pytest_operator/plugin.py", line 1028, in build_charms
charms = await asyncio.gather(
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pytest_operator/plugin.py", line 1011, in build_charm
raise RuntimeError(
RuntimeError: Failed to build charm /home/runner/work/kfp-operators/kfp-operators/charms/kfp-viz:
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='20.04' architectures=['amd64'] (may take a while the first time but it's reusable)
charmcraft internal error: LXDError(brief="Failed to create project 'charmcraft'.", details='* Command that failed: \'lxc --project charmcraft project create local:charmcraft\'\n* Command exit code: 1\n* Command standard error output: b\'Error: Failed creating project "charmcraft": Failed adding database record: This "projects" entry already exists\\n\'', resolution=None)
Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20230628-102516.095719.log'
```
here is an example run:
https://github.com/canonical/kfp-operators/actions/runs/5399918594/jobs/9807754324
### To Reproduce
I cannot reproduce it locally, but this is in our CI: https://github.com/canonical/kfp-operators/actions/runs/5399918594/jobs/9807754324
### Environment
lxd 5.15-3fe7435
Ubuntu 20.04
### charmcraft.yaml
```shell
# Learn more about charmcraft.yaml configuration at:
# https://juju.is/docs/sdk/charmcraft-config
type: "charm"
bases:
- build-on:
- name: "ubuntu"
channel: "20.04"
run-on:
- name: "ubuntu"
channel: "20.04"
parts:
charm:
charm-python-packages: [setuptools, pip] # Fixes install of some packages
```
### Relevant log output
```shell
-
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | The concurrent LXD project-creation race was fixed in Charmcraft edge and 2.4.0; a follow-up instance-copy race was addressed by craft-providers PR #357. | |
| stealth/space-bunny-alpha | — | closed resolved | Concurrent LXD packing was fixed through craft-providers changes, including #357, and released in Charmcraft 2.4.0/edge; maintainers confirmed resolution in October 2023. | |
| stealth/space-bunny-alpha | — | closed resolved | Concurrent charm packing was fixed in Charmcraft and craft-providers. The reported LXD project-creation error was addressed in edge and Charmcraft 2.4.0. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Fixed in charmcraft and craft-providers. Parallel LXD commands like lxc project create and lxc copy caused race conditions during simultaneous charm packing. Concurrency controls resolved the issue in charmcraft edge and candidate 2.4.0. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed as fixed. Maintainers resolved the LXD race condition causing parallel charm build failures in charmcraft and craft-providers. The fix was deployed to the edge channel, addressing project creation and instance busy operation errors. | |
| qwen/qwen3.6-35b-a3b | — | — | Fixed in craft-providers and charmcraft edge builds. The LXD race condition causing parallel charm packing failures was resolved by addressing database and instance creation conflicts. Maintainers verified the fix and closed the issue. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Resolved in charmcraft edge and candidate 2.4.0. Concurrency safeguards were implemented to prevent LXD race conditions during parallel charm packing, fixing both project creation and instance copy failures. |
Update history
No update history recorded yet.
Related work
-
Related To:
canonical/charmcraft#1138
(confidence 99%)
Follow-up issue documenting the remaining 'Instance is busy' failure during parallel packing.
-
Likely Fixed By:
canonical/craft-providers#357
(confidence 95%)
Fixed the follow-up LXD concurrency failure encountered when copying build instances.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1138 Packing multiple charms (?) on CI fails with "Instance is busy" message | charmcraft | closed | Parallel charm packing’s LXD race was fixed in craft-providers PR #357 for #341; Charmcraft adopted the provider fix after #1132 proved insufficient. | |
| #554 Issues with concurrent builds | charmcraft | closed | Concurrent LXD setup races were fixed by Craft Providers #158 (commit 3069492), and the maintainer confirmed the fix was included in Charmcraft. | |
| #341 lxd: race condition when running multiple instances of craft-providers | craft-providers | closed | Fixed by merged PR #357, which rechecks LXC project existence after creation fails, preventing concurrent project-creation races (CRAFT-1849). | |
| #925 providers: set lxd project to 'charmcraft' | charmcraft | merged | Merged PR #925 in d82b9bb8, fixing the LXD provider regression by using the charmcraft project instead of default. | |
| #1640 Incompatibility with latest LXD LTS | charmcraft | closed | Fixed in Charmcraft 2.6.0 by merged PR #1641 (commit 2248543f), shipping the craft-providers 1.23.1 fix for LXD versions with the LTS suffix; released to stable. |