lxd: race condition when running multiple instances of craft-providers
Metadata
Current evaluation
Fixed by merged PR #357, which rechecks LXC project existence after creation fails, preventing concurrent project-creation races (CRAFT-1849).
Resolution: closed resolved
Reason: Merged PR canonical/craft-providers#357 resolves the race by rechecking whether the LXD project exists when creation fails, treating a concurrently created project as success.
Evaluated by: mr-cal-bot
No scores available.
Issue body
### Overview
New race condition reported from charmcraft: https://github.com/canonical/charmcraft/issues/1132
### Details
This is similar to an issue from 2022:
- https://github.com/canonical/charmcraft/issues/554
- #96
- #158
The charm's CI pipeline runs multiple instances of charmcraft in parallel and create a race condition when executing lxc commands. In the prior bug, the race condition was caused by `lxc remote add`. In this new report, the race condition is caused by `lxc project create`.
### Logs
https://github.com/canonical/kfp-operators/actions/runs/5399918594/jobs/9807754324
```
Traceback (most recent call last):
File "/home/runner/work/kfp-operators/kfp-operators/tests/integration/test_bundle.py", line 39, in test_build_and_deploy
built_charms = await ops_test.build_charms(*charms_to_build.values())
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pytest_operator/plugin.py", line 1028, in build_charms
charms = await asyncio.gather(
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pytest_operator/plugin.py", line 1011, in build_charm
raise RuntimeError(
RuntimeError: Failed to build charm /home/runner/work/kfp-operators/kfp-operators/charms/kfp-viz:
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='20.04' architectures=['amd64'] (may take a while the first time but it's reusable)
charmcraft internal error: LXDError(brief="Failed to create project 'charmcraft'.", details='* Command that failed: \'lxc --project charmcraft project create local:charmcraft\'\n* Command exit code: 1\n* Command standard error output: b\'Error: Failed creating project "charmcraft": Failed adding database record: This "projects" entry already exists\\n\'', resolution=None)
Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20230628-102516.095719.log'
```
### Solution
Craft-providers can use the same race condition handling that was re-used for the prior issue (PR linked above).
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Fixed by merged PR #357, which rechecks LXC project existence after creation fails, preventing concurrent project-creation races (CRAFT-1849). | |
| qwen/qwen3.6-35b-a3b | — | closed resolved | Fixed in craft-providers #357 by rechecking project existence on creation failure. Charmcraft #1132 also resolved via concurrency controls. |
Update history
| Date | Change |
|---|---|
| created |
Related work
-
Likely Fixed By:
canonical/craft-providers#357
(confidence 99%)
Merged fix for CRAFT-1849 rechecked LXD project existence after a failed create.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #357 lxc: recheck when creating project failed | craft-providers | merged | PR #357 (maintainer syu-w, CRAFT-1849) merged into craft-providers main, adding a re-check of LXC project existence after a failed creation to avoid races when concurrent callers create the same project. | |
| #376 lxc: create status data in lxd to avoid race | craft-providers | merged | Merged PR #376 (commit 9a831a5) adds user.craft_providers.status/timer LXD config keys plus instance restart() to avoid duplicate-instance race; fixes #341. | |
| #554 Issues with concurrent builds | charmcraft | closed | Concurrent LXD setup races were fixed by Craft Providers #158 (commit 3069492), and the maintainer confirmed the fix was included in Charmcraft. | |
| #419 providers: gracefully handle when LXD not installed (CRAFT-329) | charmcraft | merged | Merged #419 to handle missing LXD: builds now require the provider, while cleanup safely skips container operations when LXD is unavailable. | |
| #1132 LXD race condition when packing multiple charms simultaneously | charmcraft | closed | The concurrent LXD project-creation race was fixed in Charmcraft edge and 2.4.0; a follow-up instance-copy race was addressed by craft-providers PR #357. | |
| #158 Avoid remote adding race condition with other processes (CRAFT-1043). | craft-providers | merged | Merged PR #158 as commit 3069492, adding retry handling for LXD remote creation to avoid races with other processes and fix #96. | |
| #856 LXDInstance.exists() can error if project doesn't exist in LXD 6.6 | craft-providers | closed | Fixed by merged PR canonical/craft-providers#874 (commit 48bd685), which changed LXDInstance.exists() to use pylxd and avoid lxc list failures for nonexistent LXD projects. | |
| #504 providers: improve error handling (CRAFT-472) | charmcraft | merged | Merged PR #504 improved LXD and Multipass provider error handling, chained exceptions, and added tests for CRAFT-472. | |
| #139 Craft-application does not create unique LXD base instances | craft-application | closed | Closed with no linked PR or commit after more than three years of inactivity; the underlying optional compatibility-tag collision remained tracked in canonical/craft-providers#454. |