Packing multiple charms (?) on CI fails with "Instance is busy" message
Metadata
Current evaluation
Parallel charm packing’s LXD race was fixed in craft-providers PR #357 for #341; Charmcraft adopted the provider fix after #1132 proved insufficient.
Resolution: closed resolved
Reason: craft-providers PR #357 implemented the LXD race fix tracked by canonical/craft-providers#341. The resolution records that Charmcraft adopted this fix after a Spread reproducer was added and #1132 was found not to address the base-instance race.
Evaluated by: mr-cal-bot
No scores available.
Issue body
### Bug Description
The Charmed Kubeflow team has a repository with 7 charms ([canonical/kfp-operators](github.com/canonical/kfp-operators)), in which the CI runs to build and deploy the seven of them. Having this many charms may be causing an issue as we are getting the following error when calling `built_charms = await ops_test.build_charms(*charms_to_build.values())`:
```
RuntimeError: Failed to build charm /home/runner/work/kfp-operators/kfp-operators/charms/kfp-schedwf:
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='20.04' architectures=['amd64'] (may take a while the first time but it's reusable)
charmcraft internal error: LXDError(brief="Failed to copy instance 'local:charmcraft-kfp-schedwf-1070911-0-0-amd64' to 'local:base-instance-charmcraft-buildd-base-v10-07bde40a62a827cdc5a4'.", details='* Command that failed: \'lxc --project charmcraft copy local:charmcraft-kfp-schedwf-1070911-0-0-amd64 local:base-instance-charmcraft-buildd-base-v10-07bde40a62a827cdc5a4\'\n* Command exit code: 1\n* Command standard error output: b\'Error: Failed creating instance record: Instance is busy running a "create" operation\\n\'', resolution=None)
Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20230704-165315.306636.log'
```
For more details go [here](https://github.com/canonical/kfp-operators/actions/runs/5456837818/jobs/9930395637#step:7:34)
It sometimes fails with:
```
charmcraft internal error: LXDError(brief="Failed to create project 'charmcraft'.", details='* Command that failed: \'lxc --project charmcraft project create local:charmcraft\'\n* Command exit code: 1\n* Command standard error output: b\'Error: Failed creating project "charmcraft": Failed adding database record: This "projects" entry already exists\\n\'', resolution=None)
Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20230704-172534.136770.log'
```
Same as #1132
For more details go [here](https://github.com/canonical/kfp-operators/actions/runs/5457119983/jobs/9930795922?pr=239#step:7:7)
#### Workaround
For me using `destructive-mode` works, of course, but it is far from ideal.
### To Reproduce
I haven't been able to reproduce in my local dev machine, but happens on GH runners:
1. Clone https://github.com/canonical/kfp-operators/tree/main
2. Apply this patch
```diff
diff --git a/.github/workflows/integrate.yaml b/.github/workflows/integrate.yaml
index 9d8b7fe..704ad15 100644
--- a/.github/workflows/integrate.yaml
+++ b/.github/workflows/integrate.yaml
@@ -133,7 +133,7 @@ jobs:
# Remove destructive-mode once these bugs are fixed:
# https://github.com/canonical/charmcraft/issues/554
# https://github.com/canonical/craft-providers/issues/96
- tox -e bundle-integration -- --model kubeflow --destructive-mode
+ tox -e bundle-integration -- --model kubeflow
- name: Get all
run: kubectl get all -A
```
3. Submit a PR and observe the "[Run Tests / Test the bundle](https://github.com/canonical/kfp-operators/actions/runs/5457119983/jobs/9930795922?pr=239)" job fail
### Environment
* GH runners on ubuntu 20.04
* microk8s channel: 1.24/stable
* juju-channel: 2.9/stable
* charmcraft-channel: latest/candidate
* bootstrap-constraints: cores=2 mem=4G
* charm-channel: latest/stable
* juju-bundle-channel: latest/stable
* lxd-channel: latest/stable
* juju-crashdump-channel: latest/stable
* microk8s-addons: storage dns rbac
### charmcraft.yaml
```shell
N/A
```
### Relevant log output
```shell
=================================== FAILURES ===================================
____________________________ test_build_and_deploy _____________________________
Traceback (most recent call last):
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/runner.py", line 341, in from_call
result: Optional[TResult] = func()
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/runner.py", line 262, in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 433, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_manager.py", line 112, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_callers.py", line 155, in _multicall
return outcome.get_result()
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_result.py", line 108, in get_result
raise exc.with_traceback(exc.__traceback__)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_callers.py", line 80, in _multicall
res = hook_impl.function(*args)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/runner.py", line 177, in pytest_runtest_call
raise e
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
item.runtest()
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/python.py", line 1788, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_hooks.py", line 433, in __call__
return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_manager.py", line 112, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_callers.py", line 155, in _multicall
return outcome.get_result()
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_result.py", line 108, in get_result
raise exc.with_traceback(exc.__traceback__)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pluggy/_callers.py", line 80, in _multicall
res = hook_impl.function(*args)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/_pytest/python.py", line 194, in pytest_pyfunc_call
result = testfunction(**testargs)
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pytest_asyncio/plugin.py", line 525, in inner
_loop.run_until_complete(task)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/runner/work/kfp-operators/kfp-operators/tests/integration/test_kfp_generic_installation.py", line 200, in test_build_and_deploy
built_charms = await ops_test.build_charms(*charms_to_build.values())
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pytest_operator/plugin.py", line 1028, in build_charms
charms = await asyncio.gather(
File "/home/runner/work/kfp-operators/kfp-operators/.tox/bundle-integration/lib/python3.8/site-packages/pytest_operator/plugin.py", line 1011, in build_charm
raise RuntimeError(
RuntimeError: Failed to build charm /home/runner/work/kfp-operators/kfp-operators/charms/kfp-api:
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='20.04' architectures=['amd64'] (may take a while the first time but it's reusable)
charmcraft internal error: LXDError(brief="Failed to create project 'charmcraft'.", details='* Command that failed: \'lxc --project charmcraft project create local:charmcraft\'\n* Command exit code: 1\n* Command standard error output: b\'Error: Failed creating project "charmcraft": Failed adding database record: This "projects" entry already exists\\n\'', resolution=None)
Full execution log: '/home/runner/.local/state/charmcraft/log/charmcraft-20230704-145156.175803.log'
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Parallel charm packing’s LXD race was fixed in craft-providers PR #357 for #341; Charmcraft adopted the provider fix after #1132 proved insufficient. | |
| stealth/space-bunny-alpha | — | closed resolved | The parallel-pack LXD race was fixed by incorporating the synchronization fix from craft-providers PR #357, linked to issue #341, after charmcraft#1132 proved incomplete. | |
| stealth/space-bunny-alpha | — | closed resolved | Parallel charm packing’s LXD race was resolved through craft-providers PR #357, linked to issue #341, after charmcraft#1132 proved insufficient. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Closed as a duplicate of #1132. The issue was not fixed by that PR. Resolution is deferred to the craft-providers library, pending a test reproducer and an upstream fix from issue #341. | |
| qwen/qwen3.6-35b-a3b | — | — | Superseded as a duplicate of #1132. The craft-providers race condition is unresolved but tracked in craft-providers issue #341. A test reproducer will be added to Spread before the fix is deployed. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed as a duplicate of #1132. The parallel packing race condition remains unfixed; resolution is deferred to craft-providers#341 pending a test reproducer. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Closed as a duplicate of #1132, but the referenced fix did not resolve the parallel packing race condition in craft-providers. Resolution requires a test reproducer and an upstream craft-providers update based on issue #341. |
Update history
No update history recorded yet.
Related work
-
Likely Fixed By:
canonical/craft-providers#357
(confidence 99%)
Implemented the provider race-condition fix used to resolve the parallel packing failure.
-
Related To:
canonical/craft-providers#341
(confidence 98%)
Upstream craft-providers issue containing the base-instance race fix.
-
Related To:
canonical/charmcraft#1132
(confidence 98%)
Initially marked as a duplicate, but later confirmed not to solve this base-instance failure.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1132 LXD race condition when packing multiple charms simultaneously | charmcraft | closed | The concurrent LXD project-creation race was fixed in Charmcraft edge and 2.4.0; a follow-up instance-copy race was addressed by craft-providers PR #357. | |
| #554 Issues with concurrent builds | charmcraft | closed | Concurrent LXD setup races were fixed by Craft Providers #158 (commit 3069492), and the maintainer confirmed the fix was included in Charmcraft. | |
| #629 Cannot pack charms with long names | charmcraft | closed | Fixed by truncating overlong instance names, stripping invalid characters, and appending a hash; Charmcraft adopted craft-providers 1.4.1 in PR #852. | |
| #823 more transparency for container build in first charmcraft pack | charmcraft | closed | Firewall-blocked LXD setup hung `charmcraft pack` indefinitely on instance network setup. No charmcraft change needed; maintainer tracked the missing timeout in craft-providers and closed the issue. |