Charmcraft hangs or fails with LXD 5.18
Metadata
Current evaluation
LXD released a fix for the 5.18 image lookup failure; Charmcraft’s separate edge retry hang was fixed by craft-providers#415 (commit da95d03).
Resolution: closed resolved
Reason: The upstream LXD defect in canonical/lxd#12295 was fixed and released. The additional Charmcraft edge hang was tracked in canonical/craft-providers#413 and fixed by merged PR canonical/craft-providers#415.
Evaluated by: mr-cal-bot
No scores available.
Issue body
### Bug Description
Charmcraft has been hanging with latest/edge lxd.
```
$ sudo snap info lxd
...
tracking: latest/edge <----------------------------
refresh-date: today at 15:28 CEST <----------------------------------
channels:
latest/stable: 5.17-e5ead86 2023-08-30 (25505) 184MB -
latest/candidate: 5.17-e5ead86 2023-08-25 (25505) 184MB -
latest/beta: ↑
latest/edge: git-87fbf8a 2023-09-18 (25653) 189MB -
5.17/stable: 5.17-e5ead86 2023-08-29 (25505) 184MB -
```
Running with charmcraft in latest/stable fails with:
```
$ charmcraft pack --debug -v
Starting charmcraft version 2.4.1
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Checking for instance 'charmcraft-mongodb-k8s-12244861-0-0-amd64' in project 'charmcraft' in remote 'local'
Instance 'charmcraft-mongodb-k8s-12244861-0-0-amd64' does not exist.
Checking for base instance 'base-instance-charmcraft-buildd-base-v10-689fe40991089812a57b' in project 'charmcraft' in remote 'local'
Base instance 'base-instance-charmcraft-buildd-base-v10-689fe40991089812a57b' does not exist.
Creating new instance from image 'core22' from remote 'craft-com.ubuntu.cloud-buildd'.
charmcraft internal error: LXDError(brief="Failed to launch instance 'charmcraft-mongodb-k8s-12244861-0-0-amd64'.", details='* Command that failed: \'lxc --project charmcraft launch craft-com.ubuntu.cloud-buildd:core22 local:charmcraft-mongodb-k8s-12244861-0-0-amd64 --config security.syscalls.intercept.mknod=true\'\n* Command exit code: 1\n* Command output: b\'Creating charmcraft-mongodb-k8s-12244861-0-0-amd64\\n\'\n* Command standard error output: b"Error: Failed instance creation: Failed getting remote image info: Failed getting image: The requested image couldn\'t be found\\n"', resolution=None)
```
Refreshing charmcraft to latest edge and it hangs on:
```
...
2023-09-18 16:26:42.128 Command '['lxc', '--project', 'charmcraft', 'launch', 'craft-com.ubuntu.cloud-buildd:core22', 'local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50', '--config', 'security.syscalls.intercept.mknod=true', '--config', 'user.craft_providers.status=STARTING', '--config', 'user.craft_providers.timer=2023-09-18T14:26:41.949784+00:00']' returned non-zero exit status 1.
2023-09-18 16:26:42.128 Failed to launch instance base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50, checking status.
2023-09-18 16:26:42.128 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:45.195 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:48.287 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:51.367 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:54.455 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:57.535 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
...
## STAYS UNTIL TIMEOUT
```
### To Reproduce
```
# Delete every image in lxc's charmcraft project, including base image
$ sudo lxc delete --force ...
$ git clone https://github.com/canonical/mongodb-operator
# Refresh lxd
$ sudo snap refresh lxd --channel=latest/edge
# Checkout charmcraft
$ sudo snap install charmcraft
# Now build
$ charmcraft pack --debug -v
# Fails, try with edge
$ sudo snap refresh charmcraft --channel=latest/edge
$ charmcraft pack --debug -v # hangs
# Try removing ~/.local/state/charmcraft
# Same results
### Environment
Ubuntu 22.04
Using LXD
### charmcraft.yaml
```shell
https://github.com/canonical/mongodb-operator/blob/d9e79262dbc8a57bd7de39c817f526a0914b4adc/charmcraft.yaml#L4
```
### Relevant log output
```shell
...
2023-09-18 16:26:42.128 Command '['lxc', '--project', 'charmcraft', 'launch', 'craft-com.ubuntu.cloud-buildd:core22', 'local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50', '--config', 'security.syscalls.intercept.mknod=true', '--config', 'user.craft_providers.status=STARTING', '--config', 'user.craft_providers.timer=2023-09-18T14:26:41.949784+00:00']' returned non-zero exit status 1.
2023-09-18 16:26:42.128 Failed to launch instance base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50, checking status.
2023-09-18 16:26:42.128 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:45.195 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:48.287 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:51.367 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:54.455 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
2023-09-18 16:26:57.535 Executing on host: lxc --project charmcraft info local:base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | LXD released a fix for the 5.18 image lookup failure; Charmcraft’s separate edge retry hang was fixed by craft-providers#415 (commit da95d03). | |
| stealth/space-bunny-alpha | — | closed not a bug | Charmcraft pack failed/hung with LXD 5.18 edge. Root cause was upstream LXD bug (canonical/lxd#12295), fixed upstream; the retry-hang on edge was tracked in craft-providers#413. Closed by lengau, no charmcraft code change. | |
| stealth/space-bunny-alpha | — | closed resolved | LXD released an upstream fix for the 5.18 image lookup failure, and Charmcraft #1260 was closed; a separate edge retry hang is tracked in craft-providers#413. | |
| stealth/space-bunny-alpha | — | closed resolved | Resolved upstream in LXD via #12295, whose fix was released; Charmcraft issue closed. A separate edge-only hang was fixed by craft-providers#415 (da95d03). | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Closed due to an upstream LXD bug. A fix was released for LXD 5.18 compatibility, resolving the hangs and failures. The investigation also uncovered a separate craft-providers issue tracked independently. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed as an upstream LXD bug. Reverting to stable LXD resolves the hang. Upstream released a fix, but the report also exposed a separate craft-providers issue. Linked for reference. | |
| qwen/qwen3.6-35b-a3b | — | — | Upstream LXD released a fix for the instance launch failure causing charmcraft to hang or fail with LXD 5.18. This issue is closed as resolved by the upstream patch, with a related craft-providers bug linked for reference. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Closed. Upstream LXD bug #12295 was fixed, resolving the hang. Rolling back to stable LXD served as a temporary workaround. The report also exposed a secondary craft-providers issue. Closed for reference. |
Update history
No update history recorded yet.
Related work
-
Likely Fixed By:
canonical/craft-providers#415
(confidence 99%)
Merged PR changing LXD provider to fail fast on non-creation errors; commit da95d03.
-
Related To:
canonical/craft-providers#413
(confidence 99%)
Follow-up issue for the edge-only indefinite retry when an LXD image was unavailable.
-
Likely Fixed By:
lxd#12295
(confidence 96%)
Upstream LXD image lookup defect; maintainers confirmed a fix had been released.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #413 Charmcraft edge build hangs indefinitely when lxd image can't be found. | craft-providers | closed | Missing-image LXD launch failures now fail fast with an error instead of polling a nonexistent instance forever, resolving the Charmcraft edge hang. | |
| #1640 Incompatibility with latest LXD LTS | charmcraft | closed | Fixed in Charmcraft 2.6.0 by merged PR #1641 (commit 2248543f), shipping the craft-providers 1.23.1 fix for LXD versions with the LTS suffix; released to stable. | |
| #560 Charmcraft pack is giving error frequently | charmcraft | closed | The failure was traced to external LXD networking blocking access to archive.ubuntu.com during snapd installation; no Charmcraft fix was merged, and the reported firewall workaround did not help. | |
| #1047 Charmcraft stuck on "launching environment" | charmcraft | closed | Craft-providers now verifies network connectivity and times out hung LXD setup, preventing indefinite waits; underlying LXD/Docker firewall conflicts may still require configuration. | |
| #92 Suddenly getting charmcraft internal error with charmcraft pack | craft-providers | closed | Resolved upstream in LXD 4.23 revision 22525+, which corrected overly strict device-name validation; `snap revert lxd` was a temporary workaround. | |
| #511 `charmcraft pack` fails running `apt update` | charmcraft | closed | Closed with no Charmcraft code change: maintainers determined the `apt update` failure came from a broken host/LXD (fan bridge) networking setup, not a Charmcraft bug; guidance posted to Charmhub Discourse and the LXD forum. | |
| #1054 Charmcraft fails to launch LXD instance with high uid (`raw.idmap=both 595156457 0`) | charmcraft | open | High host UID causes LXD launch failure via raw.idmap=both; diagnosed as craft-providers idmap issue, fixed upstream by separate uid/gid maps and charmcraft now depends on fixed version. | |
| #925 providers: set lxd project to 'charmcraft' | charmcraft | merged | Merged PR #925 in d82b9bb8, fixing the LXD provider regression by using the charmcraft project instead of default. | |
| #823 more transparency for container build in first charmcraft pack | charmcraft | closed | Firewall-blocked LXD setup hung `charmcraft pack` indefinitely on instance network setup. No charmcraft change needed; maintainer tracked the missing timeout in craft-providers and closed the issue. | |
| #647 `charmcraft build` sometimes fails on `apt install` inside the lxd container | charmcraft | closed | Closed as a duplicate of #567; its dpkg-lock fix merged via PR #745 (commit b1adccd5), adding provider warmup to avoid competing APT updates. |