← Back to issue list

fix(lxd): retry instance start on forkstart errors

View original Github issue

Metadata

Project
craft-providers
Number
#1033
Type
pull request
State
open
Author
GreyViperTooth
Labels
—
Created
Updated
Closed
—

Current evaluation

No evaluation has been recorded for this issue yet.

Issue body

LXD occasionally fails to start an instance with a transient `forkstart` error (`Error: Failed to run: /snap/lxd/current/sbin/lxd forkstart ... lxc.conf: exit status 1`). On OpenStack CI runners this hits roughly 0.5% of starts, while other starts in the same run succeed. The root cause in LXD is still unknown (canonical/lxd#18857), so this retries the start rather than addressing it. `LXC.start()` and `LXC.restart()` now share a helper that: - retries only when stderr contains `forkstart`, up to `MAX_RETRIES` (3) attempts. Any other error is raised immediately, exactly as before. - before each retry, waits up to 10 s for the instance to read as `STOPPED`, because LXD can briefly report it as running after a failed start (canonical/lxd#14042). If the status can't be read (for example, an ephemeral instance that was deleted), it stops waiting and retries. - backs off 1 s, then 2 s, outside `_run_lxc`, so the lxc lock isn't held while sleeping. - retries a failed `restart` with `lxc start`, because a failed restart leaves the instance stopped and `lxc restart` refuses to act on a stopped instance (`Error: The instance is already stopped`, verified on LXD 5.21.8). - on the final failure, raises the same `LXDError` as before (`Failed to start '<name>'.` / `Failed to restart '<name>'.`), with details from the last error. **Backward compatibility:** the retry is always on. It only triggers on `forkstart` errors, which currently fail the build outright. Callers need no changes, and every other failure behaves exactly as before. The only cost is a few extra seconds before a persistent forkstart failure is reported. **Testing:** new unit tests in `tests/unit/lxd/test_lxc.py` cover start and restart for: forkstart then success, retries exhausted, non-forkstart errors, waiting for `STOPPED`, the wait timing out, and `lxc info` failing. `make test-fast` passes (1693 passed). Ruff, ty and codespell pass, run directly because `make lint` needs `sudo snap install`. The forkstart failure itself can't be reproduced on demand, so the retry hasn't been exercised against a real failing LXD. Refs canonical/snapcraft#6381, canonical/lxd#18857 --- - [x] I've followed the [contribution guidelines](https://github.com/canonical/starbase/blob/main/CONTRIBUTING.md). - [x] I've signed the [CLA](http://www.ubuntu.com/legal/contributors/). - [x] I've successfully run `make lint && make test`. - [x] I've added or updated any relevant documentation. - [x] In documents I changed, I [added a meta description](https://canonical-starflow.readthedocs-hosted.com/how-to/add-a-page-meta-description/) if one was missing. - [x] I've updated the relevant release notes. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Evaluation history

No evaluation history available.

Update history

Date Change
updated
updated
updated
created

Related issues

No embedding available for this issue yet. Run scripts/run_llm.py evaluate to enable similarity search.