fix(lxd): retry instance start on forkstart errors
Metadata
Current evaluation
No evaluation has been recorded for this issue yet.
Issue body
LXD occasionally fails to start an instance with a transient `forkstart` error (`Error: Failed to run: /snap/lxd/current/sbin/lxd forkstart ... lxc.conf: exit status 1`). On OpenStack CI runners this hits roughly 0.5% of starts, while other starts in the same run succeed. The root cause in LXD is still unknown (canonical/lxd#18857), so this retries the start rather than addressing it.
`LXC.start()` and `LXC.restart()` now share a helper that:
- retries only when stderr contains `forkstart`, up to `MAX_RETRIES` (3) attempts. Any other error is raised immediately, exactly as before.
- before each retry, waits up to 10 s for the instance to read as `STOPPED`, because LXD can briefly report it as running after a failed start (canonical/lxd#14042). If the status can't be read (for example, an ephemeral instance that was deleted), it stops waiting and retries.
- backs off 1 s, then 2 s, outside `_run_lxc`, so the lxc lock isn't held while sleeping.
- retries a failed `restart` with `lxc start`, because a failed restart leaves the instance stopped and `lxc restart` refuses to act on a stopped instance (`Error: The instance is already stopped`, verified on LXD 5.21.8).
- on the final failure, raises the same `LXDError` as before (`Failed to start '<name>'.` / `Failed to restart '<name>'.`), with details from the last error.
**Backward compatibility:** the retry is always on. It only triggers on `forkstart` errors, which currently fail the build outright. Callers need no changes, and every other failure behaves exactly as before. The only cost is a few extra seconds before a persistent forkstart failure is reported.
**Testing:** new unit tests in `tests/unit/lxd/test_lxc.py` cover start and restart for: forkstart then success, retries exhausted, non-forkstart errors, waiting for `STOPPED`, the wait timing out, and `lxc info` failing. `make test-fast` passes (1693 passed). Ruff, ty and codespell pass, run directly because `make lint` needs `sudo snap install`. The forkstart failure itself can't be reproduced on demand, so the retry hasn't been exercised against a real failing LXD.
Refs canonical/snapcraft#6381, canonical/lxd#18857
---
- [x] I've followed the [contribution guidelines](https://github.com/canonical/starbase/blob/main/CONTRIBUTING.md).
- [x] I've signed the [CLA](http://www.ubuntu.com/legal/contributors/).
- [x] I've successfully run `make lint && make test`.
- [x] I've added or updated any relevant documentation.
- [x] In documents I changed, I [added a meta description](https://canonical-starflow.readthedocs-hosted.com/how-to/add-a-page-meta-description/) if one was missing.
- [x] I've updated the relevant release notes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Evaluation history
No evaluation history available.
Update history
| Date | Change |
|---|---|
| updated | |
| updated | |
| updated | |
| created |
Related issues
No embedding available for this issue yet. Run scripts/run_llm.py evaluate to enable similarity search.