remote build: retry launchpad queries
Metadata
Current evaluation
Added exponential backoff for Launchpad queries, with delays of 2, 4, 8, 16, and 32 seconds, via merged PR #355 (commit b97d76d6).
Resolution: closed resolved
Reason: PR #355 was merged as commit b97d76d6, adding Launchpad retries with exponential backoff and a maximum total sleep of 62 seconds.
Evaluated by: mr-cal-bot
No scores available.
Issue body
### Bug Description
Sometimes launchpad queries getting a project, repo or recipe fails and just needs a backoff before trying again. We can probably retry 3 times before fully failing.
### To Reproduce
Check charmcraft remote-build spread tests
### part yaml
_No response_
### Relevant log output
```shell
Total 18 (delta 0), reused 0 (delta 0), pack-reused 0
remote:
remote: Create a merge proposal for 'main' on Launchpad by visiting:
remote: https://code.launchpad.net/~lengau/lengau-craft-remote-build/+git/charmcraft-charm-9bae065def77f3d680f53bbcb0154b9e/+ref/main/+register-merge
remote:
To https://git.launchpad.net/~lengau/lengau-craft-remote-build/+git/charmcraft-charm-9bae065def77f3d680f53bbcb0154b9e
* [new branch] master -> main
charmcraft internal error: ValueError("Could not find charm recipe 'charmcraft-charm-9bae065def77f3d680f53bbcb0154b9e' in project 'lengau-craft-remote-build' with owner 'lengau'")
Full execution log: '/root/.local/state/charmcraft/log/charmcraft-20240319-013323.238305.log'
Traceback (most recent call last):
File "/snap/charmcraft/x1/bin/charmcraft", line 8, in <module>
sys.exit(main())
File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/application/main.py", line 141, in main
return app.run()
File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/application.py", line 476, in run
return_code = dispatcher.run() or 0
File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_cli/dispatcher.py", line 487, in run
return self._loaded_command.run(self._parsed_command_args)
File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/commands/base.py", line 182, in run
result = self._run(parsed_args, **kwargs) or result
File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/application/commands/remote.py", line 125, in _run
builds = builder.start_builds(project_dir)
File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/services/remotebuild.py", line 91, in start_builds
self._recipe = self._new_recipe(
File "/snap/charmcraft/x1/lib/python3.10/site-packages/charmcraft/services/remotebuild.py", line 75, in _new_recipe
return self.lp.get_recipe(
File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/launchpad/launchpad.py", line 161, in get_recipe
return models.CharmRecipe.get(self, name, owner, project)
File "/snap/charmcraft/x1/lib/python3.10/site-packages/craft_application/launchpad/models/recipe.py", line 398, in get
raise ValueError(
ValueError: Could not find charm recipe 'charmcraft-charm-9bae065def77f3d680f53bbcb0154b9e' in project 'lengau-craft-remote-build' with owner 'lengau'
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Added exponential backoff for Launchpad queries, with delays of 2, 4, 8, 16, and 32 seconds, via merged PR #355 (commit b97d76d6). | |
| stealth/space-bunny-alpha | — | closed resolved | Launchpad retries were fixed by merged PR #355 (commit b97d76d6), adding exponential delays of 2–32 seconds and up to 62 seconds total before failure. | |
| stealth/space-bunny-alpha | — | closed resolved | Fixed in canonical/craft-application#355 via commit b97d76d6, adding exponential backoff for Launchpad queries. | |
| stealth/space-bunny-alpha | — | closed resolved | Flaky Launchpad queries in remote-build caused spurious 'Could not find charm recipe' failures. Resolved by canonical/craft-application#355 (commit b97d76d6), merged into main, adding exponential backoff retries (2–32s, ~62s total) for project/repo/recipe lookups. | |
| stealth/space-bunny-alpha | — | closed resolved | Fixed in canonical/craft-application#355 (commit b97d76d6), adding exponential backoff for Launchpad queries to handle intermittent remote-build failures. | |
| stealth/space-bunny-alpha | — | closed resolved | Fixed intermittent remote-build Launchpad query failures via merged craft-application#355 (commit b97d76d6), adding exponential backoff for retries. | |
| qwen/qwen3.6-35b-a3b | — | — | Resolved intermittent Launchpad query failures during remote builds by implementing exponential backoff retry logic. The fix was merged via canonical/craft-application#355, automatically retrying failed requests before timing out. | |
| qwen/qwen3.6-35b-a3b | — | — | Implemented exponential backoff and retry logic for Launchpad queries during remote builds. Resolved via merged PR #355 in canonical/craft-application. |
Update history
No update history recorded yet.
Related work
-
Likely Fixed By:
canonical/craft-application#355
(confidence 100%)
Merged PR implementing exponential retries for Launchpad operations.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #355 feat: exponential backoff for launchpad retries | craft-application | merged | Merged PR #355 as commit b97d76d6, adding Launchpad retry delays of 2, 4, 8, 16, and 32 seconds for a maximum total sleep of 62 seconds. | |
| #352 feat: retry more launchpad queries | craft-application | closed | PR #352 (retry() refactor plus retries on more Launchpad calls) was closed unmerged but its work landed via follow-up PR #355, merged in commit b97d76d6 adding exponential backoff (2/4/8/16/32s, ~62s cap). | |
| #2633 remote-build: increase number of launchpad start_build request attempts | snapcraft | closed | Closed unmerged: more start_build retries were abandoned after the author traced repeated failures to an uncaught Launchpad backend error that the change would not address. |