fix(sources): retry HTTP(S) downloads on transient failures
Metadata
Current evaluation
Maintainer-authored PR adding retry-with-backoff to HTTP(S) source downloads (addresses open issue #1666). AI-agent generated; after several model-review rounds includes temp-file, checksum, and symlink/TOCTOU hardening. CI green (0 failing), 0 human reviews yet.
Suggested action: needs review
Reason: CI is fully green (0 failing; only some checks still pending) and there are 0 unresolved review comments, with the maintainer/author lengau having iterated through several AI-review rounds that now report 'NO NEW ISSUES FOUND' and 'All 235 tests pass.' The change directly addresses the still-open issue #1666 ('Source pull step doesn't retry on transient network failures'). However, review status shows 'pending (0 reviewers)' — all reviews are from copilot-swe-agent, not a human maintainer — so it is ready for human review but not yet approved to land.
Evaluated by: mr-cal-bot
Impact:
55
Quick Win:
19
Complexity:
55
Confidence:
82
Actionability:
78
Issue body
> [!NOTE]
> This PR was generated by an AI agent (GitHub Copilot).
## What
Adds retry-with-backoff to `SourceHandler.download()` (used by HTTP(S)-based sources such as `TarSource`) for transient network failures during the pull step.
Previously a single `requests.get()` call was made with no retry logic. Any transient issue (proxy `502`/`503`/`504`, rate limiting `429`, connection reset, network unreachable, etc.) during a pull step would fail the build immediately, even though a retry a moment later would very likely succeed.
## Changes
- `craft_parts/sources/base.py`: `FileSourceHandler.download()` now retries up to 5 times (with exponential backoff: 1s, 2s, 4s, 8s) when:
- the HTTP response status is one of `429, 500, 502, 503, 504`, or
- a `requests.RequestException` occurs (connection errors, timeouts, chunked-encoding errors mid-stream, etc.)
- Non-retriable failures (`404` → `SourceNotFound`, other HTTP errors) still raise immediately, preserving existing behavior.
- Any partially-downloaded file is discarded before each retry attempt so retries always start from a clean state.
- `tests/unit/sources/test_base.py`: added tests covering retry-then-success for transient HTTP status codes, giving up after exhausting retries, and retrying on a network-level exception. Updated existing tests that trigger a retriable code path to mock `time.sleep` so they stay fast.
## Scope
This addresses only the HTTP(S) case described in #1666. The subprocess-based sources (e.g. `GitSource`) are a separate follow-up per that issue.
Closes (partially) #1666
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha |
Impact:
55
Quick Win:
19
Complexity:
55
Confidence:
82
Actionability:
78
|
needs review | Maintainer-authored PR adding retry-with-backoff to HTTP(S) source downloads (addresses open issue #1666). AI-agent generated; after several model-review rounds includes temp-file, checksum, and symlink/TOCTOU hardening. CI green (0 failing), 0 human reviews yet. | |
| stealth/space-bunny-alpha |
Impact:
65
Quick Win:
13
Complexity:
70
Confidence:
72
Actionability:
66
|
needs review | Adds retry-with-backoff to FileSourceHandler.download() for transient HTTP failures, plus atomic O_EXCL/O_NOFOLLOW temp-file hardening in url_utils.download_request(). 3 files, +448/-22, CI green, awaiting human review; closes only the HTTP half of #1666. | |
| stealth/space-bunny-alpha |
Impact:
54
Quick Win:
24
Complexity:
46
Confidence:
92
Actionability:
82
|
needs review | Adds bounded exponential-backoff retries for HTTP(S) source downloads, with hardened temporary-file handling and unit coverage. Final review found no issues; all completed CI checks pass, with seven pending. | |
| qwen3.6-35b-a3b-mtp-q6 |
Impact:
55
Quick Win:
19
Staleness:
45
Complexity:
65
Confidence:
70
|
needs review | Adds retry-with-backoff to SourceHandler.download() for transient HTTP(S) failures, with security hardening against symlink TOCTOU attacks. CI passing, awaiting human review. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
40
Confidence:
85
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient failures, fixing fd leaks, checksum verification order, and symlink TOCTOU vulnerabilities. Currently awaiting human maintainer review after multiple AI agent validations and passing CI. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
0
Complexity:
45
Confidence:
85
|
needs review | Adds retry-with-backoff to HTTP(S) downloads for transient failures. Includes robust temp file handling, permission fixes, and checksum verification. Currently awaiting human review with passing CI and multiple AI review rounds completed. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient network failures. Currently polished after multiple review cycles, passing CI, and awaiting maintainer approval. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
40
Confidence:
85
|
needs review | Implements exponential backoff retry for HTTP(S) downloads on transient failures. Passes CI, author has iteratively refined edge cases, currently awaiting maintainer review. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient network failures. Currently under active development with unresolved review comments and pending CI checks. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
85
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient network errors. Currently under active development with recent author updates addressing timeout and socket leak fixes. CI is passing or pending. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
40
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads in SourceHandler.download() to handle transient network failures. Currently pending maintainer review with one unresolved comment and passing CI. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads in the source handler to handle transient network failures. Currently awaiting maintainer review with CI passing and pending test jobs. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads in SourceHandler.download() to handle transient network failures. Currently awaiting maintainer review with passing CI checks. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient failures. Currently pending maintainer review with one unresolved comment and multiple CI checks still running. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
85
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient failures, discarding partial files on retry. Currently pending maintainer review with 4 unresolved comments and passing CI. |
Update history
| Date | Change |
|---|---|
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| created |
Related work
-
Related To:
craft-parts#1666
(confidence 97%)
Linked open issue this PR closes (partially); the subprocess-based source (e.g. GitSource) retry is explicitly deferred to a follow-up per that issue.
Related issues
No related issues found above the similarity threshold.