fix(sources): retry HTTP(S) downloads on transient failures
Metadata
Current evaluation
Adds retry-with-backoff to SourceHandler.download() for transient HTTP(S) failures, with security hardening against symlink TOCTOU attacks. CI passing, awaiting human review.
Suggested action: needs review
Reason: The PR has passing CI and multiple rounds of AI-generated reviews confirming correctness, but has zero human reviewers after 24 days. It is a maintainer-authored fix addressing transient download failures (#1666) with substantive changes including retry logic, checksum-before-replace ordering, and symlink TOCTOU hardening — all warrant human maintainer review before merge.
Impact:
55
Quick Win:
19.25
Staleness:
45
Complexity:
65
Confidence:
70
Issue body
> [!NOTE]
> This PR was generated by an AI agent (GitHub Copilot).
## What
Adds retry-with-backoff to `SourceHandler.download()` (used by HTTP(S)-based sources such as `TarSource`) for transient network failures during the pull step.
Previously a single `requests.get()` call was made with no retry logic. Any transient issue (proxy `502`/`503`/`504`, rate limiting `429`, connection reset, network unreachable, etc.) during a pull step would fail the build immediately, even though a retry a moment later would very likely succeed.
## Changes
- `craft_parts/sources/base.py`: `FileSourceHandler.download()` now retries up to 5 times (with exponential backoff: 1s, 2s, 4s, 8s) when:
- the HTTP response status is one of `429, 500, 502, 503, 504`, or
- a `requests.RequestException` occurs (connection errors, timeouts, chunked-encoding errors mid-stream, etc.)
- Non-retriable failures (`404` → `SourceNotFound`, other HTTP errors) still raise immediately, preserving existing behavior.
- Any partially-downloaded file is discarded before each retry attempt so retries always start from a clean state.
- `tests/unit/sources/test_base.py`: added tests covering retry-then-success for transient HTTP status codes, giving up after exhausting retries, and retrying on a network-level exception. Updated existing tests that trigger a retriable code path to mock `time.sleep` so they stay fast.
## Scope
This addresses only the HTTP(S) case described in #1666. The subprocess-based sources (e.g. `GitSource`) are a separate follow-up per that issue.
Closes (partially) #1666
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen3.6-35b-a3b-mtp-q6 |
Impact:
55
Quick Win:
19.25
Staleness:
45
Complexity:
65
Confidence:
70
|
needs review | Adds retry-with-backoff to SourceHandler.download() for transient HTTP(S) failures, with security hardening against symlink TOCTOU attacks. CI passing, awaiting human review. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
40
Confidence:
85
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient failures, fixing fd leaks, checksum verification order, and symlink TOCTOU vulnerabilities. Currently awaiting human maintainer review after multiple AI agent validations and passing CI. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
0
Complexity:
45
Confidence:
85
|
needs review | Adds retry-with-backoff to HTTP(S) downloads for transient failures. Includes robust temp file handling, permission fixes, and checksum verification. Currently awaiting human review with passing CI and multiple AI review rounds completed. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient network failures. Currently polished after multiple review cycles, passing CI, and awaiting maintainer approval. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
40
Confidence:
85
|
needs review | Implements exponential backoff retry for HTTP(S) downloads on transient failures. Passes CI, author has iteratively refined edge cases, currently awaiting maintainer review. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient network failures. Currently under active development with unresolved review comments and pending CI checks. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
85
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient network errors. Currently under active development with recent author updates addressing timeout and socket leak fixes. CI is passing or pending. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
40
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads in SourceHandler.download() to handle transient network failures. Currently pending maintainer review with one unresolved comment and passing CI. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
45
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads in the source handler to handle transient network failures. Currently awaiting maintainer review with CI passing and pending test jobs. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads in SourceHandler.download() to handle transient network failures. Currently awaiting maintainer review with passing CI checks. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
90
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient failures. Currently pending maintainer review with one unresolved comment and multiple CI checks still running. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
5
Complexity:
35
Confidence:
85
|
needs review | Adds exponential backoff retry logic to HTTP(S) downloads for transient failures, discarding partial files on retry. Currently pending maintainer review with 4 unresolved comments and passing CI. |
Update history
| Date | Change |
|---|---|
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| updated | |
| created |
Related work
-
Related To:
canonical/craft-parts#1666
(confidence 90%)
PR explicitly closes this issue partially, addressing only the HTTP(S) case; subprocess-based sources like GitSource are noted as a separate follow-up.
Related issues
No related issues found above the similarity threshold.