← Back to issue list

`request` should be able to retry upon broken connections

View original Github issue

Metadata

Project
craft-application
Number
#993
Type
issue
State
open
Author
zhijie-yang
Labels
—
Created
Updated
Closed
—

Current evaluation

remote-build crashes with unhandled ChunkedEncodingError when artifact download breaks mid-stream; request service has no retry support. Still open, no maintainer response, code path unchanged.

Suggested action: needs triage

Reason: The issue has zero comments, no labels, no assignee, and no maintainer response in 249 days, so it has never been triaged despite being a well-written bug report with a full traceback. The defect is still present at HEAD: request.py line 71 streams via `download.iter_content(None)` with no retry and the Session has no HTTPAdapter/max_retries configured. However, the reporter proposes only the desired behaviour, not a concrete design, so implementing it needs maintainer decisions on retry scope and how to handle a partially written destination file and already-advanced progress bar.

Evaluated by: mr-cal-bot

Impact: 52 Quick Win: 16 Complexity: 45 Confidence: 82 Actionability: 55

Issue body

### What needs to get done When using `remote-build` service in `rockcraft`, there are often build failures caused by network instabilities. The network instability causes the unhandled `ChunkedEncodingError` exception inside `craft_application/services/request.py", line 71, in download_chunks`. When the `*craft` tool is downloading the artifact from the build farm, it should be able to retry upon broken connections, such that when using `*craft` tools in CI, the occasional failures happening during the download would not lead to a full rebuild. A log of such an error is attached below: ``` 2026-01-23 17:16:25.270 Succeeded: amd64, arm64, armhf, ppc64el, riscv64, s390x 2026-01-23 17:16:25.271 Fetching build artifacts... 2026-01-23 17:16:31.933 Downloading 6 files (--->) 2026-01-23 17:17:38.222 Downloading 6 files (<---) 2026-01-23 17:17:38.228 rockcraft internal error: ChunkedEncodingError(ProtocolError('Connection broken: IncompleteRead(14610378 bytes read, 25147446 more expected)', IncompleteRead(14610378 bytes read, 25147446 more expected))) 2026-01-23 17:17:38.255 Traceback (most recent call last): 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 779, in _error_catcher 2026-01-23 17:17:38.255 yield 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 904, in _raw_read 2026-01-23 17:17:38.255 data = self._fp_read(amt, read1=read1) if not fp_closed else b"" 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 887, in _fp_read 2026-01-23 17:17:38.255 return self._fp.read(amt) if amt is not None else self._fp.read() 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/current/usr/lib/python3.12/http/client.py", line 495, in read 2026-01-23 17:17:38.255 s = self._safe_read(self.length) 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/current/usr/lib/python3.12/http/client.py", line 642, in _safe_read 2026-01-23 17:17:38.255 raise IncompleteRead(data, amt-len(data)) 2026-01-23 17:17:38.255 http.client.IncompleteRead: IncompleteRead(14610378 bytes read, 25147446 more expected) 2026-01-23 17:17:38.255 2026-01-23 17:17:38.255 The above exception was the direct cause of the following exception: 2026-01-23 17:17:38.255 Traceback (most recent call last): 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/requests/models.py", line 820, in generate 2026-01-23 17:17:38.255 yield from self.raw.stream(chunk_size, decode_content=True) 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 1091, in stream 2026-01-23 17:17:38.255 data = self.read(amt=amt, decode_content=decode_content) 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 980, in read 2026-01-23 17:17:38.255 data = self._raw_read(amt) 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 903, in _raw_read 2026-01-23 17:17:38.255 with self._error_catcher(): 2026-01-23 17:17:38.255 File "/snap/rockcraft/current/usr/lib/python3.12/contextlib.py", line 158, in __exit__ 2026-01-23 17:17:38.255 self.gen.throw(value) 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/urllib3/response.py", line 806, in _error_catcher 2026-01-23 17:17:38.255 raise ProtocolError(f"Connection broken: {e!r}", e) from e 2026-01-23 17:17:38.255 urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(14610378 bytes read, 25147446 more expected)', IncompleteRead(14610378 bytes read, 25147446 more expected)) 2026-01-23 17:17:38.255 2026-01-23 17:17:38.255 During handling of the above exception, another exception occurred: 2026-01-23 17:17:38.255 Traceback (most recent call last): 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/application.py", line 669, in run 2026-01-23 17:17:38.255 return_code = self._run_inner() 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/application.py", line 646, in _run_inner 2026-01-23 17:17:38.255 return_code = dispatcher.run() or os.EX_OK 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_cli/dispatcher.py", line 564, in run 2026-01-23 17:17:38.255 return self._loaded_command.run(self._parsed_command_args) 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/commands/base.py", line 222, in run 2026-01-23 17:17:38.255 result = self._run(parsed_args, **kwargs) or result 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/commands/remote.py", line 165, in _run 2026-01-23 17:17:38.255 returncode = self._monitor_and_complete(builds=builds) 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/commands/remote.py", line 224, in _monitor_and_complete 2026-01-23 17:17:38.255 artifacts = builder.fetch_artifacts(pathlib.Path.cwd()) 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/services/remotebuild.py", line 207, in fetch_artifacts 2026-01-23 17:17:38.255 return self.request.download_files_with_progress(artifact_downloads).values() 2026-01-23 17:17:38.255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/services/request.py", line 109, in download_files_with_progress 2026-01-23 17:17:38.255 for chunk_size in dl: 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/craft_application/services/request.py", line 71, in download_chunks 2026-01-23 17:17:38.255 for chunk in download.iter_content(None): 2026-01-23 17:17:38.255 File "/snap/rockcraft/3974/lib/python3.12/site-packages/requests/models.py", line 822, in generate 2026-01-23 17:17:38.255 raise ChunkedEncodingError(e) 2026-01-23 17:17:38.255 requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(14610378 bytes read, 25147446 more expected)', IncompleteRead(14610378 bytes read, 25147446 more expected)) ``` ### Why it needs to get done As `remote-build` can take a long time to be built, purging a successful build with a single failed download attempt seems very inefficient and wasteful. We should let `*craft` tools give multiple tries to improve the chances of successful builds.

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha
Impact: 52
Quick Win: 16
Complexity: 45
Confidence: 82
Actionability: 55
needs triage remote-build crashes with unhandled ChunkedEncodingError when artifact download breaks mid-stream; request service has no retry support. Still open, no maintainer response, code path unchanged.
stealth/space-bunny-alpha
Impact: 55
Quick Win: 19
Complexity: 55
Confidence: 85
Actionability: 75
needs triage RequestService.download_chunks has no retry, so a ChunkedEncodingError mid-download aborts remote-build artifact fetching and wastes a completed build. Untouched since filing: no labels, no comments, no maintainer response.
stealth/space-bunny-alpha
Impact: 55
Quick Win: 23
Complexity: 35
Confidence: 72
Actionability: 65
needs triage RequestService.download_chunks has no retry, so ChunkedEncodingError from network flakiness fails the whole remote-build artifact/log fetch, discarding a successful build. Awaiting maintainer triage.
qwen/qwen3.6-35b-a3b
Impact: 65
Quick Win: 27
Complexity: 35
Confidence: 75
Actionability: 65
needs triage Feature request to add retry logic for ChunkedEncodingError during artifact downloads in remote-build service
qwen/qwen3.8-27b
Impact: 55
Quick Win: 30
Staleness: 65
Complexity: 45
Confidence: 80
Support Request: 10
needs triage Request service download_chunks crashes with unhandled ChunkedEncodingError when a remote-build artifact download breaks mid-transfer, forcing a full rebuild. No retry logic exists in the code at HEAD; issue is unlabeled with no maintainer activity for ~7 months.
qwen/qwen3.6-35b-a3b
Staleness: 70
Complexity: 45
Confidence: 90
Support Request: 10
needs triage Feature request to implement retry logic for broken network connections during artifact downloads. Issue is unlabelled, has zero maintainer comments, and has been inactive for over 6 months.
qwen3.6-35b-a3b-mtp-q6
Staleness: 75
Complexity: 40
Confidence: 90
Support Request: 10
needs triage Request to implement retry logic for transient network errors during artifact downloads in craft_application. The issue is unlabelled, has zero comments, and has been inactive for 164 days.

Update history

No update history recorded yet.

Related work

  • Related To: craft-application#192 (confidence 70%)

    Commit 4e2f7142 'feat: request service (#192)' introduced the download_chunks path that lacks retries.

  • Related To: craft-application#278 (confidence 55%)

    Prior art in the same repo for retrying flaky network calls (ac19c13e, b589a107, b97d76d6), which established the craft_application.util.retry helper this fix would likely reuse.

Related issues

No related issues found above the similarity threshold.