← Back to issue list

FileSourceHandler is slow for large sources

View original Github issue

Metadata

Project
craft-parts
Number
#1175
Type
issue
State
open
Author
enriqueesanchz
Labels
Status: Triaged Type: Enhancement
Created
Updated
Closed
—

Current evaluation

FileSourceHandler downloads/extracts large sources very slowly (requests+tarfile,1KB chunks, 3600s timeout): ~6h vs ~1.5h with wget on RISC-V. Triaged enhancement, root cause identified, optimizations (piped download+extract, threaded decompression) not yet implemented.

Suggested action: keep open

Reason: Labels 'Status: Triaged'/'Type: Enhancement' plus substantive maintainer input from @steinbro (asking about the 1-hour timeout rationale and an `extract-patterns` key) and @lengau (scoping the work to 'download and begin extraction simultaneously', 'multi-threaded decompression where feasible', 'separate threads') confirm a live, valid enhancement tracked as CRAFT-4689. Code inspection shows the slow path is unchanged: `craft_parts/sources/base.py:313-314` still uses `requests.get(..., timeout=3600)` with `url_utils.download_request` writing 1024-byte chunks, and `craft_parts/sources/tar_source.py` still extracts via single-threaded `tarfile.extractall`; `git log_path` on both files shows no performance work since the original `81263be5 lint: set a timeout in requests.get() (#270)`. No commit/PR resolves it, and415 days of inactivity is not grounds for closing. Impact is moderate (severe build-time regressions only on slow/emulated builders like LP riscv64); complexity is high because the proposed pipelining changes checksum, caching, resume, and error-handling semantics and may need new schema keys.

Evaluated by: mr-cal-bot

Impact: 52 Quick Win: 11 Complexity: 68 Confidence: 84 Actionability: 64

Issue body

### Bug Description When doing `snapcraft pull` in a riscv64 emulated machine (Ubuntu 24.04.2 6.14.0-15-generic) it downloads the part in ~30 mins from the source. We can see that the process is still writing the file after that and it takes ~6 hours to complete it. `tcpdump` produces no output when we finish the download. We can see the file size increasing during that ~6 hours: ```bash $ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz -rw-r--r-- 1 root root 3304247296 Jul 7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz $ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz -rw-r--r-- 1 root root 3306123264 Jul 7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz $ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz -rw-r--r-- 1 root root 3307675648 Jul 7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz $ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz -rw-r--r-- 1 root root 3310784512 Jul 7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz ``` As a workaround I changed `snapcraft.yaml` to use the `nil` plugin with `source: .` and `wget & tar` the file in the `override-build` step. This takes ~1.5h. What is `snapcraft pull` doing, shouldn't it be equivalent? ### To Reproduce To reproduce the behavior you can use a launchpad riscv64 builder from production or staging and trigger a build using the `snapcraft.yaml` below. ### Environment Snapcraft is running in an emulated riscv64 machine using Ubuntu 24.04.2 6.14.0-15-generic. It runs inside an LXC container. This is the setup from launchpad riscv64 builders. ### snapcraft.yaml ```yaml name: llvm-chromium version: "21" summary: Clang used by Google's build of the Chromium web browser source-code: https://code.launchpad.net/~chromium-team/+git/llvm-chromium/+ref/master description: | Chromium and Chromium-ffmpeg snaps build with LLVM, but upstream relies on bleeding edge LLVM, in particular Clang. LLVM from the Ubuntu archive often cannot keep pace, therefore this snap. This snap exposes no commands. To use it, invoke the binaries directly, e.g. /snap/llvm-chromium/current/bin/clang or use the environment, i.e. * CC=/snap/llvm-chromium/current/bin/clang * CXX=/snap/llvm-chromium/current/bin/clang++ * AR=/snap/llvm-chromium/current/bin/llvm-ar * ... confinement: strict base: core24 parts: llvm: plugin: nil # Roll when CLANG_REVISION changes. source: https://commondatastorage.googleapis.com/chromium-browser-official/chromium-135.0.7023.0.tar.xz build-packages: - chrpath - cmake - elfutils - g++ - gperf - ninja-build - pkg-config - python3-pkg-resources - quilt - wget - yasm override-build: | # Find all patches that don't apply anymore before running Quilt, as it # stops at the 1st unappliable and doesn't tells us how the file looks. d=$CRAFT_PROJECT_DIR/patches while IFS= read -r p; do < "$d/$p" patch --dry-run -p1 -o "$d/$p.out" || cat "$d/$p.out" done < "$d/series" QUILT_PATCHES=$d quilt push -a # --pic for building shared library python3 tools/clang/scripts/build.py --skip-checkout --bootstrap \ --disable-asserts --pgo --without-android --without-fuchsia \ --use-system-cmake --with-ml-inliner-model= --pic --no-tools #Restore this when LP:2061075 is fixed #version="$(python3 tools/clang/scripts/update.py --print-revision|head -c31)" ## Trailing 'e' avoids "invalid snap version: must end with an ASCII ## alphanumeric or one of '+' or '~' (and not '-')" #craftctl set version="$version"e #craftctl set version=llvmorg-19-smth-smth arch=$(uname -m) # Save space by removing libraries from other architectures # XXX: bashism shopt -s extglob rm -rf third_party/llvm-build/Release+Asserts/lib/clang/*/lib/!(${arch%?}*gnu*) mkdir -p $CRAFT_PART_INSTALL mv third_party/llvm-build/Release+Asserts/bin $CRAFT_PART_INSTALL mv third_party/llvm-build/Release+Asserts/lib $CRAFT_PART_INSTALL ``` ### Relevant log output ```shell 2025-07-07 09:58:33.506 verify plugin environment for part 'llvm' 2025-07-07 09:58:33.546 Pulling llvm 2025-07-07 09:58:33.565 execute action llvm:Action(part_name='llvm', step=Step.PULL, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None)) 2025-07-07 09:58:35.165 Downloading '/build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz' ``` ### Additional context _No response_

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha
Impact: 52
Quick Win: 11
Complexity: 68
Confidence: 84
Actionability: 64
keep open FileSourceHandler downloads/extracts large sources very slowly (requests+tarfile,1KB chunks, 3600s timeout): ~6h vs ~1.5h with wget on RISC-V. Triaged enhancement, root cause identified, optimizations (piped download+extract, threaded decompression) not yet implemented.
qwen/qwen3.6-35b-a3b
Impact: 70
Quick Win: 34
Complexity: 40
Confidence: 80
Actionability: 80
keep open FileSourceHandler download performance is slow for large sources on RISC-V, taking ~6 hours vs ~1.5h with wget workaround. Issue is triaged and under active discussion.
qwen/qwen3.8-27b
Impact: 45
Quick Win: 20
Staleness: 55
Complexity: 55
Confidence: 78
Support Request: 5
keep open FileSourceHandler download/extraction is CPU-bound and slow on resource-constrained hardware (riscv64). Triaged as enhancement with Jira ticket CRAFT-4689. Discussion identified multi-threaded decompression and piped download as solutions. No fix committed; code still uses requests.get with 1-hour timeout and single-threaded tarfile.
qwen/qwen3.6-35b-a3b
Staleness: 80
Complexity: 65
Confidence: 85
Support Request: 10
keep open Performance issue with FileSourceHandler causing slow downloads and extraction on resource-constrained hardware. Currently under discussion with maintainers exploring multi-threading and piping optimizations. Internal ticket CRAFT-4689 created.
qwen3.6-35b-a3b-mtp-q6
Staleness: 45
Complexity: 75
Confidence: 85
Support Request: 0
keep open Performance enhancement for FileSourceHandler to speed up downloading and extracting large archives on constrained hardware. Currently triaged with an internal Jira ticket and under discussion for parallel processing and multi-threaded decompression.

Update history

No update history recorded yet.

Related issues

No related issues found above the similarity threshold.