← Back to issue list

FileSourceHandler is slow for large sources

View original Github issue

Metadata

Project
craft-parts
Number
#1175
Type
issue
State
open
Author
enriqueesanchz
Labels
Status: Triaged Type: Enhancement
Created
2025-07-11 16:09:32+00:00
Updated
2025-08-13 16:17:01+00:00
Closed

Current evaluation

No evaluation has been recorded for this issue yet.

Issue body

### Bug Description When doing `snapcraft pull` in a riscv64 emulated machine (Ubuntu 24.04.2 6.14.0-15-generic) it downloads the part in ~30 mins from the source. We can see that the process is still writing the file after that and it takes ~6 hours to complete it. `tcpdump` produces no output when we finish the download. We can see the file size increasing during that ~6 hours: ```bash $ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz -rw-r--r-- 1 root root 3304247296 Jul 7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz $ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz -rw-r--r-- 1 root root 3306123264 Jul 7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz $ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz -rw-r--r-- 1 root root 3307675648 Jul 7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz $ ls -la /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz -rw-r--r-- 1 root root 3310784512 Jul 7 10:46 /build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz ``` As a workaround I changed `snapcraft.yaml` to use the `nil` plugin with `source: .` and `wget & tar` the file in the `override-build` step. This takes ~1.5h. What is `snapcraft pull` doing, shouldn't it be equivalent? ### To Reproduce To reproduce the behavior you can use a launchpad riscv64 builder from production or staging and trigger a build using the `snapcraft.yaml` below. ### Environment Snapcraft is running in an emulated riscv64 machine using Ubuntu 24.04.2 6.14.0-15-generic. It runs inside an LXC container. This is the setup from launchpad riscv64 builders. ### snapcraft.yaml ```yaml name: llvm-chromium version: "21" summary: Clang used by Google's build of the Chromium web browser source-code: https://code.launchpad.net/~chromium-team/+git/llvm-chromium/+ref/master description: | Chromium and Chromium-ffmpeg snaps build with LLVM, but upstream relies on bleeding edge LLVM, in particular Clang. LLVM from the Ubuntu archive often cannot keep pace, therefore this snap. This snap exposes no commands. To use it, invoke the binaries directly, e.g. /snap/llvm-chromium/current/bin/clang or use the environment, i.e. * CC=/snap/llvm-chromium/current/bin/clang * CXX=/snap/llvm-chromium/current/bin/clang++ * AR=/snap/llvm-chromium/current/bin/llvm-ar * ... confinement: strict base: core24 parts: llvm: plugin: nil # Roll when CLANG_REVISION changes. source: https://commondatastorage.googleapis.com/chromium-browser-official/chromium-135.0.7023.0.tar.xz build-packages: - chrpath - cmake - elfutils - g++ - gperf - ninja-build - pkg-config - python3-pkg-resources - quilt - wget - yasm override-build: | # Find all patches that don't apply anymore before running Quilt, as it # stops at the 1st unappliable and doesn't tells us how the file looks. d=$CRAFT_PROJECT_DIR/patches while IFS= read -r p; do < "$d/$p" patch --dry-run -p1 -o "$d/$p.out" || cat "$d/$p.out" done < "$d/series" QUILT_PATCHES=$d quilt push -a # --pic for building shared library python3 tools/clang/scripts/build.py --skip-checkout --bootstrap \ --disable-asserts --pgo --without-android --without-fuchsia \ --use-system-cmake --with-ml-inliner-model= --pic --no-tools #Restore this when LP:2061075 is fixed #version="$(python3 tools/clang/scripts/update.py --print-revision|head -c31)" ## Trailing 'e' avoids "invalid snap version: must end with an ASCII ## alphanumeric or one of '+' or '~' (and not '-')" #craftctl set version="$version"e #craftctl set version=llvmorg-19-smth-smth arch=$(uname -m) # Save space by removing libraries from other architectures # XXX: bashism shopt -s extglob rm -rf third_party/llvm-build/Release+Asserts/lib/clang/*/lib/!(${arch%?}*gnu*) mkdir -p $CRAFT_PART_INSTALL mv third_party/llvm-build/Release+Asserts/bin $CRAFT_PART_INSTALL mv third_party/llvm-build/Release+Asserts/lib $CRAFT_PART_INSTALL ``` ### Relevant log output ```shell 2025-07-07 09:58:33.506 verify plugin environment for part 'llvm' 2025-07-07 09:58:33.546 Pulling llvm 2025-07-07 09:58:33.565 execute action llvm:Action(part_name='llvm', step=Step.PULL, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None)) 2025-07-07 09:58:35.165 Downloading '/build/llvm-chromium/parts/llvm/src/chromium-135.0.7023.0.tar.xz' ``` ### Additional context _No response_

Evaluation history

No evaluation history available.