← Back to issue list

Unwanted source files copy in `build`

View original Github issue

Metadata

Project
craft-parts
Number
#1609
Type
issue
State
open
Author
imatrisciano
Labels
—
Created
Updated
Closed
—

Current evaluation

nil/dump plugins copy the full source tree into build/, tripling disk use for large assets (100GB model → 300GB). Reproducer provided; maintainer proposed a new `static-assets` plugin but no implementation exists yet.

Suggested action: keep open

Reason: The report is confirmed in code: craft_parts/executor/part_handler.py:469 still does `shutil.copytree(src, build, ...)` whenever `not update and not self._plugin.get_out_of_source_build()`, and neither nil nor dump overrides that (dump_plugin.py only defines get_build_commands). mr-cal engaged twice, explicitly validating the problem ('300GB of disk use across 3 copies of the model for a 100 GB model'), and the team converged on a `static-assets` plugin; the reporter agreed and added a blocker (LXD cannot hardlink across devices), so the discussion is live and design is still being nailed down. grep for 'static-assets'/'static_assets' and git log for 'hardlink'/'asset' show no implementing commit or PR, so it cannot be closed as resolved — the only blocker is unfinished design, not obsolescence.

Evaluated by: mr-cal-bot

Impact: 55 Quick Win: 21 Complexity: 45 Confidence: 80 Actionability: 70

Issue body

### Bug Description At the beginning of the **build** stage, a part's source files are copied into the `build` directory (see the [relevant source code](https://github.com/canonical/craft-parts/blob/139cbce69c1ef9e62c0ecf36b7567061299f3444/craft_parts/executor/part_handler.py#L463)). This happens if the build is not an update and if the plugin is not **out-of-source**. Out-of-source plugins are those which are known to not modify source files during build (e.g. `cmake`, `qmake`, `go-use`, `meson`). The issue here is that the `nil` plugin is supposed to be an empty plugin, where every instruction is explicitly set by the developer. Instead, if the `nil` plugin is used, the copy operation mentioned above is performed anyway as part of the **build** stage, and there is nothing a developer can do to prevent that. This is an issue if the project packs large files, like [inference snaps](https://documentation.ubuntu.com/inference-snaps/) do when packing larger models. This behavior is unexpected and can leave a developer wondering why their build takes more space than anticipated. The same could be said about the `dump` plugin, whose purpose is to move files from **src** to **install**, there's arguably no need for it to copy source files into **build**. ### Workaround A **workaround** for this issue is to use an **out-of-source** plugin (such as `cmake`) so the copy in **build** is skipped at the cost of confusion in the _snapcraft.yaml_ file, and unneeded build dependencies. Dropping the `source` keyword in the part description also prevents the copy from happening, but will also prevent the build system from noticing if the files on disk change to trigger an update. ### To Reproduce Setup an example project with a 1 GiB data file: ```bash mkdir -p ~/source/craft_size_playground cd ~/source/craft_size_playground mkdir data # Create large file (1 GiB) dd if=/dev/random of=data/my_file bs=1G count=1 mkdir snap touch snap/snapcraft.yaml ``` Use the following as your snapcraft.yaml: ```yaml name: craft-size-playground base: core24 version: '0.1' summary: Size test snap description: | Test snap. grade: devel confinement: strict compression: lzo parts: my-file: plugin: nil source: data override-build: | cp -rf --archive --link --no-dereference ${CRAFT_PART_SRC}/* ${CRAFT_PART_INSTALL} organize: my_file: (component/my-component) components: my-component: type: standard summary: Test component description: Just a file size test ``` Then, pack it with `snapcraft pack`. If doing multiple tests, make sure to `snapcraft clean` in between. Finally, check the build environment disk usage: ```bash sudo su cd /var/snap/lxd/common/lxd/storage-pools/default/containers/snapcraft_snapcraft-craft-size-playground-amd64-*/rootfs/root/parts du . -d 3 -h ``` The disk usage report will look like this: ``` 28K ./my-file/state 4.0K ./my-file/install 4.0K ./my-file/export 1.1G ./my-file/src 4.0K ./my-file/layer 1.1G ./my-file/build 4.0K ./my-file/run 2.1G ./my-file 2.1G . ``` The report above shows that the large file was copied two times: once in the **src** directory and another time in the **build** directory. The first copy is expected, because we are not using destructive build, and an hard-link to **src** from the project directory is not possible because it would be a cross-device link; but the second copy in **build** is unexpected and, in this example, unwanted. ### part yaml ```shell ``` ### Relevant log output ```shell Starting snapcraft, version 9.0.0.post19+dirty Logging execution to '/home/ubuntu/.local/state/snapcraft/log/snapcraft-20260605-161006.529105.log' Launching managed ubuntu 24.04 instance... Creating instance from base instance Starting instance Starting snapcraft, version 9.0.0.post19+dirty Logging execution to '/tmp/snapcraft.log' Initializing lifecycle Installing build-packages Installing build-snaps Pulling my-file Building my-file :: + cp -rf --archive --link --no-dereference /root/parts/my-file/src/my_file /root/parts/my-file/install/ Staging my-file Priming my-file Packing... Reading snap metadata... Running linters... Running linter: classic Running linter: gpu Running linter: library Running linter: metadata Creating snap package... Packed: craft-size-playground_0.1_amd64.snap, craft-size-playground+my-component.comp ```

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha
Impact: 55
Quick Win: 21
Complexity: 45
Confidence: 80
Actionability: 70
keep open nil/dump plugins copy the full source tree into build/, tripling disk use for large assets (100GB model → 300GB). Reproducer provided; maintainer proposed a new `static-assets` plugin but no implementation exists yet.
stealth/space-bunny-alpha
Impact: 60
Quick Win: 24
Complexity: 48
Confidence: 94
Actionability: 76
keep open nil and dump still copy large sources into build, potentially requiring 300 GB for a 100 GB model. A static-assets plugin was proposed in discussion, but no implementation is present.
qwen3.6-35b-a3b-mtp-q6
Impact: 75
Quick Win: 34
Staleness: 35
Complexity: 55
Confidence: 75
Support Request: 10
keep open nil plugin copies source files into build directory unnecessarily, wasting disk space for large files. Maintainer proposed static-assets plugin or hardlink-source attribute; neither implemented yet.
qwen/qwen3.6-35b-a3b
Staleness: 10
Complexity: 40
Confidence: 90
Support Request: 0
keep open Bug report: unnecessary source file copying in build directory with nil plugin, causing high disk usage. Actively discussed with a proposed new plugin solution.
qwen3.6-35b-a3b-mtp-q6
Staleness: 20
Complexity: 65
Confidence: 90
Support Request: 10
keep open Maintainer proposed a new static-assets plugin to avoid copying large source files into the build directory when using nil or dump plugins. Currently under discussion with the author regarding LXD hardlink limitations.

Update history

No update history recorded yet.

Related issues

No related issues found above the similarity threshold.