← Back to issue list

When cross-compiling incorrect architecture `stage-packages` are fetched

View original Github issue

Metadata

Project
snapcraft
Number
#4542
Type
issue
State
closed
Author
dilyn-corner
Labels
—
Created
Updated
Closed

Current evaluation

Closed without resolution. Internal ticket CRAFT-2620 tracks the bug where cross-compilation fetches host architecture stage-packages instead of the target. Manual repository configuration and explicit architecture variables work around it.

Suggested action: —

No scores available.

Issue body

### Bug Description When building a snap on an AMD64 machine for an ARM64 device, it was noticed by the library linter that a bunch of libraries and other files were unused -- oddly enough, they were located in `/usr/lib/x86_64-linxu-gnu` instead of the anticipated ARM64 triple. This bug has been noticed on several different snaps. I have included references to those in the additional context section. ### To Reproduce ```shell git clone --branch core22-bump -d 1 https://github.com/canonical/iot-field-example-snaps cd iot-field-example-snaps/qt-imx snapcraft --build-for=arm64 --use-lxd ``` ### Environment Ubuntu 23.10 running snapcraft rev. 10445 with `--use-lxd` ### snapcraft.yaml See the repository specified in the reproduction steps. ### Relevant log output See #4541 and add the below to the netplan `snapcraft.yaml`: ```shell architectures: - build-on: amd64 build-for: arm64 ``` Do the build (it may fail, that's fine) and see what packages are in `parts/netplan/stage_packages`; below are the details (and some extra steps I tried to remediate, to non avail). <details> <summary>Building netplan for arm64</summary> <br> ```shell root@testing-vm:~/netplan# snapcraft --destructive-mode --build-for=arm64 --verbosity=trace 2024-01-24 21:06:40.982 Starting Snapcraft 8.0.2.post11+gitbbf8ae08 2024-01-24 21:06:40.982 Logging execution to '/root/.local/state/snapcraft/log/snapcraft-20240124-210640.975513.log' 2024-01-24 21:06:40.982 Raw pre-parsed sysargs: args={'help': False, 'verbose': False, 'quiet': False, 'verbosity': 'trace', 'version': False, 'trace': False} filtered=['--destructive-mode', '--build-for=arm64', '--enable-manifest'] 2024-01-24 21:06:40.983 Using default command: 'pack' 2024-01-24 21:06:40.983 General parsed sysargs: command='pack' args=['--destructive-mode', '--build-for=arm64', '--enable-manifest'] 2024-01-24 21:06:40.984 Command parsed sysargs: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:06:40.984 lifecycle command: 'pack', arguments: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:06:40.984 command: pack, arguments: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:06:41.002 Created build plan: build-on: amd64 build-for: arm64 2024-01-24 21:06:41.003 CPU count (from process affinity): 1 2024-01-24 21:06:41.003 Invalid SNAPCRAFT_MAX_PARALLEL_BUILD_COUNT '' 2024-01-24 21:06:41.003 Running on amd64 for arm64 2024-01-24 21:06:41.005 Initializing parts lifecycle 2024-01-24 21:06:41.005 is_snap: True, SNAP_NAME set to snapcraft 2024-01-24 21:06:41.006 process netplan:Step.PULL 2024-01-24 21:06:41.006 add action netplan:Step.PULL(ActionType.RUN) 2024-01-24 21:06:41.010 process netplan:Step.BUILD 2024-01-24 21:06:41.010 add action netplan:Step.BUILD(ActionType.RUN) 2024-01-24 21:06:41.011 process netplan:Step.STAGE 2024-01-24 21:06:41.014 add action netplan:Step.STAGE(ActionType.RUN) 2024-01-24 21:06:41.015 process netplan:Step.PRIME 2024-01-24 21:06:41.015 add action netplan:Step.PRIME(ActionType.RUN) 2024-01-24 21:06:41.015 Installing build-packages 2024-01-24 21:06:41.016 Installing build-snaps 2024-01-24 21:06:41.579 verify plugin environment for part 'netplan' 2024-01-24 21:06:41.581 Pulling netplan 2024-01-24 21:06:41.590 execute action netplan:Action(part_name='netplan', step=Step.PULL, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None)) 2024-01-24 21:06:41.591 Fetching stage-packages 2024-01-24 21:06:41.591 Requested stage-packages: ['libnetplan0:arm64', 'libpython3-stdlib:arm64', 'libpython3.11-minimal:arm64', 'libpython3.11-stdlib:arm64', 'netplan.io:arm64', 'python3-dbus:arm64', 'python3-minimal:arm64', 'python3-netifaces:arm64', 'python3-yaml:arm64', 'python3.11-minimal:arm64', 'python3:arm64'] 2024-01-24 21:06:41.691 Executing: ['apt-get', 'update'] 2024-01-24 21:06:42.088 :: Hit:1 http://archive.ubuntu.com/ubuntu mantic InRelease 2024-01-24 21:06:42.094 :: Hit:2 http://archive.ubuntu.com/ubuntu mantic-updates InRelease 2024-01-24 21:06:42.114 :: Hit:3 http://archive.ubuntu.com/ubuntu mantic-backports InRelease 2024-01-24 21:06:42.235 :: Hit:4 http://security.ubuntu.com/ubuntu mantic-security InRelease 2024-01-24 21:06:45.560 :: Reading package lists... 2024-01-24 21:06:46.370 Marking libpython3.11-stdlib:arm64 (and its dependencies) to be fetched 2024-01-24 21:06:46.374 Stage package not found in part 'netplan': libpython3.11-stdlib:arm64. 2024-01-24 21:06:46.384 Traceback (most recent call last): 2024-01-24 21:06:46.384 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 897, in _fetch_stage_packages 2024-01-24 21:06:46.385 fetched_packages = packages.Repository.fetch_stage_packages( 2024-01-24 21:06:46.385 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 602, in fetch_stage_packages 2024-01-24 21:06:46.385 return cls._fetch_stage_debs( # type: ignore[no-any-return] 2024-01-24 21:06:46.385 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 262, in wrapped 2024-01-24 21:06:46.385 return method(*args, **kwargs) 2024-01-24 21:06:46.385 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 647, in _fetch_stage_debs 2024-01-24 21:06:46.386 apt_cache.mark_packages(set(package_names)) 2024-01-24 21:06:46.386 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/apt_cache.py", line 309, in mark_packages 2024-01-24 21:06:46.386 raise errors.PackageNotFound(name_arch) 2024-01-24 21:06:46.386 craft_parts.packages.errors.PackageNotFound: Package not found: libpython3.11-stdlib:arm64. 2024-01-24 21:06:46.386 2024-01-24 21:06:46.390 During handling of the above exception, another exception occurred: 2024-01-24 21:06:46.390 Traceback (most recent call last): 2024-01-24 21:06:46.391 File "/snap/snapcraft/10445/lib/python3.10/site-packages/snapcraft/parts/parts.py", line 190, in run 2024-01-24 21:06:46.391 aex.execute(action, stdout=stream, stderr=stream) 2024-01-24 21:06:46.391 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 318, in execute 2024-01-24 21:06:46.391 self._executor.execute(actions, stdout=stdout, stderr=stderr) 2024-01-24 21:06:46.391 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 137, in execute 2024-01-24 21:06:46.392 self._run_action(act, stdout=stdout, stderr=stderr) 2024-01-24 21:06:46.392 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 205, in _run_action 2024-01-24 21:06:46.392 handler.run_action(action, stdout=stdout, stderr=stderr) 2024-01-24 21:06:46.392 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 172, in run_action 2024-01-24 21:06:46.392 state = handler(step_info, stdout=stdout, stderr=stderr) 2024-01-24 21:06:46.393 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 194, in _run_pull 2024-01-24 21:06:46.393 fetched_packages = self._fetch_stage_packages(step_info=step_info) 2024-01-24 21:06:46.393 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 905, in _fetch_stage_packages 2024-01-24 21:06:46.393 raise errors.StagePackageNotFound( 2024-01-24 21:06:46.394 craft_parts.errors.StagePackageNotFound: Stage package not found in part 'netplan': libpython3.11-stdlib:arm64. 2024-01-24 21:06:46.394 Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240124-210640.975513.log' ``` </details> Specifying all stage-packages as `:${CRAFT_ARCH_BUILD_FOR}`, adding the ports repositories for arm64 to `/etc/apt/sources.list`, and rebuilding: <details> <summary>Doing some extra work when building netplan for arm64</summary> <br> ```shell root@testing-vm:~/netplan# snapcraft --destructive-mode --build-for=arm64 --enable-manifest --verbosity=trace 2024-01-24 21:09:02.870 Starting Snapcraft 8.0.2.post11+gitbbf8ae08 2024-01-24 21:09:02.870 Logging execution to '/root/.local/state/snapcraft/log/snapcraft-20240124-210902.863075.log' 2024-01-24 21:09:02.871 Raw pre-parsed sysargs: args={'help': False, 'verbose': False, 'quiet': False, 'verbosity': 'trace', 'version': False, 'trace': False} filtered=['--destructive-mode', '--build-for=arm64', '--enable-manifest'] 2024-01-24 21:09:02.871 Using default command: 'pack' 2024-01-24 21:09:02.871 General parsed sysargs: command='pack' args=['--destructive-mode', '--build-for=arm64', '--enable-manifest'] 2024-01-24 21:09:02.872 Command parsed sysargs: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:09:02.872 lifecycle command: 'pack', arguments: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:09:02.872 command: pack, arguments: Namespace(destructive_mode=True, use_lxd=False, debug=False, enable_manifest=True, manifest_image_information=None, bind_ssh=False, build_for='arm64', http_proxy=None, https_proxy=None, ua_token=None, enable_experimental_ua_services=False, enable_experimental_plugins=False, enable_experimental_extensions=False, enable_developer_debug=False, enable_experimental_target_arch=False, target_arch=None, provider=None, directory=None, output=None) 2024-01-24 21:09:02.902 Created build plan: build-on: amd64 build-for: arm64 2024-01-24 21:09:02.903 CPU count (from process affinity): 1 2024-01-24 21:09:02.903 Invalid SNAPCRAFT_MAX_PARALLEL_BUILD_COUNT '' 2024-01-24 21:09:02.903 Running on amd64 for arm64 2024-01-24 21:09:02.905 Initializing parts lifecycle 2024-01-24 21:09:02.905 is_snap: True, SNAP_NAME set to snapcraft 2024-01-24 21:09:02.906 process netplan:Step.PULL 2024-01-24 21:09:02.907 add action netplan:Step.PULL(ActionType.RUN) 2024-01-24 21:09:02.907 process netplan:Step.BUILD 2024-01-24 21:09:02.907 add action netplan:Step.BUILD(ActionType.RUN) 2024-01-24 21:09:02.908 process netplan:Step.STAGE 2024-01-24 21:09:02.908 add action netplan:Step.STAGE(ActionType.RUN) 2024-01-24 21:09:02.908 process netplan:Step.PRIME 2024-01-24 21:09:02.910 add action netplan:Step.PRIME(ActionType.RUN) 2024-01-24 21:09:02.911 Installing build-packages 2024-01-24 21:09:02.911 Installing build-snaps 2024-01-24 21:09:03.470 verify plugin environment for part 'netplan' 2024-01-24 21:09:03.470 Pulling netplan 2024-01-24 21:09:03.478 execute action netplan:Action(part_name='netplan', step=Step.PULL, action_type=ActionType.RUN, reason=None, project_vars=None, properties=ActionProperties(changed_files=None, changed_dirs=None)) 2024-01-24 21:09:03.478 remove directory /root/netplan/parts/netplan/src 2024-01-24 21:09:03.479 Fetching stage-packages 2024-01-24 21:09:03.479 Requested stage-packages: ['libnetplan0:arm64', 'libpython3-stdlib:arm64', 'libpython3.11-minimal:arm64', 'libpython3.11-stdlib:arm64', 'netplan.io:arm64', 'python3-dbus:arm64', 'python3-minimal:arm64', 'python3-netifaces:arm64', 'python3-yaml:arm64', 'python3.11-minimal:arm64', 'python3:arm64'] 2024-01-24 21:09:03.581 Executing: ['apt-get', 'update'] 2024-01-24 21:09:04.160 :: Hit:1 http://archive.ubuntu.com/ubuntu mantic InRelease 2024-01-24 21:09:04.160 :: Hit:2 http://security.ubuntu.com/ubuntu mantic-security InRelease 2024-01-24 21:09:04.178 :: Hit:3 http://ports.ubuntu.com/ubuntu-ports mantic InRelease 2024-01-24 21:09:04.225 :: Hit:4 http://archive.ubuntu.com/ubuntu mantic-updates InRelease 2024-01-24 21:09:04.275 :: Hit:5 http://ports.ubuntu.com/ubuntu-ports mantic-updates InRelease 2024-01-24 21:09:04.322 :: Hit:6 http://archive.ubuntu.com/ubuntu mantic-backports InRelease 2024-01-24 21:09:04.369 :: Hit:7 http://ports.ubuntu.com/ubuntu-ports mantic-backports InRelease 2024-01-24 21:09:04.462 :: Hit:8 http://ports.ubuntu.com/ubuntu-ports mantic-security InRelease 2024-01-24 21:09:13.174 :: Reading package lists... 2024-01-24 21:09:22.516 Marking libpython3.11-minimal:arm64 (and its dependencies) to be fetched 2024-01-24 21:09:22.518 package: libpython3.11-minimal:arm64 2024-01-24 21:09:22.998 Marking libpython3.11-stdlib:arm64 (and its dependencies) to be fetched 2024-01-24 21:09:22.999 package: libpython3.11-stdlib:arm64 2024-01-24 21:09:23.510 Marking netplan.io:arm64 (and its dependencies) to be fetched 2024-01-24 21:09:23.511 package: netplan.io:arm64 2024-01-24 21:09:23.714 Stage package not found in part 'netplan': netplan.io:arm64. 2024-01-24 21:09:23.720 Traceback (most recent call last): 2024-01-24 21:09:23.725 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 897, in _fetch_stage_packages 2024-01-24 21:09:23.726 fetched_packages = packages.Repository.fetch_stage_packages( 2024-01-24 21:09:23.726 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 602, in fetch_stage_packages 2024-01-24 21:09:23.726 return cls._fetch_stage_debs( # type: ignore[no-any-return] 2024-01-24 21:09:23.726 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 262, in wrapped 2024-01-24 21:09:23.726 return method(*args, **kwargs) 2024-01-24 21:09:23.726 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/deb.py", line 647, in _fetch_stage_debs 2024-01-24 21:09:23.726 apt_cache.mark_packages(set(package_names)) 2024-01-24 21:09:23.727 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/apt_cache.py", line 330, in mark_packages 2024-01-24 21:09:23.727 _verify_marked_install(package) 2024-01-24 21:09:23.727 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/packages/apt_cache.py", line 377, in _verify_marked_install 2024-01-24 21:09:23.727 raise errors.PackageNotFound(package.name) 2024-01-24 21:09:23.727 craft_parts.packages.errors.PackageNotFound: Package not found: netplan.io:arm64. 2024-01-24 21:09:23.727 2024-01-24 21:09:23.727 During handling of the above exception, another exception occurred: 2024-01-24 21:09:23.727 Traceback (most recent call last): 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/snapcraft/parts/parts.py", line 190, in run 2024-01-24 21:09:23.728 aex.execute(action, stdout=stream, stderr=stream) 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 318, in execute 2024-01-24 21:09:23.728 self._executor.execute(actions, stdout=stdout, stderr=stderr) 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 137, in execute 2024-01-24 21:09:23.728 self._run_action(act, stdout=stdout, stderr=stderr) 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/executor.py", line 205, in _run_action 2024-01-24 21:09:23.728 handler.run_action(action, stdout=stdout, stderr=stderr) 2024-01-24 21:09:23.728 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 172, in run_action 2024-01-24 21:09:23.729 state = handler(step_info, stdout=stdout, stderr=stderr) 2024-01-24 21:09:23.729 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 194, in _run_pull 2024-01-24 21:09:23.729 fetched_packages = self._fetch_stage_packages(step_info=step_info) 2024-01-24 21:09:23.734 File "/snap/snapcraft/10445/lib/python3.10/site-packages/craft_parts/executor/part_handler.py", line 905, in _fetch_stage_packages 2024-01-24 21:09:23.734 raise errors.StagePackageNotFound( 2024-01-24 21:09:23.734 craft_parts.errors.StagePackageNotFound: Stage package not found in part 'netplan': netplan.io:arm64. 2024-01-24 21:09:23.734 Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240124-210902.863075.log' ``` </details> ### Additional context This has been discussed for a long while, I have two reproduction cases and some chat history to share. An example and some additional information about what I've discovered thus far can be found in [this PR](https://github.com/canonical/iot-field-example-snaps/pull/10) -- to replicate, build the qt-imx snap on the core22-bump branch. I have additionally attached a `--verbosity=trace` log of an example build of this snap to this issue. Given that log file, I suspect that the bug is actually more a problem with apt than with snapcraft. However, I am not certain. [snapcraft-20231213-114904.997147.log.tar.gz](https://github.com/snapcore/snapcraft/files/14043395/snapcraft-20231213-114904.997147.log.tar.gz) There was an earlier discussion of this issue on the Charmhub snapcraft channel [here](https://chat.charmhub.io/charmhub/pl/9a1i7wmputfkmpaj997awyqxic). There are also the following bug reports on Launchpad (since 2020): [1881515](https://bugs.launchpad.net/snapcraft/+bug/1881515) [1921418](https://bugs.launchpad.net/snapcraft/+bug/1921418) [2012503](https://bugs.launchpad.net/snapcraft/+bug/2012503) [2018609](https://bugs.launchpad.net/launchpad/+bug/2018609/comments/5) For another example where this was noticed, see issue #4541

Evaluation history

Date Model Scores Action Summary
qwen/qwen3.6-35b-a3b — — Closed without resolution. Internal ticket CRAFT-2620 tracks the bug where cross-compilation fetches host architecture stage-packages instead of the target. Manual repository configuration and explicit architecture variables work around it.
qwen/qwen3.6-35b-a3b — — Closed. Root cause identified in craft-parts where cross-compilation incorrectly fetched host-architecture stage-packages. Issue tracked internally via Jira ticket CRAFT-2620 for a target architecture repository fix.
qwen/qwen3.6-35b-a3b — — Closed without a confirmed fix. Discussion identified a craft-parts adjustment to route stage-packages to the target architecture during cross-compilation, but no resolution or merge was documented.
qwen3.6-35b-a3b-mtp-q6 — — Closed without resolution. Cross-compilation fetches host architecture stage-packages instead of target. Tracking moved to internal Jira ticket CRAFT-2620. No fix or merge provided.
qwen3.6-35b-a3b-mtp-q6 — — Closed and deferred to internal ticket CRAFT-2620. No fix provided; users must manually specify target architecture in stage-packages and configure package-repositories to avoid fetching host-architecture libraries during cross-compilation.

Update history

No update history recorded yet.

Related issues

Issue Project State Summary Similarity
#2012503 wrong arch of stage packages snapcraft (launchpad) open Snapcraft 7.3 picks host architecture for stage packages when cross-building (build-on amd64, build-for arm64). No maintainer response in 3.4 years. Nearly identical issue #5335 was closed as expected behavior requiring explicit :$CRAFT_ARCH_BUILD_FOR suffix on stage packages.
73%
#258 packages: fetch stage-packages from host arch's repo craft-parts merged Merged to resolve cross-compilation failures by fetching stage-packages from the host architecture repository. Approved, passed CI, and closes LP #1983009 and CRAFT-1216.
73%
#1737561 stage-packages do not honor --target-arch snapcraft (launchpad) open Cross-building stage-packages pulls host architecture packages instead of target architecture. Issue is 8.7 years old with no maintainer interaction or comments. Old cross-build mode deprecated in favor of core24/craft-parts architecture.
72%
#1660903 using an invalid architecture in stage-packages causes a mess snapcraft (launchpad) closed Closed. The issue was abandoned due to lack of comments or resolution details.
71%