← Back to issue list

repo: cache stage packages for faster future use.

View original Github issue

Metadata

Project
snapcraft
Number
#1122
Type
pull request
State
merged
Author
kyrofa
Labels
—
Created
Updated
Closed

Current evaluation

Merged PR #1122 (20b422459), reusing cached stage packages and copying the apt cache into cleanbuild containers to fix LP #1661383.

Resolution: closed resolved

Reason: PR #1122 was merged as commit 20b422459. It now consumes the stage-package cache, fetches packages with Version.fetch_binary(), and makes the cache available to cleanbuild.

Evaluated by: mr-cal-bot

No scores available.

Issue body

Snapcraft currently caches stage packages, but doesn't actually use them. This PR fixes LP: [#1661383](https://bugs.launchpad.net/snapcraft/+bug/1661383) by using them, and making sure cleanbuild can use the cache as well. The way this ended up being done isn't ideal. Since `apt.Cache().fetch_archives()` seems to mangle (some) package names for no apparent reason, it's impossible to fetch stage packages with it and later determine which packages were pulled as the filenames on disk don't match up to the filenames in the archive. The settled-upon solution was to pull stage-packages using `apt.package.Version().fetch_binary()`. This works well and actually simplifies the caching mechanism, but has a few drawbacks: - It prints directly to stdout when a package is already cached. - Fetching stage packages one-by-one means one loses a sense of how long the overall process will actually take. Note also that this PR **copies** the cache into the cleanbuild container as opposed to bind-mounting read-only. This was done so the caching code was more robust. Otherwise it would need to account for being unable to do the following things: - Being unable to clean the cache when sources changes - Being unable to update the index - Being unable to fetch packages And have backup plans. Such code would be complex and difficult to test, and it seemed it wasn't worth it for a relatively small efficiency gain, particularly considering that ideally the cache would be editable from the container anyway, that just doesn't seem possible today without modifications to /`etc/subgid`. Of course this is up for debate!

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha — closed resolved Merged PR #1122 (20b422459), reusing cached stage packages and copying the apt cache into cleanbuild containers to fix LP #1661383.
stealth/space-bunny-alpha — closed resolved Merged PR #1122 (commit 20b422459), fixing LP bug #1661383 by reusing cached stage packages and copying the cache into cleanbuild containers.
qwen/qwen3.6-35b-a3b — — Merged to fix Snapcraft ignoring cached stage packages. The implementation uses fetch_binary for reliable retrieval and copies the cache into cleanbuild containers for robustness. Approved by two reviewers.
qwen3.6-35b-a3b-mtp-q6 — — Merged to resolve LP #1661383, enabling Snapcraft to utilize cached stage packages for faster builds. The implementation uses apt.package.Version().fetch_binary() and copies the cache into cleanbuild containers for robustness.

Update history

No update history recorded yet.

Related work

  • Related To: canonical/snapcraft#1216 (confidence 98%)

    Later changed cleanbuild to avoid copying the cache into the container.

Related issues

Issue Project State Summary Similarity
#1113 repo: cache stage packages for faster future use. snapcraft closed Closed unmerged in favor of #1122, which merged the stage-package caching and cleanbuild improvements as commit 20b422459.
87%
#1661383 stage package cache no longer working snapcraft (launchpad) closed Stage-package cache reuse was restored: snapcraft PR #1122 (commit 20b422459, 'repo: cache stage packages for faster future use') merged, reusing cached debs and copying the apt cache into cleanbuild containers, closing LP #1661383.
79%
#3110 repo: use persistent cache for fetching stage packages snapcraft closed PR #3110 (persistent cache for fetching stage packages) was closed unmerged with 0 reviews and 1 open maintainer question. Its dependency #3109 also closed unmerged; the feature was later delivered by merged PR #3585 (CRAFT-488) using host /var/lib/apt state.
76%
#3030 repo: drop _AptCache and add migrate to install_stage_packages() snapcraft merged Merged in PR #3030 as commit 66467629d, replacing custom _AptCache handling with install_stage_packages() and updating package staging, tests, and ROS plugins.
72%
#3086 repo: restore marked-install strategy for apt-cache snapcraft merged Merged PR #3086 (17cc75aae), restoring AptCache marked installs for build and stage packages and fixing primed-stage-packages name/version parsing from ':' to '='.
72%
#118 packages/apt: use host state for apt cache (CRAFT-487) craft-parts merged Merged PR #118 unified stage and host APT caches using /var/lib/apt, removed the update interface, and made stage package-list refresh a no-op.
72%
#2119 repo: automatically prune unneeded stage-packages snapcraft merged Merged as b9faf80bd, adding APT cache analysis to remove automatically installed stage-packages left unneeded after alternative dependency resolution.
72%
#3585 repo: use host state for apt cache (CRAFT-488) snapcraft merged Merged #3585 via f817e34f2 (merge 0ae235a09), replacing split stage APT state with the host cache at /var/lib/apt for CRAFT-488.
70%