← Back to issue list

`charm-python-packages` does not guarantee these packages are installed before packages in `requirements.txt`

View original Github issue

Metadata

Project
charmcraft
Number
#1664
Type
issue
State
closed
Author
DnPlas
Labels
—
Created
Updated
Closed

Current evaluation

Fixed in Charmcraft 3.2 by merged PR #1727 (commit 51a55f73), which installs charm-python-packages before requirements and charm libraries.

Resolution: closed resolved

Reason: PR #1727 removed the single-step non-strict installation and split it into staged installs. The change was merged as commit 51a55f73 and shipped in Charmcraft 3.2.

Evaluated by: mr-cal-bot

No scores available.

Issue body

### Bug Description I have noticed that the list of packages in `charm-python-packages` may not guarantee they will be installed before charm requirements, as the [docs suggest](https://juju.is/docs/sdk/charmcraft-yaml#heading--the-charm-plugin). The behaviour I am getting is that the packages listed in `charm-python-packages` get collected and it looks like they are being installed as the very first thing, for example: ``` :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=anyio,attrs,certifi,charmed-kubeflow-chisme,charset-normalizer,deepdiff,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jinja2,jsonschema,lightkube,lightkube-models,markupsafe,oci-image,ops,ordered-set,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,ruamel-yaml,ruamel-yaml-clib,serialized-data-interface,setuptools,sniffio,tenacity,urllib3,websocket-client,wheel,zipp', '--requirement=requirements.txt', 'pip>23.0', 'setuptools>69.0', 'wheel'] :: :: Collecting pip>23.0 :: :: Using cached pip-24.0.tar.gz (2.1 MB) :: :: Installing build dependencies: started :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing wheel metadata: started :: :: Preparing wheel metadata: finished with status 'done' :: :: Collecting setuptools>69.0 :: :: Using cached setuptools-69.5.1.tar.gz (2.3 MB) :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Installing backend dependencies: started :: :: Installing backend dependencies: finished with status 'done' :: :: Preparing wheel metadata: started :: :: Preparing wheel metadata: finished with status 'done' :: :: Collecting jinja2==3.1.2 :: :: Using cached Jinja2-3.1.2.tar.gz (268 kB) ``` but in reality they get installed after collecting all packages, including the ones listed in `requirements.txt`. ``` # There is a long list of `:: :: Collecting <package-name>...` before the following lines :: :: Successfully built wheel setuptools pip :: :: Installing collected packages: wheel, setuptools, pip :: :: Attempting uninstall: setuptools :: :: Found existing installation: setuptools 44.0.0 :: :: Uninstalling setuptools-44.0.0: :: :: Successfully uninstalled setuptools-44.0.0 :: :: Attempting uninstall: pip :: :: Found existing installation: pip 20.0.2 :: :: Uninstalling pip-20.0.2: :: :: Successfully uninstalled pip-20.0.2 ``` This could result in errors if any of the charm requirements depends on a `charm-python-package` to be installed (or upgraded) a priori. Please NOTE I concluded this was an issue after trying to solve [this other issue](https://github.com/pallets/jinja/issues/1496) with the suggestions from [this post](https://discourse.charmhub.io/t/install-or-update-python-packages-before-packing-a-charm/5158). Upgrading `setuptools` fixes the issue with `markupsafe` and `jinja2`, I tested it in a virtual env, but it was not possible in the charm. ### To Reproduce 1. Take [this charm](https://github.com/canonical/kfp-operators/tree/main/charms/kfp-api). 2. `charmcraft pack -v` 3. Check the build logs and observe when the `setuptools` and `pip` packages get actually installed and upgraded. ### Environment * Ubuntu 20.04 * charmcraft latest/candidate ### charmcraft.yaml ```shell type: "charm" bases: - build-on: - name: "ubuntu" channel: "20.04" run-on: - name: "ubuntu" channel: "20.04" parts: charm: charm-python-packages: [setuptools>69.0, pip>23.0] # Fixes install of some packages ``` ### Relevant log output Logs: https://pastebin.canonical.com/p/DhpBRNn7pz/

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha — closed resolved Fixed in Charmcraft 3.2 by merged PR #1727 (commit 51a55f73), which installs charm-python-packages before requirements and charm libraries.
stealth/space-bunny-alpha — closed resolved Charmcraft now installs charm-python-packages before charm requirements; the fix shipped in 3.2 via #1727 (51a55f73).
qwen3.6-35b-a3b-mtp-q6 — — Fixed in Charmcraft 3.2. Maintainers confirmed the charm-python-packages installation ordering bug was resolved through related PRs and subsequent releases. The issue is closed.
qwen/qwen3.6-35b-a3b — — Closed. Maintainers investigated persistent build failures where charm-python-packages installed after requirements.txt. The issue was closed after a maintainer noted it may have been resolved in Charmcraft 3.2.
qwen/qwen3.6-35b-a3b — — Reported that charm-python-packages install order conflicts with requirements.txt. Multiple PRs attempted fixes but errors persisted. The issue was closed after maintainers confirmed the behavior was corrected in Charmcraft 3.2.
qwen3.6-35b-a3b-mtp-q6 — — charm-python-packages were not guaranteed to install before requirements.txt, causing build failures. The maintainer confirmed the bug was fixed in Charmcraft 3.2.

Update history

No update history recorded yet.

Related work

  • Likely Fixed By: canonical/charmcraft#1727 (confidence 100%)

    Merged fix that changed package installation order and explicitly addressed #1664.

Related issues

Issue Project State Summary Similarity
#696 Allow for modifications to build environment of "charmcraft build" to accomodate for extra prerequisites charmcraft closed Closed as already supported: charm-python-packages installs Python prerequisites before charm requirements. Ordering was later made explicit in Charmcraft 3.2 by PR #1727.
76%
#553 parts,charm_builder: add charm-python-packages plugin property (CRAFT-552) charmcraft merged Merged PR #553 added charm-python-packages, installing listed Python packages before requirements dependencies; implementation landed in commit 4b2d291b.
74%
#470 [REGRESSION] charmcraft not installing dependencies from requirements.txt anymore charmcraft closed Fixed by merged PR #472 (commit 57b74577, "build: include venv in prime if charm-requirements defined"), which restored requirements.txt deps appearing in packed charms; maintainer closed after release in edge.
70%
#342 When installing requirements, pip should avoid using wheels charmcraft closed Fixed in Charmcraft 1.3 by merged PR #541 (commit 0d14d8a2), which disabled binary packages so dependencies are built from source during charm packing.
70%