← Back to issue list

`charm-python-packages` does not guarantee these packages are installed before packages in `requirements.txt`

View original Github issue

Metadata

Project
charmcraft
Number
#1664
Type
issue
State
closed
Author
DnPlas
Labels
Created
Updated
Closed

Current evaluation

Fixed in Charmcraft 3.2. Maintainers confirmed the charm-python-packages installation ordering bug was resolved through related PRs and subsequent releases. The issue is closed.

Suggested action:

No scores available.

Issue body

### Bug Description I have noticed that the list of packages in `charm-python-packages` may not guarantee they will be installed before charm requirements, as the [docs suggest](https://juju.is/docs/sdk/charmcraft-yaml#heading--the-charm-plugin). The behaviour I am getting is that the packages listed in `charm-python-packages` get collected and it looks like they are being installed as the very first thing, for example: ``` :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=anyio,attrs,certifi,charmed-kubeflow-chisme,charset-normalizer,deepdiff,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jinja2,jsonschema,lightkube,lightkube-models,markupsafe,oci-image,ops,ordered-set,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,ruamel-yaml,ruamel-yaml-clib,serialized-data-interface,setuptools,sniffio,tenacity,urllib3,websocket-client,wheel,zipp', '--requirement=requirements.txt', 'pip>23.0', 'setuptools>69.0', 'wheel'] :: :: Collecting pip>23.0 :: :: Using cached pip-24.0.tar.gz (2.1 MB) :: :: Installing build dependencies: started :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing wheel metadata: started :: :: Preparing wheel metadata: finished with status 'done' :: :: Collecting setuptools>69.0 :: :: Using cached setuptools-69.5.1.tar.gz (2.3 MB) :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Installing backend dependencies: started :: :: Installing backend dependencies: finished with status 'done' :: :: Preparing wheel metadata: started :: :: Preparing wheel metadata: finished with status 'done' :: :: Collecting jinja2==3.1.2 :: :: Using cached Jinja2-3.1.2.tar.gz (268 kB) ``` but in reality they get installed after collecting all packages, including the ones listed in `requirements.txt`. ``` # There is a long list of `:: :: Collecting <package-name>...` before the following lines :: :: Successfully built wheel setuptools pip :: :: Installing collected packages: wheel, setuptools, pip :: :: Attempting uninstall: setuptools :: :: Found existing installation: setuptools 44.0.0 :: :: Uninstalling setuptools-44.0.0: :: :: Successfully uninstalled setuptools-44.0.0 :: :: Attempting uninstall: pip :: :: Found existing installation: pip 20.0.2 :: :: Uninstalling pip-20.0.2: :: :: Successfully uninstalled pip-20.0.2 ``` This could result in errors if any of the charm requirements depends on a `charm-python-package` to be installed (or upgraded) a priori. Please NOTE I concluded this was an issue after trying to solve [this other issue](https://github.com/pallets/jinja/issues/1496) with the suggestions from [this post](https://discourse.charmhub.io/t/install-or-update-python-packages-before-packing-a-charm/5158). Upgrading `setuptools` fixes the issue with `markupsafe` and `jinja2`, I tested it in a virtual env, but it was not possible in the charm. ### To Reproduce 1. Take [this charm](https://github.com/canonical/kfp-operators/tree/main/charms/kfp-api). 2. `charmcraft pack -v` 3. Check the build logs and observe when the `setuptools` and `pip` packages get actually installed and upgraded. ### Environment * Ubuntu 20.04 * charmcraft latest/candidate ### charmcraft.yaml ```shell type: "charm" bases: - build-on: - name: "ubuntu" channel: "20.04" run-on: - name: "ubuntu" channel: "20.04" parts: charm: charm-python-packages: [setuptools>69.0, pip>23.0] # Fixes install of some packages ``` ### Relevant log output Logs: https://pastebin.canonical.com/p/DhpBRNn7pz/

Evaluation history

Date Model Scores Action Summary
qwen3.6-35b-a3b-mtp-q6 Fixed in Charmcraft 3.2. Maintainers confirmed the charm-python-packages installation ordering bug was resolved through related PRs and subsequent releases. The issue is closed.
qwen/qwen3.6-35b-a3b Closed. Maintainers investigated persistent build failures where charm-python-packages installed after requirements.txt. The issue was closed after a maintainer noted it may have been resolved in Charmcraft 3.2.
qwen/qwen3.6-35b-a3b Reported that charm-python-packages install order conflicts with requirements.txt. Multiple PRs attempted fixes but errors persisted. The issue was closed after maintainers confirmed the behavior was corrected in Charmcraft 3.2.
qwen3.6-35b-a3b-mtp-q6 charm-python-packages were not guaranteed to install before requirements.txt, causing build failures. The maintainer confirmed the bug was fixed in Charmcraft 3.2.
qwen3.6-35b-a3b-mtp-q6 Reported that charm-python-packages install after requirements.txt, causing build failures. After multiple attempted fixes, the issue was closed, confirming resolution in Charmcraft 3.2.

Update history

No update history recorded yet.

Related issues

Issue Project State Summary Similarity
#1781 charmcraft `3.1.1` "successfully" packs charm, but python packages are missing in the venv charmcraft closed Missing Python packages in charmcraft 3.1.1 venvs resulted from a strict dependencies bug. The developer identified the missing pip check step, implemented a fix, and closed the report after updating validation logic.
71%
#553 parts,charm_builder: add charm-python-packages plugin property (CRAFT-552) charmcraft merged Merged addition of the charm-python-packages plugin property, installing listed Python packages before requirements file dependencies. Mirrors standard python plugin behavior. Approved, passed CI, and merged by maintainer. Available in edge snap.
70%
#551 [charmcraft 1.3]: Cannot pack a charm with Jinja2 as dependency charmcraft closed Charmcraft 1.3 failed to pack charms with Jinja2 due to dependency issues. Closed with a setuptools workaround. Resolved via PR #553, which adds a charm-python-packages property to pre-install dependencies before requirements.
70%