`charm-python-packages` does not guarantee these packages are installed before packages in `requirements.txt`
Metadata
Current evaluation
Fixed in Charmcraft 3.2. Maintainers confirmed the charm-python-packages installation ordering bug was resolved through related PRs and subsequent releases. The issue is closed.
Suggested action: —
No scores available.
Issue body
### Bug Description
I have noticed that the list of packages in `charm-python-packages` may not guarantee they will be installed before charm requirements, as the [docs suggest](https://juju.is/docs/sdk/charmcraft-yaml#heading--the-charm-plugin).
The behaviour I am getting is that the packages listed in `charm-python-packages` get collected and it looks like they are being installed as the very first thing, for example:
```
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=anyio,attrs,certifi,charmed-kubeflow-chisme,charset-normalizer,deepdiff,exceptiongroup,h11,httpcore,httpx,idna,importlib-resources,jinja2,jsonschema,lightkube,lightkube-models,markupsafe,oci-image,ops,ordered-set,pip,pkgutil-resolve-name,pyrsistent,pyyaml,requests,ruamel-yaml,ruamel-yaml-clib,serialized-data-interface,setuptools,sniffio,tenacity,urllib3,websocket-client,wheel,zipp', '--requirement=requirements.txt', 'pip>23.0', 'setuptools>69.0', 'wheel']
:: :: Collecting pip>23.0
:: :: Using cached pip-24.0.tar.gz (2.1 MB)
:: :: Installing build dependencies: started
:: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing wheel metadata: started
:: :: Preparing wheel metadata: finished with status 'done'
:: :: Collecting setuptools>69.0
:: :: Using cached setuptools-69.5.1.tar.gz (2.3 MB)
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Installing backend dependencies: started
:: :: Installing backend dependencies: finished with status 'done'
:: :: Preparing wheel metadata: started
:: :: Preparing wheel metadata: finished with status 'done'
:: :: Collecting jinja2==3.1.2
:: :: Using cached Jinja2-3.1.2.tar.gz (268 kB)
```
but in reality they get installed after collecting all packages, including the ones listed in `requirements.txt`.
```
# There is a long list of `:: :: Collecting <package-name>...` before the following lines
:: :: Successfully built wheel setuptools pip
:: :: Installing collected packages: wheel, setuptools, pip
:: :: Attempting uninstall: setuptools
:: :: Found existing installation: setuptools 44.0.0
:: :: Uninstalling setuptools-44.0.0:
:: :: Successfully uninstalled setuptools-44.0.0
:: :: Attempting uninstall: pip
:: :: Found existing installation: pip 20.0.2
:: :: Uninstalling pip-20.0.2:
:: :: Successfully uninstalled pip-20.0.2
```
This could result in errors if any of the charm requirements depends on a `charm-python-package` to be installed (or upgraded) a priori.
Please NOTE I concluded this was an issue after trying to solve [this other issue](https://github.com/pallets/jinja/issues/1496) with the suggestions from [this post](https://discourse.charmhub.io/t/install-or-update-python-packages-before-packing-a-charm/5158). Upgrading `setuptools` fixes the issue with `markupsafe` and `jinja2`, I tested it in a virtual env, but it was not possible in the charm.
### To Reproduce
1. Take [this charm](https://github.com/canonical/kfp-operators/tree/main/charms/kfp-api).
2. `charmcraft pack -v`
3. Check the build logs and observe when the `setuptools` and `pip` packages get actually installed and upgraded.
### Environment
* Ubuntu 20.04
* charmcraft latest/candidate
### charmcraft.yaml
```shell
type: "charm"
bases:
- build-on:
- name: "ubuntu"
channel: "20.04"
run-on:
- name: "ubuntu"
channel: "20.04"
parts:
charm:
charm-python-packages: [setuptools>69.0, pip>23.0] # Fixes install of some packages
```
### Relevant log output
Logs: https://pastebin.canonical.com/p/DhpBRNn7pz/
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen3.6-35b-a3b-mtp-q6 | — | — | Fixed in Charmcraft 3.2. Maintainers confirmed the charm-python-packages installation ordering bug was resolved through related PRs and subsequent releases. The issue is closed. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed. Maintainers investigated persistent build failures where charm-python-packages installed after requirements.txt. The issue was closed after a maintainer noted it may have been resolved in Charmcraft 3.2. | |
| qwen/qwen3.6-35b-a3b | — | — | Reported that charm-python-packages install order conflicts with requirements.txt. Multiple PRs attempted fixes but errors persisted. The issue was closed after maintainers confirmed the behavior was corrected in Charmcraft 3.2. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | charm-python-packages were not guaranteed to install before requirements.txt, causing build failures. The maintainer confirmed the bug was fixed in Charmcraft 3.2. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Reported that charm-python-packages install after requirements.txt, causing build failures. After multiple attempted fixes, the issue was closed, confirming resolution in Charmcraft 3.2. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1781 charmcraft `3.1.1` "successfully" packs charm, but python packages are missing in the venv | charmcraft | closed | Missing Python packages in charmcraft 3.1.1 venvs resulted from a strict dependencies bug. The developer identified the missing pip check step, implemented a fix, and closed the report after updating validation logic. | |
| #553 parts,charm_builder: add charm-python-packages plugin property (CRAFT-552) | charmcraft | merged | Merged addition of the charm-python-packages plugin property, installing listed Python packages before requirements file dependencies. Mirrors standard python plugin behavior. Approved, passed CI, and merged by maintainer. Available in edge snap. | |
| #551 [charmcraft 1.3]: Cannot pack a charm with Jinja2 as dependency | charmcraft | closed | Charmcraft 1.3 failed to pack charms with Jinja2 due to dependency issues. Closed with a setuptools workaround. Resolved via PR #553, which adds a charm-python-packages property to pre-install dependencies before requirements. |