← Back to issue list

Charm venv shadows system python dependencies

View original Github issue

Metadata

Project
charmcraft
Number
#1919
Type
issue
State
open
Author
silverdrake11
Labels
—
Created
Updated
Closed
—

Current evaluation

Reported: charm plugin's PYTHONPATH=lib:venv let Juju charm venv shadow system Python packages (pydantic ImportError). lengau confirmed and pointed to new dispatch-based plugins; dispatch now sets PYTHONPATH=lib:src, so the fix has landed.

Suggested action: close resolved

Reason: Maintainer lengau explicitly acknowledged the root cause — "this is an unfortunate side effect of the way the `charm` plugin works, as it uses `PYTHONPATH` to simulate being in the virtual environment" — and stated "the next minor release of Charmcraft will include different plugins that use a dispatch file that activates the virtual environment correctly. It still uses `PYTHONPATH`, but only for adding charmlibs and the charm itself." That fix is verified present in the tree: `charmcraft/dispatch.py` sets `export PYTHONPATH="${dispatch_path}/lib:${dispatch_path}/src"` (no `venv`) and execs `venv/bin/python` directly, added in commit b4f91c7b (PR #1898) and hardened in ad438d60 (#1972) and 7f238725 (#2017). The only remaining `PYTHONPATH=lib:venv` is the legacy `DISPATCH_CONTENT` in `charmcraft/const.py`, which the maintainer said cannot be changed.

Evaluated by: mr-cal-bot

Impact: 55 Quick Win: 27 Complexity: 45 Confidence: 90 Actionability: 90

Issue body

### Bug Description When running subprocesses within the charm code, the Juju Python virtual environment `/var/lib/juju/agents/unit-landscape-server-0/charm/venv/` is included in the system’s PYTHONPATH. This causes conflicts, as we need to use the system Python and its dependencies, not those from the Juju environment. For example this bug https://bugs.launchpad.net/landscape-charm/+bug/2076926, installing a later version of the pydantic package via apt doesn't do anything because the version from the Juju charm virtual environment takes precedence. A similar issue occurred with the distutils package, where the system version was overshadowed. ### To Reproduce We used this workaround to make sure the system Python paths are used by removing Juju's virtual environment from PYTHONPATH before invoking subprocesses: ```python def get_modified_env_vars(): """ Because the python path gets munged by the juju env, this grabs the current env vars and returns a copy with the juju env removed from the python paths """ env_vars = os.environ.copy() logging.info("Fixing python paths") new_paths = [path for path in sys.path if "juju" not in path] env_vars["PYTHONPATH"] = ":".join(new_paths) return env_vars subprocess.check_call(call, env=get_modified_env_vars()) ``` ### Environment 22.04 juju --version 3.5.3-genericlinux-amd64 ### charmcraft.yaml ```yaml N/a ``` ### Relevant log output ```shell 2024-08-13 08:25:19 WARNING unit.landscape-server/0.db-relation-changed logger.go:60 from canonical.landscape.model.main.oidc import ( 2024-08-13 08:25:19 WARNING unit.landscape-server/0.db-relation-changed logger.go:60 File "/opt/canonical/landscape/canonical/landscape/model/main/oidc.py", line 6, in <module> 2024-08-13 08:25:19 WARNING unit.landscape-server/0.db-relation-changed logger.go:60 from pydantic import AnyUrl, BaseModel, Field, model_validator 2024-08-13 08:25:19 WARNING unit.landscape-server/0.db-relation-changed logger.go:60 ImportError: cannot import name 'model_validator' from 'pydantic' (/var/lib/juju/agents/unit-landscape-server-0/charm/venv/pydantic/__init__.py) 2024-08-13 08:25:19 ERROR unit.landscape-server/0.juju-log server.go:325 db:8: Landscape Server schema update failed with return code 1 ```

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha
Impact: 55
Quick Win: 27
Complexity: 45
Confidence: 90
Actionability: 90
close resolved Reported: charm plugin's PYTHONPATH=lib:venv let Juju charm venv shadow system Python packages (pydantic ImportError). lengau confirmed and pointed to new dispatch-based plugins; dispatch now sets PYTHONPATH=lib:src, so the fix has landed.
qwen/qwen3.6-35b-a3b
Impact: 70
Quick Win: 39
Complexity: 30
Confidence: 85
Actionability: 80
close resolved Juju venv PYTHONPATH shadows system Python dependencies in subprocesses. Fixed by new dispatch mechanism in PR #1898/#2017 which sets PYTHONPATH=lib:src instead of lib:venv.
qwen/qwen3.8-27b
Impact: 50
Quick Win: 20
Staleness: 55
Complexity: 60
Confidence: 70
Support Request: 10
keep open Charm venv on PYTHONPATH shadows system Python deps for subprocesses; maintainer confirmed it's an unavoidable side effect of the charm plugin, with new plugins using a dispatch file to activate the venv correctly in a future release.
qwen/qwen3.6-35b-a3b
Staleness: 80
Complexity: 30
Confidence: 85
Support Request: 20
keep open Reported that Juju's charm venv PYTHONPATH shadows system Python dependencies, causing import conflicts. Acknowledged as a known plugin limitation with a planned architectural fix in a future release.
qwen3.6-35b-a3b-mtp-q6
Staleness: 60
Complexity: 25
Confidence: 90
Support Request: 30
keep open Juju Python venv shadows system dependencies via PYTHONPATH, causing import errors. Maintainer acknowledged it as a known plugin limitation with a fix planned for the next release.
qwen3.6-35b-a3b-mtp-q6
Staleness: 85
Complexity: 20
Confidence: 90
Support Request: 70
close not a bug Juju charm venv PYTHONPATH shadows system Python dependencies in subprocesses. Maintainer confirmed this is expected plugin behavior, not a bug. A future plugin redesign will resolve the mechanism. Issue remains open but unactionable.

Update history

No update history recorded yet.

Related work

  • Likely Fixed By: canonical/charmcraft#1898 (confidence 92%)

    Commit b4f91c7b 'feat(package): create a dispatch file if none exists' introduced dispatch.py, whose template drops `venv` from PYTHONPATH and execs the venv python explicitly — exactly the remedy lengau described.

  • Likely Fixed By: canonical/charmcraft#2017 (confidence 78%)

    Commit 7f238725 'fix(dispatch): fix the dispatch script' — follow-up hardening of the same dispatch template, indicating the mechanism shipped and was refined.

  • Related To: canonical/charmcraft#1972 (confidence 70%)

    Commit ad438d60 'fix(dispatch): fix the dispatch script if bins aren't kept' — same dispatch file, keeps the venv/bin/python symlink creation working in the shipped charm.

Related issues

Issue Project State Summary Similarity
#1077 Charms can import packages not installed in `venv` charmcraft closed Charm-plugin behavior remains unchanged; Python and Poetry plugins add venv isolation, available in Charmcraft 3.3 via #1899 (8b748f3e).
71%