← Back to issue list

(CI) Parallel build test is flaky

View original Github issue

Metadata

Project
charmcraft
Number
#1332
Type
issue
State
open
Author
lengau
Labels
—
Created
Updated
Closed
—

Current evaluation

Parallel-pack smoke-test flakiness was traced to a craft-providers base-instance race and addressed by the v1.18.0 upgrade; the test now runs separately.

Suggested action: close resolved

Reason: The related #1295 investigation identified a craft-providers race, and PR #1294 resolved it by upgrading to v1.18.0 in commit 59b70f01. Later PR #1600 moved the still-potentially-flaky smoke test to weekly/manual CI, but did not leave this reported race unresolved.

Evaluated by: mr-cal-bot

Impact: 35 Quick Win: 29 Complexity: 15 Confidence: 94 Actionability: 98

Issue body

### Bug Description The parallel build smoke test doesn't always succeed. ### To Reproduce Run the smoke tests in Google ### Environment Google environment (cannot replicate on multipass on my machine ### charmcraft.yaml ```shell N/A ``` ### Relevant log output ```shell 2023-10-17 01:04:32 Error executing google:ubuntu-20.04-64:tests/spread/smoketests/parallel-build (oct170052-258541) : ----- + sleep 60 + cd charm2 + charmcraft pack --verbose + cd charm1 + charmcraft pack --verbose Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.512686.log' Packing the charm. Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.511956.log' Packing the charm. Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable) Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable) Creating new instance from remote Creating new base instance from remote Creating new instance from remote Creating new base instance from remote Failed to launch instance 'base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50'. Full execution log: '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.511956.log' + wait + cd charm3 + charmcraft pack --verbose Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-010051.612722.log' Packing the charm. Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable) Creating new instance from base instance Creating instance from base instance Starting instance Starting instance Packing the charm Packing the charm Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/tmp/charmcraft.log' Packing the charm. Building charm in '/root' Installing build-packages Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/tmp/charmcraft.log' Packing the charm. Building charm in '/root' Installing build-packages Running step PULL for part 'charm' Execute action Running step BUILD for part 'charm' Execute action :: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post77+git81e1ff9 /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt Running step PULL for part 'charm' Execute action Running step BUILD for part 'charm' Execute action :: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post77+git81e1ff9 /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt :: Starting charm builder :: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z' :: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z' :: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z' :: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z' :: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z' :: Translated .jujuignore 8 '/build/' => '/build\\Z' :: Translated .jujuignore 9 '/revision' => '/revision\\Z' :: Translated .jujuignore 10 '/venv' => '/venv\\Z' :: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z' :: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z' :: Collected charmlib dependencies: set() :: Building charm in '/root/parts/charm/install' :: Linking in generic paths :: Creating the dispatch mechanism :: Creating the 'upgrade-charm' hook script pointing to dispatch :: Creating the 'install' hook script pointing to dispatch :: Creating the 'start' hook script pointing to dispatch :: Handling dependencies :: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d' :: Dependencies directory not found :: Installing dependencies :: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv'] :: Starting charm builder :: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z' :: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z' :: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z' :: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z' :: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z' :: Translated .jujuignore 8 '/build/' => '/build\\Z' :: Translated .jujuignore 9 '/revision' => '/revision\\Z' :: Translated .jujuignore 10 '/venv' => '/venv\\Z' :: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z' :: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z' :: Collected charmlib dependencies: set() :: Building charm in '/root/parts/charm/install' :: Linking in generic paths :: Creating the dispatch mechanism :: Creating the 'install' hook script pointing to dispatch :: Creating the 'start' hook script pointing to dispatch :: Creating the 'upgrade-charm' hook script pointing to dispatch :: Handling dependencies :: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d' :: Dependencies directory not found :: Installing dependencies :: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv'] :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel'] :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel'] :: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2) :: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0) :: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2) :: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0) :: :: Collecting wheel :: :: Collecting wheel :: :: Downloading wheel-0.41.2.tar.gz (98 kB) :: :: Downloading wheel-0.41.2.tar.gz (98 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.5 MB/s eta 0:00:00 :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.5 MB/s eta 0:00:00 :: :: Installing build dependencies: started :: :: Installing build dependencies: started :: :: Installing build dependencies: finished with status 'done' :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started:: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Collecting ops~=2.5 :: :: Collecting ops~=2.5 :: :: Downloading ops-2.7.0.tar.gz (240 kB) :: :: Downloading ops-2.7.0.tar.gz (240 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.2/240.2 KB 6.0 MB/s eta 0:00:00 :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.2/240.2 KB 6.0 MB/s eta 0:00:00 :: :: Installing build dependencies: started :: :: Installing build dependencies: started :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Collecting websocket-client==1.* :: :: Downloading websocket_client-1.6.4-py3-none-any.whl (57 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.3/57.3 KB 6.6 MB/s eta 0:00:00 :: :: Collecting PyYAML==6.* :: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB) :: :: Collecting PyYAML==6.* :: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 8.7 MB/s eta 0:00:00 :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 7.6 MB/s eta 0:00:00 :: :: Collecting websocket-client==1.* :: :: Using cached websocket_client-1.6.4-py3-none-any.whl (57 kB) :: :: Building wheels for collected packages: wheel, ops :: :: Building wheel for wheel (pyproject.toml): started :: :: Building wheels for collected packages: wheel, ops :: :: Building wheel for wheel (pyproject.toml): started :: :: Building wheel for wheel (pyproject.toml): finished with status 'done' :: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8 :: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90 :: :: Building wheel for ops (pyproject.toml): started :: :: Building wheel for wheel (pyproject.toml): finished with status 'done' :: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8 :: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90 :: :: Building wheel for ops (pyproject.toml): started :: :: Building wheel for ops (pyproject.toml): finished with status 'done' :: :: Created wheel for ops: filename=ops-2.7.0-py3-none-any.whl size=146985 sha256=d20224550648efe9859f8ed90fdbcf617431e7af30ff4b56391216a9fc9710e8 :: :: Stored in directory: /root/.cache/pip/wheels/98/fb/70/4c89833e50cef8b667100e5fdf9378d658284e9659803980ba :: :: Successfully built wheel ops :: :: Installing collected packages: wheel, websocket-client, PyYAML, ops :: :: Building wheel for ops (pyproject.toml): finished with status 'done' :: :: Created wheel for ops: filename=ops-2.7.0-py3-none-any.whl size=146985 sha256=d20224550648efe9859f8ed90fdbcf617431e7af30ff4b56391216a9fc9710e8 :: :: Stored in directory: /root/.cache/pip/wheels/98/fb/70/4c89833e50cef8b667100e5fdf9378d658284e9659803980ba :: :: Successfully built wheel ops :: :: Installing collected packages: wheel, websocket-client, PyYAML, ops :: :: Successfully installed PyYAML-6.0.1 ops-2.7.0 websocket-client-1.6.4 wheel-0.41.2 :: :: Successfully installed PyYAML-6.0.1 ops-2.7.0 websocket-client-1.6.4 wheel-0.41.2 Running step STAGE for part 'charm' Execute action Running step STAGE for part 'charm' Execute action Running step PRIME for part 'charm' Execute action Running step PRIME for part 'charm' Execute action Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language). Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework). Creating the package itself Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language). Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework). Creating the package itself Created 'charm3_ubuntu-22.04-amd64.charm'. Created 'charm1_ubuntu-22.04-amd64.charm'. Charm packed ok Charms packed: charm1_ubuntu-22.04-amd64.charm Charm packed ok Charms packed: charm3_ubuntu-22.04-amd64.charm + pushd charm1 /charmcraft/tests/spread/smoketests/parallel-build/charm1 /charmcraft/tests/spread/smoketests/parallel-build + test -f charm1_ubuntu-22.04-amd64.charm + MATCH venv/ops/charm.py + unzip -l charm1_ubuntu-22.04-amd64.charm + test '!' -d build + popd /charmcraft/tests/spread/smoketests/parallel-build + pushd charm2 /charmcraft/tests/spread/smoketests/parallel-build/charm2 /charmcraft/tests/spread/smoketests/parallel-build + test -f 'charm*.charm' + echo 'charm2 failed to build' charm2 failed to build + exit 1 ----- . ```

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha
Impact: 35
Quick Win: 29
Complexity: 15
Confidence: 94
Actionability: 98
close resolved Parallel-pack smoke-test flakiness was traced to a craft-providers base-instance race and addressed by the v1.18.0 upgrade; the test now runs separately.
qwen/qwen3.6-35b-a3b
Impact: 30
Quick Win: 14
Complexity: 40
Confidence: 95
Actionability: 80
close resolved Flaky parallel build smoke test in CI. The issue was closed after craft-providers 1.18 fixed the underlying race condition causing base detection errors.
qwen/qwen3.8-27b
Impact: 20
Quick Win: 9
Staleness: 60
Complexity: 55
Confidence: 75
Support Request: 0
needs triage Flaky parallel-build spread test (concurrent base-instance creation fails on Google CI). Mitigated by moving to weekly CI in #1600, but root cause unfixed and test still marked flaky. No labels, comments, or assignee after ~3 years.
qwen/qwen3.8-27b
Impact: 30
Quick Win: 9
Staleness: 70
Complexity: 70
Confidence: 75
Support Request: 0
keep open Flaky parallel-build spread smoke test in Google CI: concurrent charmcraft pack runs race on LXC base-instance creation, causing 'Failed to launch instance' and a missing charm2 artifact. Mitigated by moving the test to a weekly manual workflow, not fixed.
qwen/qwen3.6-35b-a3b
Staleness: 90
Complexity: 40
Confidence: 65
Support Request: 5
needs triage CI parallel build smoke test fails intermittently in Google environment. Opened by maintainer nearly three years ago with zero comments, no labels, and no activity for 340 days. Requires maintainer review to assess current relevance.
qwen3.6-35b-a3b-mtp-q6
Staleness: 92
Complexity: 65
Confidence: 75
Support Request: 5
needs triage CI parallel build smoke test fails intermittently in Google Cloud due to instance launch failures. Open for nearly three years with no maintainer labels or comments.
qwen3.6-35b-a3b-mtp-q6
Staleness: 90
Complexity: 40
Confidence: 90
Support Request: 0
close stale Flaky CI parallel build test in Google environment showing instance launch failures. No maintainer response in over a year.

Update history

No update history recorded yet.

Related work

  • Duplicate Of: charmcraft#1295 (confidence 99%)

    Describes the same parallel-build flakiness and traces it to a craft-providers race.

  • Likely Fixed By: charmcraft#1294 (confidence 99%)

    Updated craft-providers to v1.18.0 in resolving commit 59b70f01.

  • Related To: charmcraft#1600 (confidence 96%)

    Moved the parallel-build smoke test to weekly CI because it remains potentially flaky.

Related issues

Issue Project State Summary Similarity
#1295 Parallel build test is flaky charmcraft closed Flakiness in the parallel-build spread test traced to a craft-providers instance race; closed after charmcraft PR #1294 (commit 59b70f01) upgraded craft-providers to v1.18.0, per @syu-w's comment.
81%