← Back to issue list

(CI) Parallel build test is flaky

View original Github issue

Metadata

Project
charmcraft
Number
#1332
Type
issue
State
open
Author
lengau
Labels
Created
Updated
Closed

Current evaluation

Flaky parallel-build spread test (concurrent base-instance creation fails on Google CI). Mitigated by moving to weekly CI in #1600, but root cause unfixed and test still marked flaky. No labels, comments, or assignee after ~3 years.

Suggested action: needs triage

Reason: PR #1600 (commit a1d285c7) moved the test to weekly CI as a mitigation, but task.yaml still carries 'manual: true # This test is somewhat flaky' and no root-cause fix for the concurrent base-instance race has landed. The issue has no labels, no comments, and no assignee after ~3 years, so a maintainer needs to decide whether to close it as mitigated or invest in fixing the underlying race.

Impact: 20 Quick Win: 9.0 Staleness: 60 Complexity: 55 Confidence: 75 Support Request: 0

Issue body

### Bug Description The parallel build smoke test doesn't always succeed. ### To Reproduce Run the smoke tests in Google ### Environment Google environment (cannot replicate on multipass on my machine ### charmcraft.yaml ```shell N/A ``` ### Relevant log output ```shell 2023-10-17 01:04:32 Error executing google:ubuntu-20.04-64:tests/spread/smoketests/parallel-build (oct170052-258541) : ----- + sleep 60 + cd charm2 + charmcraft pack --verbose + cd charm1 + charmcraft pack --verbose Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.512686.log' Packing the charm. Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.511956.log' Packing the charm. Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable) Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable) Creating new instance from remote Creating new base instance from remote Creating new instance from remote Creating new base instance from remote Failed to launch instance 'base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50'. Full execution log: '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.511956.log' + wait + cd charm3 + charmcraft pack --verbose Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-010051.612722.log' Packing the charm. Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable) Creating new instance from base instance Creating instance from base instance Starting instance Starting instance Packing the charm Packing the charm Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/tmp/charmcraft.log' Packing the charm. Building charm in '/root' Installing build-packages Starting charmcraft version 2.4.1.post77+git81e1ff9 Logging execution to '/tmp/charmcraft.log' Packing the charm. Building charm in '/root' Installing build-packages Running step PULL for part 'charm' Execute action Running step BUILD for part 'charm' Execute action :: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post77+git81e1ff9 /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt Running step PULL for part 'charm' Execute action Running step BUILD for part 'charm' Execute action :: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post77+git81e1ff9 /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt :: Starting charm builder :: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z' :: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z' :: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z' :: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z' :: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z' :: Translated .jujuignore 8 '/build/' => '/build\\Z' :: Translated .jujuignore 9 '/revision' => '/revision\\Z' :: Translated .jujuignore 10 '/venv' => '/venv\\Z' :: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z' :: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z' :: Collected charmlib dependencies: set() :: Building charm in '/root/parts/charm/install' :: Linking in generic paths :: Creating the dispatch mechanism :: Creating the 'upgrade-charm' hook script pointing to dispatch :: Creating the 'install' hook script pointing to dispatch :: Creating the 'start' hook script pointing to dispatch :: Handling dependencies :: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d' :: Dependencies directory not found :: Installing dependencies :: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv'] :: Starting charm builder :: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z' :: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z' :: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z' :: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z' :: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z' :: Translated .jujuignore 8 '/build/' => '/build\\Z' :: Translated .jujuignore 9 '/revision' => '/revision\\Z' :: Translated .jujuignore 10 '/venv' => '/venv\\Z' :: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z' :: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z' :: Collected charmlib dependencies: set() :: Building charm in '/root/parts/charm/install' :: Linking in generic paths :: Creating the dispatch mechanism :: Creating the 'install' hook script pointing to dispatch :: Creating the 'start' hook script pointing to dispatch :: Creating the 'upgrade-charm' hook script pointing to dispatch :: Handling dependencies :: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d' :: Dependencies directory not found :: Installing dependencies :: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv'] :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel'] :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel'] :: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2) :: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0) :: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2) :: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0) :: :: Collecting wheel :: :: Collecting wheel :: :: Downloading wheel-0.41.2.tar.gz (98 kB) :: :: Downloading wheel-0.41.2.tar.gz (98 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.5 MB/s eta 0:00:00 :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.5 MB/s eta 0:00:00 :: :: Installing build dependencies: started :: :: Installing build dependencies: started :: :: Installing build dependencies: finished with status 'done' :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started:: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Collecting ops~=2.5 :: :: Collecting ops~=2.5 :: :: Downloading ops-2.7.0.tar.gz (240 kB) :: :: Downloading ops-2.7.0.tar.gz (240 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.2/240.2 KB 6.0 MB/s eta 0:00:00 :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.2/240.2 KB 6.0 MB/s eta 0:00:00 :: :: Installing build dependencies: started :: :: Installing build dependencies: started :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Collecting websocket-client==1.* :: :: Downloading websocket_client-1.6.4-py3-none-any.whl (57 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.3/57.3 KB 6.6 MB/s eta 0:00:00 :: :: Collecting PyYAML==6.* :: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB) :: :: Collecting PyYAML==6.* :: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 8.7 MB/s eta 0:00:00 :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 7.6 MB/s eta 0:00:00 :: :: Collecting websocket-client==1.* :: :: Using cached websocket_client-1.6.4-py3-none-any.whl (57 kB) :: :: Building wheels for collected packages: wheel, ops :: :: Building wheel for wheel (pyproject.toml): started :: :: Building wheels for collected packages: wheel, ops :: :: Building wheel for wheel (pyproject.toml): started :: :: Building wheel for wheel (pyproject.toml): finished with status 'done' :: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8 :: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90 :: :: Building wheel for ops (pyproject.toml): started :: :: Building wheel for wheel (pyproject.toml): finished with status 'done' :: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8 :: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90 :: :: Building wheel for ops (pyproject.toml): started :: :: Building wheel for ops (pyproject.toml): finished with status 'done' :: :: Created wheel for ops: filename=ops-2.7.0-py3-none-any.whl size=146985 sha256=d20224550648efe9859f8ed90fdbcf617431e7af30ff4b56391216a9fc9710e8 :: :: Stored in directory: /root/.cache/pip/wheels/98/fb/70/4c89833e50cef8b667100e5fdf9378d658284e9659803980ba :: :: Successfully built wheel ops :: :: Installing collected packages: wheel, websocket-client, PyYAML, ops :: :: Building wheel for ops (pyproject.toml): finished with status 'done' :: :: Created wheel for ops: filename=ops-2.7.0-py3-none-any.whl size=146985 sha256=d20224550648efe9859f8ed90fdbcf617431e7af30ff4b56391216a9fc9710e8 :: :: Stored in directory: /root/.cache/pip/wheels/98/fb/70/4c89833e50cef8b667100e5fdf9378d658284e9659803980ba :: :: Successfully built wheel ops :: :: Installing collected packages: wheel, websocket-client, PyYAML, ops :: :: Successfully installed PyYAML-6.0.1 ops-2.7.0 websocket-client-1.6.4 wheel-0.41.2 :: :: Successfully installed PyYAML-6.0.1 ops-2.7.0 websocket-client-1.6.4 wheel-0.41.2 Running step STAGE for part 'charm' Execute action Running step STAGE for part 'charm' Execute action Running step PRIME for part 'charm' Execute action Running step PRIME for part 'charm' Execute action Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language). Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework). Creating the package itself Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language). Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework). Creating the package itself Created 'charm3_ubuntu-22.04-amd64.charm'. Created 'charm1_ubuntu-22.04-amd64.charm'. Charm packed ok Charms packed: charm1_ubuntu-22.04-amd64.charm Charm packed ok Charms packed: charm3_ubuntu-22.04-amd64.charm + pushd charm1 /charmcraft/tests/spread/smoketests/parallel-build/charm1 /charmcraft/tests/spread/smoketests/parallel-build + test -f charm1_ubuntu-22.04-amd64.charm + MATCH venv/ops/charm.py + unzip -l charm1_ubuntu-22.04-amd64.charm + test '!' -d build + popd /charmcraft/tests/spread/smoketests/parallel-build + pushd charm2 /charmcraft/tests/spread/smoketests/parallel-build/charm2 /charmcraft/tests/spread/smoketests/parallel-build + test -f 'charm*.charm' + echo 'charm2 failed to build' charm2 failed to build + exit 1 ----- . ```

Evaluation history

Date Model Scores Action Summary
qwen/qwen3.8-27b
Impact: 20
Quick Win: 9.0
Staleness: 60
Complexity: 55
Confidence: 75
Support Request: 0
needs triage Flaky parallel-build spread test (concurrent base-instance creation fails on Google CI). Mitigated by moving to weekly CI in #1600, but root cause unfixed and test still marked flaky. No labels, comments, or assignee after ~3 years.
qwen/qwen3.8-27b
Impact: 30
Quick Win: 9.0
Staleness: 70
Complexity: 70
Confidence: 75
Support Request: 0
keep open Flaky parallel-build spread smoke test in Google CI: concurrent charmcraft pack runs race on LXC base-instance creation, causing 'Failed to launch instance' and a missing charm2 artifact. Mitigated by moving the test to a weekly manual workflow, not fixed.
qwen/qwen3.6-35b-a3b
Staleness: 90
Complexity: 40
Confidence: 65
Support Request: 5
needs triage CI parallel build smoke test fails intermittently in Google environment. Opened by maintainer nearly three years ago with zero comments, no labels, and no activity for 340 days. Requires maintainer review to assess current relevance.
qwen3.6-35b-a3b-mtp-q6
Staleness: 92
Complexity: 65
Confidence: 75
Support Request: 5
needs triage CI parallel build smoke test fails intermittently in Google Cloud due to instance launch failures. Open for nearly three years with no maintainer labels or comments.
qwen3.6-35b-a3b-mtp-q6
Staleness: 90
Complexity: 40
Confidence: 90
Support Request: 0
close stale Flaky CI parallel build test in Google environment showing instance launch failures. No maintainer response in over a year.
qwen3.6-35b-a3b-mtp-q6
Staleness: 90
Complexity: 50
Confidence: 75
Support Request: 5
close stale CI parallel build smoke test fails intermittently in Google environments due to concurrent instance launch errors. Open, unassigned, and stalled for over 300 days. Requires investigation or triage to diagnose and fix.

Update history

No update history recorded yet.

Related work

  • Related To: charmcraft#1600 (confidence 85%)

    PR #1600 moved the flaky parallel-build test to weekly CI as a mitigation but did not fix the concurrent base-instance creation failure.

Related issues

Issue Project State Summary Similarity
#1295 Parallel build test is flaky charmcraft closed The flaky parallel build test failure was resolved. The issue closed after craft-providers 1.18 fixed the underlying race condition causing incompatible base detection errors in CI.
75%
#1600 test(spread): move flaky parallel-build test to weeklies charmcraft merged Merged to address issue #1332 by relocating a flaky parallel-build test to the weekly suite. Approved by two reviewers, the change modifies two files (+19/-2) to stabilize CI runs.
70%