(CI) Parallel build test is flaky
Metadata
Current evaluation
Parallel-pack smoke-test flakiness was traced to a craft-providers base-instance race and addressed by the v1.18.0 upgrade; the test now runs separately.
Suggested action: close resolved
Reason: The related #1295 investigation identified a craft-providers race, and PR #1294 resolved it by upgrading to v1.18.0 in commit 59b70f01. Later PR #1600 moved the still-potentially-flaky smoke test to weekly/manual CI, but did not leave this reported race unresolved.
Evaluated by: mr-cal-bot
Impact:
35
Quick Win:
29
Complexity:
15
Confidence:
94
Actionability:
98
Issue body
### Bug Description
The parallel build smoke test doesn't always succeed.
### To Reproduce
Run the smoke tests in Google
### Environment
Google environment (cannot replicate on multipass on my machine
### charmcraft.yaml
```shell
N/A
```
### Relevant log output
```shell
2023-10-17 01:04:32 Error executing google:ubuntu-20.04-64:tests/spread/smoketests/parallel-build (oct170052-258541) :
-----
+ sleep 60
+ cd charm2
+ charmcraft pack --verbose
+ cd charm1
+ charmcraft pack --verbose
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.512686.log'
Packing the charm.
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.511956.log'
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Creating new instance from remote
Creating new base instance from remote
Creating new instance from remote
Creating new base instance from remote
Failed to launch instance 'base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50'.
Full execution log: '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.511956.log'
+ wait
+ cd charm3
+ charmcraft pack --verbose
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-010051.612722.log'
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Creating new instance from base instance
Creating instance from base instance
Starting instance
Starting instance
Packing the charm
Packing the charm
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/tmp/charmcraft.log'
Packing the charm.
Building charm in '/root'
Installing build-packages
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/tmp/charmcraft.log'
Packing the charm.
Building charm in '/root'
Installing build-packages
Running step PULL for part 'charm'
Execute action
Running step BUILD for part 'charm'
Execute action
:: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post77+git81e1ff9 /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt
Running step PULL for part 'charm'
Execute action
Running step BUILD for part 'charm'
Execute action
:: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post77+git81e1ff9 /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt
:: Starting charm builder
:: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z'
:: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z'
:: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z'
:: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z'
:: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z'
:: Translated .jujuignore 8 '/build/' => '/build\\Z'
:: Translated .jujuignore 9 '/revision' => '/revision\\Z'
:: Translated .jujuignore 10 '/venv' => '/venv\\Z'
:: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z'
:: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z'
:: Collected charmlib dependencies: set()
:: Building charm in '/root/parts/charm/install'
:: Linking in generic paths
:: Creating the dispatch mechanism
:: Creating the 'upgrade-charm' hook script pointing to dispatch
:: Creating the 'install' hook script pointing to dispatch
:: Creating the 'start' hook script pointing to dispatch
:: Handling dependencies
:: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d'
:: Dependencies directory not found
:: Installing dependencies
:: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv']
:: Starting charm builder
:: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z'
:: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z'
:: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z'
:: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z'
:: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z'
:: Translated .jujuignore 8 '/build/' => '/build\\Z'
:: Translated .jujuignore 9 '/revision' => '/revision\\Z'
:: Translated .jujuignore 10 '/venv' => '/venv\\Z'
:: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z'
:: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z'
:: Collected charmlib dependencies: set()
:: Building charm in '/root/parts/charm/install'
:: Linking in generic paths
:: Creating the dispatch mechanism
:: Creating the 'install' hook script pointing to dispatch
:: Creating the 'start' hook script pointing to dispatch
:: Creating the 'upgrade-charm' hook script pointing to dispatch
:: Handling dependencies
:: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d'
:: Dependencies directory not found
:: Installing dependencies
:: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv']
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel']
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel']
:: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2)
:: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0)
:: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2)
:: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0)
:: :: Collecting wheel
:: :: Collecting wheel
:: :: Downloading wheel-0.41.2.tar.gz (98 kB)
:: :: Downloading wheel-0.41.2.tar.gz (98 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.5 MB/s eta 0:00:00
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.5 MB/s eta 0:00:00
:: :: Installing build dependencies: started
:: :: Installing build dependencies: started
:: :: Installing build dependencies: finished with status 'done'
:: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Collecting ops~=2.5
:: :: Collecting ops~=2.5
:: :: Downloading ops-2.7.0.tar.gz (240 kB)
:: :: Downloading ops-2.7.0.tar.gz (240 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.2/240.2 KB 6.0 MB/s eta 0:00:00
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.2/240.2 KB 6.0 MB/s eta 0:00:00
:: :: Installing build dependencies: started
:: :: Installing build dependencies: started
:: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Collecting websocket-client==1.*
:: :: Downloading websocket_client-1.6.4-py3-none-any.whl (57 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.3/57.3 KB 6.6 MB/s eta 0:00:00
:: :: Collecting PyYAML==6.*
:: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
:: :: Collecting PyYAML==6.*
:: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 8.7 MB/s eta 0:00:00
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 7.6 MB/s eta 0:00:00
:: :: Collecting websocket-client==1.*
:: :: Using cached websocket_client-1.6.4-py3-none-any.whl (57 kB)
:: :: Building wheels for collected packages: wheel, ops
:: :: Building wheel for wheel (pyproject.toml): started
:: :: Building wheels for collected packages: wheel, ops
:: :: Building wheel for wheel (pyproject.toml): started
:: :: Building wheel for wheel (pyproject.toml): finished with status 'done'
:: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8
:: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90
:: :: Building wheel for ops (pyproject.toml): started
:: :: Building wheel for wheel (pyproject.toml): finished with status 'done'
:: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8
:: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90
:: :: Building wheel for ops (pyproject.toml): started
:: :: Building wheel for ops (pyproject.toml): finished with status 'done'
:: :: Created wheel for ops: filename=ops-2.7.0-py3-none-any.whl size=146985 sha256=d20224550648efe9859f8ed90fdbcf617431e7af30ff4b56391216a9fc9710e8
:: :: Stored in directory: /root/.cache/pip/wheels/98/fb/70/4c89833e50cef8b667100e5fdf9378d658284e9659803980ba
:: :: Successfully built wheel ops
:: :: Installing collected packages: wheel, websocket-client, PyYAML, ops
:: :: Building wheel for ops (pyproject.toml): finished with status 'done'
:: :: Created wheel for ops: filename=ops-2.7.0-py3-none-any.whl size=146985 sha256=d20224550648efe9859f8ed90fdbcf617431e7af30ff4b56391216a9fc9710e8
:: :: Stored in directory: /root/.cache/pip/wheels/98/fb/70/4c89833e50cef8b667100e5fdf9378d658284e9659803980ba
:: :: Successfully built wheel ops
:: :: Installing collected packages: wheel, websocket-client, PyYAML, ops
:: :: Successfully installed PyYAML-6.0.1 ops-2.7.0 websocket-client-1.6.4 wheel-0.41.2
:: :: Successfully installed PyYAML-6.0.1 ops-2.7.0 websocket-client-1.6.4 wheel-0.41.2
Running step STAGE for part 'charm'
Execute action
Running step STAGE for part 'charm'
Execute action
Running step PRIME for part 'charm'
Execute action
Running step PRIME for part 'charm'
Execute action
Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language).
Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework).
Creating the package itself
Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language).
Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework).
Creating the package itself
Created 'charm3_ubuntu-22.04-amd64.charm'.
Created 'charm1_ubuntu-22.04-amd64.charm'.
Charm packed ok
Charms packed:
charm1_ubuntu-22.04-amd64.charm
Charm packed ok
Charms packed:
charm3_ubuntu-22.04-amd64.charm
+ pushd charm1
/charmcraft/tests/spread/smoketests/parallel-build/charm1 /charmcraft/tests/spread/smoketests/parallel-build
+ test -f charm1_ubuntu-22.04-amd64.charm
+ MATCH venv/ops/charm.py
+ unzip -l charm1_ubuntu-22.04-amd64.charm
+ test '!' -d build
+ popd
/charmcraft/tests/spread/smoketests/parallel-build
+ pushd charm2
/charmcraft/tests/spread/smoketests/parallel-build/charm2 /charmcraft/tests/spread/smoketests/parallel-build
+ test -f 'charm*.charm'
+ echo 'charm2 failed to build'
charm2 failed to build
+ exit 1
-----
.
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha |
Impact:
35
Quick Win:
29
Complexity:
15
Confidence:
94
Actionability:
98
|
close resolved | Parallel-pack smoke-test flakiness was traced to a craft-providers base-instance race and addressed by the v1.18.0 upgrade; the test now runs separately. | |
| qwen/qwen3.6-35b-a3b |
Impact:
30
Quick Win:
14
Complexity:
40
Confidence:
95
Actionability:
80
|
close resolved | Flaky parallel build smoke test in CI. The issue was closed after craft-providers 1.18 fixed the underlying race condition causing base detection errors. | |
| qwen/qwen3.8-27b |
Impact:
20
Quick Win:
9
Staleness:
60
Complexity:
55
Confidence:
75
Support Request:
0
|
needs triage | Flaky parallel-build spread test (concurrent base-instance creation fails on Google CI). Mitigated by moving to weekly CI in #1600, but root cause unfixed and test still marked flaky. No labels, comments, or assignee after ~3 years. | |
| qwen/qwen3.8-27b |
Impact:
30
Quick Win:
9
Staleness:
70
Complexity:
70
Confidence:
75
Support Request:
0
|
keep open | Flaky parallel-build spread smoke test in Google CI: concurrent charmcraft pack runs race on LXC base-instance creation, causing 'Failed to launch instance' and a missing charm2 artifact. Mitigated by moving the test to a weekly manual workflow, not fixed. | |
| qwen/qwen3.6-35b-a3b |
Staleness:
90
Complexity:
40
Confidence:
65
Support Request:
5
|
needs triage | CI parallel build smoke test fails intermittently in Google environment. Opened by maintainer nearly three years ago with zero comments, no labels, and no activity for 340 days. Requires maintainer review to assess current relevance. | |
| qwen3.6-35b-a3b-mtp-q6 |
Staleness:
92
Complexity:
65
Confidence:
75
Support Request:
5
|
needs triage | CI parallel build smoke test fails intermittently in Google Cloud due to instance launch failures. Open for nearly three years with no maintainer labels or comments. | |
| qwen3.6-35b-a3b-mtp-q6 |
Staleness:
90
Complexity:
40
Confidence:
90
Support Request:
0
|
close stale | Flaky CI parallel build test in Google environment showing instance launch failures. No maintainer response in over a year. |
Update history
No update history recorded yet.
Related work
-
Duplicate Of:
charmcraft#1295
(confidence 99%)
Describes the same parallel-build flakiness and traces it to a craft-providers race.
-
Likely Fixed By:
charmcraft#1294
(confidence 99%)
Updated craft-providers to v1.18.0 in resolving commit 59b70f01.
-
Related To:
charmcraft#1600
(confidence 96%)
Moved the parallel-build smoke test to weekly CI because it remains potentially flaky.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1295 Parallel build test is flaky | charmcraft | closed | Flakiness in the parallel-build spread test traced to a craft-providers instance race; closed after charmcraft PR #1294 (commit 59b70f01) upgraded craft-providers to v1.18.0, per @syu-w's comment. |