Flaky parallel-build spread test (concurrent base-instance creation fails on Google CI). Mitigated by moving to weekly CI in #1600, but root cause unfixed and test still marked flaky. No labels, comments, or assignee after ~3 years.
Suggested action: needs triage
Reason: PR #1600 (commit a1d285c7) moved the test to weekly CI as a mitigation, but task.yaml still carries 'manual: true # This test is somewhat flaky' and no root-cause fix for the concurrent base-instance race has landed. The issue has no labels, no comments, and no assignee after ~3 years, so a maintainer needs to decide whether to close it as mitigated or invest in fixing the underlying race.
### Bug Description
The parallel build smoke test doesn't always succeed.
### To Reproduce
Run the smoke tests in Google
### Environment
Google environment (cannot replicate on multipass on my machine
### charmcraft.yaml
```shell
N/A
```
### Relevant log output
```shell
2023-10-17 01:04:32 Error executing google:ubuntu-20.04-64:tests/spread/smoketests/parallel-build (oct170052-258541) :
-----
+ sleep 60
+ cd charm2
+ charmcraft pack --verbose
+ cd charm1
+ charmcraft pack --verbose
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.512686.log'
Packing the charm.
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.511956.log'
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Creating new instance from remote
Creating new base instance from remote
Creating new instance from remote
Creating new base instance from remote
Failed to launch instance 'base-instance-charmcraft-buildd-base-v20-2046f14fb4a00aa9ce50'.
Full execution log: '/root/.local/state/charmcraft/log/charmcraft-20231017-005949.511956.log'
+ wait
+ cd charm3
+ charmcraft pack --verbose
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20231017-010051.612722.log'
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Creating new instance from base instance
Creating instance from base instance
Starting instance
Starting instance
Packing the charm
Packing the charm
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/tmp/charmcraft.log'
Packing the charm.
Building charm in '/root'
Installing build-packages
Starting charmcraft version 2.4.1.post77+git81e1ff9
Logging execution to '/tmp/charmcraft.log'
Packing the charm.
Building charm in '/root'
Installing build-packages
Running step PULL for part 'charm'
Execute action
Running step BUILD for part 'charm'
Execute action
:: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post77+git81e1ff9 /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt
Running step PULL for part 'charm'
Execute action
Running step BUILD for part 'charm'
Execute action
:: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post77+git81e1ff9 /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt
:: Starting charm builder
:: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z'
:: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z'
:: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z'
:: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z'
:: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z'
:: Translated .jujuignore 8 '/build/' => '/build\\Z'
:: Translated .jujuignore 9 '/revision' => '/revision\\Z'
:: Translated .jujuignore 10 '/venv' => '/venv\\Z'
:: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z'
:: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z'
:: Collected charmlib dependencies: set()
:: Building charm in '/root/parts/charm/install'
:: Linking in generic paths
:: Creating the dispatch mechanism
:: Creating the 'upgrade-charm' hook script pointing to dispatch
:: Creating the 'install' hook script pointing to dispatch
:: Creating the 'start' hook script pointing to dispatch
:: Handling dependencies
:: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d'
:: Dependencies directory not found
:: Installing dependencies
:: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv']
:: Starting charm builder
:: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z'
:: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z'
:: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z'
:: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z'
:: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z'
:: Translated .jujuignore 8 '/build/' => '/build\\Z'
:: Translated .jujuignore 9 '/revision' => '/revision\\Z'
:: Translated .jujuignore 10 '/venv' => '/venv\\Z'
:: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z'
:: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z'
:: Collected charmlib dependencies: set()
:: Building charm in '/root/parts/charm/install'
:: Linking in generic paths
:: Creating the dispatch mechanism
:: Creating the 'install' hook script pointing to dispatch
:: Creating the 'start' hook script pointing to dispatch
:: Creating the 'upgrade-charm' hook script pointing to dispatch
:: Handling dependencies
:: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d'
:: Dependencies directory not found
:: Installing dependencies
:: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv']
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel']
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel']
:: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2)
:: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0)
:: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2)
:: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0)
:: :: Collecting wheel
:: :: Collecting wheel
:: :: Downloading wheel-0.41.2.tar.gz (98 kB)
:: :: Downloading wheel-0.41.2.tar.gz (98 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.5 MB/s eta 0:00:00
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.5 MB/s eta 0:00:00
:: :: Installing build dependencies: started
:: :: Installing build dependencies: started
:: :: Installing build dependencies: finished with status 'done'
:: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Collecting ops~=2.5
:: :: Collecting ops~=2.5
:: :: Downloading ops-2.7.0.tar.gz (240 kB)
:: :: Downloading ops-2.7.0.tar.gz (240 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.2/240.2 KB 6.0 MB/s eta 0:00:00
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.2/240.2 KB 6.0 MB/s eta 0:00:00
:: :: Installing build dependencies: started
:: :: Installing build dependencies: started
:: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Collecting websocket-client==1.*
:: :: Downloading websocket_client-1.6.4-py3-none-any.whl (57 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.3/57.3 KB 6.6 MB/s eta 0:00:00
:: :: Collecting PyYAML==6.*
:: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
:: :: Collecting PyYAML==6.*
:: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 8.7 MB/s eta 0:00:00
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 7.6 MB/s eta 0:00:00
:: :: Collecting websocket-client==1.*
:: :: Using cached websocket_client-1.6.4-py3-none-any.whl (57 kB)
:: :: Building wheels for collected packages: wheel, ops
:: :: Building wheel for wheel (pyproject.toml): started
:: :: Building wheels for collected packages: wheel, ops
:: :: Building wheel for wheel (pyproject.toml): started
:: :: Building wheel for wheel (pyproject.toml): finished with status 'done'
:: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8
:: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90
:: :: Building wheel for ops (pyproject.toml): started
:: :: Building wheel for wheel (pyproject.toml): finished with status 'done'
:: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8
:: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90
:: :: Building wheel for ops (pyproject.toml): started
:: :: Building wheel for ops (pyproject.toml): finished with status 'done'
:: :: Created wheel for ops: filename=ops-2.7.0-py3-none-any.whl size=146985 sha256=d20224550648efe9859f8ed90fdbcf617431e7af30ff4b56391216a9fc9710e8
:: :: Stored in directory: /root/.cache/pip/wheels/98/fb/70/4c89833e50cef8b667100e5fdf9378d658284e9659803980ba
:: :: Successfully built wheel ops
:: :: Installing collected packages: wheel, websocket-client, PyYAML, ops
:: :: Building wheel for ops (pyproject.toml): finished with status 'done'
:: :: Created wheel for ops: filename=ops-2.7.0-py3-none-any.whl size=146985 sha256=d20224550648efe9859f8ed90fdbcf617431e7af30ff4b56391216a9fc9710e8
:: :: Stored in directory: /root/.cache/pip/wheels/98/fb/70/4c89833e50cef8b667100e5fdf9378d658284e9659803980ba
:: :: Successfully built wheel ops
:: :: Installing collected packages: wheel, websocket-client, PyYAML, ops
:: :: Successfully installed PyYAML-6.0.1 ops-2.7.0 websocket-client-1.6.4 wheel-0.41.2
:: :: Successfully installed PyYAML-6.0.1 ops-2.7.0 websocket-client-1.6.4 wheel-0.41.2
Running step STAGE for part 'charm'
Execute action
Running step STAGE for part 'charm'
Execute action
Running step PRIME for part 'charm'
Execute action
Running step PRIME for part 'charm'
Execute action
Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language).
Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework).
Creating the package itself
Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language).
Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework).
Creating the package itself
Created 'charm3_ubuntu-22.04-amd64.charm'.
Created 'charm1_ubuntu-22.04-amd64.charm'.
Charm packed ok
Charms packed:
charm1_ubuntu-22.04-amd64.charm
Charm packed ok
Charms packed:
charm3_ubuntu-22.04-amd64.charm
+ pushd charm1
/charmcraft/tests/spread/smoketests/parallel-build/charm1 /charmcraft/tests/spread/smoketests/parallel-build
+ test -f charm1_ubuntu-22.04-amd64.charm
+ MATCH venv/ops/charm.py
+ unzip -l charm1_ubuntu-22.04-amd64.charm
+ test '!' -d build
+ popd
/charmcraft/tests/spread/smoketests/parallel-build
+ pushd charm2
/charmcraft/tests/spread/smoketests/parallel-build/charm2 /charmcraft/tests/spread/smoketests/parallel-build
+ test -f 'charm*.charm'
+ echo 'charm2 failed to build'
charm2 failed to build
+ exit 1
-----
.
```
Evaluation history
Date
Model
Scores
Action
Summary
qwen/qwen3.8-27b
Impact:
20
Quick Win:
9.0
Staleness:
60
Complexity:
55
Confidence:
75
Support Request:
0
needs triage
Flaky parallel-build spread test (concurrent base-instance creation fails on Google CI). Mitigated by moving to weekly CI in #1600, but root cause unfixed and test still marked flaky. No labels, comments, or assignee after ~3 years.
qwen/qwen3.8-27b
Impact:
30
Quick Win:
9.0
Staleness:
70
Complexity:
70
Confidence:
75
Support Request:
0
keep open
Flaky parallel-build spread smoke test in Google CI: concurrent charmcraft pack runs race on LXC base-instance creation, causing 'Failed to launch instance' and a missing charm2 artifact. Mitigated by moving the test to a weekly manual workflow, not fixed.
qwen/qwen3.6-35b-a3b
Staleness:
90
Complexity:
40
Confidence:
65
Support Request:
5
needs triage
CI parallel build smoke test fails intermittently in Google environment. Opened by maintainer nearly three years ago with zero comments, no labels, and no activity for 340 days. Requires maintainer review to assess current relevance.
qwen3.6-35b-a3b-mtp-q6
Staleness:
92
Complexity:
65
Confidence:
75
Support Request:
5
needs triage
CI parallel build smoke test fails intermittently in Google Cloud due to instance launch failures. Open for nearly three years with no maintainer labels or comments.
qwen3.6-35b-a3b-mtp-q6
Staleness:
90
Complexity:
40
Confidence:
90
Support Request:
0
close stale
Flaky CI parallel build test in Google environment showing instance launch failures. No maintainer response in over a year.
qwen3.6-35b-a3b-mtp-q6
Staleness:
90
Complexity:
50
Confidence:
75
Support Request:
5
close stale
CI parallel build smoke test fails intermittently in Google environments due to concurrent instance launch errors. Open, unassigned, and stalled for over 300 days. Requires investigation or triage to diagnose and fix.
The flaky parallel build test failure was resolved. The issue closed after craft-providers 1.18 fixed the underlying race condition causing incompatible base detection errors in CI.
Merged to address issue #1332 by relocating a flaky parallel-build test to the weekly suite. Approved by two reviewers, the change modifies two files (+19/-2) to stabilize CI runs.