Parallel build test is flaky
Metadata
Current evaluation
The flaky parallel build test failure was resolved. The issue closed after craft-providers 1.18 fixed the underlying race condition causing incompatible base detection errors in CI.
Suggested action: —
No scores available.
Issue body
### Bug Description
The parallel build spread test seems to be failing a lot in CI.
### To Reproduce
`spread google:ubuntu-22.04-64:tests/spread/smoketests/parallel-build`
### Environment
CI
### charmcraft.yaml
```shell
n/a
```
### Relevant log output
```shell
2023-09-28 20:12:52 Error executing google:ubuntu-22.04-64:tests/spread/smoketests/parallel-build (sep282006-904183) :
-----
+ sleep 60
+ cd charm2
+ charmcraft pack --verbose
+ cd charm1
+ charmcraft pack --verbose
Starting charmcraft version 2.4.1.post50+git959209b
Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20230928-200838.810338.log'
Starting charmcraft version 2.4.1.post50+git959209b
Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20230928-200838.813509.log'
Packing the charm.
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Creating new instance from remote
Creating new base instance from remote
Creating new instance from remote
Creating new base instance from remote
+ wait
+ cd charm3
+ charmcraft pack --verbose
Starting charmcraft version 2.4.1.post50+git959209b
Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20230928-200939.375926.log'
Packing the charm.
Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable)
Creating instance from base instance
Starting instance
Incompatible base detected: instance is marked as not setup.
Recommended resolution: Clean incompatible instance and retry the requested operation.
Full execution log: '/root/.local/state/charmcraft/log/charmcraft-20230928-200939.375926.log'
Creating new instance from base instance
Creating new instance from base instance
Starting instance
Starting instance
Packing the charm
Packing the charm
Starting charmcraft version 2.4.1.post50+git959209b
Logging execution to '/tmp/charmcraft.log'
Starting charmcraft version 2.4.1.post50+git959209b
Logging execution to '/tmp/charmcraft.log'
Packing the charm.
Building charm in '/root'
Installing build-packages
Packing the charm.
Building charm in '/root'
Installing build-packages
Running step PULL for part 'charm'
Execute action
Running step BUILD for part 'charm'
Execute action
:: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post50+git959209b /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt
Running step PULL for part 'charm'
Execute action
Running step BUILD for part 'charm'
Execute action
:: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post50+git959209b /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt
:: Starting charm builder
:: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z'
:: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z'
:: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z'
:: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z'
:: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z'
:: Translated .jujuignore 8 '/build/' => '/build\\Z'
:: Translated .jujuignore 9 '/revision' => '/revision\\Z'
:: Translated .jujuignore 10 '/venv' => '/venv\\Z'
:: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z'
:: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z'
:: Collected charmlib dependencies: set()
:: Building charm in '/root/parts/charm/install'
:: Linking in generic paths
:: Creating the dispatch mechanism
:: Creating the 'start' hook script pointing to dispatch
:: Creating the 'upgrade-charm' hook script pointing to dispatch
:: Creating the 'install' hook script pointing to dispatch
:: Handling dependencies
:: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d'
:: Dependencies directory not found
:: Installing dependencies
:: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv']
:: Starting charm builder
:: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z'
:: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z'
:: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z'
:: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z'
:: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z'
:: Translated .jujuignore 8 '/build/' => '/build\\Z'
:: Translated .jujuignore 9 '/revision' => '/revision\\Z'
:: Translated .jujuignore 10 '/venv' => '/venv\\Z'
:: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z'
:: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z'
:: Collected charmlib dependencies: set()
:: Building charm in '/root/parts/charm/install'
:: Linking in generic paths
:: Creating the dispatch mechanism
:: Creating the 'upgrade-charm' hook script pointing to dispatch
:: Creating the 'start' hook script pointing to dispatch
:: Creating the 'install' hook script pointing to dispatch
:: Handling dependencies
:: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d'
:: Dependencies directory not found
:: Installing dependencies
:: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv']
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel']
:: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel']
:: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2)
:: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0)
:: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2)
:: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0)
:: :: Collecting wheel
:: :: Downloading wheel-0.41.2.tar.gz (98 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.7 MB/s eta 0:00:00
:: :: Collecting wheel
:: :: Using cached wheel-0.41.2.tar.gz (98 kB)
:: :: Installing build dependencies: started
:: :: Installing build dependencies: started
:: :: Installing build dependencies: finished with status 'done':: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Collecting ops~=2.5
:: :: Collecting ops~=2.5
:: :: Downloading ops-2.6.0.tar.gz (239 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 KB 9.9 MB/s eta 0:00:00
:: :: Downloading ops-2.6.0.tar.gz (239 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 KB 3.5 MB/s eta 0:00:00
:: :: Installing build dependencies: started
:: :: Installing build dependencies: started
:: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Collecting websocket-client==1.*
:: :: Downloading websocket_client-1.6.3-py3-none-any.whl (57 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.3/57.3 KB 9.3 MB/s eta 0:00:00
:: :: Collecting PyYAML==6.*
:: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
:: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 28.5 MB/s eta 0:00:00
:: :: Building wheels for collected packages: wheel, ops
:: :: Building wheel for wheel (pyproject.toml): started
:: :: Building wheel for wheel (pyproject.toml): finished with status 'done'
:: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8
:: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90
:: :: Building wheel for ops (pyproject.toml): started
:: :: Installing build dependencies: finished with status 'done'
:: :: Getting requirements to build wheel: started
:: :: Building wheel for ops (pyproject.toml): finished with status 'done'
:: :: Created wheel for ops: filename=ops-2.6.0-py3-none-any.whl size=146675 sha256=1bca32e7eabc7d448298b3bfc20f82cba0ac6824efa5878279d9c0f5057420fe
:: :: Stored in directory: /root/.cache/pip/wheels/5d/6e/a1/df401c8bc69df5a67e17e17fff84687d939fc8d68fbd9c323d
:: :: Successfully built wheel ops
:: :: Installing collected packages: wheel, websocket-client, PyYAML, ops
:: :: Getting requirements to build wheel: finished with status 'done'
:: :: Preparing metadata (pyproject.toml): started
:: :: Preparing metadata (pyproject.toml): finished with status 'done'
:: :: Successfully installed PyYAML-6.0.1 ops-2.6.0 websocket-client-1.6.3 wheel-0.41.2
:: :: Collecting PyYAML==6.*
:: :: Using cached PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
:: :: Collecting websocket-client==1.*
:: :: Using cached websocket_client-1.6.3-py3-none-any.whl (57 kB)
:: :: Building wheels for collected packages: wheel, ops
:: :: Building wheel for wheel (pyproject.toml): started
:: :: Building wheel for wheel (pyproject.toml): finished with status 'done'
:: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8
:: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90
:: :: Building wheel for ops (pyproject.toml): started
:: :: Building wheel for ops (pyproject.toml): finished with status 'done'
:: :: Created wheel for ops: filename=ops-2.6.0-py3-none-any.whl size=146675 sha256=a11f27b75c56100385737082e22f4c76bce3f529c0b303cfd69613e6fb08c5d4
:: :: Stored in directory: /root/.cache/pip/wheels/5d/6e/a1/df401c8bc69df5a67e17e17fff84687d939fc8d68fbd9c323d
:: :: Successfully built wheel ops
:: :: Installing collected packages: wheel, websocket-client, PyYAML, ops
:: :: Successfully installed PyYAML-6.0.1 ops-2.6.0 websocket-client-1.6.3 wheel-0.41.2
Running step STAGE for part 'charm'
Execute action
Running step PRIME for part 'charm'
Execute action
Running step STAGE for part 'charm'
Execute action
Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language).
Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework).
Creating the package itself
Running step PRIME for part 'charm'
Execute action
Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language).
Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework).
Creating the package itself
Created 'charm2_ubuntu-22.04-amd64.charm'.
Created 'charm1_ubuntu-22.04-amd64.charm'.
Charm packed ok
Charms packed:
charm1_ubuntu-22.04-amd64.charm
Charm packed ok
Charms packed:
charm2_ubuntu-22.04-amd64.charm
+ pushd charm1
/charmcraft/tests/spread/smoketests/parallel-build/charm1 /charmcraft/tests/spread/smoketests/parallel-build
+ test -f charm1_ubuntu-22.04-amd64.charm
+ MATCH venv/ops/charm.py
+ unzip -l charm1_ubuntu-22.04-amd64.charm
+ test '!' -d build
+ popd
/charmcraft/tests/spread/smoketests/parallel-build
+ pushd charm2
/charmcraft/tests/spread/smoketests/parallel-build/charm2 /charmcraft/tests/spread/smoketests/parallel-build
+ test -f charm2_ubuntu-22.04-amd64.charm
+ MATCH venv/ops/charm.py
+ unzip -l charm2_ubuntu-22.04-amd64.charm
+ test '!' -d build
+ popd
/charmcraft/tests/spread/smoketests/parallel-build
+ pushd charm3
/charmcraft/tests/spread/smoketests/parallel-build/charm3 /charmcraft/tests/spread/smoketests/parallel-build
+ test -f 'charm*.charm'
+ echo 'charm3 failed to build'
charm3 failed to build
+ exit 1
-----
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen3.6-35b-a3b-mtp-q6 | — | — | The flaky parallel build test failure was resolved. The issue closed after craft-providers 1.18 fixed the underlying race condition causing incompatible base detection errors in CI. | |
| qwen/qwen3.6-35b-a3b | — | — | Resolved by upgrading to craft-providers 1.18, which fixed the incompatible base instance errors causing the flaky parallel build test failures in CI. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed and resolved by upgrading to craft-providers 1.18, which fixed the incompatible base instance state causing concurrent charmcraft pack failures in the parallel build test. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Resolved by upgrading to craft-providers 1.18, which fixed the underlying race condition causing the flaky parallel build test failures in CI. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | The flaky parallel build test failure was resolved and the issue closed. The root cause was an incompatible base instance state in craft-providers, which was fixed in version 1.18. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1332 (CI) Parallel build test is flaky | charmcraft | open | Flaky parallel-build spread test (concurrent base-instance creation fails on Google CI). Mitigated by moving to weekly CI in #1600, but root cause unfixed and test still marked flaky. No labels, comments, or assignee after ~3 years. |