← Back to issue list

Parallel build test is flaky

View original Github issue

Metadata

Project
charmcraft
Number
#1295
Type
issue
State
closed
Author
lengau
Labels
Created
Updated
Closed

Current evaluation

The flaky parallel build test failure was resolved. The issue closed after craft-providers 1.18 fixed the underlying race condition causing incompatible base detection errors in CI.

Suggested action:

No scores available.

Issue body

### Bug Description The parallel build spread test seems to be failing a lot in CI. ### To Reproduce `spread google:ubuntu-22.04-64:tests/spread/smoketests/parallel-build` ### Environment CI ### charmcraft.yaml ```shell n/a ``` ### Relevant log output ```shell 2023-09-28 20:12:52 Error executing google:ubuntu-22.04-64:tests/spread/smoketests/parallel-build (sep282006-904183) : ----- + sleep 60 + cd charm2 + charmcraft pack --verbose + cd charm1 + charmcraft pack --verbose Starting charmcraft version 2.4.1.post50+git959209b Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20230928-200838.810338.log' Starting charmcraft version 2.4.1.post50+git959209b Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20230928-200838.813509.log' Packing the charm. Packing the charm. Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable) Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable) Creating new instance from remote Creating new base instance from remote Creating new instance from remote Creating new base instance from remote + wait + cd charm3 + charmcraft pack --verbose Starting charmcraft version 2.4.1.post50+git959209b Logging execution to '/root/.local/state/charmcraft/log/charmcraft-20230928-200939.375926.log' Packing the charm. Launching environment to pack for base name='ubuntu' channel='22.04' architectures=['amd64'] (may take a while the first time but it's reusable) Creating instance from base instance Starting instance Incompatible base detected: instance is marked as not setup. Recommended resolution: Clean incompatible instance and retry the requested operation. Full execution log: '/root/.local/state/charmcraft/log/charmcraft-20230928-200939.375926.log' Creating new instance from base instance Creating new instance from base instance Starting instance Starting instance Packing the charm Packing the charm Starting charmcraft version 2.4.1.post50+git959209b Logging execution to '/tmp/charmcraft.log' Starting charmcraft version 2.4.1.post50+git959209b Logging execution to '/tmp/charmcraft.log' Packing the charm. Building charm in '/root' Installing build-packages Packing the charm. Building charm in '/root' Installing build-packages Running step PULL for part 'charm' Execute action Running step BUILD for part 'charm' Execute action :: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post50+git959209b /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt Running step PULL for part 'charm' Execute action Running step BUILD for part 'charm' Execute action :: + env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/charmcraft/x1/libexec/charmcraft:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin SNAP=/snap/charmcraft/x1 SNAP_ARCH=amd64 SNAP_NAME=charmcraft SNAP_VERSION=2.4.1.post50+git959209b /snap/charmcraft/x1/bin/python3 -u -I /snap/charmcraft/x1/lib/charmcraft/charm_builder.py --builddir /root/parts/charm/build --installdir /root/parts/charm/install --entrypoint /root/parts/charm/build/src/charm.py -p pip -p setuptools -p wheel -r requirements.txt :: Starting charm builder :: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z' :: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z' :: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z' :: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z' :: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z' :: Translated .jujuignore 8 '/build/' => '/build\\Z' :: Translated .jujuignore 9 '/revision' => '/revision\\Z' :: Translated .jujuignore 10 '/venv' => '/venv\\Z' :: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z' :: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z' :: Collected charmlib dependencies: set() :: Building charm in '/root/parts/charm/install' :: Linking in generic paths :: Creating the dispatch mechanism :: Creating the 'start' hook script pointing to dispatch :: Creating the 'upgrade-charm' hook script pointing to dispatch :: Creating the 'install' hook script pointing to dispatch :: Handling dependencies :: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d' :: Dependencies directory not found :: Installing dependencies :: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv'] :: Starting charm builder :: Translated .jujuignore 2 '.git' => '.*/\\.git\\Z' :: Translated .jujuignore 3 '.svn' => '.*/\\.svn\\Z' :: Translated .jujuignore 4 '.hg' => '.*/\\.hg\\Z' :: Translated .jujuignore 5 '.bzr' => '.*/\\.bzr\\Z' :: Translated .jujuignore 6 '.tox' => '.*/\\.tox\\Z' :: Translated .jujuignore 8 '/build/' => '/build\\Z' :: Translated .jujuignore 9 '/revision' => '/revision\\Z' :: Translated .jujuignore 10 '/venv' => '/venv\\Z' :: Translated .jujuignore 12 '.jujuignore' => '.*/\\.jujuignore\\Z' :: Translated .jujuignore 1 '/staging-venv' => '/staging\\-venv\\Z' :: Collected charmlib dependencies: set() :: Building charm in '/root/parts/charm/install' :: Linking in generic paths :: Creating the dispatch mechanism :: Creating the 'upgrade-charm' hook script pointing to dispatch :: Creating the 'start' hook script pointing to dispatch :: Creating the 'install' hook script pointing to dispatch :: Handling dependencies :: Current dependencies hash: '5ab110c82c25c8cf0dad7b8e9c827444078a759d' :: Dependencies directory not found :: Installing dependencies :: Running external command ['python3', '-m', 'venv', '/root/parts/charm/build/staging-venv'] :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel'] :: Running external command ['/root/parts/charm/build/staging-venv/bin/pip', 'install', '--no-binary=ops,pip,setuptools,wheel', '--requirement=requirements.txt', 'pip', 'setuptools', 'wheel'] :: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2) :: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0) :: :: Requirement already satisfied: pip in ./staging-venv/lib/python3.10/site-packages (22.0.2) :: :: Requirement already satisfied: setuptools in ./staging-venv/lib/python3.10/site-packages (59.6.0) :: :: Collecting wheel :: :: Downloading wheel-0.41.2.tar.gz (98 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.0/98.0 KB 1.7 MB/s eta 0:00:00 :: :: Collecting wheel :: :: Using cached wheel-0.41.2.tar.gz (98 kB) :: :: Installing build dependencies: started :: :: Installing build dependencies: started :: :: Installing build dependencies: finished with status 'done':: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Collecting ops~=2.5 :: :: Collecting ops~=2.5 :: :: Downloading ops-2.6.0.tar.gz (239 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 KB 9.9 MB/s eta 0:00:00 :: :: Downloading ops-2.6.0.tar.gz (239 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 KB 3.5 MB/s eta 0:00:00 :: :: Installing build dependencies: started :: :: Installing build dependencies: started :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Collecting websocket-client==1.* :: :: Downloading websocket_client-1.6.3-py3-none-any.whl (57 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.3/57.3 KB 9.3 MB/s eta 0:00:00 :: :: Collecting PyYAML==6.* :: :: Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB) :: :: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 KB 28.5 MB/s eta 0:00:00 :: :: Building wheels for collected packages: wheel, ops :: :: Building wheel for wheel (pyproject.toml): started :: :: Building wheel for wheel (pyproject.toml): finished with status 'done' :: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8 :: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90 :: :: Building wheel for ops (pyproject.toml): started :: :: Installing build dependencies: finished with status 'done' :: :: Getting requirements to build wheel: started :: :: Building wheel for ops (pyproject.toml): finished with status 'done' :: :: Created wheel for ops: filename=ops-2.6.0-py3-none-any.whl size=146675 sha256=1bca32e7eabc7d448298b3bfc20f82cba0ac6824efa5878279d9c0f5057420fe :: :: Stored in directory: /root/.cache/pip/wheels/5d/6e/a1/df401c8bc69df5a67e17e17fff84687d939fc8d68fbd9c323d :: :: Successfully built wheel ops :: :: Installing collected packages: wheel, websocket-client, PyYAML, ops :: :: Getting requirements to build wheel: finished with status 'done' :: :: Preparing metadata (pyproject.toml): started :: :: Preparing metadata (pyproject.toml): finished with status 'done' :: :: Successfully installed PyYAML-6.0.1 ops-2.6.0 websocket-client-1.6.3 wheel-0.41.2 :: :: Collecting PyYAML==6.* :: :: Using cached PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB) :: :: Collecting websocket-client==1.* :: :: Using cached websocket_client-1.6.3-py3-none-any.whl (57 kB) :: :: Building wheels for collected packages: wheel, ops :: :: Building wheel for wheel (pyproject.toml): started :: :: Building wheel for wheel (pyproject.toml): finished with status 'done' :: :: Created wheel for wheel: filename=wheel-0.41.2-py3-none-any.whl size=64848 sha256=75909db2664838d015e3d9139004ee16711748a52c8f336b52882266540215d8 :: :: Stored in directory: /root/.cache/pip/wheels/b2/5f/6b/dbad923fcf0045db2edbfe696e533deadceeb788b173cdec90 :: :: Building wheel for ops (pyproject.toml): started :: :: Building wheel for ops (pyproject.toml): finished with status 'done' :: :: Created wheel for ops: filename=ops-2.6.0-py3-none-any.whl size=146675 sha256=a11f27b75c56100385737082e22f4c76bce3f529c0b303cfd69613e6fb08c5d4 :: :: Stored in directory: /root/.cache/pip/wheels/5d/6e/a1/df401c8bc69df5a67e17e17fff84687d939fc8d68fbd9c323d :: :: Successfully built wheel ops :: :: Installing collected packages: wheel, websocket-client, PyYAML, ops :: :: Successfully installed PyYAML-6.0.1 ops-2.6.0 websocket-client-1.6.3 wheel-0.41.2 Running step STAGE for part 'charm' Execute action Running step PRIME for part 'charm' Execute action Running step STAGE for part 'charm' Execute action Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language). Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework). Creating the package itself Running step PRIME for part 'charm' Execute action Check result: language [attribute] python (The charm is written with Python.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--language). Check result: framework [attribute] operator (The charm is based on the Operator Framework.; see more at https://juju.is/docs/sdk/charmcraft-analyzers-and-linters#heading--framework). Creating the package itself Created 'charm2_ubuntu-22.04-amd64.charm'. Created 'charm1_ubuntu-22.04-amd64.charm'. Charm packed ok Charms packed: charm1_ubuntu-22.04-amd64.charm Charm packed ok Charms packed: charm2_ubuntu-22.04-amd64.charm + pushd charm1 /charmcraft/tests/spread/smoketests/parallel-build/charm1 /charmcraft/tests/spread/smoketests/parallel-build + test -f charm1_ubuntu-22.04-amd64.charm + MATCH venv/ops/charm.py + unzip -l charm1_ubuntu-22.04-amd64.charm + test '!' -d build + popd /charmcraft/tests/spread/smoketests/parallel-build + pushd charm2 /charmcraft/tests/spread/smoketests/parallel-build/charm2 /charmcraft/tests/spread/smoketests/parallel-build + test -f charm2_ubuntu-22.04-amd64.charm + MATCH venv/ops/charm.py + unzip -l charm2_ubuntu-22.04-amd64.charm + test '!' -d build + popd /charmcraft/tests/spread/smoketests/parallel-build + pushd charm3 /charmcraft/tests/spread/smoketests/parallel-build/charm3 /charmcraft/tests/spread/smoketests/parallel-build + test -f 'charm*.charm' + echo 'charm3 failed to build' charm3 failed to build + exit 1 ----- ```

Evaluation history

Date Model Scores Action Summary
qwen3.6-35b-a3b-mtp-q6 The flaky parallel build test failure was resolved. The issue closed after craft-providers 1.18 fixed the underlying race condition causing incompatible base detection errors in CI.
qwen/qwen3.6-35b-a3b Resolved by upgrading to craft-providers 1.18, which fixed the incompatible base instance errors causing the flaky parallel build test failures in CI.
qwen/qwen3.6-35b-a3b Closed and resolved by upgrading to craft-providers 1.18, which fixed the incompatible base instance state causing concurrent charmcraft pack failures in the parallel build test.
qwen3.6-35b-a3b-mtp-q6 Resolved by upgrading to craft-providers 1.18, which fixed the underlying race condition causing the flaky parallel build test failures in CI.
qwen3.6-35b-a3b-mtp-q6 The flaky parallel build test failure was resolved and the issue closed. The root cause was an incompatible base instance state in craft-providers, which was fixed in version 1.18.

Update history

No update history recorded yet.

Related issues

Issue Project State Summary Similarity
#1332 (CI) Parallel build test is flaky charmcraft open Flaky parallel-build spread test (concurrent base-instance creation fails on Google CI). Mitigated by moving to weekly CI in #1600, but root cause unfixed and test still marked flaky. No labels, comments, or assignee after ~3 years.
75%