Updated directory missing in `charmcraft` pull step
Metadata
Current evaluation
Closed and resolved by fixing in-place list modification during iteration in craft_parts/sources/local_source.py. This corrected directory tracking during the charmcraft pull step, preventing build failures.
Suggested action: —
No scores available.
Issue body
Charmcraft logs in https://github.com/canonical/data-platform-workflows/issues/32 show
```
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:14.786 process charm:Step.PULL
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:14.786 check if Part('charm'):Step.PULL is dirty
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:14.786 check if Part('charm'):Step.PULL is outdated
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:14.786 ignore patterns: ['*.charm']
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:14.787 updated files: {'tests/unit/test_charm.py', 'metadata.yaml', 'tests/unit/test_tls_lib.py', 'tests/unit/__init__.py', '.gitignore', 'documentation/documentation_landing.md', 'src/machine_helpers.py', 'documentation/tutorial.md', 'tests/unit/test_mongodb_lib.py', 'tests/unit/test_mongodb_helpers.py', 'tests/unit/test_mongodb_backups.py', 'LICENSE', 'config.yaml', 'charmcraft.yaml', 'README.md', 'src/charm.py', '.jujuignore', 'actions.yaml', 'pyproject.toml', 'tests/unit/test_mongodb_provider.py', 'requirements.txt', 'tox.ini', 'tests/unit/helpers.py', 'CONTRIBUTING.md', 'src/grafana_dashboards/MongoDB_Cluster_Summary.json'}
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:14.787 updated directories: {'.github', 'tests/integration', 'tests/unit/data', 'src/alert_rules', 'tests/data', '.git', 'lib'}
```
Note that updated files contains (but should not contain) `'src/grafana_dashboards/MongoDB_Cluster_Summary.json'` and updated directories does not contain (but should contain) `'src/grafana_dashboards'`.
This causes this error later in the logs
```
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:15.747 Parts processing error: Failed to copy '/root/parts/charm/src/src/grafana_dashboards/MongoDB_Cluster_Summary.json': no such file or directory.
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:15.752 Traceback (most recent call last):
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/craft_parts/utils/file_utils.py", line 172, in copy
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:15.752 shutil.copy2(source, destination, follow_symlinks=follow_symlinks)
2023-03-30 14:49:16.047 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/usr/lib/python3.8/shutil.py", line 435, in copy2
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 copyfile(src, dst, follow_symlinks=follow_symlinks)
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/usr/lib/python3.8/shutil.py", line 264, in copyfile
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 FileNotFoundError: [Errno 2] No such file or directory: '/root/parts/charm/build/src/grafana_dashboards/MongoDB_Cluster_Summary.json'
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 The above exception was the direct cause of the following exception:
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 Traceback (most recent call last):
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/charmcraft/parts.py", line 397, in run
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 aex.execute([act], stdout=stream, stderr=stream)
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/craft_parts/executor/executor.py", line 301, in execute
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 self._executor.execute(actions, stdout=stdout, stderr=stderr)
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/craft_parts/executor/executor.py", line 126, in execute
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 self._run_action(act, stdout=stdout, stderr=stderr)
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/craft_parts/executor/executor.py", line 189, in _run_action
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 handler.run_action(action, stdout=stdout, stderr=stderr)
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/craft_parts/executor/part_handler.py", line 130, in run_action
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 self._update_action(
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/craft_parts/executor/part_handler.py", line 535, in _update_action
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 handler(step_info, stdout=stdout, stderr=stderr)
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/craft_parts/executor/part_handler.py", line 627, in _update_build
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 source.update()
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/craft_parts/sources/local_source.py", line 163, in update
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 self.copy_function(
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 File "/snap/charmcraft/1171/lib/craft_parts/utils/file_utils.py", line 174, in copy
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 raise errors.CopyFileNotFound(source) from err
2023-03-30 14:49:16.048 :: 2023-03-30 14:49:15.752 craft_parts.errors.CopyFileNotFound: Failed to copy '/root/parts/charm/src/src/grafana_dashboards/MongoDB_Cluster_Summary.json': no such file or directory.
```
Special attention to the first error in the traceback: `FileNotFoundError: [Errno 2] No such file or directory: '/root/parts/charm/build/src/grafana_dashboards/MongoDB_Cluster_Summary.json'`
I believe this is because the `/src/grafana_dashboards/` directory does not exist in `/root/parts/charm/build/`. To confirm this, I ran `sudo mkdir /var/snap/lxd/common/lxd/containers/charmcraft_charmcraft-mongodb-785608-0-0-amd64/rootfs/root/parts/charm/build/src/grafana_dashboards/` and ran `charmcraft pack` again and the build succeeded.
## Source of bug
I believe this issue is caused by this block of code:
https://github.com/canonical/craft-parts/blob/67d03ad7babc8ef82d1bbedd56b3ce5d8ae7153c/craft_parts/sources/local_source.py#L121-L126
The `directories` list is modified in-place while iterating over it, which means that not all items in the list will be iterated over.
For example:
```
>>> directories = [1, 2, 3, 4, 5]
>>> for directory in directories:
... print(directory)
... directories.remove(directory)
...
1
3
5
>>> print(directories)
[2, 4]
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| qwen/qwen3.6-35b-a3b | — | — | Closed and resolved by fixing in-place list modification during iteration in craft_parts/sources/local_source.py. This corrected directory tracking during the charmcraft pull step, preventing build failures. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed. Fixed by correcting in-place list modification during iteration in craft-parts, resolving the directory tracking bug that caused pull step failures. | |
| qwen/qwen3.6-35b-a3b | — | — | Fixed. Resolved an in-place list modification bug in local_source.py that caused directories to be misclassified as files during the pull step, preventing directory creation and build failures. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Closed and fixed. Resolved by correcting an in-place list modification bug in craft-parts that misclassified directories as files during the pull step, preventing the subsequent copy failure. |
Update history
No update history recorded yet.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #501 charmcraft pack fails trying to include a non-existing charm/src/.tox/ | charmcraft | closed | charmcraft pack failed when copying non-existent build directories or files. Users worked around it with charmcraft clean. The issue was closed as fixed upstream in the underlying parts lifecycle code. | |
| #514 `charmcraft build` silently omits files called out in `charmcraft.yaml` if executed from an external directory | charmcraft | closed | Fixed in PR #525. Resolved a bug where charmcraft build and pack silently omitted glob-patterned files when run externally. The fix corrects path misalignment between Charmcraft and the parts processor, ensuring proper glob expansion. |