CI: `charm-upload-and-release` test is flaky
Metadata
Current evaluation
Added propagation delays and status retries in PR #1366 (commit 05b18a03); later store-test hardening landed in PR #1460 (fd5bd6ba).
Resolution: closed resolved
Reason: PR #1366 updated the test to wait before checking the edge channel and retry up to 10 times, resolving Charm Store propagation delays. Commit 05b18a03 contains the fix, with further store-test robustness added in fd5bd6ba via PR #1460.
Evaluated by: mr-cal-bot
No scores available.
Issue body
### Bug Description
The charm upload test seems to be fairly frequently failing due to the charm store reporting the release as having updated immediately after publish.
Likely we just need to sleep for a few seconds and maybe do some retries to prevent the test from getting the revision before the values have propagated.
### To Reproduce
`spread google:ubuntu-22.04-64:tests/spread/store/charm-upload-and-release`
See: https://github.com/canonical/charmcraft/actions/runs/6644905285/job/18055442120?pr=1352
### Environment
CI
### charmcraft.yaml
```shell
N/A
```
### Relevant log output
```shell
++ charmcraft upload gh-ci-charmcraft-charm_ubuntu-22.04-amd64.charm --format=json
Starting to push 'gh-ci-charmcraft-charm_ubuntu-22.04-amd64.charm'
Uploading... (--->)
Uploading... (<---)
Uploading bytes ended, id 00bbc7f6-1081-45a1-8ee0-e45f381ccabc
Upload 00bbc7f6-1081-45a1-8ee0-e45f381ccabc started, got status url /v1/charm/gh-ci-charmcraft-charm/revisions/review?upload-id=00bbc7f6-1081-45a1-8ee0-e45f381ccabc
Status checked: {'revisions': [{'errors': None, 'revision': None, 'status': 'new', 'upload-id': '00bbc7f6-1081-45a1-8ee0-e45f381ccabc'}]}
Status checked: {'revisions': [{'errors': None, 'revision': 1025, 'status': 'scanned', 'upload-id': '00bbc7f6-1081-45a1-8ee0-e45f381ccabc'}]}
Status checked: {'revisions': [{'errors': None, 'revision': 1025, 'status': 'approved', 'upload-id': '00bbc7f6-1081-45a1-8ee0-e45f381ccabc'}]}
+ uploaded_revno=1025
++ jq -r --arg revno 1025 '.[] | select(.revision|tostring==$revno)'
++ charmcraft revisions gh-ci-charmcraft-charm --format=json
+ up_revision='{
"revision": 1025,
"version": "1025",
"created_at": "2023-10-25T19:15:16Z",
"status": "approved"
}'
++ jq -r .created_at
++ echo '{' '"revision":' 1025, '"version":' '"1025",' '"created_at":' '"2023-10-25T19:15:16Z",' '"status":' '"approved"' '}'
+ up_revision_created=2023-10-25T19:15:16Z
+ [[ 2023-10-25T19:12:13+00:00 < 2023-10-25T19:15:16Z ]]
+ charmcraft release gh-ci-charmcraft-charm -r 1025 -c edge
Revision 1025 of 'gh-ci-charmcraft-charm' released to edge
++ jq -r '.[] | select(.track=="latest") | .mappings[0].releases | .[] | select(.channel=="latest/edge")'
++ charmcraft status gh-ci-charmcraft-charm --format=json
+ edge_release='{
"status": "open",
"channel": "latest/edge",
"version": "1024",
"revision": 1024,
"resources": [
{
"name": "example-file",
"revision": 1
},
{
"name": "example-image",
"revision": 1
}
],
"expires_at": null
}'
++ jq -r .revision
++ echo '{' '"status":' '"open",' '"channel":' '"latest/edge",' '"version":' '"1024",' '"revision":' 1024, '"resources":' '[' '{' '"name":' '"example-file",' '"revision":' 1 '},' '{' '"name":' '"example-image",' '"revision":' 1 '}' '],' '"expires_at":' null '}'
+ edge_revision=1024
+ test 1024 -ge 1025
```
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Added propagation delays and status retries in PR #1366 (commit 05b18a03); later store-test hardening landed in PR #1460 (fd5bd6ba). | |
| stealth/space-bunny-alpha | — | closed resolved | Fixed the flaky charm-upload-and-release Spread test in PR #1460 (commit fd5bd6ba), making store tests less timing-sensitive. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Closed without resolution, likely due to inactivity. The flaky CI test was not fixed, and no pull request or workaround was implemented. | |
| qwen/qwen3.6-35b-a3b | — | — | Abandoned and closed due to inactivity. The flaky CI test failure, caused by a race condition in charm store release propagation, remains unaddressed. Proposed mitigations like adding delays or retries were never implemented. | |
| qwen/qwen3.6-35b-a3b | — | — | Closed without resolution. The flaky CI test charm-upload-and-release was abandoned after the maintainer suggested adding sleep and retry logic to handle charm store propagation delays. No fixes or discussions were implemented. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Closed without resolution. The flaky charm-upload-and-release CI test failed due to charm store propagation delays. A sleep and retry workaround was proposed, but no fix was merged or implemented. |
Update history
No update history recorded yet.
Related work
-
Likely Fixed By:
canonical/charmcraft#1366
(confidence 99%)
Commit 05b18a03 made the release test less flaky by adding delays and retries.
-
Related To:
canonical/charmcraft#1460
(confidence 95%)
Commit fd5bd6ba later made the broader set of store Spread tests less timing-sensitive.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1469 Charm upload/release spread test is flaky | charmcraft | closed | Fixed the flaky store spread test by waiting and retrying release checks; merged in canonical/charmcraft#1460 (fd5bd6ba). | |
| #1429 spread: charm upload and release store test is flaky when the store is busy | charmcraft | closed | Fixed in merged PR #1460 (commit fd5bd6ba), which reduced timing sensitivity in store upload and release Spread tests. |