← Back to issue list

macOS CI is failing weekly tests

View original Github issue

Metadata

Project
craft-providers
Number
#467
Type
issue
State
open
Author
mr-cal
Labels
Type: Bug
Created
Updated
Closed
—

Current evaluation

Maintainer-tracked CI issue: macOS runners drop connection after ~1h losing all logs, so flaky macOS integration tests were pushed out of PR CI (#452) into the weekly workflow. Still open, no logs or reproducer captured.

Suggested action: needs triage

Reason: The report is labelled 'Type: Bug' but has0 comments, no assignee, and no activity for 163 days, and every evidence field is empty ('To Reproduce: n/a', 'Relevant log output: n/a'), so there is no record of it ever being assessed or diagnosed. The only concrete lead is indirect: Makefile lines 120-126 state test_multipass_instance.py 'consistently fail[s] and exhaust[s] [its] reruns, which multiplies out over many hours', which plausibly explains the ~1h runner disconnect, but nothing in the issue ties the two. A maintainer needs to decide scope before work can start — shard/timeout the weekly macOS jobs (integration-slow-macos still has no timeout-minutes at all) or treat it as upstream GitHub-hosted-runner flakiness that is not fixable in-repo.

Evaluated by: mr-cal-bot

Impact: 35 Quick Win: 3 Complexity: 70 Confidence: 58 Actionability: 25

Issue body

### Bug Description The macOS tests in PRs were failing (see #452). This was addressed by 3 PRs: #458, #459, and #460. Now, the failing tests in the PRs have been moved to a failing weekly test. The macOS runner loses connection after an hour (±15 minutes) and all logs are lost. ### To Reproduce n/a ### part yaml _No response_ ### Relevant log output ```shell n/a ```

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha
Impact: 35
Quick Win: 3
Complexity: 70
Confidence: 58
Actionability: 25
needs triage Maintainer-tracked CI issue: macOS runners drop connection after ~1h losing all logs, so flaky macOS integration tests were pushed out of PR CI (#452) into the weekly workflow. Still open, no logs or reproducer captured.
stealth/space-bunny-alpha
Impact: 38
Quick Win: 7
Complexity: 62
Confidence: 68
Actionability: 48
keep open Maintainer-tracked CI infra bug: macOS weekly multipass jobs (tests-weekly.yaml, macos-14-large) drop connection after ~1h and lose logs; PR CI stabilized by #458-#460 but root cause open.
stealth/space-bunny-alpha
Impact: 32
Quick Win: 5
Complexity: 60
Confidence: 57
Actionability: 38
keep open macOS weekly test job fails because the runner drops connection after ~60min and loses all logs; PR CI stabilized via #458-#460 but root cause remains unfixed
stealth/space-bunny-alpha
Impact: 45
Quick Win: 6
Complexity: 70
Confidence: 84
Actionability: 45
keep open macOS weekly multipass tests disconnect after roughly 60 minutes and lose all logs; PR CI was stabilized by #458–#460, but the underlying runner failure remains unresolved.
qwen3.6-35b-a3b-mtp-q6
Impact: 30
Quick Win: 18
Staleness: 85
Complexity: 40
Confidence: 65
Support Request: 10
close stale macOS CI weekly tests failing due to runner timeouts; tests temporarily disabled via PR #982; issue remains open but inactive for 136 days.
qwen/qwen3.6-35b-a3b
Staleness: 85
Complexity: 30
Confidence: 75
Support Request: 0
close stale macOS CI weekly tests failing due to runner connection loss. Issue is 965 days old with no comments, likely stale.
qwen3.6-35b-a3b-mtp-q6
Staleness: 85
Complexity: 35
Confidence: 75
Support Request: 5
needs triage macOS CI weekly tests fail due to runner connection drops after ~1 hour, causing lost logs. Open for over two years with no maintainer comments or assignee.

Update history

No update history recorded yet.

Related work

  • Related To: craft-providers#452 (confidence 95%)

    Explicitly cited in the body as the parent issue; its summary confirms macOS PR CI failures remain open after an accidental closure, so the two are still entangled.

  • Likely Fixed By: craft-providers#460 (confidence 60%)

    Commit bc946f0 'ci(github): add weekly multipass integration tests (#460)' created tests-weekly.yaml — the mitigation described in the body, but it relocated the failures rather than fixing them, and those jobs still exist.

  • Related To: craft-providers#982 (confidence 55%)

    Commit a3af10b 'ci: temporarily disable macos smoke tests (#982)' commented out the macos-smoketest and test-multipass jobs in qa.yaml, further shrinking macOS coverage this issue tracks.

Related issues

Issue Project State Summary Similarity
#452 macOS CI is failing in PRs craft-providers open macOS PR CI fails without logs; qa.yaml multipass/smoketest jobs commented out as a temporary workaround (#982), but root cause (runner timeout/disconnect) unfixed and weekly macOS jobs still fail per #467. Reopened by bepri.
81%
#460 tests(ci): add weekly slow tests for macOS craft-providers merged Merged weekly slow macOS integration tests into CI via PR #460, commit bc946f0.
72%