← Back to issue list

remote build: stale git repositories on launchpad

View original Github issue

Metadata

Project
craft-application
Number
#1205
Type
issue
State
open
Author
enriqueesanchz
Labels
—
Created
Updated
Closed
—

Current evaluation

New, unlabelled report documents several remote-build paths orphaning about 20,000 Launchpad git repositories and proposes a best-effort, bounded cleanup sweep; no maintainer response yet.

Suggested action: needs triage

Reason: The code confirms that cleanup runs only after normal monitoring or confirmed cancellation, while start failures, timeouts, and declined cancellation retain resources; cleanup also deletes the recipe before the repository, allowing recipe deletion errors to prevent repository deletion. The detached-build reproducer and proposed sweep are concrete, but safe stale-resource criteria need maintainer review, and the issue is new, unlabelled, and has no maintainer response.

Evaluated by: mr-cal-bot

Impact: 64 Quick Win: 19 Complexity: 62 Confidence: 93 Actionability: 78

Issue body

### Bug Description Hi, I've found ~20k stale Launchpad git repositories that were used by a remote build but failed to be cleaned. The only deletion for these repositories is `RemoteBuildService.cleanup()`, which runs in two places in `commands/remote.py` : normal completion, and `Ctrl-C` followed by `"Cancel builds?" = yes`. These paths leak: • **Failed start**. After the repository is pushed, a failure in recipe creation, in the build request, or on a timeout leaves the repository behind. A failed push leaves it behind too. • **Failed recipe deletion**. `cleanup()` aborted before deleting the repository when `recipe.delete()`  raised. • **Detached builds**. Declining "Cancel builds?" or timing out intentionally keeps resources for  `--recover`. If the user never recovers, they stay forever. The repository name embeds a hash of the project, so any change to the project also orphans the old repository. I'd suggest to check the `craft-remote-build` project each time a user triggers a remote build to clean past stale git repositories. We can make it best-effort and have limits to the number of repositories that are cleaned each time. I've checked https://github.com/canonical/craft-application/issues/249 that suggest user manual cleanup while this is fully automatic. That would fix the cause and the backlog while only adding some lightweight best-effort API calls that can make the remote build fail. ### To Reproduce The easiest way to reproduce is the **Detached builds** leak from the above list. You can just create a remote build and detach from it. If the user never runs `--recover` the Launchpad git repository will be there forever. ### part yaml ```shell ``` ### Relevant log output ```shell N/A ```

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha
Impact: 64
Quick Win: 19
Complexity: 62
Confidence: 93
Actionability: 78
needs triage New, unlabelled report documents several remote-build paths orphaning about 20,000 Launchpad git repositories and proposes a best-effort, bounded cleanup sweep; no maintainer response yet.
qwen/qwen3.8-27b
Impact: 45
Quick Win: 13
Complexity: 55
Confidence: 80
Actionability: 65
needs triage ~20k stale Launchpad git repos leak from remote-build: cleanup() only runs on success or confirmed cancel, so failed starts, recipe-delete errors, and detached builds orphan repos. Proposes best-effort sweep per build. New, unlabelled, no maintainer response.

Update history

Date Change
created

Related work

  • Related To: craft-application#249 (confidence 97%)

    The open #249 enhancement covers cleaning stopped remote builds left by crashes, overlapping this leak but focusing on a user cleanup command rather than automatic bounded repository sweeping.

Related issues

Issue Project State Summary Similarity
#1206 Issue 1205 remote clean build stale repos craft-application open Adds best-effort cleanup of stale Launchpad git repositories during remote builds; under review, CI title check failing and tests pending.
78%
#4481 Clean Launchpad repositories when remote-build errors are raised snapcraft closed Launchpad cleanup for remote-build errors was implemented in commit 663dcdfab; the author confirmed the request was completed.
74%