← Back to issue list

Flaky TestServerListening in control due to unclosed servers and timing race

View original Github issue

Metadata

Project
fetch-service
Number
#662
Type
issue
State
open
Author
lengau
Labels
—
Created
Updated
Closed
—

Current evaluation

control.TestServerListening flakes in CI: hardcoded port 18111 reused by three tests, servers never Stop()ed, and a 500ms sleep instead of waiting on ctl2.Dying(). Proposed fix with root cause analysis.

Suggested action: needs triage

Reason: The report is unlabelled with zero comments, so no maintainer triage is recorded yet, but the body is a complete bug report: an exact failure output (ctl2.Err() == "tomb: still alive" vs expected "bind: address already in use"), a CI run link, root cause, and a three-part fix. I verified the cause against the code at HEAD — control/control_test.go uses hardcoded port 18111 in TestStartStop, TestServerError and TestServerListening, and only TestStartStop calls Stop(); TestServerError leaves ListenAndServe bound in a tomb goroutine, so TestServerListening's first ctl can fail to bind and the fixed time.Sleep(500ms) races the asynchronous ListenAndServe in control.go's Start(). Impact is limited to CI reliability (test-only change, no production code involved), hence moderate impact and low complexity.

Evaluated by: mr-cal-bot

Impact: 32 Quick Win: 23 Complexity: 18 Confidence: 86 Actionability: 86

Issue body

`controlSuite.TestServerListening` intermittently fails in CI with: ```text FAIL: control_test.go:76: controlSuite.TestServerListening control_test.go:88: c.Assert(ctl2.Err(), ErrorMatches, ".* bind: address already in use") ... error string = "tomb: still alive" ... regex string = ".* bind: address already in use" OOPS: 10 passed, 1 FAILED --- FAIL: Test (0.51s) FAIL github.com/canonical/fetch-service/control 0.520s ``` Example run: https://github.com/canonical/fetch-service/actions/runs/36642058384/job/109657548197?pr=661 --- ### Root Cause In `control/control_test.go`: ```go func (t *controlSuite) TestServerListening(c *C) { ch := make(chan any, 1) ctl := control.NewServer(18111, ch, "user:password") ctl.Start() c.Assert(ctl.Alive(), Equals, true) ctl2 := control.NewServer(18111, ch, "user:password") ctl2.Start() time.Sleep(500 * time.Millisecond) c.Assert(ctl2.Err(), ErrorMatches, ".* bind: address already in use") c.Assert(ctl2.Alive(), Equals, false) } ``` Two problems contribute to this failure: 1. **Unstopped servers & port reuse**: - Neither `ctl` nor `ctl2` calls `.Stop()`. When the test finishes, `ctl.server.ListenAndServe()` remains running in a background goroutine, keeping port `18111` bound. - The port number `18111` is hardcoded and shared with `TestStartStop` and `TestServerError`. - When tests or runner jobs execute concurrently or in rapid succession, port `18111` may still be occupied or in a lingering/`TIME_WAIT` state. 2. **Fixed sleep synchronization race**: - `ctl2.Start()` spawns `c.server.ListenAndServe()` asynchronously in `tomb.Go()`. - The test waits on an arbitrary `time.Sleep(500 * time.Millisecond)`. On a loaded runner, `ctl2` may take longer than 500ms to attempt the bind, fail, and update the tomb error state, resulting in `ctl2.Err()` returning `"tomb: still alive"`. --- ### Suggested Fix 1. **Ensure cleanup**: Add `defer ctl.Stop()` and `defer ctl2.Stop()` to release the listener and tomb. 2. **Deterministic wait instead of `time.Sleep`**: Wait on `ctl2.Dying()` with a timeout rather than an arbitrary sleep, e.g.: ```go select { case <-ctl2.Dying(): case <-time.After(5 * time.Second): c.Fatal("timeout waiting for server to fail binding") } ``` 3. **Dynamic port selection**: Use a free ephemeral port (`net.Listen("tcp", ":0")` to allocate) instead of a hardcoded `18111`. --- _Note: This issue was toaster-created using opencode with google/gemini-3.8-flash._

Evaluation history

Date Model Scores Action Summary
stealth/space-bunny-alpha
Impact: 32
Quick Win: 23
Complexity: 18
Confidence: 86
Actionability: 86
needs triage control.TestServerListening flakes in CI: hardcoded port 18111 reused by three tests, servers never Stop()ed, and a 500ms sleep instead of waiting on ctl2.Dying(). Proposed fix with root cause analysis.
stealth/space-bunny-alpha
Impact: 45
Quick Win: 29
Complexity: 28
Confidence: 85
Actionability: 88
keep open controlSuite.TestServerListening flakes in CI: ctl/ctl2 are never stopped and port 18111 is hardcoded, plus a 500ms sleep races the async bind failure, so ctl2.Err() returns "tomb: still alive". Maintainer-authored with root cause and a concrete fix.

Update history

Date Change
created

Related work

  • Related To: fetch-service#661 (confidence 55%)

    The failing Actions run linked in the body ends with ?pr=661, so the flake surfaced on that pull request; not verified as the source of the port conflict.

Related issues

No related issues found above the similarity threshold.