list-plugins integration test is broken when run in adt
Metadata
Current evaluation
Fixed by merged PR #607 (commit 3fa8595f0), replacing brittle list-plugins output assertions; snapcraft 2.12 was verified and released to xenial -updates.
Resolution: closed resolved
Reason: PR canonical/snapcraft#607 merged as commit 3fa8595f0 and replaced exact output-format comparisons with per-plugin assertions. Launchpad comments confirm snapcraft 2.12 was accepted into xenial-proposed, verified, and released to -updates.
Evaluated by: mr-cal-bot
No scores available.
Issue body
======================================================================
FAIL: test_list_plugins (test_list_plugins.ListPluginsTestCase)
test_list_plugins.ListPluginsTestCase.test_list_plugins
----------------------------------------------------------------------
testtools.testresult.real._StringException: Traceback (most recent call last):
File "/tmp/adt-run.1qGfm6/build.Rmd/real-tree/integration_tests/test_list_plugins.py", line 28, in test_list_plugins
self.assertEqual(expected, output)
File "/usr/lib/python3/dist-packages/testtools/testcase.py", line 350, in assertEqual
self.assertThat(observed, matcher, message)
File "/usr/lib/python3/dist-packages/testtools/testcase.py", line 435, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: !=:
reference = '''\
ant cmake gulp kernel nil python3 tar-content
autotools copy jdk make nodejs qmake
catkin go kbuild maven python2 scons
'''
actual = '''\
ant catkin copy gulp kbuild make nil python2 qmake tar-content
autotools cmake go jdk kernel maven nodejs python3 scons
'''
[Impact]
* A failed test in autopkg will block the release.
* This problem started in master but wasn't released to the distro.
[Test Case]
* Run the integration suite.
[Regression Potential]
* None. This affects only the test, the feature is correct. If the test doesn't start passing, the release will be blocked.
Evaluation history
| Date | Model | Scores | Action | Summary |
|---|---|---|---|---|
| stealth/space-bunny-alpha | — | closed resolved | Fixed by merged PR #607 (commit 3fa8595f0), replacing brittle list-plugins output assertions; snapcraft 2.12 was verified and released to xenial -updates. | |
| stealth/space-bunny-alpha | — | closed resolved | Merged snapcraft#607 (3fa8595f0), replacing brittle list-plugins formatting checks with per-plugin assertions. Snapcraft 2.12 passed SRU verification and was released to Ubuntu -updates. | |
| stealth/space-bunny-alpha | — | closed resolved | Fixed by merged PR canonical/snapcraft#607 (commit 3fa8595f0), replacing brittle full-output checks with per-plugin assertions. Snapcraft 2.12 was released to Ubuntu updates. | |
| stealth/space-bunny-alpha | — | closed resolved | Fixed the release-blocking list-plugins integration test by removing its dependence on ADT-specific output ordering. Merged in PR #607 as commit 3fa8595f0. | |
| qwen/qwen3.6-35b-a3b | — | — | Resolved the list-plugins integration test failure caused by plugin output ordering differences in adt. The fix was verified, clearing the release-blocking autopkg failure with no regression risk. | |
| qwen/qwen3.6-35b-a3b | — | — | Fixed the list-plugins integration test by updating expected output to match actual plugin ordering. Verified the change resolves the autopkg failure and unblocks the release. | |
| qwen/qwen3.6-35b-a3b | — | — | Resolved by updating test expectations to match actual plugin listing order. Verified and closed, fixing the adt test failure and unblocking the release. | |
| qwen/qwen3.6-35b-a3b | — | — | The broken integration test was fixed by resolving output ordering differences. The issue was closed with the verification-done label, confirming the test now passes in adt environments without impacting the feature, unblocking the release. | |
| qwen3.6-35b-a3b-mtp-q6 | — | — | Resolved the adt test failure by updating list-plugins to handle unordered output. Closed with verification-done, confirming the fix passes autopkg and unblocks the release. |
Update history
| Date | Change |
|---|---|
| created |
Related work
-
Likely Fixed By:
canonical/snapcraft#607
(confidence 100%)
Merged fix for the ADT-dependent list-plugins integration test.
-
Related To:
canonical/snapcraft#606
(confidence 100%)
Earlier single-column-output approach was closed unmerged and superseded by PR #607.
Related issues
| Issue | Project | State | Summary | Similarity |
|---|---|---|---|---|
| #1795859 make list plugins tests use mocked plugins | snapcraft (launchpad) | closed | PR snapcraft#2315 was merged as commit 5eb674f4d, updating list-plugins tests to use mocked results so plugin additions or removals no longer require formatting-test edits. | |
| #607 Simplify the list plugins integration test | snapcraft | merged | Merged PR #607 as commit 3fa8595f0, simplifying the list-plugins integration test by removing brittle output-format assertions that required updates whenever plugins were added. | |
| #1510160 list-plugins command | snapcraft (launchpad) | closed | Added the list-plugins command in commit 32669b7a3 and released it in Snapcraft 0.4; the issue was closed after test confirmation. |