- [x] Have you followed the [guidelines for contributing](https://github.com/canonical/snapcraft/blob/main/CONTRIBUTING.md)? - [x] Have you signed the [CLA](http://www.ubuntu.com/legal/contributors/)? - [x] Have you successfully run `make lint`? - [x] Have you successfully run `make test`? --- Update/rework kernel plugin for releases later than core18. The plugin has been removed from the experimental plugins list. Additionally, a new plugin named initrd is added which enables a simple process for having a part construct an initrd and, optionally, a UKI. This plugin is essentially a de facto requirement for any successful strategy for building kernel snaps, as the initrd is inextricably linked to both the kernel itself as well as the particular Ubuntu Core release one may be building a kernel snap for. This plugin provides a uniform and reproducible interface for building that requirement. Earlier iterations of these plugins contained many options and very challenging to read and debug code. This work largely removes or consolidates those options to streamline the building workflow for snappers, and refactors most of the critical functionality to discrete and independent scripts. This makes the plugin's behavior easier to debug and more closely resembles the kind of behavior one would have to implement in an `override` of some sort to replicate this behavior. Some rudimentary tests have been written but these are not finished. Still working on some failing tests. Work done with @atomcult, they made sure my sh was up to snuff and my python wasn't Bad :) Outstanding work: - [x] Finish new behavior tests (non-legacy tests) - [ ] [possibly] directly test script functions - [ ] End-to-end tests of kernel/initrd builds, using image garden? Boot a fully-built image? - [x] Add tests for building actual snaps (simple case, hard case. Already exist, just need adding). - [x] https://github.com/dilyn-corner/snapcraft/pull/1#pullrequestreview-3291809835 - [x] Documentation - [x] https://github.com/dilyn-corner/snapcraft/pull/1#discussion_r2393445187 - [x] Refactor/simplify mas? - [x] https://github.com/dilyn-corner/snapcraft/pull/1#discussion_r2393445519 Supersedes #4302 --- <details> <summary> Useful synopsis of how these plugins should work</summary> ### Overview A quick description of each piece in some more substantive detail. #### kernel The kernel plugin accepts five options: - kernel-kdefconfig - provided as a list, default `["defconfig"]` - can be any "usual" set of _config files in the "usual" locations - kernel-kconfigflavour - provided as a string, default `"generic"` - Debian-kernel-specific, depends on the kernel source used - kernel-kconfigs - provided as a list, default `[]` - kernel-enable-zfs-support - bool, default `False` - builds ZFS as out-of-tree kernel module - kernel-enable-perf - bool, default `False` - builds the perf binary The values of these options are directly passed to a build script. This build script is shared between both the legacy plugin and the current plugin. Currently this script is `snapcraft/parts/plugins/kernel_build.sh` and invoked by the plugin as `$SNAP/lib/python3.12/site-packages/snapcraft/parts/plugins/kernel_build.sh`. This location is just what "I" (snapcraft) chose. The script itself does all the major kernel-y things. At a high level, it makes sure the "correct" config is generated (it breaks ties between defconfig, flavour, and kconfig as a developer would expect them to be done). It checks that the config supports the target (snaps, apparmor) but *does not* **fail** if the kernel wouldn't support those things - we assume you know what you're doing if you deviate from the default. After some scaffolding, the kernel build is itself kicked off (with the relevant targets determined by the plugin, and the environment determined by the script). The plugin does not assume that any other plugin will be used, but does make some opinionated choices. Specifically regarding the final location of everything it installs - the kernel image must be stored as `/kernel.img`, and the modules and firmware must be available in `/{firmware,modules}`. These are for snapd and Ubuntu Core. Current limitations: basically none, although it isn't necessarily the most space-efficient (full `cp` instead of hard links may be used, for instance). Previous versions of the plugin may supported other kernel options or kernel `source`s; these have been dropped and most behavior is now explicit aside from one key caveat: there is a "secret option" for a developer to provide their own `.config` for the kernel and skip config generation. This is not an option supported by the plugin directly but instead a thing the called script checks. It is **not** meant for end-users, but rather for developers trying to rapidly reproduce something (that is to say, nobody should be doing this in a production environment; they should be using the standard facilities explicitly supported by the plugin). #### initrd The initrd accepts three options for core22 and earlier bases: - initrd-addons - provided as a list, default `[]` - allows adding arbitrary files, a specific example would be when adding FDE support to a kernel snap requires adding things like libTEEc and an `/bin/fde-reveal-key` etc - initrd-firmware - provided as a list, default `[]` - these are firmware blobs which would be required specifically in the initrd. This is rarely necessary, but sometimes must be supported (and should be) - IYKYK - initrd-modules - provided as a list, default `[]` - this list must include at a minimum what you need to mount the rootfs and exec `/bin/init` if they are built as modules for your target kernel, and these modules must be ABI compatible with that kernel - as such, the initrd and kernel plugins are tightly related (if you are using one, you are probably using the other) Three more options are added for core24 and later: - initrd-build-efi-image - bool, default `False` - if `True`, a `kernel.img` will be used from `$CRAFT_STAGE` to build a UKI (some UEFI binary containing the kernel/dtb and initrd, usuaully used on x86_64) - initrd-efi-image-{cert,key} - provided as a string, default is a snakeoil key/cert combo - if provided, `initrd-build-efi-image` must be `True`. The key and cert would in this case be provided by the builder somehow, either through some override to the part using this plugin or via some part run before this one The values of these options are directly passed to a build script. This build script is shared between both the legacy plugin and the current plugin. Currently this script is `snapcraft/parts/plugins/initrd_build.sh` and invoked by the plugin as `$SNAP/lib/python3.12/site-packages/snapcraft/parts/plugins/initrd_build.sh`. This location is just what "I" (snapcraft) chose. The paths for the addons, firmware, and key/cert are all relative to `$CRAFT_STAGE/foo`, where foo is one of `addons/`, `firmware/`, or `signing/` as they are expected to be provided by other parts which build or otherwise supply these objects. The script itself does some semi-complex things, as it must replicate the eventual runtime environment as it uses ubuntu-core-initramfs under the hood, and that tool operates similarly to other initramfs build tools. A notable "odd" thing about this plugin is that it does a lot of its work in the `$CRAFT_PART_SRC` directory, *not* `$CRAFT_PART_BUILD`. This is largely done to improve the iterative build experience, as `$CRAFT_PART_SRC` won't be mangled if the build step for the part fails. After some scaffolding (including constructing and preppiing a chroot for the target architecture), the initrd build itself is kicked off via ubuntu-core-initramfs. Most of the work this plugin does is to support using this tool, which is the official tool for building Ubuntu Core initrds. The plugin does not assume that any other plugin will be used, but it does assume that other work has been done if some options are used; specifically the aforementioned `addons|firmware|signing` requirements. Current limitations: the implementation is kind of shaky in that chroots can sometimes be unreliable; there are active bugs in LXD and fakechroot, and sometimes weird things happen (mostly, the PPA failing to be added to the chroot environment). These are outside the plugin's control, but have been reported to the appropriate upstreams. Workarounds have been done where required, and have been done in such a way that they are fine, just not "optimal" for certain circumstances. If you can't tell what I mean, then it worked ;) ). Previous versions of this plugin may have supported other options. This implementation consolidates or removes those options, as it is more opinionated. Similar to the kernel plugin, this plugin provides a "secret option" for providing your own ubuntu-core-initramfs binary. This is intended only for testing changes to ubuntu-core-initramfs without burdening the developer of uploading those changes to some PPA (as this plugin does not support adding additional PPAs to the chroot environment, though this could be changed in the future). As such, this option remains undocumented and is **not** intended for end-users. </details> (any force pushes are after resolving all suggestions and fixing the history)