Luca Boccassi, Kai Lüke and Maanya Goenka
Using an image-based OS brings advantages and challenges. One challenge is the customization of a read-only image with additional host-level software and configuration, and how to manage this customization through the lifetime of a machine.
For deeper changes in /usr, users might build their own images instead of following the official image updates. For common scenarios, the vendor may choose to offer multiple image flavors. Simpler user customization can live outside of the read-only /usr, scattered as config files and binaries in /etc and /opt. Configuration management tools struggle with reliable (re)configuration because tracking filesystem state is hard.
The systemd project now supports a mechanism for extension images. There are two types; system extensions create an overlay for /usr or /opt and configuration extensions create an overlay for /etc. Through the overlay, users can thus change the read-only /usr without building custom OS images. Vendors can also offer their supported flavors as extensions instead of different OS images, even as composable stack where the user can choose optional parts. Users can manage their configuration by replacing the extension images atomically. Since the images bundle all files, this prevents old files lingering around or a system in a half-finished state. The read-only extension images help with setting up attestation and integrity enforcement for their contents. For distributions providing prebuilt initrds (e.g., the Fedora mkosi-initrd proposal), extensions allow initrd customization provided by the distribution or user.
The presentation will give an overview, share use cases and examples, and discuss future improvements for extension images.
A recent addition to the systemd toolbox was systemd-sysext for system extensions through overlay images mounted on /usr. Even newer is systemd-confext for configuration extensions through overlay images mounted on /etc.
The main use case for systemd-sysext is the customization or deployment of additional software on an image-based OS where /usr is read-only. The use of single images that contain all files allows to reliably manage the changes compared to unpacking files to the root filesystem. Optional dm-verity protection ensures the integrity of the extensions. A simple version matching scheme allows to either couple the extension to the OS version or not. The first case is useful for officially released OS extensions or dynamic linking, the second for static linking and only few assumptions about the host.
For systemd-confext the use case is similar as with systemd-sysext but it focuses on configuration in /etc. Here again, the use of single image files makes configuration changes more reliable. Ideally the use of configuration images should allow to have /etc read-only at runtime, following the idea of immutable infra. However, not all software and workflows are prepared for that, and the goal is to introduce different modes for the overlay to, e.g., support ephemeral or persistent changes.
Other plans are to set up the overlay mount from the initrd already to have all configuration in place as early as possible, and to improve the live reload behavior through atomic mount operations and system reload actions.
The presentation will show how to use systemd-sysext/confext and share some examples from Flatcar Container Linux and an embedded Linux platform for both coupled and decoupled extensions.