Xen support #387
Xen support #387
Conversation
|
This is wonderful; thank you!
I'm using the same
I asked Ryan Lahfa in the Bootspec room when I was figuring out the stuff needed for the systemd-boot builder, and they mentioned that you can look into the |
I don't have "with Xen Hypervisor" here, sort-key was not enough, I have added version & renamed entry file name (== entry id) to make it sort correctly. My primary system has xen enabled, and I have specialization with noxen tag, and it now orders entries exactly as with plain UKI images. |
Lanzaboote uses top-level boot.json, where specializations are described using "org.nixos.specialisation.v1" key. It is possible to look directly in specialization directories (/run/current-system/specialisation/X/boot.json), but it looks wrong, toplevel definition (/run/current-system/boot.json) should work too. |
|
For reference, current |
|
Marking as ready for review, as I don't plan to change too much implementation-wise here. Only toplevel system having xen sucks, but I believe it may need to be resolved in some more generic way, maybe it would be better to have xen installed the same way as memtest/netboot.xyz, instead of intercepting nixos entry generation, and then the solution will be the same as for #273? Anyway, opening this PR for feedback on those topics, hoping some better solution might arise for those issues. |
That was the original idea before the UKI shenanigans in NixOS/nixpkgs#324911. This isn't an ideal solution, as it's expected that end users be able to roll back to a previous generation while keeping their Xen domains running. |
No, I didn't meant it like that, I meant some generic way of producing per-entry netboot.xyz/memtest don't need to be per-entry, but the solution might be generic enough to enable both. |
|
How it may look in bootspec: // Defined in top-level boot.json, no-op if defined on specialization level.
"org.nix-community.lanzaboote.extra-entries": {
// Enabling xen in nixos will make two bootloader entries - one with xen bootloader, one without for troubleshooting.
// I don't think it will be good to make it easy to have user locked in xen with no recovery options.
xen1: {
// generator kind describes how to populate efi sections, various kinds will be defined in lanzaboote
// by code, I don't think it makes sense to make that fully declarative.
generator: {
kind: "xen",
// Null for top-level
specialization: null,
},
efi: "/nix/store/blah-blah-1/xen.efi"
},
// Instead of assigning xen config to specialization, xen bootloader entry will specify specialization which it
// should use.
xen2: {
generator: {
kind: "xen",
specialization: "specialization-name",
},
efi: "/nix/store/blah-blah-2/xen.efi"
},
memtest: {
efi: "/nix/store/blah-blah-3/memtest.efi"
}
}boot.lanzaboote.configurationLimit = 20;
boot.lanzaboote.extraEntries.xen1 = {
configurationLimit = 5;
efi = "${pkgs.xen}/xen.efi";
generator = {
kind = "xen";
specialization = "specialization-name";
};
};
# Or, for compatibility with configuration implemented currently in nixos (pseudocode)
boot.lanzaboote.extraEntries = prefixAttrs "xen-" (filterMapAttrs config.specialisation
(spec: spec.configuration.boot.xen.enable)
(spec: {
efi = "${spec.configuration.boot.xen.package}/xen.efi";
})
);
boot.lanzaboote.extraEntries.memtest = {
configurationLimit = 1;
efi = "${pkgs.memtest}/memtest.efi";
}; |
|
@CertainLach thanks for your work here! Is this a working example you posted? What would be needed to give this PR a test run with an existing |
|
@gador The thing I posted is for the better, future implementation of the same functionality. |
|
Awesome, thanks, Ill give it a try 👍 |
CertainLach
force-pushed
the
feat/xen
branch
from
17ae11f to
37e838c
Compare
|
Rebased & matched behavior with nixpkgs implementation - it is now produces two boot entries per generation if xen is enabled. |
CertainLach
force-pushed
the
feat/xen
branch
from
37e838c to
e80e3f3
Compare
|
CI failures seem to be unrelated, looking at the other PRs. |
|
I finally had time to try and boot into this PR. I have lanzaboote flake input pointed to I also now got Hope this helps. Let me know if I can/should do more debugging |
This is normal, as generated image is not a kernel image (K in UKI), instead it is booted from .conf entry in /boot/entries Do you see new (With xen hypervisor) entry appearing in bootloader menu? |
|
You got it. I didn't select the new generation on boot. I have xen enabled now, thanks! Do you know how to set it as default? Usually, the new generations always get selected by default, but this time, it didn't? Also, maybe unrelated, I got a kernel bug, which I've never had before: |
(With xen support) entry should be on top, it isn't for you? |
|
It is. It just isn't chosen as default (The second entry is selected by default) |
|
After rebase, noticed that GC does no longer work, fixed that. Latest commit might also change the ordering issue, though I haven't encountered that, probably due to customized loader.conf |
|
Tried after your change: Now, no new entry is generated and also the old ones do not get removed |
|
Entries generated using the old version need to be removed manually, rm /boot/loader/entries/xen-* /boot/efi/Linux/xen-* As for entries not generating - can you try to change your nixos configuration a little, it is possible it doesn't think that is is the new system generation for some reason. |
|
Removing the entries manually did work, thanks. And commands like |
|
Huh, are you sure you install correct lanzaboote? inputs.lanzaboote.url = "github:CertainLach/lanzaboote/feat/xen"; Do you have xen extension in your /run/current-system/boot.json? |
|
arrg. Im dump. When I switched back to the original branch I commented your input and forgot to uncomment :-/ |
|
can we rebase this branch to |
As defined in NixOS/nixpkgs#324911
Makes it possible to have system specialisation without xen, but now, only the toplevel system derivation can have xen installed, xen extension is always ignored for specialisations. Issue: DeterminateSystems/bootspec#147
|
Rebased to latest lanzaboote master. As for merging... @blitz @RaitoBezarius sorry for pinging, but this is really unfortunate that Xen in NixOS doesn't work with lanzaboote, and this PR is required for that. I have failed to get anyone's attention in the matrix room, and I still want some feedback about my implementation and/or just to have this PR merged. |
There was a problem hiding this comment.
For general architecture guidance, we do not accept system-specific code in the systemd (and shared crate when possible, e.g. bootspec extensions).
To make progress on this PR, I'd like you to do the following things:
(1) Move the Xen specific code into a xen/src subcrate in rust/tool, even if you need to copy or move "more" code from systemd/src into the shared/src crate when it makes more sense.
(2) Add a README in that Xen crate to explain that "Lanzaboote Xen backend" is maintained by your team and is not officially supported (beyond the fact that it's available here in this codebase as a matter of convenience).
The idea is the following, the lanzaboote maintainers officially maintain most of the codebase but we recognize that some folks want to support different backends out there:
- GRUB (in a PR)
- U-Boot (that I wanted to support for embedded systems, dropped the prototype for complicated reasons)
- VBE: Verified Boot For Embedded Systems
etc.
Xen is yet one another special backend to me here. Feel free to ping me over Matrix for sync discussions if needed, my bandwidth is not great these days alas.
|
This is not a backend, it can be treated as another nixos boot configuration, xen replaces standard nixos linux kernel here. Its implementation can be moved into another crate, but systemd-boot backend will have it as a dependency, I don't see how it is possible to decouple them. |
If Xen was simply a different kernel, the modifications to the shared crate should directly go into NixOS and lanzaboote should not be aware of Xen as a different kernel. But from what I gather of this PR, it's more complicated than that. On the top of that, lzbt-systemd is modified to support bespoke entries that seems also required in Xen case which are not necessary with standard partial kernel images. We would like to keep lzbt-systemd standard and close to the concept of unified kernel images. If Xen cannot accommodate for this, it cannot mix in the lzbt-systemd backend. It can be a lzbt-systemd-xen backend FWIW. It's fine if code gets copied at some point, but the idea here is to get the responsibilities & scope right for the different "backend" however you want to call it. If you enable Xen + systemd-boot, you can easily turn on the lzbt-systemd-xen implementation in the upstream NixOS modules or anything. That's fine as well. |
Linux: kernel + boot params get packed into UKI, UKI gets found by systemd-boot and then booted Xen: kernel + boot params + xen efi + initrd get packed into UXKI (Unified Xen Kernel Image, https://xenbits.xenproject.org/docs/unstable/misc/efi.html), resulting in efi image, which can be booted by systemd, but won't be detected automatically as this is not a UKI, thus a dedicated type #1 entry file is required. The problem is, enabling Xen should not disable NixOS entries generation. I've got an idea, lzbt-systemd-xen should only generate Xen-dependent files, and then delegate rest of the work to the lzbt-systemd. The only problem remains is the garbage collector. lzbt-systemd still needs to be taught to garbage-collect /boot/loader/entries, even if that directory is not populated by it right now (Until lanzaboote supports managing memtest and other plain efi binaries). Still, it would be much cleaner to just add this code to lzbt-systemd, but I understand that Xen might be too niche, and not everyone wants to deal with code supporting it. |
NixOS/nixpkgs#324911 implements Xen hypervisor support in NixOS, this PR implements Xen support in lanzaboote when used together with mentioned PR.
It is impossible to have xen enabled for specialization, only primary system can have it, due to Extensions for specializationn DeterminateSystems/bootspec#147
Current implementation produces very large EFI binaries (>160M, prepare your /boot partitions), because Xen can't verify linux image by itself, and it needs to be included in Xen unified kernel image (https://xenbits.xenproject.org/docs/unstable/misc/efi.html), which only supports unpacked (vmlinux) kernels (as far as I can see, using any other format (vmlinuz/bzImage) results in "Not an ELF binary" error).
Because there is no path to vmlinux image present in boot.json, extra configuration is required:
boot.bootspec.extensions."org.xenproject.bootspec.v1".vmlinux = "${config.system.build.kernel.dev}/vmlinux";
Created xen boot entries are located lower than entries obtained from plain nixos UKI images, because I haven't figured how to provide systemd-boot with sort key.
Garbage collection doesn't work for /boot/loader/entries (I'm not sure how it is supposed to work).
Cc: @SigmaSquadron