All newer physical servers of our infrastructure are KVM hypervisors. However, given that we also have a bunch of older hardware around, that do not yet have the right CPU extensions for Hardware virtualization, we are still running some Xen-based hypervisors. As RedHat (and so CentOS – in an initial phase) discontinued support for xen Dom0 (since EL6 only Xen guests are supported), we’re running these hypervisors on Debian Stable. Which at the moment is wheezy, running Xen 4.1.

So far we rolled out new hosts (or updating existing ones) only on kvm based hypervisors and this worked fine. Yesterday, we wanted to update an existing EL6 guest on a Xen-based hypervisor to an EL7 guest, by simply re-installing it and reapplying our automation.

Installation went fine, however afterwards we were not able to boot the installed system. We are using pygrub as a bootloader to extract the kernel from within the guest for xen. And it looked like pygrub wasn’t able to correctly extract the kernel:

# virsh start guest
error: Failed to start domain guest
error: POST operation failed: xend_post: error from xen daemon: (xend.err "Boot loader didn't return any data!")
# /usr/lib/xen-default/bin/pygrub /dev/mapper/guest_rootfs 
Traceback (most recent call last):
  File "/usr/lib/xen-default/bin/pygrub", line 704, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/lib/xen-default/bin/pygrub", line 551, in run_grub
    g = Grub(file, fs)
  File "/usr/lib/xen-default/bin/pygrub", line 206, in __init__
    self.read_config(file, fs)
  File "/usr/lib/xen-default/bin/pygrub", line 410, in read_config
    raise RuntimeError, "couldn't find bootloader config file in the image provided."
RuntimeError: couldn't find bootloader config file in the image provided.

Searching for the error didn’t help much, but it became clear that Debian stable’s pygrub version is not (yet) able to read the newer grub2 config file within the EL7 guest. Locally patching pygrub didn’t really help nor is there a more current version (supporting the newer grub2 config file) file available.

Luckily someone already had the same issue with Fedora 20, which is the basis for EL7. And meanwhile he also had the proper solution in a post kickstart section for EL7.

Integrating this in our kickstart process, made the re-installed guests also booting on a Debian stable Xen 4.1.