Immerda Techblog

To authenticate users on our SMTP Server (exim), we are using the embedded perl interpreter to do a simple login with the passed credentials against our imap server. So in the end dovecot will do the whole authentication dance.

We used this solution for years, however while deploying newer EL7 based relay smtp servers and testing authentication, it became apparent while looking at the logs that there seems to be an issue with validating the certificate of the IMAP server:

 *******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/share/perl5/vendor_perl/Net/IMAP/Simple.pm line 135.

So it seems that newer perl-Net-IO-Socket-SSL versions finally do some proper validation and the old ones never did. Actually, the state of ssl in perl is/was quite worrisome, but things got fixed for the better in the last few years.

So all we had to do was to adapt our authentication-code (Part of the exim_imap_auth module) so that it now enforces validation of certificates. This required us to also package the latest version of perl-Net-IMAP-Simple for our distributions, as none of them have picked up the new versions so far.

All newer physical servers of our infrastructure are KVM hypervisors. However, given that we also have a bunch of older hardware around, that do not yet have the right CPU extensions for Hardware virtualization, we are still running some Xen-based hypervisors. As RedHat (and so CentOS – in an initial phase) discontinued support for xen Dom0 (since EL6 only Xen guests are supported), we’re running these hypervisors on Debian Stable. Which at the moment is wheezy, running Xen 4.1.

So far we rolled out new hosts (or updating existing ones) only on kvm based hypervisors and this worked fine. Yesterday, we wanted to update an existing EL6 guest on a Xen-based hypervisor to an EL7 guest, by simply re-installing it and reapplying our automation.

Installation went fine, however afterwards we were not able to boot the installed system. We are using pygrub as a bootloader to extract the kernel from within the guest for xen. And it looked like pygrub wasn’t able to correctly extract the kernel:

# virsh start guest
error: Failed to start domain guest
error: POST operation failed: xend_post: error from xen daemon: (xend.err "Boot loader didn't return any data!")
# /usr/lib/xen-default/bin/pygrub /dev/mapper/guest_rootfs 
Traceback (most recent call last):
  File "/usr/lib/xen-default/bin/pygrub", line 704, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/lib/xen-default/bin/pygrub", line 551, in run_grub
    g = Grub(file, fs)
  File "/usr/lib/xen-default/bin/pygrub", line 206, in __init__
    self.read_config(file, fs)
  File "/usr/lib/xen-default/bin/pygrub", line 410, in read_config
    raise RuntimeError, "couldn't find bootloader config file in the image provided."
RuntimeError: couldn't find bootloader config file in the image provided.

Searching for the error didn’t help much, but it became clear that Debian stable’s pygrub version is not (yet) able to read the newer grub2 config file within the EL7 guest. Locally patching pygrub didn’t really help nor is there a more current version (supporting the newer grub2 config file) file available.

Luckily someone already had the same issue with Fedora 20, which is the basis for EL7. And meanwhile he also had the proper solution in a post kickstart section for EL7.

Integrating this in our kickstart process, made the re-installed guests also booting on a Debian stable Xen 4.1.