Ehlo Onion

Email transport security in 2016, it’s still a thing! The last mile is fortified — no reasonable provider accepts plaintext smtp, pop, or imap from a client. But what about transport? It’s still opportunistic, downgradeable, interceptable, and correlateable. It’s time to put some more band-aids on this wound!

SMTP delivery to Tor Hidden Services

Moving forward our MX is reachable at ysp4gfuhnmj6b4mb.onion:25. We are happy to accept mail for all our domains (ie. all domains where mail.immerda.ch is the MX) there! Do you want to know how to do that? Or even how to make your system reachable through tor? There is a tutorial for Exim at the bottom of this post and there is another one for Postix. But wait, there is more!

Ehlo

So how about everyone on the internet does this? How about using the following format to publish Onion Service MX records in dns:

_onion-mx._tcp.immerda.ch. 3600 IN SRV 0 5 25 ysp4gfuhnmj6b4mb.onion.

Fair enough, dns can be spoofed. But we still get all the other benefits when it works, we make correlation much harder and improve the meta-data leakage. And we exclude the attacker who can only tamper with the SMTP session but not the DNS query.

At the moment this is just a proposal but we are eager to collaborate on this if you reach out to us!

Howto? (Exim)

Now let’s see how we can configure Exim send mail to a tor Onion Service for a manually curated set of domains.

There is previous work, but since the exim 4.87 release an easier approach is possible. Here is the high level overview of what we need:

  1. A static mapping between email domain and MX onion address
  2. A router to prepare the submission using Tor’s AutomapHostsOnResolve feature: The router performs a programatic DNS lookup with the tor daemon. The returned IP is being mapped to the correct onion url by tor.
  3. A transport sending emails via the tor socks proxy using the above IP as destination.

Here’s how we adjusted our exim setup for outgoing mail (and you should be able to do it in a similar way):

  • First create the mapping of recipient domains to onion addresses:

/etc/exim/onionrelay.txt

immerda.ch ysp4gfuhnmj6b4mb.onion
lists.immerda.ch ysp4gfuhnmj6b4mb.onion
  • Then convert it to cdb for faster lookups:

    cdb -m -c -t /tmp/onionrelay.tmp /etc/exim/onionrelay.cdb /etc/exim/onionrelay.txt

  • Install and configure Tor for Onion Service DNS mapping and have the local daemon running:

/etc/torrc

AutomapHostsOnResolve 1
DNSPort 5300
DNSListenAddress 127.0.0.1
...
  • Configure Exim:

/etc/exim/conf.d/perl

perl_startup = do '/etc/exim/perl-routines.pl'
perl_at_start

/etc/exim/perl-routines.pl

use Net::DNS::Resolver;
sub onionLookup {
  my $hostname = shift;
  my $res = Net::DNS::Resolver->new(nameservers => [qw(127.0.0.1)],);
  $res->port(5300);
  my $query = $res->search($hostname);
  foreach my $rr ($query->answer) {
    next unless $rr->type eq "A";
    return $rr->address;
  }
  return 'no_such_host';
}

/etc/exim/conf.d/domainlists

ONION_RELAYDB=/etc/exim/onionrelay.cdb
domainlist onion_relays     = cdb;ONION_RELAYDB
...

/etc/exim/conf.d/router

# send things over tor where we have an entry for it
onionrelays:
  driver    = manualroute
  domains   = +onion_relays
  transport = onion_relay
  # get the automap IP for the onion address from the tor daemon
  route_data = ${perl{onionLookup}{${lookup{$domain}cdb{ONION_RELAYDB}}}}
  no_more
...

/etc/exim/conf.d/transports

onion_relay:
  driver = smtp
  socks_proxy = 127.0.0.1 port=9050
...

Running OnionService MX

To receive mail all you need to do is set up a Tor Onion Service (there are plenty of tutorials out there) which listens on port 25 and publish the address to the world.

We strongly advice to run this hidden service on a separate VM and internally forward to your MX to avoid running an open relay.

You could also configure the Onion Service directly on the MX but then you need to be extra careful since connections will appear to come from 127.0.0.1. Most mail servers treat localhost in a privileged way and you want to avoid that. Possible workarounds are to either locally map to a different port or bind tor daemon to another ip (eg. 127.0.0.2).

How we configure services: ibox types

As previously mentioned we are using the ibox project as a way to refactor, modernize and share our automation setup with other interested folks. As we are looking back to around 10 years of automating our services using puppet, there might one or the other place where it’s time to do such a refactor. So this whole project is a slow but steady process to make our plans happen: That we – internally, but also others – are able to replicate parts of our infrastructure on a local environment to easily renew, improve, debug or extend it.

Since our last posting we migrated quite a bunch of services to the ibox structure and so it might be a good time to share what you can do with it now. Also it’s a good opportunity for us to get things at least somehow a bit in a shape we can share it (read document).

ibox types

ibox types are somewhat our puppet roles and within our infrastructure in general we try to isolate things as it makes sense. This means we might actually run different ibox types together on a certain node type and some ibox types might be purely isolated within their VM. The same is possible with the ibox allowing you quite nice setups as you might see later.

Though something should be mentioned: we are never removing types from our nodes and so this is also not something we want to enable within the ibox. Removing craft respecting all kinds of different combination can become very cumbersome and given that the ibox’s idea is to easily replicate something within a local development environment, we actually don’t support that here either.

You should be able to easily get an overview of the different types by looking at the ibox::types:: space within the ibox module. Also we try to share certain examples within the vagrant.yaml.example file.

As you might see there is broad range of our services available and more will come in the future. To give you some idea on how to start using that concept, we give you an example how to use a few of the available types.

webhosting & webservices

Besides what most people are using from us – email & other communication services – we are also offering webhosting for your static or php-based websites. Most of the things that are driving the setup for such a webhosting are part of the webhosting and apache modules. You can easily use them to for example replicate your configuration of a webhosting with us locally on your machine using the webhosting type and the following hiera data example.

In short what this does is:

  1. configure a few defaults for a webhosting host, especially with regards to SFTP access.
  2. Setup the databases required for the hostings.
  3. Configure a simple php hosting with installing the php security checker of sektions eins to be automatically installed (Browse to http://php56.ibox-one.local/phpconfigcheck.php to verify our configuration). We activated only one as it should show what the others do as well.
  4. Automatically install a full WordPress installation, that is setup and ready to serve your blogpost. You can retrieve wordpress’ admin password using trocla get wordpress_adminuser_wp.ibox-one.local plain
  5. Automatically install a full Mediawiki installation, that is setup and ready to organize your content.
  6. And last but not least: A SimpleMachine Forum installation ready to go through the web installer. Something which we didn’t yet automate to the end, but we would take merge requests for that! 🙂

All these installations match what and how we are installing things on our webhostings. So if you might have problem debugging something, this might be a good opportunity to get a real shell for your webhosting 🙂

# a webhosting type with mysql
# to illustrate we also install a bunch of hostings
ibox::types: ['webhosting','dbserver']
# we don't need postgres here
ibox::types::dbserver::is_postgres_server: false
# let's get some space for the webhostings
ib_disks::datavgs::www::size_data: 12G
# let's make sure our hostings can login
# over ssh but only using sftp and are chrooted
sshd::sftp_subsystem: 'internal-sftp'
sshd::allowed_groups: 'root sftponly'
sshd::use_pam: 'yes'
sshd::hardened: '+sha1'
sshd::tail_additional_options: |
  Match Group root
         PasswordAuthentication no

  Match Group sftponly
         PasswordAuthentication yes
         ChrootDirectory %h
         ForceCommand internal-sftp
         AllowTcpForwarding no
# the databases we need
ib_mysql::server::default_databases:
  'wp_test': {}
  'wiki_test': {}
  'smf_test': {}
ib_webhosting::hostings::php:
#  'php.ibox-one.local':
#    git_repo: 'https://github.com/sektioneins/pcc'
#  'php54.ibox-one.local':
#    git_repo: 'https://github.com/sektioneins/pcc'
#    php_installation: "scl54"
#  'php55.ibox-one.local':
#    git_repo: 'https://github.com/sektioneins/pcc'
#    php_installation: "scl55"
  'php56.ibox-one.local':
    git_repo: 'https://github.com/sektioneins/pcc'
    php_installation: "scl56"
# setup a wordpress hosting fully automatic
# mind the database above
ib_webhosting::hostings::wordpress:
  'wp.ibox-one.local':
    blog_options:
      dbname: 'wp_test'
      adminemail: 'admin@ibox.local'
# setup a mediawiki fully automatic
# mind again the database above
ib_webhosting::hostings::mediawiki:
  'mw.ibox-one.local':
    db_name: 'wiki_test'
    contact: 'admin@ibox.local'
    sitename: 'mw'
    db_server: '127.0.0.1'
# install a smf hosting, ready to be clicked
# through the webinstaller
ib_webhosting::hostings::simplemachine:
  'smf.ibox-one.local': {}
# setup all diffrent kind of php hostings, either using
# the system php installation or scl installations

Webservices are managed services that we are fully running and providing to our users. Such a service is for example coquelicot powering our dl.immerda.ch service. And you can actually setup your own, by adding the following hieradata:

# webservices
ibox::types: ['webservices',]
#ib_apache::services::webhtpasswd::htpasswd_name: 'ht.ibox-one.local'
# get a coquelicot up and running
ib_apache::services::coquelicot::instances:
  'dl.ibox-one.local': {}

And as an example we have another service htpasswd.immerda.ch of us added there as well.

tor onion services

Since last week we also integrated the way how we are setting up onion services into ibox. And so you can actually make your ibox also available as an onion service:

ibox::types: ['onion_service']
ib_tor::onion::services:
  "%{hostname}":
    '22': {}

So this will create an onionserver called ibox-one (you can get the onion address @ /var/lib/tor/ibox-one/hostname) which makes ssh accessible.

You can combine that and make for example a wordpress that you installed available over an onion service:

ibox::types: ['webhosting','dbserver','onion_service']
# we don't need postgres here
ibox::types::dbserver::is_postgres_server: false
# let's get some space for the webhostings
ib_disks::datavgs::www::size_data: 12G
# the databases we need
ib_mysql::server::default_databases:
  'wp_test': {}
# setup a wordpress hosting fully automatic
# mind the database above
ib_webhosting::hostings::wordpress:
  'wp.ibox-one.local':
    domainalias: ['*.onion']
    blog_options:
      dbname: 'wp_test'
      adminemail: 'admin@ibox.local'
ib_tor::onion::services:
  "wp_os":
    '80': {}

For simplicity reasons we are serving any .onion address for the wordpress host, so we don’t need to read out the generated onion address and rerun puppet, but you might get the idea. Please note that some components might need additional tuning to make them safe, when being used through a local onion service, as lot’s of service assume localhost to be a safe place.

And so you get your wordpress served through an onion service:

$ torsocks curl -I "$(vagrant ssh -c 'cat /var/lib/tor/wp_os/hostname' | cut -c 1-22)/"
HTTP/1.1 200 OK
Date: Fri, 18 Nov 2016 15:12:01 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9
Link: <http://wp.ibox-one.local/?rest_route=/>; rel="https://api.w.org/"
Content-Type: text/html; charset=UTF-8

XMPP man-in-the-middle via tor

Update: It was pointed out to us that the word ‘wide-spread’ below is misleading since the cumulative exit probability of those nodes was probably below .5%. What we wanted to say instead is that the number of domains affected was large, when a bad exit was involved.

We saw some wide-spread XMPP man-in-the-middle via malicious tor exit nodes during the last 24h. The attacks where only targeting starttls connections on port 5222. The mitm served forged self-signed certificates for various Jabber domains, one of them being our imsg.ch. The attack was orchestrated between multiple exit nodes acting in sync. All of them served the same set of forged certificates, allegedly created around midnight March 2nd to 3rd, using common names tailored to various XMPP servers.

We tried a small sample of XMPP servers. Out of which we recorded the following domains being intercepted:

  • freifunk.im
  • jabber.ccc.de
  • jabber.systemli.org
  • jappix.org
  • jodo.im
  • pad7.de
  • swissjabber.ch
  • tigase.me

For a handful other domains the connection attempts where dropped and google xmpp was the only one we found to be unaffected.

The exit nodes involved in this attack were reported to the tor project and seem to be dysfunctional by now. The ones we know of are:

  • FAFE24D8CF973BC38B54500DA666EEE44F02C642
  • 6269E9B3549012C44F518D2D123E41A4F320157E
  • 04DDEEAB315956AD956AF046338FB8E5B52B2DAF
  • DB213B8BD383A955CC383CAB76F8DD71A7198F47
  • 0276C54A43ABF27AB0247AF377952A306605FB8A
  • A1B3C065339D11AC361D85B0A3F9B59A23BD818A
  • CAD72B06527F8534640E8E47AEE38E2A3321B2D4
  • 5CBDC2AB702784154E7EFE7E6F87645EB107E8FA
  • BAB1FA3492162DB8058F464AD99203144F2AAF87
  • 187C2945109982516002FE6620E46A94B9B81A6E
  • 6269E9B3549012C44F518D2D123E41A4F320157E
  • FAFE24D8CF973BC38B54500DA666EEE44F02C642
  • FB134ED5DEE131B707270D9E99D72201CC96D668
  • E46021F7921EBB836B318863BDD432982BAA7BD0

Here is the certificate which was presented when you tried to access imsg.ch:

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 12273528369637281981 (0xaa54550634e8d0bd)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=imsg.ch
        Validity
            Not Before: Mar  3 12:08:43 2016 GMT
            Not After : Jan 10 12:08:43 2026 GMT
        Subject: CN=imsg.ch
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
        [...]
SHA1 Fingerprint=2C:F2:07:E8:19:ED:4E:CA:81:59:6E:3F:D8:59:52:B8:12:22:88:DB

What was curious is that the mitm SSL endpoint was sending a TLS session ticket. Does anybody have an idea if that could lead to an additional attack being carried out, or if it was merely an artifact of their SSL stack. E.g. one explanation we have is that the SSL terminator might have seen all packets originating from the same local tor daemon IP.

Here’s more log, for your convenience:

Certificate chain
 0 s:/CN=imsg.ch
   i:/CN=imsg.ch
-----BEGIN CERTIFICATE-----
MIICoDCCAYgCCQCqVFUGNOjQvTANBgkqhkiG9w0BAQUFADASMRAwDgYDVQQDEwdp
bXNnLmNoMB4XDTE2MDMwMzEyMDg0M1oXDTI2MDExMDEyMDg0M1owEjEQMA4GA1UE
AxMHaW1zZy5jaDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALqSTvWG
ohTiP9DJztuSki2NTLRUfC9/1gbby2TK9kKPFKLyeIyZDtvgQvigT+ToNfvOl2EW
XX4WwlQ9sWGY+C2nZrX5EmttE7zV1/v3tYcPf9Vdir83WXX8/IRauewMxghHd2kH
LTQCTjYtOao+b2VEBDf6QsOV3DEAeM3tbitgFHazmxGMBI0LvadT2NWDVK36gVri
nD4FTV52RydamGMN9hwLP8Lp5RFcmoIOf8xJcYkMRpusvTNDCWERJYHrSnq0fppE
Y2i6EIMBLKpfbatO7IVyy4E1zAMgcVzvnDBQ2DsmcFpjLrzBVmW73YnH7GsGcuPk
Z9ha1eCGML2VncsCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAG98eWZQpv0V7e8Lt
i/ajKnZrFiv3kh6GwuJAXOH0dimhoNOaCMfwY2e30e3rxSNpXGr09mx6qozH2oh4
CbsZt2zd3jfeFeFe7bEWZijkigMcr+zI4IPpVeHfnZLidaMo/Pr5WMtaHgfO/kOC
UGboN3YbnVVQ7aFvrpvFhyfoemZD44O7ieFObKcmKHySXNyLrIYL1y9LNpWNZF5X
c6JsGHjFrefWI0smcd/aIyus0UTNJ/UaZwRxNbdlnSpXL0+nptnFadXqCo7ygZeE
ciqz/ckEIY0i4S39O1hO3LXEpT6gmlZnJt9Kffc3zLw5nS3IE+LpCbxnGNn53MCG
R977LA==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=imsg.ch
issuer=/CN=imsg.ch
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1948 bytes and written 490 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: [... 64 bytes ...]
    Session-ID-ctx: 
    Master-Key: [... 96 bytes ...]
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket: [... 0x100 bytes ...]

    Start Time: [...]
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)

Another pygrub adventure

So the bug from the article Get Centos 7 DomU guests booting on Xen 4.1 hit us again, after a while as we wanted to reboot a few guests.

Everything still seemed to be fine within the guest, nevertheless pygrub on host wasn’t able to find a kernel to boot.

Revisiting one of the old xen bus that we already searched on our previous experience, showed us that we also need to patch pygrub on the host:

> -            title_match = re.match('^menuentry ["\'](.*)["\'] (.*){', l)
> +            title_match = re.match('^menuentry ["\'](.*?)["\'] (.*){', l)

in /usr/lib/xen-4.1/bin/pygrub and your guests boot again.

Validate certificates with perl-Net-Imap

To authenticate users on our SMTP Server (exim), we are using the embedded perl interpreter to do a simple login with the passed credentials against our imap server. So in the end dovecot will do the whole authentication dance.

We used this solution for years, however while deploying newer EL7 based relay smtp servers and testing authentication, it became apparent while looking at the logs that there seems to be an issue with validating the certificate of the IMAP server:

 *******************************************************************
 Using the default of SSL_verify_mode of SSL_VERIFY_NONE for client
 is deprecated! Please set SSL_verify_mode to SSL_VERIFY_PEER
 together with SSL_ca_file|SSL_ca_path for verification.
 If you really don't want to verify the certificate and keep the
 connection open to Man-In-The-Middle attacks please set
 SSL_verify_mode explicitly to SSL_VERIFY_NONE in your application.
*******************************************************************
  at /usr/share/perl5/vendor_perl/Net/IMAP/Simple.pm line 135.

So it seems that newer perl-Net-IO-Socket-SSL versions finally do some proper validation and the old ones never did. Actually, the state of ssl in perl is/was quite worrisome, but things got fixed for the better in the last few years.

So all we had to do was to adapt our authentication-code (Part of the exim_imap_auth module) so that it now enforces validation of certificates. This required us to also package the latest version of perl-Net-IMAP-Simple for our distributions, as none of them have picked up the new versions so far.

Get Centos 7 DomU guests booting on Xen 4.1

All newer physical servers of our infrastructure are KVM hypervisors. However, given that we also have a bunch of older hardware around, that do not yet have the right CPU extensions for Hardware virtualization, we are still running some Xen-based hypervisors. As RedHat (and so CentOS – in an initial phase) discontinued support for xen Dom0 (since EL6 only Xen guests are supported), we’re running these hypervisors on Debian Stable. Which at the moment is wheezy, running Xen 4.1.

So far we rolled out new hosts (or updating existing ones) only on kvm based hypervisors and this worked fine. Yesterday, we wanted to update an existing EL6 guest on a Xen-based hypervisor to an EL7 guest, by simply re-installing it and reapplying our automation.

Installation went fine, however afterwards we were not able to boot the installed system. We are using pygrub as a bootloader to extract the kernel from within the guest for xen. And it looked like pygrub wasn’t able to correctly extract the kernel:

# virsh start guest
error: Failed to start domain guest
error: POST operation failed: xend_post: error from xen daemon: (xend.err "Boot loader didn't return any data!")
# /usr/lib/xen-default/bin/pygrub /dev/mapper/guest_rootfs 
Traceback (most recent call last):
  File "/usr/lib/xen-default/bin/pygrub", line 704, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/lib/xen-default/bin/pygrub", line 551, in run_grub
    g = Grub(file, fs)
  File "/usr/lib/xen-default/bin/pygrub", line 206, in __init__
    self.read_config(file, fs)
  File "/usr/lib/xen-default/bin/pygrub", line 410, in read_config
    raise RuntimeError, "couldn't find bootloader config file in the image provided."
RuntimeError: couldn't find bootloader config file in the image provided.

Searching for the error didn’t help much, but it became clear that Debian stable’s pygrub version is not (yet) able to read the newer grub2 config file within the EL7 guest. Locally patching pygrub didn’t really help nor is there a more current version (supporting the newer grub2 config file) file available.

Luckily someone already had the same issue with Fedora 20, which is the basis for EL7. And meanwhile he also had the proper solution in a post kickstart section for EL7.

Integrating this in our kickstart process, made the re-installed guests also booting on a Debian stable Xen 4.1.

Manage and purge root ssh authorized_keys in an ibox

So far the ibox modules didn’t manage any authorized_keys for the user root, while the sshd module enforced that the root user can’t login using the password. We had the functionality already implemented in our internal code base, but it originated from the very early puppet days and it lacked several features, like purging non-managed keys or being able to selectively allow certain keys only on certain hosts.

With a recent commit to the ibox modules we introduced this functionality also on the ibox with all the missing features.

We introduced a new variable within the main ibox class called $root_keys and if this variable is not empty, we will generate a set of sshd::autorized_keys resources.

Furthermore, it will activate the purge_ssh_keys feature for the user root, purging any unmanaged keys in the authorized_keys file of the user root.

An implementation detail is that $root_keys is not a class variable, but just a normal variable within the class. This variable is being looked up using hiera_hash(). hiera_hash() and has the advantage that puppet will traverse your whole hiera-hierarchy and merge all the found hashes together into one. This will allow us to configure a set of keys for some users on all systems (putting them under ibox::root_keys in the defaults file), while allowing a key for a a user – that should have access to only one or a few systems – into the specific place in the hierarchy. Like:

$ cat hieradata/hosts/ibox-two.local.yaml
ibox::root_keys:
  'user1':
    key: 'AAAAA......'
$ cat hieradata/defaults.yaml
ibox::root_keys:
  'userA':
    key: 'AAAAA......'
  'userB':
    key: 'AAAAA......'

This will add the keys for userA & userB on all systems, while user1 only gets added to the host ibox-two.local. userA & userB will be added there as well.

Still: Why not a class parameter? If we would define $root_keys as a class parameter of the class ibox, puppet would start looking up the value for ibox::root_keys using its normal internal puppet hiera lookup mechanism and only reaching out to the code-default (which is hiera_hash('ibox::root_keys',{}) if nothing would be found. However, if we use the same lookup key also for the hiera_hash-lookup, puppet will already find something in the first face and this won’t be merged, because the lookup mechanism is using the standard priority mechanism.

We could have chosen another name either for the variable or the merge lookup key, which would have worked and done what we wanted. On the other hand: puppet would have still used its internal lookup mechanism first and we would have traversed the whole hierarchy without any need. Also it would have been possible to ignore the merge lookup by just setting a value for the wrong key in hiera. By not defining it as a class parameter, it makes the key more consistent, while avoiding any accidental overwrites of the features. Plus it saves us a few hiera calls.

Another implementation detail is the usage of stages. While stages seem to be nice to be able to have implicit dependencies without having to declare too many require/before statements, they can become very quickly quite ugly. This is one of the reasons, why the official documentation doesn’t recommend them. Except for package repository setups, which is exactly what we use them for. However, because we also manage a few files owned by the user root as part of the yum stage, this stage also requires the user root to be managed, as puppet’s autorequire feature will kick in and you hence will end up in a dependency loop. This is why we had to put the management of the user root (and it’s associated keys) to a dedicated stage that precedes the yum stage.

varnish host stats – with what is your varnish busy

Every website or http-application hosted in our environment is fronted by a varnish instance (and a nginx to varnish pair if we talk https). We use it to route traffic to the right backend, but also to introduce some easy – but very effective – caching to boost websites.

Sometimes when dealing with lots of requests to a certain domain you might get interested in with what your varnish is busy and then being able to identify traffic even further. Because of the proxy/cache cascade, it’s usually not very effective to do this on the backend and hence you need to analyze what varnish is busy with.

This is where varnishHostStat might become handy, so that you can see which websites or even which parts of a website are causing which amount of traffic. See the README for a detailed description what it can do.

To be able to easier distribute varnishStats onto our systems, we created a simple SPEC file for EL6 & 7 to package this up and distribute on our systems. So packages can also be found in our yum repository

Building the common ground – a basic iBox

In the previous post we introduced the building blocks for the iBox, which can also be used for local development. As an example we brought up the CentOS 7 integration into our environment. So how does this work?

Background

We run a couple of services and they all run on either one or multiple systems. So, if we talk about systems in our setup, we talk about many VirtualMachines all fulfilling some kind of role or – as we call it – type. VMs give us for example a way of separation and isolation, as we don’t want to keep mailboxes alongside user-maintained CMSs, that tend to be outdated and hacked/defaced/… from time to time. But it also eases management, as VMs are much easier to handle, than baremetal systems – even if it’s just to avoid the usually long BIOS roundtrip time.

Among all these different types, you have usually a set of configurations that need to be applied to all systems. Things like: sshd configuration, firewall, ntp, repositories and so on. In the end this means that a VM consists of two parts: a base system – equal among all systems – and the additional software/configuration – representing the specific type of workload.

And this common ground is often the first part, that we need to adapt and hence implement for a new major release of a distribution. As the new version might use newer or other software, tends to solve problems differently and sometimes we can even drop a few workarounds that we put in place on earlier versions.

Getting a box with the basic OS configuration

If we look at how our configuration management is set up, this means that if we start a box without a type, but still run our configuration management on top of it, we will simply get our common ground. And getting such a box is as simple as described in the README of the ibox_base repository.

  1. Install vagrant
  2. Import a stemcell from our stemcell repository
  3. Checkout the repository and its submodules.
  4. Run vagrant up
  5. Profit

After running vagrant up, vagrant will start a VM and once it got access to it, starting to apply our base configuration, that every system should have. We haven’t yet ported every detail of our base configuration to the ibox modules, but the most important things are there, the rest will be added as we move on.

Hiera

Nearly all aspects of our setup can be tuned using hiera and the sames applies to the local development environment you get with iBox.

There is a basic configuration that defines a sane hierarchy for a development environment (our production environment has a bit more levels, but the idea should be clear), which is used while running puppet. The exisiting default.yaml (in manifests/hieradata/default.yaml) should contain everything which is our default configuration of the puppet modules we use.

Furthermore, we ship a sample vagrant.yaml (located in manifests/hieradata/vagrant.yaml.sample) file, which should show you common configuration options, that we have so far ported to iBox. Simply make a copy of it to vagrant.yaml and start editing for what you like to try out.

Types

A basic configuration of the operating system is usually nothing spectacular. More interesting are usually the applications that you run on top of it.

As we move further with our adoption of CentOS 7, we have to verify the configuration of our types on the new major release, which usually means it is a good point to also release our so far secret – but still share able parts – into iBox building blocks.

As an example we can take the dbserver type, representing a database server nodes running (either or both) MariaDB and PostgreSQL.

# enables the type dbserver
ibox::types: ['dbserver']

# configure a postgres admin user called test2 (superuser role)
ib_postgres::server::admin_users:
  'test2': {}
# configure a postgres database called test and
# a role test owning that database
ib_postgres::server::default_databases:
  'test': {}

# setup a mysql user have all privileges on all databases
ib_mysql::server::admin_users:
  'user1': {}
# setup 2 MariaDB databases and for each of them a user
# with the same name and with full privileges on the specific database
ib_mysql::server::default_databases:
  'testdb1': {}
  'testdb2': {}

Having the above in the manifests/hieradata/vagrant.yaml file and a simple vagrant up will give you an iBox with 2 fully configured database servers (e.g. incl. backup) and a few resources (like databases) configured.

We will talk about types and further ports to the iBox as we move forward with releasing the parts.

Introducing iBox – stemcells

As a lot of people might know: We are happy puppet users @ immerda for ages. Also since the very beginning, we try to publish as much as possible of our puppet work and collaborate with others on improving these puppet modules and make them available and (most important!) useful for others as well.

Background

As (for obvious reasons) we can’t publish the whole configuration of our infrastructure, we went with a dual-modules setup: public and private (site-specific) modules. While this was fine in the beginning and it let us at least publish the common ground for our infrastructure, it was very hard to see how in the end all these public modules build up our infrastructure. Within the last 6 years puppet and its ecosystem developed and improved heavily and made a lot of things easier. Drastic changes within the language (e.g. 2.7 to 3.0) made it hard to keep things updated and staying on top of the current development, while still running a platform in production. But once you made the big step, a lot of things became way easier. Especially, some of these changes made it easier to share more of our infrastructure and especially publish our usage of our public modules. Furthermore, it becomes more and more important for us to be able to quickly try something new out, to develop and integrate a new feature into our infrastructure, while not necessarily be too disruptive to it. In the end our users – and also we – are using our infrastructure daily for our digital communication and we don’t want to be too disruptive to it, which would just bury us in (unnecessary) work.

Additionally, more and more people became interested in how we set things up and would like to help out and contribute to our infrastructure in one or another way. So we had a new goal: Make more of our internal (so far secret) sauce publicly available, so that people can inspect it, fix it, contribute to it or even just use it to run their very own version of our infrastructure.

iBox

This lead us to kick off (another) new internal project, where we refactor our current puppet code base in such a way that we can publish more parts of our infrastructure. In the end this could mean that anyone could use our codebase to run her very own instance of our infrastructure. And the basis for that would be something that we named: iBox.

It is a very ambitious initiative and we are far away from the final goal. However, we see it more as a project, that will accompany us in the next years, while improving the infrastructure further. We don’t plan to dedicate too much concentrated effort for it, more it should grow as we grow and move on. So in the past few months we started working on it and are now in a state where we are able to publish first, but still very basic parts of what we envision. And given that CentOS 7 was recently published, the integration of the new version of our most used distribution, was also a good starting point for our effort. We anyway would have to do the integration of the new version into our infrastructure, so why not kick off the iBox initiative and use it to develop the CentOS 7 integration into our infrastructure. Eat your own dog food! 😉

This and a few more upcoming blog posts will introduce you (and also some members of our collective) to what we have currently developed, how we built it and how you can use it. It is far from being useful, but it is a start and we expect to grow and integrate more and more things, as we move on with our infrastructure. We don’t have a concrete time line nor road map (we luckily never have!), but don’t expect it to just die after the few things that we publish.

stemcells

If we look at all the prerequisites that one needs to be able to run the code for our infrastructure, it starts with a very simple thing: a server or at least a virtualmachine (VM). You need an operating system, where you can run puppet to setup all the parts that build an infrastructure. Luckily, with today’s availability of virtualization it is very easy to run VMs on a local desktop computer and in the recent years a lot of nice tools appeared, addressing all the boring and cumbersome steps. One of them is Vagrant, which makes it easy to setup and run development VMs on your local desktop. More about that later.

Vagrant is based on the idea, that you have a set of base-images, which you use as a starting point to apply further modifications using a configuration management tool or good old simple scripts. There are plenty of boxes available, but usually projects established their own process and guidelines how a box should look like (e.g. filesystem-wise). So did we and this is what we refer to as a stemcell. Once you have such a stemcell for different virtualization providers, you can publish them somewhere and people can consume them for their vagrant setups, without redoing the OS installation on their own. Building such a stemcell is no magic and there are different tools available, that make this process very easy: e.g. Veewee or Packer.

We first went with VeeWee, but given that packer is more or less the successor and makes a few things much easier, we ported things over to packer. We already have our own internal kickstarting server (called iBoot), that makes it very easy for us to kickstart (or preseed) new VMs or physical machines based on such a file. So that each installed machine looks the same. Hence, it was easy to just adapt these existing kickstart files for a setup with packer. This is what the ibox-stemcells repository contains.

packer

Packer works in the way, that you write a template, which describes how your images (in our terms: stemcells) should be built on different virtualization platforms. For each of them, you describe a builder. Packer also comes with an internal http server, that can be used to serve the kickstart files or anything else that should be available at installation time. Packer also downloads your (net-)install images, boots the new VM using them, enters the right boot parameter, kicks off the installation and so on.

Once the installation is done, packer will apply a set of defined provisioners that could do further modification of the installed system and could even kick off a configuration management tool. However, as we just want to have a very basic image, which matches what we have after the very basic OS installation, we only apply a few things through a provisioner as a postinstall script. The basis should be as clean as possible, so that we can develop our puppet modules on top of it using Vagrant. This process will be part of a next blog post. Using packer, we now have 2 CentOS 7 stemcells available: One for kvm (qemu) and another one for Virtualbox, which should make it very easy for most people to use them.

There is also already a builder for a vmWare stemcell in place. However, the author failed to get an usable vmWare installation running in an appropriate timeframe and left it out as an exercise for the interested people.

Also it should be very easy to create stemcells for Debian, previous CentOS versions or any other intersting distribution. This should make it really easy to be able to not only develop our manifests on CentOS 7, but adding more support for other distributions.