How we configure services: ibox types

As previously mentioned we are using the ibox project as a way to refactor, modernize and share our automation setup with other interested folks. As we are looking back to around 10 years of automating our services using puppet, there might one or the other place where it’s time to do such a refactor. So this whole project is a slow but steady process to make our plans happen: That we – internally, but also others – are able to replicate parts of our infrastructure on a local environment to easily renew, improve, debug or extend it.

Since our last posting we migrated quite a bunch of services to the ibox structure and so it might be a good time to share what you can do with it now. Also it’s a good opportunity for us to get things at least somehow a bit in a shape we can share it (read document).

ibox types

ibox types are somewhat our puppet roles and within our infrastructure in general we try to isolate things as it makes sense. This means we might actually run different ibox types together on a certain node type and some ibox types might be purely isolated within their VM. The same is possible with the ibox allowing you quite nice setups as you might see later.

Though something should be mentioned: we are never removing types from our nodes and so this is also not something we want to enable within the ibox. Removing craft respecting all kinds of different combination can become very cumbersome and given that the ibox’s idea is to easily replicate something within a local development environment, we actually don’t support that here either.

You should be able to easily get an overview of the different types by looking at the ibox::types:: space within the ibox module. Also we try to share certain examples within the vagrant.yaml.example file.

As you might see there is broad range of our services available and more will come in the future. To give you some idea on how to start using that concept, we give you an example how to use a few of the available types.

webhosting & webservices

Besides what most people are using from us – email & other communication services – we are also offering webhosting for your static or php-based websites. Most of the things that are driving the setup for such a webhosting are part of the webhosting and apache modules. You can easily use them to for example replicate your configuration of a webhosting with us locally on your machine using the webhosting type and the following hiera data example.

In short what this does is:

  1. configure a few defaults for a webhosting host, especially with regards to SFTP access.
  2. Setup the databases required for the hostings.
  3. Configure a simple php hosting with installing the php security checker of sektions eins to be automatically installed (Browse to http://php56.ibox-one.local/phpconfigcheck.php to verify our configuration). We activated only one as it should show what the others do as well.
  4. Automatically install a full WordPress installation, that is setup and ready to serve your blogpost. You can retrieve wordpress’ admin password using trocla get wordpress_adminuser_wp.ibox-one.local plain
  5. Automatically install a full Mediawiki installation, that is setup and ready to organize your content.
  6. And last but not least: A SimpleMachine Forum installation ready to go through the web installer. Something which we didn’t yet automate to the end, but we would take merge requests for that! 🙂

All these installations match what and how we are installing things on our webhostings. So if you might have problem debugging something, this might be a good opportunity to get a real shell for your webhosting 🙂

# a webhosting type with mysql
# to illustrate we also install a bunch of hostings
ibox::types: ['webhosting','dbserver']
# we don't need postgres here
ibox::types::dbserver::is_postgres_server: false
# let's get some space for the webhostings
ib_disks::datavgs::www::size_data: 12G
# let's make sure our hostings can login
# over ssh but only using sftp and are chrooted
sshd::sftp_subsystem: 'internal-sftp'
sshd::allowed_groups: 'root sftponly'
sshd::use_pam: 'yes'
sshd::hardened: '+sha1'
sshd::tail_additional_options: |
  Match Group root
         PasswordAuthentication no

  Match Group sftponly
         PasswordAuthentication yes
         ChrootDirectory %h
         ForceCommand internal-sftp
         AllowTcpForwarding no
# the databases we need
ib_mysql::server::default_databases:
  'wp_test': {}
  'wiki_test': {}
  'smf_test': {}
ib_webhosting::hostings::php:
#  'php.ibox-one.local':
#    git_repo: 'https://github.com/sektioneins/pcc'
#  'php54.ibox-one.local':
#    git_repo: 'https://github.com/sektioneins/pcc'
#    php_installation: "scl54"
#  'php55.ibox-one.local':
#    git_repo: 'https://github.com/sektioneins/pcc'
#    php_installation: "scl55"
  'php56.ibox-one.local':
    git_repo: 'https://github.com/sektioneins/pcc'
    php_installation: "scl56"
# setup a wordpress hosting fully automatic
# mind the database above
ib_webhosting::hostings::wordpress:
  'wp.ibox-one.local':
    blog_options:
      dbname: 'wp_test'
      adminemail: 'admin@ibox.local'
# setup a mediawiki fully automatic
# mind again the database above
ib_webhosting::hostings::mediawiki:
  'mw.ibox-one.local':
    db_name: 'wiki_test'
    contact: 'admin@ibox.local'
    sitename: 'mw'
    db_server: '127.0.0.1'
# install a smf hosting, ready to be clicked
# through the webinstaller
ib_webhosting::hostings::simplemachine:
  'smf.ibox-one.local': {}
# setup all diffrent kind of php hostings, either using
# the system php installation or scl installations

Webservices are managed services that we are fully running and providing to our users. Such a service is for example coquelicot powering our dl.immerda.ch service. And you can actually setup your own, by adding the following hieradata:

# webservices
ibox::types: ['webservices',]
#ib_apache::services::webhtpasswd::htpasswd_name: 'ht.ibox-one.local'
# get a coquelicot up and running
ib_apache::services::coquelicot::instances:
  'dl.ibox-one.local': {}

And as an example we have another service htpasswd.immerda.ch of us added there as well.

tor onion services

Since last week we also integrated the way how we are setting up onion services into ibox. And so you can actually make your ibox also available as an onion service:

ibox::types: ['onion_service']
ib_tor::onion::services:
  "%{hostname}":
    '22': {}

So this will create an onionserver called ibox-one (you can get the onion address @ /var/lib/tor/ibox-one/hostname) which makes ssh accessible.

You can combine that and make for example a wordpress that you installed available over an onion service:

ibox::types: ['webhosting','dbserver','onion_service']
# we don't need postgres here
ibox::types::dbserver::is_postgres_server: false
# let's get some space for the webhostings
ib_disks::datavgs::www::size_data: 12G
# the databases we need
ib_mysql::server::default_databases:
  'wp_test': {}
# setup a wordpress hosting fully automatic
# mind the database above
ib_webhosting::hostings::wordpress:
  'wp.ibox-one.local':
    domainalias: ['*.onion']
    blog_options:
      dbname: 'wp_test'
      adminemail: 'admin@ibox.local'
ib_tor::onion::services:
  "wp_os":
    '80': {}

For simplicity reasons we are serving any .onion address for the wordpress host, so we don’t need to read out the generated onion address and rerun puppet, but you might get the idea. Please note that some components might need additional tuning to make them safe, when being used through a local onion service, as lot’s of service assume localhost to be a safe place.

And so you get your wordpress served through an onion service:

$ torsocks curl -I "$(vagrant ssh -c 'cat /var/lib/tor/wp_os/hostname' | cut -c 1-22)/"
HTTP/1.1 200 OK
Date: Fri, 18 Nov 2016 15:12:01 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9
Link: <http://wp.ibox-one.local/?rest_route=/>; rel="https://api.w.org/"
Content-Type: text/html; charset=UTF-8

XMPP man-in-the-middle via tor

Update: It was pointed out to us that the word ‘wide-spread’ below is misleading since the cumulative exit probability of those nodes was probably below .5%. What we wanted to say instead is that the number of domains affected was large, when a bad exit was involved.

We saw some wide-spread XMPP man-in-the-middle via malicious tor exit nodes during the last 24h. The attacks where only targeting starttls connections on port 5222. The mitm served forged self-signed certificates for various Jabber domains, one of them being our imsg.ch. The attack was orchestrated between multiple exit nodes acting in sync. All of them served the same set of forged certificates, allegedly created around midnight March 2nd to 3rd, using common names tailored to various XMPP servers.

We tried a small sample of XMPP servers. Out of which we recorded the following domains being intercepted:

  • freifunk.im
  • jabber.ccc.de
  • jabber.systemli.org
  • jappix.org
  • jodo.im
  • pad7.de
  • swissjabber.ch
  • tigase.me

For a handful other domains the connection attempts where dropped and google xmpp was the only one we found to be unaffected.

The exit nodes involved in this attack were reported to the tor project and seem to be dysfunctional by now. The ones we know of are:

  • FAFE24D8CF973BC38B54500DA666EEE44F02C642
  • 6269E9B3549012C44F518D2D123E41A4F320157E
  • 04DDEEAB315956AD956AF046338FB8E5B52B2DAF
  • DB213B8BD383A955CC383CAB76F8DD71A7198F47
  • 0276C54A43ABF27AB0247AF377952A306605FB8A
  • A1B3C065339D11AC361D85B0A3F9B59A23BD818A
  • CAD72B06527F8534640E8E47AEE38E2A3321B2D4
  • 5CBDC2AB702784154E7EFE7E6F87645EB107E8FA
  • BAB1FA3492162DB8058F464AD99203144F2AAF87
  • 187C2945109982516002FE6620E46A94B9B81A6E
  • 6269E9B3549012C44F518D2D123E41A4F320157E
  • FAFE24D8CF973BC38B54500DA666EEE44F02C642
  • FB134ED5DEE131B707270D9E99D72201CC96D668
  • E46021F7921EBB836B318863BDD432982BAA7BD0

Here is the certificate which was presented when you tried to access imsg.ch:

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 12273528369637281981 (0xaa54550634e8d0bd)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=imsg.ch
        Validity
            Not Before: Mar  3 12:08:43 2016 GMT
            Not After : Jan 10 12:08:43 2026 GMT
        Subject: CN=imsg.ch
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
        [...]
SHA1 Fingerprint=2C:F2:07:E8:19:ED:4E:CA:81:59:6E:3F:D8:59:52:B8:12:22:88:DB

What was curious is that the mitm SSL endpoint was sending a TLS session ticket. Does anybody have an idea if that could lead to an additional attack being carried out, or if it was merely an artifact of their SSL stack. E.g. one explanation we have is that the SSL terminator might have seen all packets originating from the same local tor daemon IP.

Here’s more log, for your convenience:

Certificate chain
 0 s:/CN=imsg.ch
   i:/CN=imsg.ch
-----BEGIN CERTIFICATE-----
MIICoDCCAYgCCQCqVFUGNOjQvTANBgkqhkiG9w0BAQUFADASMRAwDgYDVQQDEwdp
bXNnLmNoMB4XDTE2MDMwMzEyMDg0M1oXDTI2MDExMDEyMDg0M1owEjEQMA4GA1UE
AxMHaW1zZy5jaDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALqSTvWG
ohTiP9DJztuSki2NTLRUfC9/1gbby2TK9kKPFKLyeIyZDtvgQvigT+ToNfvOl2EW
XX4WwlQ9sWGY+C2nZrX5EmttE7zV1/v3tYcPf9Vdir83WXX8/IRauewMxghHd2kH
LTQCTjYtOao+b2VEBDf6QsOV3DEAeM3tbitgFHazmxGMBI0LvadT2NWDVK36gVri
nD4FTV52RydamGMN9hwLP8Lp5RFcmoIOf8xJcYkMRpusvTNDCWERJYHrSnq0fppE
Y2i6EIMBLKpfbatO7IVyy4E1zAMgcVzvnDBQ2DsmcFpjLrzBVmW73YnH7GsGcuPk
Z9ha1eCGML2VncsCAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAG98eWZQpv0V7e8Lt
i/ajKnZrFiv3kh6GwuJAXOH0dimhoNOaCMfwY2e30e3rxSNpXGr09mx6qozH2oh4
CbsZt2zd3jfeFeFe7bEWZijkigMcr+zI4IPpVeHfnZLidaMo/Pr5WMtaHgfO/kOC
UGboN3YbnVVQ7aFvrpvFhyfoemZD44O7ieFObKcmKHySXNyLrIYL1y9LNpWNZF5X
c6JsGHjFrefWI0smcd/aIyus0UTNJ/UaZwRxNbdlnSpXL0+nptnFadXqCo7ygZeE
ciqz/ckEIY0i4S39O1hO3LXEpT6gmlZnJt9Kffc3zLw5nS3IE+LpCbxnGNn53MCG
R977LA==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=imsg.ch
issuer=/CN=imsg.ch
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1948 bytes and written 490 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: [... 64 bytes ...]
    Session-ID-ctx: 
    Master-Key: [... 96 bytes ...]
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket: [... 0x100 bytes ...]

    Start Time: [...]
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)