Immerda Techblog

Background

At immerda.ch, we try to automate every aspect of our infrastructure, so we can work on more interesting things and let the repetitive and boring work be done by puppet. This means that we also manage a lot of users, required by the different services, with puppet, whether these are plain system-, mysql- or any other kind of users. For some of them, e.g. the SFTP-Users for the webhostings, we are also managing the passwords.

Up to now, we generated and hashed the passwords by hand and put them in our manifests, which means that the password hashes also ended up being version controlled by git. Managing the users and their passwors with puppet works very well and have proven to be a very stable solution. However, it has the disadvantage that a lot of (mainly hashed) passwords are laying around in different places in our infrastracture:

1. The host on which they are used: In the actual backend (shadow, mysql, …) and the puppet catalog.
2. On the puppet master: In the manifests that are checked out from git.
3. In the git repository: This means a) on our internal git server, but b) also checked out on different systems of immerda admins.

Point 1 and 2 are obvious and can’t be changed given that we want local authentication (no central ldap) for most parts of our infrastructure and given that we want to run puppet in master/agent mode as our source of truth for various reasons. Although, we take the protection of the data we handle serious and no immerda admin should ever work with any content from our systems on disks without strong encryption, we think that it is better to not spread data more than it is necessary. So point 3 was in our eyes always a bit annoying.

Meet trocla

To address this issue and also to make password generation a bit more comfortable, we use trocla. Trocla has 2 main parts:

1. A gem that provides all the logic and a little cli to work with the data
2. A puppet function to query trocla while compiling the manifests, which fetches the passwords from trocla (and thus generates them if not yet existing).

So instead of generating and hashing the passwords ourself and keeping them in our puppet manifests, read: in our git repository, we simply use a puppet function that will do all of these steps for us and keep our git repositories password free.

How trocla works

Trocla is a wrapper around a key/value storage. Actually, it was built that you can use any kind of key/value storage that is supported by a newer not yet released version of the moneta-gem. By default trocla uses a yaml backend, which should be sufficient for most use-cases. The keys are used in the manifests to lookup the passwords from trocla and the value would be the stored password. That’s more or less the big picture.

However, with only that feature set we could also simply stick with something like extlookup or hiera (or hiera-gpg) and just put our values in a storage file, that is not in our git repository. But lets get a step back and look at all the steps that need to be performed, if we set somewhere a password:

1. The plaintext password (which a user can later user to login)
2. The password in the format of the actual service. So for example for local users we use salted SHA-512 passwords, MySQL passwords are stored using a simple SHA1, etc.

Trocla extends this simply key/value lookup with a third argument named format. This argument refers to the format of the password that we are interested in and is used by the service/user we are managing. The format option actually refers to the algorithm that have been used to hash the password. And to automate things a little bit further: trocla will generate a random password, if it does not yet find a password for the key.

In short we can describe trocla’s workflow as followed:

1. Do I have the key/format tuple stored? Yes? -> Return the stored value.
2. Do I have the value for the key stored in the plain format? Yes? -> Generate the requested format, store it (for later lookups) and go to 1.
3. Otherwise: Generate a new random password, store it as plain format for that key and go to 2.

We need to store the hashed passwords, as we always want to return the same password hash for a certain key during multiple runs, so we don’t have to challenge puppet’s requirement for idempotency. Also, as mentioned above at some point (mainly in the beginning) you are usually also interested in the plain password, hence we store that one as well.

Workflow

Now, by using trocla, we are able to get rid off any passwords in our manifests and replace them with puppet-trocla-function calls and puppet will retrieve the passwords during catalog compilation in the right format. This means that the passwords are now only stored in 2 places:

1. On the host itself: In the compiled catalog and the backend.
2. On the puppet master: As hashed version and as plaintext password.

So, the only place where the plaintext password is stored is on the puppetmaster, which is anyway our source of truth and central point to manage all our systems. However, if we don’t need the plaintext password on the target host itself, it is not really necessary to keep the password on the puppetmaster. Still, our users should get the plaintext password, so they can actually login and use the service. Would be nice, not? 😉

If we keep trocla’s lookup in mind: Once the hashed password is generated and the plaintext password is not used in any place in the puppet manifests, there is no need to keep the plaintext password on the puppetmaster. As mentioned in the beginning, the trocla gem comes also with a little cli tool to work with its storage. All the different actions of that cli are explained in the README file and the one we are interested in is delete:

$ trocla delete user1 plain
# This will delete the plain password of the key user1 and return it.

The last part of how that command works is the most interesting: This action will not only delete the value of the supplied format, but will also return (read display) it! So we can get the plaintext password, while removing it the from the puppetmaster. 2 important things to remember at this point:

1. In the manifests, we usually only query the hashed format.
2. If the hashed password is once stored, trocla will directly return that stored format.

So to wrap up our workflow for generating passwords for our users works now the following way:

1. Add the new user to the puppet manifests and use the trocla function to query the passwords.
2. Let puppet run on the target host, so puppet manages the user, hence queries trocla for the password, which will generate the passwords in the first run, but subsequently directly return the hashed password.
3. Login on the puppet master and query the plaintext password by deleting it. This has the advantage that you not only got the password, but that it’s also not anymore stored on puppetmaster.

Note: Beware that you always delete only the plain format and not hashed format, or no format. The latter will delete and return all stored formats for that key, which is the same as a password reset and deleting a hashed format is only interesting if the format uses a salt and you’d like to resalt the hashed password, but keeping the plaintext format. However for both issues, there are other actions provided by the trocla cli.

Supported hashes and more

Trocla currently only supports a few hashes:

• bcrypt: -hashed passwords
• md5crypt: salted MD5-shadow passwords
• mysql: SHA1-Hashes for MySQL-Users
• pgsql: MD5 hashed passwords for PostgreSQL, that are salted with the username, which you need to pass as an option
• sha256crypt: salted SHA256-shadow passwords
• sha512crypt: salted SHA512-shadow passwords

However, trocla is built-in mind to easily extend it with further formats and if you look at the various formats you should be able to quickly get an idea how to extend trocla with further formats. Git pull requests are always welcome!

And to finish a few examples, how trocla is used in our manifests:

# common usage:
webhosting::static{'www.immerda.ch':
  ...
  password => trocla('webhosting_www.immerda.ch','sha512crypt');
}
# format requires an option:
postgres::role{'some_user':
  ...
  password => trocla('postgres_some_user','pgsql','username: some_user');
}

But we took that part even a step further and integrated the usage of trocla in completely transparent manner into our manifests. Examples can be found in the mysql module or the webhosting module.

Future

Trocla gives us now an automated integration of password storing and generation into puppet manifests. If you take the steps taken to that point a little bit further, we see plenty of more options to automate various things further and probably also to integrate them with other interfaces (to users?). So that various configuration parts of webhostings could be done by the users themselves, but would still be managed by puppet.