What is Ansible?

If you are reading this blog you probably know what Ansible is but in case this is new to you, let me give you a brief introduction.

In the past servers were installed and configured manually. This was quite tedious but ok when there were only a few servers to manage. However nowadays, the number of servers and their complexity, under management in the average company, has increased exponentially. Even more so when we talk about Infrastructure As Code when the servers are transient.

Also doing things manually often leads to errors and discrepancies between configurations and servers.

That is how automation came to be. There are multiple options these days probably the most widely used are Puppet, Chef and Ansible. All three allow us to manage the configuration of multiple servers in a way that is repeatable to ensure all servers have the same settings and that any new server we add into the mix will be identical to the others.

However the orchestration software is only going to be as good as the version and code management. If you do not keep track of the changes you’re making to (in our case) the Ansible code you will eventually have different configurations on servers and unrepeatable infrastructure.

The above example is a very simple playbook for installing nginx which reads the environment parameters from a file imported on runtime based on the env variable.

Version control

The most common way of keeping track of your changes to Ansible is using version control and the best version control software at the moment is git. People starting up with git find it slightly daunting to begin with but it is pretty powerful and used around the world.

By keeping your Ansible code in a git repository you will be able to track changes to the code. If you’re working on a project with little collaboration it is easy to fall into the temptation of committing all your changes straight into the master branch. After all, it’s just you and you know what you have done, right?

It may well be you have a fantastic memory and you are able to keep track but once multiple people start working on the same repository you will very quickly lose sight. Furthermore your configuration changes will no longer be repeatable. You cannot (easily) go back to the code you created two months ago and use it to set up a server. See the use case below:

Use Case

Let’s have a look at a use case and see what would happen depending on whether you are using versioned code or not (a bit more on versioning in the next section).

You have 10 servers in development and 20 in production. Your production servers have been running for the last year with no issues and very few updates. In the meantime you’ve been working on a new feature and testing it in the development servers.

Suddenly you’re in urgent need of building 5 more servers in production:

No Versioning

• The code in the git repository is no longer the same as you used to build the production servers
• The code is riddled with bugs because after all you’re working on new features
• Result: The new servers you just built don’t work or they work a different way

Versioning

• You know you used version 1.2.3 of the Ansible code last time the production servers were built
• You build the new servers using said version
• Result: You pat yourself in the back for a job well done!

As you can see having a versioned deployment would have helped in this case. This is a very simplistic way of explaining it but you can probably see how much of an advantage it is to use versions. Knowing what’s on each of your environments as oppose to thinking you know will add a large amount of peace of mind to your daily work.

Git Versions

Companies and individuals may take different approaches at versioning the git repositories. At the core of our version control we use branches and tags. We use branches to separate the work stream between individuals or projects and tags to mark a fixed point in time, for example, project end.

A branch is simply a fork of your code you keep separated from the main branch (usually called master ) where you can record your changes until they are ready for mainstream use at which point you would merge them with the master branch.

A tag by contract is a fixed point in time. Tags are immutable. Once created they have no further history or commits.

We allow deployments into development from git branches but we don’t allow deployments into the rest of the environments other than from tags (known versions).

We prefer to use tags in the format MAJOR.MINOR.HOTFIX (ie, 1.1.0). This type of versioning is called semantic versioning.

Hot fixing

I’m not going to explain how to create tags but I will go into some detail on how we manage hot fixes as this is quite different between companies. In this scenario we have a product called productX and we’re running version 2.0.0 on production.

We have confirmed there is bug and we need to update a single parameters on our Ansible code. If we take the current code on our repository and tag it as 2.13.0, which would be the next logical version number, we will be taking with us all changes between versions and the HEAD of the git repository, many of which have never gone through testing. What we do instead is we create a tag using the current version as a base. That way your version will be identical to the production version except for the fix you just introduced.

Ansible Playbooks, Ansible Roles and Ansible variables

Before we can talk about versioning our code, let’s take it apart. There are three areas where we do versioning separately:

• Playbook: it is a list of tasks, roles and configuration options (variables) you can apply to a server
• Variables: options you can use to customise your playbooks and roles
• Roles: instead of keeping everything in one playbook which can be quite difficult to manage you can subdivide them in roles

When making changes to Ansible code you will most likely be updating one or more of the above resources. We therefore need to keep track of everything keeping in mind that some areas like the roles are shared between deployments.

We separated the roles from the rest of the playbook. Each role is a git repository in its own right with a git tag for versioning. And we use ansible-galaxy at runtime to download the required versions every time the playbook is run.

Ansible Galaxy

Ansible Galaxy uses a simple yaml configuration file to list all the roles. Whilst you can use Ansible Tower or AWX this is not required. This is the prefer approach as it decreases the complexity and the number of servers we need to support.

Versions can be either a branch name or a tag. This adds the flexibility to test new features in the development environment without the need to update the requirements.yml file every time with a new tag.

Each of your roles will also need to be configured for Galaxy. It needs an additional file, meta/main.yml with a format like

If your role requires another one to run (dependent), you can add them to the dependencies section. You can also use SCM here for downloading the roles, though I would not recommend this as it will clash with the config in requirements.yml and you will end up having to maintain two different configurations.

The screenshot below represents a sample deployment which we refer to a product. You may have noticed there are no roles defined in this directory. We have the different variables, the tasks and finally the requirements.yml. As explained above, we keep them on their own git repositories and we include them with Ansible Galaxy on demand.

The product git repository is tagged every time any of the files it contains changes (except during development when we use branches) and this becomes the version we control to keep track of changes into our different environments.

We now have the two main components joined up.

As you can see in the diagram below we have one single version for the whole product, which in turn contains all the roles with their versions. Whenever we make a change we will always need to update the product repository and therefore a new version (tag) is created

Multiple environment configs

The best way in this scenario is to either have one playbook git repository per environment (preferred option) or to have one per environment.

Be aware that multiple is probably a good idea for large deployments but it can be quite painful to keep environments in sync. Many times I have seen the versions between environments become very different and unfortunately there is no magic pill to fix this other than to ensure there are good practices and that the whole team follows them. Automation is key.

Running Ansible

When using Ansible with Ansible Galaxy for role management there is an extra step before you can run the playbook which is downloading all roles referenced in the    requirements.yml. This is done using the ansible-galaxy command:

There are a couple of additional options worth mentioning:

• –force: by default ansible-galaxy will not override existing roles. If you previously downloaded let’s say version 1.0.0 and now you want 1.2.0 you’ll need to add this option to the above command. Otherwise you just get a warning in the screen but no updated repo.
• -p: the default is to download the roles to ~/.ansible/roles or whatever is set on the ansible.cfg but you can override the path with this option

Jenkins and Rundeck

We prefer to automate as much as we can, including running Ansible. Also we don’t encourage manual intervention. What I mean is we try not to log into servers whenever possible and use centralised tools such as Jenkins and Rundeck to run any command on the servers.

There are many advantages to automation tools such as Jenkins and Rundeck. To list a few:

• Access control: we control who can run Ansible
• Accountability: we record who ran Ansible and when
• Error checking: we can check the parameters are correct before proceeding
• Enforcing: we can enforce some basic standards such as ensuring code is run from a valid branch or tag
• Scheduled runs: we can schedule to run Ansible at certain times
• Notifications: Slack, PagerDuty, etc. If Ansible fails we want to know

Conclusion

Pretty much everyone is reluctant to introduce versioning into their code. After all, commit to master and run Ansible, what’s the worst that could happen? The worst will happen, it is only a matter of time. The good news is that implementing good DevOps principals is easy and once you build your automation around it, it becomes easy to manage.

The next time you need to rollback your code you will be grateful you can do so without having to cherry pick your last 100 git commits.

Be safe.

If you would like to know more about how to implement modern data and cloud technologies, into to your business, we at Digitalis do it all: from cloud migration to fully managed services, we can help you modernize your operations, data, and applications. We provide consulting and managed services on cloud, data, and DevOps for any business type. Contact us today for more information or learn more about each of our services here.