Ansible Versioning
l

4 Mar, 2021

LinkedInTwitter

What is Ansible?

If you are reading this blog you probably know what Ansible is but in case this is new to you, let me give you a brief introduction.

In the past servers were installed and configured manually. This was quite tedious but ok when there were only a few servers to manage. However nowadays, the number of servers and their complexity, under management in the average company, has increased exponentially. Even more so when we talk about Infrastructure As Code when the servers are transient.

Also doing things manually often leads to errors and discrepancies between configurations and servers.

That is how automation came to be. There are multiple options these days probably the most widely used are Puppet, Chef and Ansible. All three allow us to manage the configuration of multiple servers in a way that is repeatable to ensure all servers have the same settings and that any new server we add into the mix will be identical to the others.

However the orchestration software is only going to be as good as the version and code management. If you do not keep track of the changes you’re making to (in our case) the Ansible code you will eventually have different configurations on servers and unrepeatable infrastructure.

- hosts: all
  vars:
    env: production
  var_files:
    - "vars/{{ env }}.yml"
  tasks:
    - name: Install nginx
      package:
        name: nginx
        state: present

The above example is a very simple playbook for installing nginx which reads the environment parameters from a file imported on runtime based on the env variable.

Version control

The most common way of keeping track of your changes to Ansible is using version control and the best version control software at the moment is git. People starting up with git find it slightly daunting to begin with but it is pretty powerful and used around the world.

By keeping your Ansible code in a git repository you will be able to track changes to the code. If you’re working on a project with little collaboration it is easy to fall into the temptation of committing all your changes straight into the master branch. After all, it’s just you and you know what you have done, right?

It may well be you have a fantastic memory and you are able to keep track but once multiple people start working on the same repository you will very quickly lose sight. Furthermore your configuration changes will no longer be repeatable. You cannot (easily) go back to the code you created two months ago and use it to set up a server. See the use case below:

Use Case

Let’s have a look at a use case and see what would happen depending on whether you are using versioned code or not (a bit more on versioning in the next section).

You have 10 servers in development and 20 in production. Your production servers have been running for the last year with no issues and very few updates. In the meantime you’ve been working on a new feature and testing it in the development servers.

Suddenly you’re in urgent need of building 5 more servers in production:

No Versioning

  • The code in the git repository is no longer the same as you used to build the production servers
  • The code is riddled with bugs because after all you’re working on new features
  • Result: The new servers you just built don’t work or they work a different way

Versioning

  • You know you used version 1.2.3 of the Ansible code last time the production servers were built
  • You build the new servers using said version
  • Result: You pat yourself in the back for a job well done!

As you can see having a versioned deployment would have helped in this case. This is a very simplistic way of explaining it but you can probably see how much of an advantage it is to use versions. Knowing what’s on each of your environments as oppose to thinking you know will add a large amount of peace of mind to your daily work.

Git Versions

Companies and individuals may take different approaches at versioning the git repositories. At the core of our version control we use branches and tags. We use branches to separate the work stream between individuals or projects and tags to mark a fixed point in time, for example, project end.

A branch is simply a fork of your code you keep separated from the main branch (usually called master ) where you can record your changes until they are ready for mainstream use at which point you would merge them with the master branch.

A tag by contract is a fixed point in time. Tags are immutable. Once created they have no further history or commits.

We allow deployments into development from git branches but we don’t allow deployments into the rest of the environments other than from tags (known versions).

We prefer to use tags in the format MAJOR.MINOR.HOTFIX (ie, 1.1.0). This type of versioning is called semantic versioning.

Major version

Major version change should only occur when it is materially different to the previous version or includes backward incompatible changes.

Minor version

Progression over last version such as new feature of improvement over existing.

Hotfix/Patch

Applies a correction to existing repository without carrying forward new code.

Hot fixing

I’m not going to explain how to create tags but I will go into some detail on how we manage hot fixes as this is quite different between companies. In this scenario we have a product called productX and we’re running version 2.0.0 on production.

Hot Fix Diag

We have confirmed there is bug and we need to update a single parameters on our Ansible code. If we take the current code on our repository and tag it as 2.13.0, which would be the next logical version number, we will be taking with us all changes between versions and the HEAD of the git repository, many of which have never gone through testing. What we do instead is we create a tag using the current version as a base. That way your version will be identical to the production version except for the fix you just introduced.

[(master)]$ git checkout -b hotfix/2.0.1 2.0.0
Switched to a new branch 'hotfix/2.0.1'
[(hotfix/2.0.1)]$ echo hotfix > README.md
[(hotfix/2.0.1)]$ git commit -am 'hotfix: fixing something broken'
[hotfix/2.0.1 3cda6d4] hotfix: fixing something broken
 1 file changed, 1 insertion(+)
[(hotfix/2.0.1)]$ git push -u origin hotfix/2.0.1
Counting objects: 3, done.
Writing objects: 100% (3/3), 258 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To [email protected]:sample-repo.git
 * [new branch]      hotfix/2.0.1 -> hotfix/2.0.1
Branch hotfix/1.0.1 set up to track remote branch hotfix/2.0.1 from origin.
Now the changes have been committed you just need to tag it in readiness to deploy:
[(hotfix/1.0.1)]$ git tag 1.0.1
[(hotfix/1.0.1)]$ git push --tags
Counting objects: 1, done.
Writing objects: 100% (1/1), 156 bytes | 0 bytes/s, done.
Total 1 (delta 0), reused 0 (delta 0)
To [email protected]:sample-repo.git
 * [new tag]         1.0.1 -> 1.0.1
 * [new tag]         3.0.0 -> 3.0.0

Ansible Playbooks, Ansible Roles and Ansible variables

Before we can talk about versioning our code, let’s take it apart. There are three areas where we do versioning separately:

  • Playbook: it is a list of tasks, roles and configuration options (variables) you can apply to a server
  • Variables: options you can use to customise your playbooks and roles
  • Roles: instead of keeping everything in one playbook which can be quite difficult to manage you can subdivide them in roles

When making changes to Ansible code you will most likely be updating one or more of the above resources. We therefore need to keep track of everything keeping in mind that some areas like the roles are shared between deployments.

We separated the roles from the rest of the playbook. Each role is a git repository in its own right with a git tag for versioning. And we use ansible-galaxy at runtime to download the required versions every time the playbook is run.

Ansible Galaxy

Ansible Galaxy uses a simple yaml configuration file to list all the roles. Whilst you can use Ansible Tower or AWX this is not required. This is the prefer approach as it decreases the complexity and the number of servers we need to support.

- src: [email protected]:mygroup/ansible-role-nginx.git
  scm: git
  version: "1.0.0"
- src: [email protected]:mygroup/ansible-role-apache.git
  scm: git
  version: "1.3.0"
- src: [email protected]:mygroup/ansible-role-cassandra.git
  scm: git
  version: "feature/AAABBB"

Versions can be either a branch name or a tag. This adds the flexibility to test new features in the development environment without the need to update the requirements.yml file every time with a new tag.

Each of your roles will also need to be configured for Galaxy. It needs an additional file, meta/main.yml with a format like

---
galaxy_info:
  author: Sergio Rua <[email protected]>
  description: Digitalis Role for Blog
  company: Digitalis.IO
  license: Apache Licese 2.0

  min_ansible_version: 2.9

  platforms:
    - name: RedHat
      versions:
        - all
    - name: Debian
      versions:
        - all

  galaxy_tags:
    - digitalis
    - blog

dependencies: []

If your role requires another one to run (dependent), you can add them to the dependencies section. You can also use SCM here for downloading the roles, though I would not recommend this as it will clash with the config in requirements.yml and you will end up having to maintain two different configurations.

dependencies:
  - role: foo
    src: [email protected]:MyOrg/ansible-foo
    scm: git
    version: 0.1.0

The screenshot below represents a sample deployment which we refer to a product. You may have noticed there are no roles defined in this directory. We have the different variables, the tasks and finally the requirements.yml. As explained above, we keep them on their own git repositories and we include them with Ansible Galaxy on demand.

Sample Deployment

The product git repository is tagged every time any of the files it contains changes (except during development when we use branches) and this becomes the version we control to keep track of changes into our different environments.

We now have the two main components joined up.

As you can see in the diagram below we have one single version for the whole product, which in turn contains all the roles with their versions. Whenever we make a change we will always need to update the product repository and therefore a new version (tag) is created

Blog diagram

Multiple environment configs

In certain circumstances you may wish to have different configurations on your environments. For example if your product lifecycle is long or it has multiple streams you may be wanting to diverge configurations for a while.

The best way in this scenario is to either have one playbook git repository per environment (preferred option) or to have one per environment.

Blog diagram

Be aware that multiple is probably a good idea for large deployments but it can be quite painful to keep environments in sync. Many times I have seen the versions between environments become very different and unfortunately there is no magic pill to fix this other than to ensure there are good practices and that the whole team follows them. Automation is key.

None of these is worth doing if the team is not following the practices.

Running Ansible

When using Ansible with Ansible Galaxy for role management there is an extra step before you can run the playbook which is downloading all roles referenced in the    requirements.yml. This is done using the ansible-galaxy command:

ansible-galaxy install -r requirements.yml

There are a couple of additional options worth mentioning:

  • –force: by default ansible-galaxy will not override existing roles. If you previously downloaded let’s say version 1.0.0 and now you want 1.2.0 you’ll need to add this option to the above command. Otherwise you just get a warning in the screen but no updated repo.
  • -p: the default is to download the roles to ~/.ansible/roles or whatever is set on the ansible.cfg but you can override the path with this option

Jenkins and Rundeck

We prefer to automate as much as we can, including running Ansible. Also we don’t encourage manual intervention. What I mean is we try not to log into servers whenever possible and use centralised tools such as Jenkins and Rundeck to run any command on the servers.

There are many advantages to automation tools such as Jenkins and Rundeck. To list a few:

  • Access control: we control who can run Ansible
  • Accountability: we record who ran Ansible and when
  • Error checking: we can check the parameters are correct before proceeding
  • Enforcing: we can enforce some basic standards such as ensuring code is run from a valid branch or tag
  • Scheduled runs: we can schedule to run Ansible at certain times
  • Notifications: Slack, PagerDuty, etc. If Ansible fails we want to know

Conclusion

Pretty much everyone is reluctant to introduce versioning into their code. After all, commit to master and run Ansible, what’s the worst that could happen? The worst will happen, it is only a matter of time. The good news is that implementing good DevOps principals is easy and once you build your automation around it, it becomes easy to manage.

The next time you need to rollback your code you will be grateful you can do so without having to cherry pick your last 100 git commits.

Be safe.

If you would like to know more about how to implement modern data and cloud technologies, into to your business, we at Digitalis do it all: from cloud migration to fully managed services, we can help you modernize your operations, data, and applications. We provide consulting and managed services on clouddata, and DevOps for any business type. Contact us today for more information or learn more about each of our services here.

Sergio Rua

Sergio Rua

Senior DevOps Engineer

Sergio has many years experience working on various development projects before joining Digitalis. He worked for large companies with complex networks and infrastructure. ‘This has  helped Sergio gain lots of experience in multiple areas from programming to networks. He especially excels in DevOps: automation is his day-to-day and Kubernetes his passion.

Categories

Archives

Related Articles

ECS Container monitoring using cAdvisor

ECS Container monitoring using cAdvisor

In this topic, I will explain how to monitor docker containers running on an ECS cluster. Even though AWS CloudWatch is the preferred tool for monitoring and collecting container metrics, in some scenarios it is required to use alternative solutions.

Incremental backups with rsync and hard links

Incremental backups with rsync and hard links

There are many different options that control the behaviour of the backup process and how it determines what files to copy, link or delete, this blog describes how to build a simple incremental backup solution using rsync and hard links.

Containerized Cassandra Cluster for local testing

Containerized Cassandra Cluster for local testing

A blog on how to build clean and simple containerised Apache Cassandra cluster for local testing, using the official docker image, while preserving the data and having the ability to change any configuration needed.