How we test Ansible roles with Molecule and Claude

When I started working for Digitalis I was asked if I wanted to write a few blog posts from time to time. I quite like doing so and I jumped on board. I mostly write about my own experience and what I have learnt from friends and colleagues across my long career. I try to make them instructive, and very often I take the opportunity to learn more by researching before I write.

One of my very early posts, and still one of the most read, is about how to use Molecule to test Ansible roles. I'm still a massive fan of Molecule. It is brilliant at finding problems before they hit the end user. And lately, thanks to my best buddy Claude, I run a lot more tests with far higher coverage.

Have a look at the v0.2.0 release run of our Ansible collection. That is 96 Molecule jobs, all green, before a single byte was published to Ansible Galaxy.

Why? Because it's easy. AI takes what can be a bit of a chore and makes the product much better.

What Molecule testing looked like a few years back

If you read the original post you'll remember the starting point. I had a simple nginx role, I added a couple of Docker images to the platforms section of molecule.yml and I ran molecule converge.

platforms:
  - name: CentOS-7
    image: centos:7
  - name: Ubuntu-18
    image: ubuntu:18.04

If the role completed on both, I was happy. That was already a big step up from my old Jenkins and Docker scripts and it caught plenty of typos before they reached anyone. But if I'm honest it only ever tested the happy path. The role ran, nothing exploded, job done. I never checked what happened when somebody passed it a wrong variable and I rarely tested more than the default configuration.

Why? Because writing tests is a chore no one wants to do, and writing tests for the things that should fail is often counterintuitive.

What our Ansible collection tests today

The digitalis-ansible-collection currently has five roles: openbao, openbao_agent, postgres_cluster, prometheus and rabbitmq. Each one has its own GitHub Actions workflow that runs Molecule against a matrix of distributions and scenarios. For postgres_cluster it looks like this:

strategy:
  fail-fast: false
  matrix:
    distro: [rockylinux9, rockylinux10, ubuntu2204, ubuntu2404, ubuntu2604, debian12, debian13]
    scenario: [default, cluster, tls, restore, invalid-config]

That's seven distributions times five scenarios, 35 jobs for one role. The default scenario is what I used to do back in the day. The others are where it gets interesting. cluster builds a real multi-node PostgreSQL cluster, tls checks the certificates are wired up correctly and restore takes a backup and restores it. The OpenBao role has an init scenario that initialises and unseals a server and an invalid-version scenario that asks for a release which does not exist.

Source: the role workflows in .github/workflows of the collection repository.

Every role except rabbitmq also has an invalid-config scenario. This is the one I'm most pleased with.

Negative tests: checking the role rejects bad input

A role that fails halfway through a run is much worse than a role that refuses to start. If someone passes the OpenBao role a listener without an address, I want it to stop with a clear message before it installs a package or touches a config file.

We describe the expected behaviour as BDD feature files, written in Gherkin, that live next to the scenario. This is a trimmed version of rejects-invalid-config.feature for OpenBao:

Feature: Reject an invalid OpenBao server configuration

  Scenario Outline: Invalid variables are rejected before the host is changed
    When the openbao role is applied with <invalid input>
    Then the role fails with a message explaining the problem
    And the OpenBao package is not installed once every example has run

    Examples:
      | invalid input                                   |
      | an empty listener list                          |
      | a listener without an address                   |
      | a listener of an unknown type                   |
      | an unpinned plugin image without a checksum     |
      | a plugin checksum that is not 64 hex characters |
      | two plugins with the same type and name         |
      | an init threshold above the number of shares    |

The real file has 26 of those cases plus two more where OpenBao itself has to reject the generated HCL. The converge playbook turns each row into a test case with the bad variables and the error message we expect back:

openbao_test_variable_cases:
  - name: a listener without an address
    vars: { listeners: [{ tls_disable: true }] }
    expect: openbao_listeners must be
  - name: an init threshold above the number of shares
    vars: { init: true, init_shares: 2, init_threshold: 3 }
    expect: openbao_init needs

A small expect-rejection.yml loops over the cases, applies the role inside a block and fails the test if the role succeeds or fails with the wrong message. Trimmed, the core of it is:

- name: "Reject {{ openbao_test_case.name }}"
  block:
    - name: "Apply openbao with {{ openbao_test_case.name }}"
      ansible.builtin.include_role:
        name: openbao
    # Only reached if the role accepted the bad input
    - name: Fail because the role accepted invalid input
      ansible.builtin.fail:
        msg: "openbao accepted {{ openbao_test_case.name }}"
  rescue:
    - name: "Assert the failure explains {{ openbao_test_case.name }}"
      ansible.builtin.assert:
        that:
          - openbao_test_case.expect in ansible_failed_result.msg

At the end it runs bao version and checks the binary is not there, which proves none of the bad inputs got far enough to install anything.

Across the whole collection there are 13 feature files with 42 scenarios between them. We would never have written all of them by hand. Not because they are hard but because they are tedious. That's exactly the kind of work where Claude shines.

How we use Claude Code to write Molecule tests

We don't just open a chat window and ask for some tests. Across the company we run Claude Code with a set of specialist agents, each one with a narrow job and its own instructions. For Ansible work there are three:

• ansible-specialist reviews every role, task and Molecule scenario before it is committed. It checks idempotency, FQCN modules, lint compliance, variable naming and that the Molecule config will actually run in CI.
• ansible-devops-reviewer is the one that cares about coverage. If a change adds a feature that no Molecule scenario exercises, it says so and writes the YAML for the missing test.
• issue-writer drafts the ticket before the work starts. Its testing section has to name the actual Molecule scenarios and platforms. "Add tests" is not accepted.

On top of the agents every repository has a CLAUDE.md file with the rules Claude must follow. Some are shared by all our projects: tests are written as Gherkin feature files with a thin step layer, they cover the happy path and the invalid and edge cases, and nothing is merged until the pipeline is green with those tests in it. Others are specific to the collection, for example:

- Do not add a variable without a spec entry; CI fails on undocumented variables.
- Do not change a molecule scenario without keeping its `.feature` file in step.

The day-to-day loop is simple. We write or change the role, then ask Claude to read meta/argument_specs.yml and the tasks and describe what the role should do as feature files. For the invalid input scenarios that means going through every variable and asking what happens if it's empty, the wrong type or out of range. That list is the bit that takes one of us an afternoon and takes Claude a couple of minutes. From the feature files it writes the Molecule scenario, we run it locally against one distribution and the specialist agents review the lot before it goes anywhere near a pull request.

When something fails we fix either the role or the test. The fact that we end up fixing the role quite often means the tests are doing their job. A good number of the validation messages in our roles exist because a test asked a question the role had no answer for.

Then the pull request goes up and GitHub Actions runs the full matrix. If a single job is red it doesn't get merged.

What to watch out for

Like people, AI is very good at writing tests that pass and leaving out the ones that would fail. That is not the same as writing tests that are right. If the role has a bug, it's perfectly possible to get a test that asserts the bug. This is why I like the feature files so much. They are written in plain English, so we can read them in a couple of minutes and spot when the behaviour described is not what we wanted.

The other cost is CI time. Most of the jobs in that run take two to five minutes, but the PostgreSQL cluster scenario on Rocky Linux 9 took 16 minutes and 45 seconds. With 96 jobs you do burn through runner minutes. For a public repository on GitHub that's free. On a private one you may want to run the full matrix only on release tags and a smaller subset on pull requests.

Where to start with AI-assisted Molecule testing

If you already have Molecule set up, pick one role and add an invalid-config scenario. Ask your AI assistant to list every variable in the role and what a bad value looks like, write it up as a feature file and review it before you let it write any YAML. You'll be surprised how many of those cases your role currently accepts without complaining.

If you don't have Molecule yet, the original post still covers the basics and our collection is a working example you're welcome to borrow from.

I for one welcome our robot overlords.