How to integrate Hashicorp Vault into your automation, builds and Kubernetes
l

by Sergio Rua

4 May, 2021

LinkedInTwitter

Introduction

Hashicorp Vault is a brilliant tool to keep your secrets stored. For me, that’s not its biggest strength. The best thing about Vault is that it provides a central place from where your application can access them. How often have you had the problem that your application failed because you updated the password in some places but not others? I had it multiple times until now.

I am not going to get into installing and setting up Vault, that could be another blog in the near future. Here I will explain how to use it in different contexts. All of the following assumes you have a Vault instance up and running and you have at least the VAULT_ADDR and VAULT_TOKEN environment variables set up.

Command line

The first step is on the command line. You can download the vault CLI from the Hashicorp’s website. Let’s use it to create our first secret, a simple user and password entry:

~$ vault kv put secret/demo username=admin password=foobar
Key              Value
---              -----
created_time     2021-04-21T09:44:36.328590564Z
deletion_time    n/a
destroyed        false
version          1

Great, now we have some data stored securely. How can we access it back? Simple, just use the vault command again:

~$ vault read secret/data/demo -format=yaml
data:
  data:
    password: foobar
    username: admin
  metadata:
    created_time: "2021-04-21T09:44:36.328590564Z"
    deletion_time: ""
    destroyed: false
    version: 1
lease_duration: 0
lease_id: ""
renewable: false
request_id: ee361d69-a5fb-dc86-1198-fb7abbd6a4eb
warnings: null

What about if we don’t have the vault CLI installed? No sweat, just use the API:

~$ curl -s -H "X-Vault-Token: $VAULT_TOKEN" ${VAULT_ADDR}/v1/secret/data/demo | jq .

{
  "request_id": "47d43632-31d8-a1d0-a8f8-cb0ff75204d1",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "data": {
      "password": "foobar",
      "username": "admin"
    },
    "metadata": {
      "created_time": "2021-04-21T09:44:36.328590564Z",
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Great, we have now learnt how to simply get a secret from the command line but what about integrations with Kubernetes, Ansible, etc? Let’s get into them. 

Ansible

With ansible we have two possibilities: 

  1. Use the vault API together with the uri module
  2. Use any of the community modules supporting Vault

Ansible URI

No need to install any community roles, just use the API similar to what was shown before:

- hosts: all
  tasks:
    - name: Pull secret
      uri:
        url: "{{ lookup('env', 'VAULT_ADDR') }}/v1/secret/data/demo"
        headers:
          X-Vault-Token:  "{{ lookup('env', 'VAULT_TOKEN') }}"
      register: secret

    - debug:
        msg: "Username is {{ secret.json.data.data.username }} and password is {{ secret.json.data.data.password }}"

Ansible Role

The community Hashicorp Ansible Role is more flexible as you can directly get your secrets into variables on a single line. It’s worth using this method if you have multiple secrets to read.

- hosts: all
  vars:
    username: "{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/data/demo:username') }}"
    password: "{{ lookup('community.hashi_vault.hashi_vault', 'secret=secret/data/demo:password) }}"
  tasks:
    - debug:
        msg: "Username is {{ username }} and password is {{ password }}"

As you can see the code is much simplified.

YAML/JSON configs with vals

This is one of my favourite tools for its simplicity but it does a brilliant job! Vals is a configuration manager that supports multiple secret backends. You can use it with most cloud providers (AWS, Google, Azure, etc) and of course with Hashicorp Vault.

Imagine you have a config file like this:

---
some: thing
other: thing
more:
  - one
  - two
  - three
# Secrets in Vault
username: ref+vault://secret/demo/username
password: ref+vault://secret/demo/password

Username and password field are references to the vault secret we have stored. Using vals you can just parse the above config and get the values:

~$ vals eval -f myconfig.yaml
more:
    - one
    - two
    - three
other: thing
some: thing
username: admin
password: foobar

As yaml is very popular to manage configs nowadays this is very handy indeed! Vals also supports JSON as both input and output. In fact, you can read a yaml and output a json. Nice, isn’t it?

~$ vals eval -f myconfig.yaml -o json | jq .
{
  "more": [
    "one",
    "two",
    "three"
  ],
  "other": "thing",
  "some": "thing",
  "username": "admin",
  "password": "foobar"
}

Kubernetes did you say? Checked!

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: default
stringData:
  username: ref+vault://secret/demo/username
  password: ref+vault://secret/demo/password
type: Opaque

And apply it with:

vals eval -f kube-secret.yaml | kubectl apply -f -

Vals is natively supported by helmfile (my preferred method of deploying helm charts) as well, so you have no excuses now! If you don’t use helmfile (why not!) it is possible to integrate Vault with regular helm but it is not as neat:

---
auth:
 username: ref+vault://secret/demo/username
 password: ref+vault://secret/demo/password

$ vals eval -f values.yml | helm install my my-helm-chart -f - 

Consul Template

Put it very simply. The consul-template daemon can query and watch Vault for changes and update your config files whenever the secret is updated. This can be very handy especially for SSL certificates using the pki engines. See the following sample configuration:

vault {
# settings to connect to vault go here
}

template {
    contents = "{{ with secret \"secret/demo\" }}username: {{ .Data.data.username }}\npassword: {{ .Data.data.password }}\n{{ end }}"
    destination = "/etc/my_app_config.yaml"
    create_dest_dirs = true
}

Now you just need to run it as a daemon and your password will never be out of sync again! consul-template will read the secret configured in the contents section and write it down in the format we specify to the file configured as destination.

vault {
# settings to connect to vault go here
}

template {
    contents = "{{ with secret \"secret/demo\" }}username: {{ .Data.data.username }}\npassword: {{ .Data.data.password }}\n{{ end }}"
    destination = "/etc/my_app_config.yaml"
    create_dest_dirs = true
}

Now you just need to run it as a daemon and your password will never be out of sync again! consul-template will read the secret configured in the contents section and write it down in the format we specify to the file configured as destination.

consul-template -config /path/to/config.hcl

And the output file /etc/my_app_config.yaml will look like:

---
username: admin
password: foobar

Jenkins

Last but not least our favourite CI tool! Yes, it also supports Vault via a plugin

node {
    // define the secrets and the env variables
    def secrets = [
        [path: 'secret/demo, engineVersion: 2, secretValues: [
            [envVar: USERNAME, vaultKey: ‘username’],
            [envVar: PASSWORD, vaultKey: ‘password’]]],
    ]

    withVault([vaultSecrets: secrets]) {
        sh “echo My username is ${USERNAME} and password is ${PASSWORD}”
    }
}

Conclusion

There are lots of interesting tools to use, both community and corporate-driven. Often the usefulness is not defined for the tool itself but how well it can adapt to your work, code and pipelines. HashiCorp Vault ticks many boxes in my opinion and it easily integrates with everything we do here.

Having a single source of truth for all our secrets drastically simplifies all our deployments as we no longer need to chase down where those passwords were stored. Furthermore, a well set up environment can take advantage of rotating passwords and secrets and keep the environment more secure.

If you would like to know more about how to implement modern data, streaming and cloud technologies into your business, we at Digitalis do it all: from cloud migration to fully managed services, we can help you modernize your operations, data, streaming and applications. We provide consulting and managed services on clouddata, and DevOps for any business type. Contact us for more information.

Categories

Archives

Related Articles