Introduction

At Digitalis we manage different databases for many customers in the financial sector where data security and access controls are extremely important.

For the databases we managed, whenever possible, we perform SSL certificate rotation making sure we don’t have long-lived certs. We also encourage our AxonOps customers to do the same with Cassandra.

But password rotation is harder to enforce because it can be more difficult and time-consuming to implement as it requires updating the passwords for all the clients and reconnecting them to the database using the new credentials.

HashiCorp Vault Database Credentials

Vault offers a feature where you can ask it to manage the credentials to a number of databases. I won’t get into details, you can read more about it on their website.

On a project I’ve been working on lately I thought it would be a good idea to use Vault to provide short-lived passwords to our DBAs to connect to the Cassandra clusters. It only took me a few minutes to set it up and it worked very well.

I have multiple Cassandra clusters and many clients running in Kubernetes. And then I thought, can I use this to provide credentials and rotate passwords for the Kubernetes clients?

Yes, I can!

Vals-Operator DbSecret

Vals-Operator is a Kubernetes operator we developed and use heavily on all our Kubernetes deployments for secrets management.

The new feature we just released provides the ability to obtain a database credential from Vault and store it as a Kubernetes Secret. It will also:

• Renew expired credentials
• Revoke credentials when the secret is deleted
• Perform a rollout (rolling restart) to any number of Deployments or StatefulSets to make them use the new credentials

apiVersion: digitalis.io/v1beta1
kind: DbSecret
metadata:
  name: cassandra
spec:
  renew: true # this is the default
  vault:
    role: readonly
    mount: cass000
  rollout: # optional: run a `rollout` to make the pods use new credentials
    - kind: Deployment
      name: cassandra-client
    - kind: StatefulSet
      name: cassandra-client-other

The example above is for Cassandra but it will work with any database supported by Vault. The syntax is the same for all. This resource definition is the equivalent of running

~$ vault read cass000/creds/readonly
Key                Value
---                -----
lease_id           cass000/creds/readonly/NiFIaq2ZOg4xowI0JUlnz0It
lease_duration     720h
lease_renewable    true
password           random_password
username           v_readonly_vrqraguqlj3p2takeh5h_1674597849

and storing the results in a Kube secret for the application to use.

Conclusion

Data security is the most important part of any project. You want to restrict who can access the data at all times. And you want to change the passwords often to avoid old “forgotten” passwords being used against you. I remember I company I worked for years ago where during a PEN testing exercise they managed to get access to the database because they found an unused password stored in a DBA's computer.

The advantage of this solution is you can set it up and forget. Vals-Operator will manage the lifecycle of your passwords. In conjunction with Vault, it is a winning strategy.

We no longer have any static passwords to access our clusters. Even the DBAs must request access through Vault and the password doesn’t last more than 1h.

And if you use AxonOps to monitor your Cassandra Clusters for access violations, you are fully covered.

‍