Introduction

I’m probably not exaggerating when I tell you that Kubernetes secrets are the most critical piece of information hosted in your cluster. It doesn’t matter how well you secure, for example, your PostgreSQL DB running in Kubernetes if a malicious person gets hold of the credentials to access it.

Furthermore, automation such as CI/CD and GitOps make things worse rather than better because credentials end up scattered around and often stored in insecure locations such as Git repositories (yes, it happens often and you know it, don’t lie to me).

What should I do?

The first thing is to protect access to the Kubernetes API making sure only the people that actually need access will be able to get hold of the secrets.

Secondly, use, when possible, a secret store. There are many around but just to list a few of the most popular ones:

• HashiCorp Vault: it’s a simple yet powerful server supporting multiple authentication methods with a great API to make integrations easy
• Mozilla SOPS: this is not a secret store strictly but it can be used as one. SOPS is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.
• AWS Secrets Manager: a key value store for credentials which also allows you to rotate passwords for AWS services such as RDS.
• Google Secrets Manager: very similar to AWS

I could go on for a long while but these are probably amongst the most widely used.

Now what?

You now have the secrets stored securely but in the wrong place. You need them in Kubernetes which is where your applications are. We at Digitalis manage Kubernetes clusters for many companies and each of them has their own preferred secret store.

As we needed a way to get the secrets securely into Kubernetes we came up with vals-operator.

What is it?

Vals-Operator is a Kubernetes Operator that retrieves secrets from a secret store and stores them as Kubernetes secrets in the desired cluster and namespace. Also, it will make sure that if the secret changes at source (password rotation or TLS certificate renewal for example) the Kubernetes secret will update accordingly.

Vals-Operator uses at its core vals, a golang library and CLI tool to inject secrets into a YAML/JSON file. We support any backend supported by vals as well as internal sync of secrets between namespaces.

Tell me more

Once you have installed it from the helm chart, it’s very easy to use. Have a look at the example below. It uses multiple backends for the demonstration and it pulls secrets from HashiCorp Vault, AWS Secrets and another namespace.

apiVersion: digitalis.io/v1
kind: ValsSecret
metadata:
  name: vals-secret-sample
  labels:
    owner: digitalis.io
spec:
  name: my-secret # Optional, default is the resource name
  ttl: 3600       # Optional, default is 60 seconds
  type: Opaque    # Default type, others supported
  data:
    username:
      ref: ref+vault://secret/database/username
      encoding: text
    password:
      ref: ref+vault://secret/database/password
      encoding: text
    ssh:
      ref: ref+vault://secret/database/ssh-private-key
      encoding: base64
    aws-user:
      ref: ref+awssecrets://kube/test#username
    aws-pass:
      ref: ref+awssecrets://kube/test#password
    ns-secret:
      ref: ref+k8s://namespace/secret#key

The syntax for ref+k8s://namespace/secret#key is unique to us and not supported by vals. This is a great way of duplicating a secret across namespaces and a great all-in-one alternative to tools such as kubed.

Demo

If you’d like to see a demo of Vals-Operator at work, check out the video below.

Conclusion

Vals-Operator closes the gap between external secret stores and Kubernetes. It also means you can have a single source of truth for all your clusters and you won’t need to worry about how to keep Kubernetes secret in sync.

Digitalis is a SUSE Rancher Gold partner and we have also added the operator to the marketplace.

Contact us if you require assistance with your enterprise-grade security implementation of Kubernetes.

‍