Introduction

At Digitalis we plan deployments with a security-first mentality. We have many customers in the financial sector and we adhere to PCI-DSS. We encrypt everything, both in transit and at rest. And sometimes we need to go the extra mile and ensure only certain applications are able to connect to the https endpoints.

This is why we use Mutual TLS (mTLS) for some ingresses.

Traefik vs NGINX

We use both on most deployments but we tend to prefer Traefik for outside load balancers because it offers more routing configurations that we often require.

HashiCorp Vault

We extensible use HashiCorp Vault. In the last couple of years, it has become core and essential in all our customer-managed deployments. As we need to issue SSL certificates to both clients and servers, the vault pki support fits nicely within our needs.

We simply configure cert-manager to issue SSL certificates via Vault. The clients can also request and get their signed SSL certificates to connect to the server. See below a sample terraform code to configure cert-manager is as follows:

resource "helm_release" "certs" {
  name       = "jetstack"
  repository = "https://charts.jetstack.io"
  chart      = "cert-manager"
  version    = "v1.11.0"

  create_namespace = true
  namespace        = "cert-manager"
  wait             = true
  wait_for_jobs    = true
}

resource "kubernetes_service_account" "vault_issuer" {
  metadata {
    name      = "vault-issuer"
    namespace = "cert-manager"
    annotations = {
      "kubernetes.io/service-account.name" = "vault-issuer"
    }
  }
  depends_on = [
    helm_release.certs,
  ]
}

resource "kubernetes_secret" "vault_issuer" {
  metadata {
    name      = "vault-issuer-token"
    namespace = "cert-manager"
    annotations = {
      "kubernetes.io/service-account.name" = kubernetes_service_account.vault_issuer.metadata[0].name
    }
  }
  type = "kubernetes.io/service-account-token"
}

resource "kubectl_manifest" "vault_issuer" {
  yaml_body = yamlencode({

    apiVersion = "cert-manager.io/v1"
    kind       = "ClusterIssuer"
    metadata = {
      name      = "vault-issuer"
      namespace = "cert-manager"
    }
    spec = {
      vault = {
        auth = {
          kubernetes = {
            mountPath = "/v1/auth/kubernetes"
            # you'll need this role in Vault with permssions to issue
            # and sign ssl certs
            role      = "vault-issuer"
            secretRef = {
              key  = "token"
              name = kubernetes_secret.vault_issuer.metadata[0].name
            }
          }
        }
        # FIXME: add here your vault config
        path   = "pki/sign/my-domain"
        server = "https://vault:8200"
      }
    }
  })
  depends_on = [
    helm_release.certs,
    kubernetes_secret.vault_issuer,
  ]
}

Traefik

Now with cert-manager up and running, the next part is to deploy your application with the mTLS configuration.

For the example below, let’s assume I’m deploying an app called httpbin for the domain httpbin.my-domain.com

We need two configuration objects, one for the ingress and one for TLSoptions. See below the most important settings.

• cert-manager.io/cluster-issuer tells the ingress controller how to request the SSL certs via cert-manager and
• cert-manager.io/common-name the domain name to use. The SSL certs will be saved into a Kubernetes secret called secretName. This is also used by the TLSoptions configuration.
• traefik.ingress.kubernetes.io/router.tls.options points to the mTLS configuration to use
• clientAuthType sets the ingress to only allow connections with the right SSL certificates

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: vault-issuer
    cert-manager.io/common-name: httpbin.my-domain.com
    cert-manager.io/private-key-size: "4096"
    external-dns.alpha.kubernetes.io/hostname: httpbin.my-domain.com
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls.options: default-mtls@kubernetescrd
  name: httpbin
  namespace: default
spec:
  ingressClassName: traefik
  rules:
  - host: httpbin.my-domain.com
    http:
      paths:
      - backend:
          service:
            name: httpbin
            port:
              number: 8080
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - httpbin.my-domain.com
    secretName: httpbin-certs

TLSOptions

apiVersion: traefik.containo.us/v1alpha1
kind: TLSOption
metadata:
  name: mtls
  namespace: default
spec:
  clientAuth:
    clientAuthType: RequireAndVerifyClientCert
    secretNames:
    - httpbin-certs
  minVersion: VersionTLS12

Gotchas

It took me a long time to find the documentation of the syntax for traefik.ingress.kubernetes.io/router.tls.options. See the example below:

traefik.ingress.kubernetes.io/router.tls.options: {namespace}-{resource-name}@kubernetescrd

Conclusion

Security can never be underestimated. TLS is challenging and mutual TLS even more. That’s why having the right automation and CI/CD is fundamental to success. Be it with Apache Cassandra or web services, we implement it always: it’s our default.

Cert-manager and HashiCorp Vault are core applications to me and for good reason. They work fantastically well and I use them as part of every single deployment I implement.

‍