Introduction

Security is one of the most important considerations when running any database. But it is also one of the most challenging.

You don’t just want to encrypt the data between nodes and clients but also to ensure the SSL certificates are not used for a long time.

The safer thing to do is to renew certificates frequently. If for some reason they are leaked, they will not be valid for too long restoring security to the data.

There are multiple solutions to this problem. Mine is to use HashiCorp Vault as a CA and consul-templates for the rotation.

HashiCorp Vault

I configure my vault cluster using terraform, it’s the easier way to manage the configuration. If you don’t use terraform, you can find instructions on how to set it up manually from HashiCorp’s website.

The code below performs all necessary configs to allow clients to issue certs for a domain called here example.com and all its subdomains.

locals {
  dns_domains = [
    "example.com",
  ]
}

resource "vault_mount" "pki" {
  path                      = "pki"
  type                      = "pki"
  default_lease_ttl_seconds = 691200  # 8 days
  max_lease_ttl_seconds     = 2764800 # 30 days
}

resource "vault_pki_secret_backend_role" "pki" {
  backend            = vault_mount.pki.path
  name               = "vdp"
  ttl                = 2419200
  allow_ip_sans      = true
  key_type           = "rsa"
  key_bits           = 4096
  allowed_domains    = local.dns_domains
  allow_subdomains   = true
  allow_glob_domains = true
}

resource "vault_pki_secret_backend_intermediate_cert_request" "axonops" {
  depends_on  = [vault_mount.pki]
  backend     = vault_mount.pki.path
  type        = "internal"
  common_name = local.dns_domain[0]
}

resource "vault_pki_secret_backend_root_cert" "axonops" {
  depends_on           = [vault_mount.pki]
  backend              = vault_mount.pki.path
  type                 = "internal"
  common_name          = "Root CA"
  ttl                  = "315360000"
  format               = "pem"
  private_key_format   = "der"
  key_type             = "rsa"
  key_bits             = 4096
  exclude_cn_from_sans = true
  ou                   = "Cassandra"
  organization         = "AxonOps"
}

resource "vault_pki_secret_backend_root_sign_intermediate" "root" {
  depends_on           = [vault_pki_secret_backend_intermediate_cert_request.axonops]
  backend              = vault_mount.pki.path
  csr                  = vault_pki_secret_backend_intermediate_cert_request.axonops.csr
  common_name          = "Intermediate CA"
  exclude_cn_from_sans = true
  ou                   = "Cassandra"
  organization         = "AxonOps"
}

resource "vault_pki_secret_backend_config_urls" "axonops" {
  backend = vault_mount.pki.path
  issuing_certificates = [
    "https://vault:8200/v1/pki/ca",
  ]
  crl_distribution_points = [
    "https://vault:8200/v1/pki/crt",
  ]
}

Consul Template

There are many ways to create SSL certs from Vault and configure them in Cassandra. You can use the following:

• a simple cron job with a script
• orchestration such as Ansible
• remote execution using tools such as Rundeck
• long etc here

I have chosen to use consul-templates. The consul-template daemon can query Vault to retrieve the SSL cert with two added bonuses: it will update the cert when it expires and it can run an arbitrary command (a script here) that I will use to reload the certificates.

You can save the template definition (contents in the example) into different files but I prefer to use it all in one as it reduces the number of config files I need to copy over to each Cassandra node.

vault {
  address = "https://vault:8200"
  token = "TOKEN"

  # must be false if using root token
  renew_token = false
  ssl {
    enabled = true
    verify = false
  }
}
template {
    contents = "{{ with secret \"pki/issue/vdp\" \"common_name=cass000.domain.com\" \"ip_sans=1.1.1.1\"}}{{ .Data.private_key }}{{ end }}"
    destination = "/opt/ssl/cassandra-certs.key"
    create_dest_dirs = true
    command = "/opt/consul_template/cassandra-certs/restart.sh"
}
template {
    contents = "{{ with secret \"pki/issue/vdp\" \"common_name=cass001.domain.com\" \"ip_sans=1.1.1.1\"}}{{ .Data.certificate }}{{ end }}"
    destination = "/opt/ssl/cassandra-certs.crt"
    create_dest_dirs = true
    command = "/opt/consul_template/cassandra-certs/restart.sh"
}
template {
    contents = "{{ with secret \"pki/issue/vdp\" \"common_name=cass002.domain.com\" \"ip_sans=1.1.1.1\"}}{{ .Data.issuing_ca }}{{ end }}"
    destination = "/opt/ssl/cassandra-certs.ca"
    create_dest_dirs = true
    command = "/opt/consul_template/cassandra-certs/restart.sh"
}

Finally, you need something to convert the PEM files into keystore.jks This is the function of the restart.sh script invoked from consul-templates:

#!/bin/bash

CA=/opt/ssl/cassandra-certs.ca
CRT=/opt/ssl/cassandra-certs.crt
KEY=/opt/ssl/cassandra-certs.key
NODE=cass001.example.com
TRUST_STORE_PASSWORD="CHANGEME"
JKS_PASSWD="CHANGEME"
TMPDIR=$(mktemp -d /tmp/key.XXXXX)
TMPKEYSTORE="$TMPDIR/keystore.jks"
TMPTRUSTSTORE="$TMPDIR/truststore.jks"
P12TMP="$TMPDIR/keystore.p12"

trap "rm -rf ${TMPDIR}" EXIT

cd /opt/ssl
openssl verify -CAfile $CA $CRT

# Generate PKCS12 keystore from private key, host cert and intermediates
openssl pkcs12 -export -in "$CRT" -inkey "$KEY" -name $NODE -out "$P12TMP" -passout pass:"$JKS_PASSWD"

# Import the PKCS12 into a JKS keystore
keytool -noprompt -importkeystore -deststorepass "$JKS_PASSWD" -destkeystore "$TMPKEYSTORE" -srckeystore "$P12TMP" -srcstoretype PKCS12 -srcstorepass "$JKS_PASSWD"
# Add the CA certificate to the keystore
keytool -noprompt -import -alias caroot -keystore "$TMPKEYSTORE" -keyalg RSA -storepass "$JKS_PASSWD" -file "$CA"

# Generate the truststore containing just the CA certificate
keytool -noprompt -import -alias caroot -keystore "$TMPTRUSTSTORE" -keyalg RSA -storepass "$TRUST_STORE_PASSWORD" -file "$CA"

cp -f $TMPKEYSTORE /opt/cassandra/ssl/keystore.jks
cp -f $TMPTRUSTSTORE /opt/cassandra/ssl/truststore.jks

/opt/cassandra/bin/nodetool -Dcom.sun.jndi.rmiURLParsing=legacy reloadssl

Note the last command in the script, this is the most important. It causes Cassandra to reload the SSL without restarting. Pretty cool!

/opt/cassandra/bin/nodetool -Dcom.sun.jndi.rmiURLParsing=legacy reloadssl

Conclusion

Improve your security with strong SSL certificates and make sure you rotate them often.

‍