Turning TLS on in Cassandra is the straightforward part. Certificates then expire by design, verification failures block handshakes by design, and partial rotations fail in ways that are easy to miss until they become incidents. This post configures AxonOps to alert on the main certificate failure modes on a Cassandra cluster: expiry, hostname and SAN mismatch, chain validation, truststore and CA expiry, on-disk drift, served-certificate mismatch after rotation, and runtime SSL errors in the logs.

Cassandra TLS internals, keystore and truststore hot reload in 4.0+, and zero-downtime rotation are covered in Apache Cassandra SSL Guide: Certificate Management and Zero-Downtime Reloading. This one is about alerting and operational detection.

Cassandra exposes two TLS endpoints, and both need checking

They differ by port and by whether the connecting side must present its own certificate.

Client-to-node encryption is usually one-way TLS on port 9042

CQL from applications, drivers and cqlsh arrives on the native transport port (native_transport_port, default 9042). Enabling client_encryption_options turns TLS on for this port; optional: true allows both plaintext and TLS.

The mode is set by require_client_auth. The default is one-way TLS: the client verifies the server and presents no certificate of its own. With require_client_auth: true it becomes mutual TLS (mTLS), where the client must present a trusted certificate. Client mTLS is common in security-conscious deployments, and in Cassandra 5.0 it also supports certificate-based authentication (MutualTlsAuthenticator).

The older dedicated SSL port (native_transport_port_ssl, default 9142) is deprecated in Cassandra 5.0, so use the single native port.

Internode encryption is usually mutual TLS on port 7000

Gossip, mutations, repair streams and hints ride the storage port (storage_port, default 7000). This is usually mTLS: each node authenticates its peers, so the connecting side must present a valid certificate too. In Cassandra 4.0+ encrypted internode traffic uses storage_port; the older ssl_storage_port (default 7001) is deprecated and is normally not bound.

Every listener check below starts with the client-to-node endpoint, because it is simplest to reach and the most visible when it fails. Run each of them a second time against the internode endpoint: switch the port to 7000 and add -cert and -key as required for mTLS. Do not treat internode TLS checking as optional. Its failures are often the quiet ones: hint backlogs, failed repairs, schema disagreement, and partial unavailability.

Eight failure modes a production certificate monitor has to cover

1. Certificate will expire within a threshold window.
2. Certificate is already expired, or notBefore is in the future.
3. Hostname or SAN does not match the address the node is serving on.
4. Chain does not validate against the truststore in use.
5. A CA in the truststore expires, invalidating every certificate it signed.
6. Keystore or truststore file on disk has drifted from expected content, or differs between nodes.
7. Keystore was rotated on disk but Cassandra is still serving the old certificate.
8. Runtime SSL errors appear in system.log.

Four AxonOps building blocks cover all eight

• Service checks run on each agent on a schedule. Shell, TCP and HTTP variants are supported. Changes are pushed automatically to agents.
• Log alerts match a regex against Cassandra and agent logs and trigger on a rate or a single occurrence.
• Metric alerts threshold on collected metrics. They are not the primary tool for certificate monitoring, but they help correlate certificate failures with application-visible symptoms.
• Integrations route alerts to Slack, Microsoft Teams, PagerDuty, OpsGenie, ServiceNow, webhook or SMTP, independently by source and severity.

Three implementation details to settle before writing any shell check

• The interpreter is controlled by the check's Shell field. Left blank it defaults to /bin/sh, which is often dash on Debian and Ubuntu. Every script below is written in POSIX sh.
• Cassandra config values are available as template variables inside the script. For example, {{.comp_listen_address}} and {{.comp_native_transport_port}} resolve at runtime on each node.
• If the agent uses disable_command_exec: true, only scripts placed in scripts_location can be executed. In that case, store scripts on disk via configuration management and reference them by path.

Anything written to stdout becomes the check output shown in the AxonOps UI and in the alert payload.

Give liveness its own TCP check, so certificate checks can stay quiet

Every certificate check below touches the SSL listener. If Cassandra is down because of a rolling restart, a node failure or a network blip, the endpoint is unreachable and there is no certificate to inspect. Returning Critical or Unknown from a long-interval certificate check creates alert noise, and can leave stale failures pinned until the next scheduled run.

The cleaner pattern is to split the two questions:

• One fast TCP check every few minutes owns the question is the SSL listener reachable?
• Certificate-content checks run every 1 to 12 hours and return OK on unreachable, alerting only on certificate-specific problems.

Go to Service Checks → Configurations → + → TCP and create:

Add a second check against :7000 for internode TLS.

Ports are hardcoded because an SSL-port template variable is not available consistently across all agent versions. On Cassandra 5.0 both endpoints are effectively single-port: encrypted client traffic rides 9042 and encrypted internode traffic rides 7000. Older clusters still using 9142 or 7001 should target those instead.

Check 1: read expiry off the live socket, not off the keystore

This check reads notBefore and notAfter from the certificate actually being served on the live socket, which covers both upcoming expiry and already-invalid timing conditions. Checking the listener rather than the keystore file makes sure the result reflects what Cassandra is currently serving.

To create the shell checks below, go to Service Checks → Configurations → + Add Shell Check.

Create a shell service check named Cassandra TLS cert expiry - client SSL with interval 1h and timeout 30s:

set -eu
 
HOST="{{.comp_rpc_address}}"
PORT=9042
WARN_DAYS=30
CRIT_DAYS=7
 
dates=$(echo | openssl s_client -connect "${HOST}:${PORT}" -servername "${HOST}" 2>/dev/null \
  | openssl x509 -noout -startdate -enddate 2>/dev/null || true)
 
if [ -z "${dates}" ]; then
  echo "OK: could not retrieve cert from ${HOST}:${PORT} (liveness owned by TCP check)"
  exit 0
fi
 
start_date=$(echo "${dates}" | sed -n 's/^notBefore=//p')
end_date=$(echo "${dates}"   | sed -n 's/^notAfter=//p')
 
start_epoch=$(date -d "${start_date}" +%s)
end_epoch=$(date -d "${end_date}" +%s)
now_epoch=$(date +%s)
 
if [ "${now_epoch}" -lt "${start_epoch}" ]; then
  echo "CRITICAL: cert on ${HOST}:${PORT} is NOT YET VALID (notBefore=${start_date})"
  exit 2
fi
 
days_left=$(( (end_epoch - now_epoch) / 86400 ))
 
if [ "${days_left}" -lt 0 ]; then
  echo "CRITICAL: cert on ${HOST}:${PORT} EXPIRED ${days_left#-} days ago (notAfter=${end_date})"
  exit 2
elif [ "${days_left}" -le "${CRIT_DAYS}" ]; then
  echo "CRITICAL: cert on ${HOST}:${PORT} expires in ${days_left} days (notAfter=${end_date})"
  exit 2
elif [ "${days_left}" -le "${WARN_DAYS}" ]; then
  echo "WARNING: cert on ${HOST}:${PORT} expires in ${days_left} days (notAfter=${end_date})"
  exit 1
fi
 
echo "OK: cert on ${HOST}:${PORT} valid (notBefore=${start_date}; expires in ${days_left} days, notAfter=${end_date})"
exit 0

For the internode variant, change the port to 7000 and add -cert <node.crt> -key <node.key> to the openssl s_client line.

Check 2: a valid certificate still fails verification if the SAN is wrong

An unexpired certificate fails verification when the SAN list does not contain the address clients or peers actually use. Internode verification is controlled by server_encryption_options.require_endpoint_verification; on the client side, hostname validation is enforced by driver SSL configuration.

That is what makes SAN drift easy to underestimate. With verification off, the gap stays invisible until a future config change turns it into a production incident. Continuous validation catches the mismatch early.

CN-only hostname verification is deprecated by RFC 6125, and modern Java and driver stacks generally require SANs when endpoint verification is enabled.

Create a shell service check named Cassandra TLS hostname-SAN match with interval 1h and timeout 30s:

set -eu
 
HOST="{{.comp_rpc_address}}"
PORT=9042
EXPECTED="${EXPECTED:-{{.comp_rpc_address}}}"
 
cert_text=$(echo | openssl s_client -connect "${HOST}:${PORT}" -servername "${EXPECTED}" 2>/dev/null \
  | openssl x509 -noout -text 2>/dev/null || true)
 
if [ -z "${cert_text}" ]; then
  echo "OK: could not retrieve cert from ${HOST}:${PORT} (liveness owned by TCP check)"
  exit 0
fi
 
san_line=$(echo "${cert_text}" | grep -A1 "Subject Alternative Name" | tail -n1 | tr -d ' ')
cn=$(echo "${cert_text}" | grep "Subject:" | sed -n 's/.*CN=\([^,]*\).*/\1/p')
 
if echo "${san_line}" | grep -qE "(DNS:|IPAddress:)${EXPECTED}([,]|$)"; then
  echo "OK: ${EXPECTED} present in SAN"
  exit 0
fi
 
if [ "${cn}" = "${EXPECTED}" ]; then
  echo "WARNING: ${EXPECTED} matches CN but is missing from SAN (modern clients require SAN)"
  exit 1
fi
 
echo "CRITICAL: ${EXPECTED} not in SAN (${san_line}) and not in CN (${cn})"
exit 2

For internode TLS, use port 7000, add -cert and -key, and set EXPECTED to the broadcast address or FQDN peers actually use.

Check 3: chain validation catches the wrong CA, not just the wrong date

This validates the live certificate against the node's truststore. It catches expired intermediates, certificates issued by the wrong CA, missing CA material, and accidental self-signed certificates.

Create a shell service check named Cassandra TLS chain validation with interval 1h and timeout 30s:

set -eu
 
HOST="{{.comp_rpc_address}}"
PORT=9042
TRUSTSTORE_PEM=/etc/cassandra/conf/truststore.pem
 
if [ ! -r "${TRUSTSTORE_PEM}" ]; then
  echo "CRITICAL: truststore PEM not readable at ${TRUSTSTORE_PEM}"
  exit 2
fi
 
result=$(echo | openssl s_client -connect "${HOST}:${PORT}" -servername "${HOST}" \
  -CAfile "${TRUSTSTORE_PEM}" -verify_return_error 2>&1 || true)
 
if ! echo "${result}" | grep -qE "Verify return code|verify error"; then
  echo "OK: could not retrieve cert from ${HOST}:${PORT} (liveness owned by TCP check)"
  exit 0
fi
 
verify_line=$(echo "${result}" | grep -E "Verify return code|verify error" | head -n1)
 
if echo "${verify_line}" | grep -q "Verify return code: 0 (ok)"; then
  echo "OK: chain verifies against ${TRUSTSTORE_PEM}"
  exit 0
fi
 
echo "CRITICAL: chain validation failed - ${verify_line}"
exit 2

If only a JKS truststore exists, export it once with keytool -list -rfc -keystore truststore.jks -storepass <pw> > truststore.pem.

Check 4: truststore CA expiry is the one expiry with a cluster-wide blast radius

Checks 1 to 3 watch the served certificate and validate its chain at connection time, but none of them warns before a CA in the truststore expires. When a trusted CA expires, every peer and client certificate signed by it fails validation at the same moment. Check 3 starts failing once that happens, with no lead time.

This check provides the advance warning. It walks each certificate in the truststore PEM and reports the soonest expiry. It reads a static file, so it is cheap.

Create a shell service check named Cassandra truststore CA expiry with interval 1h and timeout 15s:

# POSIX sh (see Check 1). Warns before a CA in the truststore expires, which the
# chain check only detects after the fact. Walks every cert in the bundle, not
# just the first.
set -eu
 
TRUSTSTORE_PEM=/etc/cassandra/conf/truststore.pem
WARN_DAYS=30
CRIT_DAYS=7
 
[ -r "${TRUSTSTORE_PEM}" ] || { echo "CRITICAL: truststore PEM not readable at ${TRUSTSTORE_PEM}"; exit 2; }
 
now_epoch=$(date +%s)
worst_rc=0
min_days=""
tmp=$(mktemp -d)
awk -v d="${tmp}" '/-----BEGIN CERTIFICATE-----/{n++} n{print > (d"/c." n)}' "${TRUSTSTORE_PEM}"
 
for f in "${tmp}"/c.*; do
  [ -e "${f}" ] || continue
  subject=$(openssl x509 -in "${f}" -noout -subject 2>/dev/null | sed 's/^subject= *//')
  end_date=$(openssl x509 -in "${f}" -noout -enddate 2>/dev/null | sed 's/notAfter=//')
  [ -n "${end_date}" ] || continue
  end_epoch=$(date -d "${end_date}" +%s)
  days_left=$(( (end_epoch - now_epoch) / 86400 ))
  if [ -z "${min_days}" ] || [ "${days_left}" -lt "${min_days}" ]; then min_days="${days_left}"; fi
  rc=0
  if [ "${days_left}" -lt 0 ]; then
    echo "CRITICAL: truststore CA EXPIRED ${days_left#-} days ago (${subject})"; rc=2
  elif [ "${days_left}" -le "${CRIT_DAYS}" ]; then
    echo "CRITICAL: truststore CA expires in ${days_left} days (${subject})"; rc=2
  elif [ "${days_left}" -le "${WARN_DAYS}" ]; then
    echo "WARNING: truststore CA expires in ${days_left} days (${subject})"; rc=1
  fi
  if [ "${rc}" -gt "${worst_rc}" ]; then worst_rc="${rc}"; fi
done
rm -rf "${tmp}"
 
if [ "${worst_rc}" -eq 0 ]; then
  echo "OK: all truststore CAs valid (soonest expiry in ${min_days:-?} days)"
fi
exit "${worst_rc}"

It reads the truststore in PEM. Export it once from your keystore format: from JKS with keytool -list -rfc -keystore truststore.jks -storepass <pw> > truststore.pem, or from PKCS12 with openssl pkcs12 -in truststore.p12 -nokeys (add -legacy if it was written with older algorithms). It walks every certificate in the bundle, so a multi-CA truststore is handled, and because it reads a static file it needs no -cert or -key and has no client or internode variant.

Check 5: file fingerprints catch the rollout that only reached half the nodes

Wire-level checks do not catch a bad file that has not been reloaded yet, or a rollout that updated only some nodes. A SHA-256 fingerprint of the keystore and truststore files gives you a simple way to compare nodes, and optionally to pin expected values.

Create a shell service check named Cassandra keystore and truststore fingerprint with interval 15m and timeout 10s:

set -eu
 
KEYSTORE=/etc/cassandra/conf/keystore.jks
TRUSTSTORE=/etc/cassandra/conf/truststore.jks
 
[ -r "${KEYSTORE}" ]   || { echo "CRITICAL: keystore unreadable at ${KEYSTORE}"; exit 2; }
[ -r "${TRUSTSTORE}" ] || { echo "CRITICAL: truststore unreadable at ${TRUSTSTORE}"; exit 2; }
 
ks=$(sha256sum "${KEYSTORE}"   | awk '{print $1}')
ts=$(sha256sum "${TRUSTSTORE}" | awk '{print $1}')
 
PIN_DIR="${PIN_DIR:-/etc/axonops/pins}"
EXPECTED_KS="${EXPECTED_KEYSTORE_SHA256:-$(cat "${PIN_DIR}/keystore.sha256" 2>/dev/null || true)}"
EXPECTED_TS="${EXPECTED_TRUSTSTORE_SHA256:-$(cat "${PIN_DIR}/truststore.sha256" 2>/dev/null || true)}"
 
msg="keystore=${ks} truststore=${ts}"
 
if [ -n "${EXPECTED_KS}" ] && [ "${ks}" != "${EXPECTED_KS}" ]; then
  echo "CRITICAL: keystore drift - ${msg}"
  exit 2
fi
if [ -n "${EXPECTED_TS}" ] && [ "${ts}" != "${EXPECTED_TS}" ]; then
  echo "CRITICAL: truststore drift - ${msg}"
  exit 2
fi
 
echo "OK: ${msg}"
exit 0

This check has two common operating modes:

• Pinned: expected hashes are recorded during rollout and any mismatch alerts immediately.
• Comparative: pins are left unset and operators compare output across nodes to spot outliers.

Check 6: a rotated keystore does not mean a rotated certificate

This catches the common rotation failure where the keystore file has been replaced, but Cassandra has not reloaded it and continues serving the old certificate from memory. Check 1 only notices that problem once the stale certificate is near expiry. This check detects it directly.

The example below assumes a PKCS12 keystore. For a JKS keystore, export the on-disk certificate with keytool -exportcert and then fingerprint it with openssl x509.

Create a shell service check named Cassandra served cert matches disk with interval 15m and timeout 30s:

set -eu
 
HOST="{{.comp_rpc_address}}"
PORT=9042
KEYSTORE="${KEYSTORE:-/etc/cassandra/conf/keystore.p12}"
 
PASS_FILE="${KEYSTORE_PASSWORD_FILE:-/etc/axonops/pins/keystore.pass}"
if [ -r "${PASS_FILE}" ]; then
  PASSIN="file:${PASS_FILE}"
else
  PASSIN="env:KEYSTORE_PASSWORD"
fi
 
live=$(echo | openssl s_client -connect "${HOST}:${PORT}" -servername "${HOST}" 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//')
if [ -z "${live}" ]; then
  echo "OK: could not retrieve cert from ${HOST}:${PORT} (liveness owned by TCP check)"
  exit 0
fi
 
disk=$(openssl pkcs12 -in "${KEYSTORE}" -clcerts -nokeys -passin "${PASSIN}" -legacy 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//')
# -legacy lets OpenSSL 3 read keystores written by older keytool
if [ -z "${disk}" ]; then
  echo "CRITICAL: cannot read leaf cert from ${KEYSTORE} (bad path or password?)"
  exit 2
fi
 
if [ "${live}" = "${disk}" ]; then
  echo "OK: served cert matches keystore on disk"
  exit 0
fi
 
echo "WARNING: served cert != keystore on disk - reload pending or failed"
exit 1

This check carries a secret: the keystore password. Read it from a file accessible only to the AxonOps agent user and pass it through -passin file: or env:, never directly on the command line.

Two assumptions are worth stating. The fingerprint comes from the first leaf certificate the keystore emits, so this assumes a single key alias. A keystore mid-rotation carrying two key aliases can leave openssl fingerprinting a different certificate than the one Cassandra serves, which is the very state this check should catch; if your rotation adds an alias rather than replacing the file, extract the specific alias with keytool -exportcert -alias <alias>, or match the live fingerprint against every leaf in the store. The -legacy flag lets OpenSSL 3 read keystores written with older keytool algorithms; it is required for those and harmless on modern ones.

A short mismatch immediately after rotation can be normal, because Cassandra polls for keystore changes rather than reloading instantly. Keep this at Warning and configure the alert rule to require the mismatch to persist beyond the expected reload window.

Check 7: the logs show what Cassandra is actually rejecting

Service checks show what the certificate on the wire looks like. Cassandra's logs show what the server is actually rejecting. That matters because Cassandra validates against its own truststore, can enforce client authentication and endpoint verification, and may hold a stale SSL context after a failed reload.

Which log lines you need depends on the SSL provider in use. Cassandra can use the JDK JSSE provider or the native OpenSSL provider. The official image ships with the native library, so OpenSSL is commonly the default. The startup line identifies the provider in use.

Create a log alert under Alerts & Notifications → Alert Definitions → Event Alert Definitions using this regex:

SSLHandshakeException|SSLPeerUnverifiedException|CertificateExpiredException|CertificateNotYetValidException|PKIX path (validation|building) failed|sun\.security\.validator\.ValidatorException|OpenSslException|CERTIFICATE_VERIFY_FAILED|(SSLv3|TLSV1)_ALERT_(UNKNOWN_CA|BAD_CERTIFICATE|CERTIFICATE_(EXPIRED|REVOKED|UNKNOWN|REQUIRED))

A practical threshold is:

• More than 5 occurrences in 5 minutes → Warning
• More than 20 occurrences in 5 minutes → Critical

Use a multi-minute window rather than a very short one, because handshake failures often arrive in bursts tied to gossip and reconnection timing.

JSSE provider:

javax.net.ssl.SSLHandshakeException: General SSLEngine problem
  Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

OpenSSL provider:

io.netty.handler.ssl.ReferenceCountedOpenSslEngine$OpenSslException: error:0A000086:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Check 8: OCSP revocation, only if a public CA is stapling

Revocation monitoring is rarely relevant for Cassandra clusters. Most internode and client-facing certificates are issued by private CAs that do not publish OCSP responses or CRLs. This check only makes sense when the certificate is issued by a public CA and Cassandra is configured to staple OCSP responses on the listener.

If either condition is false, leave this check disabled. Otherwise it will return Unknown on every run.

Create a shell service check named Cassandra TLS OCSP revocation - client SSL with interval 12h and timeout 30s:

EXIT_OK=0
EXIT_WARNING=1
EXIT_CRITICAL=2
EXIT_UNKNOWN=3
 
HOST="{{.comp_rpc_address}}"
PORT=9042
 
output=$(printf '' | openssl s_client -connect "${HOST}:${PORT}" -servername "${HOST}" -status 2>&1 || true)
 
if [ -z "${output}" ] || ! echo "${output}" | grep -q "BEGIN CERTIFICATE"; then
  echo "OK: could not retrieve cert from ${HOST}:${PORT} (liveness owned by TCP check)"
  exit $EXIT_OK
fi
 
if echo "${output}" | grep -q "OCSP Response Status: successful"; then
  cert_status=$(echo "${output}" | grep -m1 "Cert Status:" | awk '{print $3}')
  case "${cert_status}" in
    good)     echo "OK: OCSP stapled, cert status good" ;;
    revoked)  echo "CRITICAL: OCSP reports cert REVOKED"; exit $EXIT_CRITICAL ;;
    unknown)  echo "WARNING: OCSP status unknown"; exit $EXIT_WARNING ;;
    *)        echo "WARNING: unparseable OCSP status: ${cert_status}"; exit $EXIT_WARNING ;;
  esac
else
  echo "UNKNOWN: server did not staple an OCSP response (stapling not configured, or private CA)"
  exit $EXIT_UNKNOWN
fi
 
if ! echo "${output}" | openssl x509 -noout -checkend 432000 >/dev/null 2>&1; then
  echo "CRITICAL: cert on ${HOST}:${PORT} expires within 5 days"
  exit $EXIT_CRITICAL
fi
if ! echo "${output}" | openssl x509 -noout -checkend 864000 >/dev/null 2>&1; then
  echo "WARNING: cert on ${HOST}:${PORT} expires within 10 days"
  exit $EXIT_WARNING
fi
 
exit $EXIT_OK

Exporting the node certificate for the internode variants

The internode variants of Checks 1 to 3 need -cert <node.crt> -key <node.key> so the client side of the mTLS handshake can complete. Cassandra usually stores these in a keystore rather than as separate PEM files, so they may need exporting.

From a PKCS12 keystore:

openssl pkcs12 -in keystore.p12 -clcerts -nokeys -passin pass:<storepass> -out node.crt
openssl pkcs12 -in keystore.p12 -nocerts -nodes  -passin pass:<storepass> -out node.key

If the node uses JKS, convert it first:

keytool -importkeystore -srckeystore keystore.jks -srcstoretype JKS \
        -destkeystore keystore.p12 -deststoretype PKCS12

node.key is an unencrypted private key in this export flow. Store it with restrictive permissions, keep it off shared storage, and regenerate it whenever the node certificate rotates.

Route by severity, and keep the whole thing in version control

AxonOps routes alerts independently by source and severity, so the same service check can send Warning notifications to chat and Critical notifications to a paging system without duplicating the check definition. Per-severity integrations are assigned under Settings → Integrations.

Manual point-and-click configuration works for a single cluster. Beyond that, version-controlled configuration improves consistency, review and rollback. The axonops-ansible-collection provides Ansible roles and YAML schemas for service checks, alert rules, dashboards and backup schedules, including SSL certificate checks.

Certificate rotation should be verifiable, not hopeful

TLS is one of the few systems that can legitimately take the whole application down by design. Certificates expire by design, verification failures block handshakes by design, and partial rotations fail in ways that are easy to miss until they become incidents.

Wired up once, these checks turn certificate rotation from something you hope completed cleanly into something you can verify continuously.