@ET-DC@eyJkeW5hbWljIjp0cnVlLCJjb250ZW50IjoicG9zdF90aXRsZSIsInNldHRpbmdzIjp7ImJlZm9yZSI6IiIsImFmdGVyIjoiIn19@[addtoany buttons="twitter", "linkedin"]

Introduction

In any environment that needs to expose endpoints, it's important to ensure that they are secured and monitored. Although there is plenty of documentation and examples on performance monitoring, there is minimal documentation on how to monitor the security of the endpoints and specifically the TLS/SSL non-HTTPS certificates. This blog outlines how you can use the Prometheus Blackbox Exporter to do this - I hope you find it useful!

As an example, when we deploy Apache Cassandra it is typical to secure the database endpoints with certificates to ensure the data is encrypted in-flight and, if required, enforce client certificate validation. Knowing when these certificates expire is important and being able to monitor and alert based on this is critical. However, this example also applies to any application that exposes a TLS/SSL endpoint like LDAP, Kafka, ELK, etc.

For the sake of simplicity, we will use a single Blackbox probe located on the same VM as our single Prometheus instance to monitor a certificate on an Apache Cassandra database.

Note:

In a high availability or production environment, it is always suggested to use multiple probes and multiple Prometheus instances. For Apache Cassandra in particular, we also suggest to use a complete monitoring system / operational toolset such as https://www.axonops.com/

First of all, let's manually check the connection from the Prometheus instance to our node, cas01.dev.db.myexample.io port 9142 for Apache Cassandra. We can do that using the OpenSSL command:

cm9vdEBwcm9tLmJsb2cubXlleGFtcGxlLmlvOn4jIGVjaG8gLW4gfCBvcGVuc3NsIHNfY2xpZW50IC1jb25uZWN0IGNhczAxLmRldi5kYi5teWV4YW1wbGUuaW86OTE0MiAyPiAvZGV2L251bGwgfCBvcGVuc3NsIHg1MDkgLW5vb3V0IC10ZXh0CkNlcnRpZmljYXRlOgogICAgRGF0YToKICAgICAgICBWZXJzaW9uOiAxICgweDApCiAgICAgICAgU2VyaWFsIE51bWJlcjogMTMzMTU2ODQ2Mzg4MDY1NzI2NzQgKDB4YjhjYWQwYTEyNTMwZmU4MikKICAgIFNpZ25hdHVyZSBBbGdvcml0aG06IHNoYTI1NldpdGhSU0FFbmNyeXB0aW9uCiAgICAgICAgSXNzdWVyOiBDPVVTLCBPPW15ZXhhbXBsZS5pbywgT1U9RGV2Q2x1c3RlciwgQ049cm9vdENBCiAgICAgICAgVmFsaWRpdHkKICAgICAgICAgICAgTm90IEJlZm9yZTogSmFuIDIyIDEzOjAxOjE1IDIwMTkgR01UCiAgICAgICAgICAgIE5vdCBBZnRlciA6IEphbiAyMSAxMzowMToxNSAyMDIxIEdNVAogICAgICAgIFN1YmplY3Q6IEM9VVMsIE89bXlleGFtcGxlLmlvLCBPVT1EZXZDbHVzdGVyLCBDTj1jYXMwMS5kZXYuZGIubXlleGFtcGxlLmlvClsuLi5d

echo -n makes OpenSSL return the prompt immediately after the command, openssl s_client connects to the endpoint to read the certificate and openssl x509 displays the certificates.

Now let's configure the TCP module on the Blackbox Exporter probe like this:

cm9vdEBwcm9tLmJsb2cubXlleGFtcGxlLmlvOn4jIGN1cmwgJ2h0dHA6Ly9wcm9tLmJsb2cubXlleGFtcGxlLmlvOjkxMTUvcHJvYmU/dGFyZ2V0PWNhczAxLmRldi5kYi5teWV4YW1wbGUuaW8lM0E5MTQyJm1vZHVsZT10Y3BfY2VydCZkZWJ1Zz10cnVlJwpMb2dzIGZvciB0aGUgcHJvYmU6CnRzPTIwMjAtMDgtMDFUMDk6Mzk6NDEuNDk4NDY0MzU5WiBjYWxsZXI9bWFpbi5nbzozMDQgbW9kdWxlPXRjcF9jZXJ0IGRhdGEtZXQtdGFyZ2V0LWxpbms9Y2FzMDEuZGV2LmRiLm15ZXhhbXBsZS5pbzo5MTQyIGxldmVsPWluZm8gbXNnPSJCZWdpbm5pbmcgcHJvYmUiIHByb2JlPXRjcCB0aW1lb3V0X3NlY29uZHM9NQp0cz0yMDIwLTA4LTAxVDA5OjM5OjQxLjQ5ODU2ODU5OFogY2FsbGVyPXRjcC5nbzo0MSBtb2R1bGU9dGNwX2NlcnQgZGF0YS1ldC10YXJnZXQtbGluaz1jYXMwMS5kZXYuZGIubXlleGFtcGxlLmlvOjkxNDIgbGV2ZWw9aW5mbyBtc2c9IlJlc29sdmluZyB0YXJnZXQgYWRkcmVzcyIgaXBfcHJvdG9jb2w9aXA2CnRzPTIwMjAtMDgtMDFUMDk6Mzk6NDEuNTAzMzg2MjkxWiBjYWxsZXI9dGNwLmdvOjQxIG1vZHVsZT10Y3BfY2VydCBkYXRhLWV0LXRhcmdldC1saW5rPWNhczAxLmRldi5kYi5teWV4YW1wbGUuaW86OTE0MiBsZXZlbD1pbmZvIG1zZz0iUmVzb2x2ZWQgdGFyZ2V0IGFkZHJlc3MiIGlwPTEwLjAuNC4yMAp0cz0yMDIwLTA4LTAxVDA5OjM5OjQxLjUwMzQxMzUwM1ogY2FsbGVyPXRjcC5nbzoxMTEgbW9kdWxlPXRjcF9jZXJ0IGRhdGEtZXQtdGFyZ2V0LWxpbms9Y2FzMDEuZGV2LmRiLm15ZXhhbXBsZS5pbzo5MTQyIGxldmVsPWluZm8gbXNnPSJEaWFsaW5nIFRDUCB3aXRoIFRMUyIKdHM9MjAyMC0wOC0wMVQwOTozOTo0MS41MjQ2MDIzNDFaIGNhbGxlcj1tYWluLmdvOjExOSBtb2R1bGU9dGNwX2NlcnQgZGF0YS1ldC10YXJnZXQtbGluaz1jYXMwMS5kZXYuZGIubXlleGFtcGxlLmlvOjkxNDIgbGV2ZWw9aW5mbyBtc2c9IlN1Y2Nlc3NmdWxseSBkaWFsZWQiCnRzPTIwMjAtMDgtMDFUMDk6Mzk6NDEuNTI0NjY5OTMxWiBjYWxsZXI9bWFpbi5nbzozMDQgbW9kdWxlPXRjcF9jZXJ0IGRhdGEtZXQtdGFyZ2V0LWxpbms9Y2FzMDEuZGV2LmRiLm15ZXhhbXBsZS5pbzo5MTQyIGxldmVsPWluZm8gbXNnPSJQcm9iZSBzdWNjZWVkZWQiIGR1cmF0aW9uX3NlY29uZHM9MC4wMjYxNDc1NDcKCgpNZXRyaWNzIHRoYXQgd291bGQgaGF2ZSBiZWVuIHJldHVybmVkOgojIEhFTFAgcHJvYmVfZG5zX2xvb2t1cF90aW1lX3NlY29uZHMgUmV0dXJucyB0aGUgdGltZSB0YWtlbiBmb3IgcHJvYmUgZG5zIGxvb2t1cCBpbiBzZWNvbmRzCiMgVFlQRSBwcm9iZV9kbnNfbG9va3VwX3RpbWVfc2Vjb25kcyBnYXVnZQpwcm9iZV9kbnNfbG9va3VwX3RpbWVfc2Vjb25kcyAwLjAwNDgzMTI1OAojIEhFTFAgcHJvYmVfZHVyYXRpb25fc2Vjb25kcyBSZXR1cm5zIGhvdyBsb25nIHRoZSBwcm9iZSB0b29rIHRvIGNvbXBsZXRlIGluIHNlY29uZHMKIyBUWVBFIHByb2JlX2R1cmF0aW9uX3NlY29uZHMgZ2F1Z2UKcHJvYmVfZHVyYXRpb25fc2Vjb25kcyAwLjAyNjE0NzU0NwojIEhFTFAgcHJvYmVfZmFpbGVkX2R1ZV90b19yZWdleCBJbmRpY2F0ZXMgaWYgcHJvYmUgZmFpbGVkIGR1ZSB0byByZWdleAojIFRZUEUgcHJvYmVfZmFpbGVkX2R1ZV90b19yZWdleCBnYXVnZQpwcm9iZV9mYWlsZWRfZHVlX3RvX3JlZ2V4IDAKIyBIRUxQIHByb2JlX2lwX3Byb3RvY29sIFNwZWNpZmllcyB3aGV0aGVyIHByb2JlIGlwIHByb3RvY29sIGlzIElQNCBvciBJUDYKIyBUWVBFIHByb2JlX2lwX3Byb3RvY29sIGdhdWdlCnByb2JlX2lwX3Byb3RvY29sIDQKIyBIRUxQIHByb2JlX3NzbF9lYXJsaWVzdF9jZXJ0X2V4cGlyeSBSZXR1cm5zIGVhcmxpZXN0IFNTTCBjZXJ0IGV4cGlyeSBkYXRlCiMgVFlQRSBwcm9iZV9zc2xfZWFybGllc3RfY2VydF9leHBpcnkgZ2F1Z2UKcHJvYmVfc3NsX2VhcmxpZXN0X2NlcnRfZXhwaXJ5IDEuNjExMjM0MDc0ZSswOQojIEhFTFAgcHJvYmVfc3VjY2VzcyBEaXNwbGF5cyB3aGV0aGVyIG9yIG5vdCB0aGUgcHJvYmUgd2FzIGEgc3VjY2VzcwojIFRZUEUgcHJvYmVfc3VjY2VzcyBnYXVnZQpwcm9iZV9zdWNjZXNzIDEKIyBIRUxQIHByb2JlX3Rsc192ZXJzaW9uX2luZm8gUmV0dXJucyB0aGUgVExTIHZlcnNpb24gdXNlZCwgb3IgTmFOIHdoZW4gdW5rbm93bgojIFRZUEUgcHJvYmVfdGxzX3ZlcnNpb25faW5mbyBnYXVnZQpwcm9iZV90bHNfdmVyc2lvbl9pbmZve3ZlcnNpb249IlRMUyAxLjIifSAxCgoKCk1vZHVsZSBjb25maWd1cmF0aW9uOgpwcm9iZXI6IHRjcAp0aW1lb3V0OiA1cwpodHRwOgogICAgaXBfcHJvdG9jb2xfZmFsbGJhY2s6IHRydWUKdGNwOgogICAgaXBfcHJvdG9jb2xfZmFsbGJhY2s6IHRydWUKICAgIHRsczogdHJ1ZQogICAgdGxzX2NvbmZpZzoKICAgICAgICBpbnNlY3VyZV9za2lwX3ZlcmlmeTogdHJ1ZQppY21wOgogICAgaXBfcHJvdG9jb2xfZmFsbGJhY2s6IHRydWUKZG5zOgogICAgaXBfcHJvdG9jb2xfZmFsbGJhY2s6IHRydWUKcm9vdEBwcm9tLmJsb2cubXlleGFtcGxlLmlvOn4jPerfect, we know that the probe works, can connect to our Apache Cassandra database and we can also see the various metrics exported.The metrics that are very useful are::

• probe_success allows us to make sure that the endpoint is reachable;
• probe_ssl_earliest_cert_expiry is the expiry time of the certificate;
• probe_duration_seconds the time that the probe took, useful to check the responsiveness of the node

Finally, let's set Prometheus to scrape our node:cm9vdEBwcm9tLmJsb2cubXlleGFtcGxlLmlvOn4jIGNhdCAvdmFyL2xpYi9wcm9tZXRoZXVzL2NvbmZpZy9wcm9tZXRoZXVzLnltbAoKWy4uLl0KLSBqb2JfbmFtZTogYmxhY2tib3hfY2FzCiAgcGFyYW1zOgogICAgbW9kdWxlOgogICAgLSB0Y3BfY2VydAogIG1ldHJpY3NfcGF0aDogL3Byb2JlCiAgc3RhdGljX2NvbmZpZ3M6CiAgLSB0YXJnZXRzOgogICAgLSBjYXMwMS5kZXYuZGIubXlleGFtcGxlLmlvOjkxNDIKICByZWxhYmVsX2NvbmZpZ3M6CiAgICAtIHNvdXJjZV9sYWJlbHM6IFtfX2FkZHJlc3NfX10KICAgICAgdGFyZ2V0X2xhYmVsOiBfX3BhcmFtX3RhcmdldAogICAgLSBzb3VyY2VfbGFiZWxzOiBbX19wYXJhbV90YXJnZXRdCiAgICAgIHRhcmdldF9sYWJlbDogaW5zdGFuY2UKICAgIC0gdGFyZ2V0X2xhYmVsOiBfX2FkZHJlc3NfXwogICAgICByZXBsYWNlbWVudDogMTI3LjAuMC4xOjkxMTU=

Summary

To visualize the metrics we can use the dashboards from the Grafana website, like: https://grafana.com/grafana/dashboards/7587 or https://grafana.com/grafana/dashboards/11529

Monitoring TLS/SSL certificates, also if alone is not sufficient for high availability or production environment, should be part of any monitoring system. No one likes to be woken up in the middle of the night because the entire production environment is down due to an expired certificate. The suggestion is always to make sure that your monitoring system displays and alerts for certificate expirations.

@ET-DC@eyJkeW5hbWljIjp0cnVlLCJjb250ZW50IjoicG9zdF9hdXRob3JfYmlvIiwic2V0dGluZ3MiOnsiYmVmb3JlIjoiTWFyaW8gTnVnbmVzIGlzIGEgbWlkZGxlIGxldmVsIERldk9wcyBFbmdpbmVlci4gTWFyaW8gaGFzIGV4dGVuc2l2ZSBleHBlcmllbmNlIHdpdGggYm90aCBsYXJnZSBhbmQgc21hbGwgY29tcGFuaWVzLiBIZSB3b3JrZWQgb24gY29tcGxleCBhbmQgcmVsYXRpdmUgc2ltcGxlIGVudmlyb25tZW50cyBhbmQgaGUgaXMgdmVyeSBrZWVuIHRvIGNvbnN0YW50bHkgaW1wcm92ZSBoaW1zZWxmIGFuZCB0aGUgc3lzdGVtcyBoZSBpcyB3b3JraW5nIG9uLiBIaXMgY29tcGV0ZW5jZSBpcyBtYWlubHkgb24gUHJvbWV0aGV1cywgQ2Fzc2FuZHJhIGFuZCBBbnNpYmxlLiAgTWFyaW8gaGFzIGFsc28gZXhwZXJpZW5jZSB3aXRoIEthZmthIGFuZCBFbGFzdGljLiIsImFmdGVyIjoiIn19@

Related Articles