We have written before about managing SSL in Kubernetes with cert-manager and, more recently, about what the public SSL changes mean for your data platform. Both cover Kubernetes and the data platform. Neither covers the VM estate that most organisations still run, and that is the subject of this post.
Kubernetes has cert-manager, which is widely used and handles certificate renewal well. VMs have no single standard approach. I have seen many organisations relying on a cron job and a script someone wrote in 2019 that nobody maintains or checks any more. That arrangement was tolerable when certificates lasted a year, but it will not cope with the renewal frequency that is coming.
How short are public certificates getting?
The CA/Browser Forum passed Ballot SC-081v3 in 2025 and the first step is already in force. Every publicly trusted CA has to follow it, so changing provider does not get you out of it.
| From | Maximum certificate lifetime | Domain validation reuse |
|---|---|---|
| 15 March 2026 (now) | 200 days | 200 days |
| 15 March 2027 | 100 days | 100 days |
| 15 March 2029 | 47 days | 10 days |
Let's Encrypt is moving faster than the ballot requires. Its published plan cuts default certificates to 64 days on 10 February 2027 and to 45 days on 16 February 2028, with authorisation reuse coming down to 10 days and later to 7 hours.
Two smaller Let's Encrypt changes add to the problem, and I suspect many teams have not noticed them. It turned off expiry notification emails in June 2025, so nobody will warn you before a certificate lapses. It also stopped issuing certificates with the TLS client authentication EKU, removing it from the default profile in February 2026 and retiring the separate tlsclient profile in July 2026. Anything that relies on a public certificate for mTLS is running on certificates that cannot be renewed in the same form.
What does 47 days mean in numbers?
The arithmetic is what makes this a real operational problem. A healthy ACME client renews at about two thirds of the certificate lifetime, so a 47-day certificate is replaced roughly every 30 days. That is 12 renewals a year per certificate, compared with one or two today for most enterprise certificates.
Take a modest estate of 300 public certificates across web servers, load balancers, Kafka listeners, Elasticsearch nodes and the odd Windows server. That is 3,600 renewal events a year, or about 14 every working day, and each one can write a bad file or fail silently when nobody is watching.
With 10-day validation reuse, almost every one of those renewals also needs a fresh DNS or HTTP challenge. Email validation and manual requests to the DNS team for a TXT record will no longer be workable.
If even 1% of those renewals go wrong, that is 36 incidents a year, and for an estate that was never built for this, 1% is an optimistic estimate.
Why isn't installing certbot enough?
Most teams start with something like this and consider the job done:
certbot certonly \
--dns-route53 \
-d kafka-01.example.com \
--deploy-hook /etc/letsencrypt/renewal-hooks/deploy/kafka.shCertbot 4.1 and later, acme.sh 3.1.4 and later, and lego all support ACME Renewal Information (ARI), which lets the CA tell your client when to renew. If your client predates ARI support, upgrade it. Many teams treat ARI as the end of the job, but it only decides when a renewal happens, not whether the service picks up the new certificate safely.
This is the kind of deploy hook we regularly find on Kafka and Cassandra nodes:
#!/usr/bin/env bash
set -euo pipefail
openssl pkcs12 -export \
-in "${RENEWED_LINEAGE}/fullchain.pem" \
-inkey "${RENEWED_LINEAGE}/privkey.pem" \
-out /etc/kafka/ssl/server.p12 \
-name kafka -passout file:/etc/kafka/ssl/.p12pass
systemctl restart kafkaIt works in a lab, but in production it has at least three problems. See if you can spot them before reading on.
The first is timing. Certificates issued together renew together. ARI picks a time inside the CA's suggested window and nothing stops five brokers landing in the same hour, so on a quiet Tuesday night they all renew and all restart. The result is under-replicated partitions, producers retrying and consumer groups rebalancing, and an incident that nobody caused by touching anything.
The second is that the hook does not check whether the new keystore is valid. A truncated file, a wrong password or a chain missing its intermediate means the service restarts and does not come back. The old certificate has already been replaced on disk, the new one is broken, and the on-call engineer is running keytool by hand at 3am.
The third is the restart itself. Kafka, Cassandra 4.0 and later, and Elasticsearch can all reload TLS material without a restart if they are configured to. Most estates we review are not configured that way, so every renewal turns into a rolling restart. With one renewal a year that is an inconvenience, but with twelve it becomes a standing change ticket for every cluster you run.
Where does DNS-01 go wrong on VMs?
Most internal VMs cannot answer an HTTP-01 challenge because the internet cannot reach them, which leaves DNS-01 as the only realistic option. The obvious implementation gives each VM credentials for your DNS provider.
That means an API key able to rewrite your public DNS zone sits on hundreds of hosts, often in a plain text file under /etc/letsencrypt. If an attacker compromises any one of those hosts, they can point your login page somewhere else and obtain a valid certificate for it.
There are well-known patterns that reduce this risk, such as delegating _acme-challenge records to a separate zone with very narrow permissions, or centralising issuance so the VMs never hold DNS credentials at all. Both work, but someone has to design the delegation and lock down the IAM policy.
Should Kafka and Cassandra use public certificates at all?
Going back to the deploy hook above, I would question why that Kafka broker has a public certificate in the first place.
A public CA exists so that browsers and other people's software can trust you. Brokers and nodes talk to each other and to clients you control, so you own the truststore at both ends, and a public CA adds nothing for that traffic.
It also costs you more than renewals. Every public certificate is published in Certificate Transparency logs, so each new broker or node publicly announces its hostname. And because public CAs are dropping the client authentication EKU, a public certificate cannot be used for Cassandra internode encryption with require_client_auth or for Kafka mTLS. From March 2027 no public CA will issue one that can.
For internal services we use a private CA, which these days is usually the PKI secrets engine in OpenBao, the open source fork of HashiCorp Vault. Private certificates sit outside the CA/Browser Forum rules, so you choose the lifetime and there is no domain validation. Hosts authenticate to OpenBao with their own identity rather than with DNS credentials, which removes the problem described in the previous section. OpenBao Agent can write the certificate to disk and run a command when it changes, so the same automation works on every VM.
In my view this is a good trade. Instead of chasing hundreds of renewals under someone else's rules, you run one certificate authority with rules you set. It still needs looking after: keep the root key offline, rotate intermediates on a schedule and get your CA bundle into every client truststore. OpenBao also needs to run highly available with auto-unseal, so that renewals continue when a node goes down overnight. Once that is in place, certificate renewal becomes something the platform handles itself rather than work scheduled into change windows.
How do you know renewal is actually working?
With expiry emails gone, your own certificate expiry monitoring is the only thing between a failed renewal and an outage. That monitoring has to check the certificate actually served on the wire rather than the file on disk, because a renewed certificate that was never reloaded looks correct from the filesystem.
You need alerts well before the renewal window closes and a view of which hosts have not renewed when their siblings have. Prometheus with the Blackbox Exporter covers part of this. Joining it to an inventory of who owns each certificate is the part most teams skip.
What should you do this quarter?
- Build an inventory. Scan your public ranges, search Certificate Transparency logs for your domains, and list every certificate, where it is served, which process renews it and how the service picks it up. Expect to find certificates nobody can explain.
- Mark the internal services that should move to a private CA. Then sort the rest into three groups: automated end to end, automated issuance with manual reload or deployment, and fully manual. The middle group carries the most risk because it appears to be finished.
- Fix your hardest service first. Pick your most difficult clustered Java service and get one renewal working with no restart and no human involvement. Once that works, the remaining services are a matter of rolling out the same pattern.
March 2027 halves the maximum lifetime and 2029 halves it again. Starting now gives you more renewal cycles to find and fix problems before they cause an outage.
