Introduction

We at Digitalis install and manage Kubernetes clusters for our customers. Customer requirements vary extraordinarily but there are a few core elements we include more often than not. These are some of the core technologies that many of them require, no matter whether they are on a cloud provider or on-premises.

Terraform

Infrastructure as Code is a must. We absolutely do not do anything that’s not automated. Nothing. Terraform is at the core and it’s the first thing we put together to be able to build the Kubernetes cluster via Rancher or directly into the cloud provider.

We also use Terraform to install some core applications (Helm charts) that I’m listing below.

Rancher

We start with SUSE Rancher. We like SUSE Rancher. It provides an easy management Web UI with a powerful RBAC system. We use Rancher for two main reasons:

• Build Kubernetes clusters using their API (via Terraform)
• Manage access to the clusters

But that’s not all. Rancher Web UI is powerful and easy to use. We find the less experience Kubernetes users like to be able to manage pods, view logs, etc.

NeuVector

Security is paramount and it’s at the core of our deployments. This is why we have adopted NeuVector and we use it to scan container images for security issues and to control what gets deployed with the built-in admission controller.

Vals-Operator

Continuing with the security chapter, we get into secrets. No matter how small the Kubernetes cluster is, the chances are there will be some secrets to manage (database credentials, TLS certs, etc).

We need to be able to source credentials from wherever they are saved (AWS SecretsManager, Vault, etc) and feed them to Kubernetes. This is the role of vals-operator.

Cert-Manager

Everything we deploy must use TLS encryption and here cert-manager is just brilliant. It can create the TLS certs automatically using the company’s internal CA, letsencrypt or, God forbid, self-signed.

External-DNS

The external-dns operator is another gem that’s a must on all our installations. As IP addressing in Kubernetes is dynamic for the most part, we need something that also updates the DNS servers automatically whenever an IP changes. This is why external-dns is at the core of the applications we use daily.

Traefik

We still use Nginx ingress in many places but we are slowly facing it out in favour of Traefik. Traefik is a more complete implementation with many more features and a dashboard useful for debugging issues. It also supports Jaeger natively rather than through a third-party plugin like nginx does.

HashiCorp Vault

If our customer does not have already a centralised secrets manager, we prefer HashiCorp Vault and this is what we deploy.

ArgoCD

Terraform installs some core elements. Amongst them, it installs and setups ArgoCD. We then use ArgoCD to manage and install the rest of the infrastructure and to provide developers continuous deployment into (CD) to Kubernetes.

Harbor

There are lots of container registries out there. You can use any provided by cloud providers such as Google, Amazon or Azure. None of them is perfect and none of them is cheap. If you want full control over your containers, security scanning, caching, container signing and other cool features, you need Harbor instead.

Tekton Pipelines

We are able to adapt and use whatever is the customer’s favourity CI/CD, from Jenkins to GitHub Actions, BitBucket pipelines, GitLab, etc. But when the CI (Continous Integration) needs running in Kubernetes we favour Tekton Pipelines. They are easy to set up and use and extremely versatily. Then we can hand over the CD (Continous Deployment) to ArgoCD.

Linkerd

Back with security. Meshed technologies can be very helpful to provide observability and security. This is why we opted for linkerd to ensure encryption is enabled between applications whilst at the same time providing us with observability to inspect the network and applications.

OpenTelemetry

As part of the monitoring stack and closely related to linkerd, we also like to install OpenTelemetry to provide us with additional information on the applications we’re running.

Calico Enterprise

Many of our customers are in the financial sector and you would expect, they face a lot of scrutiny both in terms of security and compliance. Compliance is often overlooked on Kubernetes and it can be difficult to achieve. This is why we recommend and install Calico Enterprise.

Grafana and Prometheus

Once everything is running, it ought to continue running. And you won’t know unless you monitor it. Grafana and Prometheus are the best tools in my opinion for monitoring Kubernetes and many of its components.

The exception is when we deploy Apache Cassandra into Kubernetes. Prometheus falls short of our high standards and we use AxonOps instead.

Logging

Kubernetes applications logs need to be available to the different areas of the business that require them for incident management, security, etc. For this we like Banzai Cloud logging operator. It’s very good and flexible. We use it to set up logging filters and destinations for backend where we will keep the logs securely.

If the customer does not already have a logging collection system, we tend to prefer ElasticSearch or OpenSearch.

Other

I could go on for a long time as there are a lot of cool technologies but the above represents our core. Of course, this is not set in stone. We do what our customers request of us and when something comes up that’s better than what we have, we have no issues on replacing it.

The Kubernetes community is very fast pace and there are lots of very interesting technologies to use. We will eventually see some consolidation and some will disappear.

If you want to know more, leave a comment or get in touch.

‍