Introduction

AxonOps is the only solution for the one-stop operation of Apache Cassandra®. Built by Cassandra experts and underpinned by a secure and efficient bi-directional protocol, it combines the essential functionality to enable Site Reliability Engineering to observe, maintain and protect their clusters. Key features include pre-configured dashboards, alerting with enterprise integration, performance-adaptive repairs and highly configurable backups with fine-grained retention management.

Cassandra is still primarily run on bare metal or virtual machines. As more companies migrate to Kubernetes (and cloud providers) they are also moving some of the database servers such as Cassandra into Kubernetes. Kubernetes presents several additional challenges and that’s why we are adding support for Kubernetes to our AxonOps products.

k8ssandra

k8ssandra is a Kubernetes Operator for running Apache Cassandra®. With K8ssandra you can quickly deploy and run Apache Cassandra® and by default, it would add grafana and prometheus for monitoring. I’ll show you below how to use AxonOps instead for richer and more complete Cassandra observability.

AxonOps

Before you start, get onto https://axonops.com and sign up for a Free Plan. Then, if you browse to the AxonOps console you should be able to grab a copy of the agent’s config file you’ll require for the next step.

Install k8ssandra operator

The k8ssandra-operator is very easy to install from helm charts. It requires cert-manager as well, if you don’t have it yet, remember to install it first. The example below uses a helmfile to install both k8ssandra-operator and cert-manager.

---
repositories:
  - name: jetstack
    url: https://charts.jetstack.io
  - name: k8ssandra
    url: https://helm.k8ssandra.io/
releases:
  - name: cert-manager
    namespace: cert-manager
    wait: true
    waitForJobs: true
    chart: jetstack/cert-manager
    labels:
      app: cert-manager
    values:
      - installCRDs: true
  - name: k8ssandra
    namespace: k8ssandra
    chart: "k8ssandra/k8ssandra-operator"
    wait: true
    values:
      - global:
          clusterScoped: true

Then save this as helmfile.yaml then just do:

helmfile -l app=cert-manager sync
helmfile sync

AxonOps Agent

Take the AxonOps Agent config you previously obtained from the AxonOps console and convert it into a configmap to be mounted into the pods:

---
apiVersion: v1
data:
  axon-agent.yml: |
    axon-server:
        hosts: "agents.axonops.cloud"
    axon-agent:
        key: AGENT_KEY
        org: ORGANIZATION
    disable_command_exec: false
kind: ConfigMap
metadata:
  name: axonops-agent-config
  namespace: axonops

Make sure you have an active plan enabled on the AxonOps Console. You can get a Free Plan that allows you to connect up to 5 nodes.

Apache Cassandra Cluster

The sample config below should work on minikube but you should obviously adjust memory, CPU and disk settings according to your set-up. Make sure (if using minikube) that there are enough memory and CPU resources available.

I’m using a MacBook to run minikube and these are the options I require. You may need slightly different arguments depending on whether you use Mac, Windows or Linux.

minikube start --memory 3072 --cpus=2 -n 1
# This example uses a single node. If you have the memory and CPU
# for more, adjust the `-n` settings to start multiple nodes

Storage Class

Please pay special attention to the storage class for the storage. If you’re running it locally (lab environment like minikube) you use either the standard k8s.io/minikube-hostpath or install instead the rancher.io/local-path provisioner.

On cloud environments running managed Kubernetes such as AWS or Google, you can usually leave the storageClassName empty to use the default configuration.

Note: I’ve encountered a bug on the minikubestorage provisioner when starting multiple nodes. You may want to use the rancher’s provisioner instead which does work as expected.

Config

The configuration below deploys an Apache Cassandra cluster using a k8ssandra resource definition. You can find some examples at the k8ssandra.io website. The important parts for adding AxonOps to the cluster are:

• initContainer: this is used to deploy the AxonOps Agent configuration to the Cassandra pods
• volumes and volumeMounts: the AxonOps Agent requires access to the Apache Cassandra logs and they also need to share a volume where the network socket and configuration are hosted.
• axon-agent: there is an additional container added to the pod running the AxonOps Agent.
• jvmOptions: we’re injecting the AxonOps Agent as a javaagent to Apache Cassandra

---
apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: axonops
  namespace: axonops
spec:
  clusterName: axonops-medium
  serverType: cassandra
  serverVersion: "4.0.3"
  managementApiAuth:
    insecure: {}
  size: 1
  podTemplateSpec:
    spec:
      initContainers:
        - name: axonops-init
          image: busybox:1.34.1
          command: ['sh', '-c', 'cp /etc/axonops/axon-agent.yml /var/lib/axonops/axon-agent.yml']
          env:
            - name: AXON_SHARED_VOLUME
              value: /var/lib/axonops
          volumeMounts:
            - name: axonops-agent-config
              mountPath: /etc/axonops
            - name: axonops-shared
              mountPath: /var/lib/axonops
          resources:
            requests:
              memory: 256Mi
              cpu: 10m
            limits:
              memory: 256Mi
              cpu: 50m
      containers:
        - name: "axon-agent"
          image: registry.axonops.com/axonops-public/axonops-docker/axon-agent:latest
          imagePullPolicy: Always
          volumeMounts:
            - name: axonops-agent-config
              mountPath: /etc/axonops
            - name: axonops-shared
              mountPath: /var/lib/axonops
              readOnly: false
            - name: axonops-logs
              mountPath: /var/log/axonops
            - name: server-logs
              mountPath: /opt/cassandra/logs
            - name: server-data
              mountPath: /var/lib/cassandra
          resources:
            requests:
              memory: 256Mi
              cpu: 10m
            limits:
              memory: 512Mi
              cpu: 20m
      volumes:
        - name: axonops-agent-config
          configMap:
            name: axonops-agent-config
        - name: axonops-shared
          emptyDir: {}
        - name: axonops-logs
          emptyDir: {}
  cassandra:
    kube-prometheus-stack:
      enabled: false
    prometheus:
      enabled: false
    grafana:
      enabled: false
    resources:
      requests:
        memory: 700Mi
        cpu: 500m
      limits:
        memory: 1Gi
        cpu: 1
  reaper:
    enabled: false
  stargate:
    enabled: false
  storageConfig:
    cassandraDataVolumeClaimSpec:
      #storageClassName: local-path
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
    additionalVolumes:
      - name: axonops-shared
        mountPath: /var/lib/axonops
        pvcSpec:
          #storageClassName: local-path
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 100Mi
      - name: axonops-logs
        mountPath: /var/log/axonops
        pvcSpec:
          #storageClassName: local-path
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 500Mi
      - mountPath: /opt/cassandra/logs
        name: server-logs
        pvcSpec:
          #storageClassName: local-path
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
  config:
    cassandra-yaml:
      authenticator: org.apache.cassandra.auth.PasswordAuthenticator
      authorizer: org.apache.cassandra.auth.CassandraAuthorizer
      role_manager: org.apache.cassandra.auth.CassandraRoleManager
      num_tokens: 8
      max_hints_file_size_in_mb: 128
      disk_optimization_strategy: ssd
    jvm-server-options:
      initial_heap_size: "700M"
      max_heap_size: "700M"
      additional-jvm-opts:
        - "-javaagent:/var/lib/axonops/axon-cassandra4.0-agent-1.0.1.jar=/var/lib/axonops/axon-agent.yml"

Running

Once you have saved this configuration to a file, you only need to apply it. Remember to create the namespace first, if it does not exist already.

kubectl create ns axonops
kubectl apply -f axonops-agent-config.yaml
kubectl apply -f axonops-cassandra.yaml

If everything worked, you should be able to list the running Cassandra pods:

kubectl get po -n axonops

As they get bootstrapped, they will start appearing in the AxonOps Dashboard.

Conclusion

As you see adding AxonOps to your Kubernetes Cassandra is very simple. This blog focuses on k8ssandra but if you don’t use the operator but a helm chart or a custom deployment, AxonOps can be added in a similar way. Get in touch if you need help.

‍