Cassandra with AxonOps on Kubernetes
Introduction
The following shows how to install AxonOps for monitoring Cassandra. This process specifically requires the official cassandra helm repository.
Using minikube
The deployment should work fine on latest versions of minikube as long as you provide enough memory for it.
bWluaWt1YmUgc3RhcnQgLS1tZW1vcnkgODE5MiAtLWNwdXM9NAptaW5pa3ViZSBhZGRvbnMgZW5hYmxlIHN0b3JhZ2UtcHJvdmlzaW9uZXI=
⚠️ Make sure you use a recent version of minikube. Also check available drivers and select the most appropriate for your platform
Helmfile
Overview
As this deployment contains multiple applications we recommend you use an automation system such as Ansible or Helmfile to put together the config. The example below uses helmfile.
Install requirements
You would need to install the following components:
Alternatively you can consider using a dockerized version of them both such as https://hub.docker.com/r/chatwork/helmfile
Config files
The values below are set for running on a laptop with minikube, adjust accordingly for larger deployments.
The values below are set for running on a laptop with minikube, adjust accordingly for larger deployments.
helmfile.yaml
LS0tCnJlcG9zaXRvcmllczoKICAtIG5hbWU6IHN0YWJsZQogICAgdXJsOiBodHRwczovL2t1YmVybmV0ZXMtY2hhcnRzLnN0b3JhZ2UuZ29vZ2xlYXBpcy5jb20KICAtIG5hbWU6IGluY3ViYXRvcgogICAgdXJsOiBodHRwczovL2t1YmVybmV0ZXMtY2hhcnRzLWluY3ViYXRvci5zdG9yYWdlLmdvb2dsZWFwaXMuY29tCiAgLSBuYW1lOiBheG9ub3BzLWhlbG0KICAgIHVybDogaHR0cHM6Ly9yZXBvLmF4b25vcHMuY29tL3B1YmxpYy9oZWxtL2hlbG0vY2hhcnRzLwogIC0gbmFtZTogYml0bmFtaQogICAgdXJsOiBodHRwczovL2NoYXJ0cy5iaXRuYW1pLmNvbS9iaXRuYW1pCnJlbGVhc2VzOgogIC0gbmFtZTogYXhvbi1lbGFzdGljCiAgICBuYW1lc3BhY2U6IHt7IGVudiAiTkFNRVNQQUNFIiB8IGRlZmF1bHQgIm1vbml0b3JpbmciIH19CiAgICBjaGFydDogImJpdG5hbWkvZWxhc3RpY3NlYXJjaCIKICAgIHdhaXQ6IHRydWUKICAgIGxhYmVsczoKICAgICAgZW52OiBtaW5pa3ViZQogICAgdmFsdWVzOgogICAgICAtIGZ1bGxuYW1lT3ZlcnJpZGU6IGF4b24tZWxhc3RpYwogICAgICAtIGltYWdlVGFnOiAiNy44LjAiCiAgICAgIC0gZGF0YToKICAgICAgICAgIHJlcGxpY2FzOiAxCiAgICAgICAgICBwZXJzaXN0ZW5jZToKICAgICAgICAgICAgc2l6ZTogMUdpCiAgICAgICAgICAgIGVuYWJsZWQ6IHRydWUKICAgICAgICAgICAgYWNjZXNzTW9kZXM6IFsgIlJlYWRXcml0ZU9uY2UiIF0KICAgICAgLSBjdXJhdG9yOgogICAgICAgICAgZW5hYmxlZDogdHJ1ZQogICAgICAtIGNvb3JkaW5hdGluZzoKICAgICAgICAgIHJlcGxpY2FzOiAxCiAgICAgIC0gbWFzdGVyOgogICAgICAgICAgcmVwbGljYXM6IDEKICAgICAgICAgIHBlcnNpc3RlbmNlOgogICAgICAgICAgICBzaXplOiAxR2kKICAgICAgICAgICAgZW5hYmxlZDogdHJ1ZQogICAgICAgICAgICBhY2Nlc3NNb2RlczogWyAiUmVhZFdyaXRlT25jZSIgXQoKICAtIG5hbWU6IGF4b25vcHMKICAgIG5hbWVzcGFjZToge3sgZW52ICJOQU1FU1BBQ0UiIHwgZGVmYXVsdCAibW9uaXRvcmluZyIgfX0KICAgIGNoYXJ0OiAiYXhvbm9wcy1oZWxtL2F4b25vcHMiCiAgICB3YWl0OiB0cnVlCiAgICBsYWJlbHM6CiAgICAgIGVudjogbWluaWt1YmUKICAgIHZhbHVlczoKICAgICAgLSB2YWx1ZXMueWFtbAoKICAtIG5hbWU6IGNhc3NhbmRyYQogICAgbmFtZXNwYWNlOiBjYXNzYW5kcmEKICAgIGNoYXJ0OiAiaW5jdWJhdG9yL2Nhc3NhbmRyYSIKICAgIHdhaXQ6IHRydWUKICAgIGxhYmVsczoKICAgICAgZW52OiBkZXYKICAgIHZhbHVlczoKICAgICAgLSB2YWx1ZXMueWFtbA==
values.yaml
LS0tCnBlcnNpc3RlbmNlOgogIGVuYWJsZWQ6IHRydWUKICBzaXplOiAxR2kKICBhY2Nlc3NNb2RlOiBSZWFkV3JpdGVNYW55Cgpwb2RTZXR0aW5nczoKICB0ZXJtaW5hdGlvbkdyYWNlUGVyaW9kU2Vjb25kczogMzAwCgppbWFnZToKICB0YWc6IDMuMTEuNgogIHB1bGxQb2xpY3k6IElmTm90UHJlc2VudAoKY29uZmlnOgogIGNsdXN0ZXJfbmFtZTogbWluaWt1YmUKICBjbHVzdGVyX3NpemU6IDMKICBzZWVkX3NpemU6IDIKICBudW1fdG9rZW5zOiAyNTYKICBtYXhfaGVhcF9zaXplOiA1MTJNCiAgaGVhcF9uZXdfc2l6ZTogNTEyTQoKZW52OgogIEpWTV9PUFRTOiAiLWphdmFhZ2VudDovdmFyL2xpYi9heG9ub3BzL2F4b24tY2Fzc2FuZHJhMy4xMS1hZ2VudC5qYXI9L2V0Yy9heG9ub3BzL2F4b24tYWdlbnQueW1sIgoKZXh0cmFWb2x1bWVzOgogIC0gbmFtZTogYXhvbm9wcy1hZ2VudC1jb25maWcKICAgIGNvbmZpZ01hcDoKICAgICAgbmFtZTogYXhvbm9wcy1hZ2VudAogIC0gbmFtZTogYXhvbm9wcy1zaGFyZWQKICAgIGVtcHR5RGlyOiB7fQogIC0gbmFtZTogYXhvbm9wcy1sb2dzCiAgICBlbXB0eURpcjoge30KICAtIG5hbWU6IGNhc3NhbmRyYS1sb2dzCiAgICBlbXB0eURpcjoge30KCmV4dHJhVm9sdW1lTW91bnRzOgogIC0gbmFtZTogYXhvbm9wcy1zaGFyZWQKICAgIG1vdW50UGF0aDogL3Zhci9saWIvYXhvbm9wcwogICAgcmVhZE9ubHk6IGZhbHNlCiAgLSBuYW1lOiBheG9ub3BzLWFnZW50LWNvbmZpZwogICAgbW91bnRQYXRoOiAvZXRjL2F4b25vcHMKICAgIHJlYWRPbmx5OiB0cnVlCiAgLSBuYW1lOiBheG9ub3BzLWxvZ3MKICAgIG1vdW50UGF0aDogL3Zhci9sb2cvYXhvbm9wcwogIC0gbmFtZTogY2Fzc2FuZHJhLWxvZ3MKICAgIG1vdW50UGF0aDogL3Zhci9sb2cvY2Fzc2FuZHJhCgpleHRyYUNvbnRhaW5lcnM6CiAgLSBuYW1lOiBheG9ub3BzLWFnZW50CiAgICBpbWFnZTogZGlnaXRhbGlzZG9ja2VyL2F4b24tYWdlbnQ6bGF0ZXN0CiAgICBlbnY6CiAgICAgIC0gbmFtZTogQVhPTl9BR0VOVF9WRVJCT1NJVFkKICAgICAgICB2YWx1ZTogIjEiCiAgICB2b2x1bWVNb3VudHM6CiAgICAgIC0gbmFtZTogYXhvbm9wcy1hZ2VudC1jb25maWcKICAgICAgICBtb3VudFBhdGg6IC9ldGMvYXhvbm9wcwogICAgICAgIHJlYWRPbmx5OiB0cnVlCiAgICAgIC0gbmFtZTogYXhvbm9wcy1zaGFyZWQKICAgICAgICBtb3VudFBhdGg6IC92YXIvbGliL2F4b25vcHMKICAgICAgICByZWFkT25seTogZmFsc2UKICAgICAgLSBuYW1lOiBheG9ub3BzLWxvZ3MKICAgICAgICBtb3VudFBhdGg6IC92YXIvbG9nL2F4b25vcHMKICAgICAgLSBuYW1lOiBjYXNzYW5kcmEtbG9ncwogICAgICAgIG1vdW50UGF0aDogL3Zhci9sb2cvY2Fzc2FuZHJhCgpheG9uLXNlcnZlcjoKICBlbGFzdGljX2hvc3Q6IGh0dHA6Ly9heG9uLWVsYXN0aWMtZWxhc3RpY3NlYXJjaC1tYXN0ZXIKICBpbWFnZToKICAgIHJlcG9zaXRvcnk6IGRpZ2l0YWxpc2RvY2tlci9heG9uLXNlcnZlcgogICAgdGFnOiBsYXRlc3QKICAgIHB1bGxQb2xpY3k6IElmTm90UHJlc2VudAoKCmF4b24tZGFzaDoKICBheG9uU2VydmVyVXJsOiBodHRwOi8vYXhvbm9wcy1heG9uLXNlcnZlcjo4MDgwCiAgc2VydmljZToKICAgICMgdXNlIE5vZGVQb3J0IGZvciBtaW5pa3ViZSwgY2hhbmdlIHRvIENsdXN0ZXJJUCBvciBMb2FkQmFsYW5jZXIgb24gZnVsbHkgZmVhdHVyZWQKICAgICMgazhzIGRlcGxveW1lbnRzIHN1Y2ggYXMgQVdTIG9yIEdvb2dsZQogICAgdHlwZTogTm9kZVBvcnQKICBpbWFnZToKICAgIHJlcG9zaXRvcnk6IGRpZ2l0YWxpc2RvY2tlci9heG9uLWRhc2gKICAgIHRhZzogbGF0ZXN0CiAgICBwdWxsUG9saWN5OiBJZk5vdFByZXNlbnQ=
axon-agent.yml
YXhvbi1zZXJ2ZXI6CiAgICBob3N0czogImF4b25vcHMtYXhvbi1zZXJ2ZXIubW9uaXRvcmluZyIgIyBTcGVjaWZ5IGF4b24tc2VydmVyIElQIGF4b24tc2VydmVyLm15Y29tcGFueS4KICAgIHBvcnQ6IDE4ODgKCmF4b24tYWdlbnQ6CiAgICBvcmc6ICJtaW5pa3ViZSIgIyBTcGVjaWZ5IHlvdXIgb3JnYW5pc2F0aW9uIG5hbWUKICAgIGh1bWFuX3JlYWRhYmxlX2lkZW50aWZpZXI6ICJheG9uX2FnZW50X2lwIiAjIG9uZSBvZiB0aGUgZm9sbG93aW5nOgoKTlRQOgogICAgaG9zdDogInBvb2wubnRwLm9yZyIgIyBTcGVjaWZ5IGEgTlRQIHRvIGRldGVybWluZSBhIE5UUCBvZmZzZXQKCmNhc3NhbmRyYToKICB0aWVyMDogIyBtZXRyaWNzIGNvbGxlY3RlZCBldmVyeSA1IHNlY29uZHMKICAgICAgbWV0cmljczoKICAgICAgICAgIGp2bV86CiAgICAgICAgICAgIC0gImphdmEubGFuZzoqIgogICAgICAgICAgY2FzXzoKICAgICAgICAgICAgLSAib3JnLmFwYWNoZS5jYXNzYW5kcmEubWV0cmljczoqIgogICAgICAgICAgICAtICJvcmcuYXBhY2hlLmNhc3NhbmRyYS5uZXQ6dHlwZT1GYWlsdXJlRGV0ZWN0b3IiCgogIHRpZXIxOgogICAgICBmcmVxdWVuY3k6IDMwMCAjIG1ldHJpY3MgY29sbGVjdGVkIGV2ZXJ5IDMwMCBzZWNvbmRzICg1bSkKICAgICAgbWV0cmljczoKICAgICAgICAgIGNhc186CiAgICAgICAgICAgIC0gIm9yZy5hcGFjaGUuY2Fzc2FuZHJhLm1ldHJpY3M6bmFtZT1Fc3RpbWF0ZWRQYXJ0aXRpb25Db3VudCwqIgoKICBibGFja2xpc3Q6ICMgWW91IGNhbiBibGFja2xpc3QgbWV0cmljcyBiYXNlZCBvbiBSZWdleCBwYXR0ZXJuLiBIaXQgdGhlIGFnZW50IG9uIGh0dHA6Ly9hZ2VudElQOjk5MTYvbWV0cmljc2xpc3QgdG8gbGlzdCBKTVggbWV0cmljcyBpdCBpcyBjb2xsZWN0aW5nCiAgICAtICJvcmcuYXBhY2hlLmNhc3NhbmRyYS5tZXRyaWNzOnR5cGU9Q29sdW1uRmFtaWx5LioiICMgZHVwbGljYXRpb24gb2YgdGFibGUgbWV0cmljcwogICAgLSAib3JnLmFwYWNoZS5jYXNzYW5kcmEubWV0cmljczouKnNjb3BlPVJlcGFpciMuKiIgIyBpZ25vcmUgZWFjaCByZXBhaXIgaW5zdGFuY2UgbWV0cmljcwogICAgLSAib3JnLmFwYWNoZS5jYXNzYW5kcmEubWV0cmljczouKm5hbWU9U25hcHNob3RzU2l6ZS4qIiAjIENvbGxlY3RpbmcgU25hcHNob3RzU2l6ZSBtZXRyaWNzIHNsb3dzIGRvd24gY29sbGVjdGlvbgogICAgLSAib3JnLmFwYWNoZS5jYXNzYW5kcmEubWV0cmljczouKk1heC4qIgogICAgLSAib3JnLmFwYWNoZS5jYXNzYW5kcmEubWV0cmljczouKk1pbi4qIgogICAgLSAiLio5OTl0aFBlcmNlbnRpbGV8Lio1MHRoUGVyY2VudGlsZXwuKkZpZnRlZW5NaW51dGVSYXRlfC4qRml2ZU1pbnV0ZVJhdGV8LipNZWFuUmF0ZXwuKk1lYW58LipPbmVNaW51dGVSYXRlfC4qU3RkRGV2IgoKICBKTVhPcGVyYXRpb25zQmxhY2tsaXN0OgogICAgLSAiZ2V0VGhyZWFkSW5mbyIKICAgIC0gImdldERhdGFjZW50ZXIiCiAgICAtICJnZXRSYWNrIgoKICBETUxFdmVudHNXaGl0ZWxpc3Q6ICMgWW91IGNhbiB3aGl0ZWxpc3Qga2V5c3BhY2VzIC8gdGFibGVzIChsaXN0IG9mICJrZXlzcGFjZSIgYW5kL29yICJrZXlzcGFjZS50YWJsZSIgdG8gbG9nIERNTCBxdWVyaWVzLiBEYXRhIGlzIG5vdCBhbmFseXNlZC4KICAjIC0gInN5c3RlbV9kaXN0cmlidXRlZCIKCiAgRE1MRXZlbnRzQmxhY2tsaXN0OiAjIFlvdSBjYW4gYmxhY2tsaXN0IGtleXNwYWNlcyAvIHRhYmxlcyBmcm9tIHRoZSBETUxFdmVudHNXaGl0ZWxpc3QgKGxpc3Qgb2YgImtleXNwYWNlIiBhbmQvb3IgImtleXNwYWNlLnRhYmxlIiB0byBsb2cgRE1MIHF1ZXJpZXMuIERhdGEgaXMgbm90IGFuYWx5c2VkLgogICMgLSBzeXN0ZW1fZGlzdHJpYnV0ZWQucGFyZW50X3JlcGFpcl9oaXN0b3J5CgogIGxvZ1N1Y2Nlc3NmdWxSZXBhaXJzOiBmYWxzZSAjIHNldCBpdCB0byB0cnVlIGlmIHlvdSB3YW50IHRvIGxvZyBhbGwgdGhlIHN1Y2Nlc3NmdWwgcmVwYWlyIGV2ZW50cy4KCiAgd2FybmluZ1RocmVzaG9sZE1pbGxpczogMjAwICMgVGhpcyB3aWxsIHdhcm4gaW4gbG9ncyB3aGVuIGEgTUJlYW4gdGFrZXMgbG9uZ2VyIHRoYW4gdGhlIHNwZWNpZmllZCB2YWx1ZS4KCiAgbG9nRm9ybWF0OiAiJTQkcyAlMSR0WS0lMSR0bS0lMSR0ZCAlMSR0SDolMSR0TTolMSR0UywlMSR0TCAlNSRzJTYkcyVuIg==
Start up
Create Axon Agent configuration
a3ViZWN0bCBjcmVhdGUgbnMgY2Fzc2FuZHJhCmt1YmVjdGwgY3JlYXRlIGNvbmZpZ21hcCBheG9ub3BzLWFnZW50IC0tZnJvbS1maWxlPWF4b24tYWdlbnQueW1sIC1uIGNhc3NhbmRyYQ==
Run helmfile
With locally installed helm and helmfile
Y2QgeW91ci9jb25maWcvZGlyZWN0b3J5CmhlbWxmaWxlIHN5bmM=
With docker image
ZG9ja2VyIHJ1biAtLXJtIAogICAgLXYgfi8ua3ViZTovcm9vdC8ua3ViZSAKICAgIC12ICR7UFdEfS8uaGVsbTovcm9vdC8uaGVsbSAKICAgIC12ICR7UFdEfS9oZWxtZmlsZS55YW1sOi9oZWxtZmlsZS55YW1sIAogICAgLXYgJHtQV0R9L3ZhbHVlcy55YW1sOi92YWx1ZXMueWFtbCAKICAgIC0tbmV0PWhvc3QgY2hhdHdvcmsvaGVsbWZpbGUgc3luYw==
Access
Minikube
If you used minikube, identify the name of the service with kubectl get svc -n monitoring and launch it with
bWluaWt1YmUgc2VydmljZSBheG9ub3BzLWF4b24tZGFzaCAtbiBtb25pdG9yaW5n
LoadBalancer
Find the DNS entry for it:
a3ViZWN0bCBnZXQgc3ZjIC1uIG1vbml0b3JpbmcgLW8gd2lkZQ==
Open your browser and copy and paste the URL.
Troubleshooting
Check the status of the pods:
a3ViZWN0bCBnZXQgcG9kIC1uIG1vbml0b3JpbmcKa3ViZWN0bCBnZXQgcG9kIC1uIGNhc3NhbmRyYQ==
Any pod which is not on state Running check it out with
a3ViZWN0bCBkZXNjcmliZSAtbiBOQU1FU1BBQ0UgcG9kIFBPRC1OQU1F
Storage
One common problem is regarding storage. If you have enabled persistent storage you may see an error about persistent volume claims (not found, unclaimed, etc). If you're using minikube make sure you enable storage with
bWluaWt1YmUgYWRkb25zIGVuYWJsZSBzdG9yYWdlLXByb3Zpc2lvbmVy
Memory
The second most common problem is not enough memory (OOMKilled). You will see this often if you're node does not have enough memory to run the containers or if the heap settings for Cassandra are not right. kubectl describe command will be showing Error 127 when this occurs.
In the values.yaml file adjust the heap options to match your hardware:
bWF4X2hlYXBfc2l6ZTogNTEyTQogIGhlYXBfbmV3X3NpemU6IDUxMk0=
Minikube
Review the way you have started up minikube and assign more memory if you can. Also check the available drivers and select the appropriate for your platform. On MacOS where I tested hyperkit or virtualbox are the best ones.
bWluaWt1YmUgc3RhcnQgLS1tZW1vcnkgMTAyNDAgLS1jcHVzPTQgLS1kcml2ZXI9aHlwZXJraXQ=
Putting it all together
This short video shows how quickly you can run Cassandra with AxonOps on a Kubernetes cluster. The video uses a "helmfile" to manage the packages and configurations. https://youtu.be/OvRZkS0FNCg
