Introduction

In a previous blog post we talked about using Kubernetes Network Policies to secure traffic between pods and namespaces, for example, between the frontend web servers and the databases.

This is not much different from what you would have done if your application was running on Virtual Machines or dedicated servers.

Whilst Kubernetes Network Policies help to resolve the problem of securing network traffic they don’t by themselves address all the issues. Remember to also implement security at the application and transport levels by encrypting traffic when possible.

The problem with Kubernetes Network Policies is that they have some limitations that for many users would be a problem:

• You cannot deny traffic, only allow
• There is no logging for security events
• You can’t route traffic through a common gateway
• You cannot block localhost traffic
• Policies are applied per namespace, no global policies

Some of these issues are addressed by third parties such as Tigera who develops Project Calico. We’re going to look at this briefly below.

Calico Products

Calico is a product by the company Tigera. If you used Kubernetes before you most likely used Calico already in one form or another as it is a well known CNI for the platform. But you have been using the Open Source edition and Tigera offers different products with more features. This blog’s lab covers only the open-source features but I do mention along the way some of the most interesting options in the Enterprise Edition.

LAB Setup

You can use either minikube or k3s / k3d if you want to try this lab locally on your computer. I’m going to demonstrate how to use Calico Network Policies and compare them against the standard Kubernetes Network Policies. This is how you start it up if you’d like to play with it.

#!/bin/bash
 
#
# Make sure the CIDR selected is not in use in your network
#
k3d cluster create --api-port 6550 \
 --agents 2 \
 --k3s-server-arg "--flannel-backend=none" \
 --k3s-server-arg "--cluster-cidr=192.168.0.0/16" \
 --k3s-server-arg "--disable-network-policy" \
 --k3s-server-arg "--disable=traefik" \
 calico
 
kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml
kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml

Wait for the installation to complete by checking the calico-system namespace.

~$ k get pod -n calico-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-typha-6c9ccfddd5-7t8zv              1/1     Running   0          14m
calico-typha-6c9ccfddd5-gm5qz              1/1     Running   0          14m
calico-typha-6c9ccfddd5-mpbrz              1/1     Running   0          14m
calico-node-pt7l7                          1/1     Running   0          14m
calico-node-pwvs8                          1/1     Running   0          14m
calico-node-cg96c                          1/1     Running   0          14m
calico-kube-controllers-86bc854bdb-zrncr   1/1     Running   0          14m

If you prefer to use minikube check out Calico website. This command should work with up-to-date versions of minkube:

~$ minikube start \
  --memory 8192 \
  --cpus=4 \
  --driver=hyperkit \
  --network-plugin=cni \
  --cni=calico

LAB Application

This shows the network diagram of the applications we’re going to secure. It’s a standard 3-tier application with a frontend (webserver), a middleware application that handles data requests and a database server.

Default Deny

As you know all traffic is allowed by default in Kubernetes, hence the first thing you should do is block all traffic. Now, this is one of the first advantages of using Calico instead of the default Network Policies. Kubernetes Network Policies need to be applied to each and every namespace where you would like to manage traffic and if you have many namespaces this can be quite cumbersome to manage. Calico offers you the choice to use a GlobalPolicy (all namespaces) or targeting individual ones.

I don’t recommend you start blocking Egress until you have all the ingress rules worked out as it will be much harder. The rule below blocks ingress only:

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
 name: default-deny-ingress
spec:
 namespaceSelector: has(kubernetes.io/metadata.name) && kubernetes.io/metadata.name not in {"kube-system", "calico-system"}
 types:
 - Ingress

If you want to go the full way, try something like the below. Please note this is excluding the namespaces “kube-system” and “calico-system” as recommended by Calico whilst also allowing DNS which would otherwise break most things.

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
 name: default-deny
spec:
 namespaceSelector: has(kubernetes.io/metadata.name) && kubernetes.io/metadata.name not in {"kube-system", "calico-system"}
 
 types:
 - Ingress
 - Egress
 egress:
 # allow all namespaces to communicate to DNS pods
 - action: Allow
   protocol: UDP
   destination:
     selector: 'k8s-app == "kube-dns"'
     ports:
     - 53

Before applying the rule I can confirm the web server can connect to the application server or database:

~$ kubectl exec -ti -n frontend webserver02 -- nc -w 3 -v appserver.application 8080
appserver.application (10.43.95.231:8080) open
~$ kubectl exec -ti -n application appserver01 -- nc -w 3 -v db.database 5432
db.database (10.43.220.54:5432) open

And right after the default deny (ingress only for now) is applied the traffic is otherwise denied:

~$ kubectl exec -ti -n frontend webserver02 -- nc -w 3 -v appserver.application 8080
nc: connect to appserver.application port 8080 (tcp) timed out: Operation in progress
command terminated with exit code 1

Now we’re ready to start building the rules to allow access.

Webserver to application

Our webservers need to connect to the application servers:

This translates to the following configuration:

apiVersion: crd.projectcalico.org/v1
kind: NetworkPolicy
metadata:
 name: allow-application-8080
 namespace: application
Spec:
 order: 100
 selector: app == 'appserver'
 ingress:
 - action: Allow
   protocol: TCP
   source:
     selector: app == 'webserver'
     namespaceSelector: name == 'frontend'
   destination:
     ports:
       - 8080

We can now confirm that webservers are able to connect to the application servers:

~$ for x in 01 02 03; do kubectl exec -ti -n frontend webserver${x} -- nc -w 3 -v appserver.application 8080;done
Connection to appserver.application 8080 port [tcp/http-alt] succeeded!
Connection to appserver.application 8080 port [tcp/http-alt] succeeded!
Connection to appserver.application 8080 port [tcp/http-alt] succeeded!

One thing you cannot do with the standard Kubernetes Network Policies is deny traffic, you can only allow it. <link to previous blog post here please>. Calico does allow you to do this. Let’s see an example.

NAME          READY   STATUS    RESTARTS   AGE    LABELS
webserver01   1/1     Running   0          106m   app=webserver,type=blue
webserver02   1/1     Running   0          106m   app=webserver,type=green
webserver03   1/1   Running  0         106m   app=webserver,type=yellow

We have labelled all pods with a type that will allow us to target individual pods when required. Let’s use that to block access from type=yellow to confirm we can apply a deny.

apiVersion: crd.projectcalico.org/v1
kind: NetworkPolicy
metadata:
 name: deny-webserver03-to-application
 namespace: application
spec:
 order: 10
 selector: app == 'appserver'
 ingress:
 - action: Deny
   protocol: TCP
   source:
     selector: app == 'webserver' && type == 'yellow'
     namespaceSelector: name == 'frontend'
   destination:
     ports:
       - 8080

Two important things to notice here:

• order: if you used traditional firewalls before you probably are aware of the importance of order. Most firewalls work on a first match rule: the first entry that successfully selects a source (for ingress) or destination (for egress) is the last one used, for example, Cisco firewalls have an implicit deny rule in their access-lists. This is why you have to be careful when ordering and for this Deny rule to work we have to set up the ordering (order: 10) to be lower than the default deny we applied earlier (order: 100)
• selector: the selectors used by Calico are more flexible allowing you to perform conditions such as the and (&&) condition we just used.

As you can see we have successfully blocked webserver03 from connection to the application server.

~$ for x in 01 02 03; do kubectl exec -ti -n frontend webserver${x} -- nc -w 3 -v appserver.application 8080;done
Connection to appserver.application 8080 port [tcp/http-alt] succeeded!
Connection to appserver.application 8080 port [tcp/http-alt] succeeded!
nc: connect to appserver.application port 8080 (tcp) timed out: Operation in progress

Logging

We can complete the rule above with logging, another improvement Calico offers over Network Policies. In order to do this we add a Log action to the NetworkPolicy:

apiVersion: crd.projectcalico.org/v1
kind: NetworkPolicy
metadata:
 name: deny-webserver03-to-application
 namespace: application
spec:
 order: 10
 selector: app == 'appserver'
 ingress:
 - action: Log
   protocol: TCP
   source:
     selector: app == 'webserver' && type == 'yellow'
     namespaceSelector: name == 'frontend'
 - action: Deny
   protocol: TCP
   source:
     selector: app == 'webserver' && type == 'yellow'
     namespaceSelector: name == 'frontend'
   destination:
     ports:
       - 8080

Logs are sent to the syslog server. You will need to set up something to be able to see the logs, be it Rancher Logging, Filebeat, Banzai Logging Operator, etc with our preferred method being Calico Enterprise which also has some very interesting features that most large companies will be interested in such as auditing and compliance reports.

Application to Database

Let’s now complete the lab configuration by adding the rules to allow traffic from the application to the database.

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
 name: allow-postgres-5432
 namespace: database
spec:
 order: 110
 selector: app == 'db'
 ingress:
 - action: Allow
   protocol: TCP
   source:
     selector: app == 'appserver'
     namespaceSelector: name == 'application'
   destination:
     ports:
       - 5432

The rule is very similar to what we used for webserver to application servers traffic. But let’s do it in a different way, something that is is only available for Calico: using service accounts as selectors:

Service account rules

When I created my application pods I assigned them to the service account named “application”:

~$ kubectl get -n application pod/appserver01 -ojsonpath={.spec.serviceAccountName}
application

We can use this service account name as a selector for our rules. The syntax is pretty similar to using labels and you can specify multiple service account names when building up the access list.

apiVersion: crd.projectcalico.org/v1
kind: NetworkPolicy
metadata:
 name: allow-postgres-5432
 namespace: database
spec:
 order: 110
 selector: app == 'db'
 ingress:
 - action: Allow
   protocol: TCP
   source:
     serviceAccounts:
       - names: application
     selector: app == 'appserver'
   destination:
     ports:
       - 5432

As you can see we are no longer using the regular selector but just the service account. I have kept the namespace selector for extra security. We can confirm the rule is in place by the usual method:

~$ kubectl exec -ti -n application appserver01 -- nc -w 3 -v db.database 5432
Connection to db.database 5432 port [tcp/postgresql] succeeded!

ICMP ping

The default Kubernetes Network Policies only support TCP and UDP. Calico also supports ICMP which will provide you with further security by making your network more difficult to scan and discover endpoints. This is very handy in combination with GlobalPolicies as you can create a policy to allow PING from certain pods or namespaces only whilst keeping the rest of the cluster locked down.

Be careful with ordering to make sure your policies are hit. Remember, like most firewalls, processing stops on the first matched rule. No further rules will be examined.

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: block-icmp
spec:
  order: 200
  selector: all()
  types:
  - Ingress
  - Egress
  ingress:
  - action: Deny
    protocol: ICMP
  - action: Deny
    protocol: ICMPv6
  egress:
  - action: Deny
    protocol: ICMP
  - action: Deny
    protocol: ICMPv6

Host level rules

Calico also provides a way of managing access to the underlying hosts. This is a good feature for people running their own infrastructure such as using RKE. It can help you block access to unexpected ports and services like SSH.

You ought to remember that there are some ports you will always need to allow. These are the ones required by either Calico itself or Kubernetes to continue working. Make sure you have these allowed!

Source: https://docs.projectcalico.org/security/protect-hosts

The example below allows SSH to the workers from just a management network CIDR you might have configured:

apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
  name: k8s-worker
spec:
  selector: "role == 'k8s-worker'"
  order: 0
  ingress:
  - action: Allow
    protocol: TCP
    source:
      nets:
      - "<your management CIDR>"
    destination:
      ports: [22]

It also allows you to set up advanced rules such as blocking external CIDR to your cluster, adding some DoS protection and even controlling access to services node ports.

All these policy types require you to define a new object kind called HostEndpoint which describes (as the name implies) a host interface. For example, the below rule defines a host set up in a minikube lab environment.

---
apiVersion: crd.projectcalico.org/v1
kind: HostEndpoint
metadata:
 name: node1-eth0
 labels:
   role: k8s-worker
spec:
 interfaceName: eth0
 node: minikube
 expectedIPs:
 - 192.168.64.74
---
apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
 name: k8s-worker
spec:
 # the selector matches the HostEndpoint definition above
 selector: "role == 'k8s-worker'"
 order: 0
 ingress:
 - action: Allow
   protocol: TCP
   source:
     nets:
     - "192.168.0.0/16"
   destination:
     ports: [8443] # kubernetes api port in minikube
 - action: Allow
   protocol: ICMP
 egress:
 - action: Allow
   protocol: TCP
   destination:
     nets:
     - "0.0.0.0/0"
 - action: Allow
   protocol: UDP
   destination:
     ports: [53, 67]

When the HostEndpoint is created, traffic to or from the interface is dropped unless the policy is in place.

We are only allowing ingress access to the Kubernetes API (or we would be locked out) and egress to DNS and DHCP (without which we would struggle to launch new pods).

Conclusion

As you can see Calico Network Policies are superior to standard Kubernetes giving you far more control over the Kubernetes cluster and the traffic crossing it. Additionally, if you opt-in for the Enterprise edition you will be able to use the Web UI for managing the rules in a friendly and visual way with access to the logs to see where your traffic is blocked and why.

Calico Enterprise has much more to offer and would make this blog quite long. But one of my favourite features is to set up rules in Draft mode. Draft mode means the rule is in place, logging and reporting, counting hits, etc but it is not enforced. We all know how scary it can be to make changes to production environments. This way you can test the rules first without breaking anything before and get the courage you need to go ahead with the changes.

Another brilliant thing about Calico Enterprise is the integration with managed cloud providers. For example, being able to restrict a Pod Egress based on an AWS security group.

Do have a look at their Policy Workflow document when you have a minute, it is worth it.

‍