Introduction

In a previous blog post, I talked about how we run Kubernetes clusters without load balancers both on-premises and in the cloud. The overwhelming response to that piece was both flattering and a tad surprising, which leads me to believe that a sequel is in order.

As much as I love Kubernetes it is not always the right tool for the job. In cases where there are only a limited number of containers, it becomes necessary to construct a minimum of three worker nodes within Kubernetes, resulting in higher associated costs compared to alternative options such as ECS, Google Cloud Run, or Azure Container Instances.

I’m currently helping several companies migrate out of cloud providers back to on-premises and they often can save quite a lot of 💵

One of the factors contributing to the higher cost of Amazon Elastic Container Service (ECS) is the necessity for load balancers when exposing containers to the network, whether it be for internal or external access, such as in the case of a web server. This requirement results in the proliferation of load balancers, which in turn escalates the operational expenses associated with ECS.

Kubernetes does not suffer of this problem as you can use an Ingress to share a single load balancer with many applications.

AWS ECS

I’m going to show you below how you can set up AWS ECS with two containers. The main container if your application and the second container is the cloudflared daemon for the ingress tunnel. But first, let’s look at why we would want to do it.

• Price: determining the exact cost of a load balancer is hard and is not within the scope of this discussion. You can try to work it out from their pricing website but but I can tell you already it is not anywhere near zero $. On the other hand, Cloudflare (depending on the user case and how many tunnels you run) could be free.

• Security: if you check out the diagrams below, you may have spotted that the connection line between the Internet and the container goes in the opposite direction: it leaves the container instead of entering. This is because the connection is initiated from the cloudflared daemon. It means you don’t need firewalls or public IP addresses. It’s all managed by cloudflare. Just like magic 🎭

Show me how

Cloudflare tunnel

The first thing you need is to create a cloudflare tunnel. You can do it from the Cloudflare dashboard or (preferably) as code using for example terraform.

You’ll also need to create a route to the application. For example, let’s say you’re deploying a website using the nginx container. You would need to create a tunnel route to http://localhost:80 (adjust port as required).

Note we are using localhost . AWS Fargate uses the awsvpc network mode. This means both of your containers (nginx + cloudflared in this example) share the same network space. This is why the cloudflared container will be able to access nginx on localhost!

Additionally, when configuring the security groups for the ECS Service, it is unnecessary to expose it for external access. Typically, access is limited to the monitoring platform exclusively, and no broader external access permissions are granted.

Containers

Now for the fun part, you’ll need to create a task definition for your ECS service where you configure two containers:

• Your application (nginx in this example)
• Cloudflared (https://hub.docker.com/r/cloudflare/cloudflared)

This is an extract from my terraform code. A couple of things I found when running this container:

• You cannot run it as root . The nonroot account will serve you well.
• You can expose port 2000 for health checks and metrics (I use the Prometheus exporter)

cloudflare_container = {
    name                                   = "${var.app.name}-cloudflared"
    cpu                                    = 256
    memory                                 = 512
    essential                              = true
    image                                  = "cloudflare/cloudflared:2024.1.5"
    enable_cloudwatch_logging              = false
    create_cloudwatch_log_group            = false
    cloudwatch_log_group_retention_in_days = var.cloudwatch_log_group_retention_in_days
    readonly_root_filesystem               = false
    memory_reservation                     = 100
    user                                   = "nonroot"

    environment = [
      {
        name  = "TUNNEL_TOKEN"
        value = cloudflare_tunnel.ecs.tunnel_token
      },
    ]

    port_mappings = [
      {
        name          = "cloudflare"
        containerPort = 2000
        protocol      = "tcp"
      }
    ]

    command = [
      "tunnel",
      "--metrics",
      "0.0.0.0:2000",
      "run"
    ]

    tags = var.tags
  }

Once deployed check out Cloudwatch for any potential configuration mistakes. If you’ve done it all right, you should be able to connect to nginx using the URL you set up when creating the Cloudflare tunnel.

Conclusion

This solution will not work for everyone. It is more limiting than running Cloudflare in Kubernetes in my opinion but it’s a neat solution for when you don’t want to use a load balancer or to save a few dollars.

The difference with cloudflared in Kubernetes is the cloudflared daemon runs outside the pods. You not only can run multiple cloudflared pods for redundancy but you can also use the same ones to route to pretty much anything in the cluster or outside of it.

I hope you enjoyed the article. Keep me posted.

I, for one welcome our new robot overlords.

‍