Introduction

Terraform is a brilliant tool to manage Infrastructure as Code we at Digitalis both love and hate in equal measure. We love it because it manages IaC very well and the amount of providers available is continuously growing in number and quality. We hate it because when it goes wrong, it goes horribly wrong and it can be quite difficult to get out of trouble.

Size

The first thing to mind is the terraform repository size. It’s a bad practice to do too much in one repository alone. Try to split the code into multiple small repositories. If you need data from one repository to be used on another you can use either data resources or remote state:

For example, you have created an AWS Instance on another terraform repository and now you need to reference it to set up some security groups. You can do:

1. Use a data resource to locate the instance based on the tag Name

// First locate the instance created by Terraform on another repo
// filtering by tag
data "aws_instance" "foo" {
  filter {
    name   = "tag:Name"
    values = ["instance-name-tag"]
  }
}
// Use the aws_instance on a security group
resource "aws_network_interface_sg_attachment" "sg_attachment" {
  security_group_id    = aws_security_group.sec_group.id
  network_interface_id = data.aws_instance.foo.primary_network_interface_id
}

2. Use a remote state

The advantage of using the remote state is you are not required to define a data block every time you need to reference another resource. You configure the remote state only once and it gives you access to everything exported by that repository.

There is one downside though. The remote state has to output the information we’re allowed to use using outputs.For the example below to work, the remote repository will need to contain

output "instance_ip" {
  value = aws_instance.foo.primary_network_interface_id
}

then we can use it like:

data "terraform_remote_state" "remote" {
  backend = "s3"
  config {
    bucket = "my-bucket-name"
    key    = "my-key-name"
    region = "my-region"
  }
}
// Use the aws_instance on a security group
resource "aws_network_interface_sg_attachment" "sg_attachment" {
  security_group_id    = aws_security_group.sec_group.id
  network_interface_id = data.terraform_remote_state.remote.outputs.instance_ip
}

Remote

The terraform state file is the most important piece. It contains a description of all the resources created by terraform. If this is lost, it won’t know what has been created and whatnot.

Terraform provides many backend types to ensure the state file is in a safe place. My default is usually an object storage such as S3, GCS, Swift, etc. But you can also use HTTP backends and many others. Choose carefully but choose a remote one.

Unique

And make it unique! Don’t use the same remote file for all your deployments. You’re increasing the risk of something going wrong every time you add a new deployment. A simple conflict on a resource name will cause havoc! Bad bad idea!

Modules

Another common problem with terraform is code repetition. There are many tasks we do all the time based on company requirements. For example, creating an AWS instance requires a single resource: aws_instance . But then you’ll probably want to add some security groups for it and attach an EBS volume as well. This simple one resource terraform is now several resources long.

What you should do in this case and construct a module and reference it from every one of the repositories requiring this. A great advantage in my opinion is also the ability to version the modules.

module "aws_instance" {
  source = github.com/mycompany/tf-aws-instance?ref=1.0.0
  name   = "myinstace"
  sg     = [security_groups_list] // attach this security groups
  volume = 10 // create volume of 10 Gb
}

Dynamic config

I’ve always been a proponent of using a Makefile to simplify the running and management of terraform code. Why?

• Simplifies CLI commands
• Avoid typos and errors
• Provides a way to dynamically reconfigure terraform
• Integrates very well with CI/CD pipelines

There are multiple examples you can find online and I’m sharing some snippets I use below.

.EXPORT_ALL_VARIABLES:
.ONESHELL:
.SHELL := /bin/bash
.PHONY: apply destroy plan prep
ENV ?= "dev"
REGION ?= "us-east-1"
VARS="params/$(REGION)/$(ENV)/params.tfvars"
REPO="my-repository"
# Dynamically set up the terraform backend
# into a S3 bucket
prep:
        @terraform init \
                -backend=true \
                -backend-config="region=${REGION}" \
                -backend-config="key=${REPO}" \
                -backend-config="bucket=${REPO}-${REGION}-${ENV}"
# run make plan and pass on some dynamic variables
# and source the rest of variables from config $VARS file
plan: prep ## Show what terraform thinks it will do
        @terraform plan \
                -detailed-exitcode \
                -out=plan.out \
                -lock=true \
                -input=false \
                -refresh=true \
                -var="env=$(ENV)" \
                -var="region=$(REGION)" \
                -var-file="$(VARS)"; \
                EXIT_CODE=$$?; \
                echo "Plan exited with status $$EXIT_CODE"; \
                echo $$EXIT_CODE > status

Conclusion

Some take-home points:

• Keep the terraform repositories small and laser-focused on a single task. You want to be able to create, update and destroy any given resource without affecting the rest of the environment. And you want this fast! If your repo grows too much you’ll see the terraform plan commands taking a long time. If the terraform plan takes more than a few seconds, you’re doing something wrong.
• Don’t repeat code, use modules. You’ll have faster deployments and release cycles.
• Use CI/CD when possible to deploy to avoid human errors when running terraform. Makefiles and custom shell scripts are the way to go.

‍