In the last post I talked about my old directory of .env files full of customer credentials and promised to deal with the ones that live on engineers' laptops. The worst of those is not even a .env file but ~/.aws/credentials. Most engineers I know have one with a handful of profiles in it. Each profile has an access key that was created the day they joined the project and never rotated since.

We already use OpenBao for dynamic database credentials and AWS works on the same principle. Instead of a key that lives forever you ask OpenBao for credentials when you need them and they stop working on their own.

How OpenBao issues AWS credentials

The AWS secrets engine keeps one set of AWS credentials for itself and uses them to create credentials for you. It supports four credential types. iam_user creates a real IAM user with an access key and deletes it when the lease ends. The other three (assumed_role, federation_token and session_token) return temporary STS credentials.

We use assumed_role. OpenBao calls sts:AssumeRole on an IAM role you define and you get back an access key, a secret and a session token. They expire after the TTL you asked for, which can be anything between 15 minutes and 12 hours. Nothing is created in IAM when you request them so there is nothing to clean up. The permissions live in ordinary IAM roles your AWS team can review like any other.

The downside is revocation. AWS has no API to cancel a single STS session so revoking the lease in OpenBao does nothing on the AWS side. If you need credentials you can kill on demand then iam_user gives you that. The price is that OpenBao needs permission to create and delete IAM users and each new user takes a few seconds to propagate. For day to day engineering access we prefer short STS credentials.

What OpenBao needs in AWS

You need two things in AWS. The first is an IAM user we call openbao. Its policy allows sts:AssumeRole on the roles OpenBao hands out plus the four IAM actions it needs to rotate its own key. The second is one IAM role per access level such as openbao-developer with ReadOnlyAccess. Its trust policy names the openbao user as the only principal allowed to assume it.

I set this up by hand first and it took me longer than I would like to admit. My first attempt failed with AccessDenied because I had given OpenBao my own IAM keys and the role only trusted the openbao user. The second failed because I had copied the trust policy from an existing role that still trusted a user in another account. Then rotate-root failed because the policy was missing iam:GetUser. None of this is hard but the error messages only tell you which call was denied. They do not tell you which of the two policies is wrong.

Two AWS limits are worth knowing before you start. A role's MaxSessionDuration defaults to one hour and AWS rejects any request for a longer session, so raise it to match the longest TTL you want to allow. OpenBao's own credentials must also belong to an IAM user rather than a role because AWS caps role chaining at one hour whatever the role allows.

Keeping the root key out of everyone's hands

The openbao user needs an access key and that key is the most sensitive thing in the whole setup. Our first idea was to create it with aws_iam_access_key in OpenTofu and pass it to the vault_aws_secret_backend resource. It works but the AWS provider stores the secret in the state file and there is no write-only version of that resource.

OpenBao has a better answer called rotate-root. It creates a new key for its own IAM user and deletes the old one. We could not drive that from OpenTofu either. Once OpenBao deletes the key OpenTofu created, the next plan sees the key has gone and creates a new one, so you end up in a loop. In the end we moved the bootstrap into a short script that OpenTofu runs once:

aws iam create-access-key --user-name "$IAM_USER" --output json |
  jq --arg region "$AWS_REGION" \
    '{access_key: .AccessKey.AccessKeyId, secret_key: .AccessKey.SecretAccessKey, region: $region}' |
  bao write "${MOUNT_PATH}/config/root" - >/dev/null

bao write -f "${MOUNT_PATH}/config/rotate-root"

The key travels from the AWS CLI to OpenBao through a pipe so it never shows up on screen, in the shell history or in the state. A few seconds later OpenBao replaces it with a key it generated itself and from then on nobody knows the credentials behind every role. The real script retries rotate-root for a minute because a brand new IAM key can take a few seconds to start working.

Running the demo

Everything above is in the repository along with a config for a local OpenBao dev server that downloads the AWS plugin when it starts:

git clone https://github.com/digitalis-io/openbao-aws-creds-demo.git
cd openbao-aws-creds-demo
make bao-dev                      # first terminal

export BAO_ADDR=http://127.0.0.1:8200 BAO_TOKEN=root
export AWS_PROFILE=my-admin       # an identity that can manage IAM
make plan && make apply
bao write aws/sts/developer ttl=1h

Access levels are a map so adding a short lived admin role only takes a few lines in terraform.tfvars:

roles = {
  developer = {
    managed_policy_arns = ["arn:aws:iam::aws:policy/ReadOnlyAccess"]
  }
  admin = {
    managed_policy_arns = ["arn:aws:iam::aws:policy/AdministratorAccess"]
    default_ttl         = 900
    max_ttl             = 3600
  }
}

Each role also gets its own OpenBao policy (aws-developer and aws-admin) which you attach to whoever should be able to request it. You can see a quick test below. I'm obscuring the sensitive bits but to be honest by the time you read this the credentials will have expired

Credentials that live in one terminal

What I wanted was credentials that exist in one terminal and nowhere else. The simplest way to get that is a subshell with the credentials in its environment:

aws-session() {
  local role="${1:?usage: aws-session <role> [ttl]}" ttl="${2:-1h}" creds
  creds=$(bao write -format=json "aws/sts/${role}" ttl="$ttl") || return 1
  env -u AWS_PROFILE \
    AWS_ACCESS_KEY_ID="$(jq -r .data.access_key <<<"$creds")" \
    AWS_SECRET_ACCESS_KEY="$(jq -r .data.secret_key <<<"$creds")" \
    AWS_SESSION_TOKEN="$(jq -r '.data.session_token // .data.security_token' <<<"$creds")" \
    AWS_SESSION_ROLE="$role" \
    "$SHELL"
}

aws-session developer 2h drops you into a shell where the AWS CLI, Terraform and anything else built on the SDK picks up the credentials. Type exit or close the window and they are gone from the machine. Nothing is written to disk and each terminal can hold a different role at the same time.

Be clear about what closing the terminal actually does. The credentials disappear from your laptop but they stay valid in AWS until the TTL runs out, which is why we keep the default at one hour.

By far my preferred way is this simple hack in ~/.aws/config:

[profile openbao-developer]
region = eu-west-2
credential_process = sh -c "bao write -format=json aws/sts/developer ttl=1h | jq '{Version: 1, AccessKeyId: .data.access_key, SecretAccessKey: .data.secret_key, SessionToken: (.data.session_token // .data.security_token), Expiration: (now + .lease_duration | todate)}'"

Then you can use it like any other AWS profile:

$ BAO_ADDR=http://localhost:8200 AWS_PROFILE=openbao-developer aws sts get-caller-identity
{
    "UserId": "AROAXXXXXXXXXXHH:vault-token-developer-1790243125-8BP6BZl2IANwr7kF4IYT",
    "Account": "000000000000",
    "Arn": "arn:aws:sts::000000000000:assumed-role/openbao-developer/vault-token-developer-1790243125-8BP6BZl2IANwr7kF4IYT"
}

The bao command needs BAO_ADDR and a token in its environment. I forgot the address the first time and all the AWS CLI told me was Expecting value: line 1 column 1 (char 0), which is what you get when the process returns nothing at all.

Stopping access at the end of the day

The last piece is the OpenBao login itself. Every lease is a child of the token that requested it and nobody can request anything once that token has expired. If engineers log in through OIDC with an eight hour token they authenticate with SSO in the morning and request as many one hour credentials as they need during the day. Shortly after they stop working they are locked out of AWS:

bao write auth/oidc/role/engineers \
  token_policies=aws-developer token_ttl=8h token_max_ttl=8h \
  ...

At worst the last credentials issued are still valid for an hour after the token expires. There is no key on the laptop to leak and with an audit device enabled every request is logged against the person who made it.

For emergencies AWS can invalidate every session for a role at once. The "Revoke active sessions" button in the IAM console adds a policy that denies anything with a token issued before that moment. It is heavy handed because it cuts off everyone using the role, but it is worth knowing where it is before you need it.

OpenBao and Vault

Everything here works on HashiCorp Vault too. Vault has the AWS engine built in while on OpenBao we run it as a plugin from the OpenBao registry. We configure OpenBao with the hashicorp/vault OpenTofu provider because the two share an API.

Not only AWS

AWS was the subject of the day but the same idea applies to most of the credentials an engineer carries around. OpenBao has secrets engines for Google Cloud and Azure that hand out short lived access tokens, service account keys or service principals in much the same way. For Kubernetes the secrets engine creates a service account token scoped to one namespace, which beats a kubeconfig with cluster admin in it that never expires. The SSH engine signs a certificate that is valid for half an hour so nobody needs to add their key to authorized_keys, and the database engine we covered in Dynamic credentials with OpenBao and Vault does the same for database users.

The pattern does not change from one to the next. OpenBao keeps the one powerful credential, you log in with SSO and whatever you get back expires on its own. Once the login and the policies are in place adding another service is mostly a case of mounting another engine. We will cover GCP in a follow up.