Automatically set Terraform environment variables

If you have spent any time managing infrastructure you have probably run into the issue of needing to set environment variables in order to connect to various resources.

For example, the AWS Terraform provider allows you to automatically source local environment variables, which solves the issue of placing secrets in places they should be, ie. writing the keys into configurations or state. This use case is pretty straight forward, you can just set the environment variables once and everything will be able to connect.

The problem arises when environments need to be changed frequently or other configurations requiring secrets or tokens to connect to other resources need to be changed.

There are various tools available to solve the issue of managing and storing secrets for connecting to AWS including aws-vault and aws-env. The main issue is that these tools are opinionated and only work with AWS and aren’t designed to work with Terraform.

Of course this problem isn’t specific to Terraform, but the solution I have discovered using ondir is generic enough that it can be applied to Terraform as well as any other task that involves setting up environment variables or running shell commands whenever a specific directory is changed to or left.

With that said, the ondir tool in combination with Terragrunt, gives you a set of tools that vastly improves the Terraform experience. There is a lot of Terragrunt reference material already so I won’t discuss it much here. The real power of my solution comes from combining ondir with Terragrunt to enhance how you use Terraform.

Note: Before getting into more details of ondir, it is worth mentioning that there are several other similar tools, the most well known of which is called direnv. Direnv provides a few other features and a nice library for completing actions so it is definitely work checking out if you are not looking for a solution specific to Terraform/Terragrunt. Other tools in the space include autoenv, smartcd and a few others.

Direnv should technically work to solve the problem, but when managing and maintaining large infrastructures, I have found it to be more tedious and cumbersome to manage.

Ondir

From the Github page, Ondir is a small program to automate tasks specific to certain directories, so isn’t designed specifically to set environment variables but makes a perfect fit doing so.

Install ondir

# OSX
brew install ondir
# Ubuntu
apt install ondir

Configure ondir

After installing, add the following lines to your ~/.zshrc (or these lines if you are using bash) and restart your shell.

val_ondir() {
  eval "`ondir \"$OLDPWD\" \"$PWD\"`"
}
chpwd_functions=( eval_ondir $chpwd_functions )

Ondir leverages an ~/.ondirrc file to configure the tool. This file basically consists of enter and leave directives for performing actions when specific directories are entered and left. To make this tool useful with Terragrunt, I have setup a simple example below for exporting a database password given a path that gets switched into (if the path ends in stage/database), which is defined in the ~/.ondirrc file. One very nice feature of ondir is that it allows regular expressions to be used to define the enter/leave directives, which makes allows for complex paths to be defined.

enter .+(stage\/database)
    echo "Setting database password"
    export TF_VAR_db_password=mypassword
leave .+(stage\/database)
    unset TF_VAR_db_password

With this logic, the shell should automatically export a database password for Terraform to use whenever you switch to the directory ending in stage/database.

cd ~/project/path/to/stage/database
# Check your environment for the variable
env | grep TF_VAR
TF_VAR_db_password=mypassword

cd ~
# When you leave the directory it should get unset
env | grep TF_VAR

In a slightly more complicated example, you may have a databases.sh script containing more logic for setting up or exporting variables or otherwise setting up environments.

#!/usr/bin/env bash

db_env=$(pwd | awk -F '/' '{print $(NF-1)"/"$NF}')

if [[ "$db_env" == "stage/database" ]]; then
    # do stuff
    export TF_VAR_db_password="mypassword"

else [[ "$db_env" == "prod/database" ]]; then
    # do other stuff
    export TF_VAR_db_password="myotherpassword"
fi

Conclusion

That’s pretty much. Now you can automatically set environment variables for Terraform to use, based on the directory we are currently and no longer need to worry about setting and unsetting environment variables manually.

I HIGHLY recommend checking out direnv as well. It offers slightly different functionality, and some neat features which in some cases may actually be better for specific tasks than ondir. For example, the direnv stdlib has some great helpers to help configure environments cleanly.

For me, using ondir for managing my Terragrunt configs and direnv for managing other projects makes the most sense.