What begins with “T”? I have been thinking about various ways of testing infrastructure and resources lately and have been having a difficult time parsing out the various tools that are available. This post is meant to be a reference for “finding the right tool for the right job” as part of testing various infrastructures.
Often times when you start talking about testing, you will hear about the testing pyramid, which is described along with some other interesting aspects of testing Terraform in this blog post, it covers a lot of the pitfalls and gotchas you might run into.
My aim originally was to find a good tool for unit testing Terraform and as part of that adventure have uncovered a number of other interesting projects, that while not directly applicable, could be very useful for a number of different testing scenarios. Read the above blog post for more info but suffice to say, unit testing (the bottom layer of the testing pyramid) is quite difficult to do with Infrastructure as Code and Terraform and at this point is mostly not a solved problem.
Here is the list of tools that I uncovered in my research. Please let me know if there are any missing.
Terratest is written and maintained by the folks at Terragrunt and it provides a comprehensive testing experience for deploying Infrastructure as Code, testing that it works as expected and then tears down the IaC when it is finished. From what I can tell this is probably the most comprehensive testing tool out there, still doesn’t cover unit tests, but is a great way to ensure things are working end to end.
Follows the BDD philosophy and spins up, tests and spins down various Terraform resources. Works in a similar fashion to Terratest to bring up the environment, test and then tear things down.
This testing tool is one of the original tools I ran across when originally mulling over the idea of unit testing infrastructure back in the days when configuration management tools like Chef, Puppet and Salt ruled the earth.
Very similiar to awspec, this tool provides a framework for testing various AWS resources. This one is nice because it uses inspec to build on and so it has a lot of extra capabilities.
Very similar to the other “spec” tools, awspec is built in the same style as serverspec/inspec and provides a very nice interface for testing various AWS resource types. Since it is modeled after serverspec, you will need to deal with Ruby.
Another Python based unit testing style framework. This tools varies from Terraform Validate in that it mainly focuses on testing the lower level server and OS, but it does have integration for testing a number of other things.
This is a nice tool for expressing various test conditions as compliance, especially if you are already familiar with Python and its testing landscape. This tool parses configs using pyhcl and allows you to write familiar unittest style tests for Terraform configurations.
This one is a bit of a wild card but could prove interesting to some readers. Pulumi provides direct integration into your language of choice (Python for me) to enable you to write pulumi code using language native unit tests. Obviously Pulumi isn’t Terraform but I had to mention it here because there is a significant amount of crossover between the tools.
If you have spent any time managing infrastructure you have probably run into the issue of needing to set environment variables in order to connect to various resources.
For example, the AWS Terraform provider allows you to automatically source local environment variables, which solves the issue of placing secrets in places they should be, ie. writing the keys into configurations or state. This use case is pretty straight forward, you can just set the environment variables once and everything will be able to connect.
The problem arises when environments need to be changed frequently or other configurations requiring secrets or tokens to connect to other resources need to be changed.
There are various tools available to solve the issue of managing and storing secrets for connecting to AWS including aws-vault and aws-env. The main issue is that these tools are opinionated and only work with AWS and aren’t designed to work with Terraform.
Of course this problem isn’t specific to Terraform, but the solution I have discovered using ondir is generic enough that it can be applied to Terraform as well as any other task that involves setting up environment variables or running shell commands whenever a specific directory is changed to or left.
With that said, the ondir tool in combination with Terragrunt, gives you a set of tools that vastly improves the Terraform experience. There is a lot of Terragrunt reference material already so I won’t discuss it much here. The real power of my solution comes from combining ondir with Terragrunt to enhance how you use Terraform.
Note: Before getting into more details of ondir, it is worth mentioning that there are several other similar tools, the most well known of which is called direnv. Direnv provides a few other features and a nice library for completing actions so it is definitely work checking out if you are not looking for a solution specific to Terraform/Terragrunt. Other tools in the space include autoenv, smartcd and a few others.
Direnv should technically work to solve the problem, but when managing and maintaining large infrastructures, I have found it to be more tedious and cumbersome to manage.
Ondir
From the Github page, Ondir is a small program to automate tasks specific to certain directories, so isn’t designed specifically to set environment variables but makes a perfect fit doing so.
Ondir leverages an ~/.ondirrc file to configure the tool. This file basically consists of enter and leave directives for performing actions when specific directories are entered and left. To make this tool useful with Terragrunt, I have setup a simple example below for exporting a database password given a path that gets switched into (if the path ends in stage/database), which is defined in the ~/.ondirrc file. One very nice feature of ondir is that it allows regular expressions to be used to define the enter/leave directives, which makes allows for complex paths to be defined.
With this logic, the shell should automatically export a database password for Terraform to use whenever you switch to the directory ending in stage/database.
cd ~/project/path/to/stage/database
# Check your environment for the variable
env | grep TF_VAR
TF_VAR_db_password=mypassword
cd ~
# When you leave the directory it should get unset
env | grep TF_VAR
In a slightly more complicated example, you may have a databases.sh script containing more logic for setting up or exporting variables or otherwise setting up environments.
#!/usr/bin/env bash
db_env=$(pwd | awk -F '/' '{print $(NF-1)"/"$NF}')
if [[ "$db_env" == "stage/database" ]]; then
# do stuff
export TF_VAR_db_password="mypassword"
else [[ "$db_env" == "prod/database" ]]; then
# do other stuff
export TF_VAR_db_password="myotherpassword"
fi
Conclusion
That’s pretty much. Now you can automatically set environment variables for Terraform to use, based on the directory we are currently and no longer need to worry about setting and unsetting environment variables manually.
I HIGHLY recommend checking out direnv as well. It offers slightly different functionality, and some neat features which in some cases may actually be better for specific tasks than ondir. For example, the direnv stdlib has some great helpers to help configure environments cleanly.
For me, using ondir for managing my Terragrunt configs and direnv for managing other projects makes the most sense.
If you have been a CoreOS user for long you will undoubtedly have noticed that there is no real package management system. If you’re not familiar, the philosophy of CoreOS is to avoid using a package manager and instead rely heavily on leveraging the power of Docker containers along with a few system level tools to manage servers. The problem that I just recently stumbled across is that the AWS SSM agent is packaged into debian and RPM formats and is assumed to be installed with a package manager, which obviously won’t work on CoreOS. In the remainder of this post I will describe the steps that I took to get the SSM agent working on a CoreOS/Dockerized server. Overall I am very happy with how well this solution turned out.
To get started, there is a nice tutorial here for using the AWS Session Manager through the the console. The most important thing that needs to be done before “installing” the SSM agent on the CoreOS host is to set up the AWS instance with the correct permissions for the agent to be able to communicate with AWS. For accomplishing this, I created a new IAM role and attached the AmazonEC2RoleForSSM policy to it through the AWS console.
After this step is done, you can bring up the ssm-agent.
Install the ssm-agent
After ensuring the correct permissions have been applied to the server that is to be manager, the next step is to bring up the agent. To do this using Docker, there are some tricks that need to be used to get things working correctly, notably, fixing the PID 1 zombie reaping problem that Docker has.
I basically lifted the Dockerfile from here originally and adapted it into my own public Docker image at jmreicha/ssm-agent:latest. In case readers want to go try this, my image is a little bit newer than the original source and has a few tweaks. The Dockerfile itself is mostly straight forward, the main difference is that the ssm-agent process won’t reap child processes in the default Debian image.
In order to work around the child reaping problem I substituted the slick Phusion Docker baseimage, which has a very simple process manager that allows shells spawned by the ssm-agent to be reaped when they get terminated. I have my Dockerfile hosted here if you want to check out how the phusion baseimage version works.
Once the child reaping problem was solved, here is the command I initially used to spin up the container, which of course still didn’t work out of the box.
docker run \
-v /var/run/dbus:/var/run/dbus \
-v /run/systemd:/run/systemd \
jmreicha/ssm-agent:latest
I received the following errors.
2018-11-05 17:42:27 INFO [OfflineService] Starting document processing engine...
2018-11-05 17:42:27 INFO [OfflineService] [EngineProcessor] Starting
2018-11-05 17:42:27 INFO [OfflineService] [EngineProcessor] Initial processing
2018-11-05 17:42:27 INFO [OfflineService] Starting message polling
2018-11-05 17:42:27 INFO [OfflineService] Starting send replies to MDS
2018-11-05 17:42:27 INFO [LongRunningPluginsManager] starting long running plugin manager
2018-11-05 17:42:27 INFO [LongRunningPluginsManager] there aren't any long running plugin to execute
2018-11-05 17:42:27 INFO [HealthCheck] HealthCheck reporting agent health.
2018-11-05 17:42:27 INFO [MessageGatewayService] Starting session document processing engine...
2018-11-05 17:42:27 INFO [MessageGatewayService] [EngineProcessor] Starting
2018-11-05 17:42:27 INFO [LongRunningPluginsManager] There are no long running plugins currently getting executed - skipping their healthcheck
2018-11-05 17:42:27 INFO [StartupProcessor] Executing startup processor tasks
2018-11-05 17:42:27 INFO [StartupProcessor] Unable to open serial port /dev/ttyS0: open /dev/ttyS0: no such file or directory
2018-11-05 17:42:27 INFO [StartupProcessor] Attempting to use different port (PV): /dev/hvc0
2018-11-05 17:42:27 INFO [StartupProcessor] Unable to open serial port /dev/hvc0: open /dev/hvc0: no such file or directory
2018-11-05 17:42:27 ERROR [StartupProcessor] Error opening serial port: open /dev/hvc0: no such file or directory
2018-11-05 17:42:27 ERROR [StartupProcessor] Error opening serial port: open /dev/hvc0: no such file or directory. Retrying in 5 seconds...
2018-11-05 17:42:27 INFO [MessageGatewayService] Successfully created ssm-user
2018-11-05 17:42:27 ERROR [MessageGatewayService] Failed to add ssm-user to sudoers file: open /etc/sudoers.d/ssm-agent-users: no such file or directory
2018-11-05 17:42:27 INFO [MessageGatewayService] [EngineProcessor] Initial processing
2018-11-05 17:42:27 INFO [MessageGatewayService] Setting up websocket for controlchannel for instance: i-0d33006836710e7ef, requestId: 2975fe0d-846d-4256-9d50-57932be03925
2018-11-05 17:42:27 INFO [MessageGatewayService] listening reply.
2018-11-05 17:42:27 INFO [MessageGatewayService] Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.us-west-2.amazonaws.com/v1/control-channel/i-0d33006836710e7ef?role=subscribe&stream=input)
2018-11-05 17:42:27 INFO [MessageGatewayService] Successfully opened websocket connection to: %!(EXTRA string=wss://ssmmessages.us-west-2.amazonaws.com/v1/control-channel/i-0d33006836710e7ef?role=subscribe&stream=input)
2018-11-05 17:42:27 INFO [MessageGatewayService] Starting receiving message from control channel
2018-11-05 17:42:32 INFO [StartupProcessor] Unable to open serial port /dev/ttyS0: open /dev/ttyS0: no such file or directory
2018-11-05 17:42:32 INFO [StartupProcessor] Attempting to use different port (PV): /dev/hvc0
2018-11-05 17:42:32 INFO [StartupProcessor] Unable to open serial port /dev/hvc0: open /dev/hvc0: no such file or directory
2018-11-05 17:42:32 ERROR [StartupProcessor] Error opening serial port: open /dev/hvc0: no such file or directory
2018-11-05 17:42:32 ERROR [StartupProcessor] Error opening serial port: open /dev/hvc0: no such file or directory. Retrying in 5 seconds...
2018-11-05 17:42:35 INFO [MessagingDeliveryService] [Association] No associations on boot. Requerying for associations after 30 seconds.
The first error that jumped out in logs is the “Unable to open serial port”. There is also an error referring to not being able to add the ssm-user to the sudoers file.
The fix for these issues is to add a Docker flag to the CoreOS serial device, “–device=/dev/ttyS0” and a volume mount to the sudoers path, “-v /etc/sudoers.d:/etc/sudoers.d”. The full Docker run command is shown below.
After fixing the errors found in the logs, and bringing up the containerized SSM agent, go ahead and create a new session in the AWS console.
The session should come up pretty much immediately and you should be able to run commands like you normally would.
The last thing to (optionally) do is run the agent as a systemd service to take advantage of some capabilities to start it up automatically if it dies or start it if the server gets rebooted. You can probably just get away with using the docker restart policy too if you aren’t interested in configuring a systemd service, which is what I have chosen to do for now.
You could even adapt this Docker image into a Kubernetes manifest and run it as a daemonset on each node of the cluster if desired to simplify things and add another layer of security. I may return to the systemd unit and/or Kubernetes manifest in the future if readers are interested.
Conclusion
The AWS Session manager is a fantastic tool for troubleshooting/debugging as well as auditing and security.
With SSM you can make sure to never expose specific servers to the internet directly, and you can also keep track of what kinds of commands have been run on the server. As a bonus, the AWS console helps keeps track of all the previous sessions that were created and if you hook up to Cloudwatch and/or S3 you can see all the commands and times that they were run with nice simple links to the log files.
SSM allows you to do a lot of other cool stuff like run scripts against either a subset of servers which can be filtered by tags or against all servers that are recognized by SSM. I’m sure there are some other features as well, I just haven’t found them yet.
There is an annoying behavior that you might run across in the Windows version of Vscode if you are using the Vsvim plugin when trying to use the default hotkey <ctrl+b> to toggle the sidebar open and closed. As far as I can tell, this glitch doesn’t exist in the OSX version of Vscode.
In vim, the shortcut for this toggle is actually used to scroll the page buffer up one screen. Obviously this makes behavior is the default for Vim but it also is annoying to not be able to open and close the sidebar. The solution is a simple hotkey remap for the keyboard shortcut in Vscode.
To do this, pull open the keyboard shortcuts by either navigating to File -> Preferences -> Keyboard shortcuts
Or by using the command pallet (ctrl + shift + p) and searching for “keyboard shortcut”.
Once the keyboard shortcuts have been pulled open just do a search for “sidebar” and you should see the default key binding.
Just click in the “Toggle Side Bar Visibility” box and remap the keybinding to Ctrl+Shift+B (since that doesn’t get used by anything by default in Vim). This menu is also really useful for just discovering all of the different keyboard shortcuts in Vscode. I’m not really a fan of going crazy and totally customizing hotkeys in Vscode just because it makes support much harder but in this case it was easy enough and works really well with my workflow.
Kubernetes is complicated, as you’ve probably already discovered if you’ve used Kubernetes before. Likewise, the Kubectl command line tool can pretty much do anything but can feel cumbersome, clunky and generally overwhelming for those that are new to the Kubernetes ecosystem. In this post I want to take some time to describe a few of the CLI tools that I have discovered that help ease the pain of working with and managing Kubernetes from the command line.
There are many more tools out there and the list keeps growing, so I will probably revisit this post in the future to add more cool stuff as the community continues to grow and evolve.
Where to find projects?
As a side note, there are a few places to check for tools and projects. The first is the CNCF Cloud Native Landscape. This site aims to keep track of all the various different projects in the Cloud/Kubernetes world. An entire post could be written about all of the features and and filters but at the highest level it is useful for exploring and discovering all the different evolving projects. Make sure to check out the filtering capabilities.
The other project I have found to be extremely useful for finding different projects is the awesome-kubernetes repo on Github. I found a number of tools mentioned in this post because of the awesome-kubernetes project. There is some overlap between the Cloud Native Landscape and awesome-kubernetes but they mostly compliment each other very nicely. For example, awesome-kubernetes has a lot more resources for working with Kubernetes and a lot of the smalller projects and utilities that haven’t made it into the Cloud Native Landscape. Definitely check this project out if you’re looking to explore more of the Kubernetes ecosystem.
Kubectl tricks
These are various little tidbits that I have found to help boost my productivity from the CLI.
Tab completion – The first thing you will probably want to get working when starting. There are just too many options to memorize and tab completion provides a nice way to look through all of the various commands when learning how Kubernetes works. To install (on OS X) run the following command.
brew install bash-completion
In zsh, adding the completion is as simple as running source <(kubectl completion bash). The same behavior can be accomplished in zsh using source <(kubectl completion zsh).
Aliases and shortcuts – One distinct flavor of Kubernetes is how cumbersome the CLI can be. If you use Zsh and something like oh-my-zsh, there is a default set of aliases that work pretty well, which you can find here. There are a many posts about aliases out there already so I won’t go into too much detail about them. I will say though that aliasing k to kubectl is one of the best time savers I have found so far. Just add the following snippet to your bash/zsh profile for maximum glory.
alias k=kubectl
kubectl –export – This is a nice hidden feature that basically allows users to switch Kubernetes from imperative (create) to declarative (apply). The --export flag will basically take an existing object and strip out unwanted/unneeded metadata like statuses and timestamps and present a clear version of what’s running, which can then be exported to a file and applied to the cluster. The biggest advantage of using declarative configs is the ability to mange and maintain them in git repos.
kubectl top – In newer versions, there is the top command, which gives a high level overview of CPU and memory utilization in the cluster. Utilization can be filtered at the node level as well as the pod level to give a very quick and dirty view into potential bottlenecks in the cluster. In older versions, Heapster needs to be installed for this functionaliity to work correctly, and in newer versions needs metrics-server to be running.
kubectl explain – This is a utility built in to Kubectl that basically provides a man page for what each Kubernetes resource does. It is a simple way to explore Kubernetes without leaving the terminal
This is an amazing little utility for quickly moving between Kubernetes contexts and namespaces. Once you start working with multiple different Kubernetes clusters, you notice how cumbersome it is to switch between environments and namespaces. Kubectx solves this problem by providing a quick and easy way to see what environments and namespaces a user is currently in and also quickly switch between them. I haven’t had any issues with this tool and it is quickly becoming one of my favorites.
Dealing with log output using Kubectl is a bit of a chore. Stern (and similarly kail) offer a much nicer user experience when dealing with logs. These tools allow users the ability to do things like show logs for multiple containers in pod, use regex matching to tail logs for specific containers, give nice colored output for distinguishing between logs, filter logs by namespaces and a bunch of other nice features.
Obviously for a full setup, using an aggregated/centralized logging solution with something like Fluenctd or Logstash would be more ideal, but for examining logs in a pinch, these tools do a great job and are among my favorites. As an added bonus, I don’t have to copy/paste log names any more.
yq is a nice little command line tool for parsing yaml files, which works in a similar way to the venerable jq. Parsing, reading, updating yaml can sometimes be tricky and this tool is a great and lightweight way to manipulate configurations. This tool is especially useful for things like CI/CD where a tag or version might change that is nested deep inside yaml.
There is also the lesser known jsonpath option that allows you to interact with the json version of a Kubernetes object, baked into kubectl. This feature is definitely less powerful than jq/yq but works well when you don’t want to overcomplicate things. Below you can see we can use it to quickly grab the name of an object.
kubectl get pods -o=jsonpath='{.items[0].metadata.name}'
Working with yaml and json for configuration in general seems to be an emerging pattern for almost all of the new projects. It is definitely worth learning a few tools like yq and jq to get better at parsing and manipulating data using these tools.
Similar to the above, ksonnet and jsonnet are basically templating tools for working with Kubernetes and json objects. These two tools work nicely for managing Kubernetes manifests and make a great fit for automating deployments, etc. with CI/CD.
ksonnet and jsonnet are gaining popularity because of their ease of use and simplicity compared to a tool like Helm, which also does templating but needs a system level permission pod running in the Kubernetes cluster. Jsonnet is all client side, which removes the added attack vector but still provides users with a lot of flexibility for creating and managing configs that a templating language provides.
More random Kubernetes tricks
Since 1.10, kubectl has the ability to port forward to resource name rather than just a pod. So instead of looking up pods that are running and connecting to one all the time, you can just grab the service name or deployment and just port forward to it.
port-forward TYPE/NAME [LOCAL_PORT:]REMOTE_PORT
k port-forward deployment/mydeployment 5000:6000
New in 1.11, which will be dropping soonish, there is a top level command called api-resource, which allows users to view and interact with API objects. This will be a nice troubleshooting tool to have if for example you are wanting to see what kinds of objects are in a namespace. The following command will show you these objects.
k api-resources --verbs=list --namespace -o name | xargs -n 1 kubectl get -o name -n foo
Another handy trick is the ability to grab a base64 string and decode it on the fly. This is useful when you are working with secrets and need to quickly look at what’s in the secret. You can adapt the following command to accomplish this (make sure you have jq installed).
k get secret my-secret --namespace default -o json | jq -r '.data | .["secret-field"]' | base64 --decode
Just replace .["secret-field"] to use your own field.
UPDATE: I just recently discovered a simple command line tool for decoding base64 on the fly called Kubernetes Secret Decode (ksd for short). This tool looks for base64 and renders it out for you automatically so you don’t have to worry about screwing around with jq and base64 to extract data out when you want to look at a secret.
k get secret my-secret --namespace default -o json | ksd
That command is much cleaner and easier to use. This utility is a Go app and there are binaries for it on the releases page, just download it and put it in your path and you are good to go.
Conclusion
The Kubernetes ecosystem is a vast world, and it only continues to grow and evolve. There are many more kubectl use cases and community to tools that I haven’t discovered yet. Feel free to let me know any other kubectl tricks you know of, and I will update them here.
I would love to grow this list over time as I get more acquainted with Kubernetes and its different tools.
