Category Archives: Command Line

Add environment variable file to fig

If you haven’t heard of fig and are using Docker, you need to check it out.  Basically Fig is a tool that allows users to quickly create development environments using Docker.  Fig alleviates the complexity and tediousness of having to manually bring containers up and down, stitch them together and basically orchestrate a Docker environment.  On top of this, Fig offers some other cool functionality, for example, the ability to scale up applications.  I am excited to see what happens with the project because it was recently merged in to the Docker project.  My guess is that there will be many new features and additions to either Docker if Fig gets rolled in to the Docker core project.  For now, you can check out Fig here if you haven’t heard of it and are interested in learning more.

One issue that I have run in to is that there is currently not a great way to handle a large number of environment variables in fig.  In Docker there is an option that allows a user to pass in an environment variable file with the –env file <filename> flag.  To do the same with Fig in its current form, you are forced to list out each individual environment variable in your configuration which can quickly become tedious and confusing.

There is a PR out for adding in the ability to pass an environment variable file in to fig via the env-file option in a fig.yml file.  This approach to me is much easier than adding each environmental variable separately to the configuration with the environment option as well as having to update the fig.yml configuration if any of the values ever change.  I know that functionality like this will get merged eventually but until then I have been using the PR as a workaround to solve this issue, I think that this is also a good opportunity to show people how to get a project working manually with custom changes.  Luckily the fix isn’t difficult.

This post will assume that you have git, python and pip installed.  If you don’t have these tools installed go ahead and get that done.  The first step is to clone the fig project on github on to your local machine, see above for the link to the PR.

git clone git@github.com:docker/fig.git

Jump in to the fig project you just cloned and edit the service.py file.  This is the file that handles the processing of environment variables.  There are a few sections that need to be updated in the config.  Check the PR to be sure, but at the time of the writing, the following code should be added.

Line 55

- supported_options = DOCKER_CONFIG_KEYS + ['build', 'expose']
+ supported_options = DOCKER_CONFIG_KEYS + ['build', 'expose', 'env-file']

Line 318

+ def _get_environment(self):
+ env = {}
+
+ if 'env-file' in self.options:
+ env = env_vars_from_file(self.options['env-file'])
+
+ if 'environment' in self.options:
+ if isinstance(self.options['environment'], list):
+ env.update(dict(split_env(e) for e in self.options['environment']))
+ else:
+ env.update(self.options['environment'])
+
+ return dict(resolve_env(k, v) for k, v in env.iteritems())
+

LIne 352

- if 'environment' in container_options:
- if isinstance(container_options['environment'], list):
- container_options['environment'] = dict(split_env(e) for e in container_options['environment'])
- container_options['environment'] = dict(resolve_env(k, v) for k, v in container_options['environment'].iteritems())
+ container_options['environment'] = self._get_environment()

Line 518

+
+
+def env_vars_from_file(filename):
+ """
+ Read in a line delimited file of environment variables.
+ """
+ env = {}
+ for line in open(filename, 'r'):
+ line = line.strip()
+ if line and not line.startswith('#') and '=' in line:
+ k, v = line.split('=', 1)
+ env[k] = v
+ return env

That should be it.  Now you should be able to install fig with the new changes.  Make sure you are in the root fig directory that contains the setup.py file.

sudo python setup.py develop

Now you should be able to edit your fig.yml file to reflect the changes that have been added to fig via env-file.  Here is what a sample configuration might look like.

testcontainer:
 image: username/testcontainer
 ports:
 - "8080:8080"
 links:
 - "mongodb:mongodb"
 env-file: "/home/username/test_vars"

Notice that nothing else changed.  But instead of having to list out environment variables one at a time we can simply read in a file.  I have found this to be very useful for my workflow, I hope others can either adapt this or use this as well.  I have a feeling this will get merged in to fig at some point but for now this workaround works.

Patching CVE-6271 (shellshock) with Chef

If you haven’t heard the news yet, a recently disclosed vulnerability has been released that exploits environmental variables in bash.  This has some far reaching implications because bash is so widespread and runs on many different types of devices, for example network gear, routers, switches, firewalls, etc.  If that doesn’t scare you then you probably don’t need to finish reading this article.  For more information you can check out this article that helped to break the story.

I have been seeing a lot “OMG the world is on fire, patch patch patch!” posts and sentiment surrounding this recently disclosed vulnerability, but basically have not seen anybody taking the time to explain how to patch and fix this issue.  It is not a difficult fix but it might not be obvious to the more casual user or those who do not have a sysadmin or security background.

Debian/Ubuntu:

Use the following commands to search through your installed packages for the correct package release.  You can check the Ubuntu USN for versions.

dpkg -l | grep '^ii' or
dpkg-query --show bash

If you are on Ubuntu 12.04 you will need update to the following version:

bash    4.2-2ubuntu2.3

If you are on Ubuntu 13.10, and have this package (or below), you are vulnerable.  Update to 14.04!

 bash 4.2-5ubuntu3

If you are on Ubuntu 14.04, be sure to update to the most recently patched patch.

bash 4.3-7ubuntu1.3

Luckily, the update process is pretty straight forward.

apt-get update
apt-get --only-upgrade install bash

If you have the luxury of managing your environment with some sort of automation or configuration management tool (get this in place if you don’t have it already!) then this process can be managed quite efficiently.

It is easy to check if a server that is being managed by Chef has the vulnerability by using knife search:

knife search node 'bash_shellshock_vulnerable:true'

From here you could createa a recipe to patch the servers or fix each one by hand.  Another cool trick is that you can blast out the update to Debian based servers with the following command:

knife ssh 'platform_family:debian' 'sudo apt-get update; sudo apt-get install -y bash'; knife ssh 'platform_family:redhat' 'sudo yum -y install bash'

This will iterate over every server in your Chef server environment that is in the Debain family (including Ubuntu) or RHEL family (including CentOS) and update the server packages so that the latest patched bash version gets pulled down and then gets updated to the latest version.

You may need to tweak the syntax a little, -x to override the ssh user and -i to feed an identity file.  This is so much faster than manually installing the update on all your servers or even fiddling around with a tool like Fabric, which is still better than nothing.

One caveat to note:  If you are not on an LTS version of Ubuntu, you will need to upgrade your server(s) first to an LTS, either 12.04 or 14.04 to qualify for this patch.  Ubuntu 13.10 went out of support in August which was about a month ago as per the time of this writing so you will want to get your OS up date.

One more thing:  The early patches to address this vulnerability did not entirely fix the issue, so make sure that you have the correct patch installed.  If you patched right away there is a good chance you may still be vulnerable, so simply rerun your knife ssh command to reapply the newest patch, now that the dust is beginning to settle.

Outside of this vulnerability, it is a good idea to get your OS on an Ubuntu LTS version anyway to continue receiving critical updates for software as well as security patches for a longer duration than the normal, 6 month release cycle of the server distribution.

Test Kitchen Tricks

test kitchen

I have been working a lot with Chef and Test kitchen lately and thus have learned a few interesting tricks when running tests with these tools.  Test Kitchen is one of my favorite tools when working with Chef configuration management because it is very easy to use and has a number of powerful features that make testing things in Chef simple and easy.

Test Kitchen itself sits on top of Vagrant and Virtualbox by default so to get started with the most basic usage example of Test Kitchen you will need to have Vagrant installed along with a few other items.

Then to install Test Kitchen.

gem install test-kitchen

That’s pretty much it.  The official docs have some pretty detailed usage and in fact I have learned many of the tricks that I will be writing about today from the docs.

Once you are comfortable with Test Kitchen you can begin leveraging some of the more powerful features, which is what the remainder of this post will cover.  There is a great talk given by the creator of test kitchen at this year’s Chefconf by the creator of Test Kitchen about some of the lessons learned and cool things that you can do with the tool.  If you haven’t already seen it, it is worth a watch.

Anyway, let’s get started.

1) Fuzzy matching

This one is great for the lazy people out there.  It basically allows you match a certain unique part of a command instead of typing out an entire command.  So for example, you can just type in a partial name for a command to return the desired full command.  Since Test Kitchen uses regular expression matching, this can be a very powerful feature.

2) Custom drivers

One reason that Test Kitchen is so flexible is because it can leverage many different plugins and drivers.  And, since it is open source, if there is functionality missing from a driver you can simply write your own.  Currently there is an awesome list of drivers available for Test Kitchen to use, and a wide variety of options available to hopefully suit most testing scenarios.

Of course, there are others as well.  These just happen to be the drivers that I have tried and can verify.  There is even support for alternate configuration management tool testing, which can be handy for those that are not using Chef specifically.  For example there is a salt driver available.

3) .kitchen.local.yml

This is a nice handy little bit that is often overlooked but allows a nice amount of control by overriding the default .kitchen.yml configuration file with specific options.  So for example, if you are using the ec2 driver in your configuration but need to test locally with Vagrant you can simply drop a .kitchen.local.yml on your dev machine and override the driver (and any other settings you might need to change). I have created the following .kitchen.local.yml for testing on a local Vagrant box using 32 bit Ubuntu to highlight the override capabilities of Test Kitchen.

driver: 
 name: vagrant 
 
platforms: 
 - name: ubuntu-1310-i386 
 - name: ubuntu-1404-i386

4) Kitchen diagnose

An awesome tool for diagnosing issues with Test Kitchen.  Running the diagnose will give you lots of juicy info about what your test machines are doing (or should be doing) and a ton of configuration information about them.  Basically, if something is misbehaving this is the first place you should look for clues.

If you want to blast info and settings for all your configurations, just run the following,

kitchen diagnose

5) Concurrency

If you have a large number of systems that need to have tests run on them then running your Test Kitchen tests in parallel is a great way to speed up your total testing time.  Turning on concurrency is pretty straight forward, just add the “-c” flag and the number of instances to run on (the default is 9999).

kitchen converge -c 5

6) Verbose logging

This one can be helpful if your kitchen run is failing with no real clues or helpful information provided by the diagnose command.  It seems obvious but getting this one to work gave me some trouble initially.  To turn on verbosity simply add the debug flag to your test kitchen command.

So for example, if you want to converge a node with verbosity turned on, you would use this command.

kitchen converge -l debug

I recommend taking a look at some or all of these tricks to help improve your integration testing with Test Kitchen.  Of course as I stated, all of this is pretty well documented.  Even if you are already familiar with this tool, sometimes it just helps to have a refresher to remind you of a great tool and to jar your memory.  Let me know if you have any other handy tricks and I will be sure to post them here.

Transitioning from bash to zsh

oh-my-zsh

I have know about zsh for a long time now but have never really had a compelling reason to switch my default shell from bash until just recently, I have been hearing more and more people talking about how powerful and awesome zsh is.  So I thought I might as well take the dive and get started since that’s what all the cool kids seem to be doing these days.  At first I thought that changing my shell was going to be a PITA with all the customizations and idiosyncrasies that I have grown accustomed to using bash but I didn’t find that to be the case at all when switching to zsh.

First and foremost, I used a tool called oh-my-zsh to help with the transition.  If you haven’t heard about it yet, oh-my-zsh aims to be a sort of framework for zsh.  This project is a nice clean way to get started with zsh because it give you a nice set of defaults out of the the box without having to do much configuration or tweaking and I found that many of my little tricks and shortcuts were already baked in to to oh-my-zsh, along with a ton of other settings and customizations that I did not have using bash.

From their github page:

oh-my-zsh is an open source, community-driven framework for managing your ZSH configuration. It comes bundled with a ton of helpful functions, helpers, plugins, themes, and few things that make you shout…

Here are just a few of the improvement that zsh/oh-my-zsh offer:

  • Improved tab completion
  • persistent history across all shells
  • Easy to use plugin system
  • Easy to use theme system
  • Autocorrect

The most obvious difference that I have noticed is the improved, out of the box tab completion, which I think should be enough on its own to convince you!  On top of that, most of my tricks and customizations were already turned on with oh-my-zsh.  Another nice touch is that themes and plugins come along as part of the package, which is really nice for easing the transition.

So after spending an afternoon with zsh I am convinced that it is the way to go (at least for my own workfolw).  Of course there are always caveats and hiccups along the way as I’ve learned there are with pretty much everything.

Tuning up tmux

Out of the box, my tmux config uses the default shell, which happens to be bash.  So I needed to modify my ~/.tmux.conf to reflect the switch over the zsh.  It is a pretty straight forward change but is something that you will need to make note of kif you use tmux and are transitioning in to using zsh.

set-option -g default-shell /usr/bin/zsh

I am using Ubuntu 14.04, so my zsh is installed to /usr/bin/zsh.  The other thing that you will need to do is make sure you kill any stale tmux processes after updating to zsh.  I found one running in the background that was blocking me from using the new coonfig.

Goodies

There is a nice command cheat sheet for zsh.  Take some time to learn these shortcuts and aliases, they are great time savers and are very usefull.

oh-my-zsh comes bundled up with a large number of goodies.  At the time of this writing there were 135 plugins as well as a variety of themes.  You can check the plugins wiki page for descriptions for the various plugins.  To turn on a specific plugin you will need to add it to your ~/.zshrc config file.  Find the following line in your config.

plugins=(git)

and add plugins separated by spaces

plugins (git vagrant chef)

You will need to reload the config for the changes to be picked up.

source ~/.zshrc

Most themes are hosted on the wiki, but there is also a web site dedicated to displaying the various themes, which is really cool.  It does a much better job of showing differences between various themes.  You can check it out here.  Themes function in a similar way to plugins.  If you want to change your theme, edit your ~/.zshrc file and select the desired them.

ZSH_THEME="clean"

You will need to reload your config for this option as well.

source ~/.zshrc

Conclusion

If you haven’t already made the switch to zsh I recommend that you at least experiment and play around with it before you make any final decisions.  You may be set in your ways and happy with bash or any other shell that you are used to but for me, all the awesomeness changed my opinion and decide to reevaluate my biases.  If you’re worried about making the switchin, using oh-my-zsh makes the transition so painless there is practically no reason not to try it out.

This post is really just the tip of the iceberg for the capabilities of this shell, I just wanted to expose readers to all of its glory.  Zsh offers so much more power and customization than I have covered in this post and is an amazing productivity tool with little learning overhead.

Let me know if you have any awesome zsh tips or tweaks that folks should know about.

7 useful but hard to remember Linux commands

I have found myself using these commands over and over so I decided I’d take the time to go ahead and document them for future me as well as readers because I find these commands pretty useful.  I just always manage to forget them, hence the title of the post.  The smart thing to do would be to create aliases for these commands but I have just been too lazy and some of them are run across different servers so it isn’t always a convenient option.

Anyway, let’s go ahead and run through the commands before I forget…

1) du -ah / | sort -n -r | head -n 50

This one is really handy for debugging space issues.  It will list the top 50 files according to file size, with the largest at the top of the list.Notice the “/” will specify the location to search so you can easily modify this one to search different locations, like “/var/log” for example if you are having trouble with growing log files.

1.5) du -sh /*

This will quickly give you an idea of how how your disk space has been allocated.  Definitely handy when you are troubleshooting.

2) git checkout — .

I don’t use this one very often, which is probably why I manage to forget it so easily.  But I really like it.  Sometimes I will be working on a git repo across different machines at the same time and will run in to conflicts committing to the repo or more likely I committed changes on one machine and just need to pull down the newest changes but can’t since I have made modifications.  For those scenarios you can run the above command quickly reset your git changes quickly and easily.

3) tmux kill-window -t 3

i use tmux for my terminal and window manager on all my workstations and love it.  If you haven’t heard of it, take a look here.  Sometimes the sessions can get stuck so it becomes necessary to close the window without destroying the tmux session.  Again, this doesn’t happen very often so it is sometimes hard for me to remember the exact syntax but this one is a handy little trick for managing tmux windows and sessions.

4) grep -r “text”

I know, I should really have this one memorized by now.  I am trying to remember but I don’t find myself using this one all that often even though it is really powerful and useful.  This will essentially search through every file recursively and spit out the text pattern that you feed to it.

5) kill $(pgrep process)

This one is handy when there are a large number of stuck processes and you need to blow them all out with one command.  For example if the chrome browser ever gets stuck with a million tabs open, there are likely a large number of processes all with the same – or similar names.  If you pass all or part of the process name in to this command pgrep will find them and kill will destroy them

6) docker rm $(docker ps -a -q)

I have been using Docker more and more recently and every once in awhile I find myself with a large number of dead Docker processes that need to be cleaned up.  This command will blow out all of these stale processes at once.  This is nice because Docker processes take up a large amount of disk space and often times can fill up your drives without you being aware.  I have been able to reclaim large amounts of disk space with this command.

7) watch -n 10 df -ah

This is another good one for checking disk space issues.  It will update you every ten seconds with the disk utilization of the system.  Pretty straight forward but a great tool to help troubleshooting space issues.

That’s all I have for now, there are lots more but these are the most useful ones that I find myself forgetting the most often, hopefully this post will serve as a nice reminder.  If you have any cool or useful commands that you would like to share feel free to comment and I will update the post to include them.