Mount an external EBS volume in AWS

Creating and attaching external volumes is one of those things in administration that is really nice to know how to do but for me is also one that doesn’t happen every day so it is really easy to forget how to do which makes it a little bit more painful, especially with deadlines and people watching over your shoulder.  So, having said that, I think it is probably worth writing a post about how to do it because it happens just enough that I have trouble getting everything straight, and I’m sure otheres run in to this as well, so that’s what I will be writing about today.

There is  good documentation for how to do this but there are a lot of separate steps so consolidating the components might be helpful to readers who stumble across this.  I’m sure there are other ways to accomplish this but I don’t think it is necessary to cover everything here.

Create your “floating volume”

This step is straight forward.  In the AWS EC2 console choose the type of volume this will be (SSD or magnetic), availability zone, and any other options here.

create ebs volume

After your volume has been created you will want to attach it to an instance.  This part is important because the changes could break your OS volume if you write to your fstab file incorrectly.  In this example I am choosing to attach the EBS volume as /dev/xvdf, but you could name it differently if it corresponds to your setup.

attach ebs volume

After the volume has been mounted you can check that it has been picked up by the OS by either checking the /dev directory or by running fdisk -l and looking for the size of the disk you just attached.

It is worth pointing out that all of the steps in the AWS console can alternatively be done with the aws-cli tool.  It is probably easier but for the sake of time and illustration I am leaving those steps out.  Feel free to reach out if you are interested in the cli tool and I can update this post.

If you run fdisk -l you will notice that the device is empty, so you will need to format the disk.  In this instance I am formatting the disk as ext4.  So use the following command to format it.

sudo mkfs.ext4 /dev/xvdf

After the volume has been formatted you can mount it to your OS.

Attaching the volume

sudo mkdir /data
sudo mount -t ext4 /dev/xvdf /data

If you need to resize the filesystem for whatever reason, you can use the resize2fs command.

sudo resize2fs <mount point>
sudo resize2fs /dev/xvdf

Here you will create the directory (if it doesn’t already exist) to mount the volume to and then mount it.  At this point it would be fine to be done if you just needed temporary access to the storage on this device.  But if you want your mount to persist and to survive a reboot then you just add an entry to your /etc/fstab file to make sure the /data directory gets the volume mounted to it after a reboot.  Something like the following would work.

/dev/xvdf       /data   ext4    defaults,nobootwait        0       0

The entry is pretty easy to follow but may be confusing for those who are not familiar with how fstab works.  I will break down the various components here.

The first parameter is the location of the volume (/dev/xvdf) and is referred to as the file_system field.

The second parameter specifies where to mount the volume to (/data) and is referred to as the dir field.

The third field is the type.  This is where you specify the file system type or device to be mounted.  If you didn’t format this volume previously it would crate problems for OS when it tried to load in your volume from this file.

The fourth section is the options for the mount.  Here, the defaults,nobootwait section is very important.  If you don’t have the nobootwait option specified here then your OS could potentially hang on boot up if it couldn’t find the specified volume, so this option helps escape it if there are any problems.

The fifth field is to either enable or disable the dump option.  Unless you are familiar with or use the dump command you will almost always set this to 0.

The last section is the pass section.  This simply tells the OS if it should run an fsck or not on this volume.  Here I have it set to 0 so it doesn’t get checked but for OS volumes, this could be important to turn on.

Next steps

There are many more things you can do with fstab so if you are interested in other options for how to mount volumes you can look at the fstab documentation for more insights and information.

If you ever wanted to float this volume to another host it would be easy to do, and would not require any new or special formatting since this was already taken care of here.  The steps would looks similar to the following.

  • Unmount volume from current OS
  • Remove entry in /etc/fstab for volume mount
  • Detach mount in AWS console from current OS
  • Attach mount to new OS
  • Mount volume manualy in new OS to test if it works
  • If the mount works add a new entry in /etc/fstab
  • Done

So that’s pretty much it.  Hopefully this is useful for everybody.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Kubernetes resize and rolling updates

If you haven’t heard of or used Kubernetes yet, I highly recommend taking a look (see the link below).  I won’t take too much time here today to talk about the Kubernetes project because there is just too much to cover.  Instead I will be writing a series of posts about how to work with Kubernetes and share some tricks and tips that I have discovered in my experiences so far with the tool.  Since the project is still very young and moving incredibly quickly, the best place to get information is either the IRC channel (#google-containers), the mailing list, or their github project.  Please go look at the github project if you are new to Kubernetes, or are interested in learning more about it, especially their docs and examples sections.

As I said, updates and progress have been extremely fast paced, so it isn’t uncommon for things in the Kubernetes project to seem obselete before they have even been implemented.  For example, the command line tool for interacting with a Kubernetes cluster has already changed faces a few times, which was confusing to me when I first started out.  Kubecfg is on the way out and the project maintainers are working on removing old references to it.  On the flip side, the kubectl command is maturing quite nicely and will be around for awhile, along with the subcommand that I will be describing.

Now that I have all the basic background stuff out of the way; the version of kubectl I am using for this demonstration is v0.9.1.  If you just discovered Kubernetes or have been using kubecfg (as explained above) you will want to make sure to get more familiar with kubectl because it is the preferred tool going forward, at least at this point.

There are a few handy subcommands that come baked in to the kubectl command.  The first is the resize command.  This command allows you to scale the number of running containers being managed by Kubernetes up or down on the fly.  Obviously this can be really powerful!  The syntax is pretty straight forward and below I have an example listed.

kubectl resize –current-replicas=6 –replicas=0 rc test-rc

The –current-replicas argument is optional, the –replicas defines the *desired* number of replicas to have runing, rc specifies this is a replication controller, and finally, test-rc is the name of the replication controller to scale.   After you scale your replication controller you can check out the status quickly via the following command.

kubectl get pod

Another handy tool to have when working with Kubernetes is the ability to deploy new images as a rolling update.

 kubectl rollingupdate test-rc -f test-rc-2.yml –update-period=”10s”

The rollingupdate command takes a few arguments.  The first is the name of the current replication controller that you would like to update.  The second is to replace it with the yml file of the new replication controller and the third optional argument is the –update-period, which allows a user to override the default time that it takes to spin up a new container and spin down an old.

Below is an example of what your test-rc-2.yml file may look like.

kind: ReplicationController
apiVersion: v1beta1
id: test-rc-2
namespace: default
 replicas: 1
   name: test-rc
   version: v2
   name: test-rc
   version: v2
 version: v1beta1
 id: test-rc
   - name: test-image
   image: test/test:new-tag
   imagePullPolicy: PullAlways
   - name: test-port
   containerPort: 8080

There are a few important things to notice.  The first is that the id must be unique, it can’t be a name that is already in use by another replication controller.  All of the label names should remain the same except for the version.  The version is used to signify the new replication controller is a running a new docker image.  The version number should be unique, which will help keep track of which image version is running.

Another thing to note.  If your original replication controller did not contain a unique key (like version) then you will need to update the original replication controller first, adding a unique key, before attempting to run the rolling update.

If both replication controllers don’t have the same format you will get an error similar to this.

test-rc.yml must specify a matching key with non-equal value in Selector for <selector name>

So that’s pretty much it for now.  I will revisit this post again in the future as new flags and subcommands are added to kubectl for managing and updating replication controllers.  I also plan on writing a few more posts about other aspects and areas of kubectl and running Kubernetes, so please check back soon!

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Setting up a pfSense NAT instance in AWS

One important aspect of cloud deployments that often get overlooked, especially at start ups is the aspect of security.  So I thought I would take some time to go through the process of setting up a NAT instance on AWS with full firewall capabilities.  There are instructions and documentation for this process which are very good but aren’t completely clear so I will attempt to fill in some of the gaps I ran in to when attempting to set this up myself.

There is one thing to take note of if you have used pfSense before.  This firewall isn’t free.  There is a slight hourly charge for this that ends up coming out to about $500/yr (which comes out to about $42/month).  If you look at other commercial solutions with similar functionality you are looking at thousands of dollars per month in costs.  Long story short, the cloud images of pfSense has a tiny tiny cost associated with it but is very much worth it.

Just for reference I put together a few comparison prices.

  • Barracuda web app firewall – ($1.04-1.76/hr) (up to ~$1300/month)
  • Vyatta  ($0.30-1.50/hr) (up to ~$1100/month)
  • Sophos UTM ($0.35-$2.80/hr) (up to ~$2000/month)
  • pfSense ($0.07/hr) ($42/month)

As you can see, pfSense is very reasonable compared to some of the other bigger players.  You can build an r3.8xlarge instance and the software price won’t change which doesn’t seem to be the case with others.  One bonus to choosing pfSense is that you automatically qualify for support by agreeing to the ToS when getting the pfSense AMI set up.

Finally, pfSense is rock solid being built on top BSD and is thoroughly tested.  I have been running pfSense on other projects outside of AWS for 5+ years and have never had an issue with it outside of a dead hard drive one time.  Other added benefits of choosing pfSense are that updates are frequent and thoroughly tested, tons of add-ons including IPS’s and VPN’s so additional functionality can be built on top and great community support as well.

Getting started

There are a few good resources that I found to be useful when working through this problem, which got me most of the way to a working setup.  They are listed below.

And here is the link to my question about how to do this on serverfault, there is some good detail in the post over there.

Setting up the NAT in pfSense

The first issue that was confusing was the issue of getting the network interfaces set up and configured.  For this setup you will need two interfaces, preferably with static IP addresses.  You will also need to make sure that you disable source/destination checks for the interface that will be acting as the LAN interface that the nat goes through.  Disabling source and destination checks is pretty straightforward and is detailed in pretty much all of the guides.

You should note that there will be tabs for firewalling for LAN as well as WAN, if you can keep these two straight it should be much easier to troubleshoot and configure your pfSense machine.  Out of the box, the firewall on pfSense will not be configured to allow your LAN interface to do any sort of NATing, you will need to manually create rules to get started.  If you check the WAN firewall tab you should notice some access rules but the LAN tab should be empty.  Most of the work we will be doing will be on the LAN firewall.

The first rule to set up to make things easier to troubleshoot is a ping rule.  There is a WAN rule for ping but not for LAN.  You can essentially copy the WAN rule into a new one and modify it to look similar to the following.

LAN firewall rule














This rule will work for the template for the other rules that need to be put in to place.  The other rules will be for outbound web access.   Just copy this rule in to a new rule and change the protcol to TCP and make one rule that allows port 80 and another that allows 443.  The resulting should look similar to what I have listed below.

firewall rules for nat

Just a quick note.  If at any point you are having trouble seeing traffic or are getting stuck in your troubleshooting, an excellent way to figure out what is going on is the logging that is provided by pfSense.  You can access all of the various logs to see what is happening by selecting Status -> System Logs and the highlighting the firewall tab.

Modifying your outbound nat

Here is what your outbound NAT rule should look like.

outbound nat














Notice the “Networks_to_NAT” value in the source section.  This is a pfSense alias that can be used as a sort of variable to help ease management.  You can either use this alias or specify the local subnet you want to use here.  To check the values in your alias you can go to Firewall -> Aliases.


This setup will provide you with a nice easy way to manage your network in AWS.  The guides for setting up a NAT are nice and are a good first step but with a Firewall in place you can do so many other things, especially auditing that just aren’t available or viable with a straight AWS nat instance or that are way out of your price range with some of the other commercial solutions available.

pfSense also provides the capability to add more advanced tools like IDS/IPS, VPN and high availability if you choose so there is nice room for expansion.  Even if you don’t take advantage of all of the additional components of pfSense you will still have a rock solid firewall and nat instance that is suitable for production workloads at a fraction of the cost of other commercial solutions.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

CLI hotkey and navigation guide

I have been meaning to write this post for quite a while now but have always managed to forget.  I have been piecing together useful terminal shortcuts, commands and productivity tools since I started using Linux back in the day.  If you spend any amount of time in the terminal you should hopefully know about some of these tricks already but more importantly, if you’re like me, are always looking for ways to improve the efficiency of your bash workflow and making your life easier.

There are a few things that I would quickly like to note.  If you use tmux as your CLI session manager you may not be able to use some of the mentioned hotkeys to get around by default if you don’t have some settings turned on in your configuration file.

You can take a look at my custom .tmux.conf file if you’re interested in screen style bindings plus configuration for hotkeys.  If you simply want to add the option that turns on the correct hotkey bindings for your terminal, add this line to your ~/.tmux.conf file

set-window-option -g xterm-keys on

Also, if you are a Mac user, and don’t already know about it, I highly recommend checking out iTerm2.  Coming primarily from a Linux background the hotkey bindings in Mac OS X are a little bit different than what I am used to and were initially a challenge for me to get accustomed to.  The transition for me took a little bit but iTerm has definitely helped me out immensely, as well as a few other ticks learned along the way.  I really haven’t dug through all the options in iTerm but there are a huge number of options and customizations that can made.

The only thing I have been interested in so far is the navigation which I will highlight below.

Adjust iTerm keybindings – As I mentioned, I am used to using Linux keybinding so a natural fit for my purposes is the option key.  The first step is to disable the custom binding in the iTerm preferences.  To do this, click iTerm -> Preferences -> Profiles -> Keys and find the binding for option left arrow and option right arrow and remove them from the default profile.

Next, add the following to your global key bindings, iTerm -> Preferences -> Keys.





Move left one word

  • Keyboard shortcut: ??
  • Action: Send Escape Sequence
  • Escape: b

Move right one word

  • Keyboard shortcut: ??
  • Action: Send Escape Sequence
  • Escape: f

Finally, it is also worth pointing out that I use zsh for my default shell.  There are some really nice additions that zsh offers over vanilla bash.  I recently ran across this blog post which has some awesome tips.  I have also written about switching to zsh here.  Anyway, here is the lis.  It will grow as I find more tips.

Basic navigation:

  • Ctrl-left/right arrow – Jump between words quickly.
  • Opt-left/right arrow – Custom iTerm binding for jumping between words quickly.
  • Alt-left/right arrow – Linux only.  Jump between words quickly.
  • Esc-b/f – Mac OS.  Similar to arrow keys, move between words quickly.
  • Alt-b – Linux only.  Jump back one word.  Handy with other hotkeys overridden.
  • Ctrl-a – Jump to the beginning of a line (doesn’t work with tmux mappings).
  • Ctrl-e – Jump to the end of a line.
  • End – SImilar to ctrl-e this will send your cursor to the end of the line.
  • Home – Similar to End, except jumps to the beginning of the line.

Intermediate navigation:

  • Ctrl-u – Copy entire command to clipboard.
  • Ctrl-y – Paste previously copied ctrl-u command in to the terminal.
  • Ctrl-w – Cut a word to the left of the cursor.
  • Alt-d – Cut after word after the cursor position

Advanced use:

  • Ctrl-x Ctrl-e – Zsh command.  Edit the current command in your $EDITOR, which for me is vim
  • Ctrl-r – Everybody hopefully should know this one.  It is basically recursive search history
  • Ctrl-k – Erase everything after the current cursor position.  Handy for long commands
  • !<command>:p – Print the last command
  • cd … – Zsh command.  This can be easily aliased but will jump up two directories
  • !$ – Quickly access the last argument of the last command.

Zsh tab completion

Tab completion with Zsh is awesome, it’s like bash completion on steroids.  I will attempt to highlight some of my favorite tab completion tricks with Zsh.

Directory shorthand – Say you need to get to a directory that is nested deeply.  You can use the first few characters that will uniquely match to that directory to navigate instead of typing out the whole command.  So for example, cd /u/lo/b will expand out to /usr/local/bin.

command specific history – This one comes in handy all the time.  If you need to grab a command that you don’t use very often you can user Ctrl-r to match the first part of the command and from there you can up arrow to locate the command you typed.

Spelling and case correction – Bash by default can get annoying if you have a long command typed out but somehow managed to typo part of the command.  In zsh this is (sometimes) corrected for you automatically when you <tab> to complete the command.  For example if you are changing dirs in to the ‘Documents’ directory you can type ‘cd ~/doc/’ and the correct location will be expanded for you.

This list will continue to grow as I find more handy shortcuts, hotkeys or generally other useful tips and tricks that I find in my day to day command line work.  I really want to build a similar list for things in Vim but my Vim skills are unfortunately lacking plus there is already some really nice documentation and guidance out there already.  If you are interested in writing up a Vim productivity post I would love to post it.  Likewise, if you have any other nice shortcuts or tips you think are worth mentioning, post them in the comments and I will try to get them added to the list.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Add environment variable file to fig

If you haven’t heard of fig and are using Docker, you need to check it out.  Basically Fig is a tool that allows users to quickly create development environments using Docker.  Fig alleviates the complexity and tediousness of having to manually bring containers up and down, stitch them together and basically orchestrate a Docker environment.  On top of this, Fig offers some other cool functionality, for example, the ability to scale up applications.  I am excited to see what happens with the project because it was recently merged in to the Docker project.  My guess is that there will be many new features and additions to either Docker if Fig gets rolled in to the Docker core project.  For now, you can check out Fig here if you haven’t heard of it and are interested in learning more.

One issue that I have run in to is that there is currently not a great way to handle a large number of environment variables in fig.  In Docker there is an option that allows a user to pass in an environment variable file with the –env file <filename> flag.  To do the same with Fig in its current form, you are forced to list out each individual environment variable in your configuration which can quickly become tedious and confusing.

There is a PR out for adding in the ability to pass an environment variable file in to fig via the env-file option in a fig.yml file.  This approach to me is much easier than adding each environmental variable separately to the configuration with the environment option as well as having to update the fig.yml configuration if any of the values ever change.  I know that functionality like this will get merged eventually but until then I have been using the PR as a workaround to solve this issue, I think that this is also a good opportunity to show people how to get a project working manually with custom changes.  Luckily the fix isn’t difficult.

This post will assume that you have git, python and pip installed.  If you don’t have these tools installed go ahead and get that done.  The first step is to clone the fig project on github on to your local machine, see above for the link to the PR.

git clone

Jump in to the fig project you just cloned and edit the file.  This is the file that handles the processing of environment variables.  There are a few sections that need to be updated in the config.  Check the PR to be sure, but at the time of the writing, the following code should be added.

Line 55

- supported_options = DOCKER_CONFIG_KEYS + ['build', 'expose']
+ supported_options = DOCKER_CONFIG_KEYS + ['build', 'expose', 'env-file']

Line 318

+ def _get_environment(self):
+ env = {}
+ if 'env-file' in self.options:
+ env = env_vars_from_file(self.options['env-file'])
+ if 'environment' in self.options:
+ if isinstance(self.options['environment'], list):
+ env.update(dict(split_env(e) for e in self.options['environment']))
+ else:
+ env.update(self.options['environment'])
+ return dict(resolve_env(k, v) for k, v in env.iteritems())

LIne 352

- if 'environment' in container_options:
- if isinstance(container_options['environment'], list):
- container_options['environment'] = dict(split_env(e) for e in container_options['environment'])
- container_options['environment'] = dict(resolve_env(k, v) for k, v in container_options['environment'].iteritems())
+ container_options['environment'] = self._get_environment()

Line 518

+def env_vars_from_file(filename):
+ """
+ Read in a line delimited file of environment variables.
+ """
+ env = {}
+ for line in open(filename, 'r'):
+ line = line.strip()
+ if line and not line.startswith('#') and '=' in line:
+ k, v = line.split('=', 1)
+ env[k] = v
+ return env

That should be it.  Now you should be able to install fig with the new changes.  Make sure you are in the root fig directory that contains the file.

sudo python develop

Now you should be able to edit your fig.yml file to reflect the changes that have been added to fig via env-file.  Here is what a sample configuration might look like.

 image: username/testcontainer
 - "8080:8080"
 - "mongodb:mongodb"
 env-file: "/home/username/test_vars"

Notice that nothing else changed.  But instead of having to list out environment variables one at a time we can simply read in a file.  I have found this to be very useful for my workflow, I hope others can either adapt this or use this as well.  I have a feeling this will get merged in to fig at some point but for now this workaround works.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.