Mount an external EBS volume in AWS

February 25, 2015June 29, 2020 Josh Reichardt 10 Comments

Creating and attaching external volumes is one of those things in administration that is really nice to know how to do but for me is also one that doesn’t happen every day so it is really easy to forget how to do which makes it a little bit more painful, especially with deadlines and people watching over your shoulder. So, having said that, I think it is probably worth writing a post about how to do it because it happens just enough that I have trouble getting everything straight, and I’m sure others run in to this as well, so that’s what I will be writing about today.

There is good documentation for how to do this but there are a lot of separate steps so consolidating the components might be helpful to readers who stumble across this. I’m sure there are other ways to accomplish this but I don’t think it is necessary to cover everything here.

Create your “floating volume”

This step is straight forward. In the AWS EC2 console choose the type of volume this will be (SSD or magnetic), availability zone, and any other options here.

After your volume has been created you will want to attach it to an instance. This part is important because the changes could break your OS volume if you write to your fstab file incorrectly. In this example I am choosing to attach the EBS volume as /dev/xvdf, but you could name it differently if it corresponds to your setup.

After the volume has been mounted you can check that it has been picked up by the OS by either checking the /dev directory or by running fdisk -l and looking for the size of the disk you just attached.

It is worth pointing out that all of the steps in the AWS console can alternatively be done with the aws-cli tool. It is probably easier but for the sake of time and illustration I am leaving those steps out. Feel free to reach out if you are interested in the cli tool and I can update this post.

If you run fdisk -l you will notice that the device is empty, so you will need to format the disk. In this instance I am formatting the disk as ext4. So use the following command to format it.

sudo mkfs.ext4 /dev/xvdf

After the volume has been formatted you can mount it to your OS.

Attaching the volume

sudo mkdir /data
sudo mount -t ext4 /dev/xvdf /data

If you need to resize the filesystem for whatever reason, you can use the resize2fs command.

sudo resize2fs <mount point>
sudo resize2fs /dev/xvdf

Here you will create the directory (if it doesn’t already exist) to mount the volume to and then mount it. At this point it would be fine to be done if you just needed temporary access to the storage on this device. But if you want your mount to persist and to survive a reboot then you just add an entry to your /etc/fstab file to make sure the /data directory gets the volume mounted to it after a reboot. Something like the following would work.

/dev/xvdf       /data   ext4    defaults,nobootwait        0       0

The entry is pretty easy to follow but may be confusing for those who are not familiar with how fstab works. I will break down the various components here.

The first parameter is the location of the volume (/dev/xvdf) and is referred to as the file_system field.

The second parameter specifies where to mount the volume to (/data) and is referred to as the dir field.

The third field is the type. This is where you specify the file system type or device to be mounted. If you didn’t format this volume previously it would crate problems for OS when it tried to load in your volume from this file.

The fourth section is the options for the mount. Here, the defaults,nobootwait section is very important. If you don’t have the nobootwait option specified here then your OS could potentially hang on boot up if it couldn’t find the specified volume, so this option helps escape it if there are any problems.

The fifth field is to either enable or disable the dump option. Unless you are familiar with or use the dump command you will almost always set this to 0.

The last section is the pass section. This simply tells the OS if it should run an fsck or not on this volume. Here I have it set to 0 so it doesn’t get checked but for OS volumes, this could be important to turn on.

Next steps

There are many more things you can do with fstab so if you are interested in other options for how to mount volumes you can look at the fstab documentation for more insights and information.

If you ever wanted to float this volume to another host it would be easy to do, and would not require any new or special formatting since this was already taken care of here. The steps would looks similar to the following.

Unmount volume from current OS
Remove entry in /etc/fstab for volume mount
Detach mount in AWS console from current OS
Attach mount to new OS
Mount volume manually in new OS to test if it works
If the mount works add a new entry in /etc/fstab
Done

So that’s pretty much it. Hopefully this is useful for everybody.

Kubernetes resize and rolling updates

February 6, 2015August 31, 2015 Josh Reichardt 1 Comment

If you haven’t heard of or used Kubernetes yet, I highly recommend taking a look (see the link below). I won’t take too much time here today to talk about the Kubernetes project because there is just too much to cover. Instead I will be writing a series of posts about how to work with Kubernetes and share some tricks and tips that I have discovered in my experiences so far with the tool. Since the project is still very young and moving incredibly quickly, the best place to get information is either the IRC channel (#google-containers), the mailing list, or their github project. Please go look at the github project if you are new to Kubernetes, or are interested in learning more about it, especially their docs and examples sections.

As I said, updates and progress have been extremely fast paced, so it isn’t uncommon for things in the Kubernetes project to seem obselete before they have even been implemented. For example, the command line tool for interacting with a Kubernetes cluster has already changed faces a few times, which was confusing to me when I first started out. Kubecfg is on the way out and the project maintainers are working on removing old references to it. On the flip side, the kubectl command is maturing quite nicely and will be around for awhile, along with the subcommand that I will be describing.

Now that I have all the basic background stuff out of the way; the version of kubectl I am using for this demonstration is v0.9.1. If you just discovered Kubernetes or have been using kubecfg (as explained above) you will want to make sure to get more familiar with kubectl because it is the preferred tool going forward, at least at this point.

There are a few handy subcommands that come baked in to the kubectl command. The first is the resize command. This command allows you to scale the number of running containers being managed by Kubernetes up or down on the fly. Obviously this can be really powerful! The syntax is pretty straight forward and below I have an example listed.

kubectl resize –current-replicas=6 –replicas=0 rc test-rc

The –current-replicas argument is optional, the –replicas defines the *desired* number of replicas to have runing, rc specifies this is a replication controller, and finally, test-rc is the name of the replication controller to scale. After you scale your replication controller you can check out the status quickly via the following command.

kubectl get pod

Another handy tool to have when working with Kubernetes is the ability to deploy new images as a rolling update.

kubectl rollingupdate test-rc -f test-rc-2.yml –update-period=”10s”

The rollingupdate command takes a few arguments. The first is the name of the current replication controller that you would like to update. The second is to replace it with the yml file of the new replication controller and the third optional argument is the –update-period, which allows a user to override the default time that it takes to spin up a new container and spin down an old.

Below is an example of what your test-rc-2.yml file may look like.

kind: ReplicationController
apiVersion: v1beta1
id: test-rc-2
namespace: default
desiredState:
 replicas: 1
 replicaSelector:
   name: test-rc
   version: v2
 podTemplate:
 labels:
   name: test-rc
   version: v2
 desiredState:
 manifest:
 version: v1beta1
 id: test-rc
 containers:
   - name: test-image
   image: test/test:new-tag
   imagePullPolicy: PullAlways
 ports:
   - name: test-port
   containerPort: 8080

There are a few important things to notice. The first is that the id must be unique, it can’t be a name that is already in use by another replication controller. All of the label names should remain the same except for the version. The version is used to signify the new replication controller is a running a new docker image. The version number should be unique, which will help keep track of which image version is running.

Another thing to note. If your original replication controller did not contain a unique key (like version) then you will need to update the original replication controller first, adding a unique key, before attempting to run the rolling update.

If both replication controllers don’t have the same format you will get an error similar to this.

test-rc.yml must specify a matching key with non-equal value in Selector for <selector name>

So that’s pretty much it for now. I will revisit this post again in the future as new flags and subcommands are added to kubectl for managing and updating replication controllers. I also plan on writing a few more posts about other aspects and areas of kubectl and running Kubernetes, so please check back soon!

Add environment variable file to fig

November 21, 2014August 31, 2015 Josh Reichardt 14 Comments

If you haven’t heard of fig and are using Docker, you need to check it out. Basically Fig is a tool that allows users to quickly create development environments using Docker. Fig alleviates the complexity and tediousness of having to manually bring containers up and down, stitch them together and basically orchestrate a Docker environment. On top of this, Fig offers some other cool functionality, for example, the ability to scale up applications. I am excited to see what happens with the project because it was recently merged in to the Docker project. My guess is that there will be many new features and additions to either Docker if Fig gets rolled in to the Docker core project. For now, you can check out Fig here if you haven’t heard of it and are interested in learning more.

One issue that I have run in to is that there is currently not a great way to handle a large number of environment variables in fig. In Docker there is an option that allows a user to pass in an environment variable file with the –env file <filename> flag. To do the same with Fig in its current form, you are forced to list out each individual environment variable in your configuration which can quickly become tedious and confusing.

There is a PR out for adding in the ability to pass an environment variable file in to fig via the env-file option in a fig.yml file. This approach to me is much easier than adding each environmental variable separately to the configuration with the environment option as well as having to update the fig.yml configuration if any of the values ever change. I know that functionality like this will get merged eventually but until then I have been using the PR as a workaround to solve this issue, I think that this is also a good opportunity to show people how to get a project working manually with custom changes. Luckily the fix isn’t difficult.

This post will assume that you have git, python and pip installed. If you don’t have these tools installed go ahead and get that done. The first step is to clone the fig project on github on to your local machine, see above for the link to the PR.

git clone [email protected]:docker/fig.git

Jump in to the fig project you just cloned and edit the service.py file. This is the file that handles the processing of environment variables. There are a few sections that need to be updated in the config. Check the PR to be sure, but at the time of the writing, the following code should be added.

Line 55

- supported_options = DOCKER_CONFIG_KEYS + ['build', 'expose']
+ supported_options = DOCKER_CONFIG_KEYS + ['build', 'expose', 'env-file']

Line 318

+ def _get_environment(self):
+ env = {}
+
+ if 'env-file' in self.options:
+ env = env_vars_from_file(self.options['env-file'])
+
+ if 'environment' in self.options:
+ if isinstance(self.options['environment'], list):
+ env.update(dict(split_env(e) for e in self.options['environment']))
+ else:
+ env.update(self.options['environment'])
+
+ return dict(resolve_env(k, v) for k, v in env.iteritems())
+

LIne 352

- if 'environment' in container_options:
- if isinstance(container_options['environment'], list):
- container_options['environment'] = dict(split_env(e) for e in container_options['environment'])
- container_options['environment'] = dict(resolve_env(k, v) for k, v in container_options['environment'].iteritems())
+ container_options['environment'] = self._get_environment()

Line 518

+
+
+def env_vars_from_file(filename):
+ """
+ Read in a line delimited file of environment variables.
+ """
+ env = {}
+ for line in open(filename, 'r'):
+ line = line.strip()
+ if line and not line.startswith('#') and '=' in line:
+ k, v = line.split('=', 1)
+ env[k] = v
+ return env

That should be it. Now you should be able to install fig with the new changes. Make sure you are in the root fig directory that contains the setup.py file.

sudo python setup.py develop

Now you should be able to edit your fig.yml file to reflect the changes that have been added to fig via env-file. Here is what a sample configuration might look like.

testcontainer:
 image: username/testcontainer
 ports:
 - "8080:8080"
 links:
 - "mongodb:mongodb"
 env-file: "/home/username/test_vars"

Notice that nothing else changed. But instead of having to list out environment variables one at a time we can simply read in a file. I have found this to be very useful for my workflow, I hope others can either adapt this or use this as well. I have a feeling this will get merged in to fig at some point but for now this workaround works.

Patching CVE-6271 (shellshock) with Chef

September 26, 2014August 31, 2015 Josh Reichardt 2 Comments

If you haven’t heard the news yet, a recently disclosed vulnerability has been released that exploits environmental variables in bash. This has some far reaching implications because bash is so widespread and runs on many different types of devices, for example network gear, routers, switches, firewalls, etc. If that doesn’t scare you then you probably don’t need to finish reading this article. For more information you can check out this article that helped to break the story.

I have been seeing a lot “OMG the world is on fire, patch patch patch!” posts and sentiment surrounding this recently disclosed vulnerability, but basically have not seen anybody taking the time to explain how to patch and fix this issue. It is not a difficult fix but it might not be obvious to the more casual user or those who do not have a sysadmin or security background.

Debian/Ubuntu:

Use the following commands to search through your installed packages for the correct package release. You can check the Ubuntu USN for versions.

dpkg -l | grep '^ii' or
dpkg-query --show bash

If you are on Ubuntu 12.04 you will need update to the following version:

bash    4.2-2ubuntu2.3

If you are on Ubuntu 13.10, and have this package (or below), you are vulnerable. Update to 14.04!

 bash 4.2-5ubuntu3

If you are on Ubuntu 14.04, be sure to update to the most recently patched patch.

bash 4.3-7ubuntu1.3

Luckily, the update process is pretty straight forward.

apt-get update
apt-get --only-upgrade install bash

If you have the luxury of managing your environment with some sort of automation or configuration management tool (get this in place if you don’t have it already!) then this process can be managed quite efficiently.

It is easy to check if a server that is being managed by Chef has the vulnerability by using knife search:

knife search node 'bash_shellshock_vulnerable:true'

From here you could createa a recipe to patch the servers or fix each one by hand. Another cool trick is that you can blast out the update to Debian based servers with the following command:

knife ssh 'platform_family:debian' 'sudo apt-get update; sudo apt-get install -y bash'; knife ssh 'platform_family:redhat' 'sudo yum -y install bash'

This will iterate over every server in your Chef server environment that is in the Debain family (including Ubuntu) or RHEL family (including CentOS) and update the server packages so that the latest patched bash version gets pulled down and then gets updated to the latest version.

You may need to tweak the syntax a little, -x to override the ssh user and -i to feed an identity file. This is so much faster than manually installing the update on all your servers or even fiddling around with a tool like Fabric, which is still better than nothing.

One caveat to note: If you are not on an LTS version of Ubuntu, you will need to upgrade your server(s) first to an LTS, either 12.04 or 14.04 to qualify for this patch. Ubuntu 13.10 went out of support in August which was about a month ago as per the time of this writing so you will want to get your OS up date.

One more thing: The early patches to address this vulnerability did not entirely fix the issue, so make sure that you have the correct patch installed. If you patched right away there is a good chance you may still be vulnerable, so simply rerun your knife ssh command to reapply the newest patch, now that the dust is beginning to settle.

Outside of this vulnerability, it is a good idea to get your OS on an Ubuntu LTS version anyway to continue receiving critical updates for software as well as security patches for a longer duration than the normal, 6 month release cycle of the server distribution.

Test Kitchen Tricks

September 2, 2014August 31, 2015 Josh Reichardt 27 Comments

I have been working a lot with Chef and Test kitchen lately and thus have learned a few interesting tricks when running tests with these tools. Test Kitchen is one of my favorite tools when working with Chef configuration management because it is very easy to use and has a number of powerful features that make testing things in Chef simple and easy.

Test Kitchen itself sits on top of Vagrant and Virtualbox by default so to get started with the most basic usage example of Test Kitchen you will need to have Vagrant installed along with a few other items.

Vagrant (your distribution)
Virtualbox (again with your distribution)
Ruby 1.9 or higher + ruby gems

Then to install Test Kitchen.

gem install test-kitchen

That’s pretty much it. The official docs have some pretty detailed usage and in fact I have learned many of the tricks that I will be writing about today from the docs.

Once you are comfortable with Test Kitchen you can begin leveraging some of the more powerful features, which is what the remainder of this post will cover. There is a great talk given by the creator of test kitchen at this year’s Chefconf by the creator of Test Kitchen about some of the lessons learned and cool things that you can do with the tool. If you haven’t already seen it, it is worth a watch.

Anyway, let’s get started.

1) Fuzzy matching

This one is great for the lazy people out there. It basically allows you match a certain unique part of a command instead of typing out an entire command. So for example, you can just type in a partial name for a command to return the desired full command. Since Test Kitchen uses regular expression matching, this can be a very powerful feature.

2) Custom drivers

One reason that Test Kitchen is so flexible is because it can leverage many different plugins and drivers. And, since it is open source, if there is functionality missing from a driver you can simply write your own. Currently there is an awesome list of drivers available for Test Kitchen to use, and a wide variety of options available to hopefully suit most testing scenarios.

Of course, there are others as well. These just happen to be the drivers that I have tried and can verify. There is even support for alternate configuration management tool testing, which can be handy for those that are not using Chef specifically. For example there is a salt driver available.

3) .kitchen.local.yml

This is a nice handy little bit that is often overlooked but allows a nice amount of control by overriding the default .kitchen.yml configuration file with specific options. So for example, if you are using the ec2 driver in your configuration but need to test locally with Vagrant you can simply drop a .kitchen.local.yml on your dev machine and override the driver (and any other settings you might need to change). I have created the following .kitchen.local.yml for testing on a local Vagrant box using 32 bit Ubuntu to highlight the override capabilities of Test Kitchen.

driver: 
 name: vagrant 
 
platforms: 
 - name: ubuntu-1310-i386 
 - name: ubuntu-1404-i386

4) Kitchen diagnose

An awesome tool for diagnosing issues with Test Kitchen. Running the diagnose will give you lots of juicy info about what your test machines are doing (or should be doing) and a ton of configuration information about them. Basically, if something is misbehaving this is the first place you should look for clues.

If you want to blast info and settings for all your configurations, just run the following,

kitchen diagnose

5) Concurrency

If you have a large number of systems that need to have tests run on them then running your Test Kitchen tests in parallel is a great way to speed up your total testing time. Turning on concurrency is pretty straight forward, just add the “-c” flag and the number of instances to run on (the default is 9999).

kitchen converge -c 5

6) Verbose logging

This one can be helpful if your kitchen run is failing with no real clues or helpful information provided by the diagnose command. It seems obvious but getting this one to work gave me some trouble initially. To turn on verbosity simply add the debug flag to your test kitchen command.

So for example, if you want to converge a node with verbosity turned on, you would use this command.

kitchen converge -l debug

I recommend taking a look at some or all of these tricks to help improve your integration testing with Test Kitchen. Of course as I stated, all of this is pretty well documented. Even if you are already familiar with this tool, sometimes it just helps to have a refresher to remind you of a great tool and to jar your memory. Let me know if you have any other handy tricks and I will be sure to post them here.