Creating new a instance-store AMI for Amazon AWS EC2

This is a HOWTO build your own instance-store backed AMI image which is suitable for creating a Paid AMI. The motivation for doing this HOWTO is simple: I tried it, and it has a lot of little gotchas, so I want some notes for myself. This HOWTO assumes you’re familiar with launching EC2 instances, logging into them, and doing basic command line tasks.

Choosing a starting AMI

There’s a whole ton of AMIs available for use with EC2, but not quite so many which are backed by instance-store storage. Why’s that? Well, EBS is a lot more flexible and scalable. The instance-store images have a fairly limited size for their root partition. For my use case, this isn’t particularly important, and for many use cases, it’s trivial to mount some EBS volumes for persistant storage.

Amazon provides some of their Amazon Linux AMIs which are backed by EBS or instance-store, but they’re based on CentOS, and frankly, I’ve had so much troubles with CentOS in the past, that I just prefer my old standby: Ubuntu. Unfortunately, I had a lot of trouble finding a vanilla Ubuntu 12.04 LTS instance-store backed image through the AWS Console. They do exist, however, and they’re provided by Canonical. Thanks  guys!

Here’s a list of all the 12.04 Precise official AMIs:
http://cloud-images.ubuntu.com/releases/precise/release/

Conveniently, there’s a Launch button right there for each AMI instance. Couldn’t be easier!

Installing the EC2 Tools

Once you’ve got an instance launched and you’re logged in and sudo‘d to root, you’ll need to install the EC2 API and AMI tools provided by Amazon. The first step is, of course, to download them. Beware! The tools available through the Ubuntu multiverse repositories are unfortunately out of date.

The latest EC2 API tools can be found here:
http://aws.amazon.com/developertools/351

The latest EC2 AMI tools can be found here:
http://aws.amazon.com/developertools/368

I like to copy the download link and use wget to download them rather than scp‘ing them from my client machine.

sudo su
mkdir -p /tmp/ec2-tools
cd /tmp/ec2-tools
wget -O ec2-api-tools.zip 'http://www.amazon.com/gp/redirect.html/ref=aws_rc_ec2tools?location=http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip&token=A80325AA4DAB186C80828ED5138633E3F49160D9'
wget -O ec2-ami-tools.zip 'http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip'

Before we can install the EC2 tools, we need to install a few packages that our vanilla Ubuntu is lacking, namely zip and Java.

apt-get install zip
apt-get install openjdk-6-jre-lib
apt-get install ruby

Once we have those installed, we need to unzip our packages and install them to the /usr/local directory.

unzip "*.zip"
find . \( -name bin -o -name lib -o -name etc \) | \
    xargs -I path cp -r path /usr/local

Lastly we have to set the EC2_HOME and the JAVA_HOME environment variables for the EC2 tools to work properly. I like to do this by editing /etc/bash.bashrc so anyone on the machine can use the tools without issue.

echo -e "\nexport EC2_HOME=/usr/local\nexport JAVA_HOME=/usr\n" >> /etc/bash.bashrc

Once we log out and back in, those variables will be set, and the EC2 tools will be working.

# exit
$ sudo su
# ec2-version
1.6.7.4 2013-02-01

Customizing Your AMI

At this point, your machine should be all set for you to do whatever customization you need to do. Install libraries, configure boot scripts, create users, get your applications set up, anything at all. Once you’ve got a nice, stable (rebootable) machine going, then you can image it.

Bundling, Uploading and Registering your AMI

This is actually pretty easy, but I’ll still go through it. The Amazon documentation is fairly clear, and I recommend following along with that as well, as it explains all the options to each command.

Here’s the official Amazon documentation:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-snapshot-s3-linux.html

1. Create an S3 bucket. This is where you’ll upload your AMI images. If you already have a bucket, you can use that.
2. Download your AWS security certificates and copy your API keys. They can be found here: https://portal.aws.amazon.com/gp/aws/securityCredentials
3. Copy your credentials to the instance you’re going to image. First, create a directory to store them in on your instance:

   mkdir -p /tmp/cert
   chmod 777 /tmp/cert

4. Then copy them from the place you downloaded them on your client machine, to your instance:

   scp -i <keypair_name> pk-*.pem cert-*.pem ubuntu@<host_name>:/tmp/cert

5. Bundle your instance image. The actual image bundle and manifest will end up in /tmp.

   cd /tmp/cert
   ec2-bundle-vol -k <private_keyfile> -c <certificate_file> \
       -u <user_id> -e <cert_location>
   cd /tmp

6. Upload your bundled image. Note that <your-s3-bucket> should include a path that is unique to this image, such as my-bucket/ami/ubuntu/my-ami-1, otherwise things will get very messy for you, because an image consists of an image.manifest.xml file and many chunks which compose the image itself, which are generically named by default when you use this tool.

   ec2-upload-bundle -b <your-s3-bucket> -m <manifest_path> \
       -a <access_key> -s <secret_key>

7. Register your new AMI.

   ec2-register <your-s3-bucket>/<path>/image.manifest.xml -n <image_name> \
       -O <your_access_key> -W <your_secret_key>

That’s it! You should be all set with a new AMI, which should also show up in the AWS Console.