Set up PEM key authentication

May 15, 2014August 31, 2015 Josh Reichardt 1 Comment

Many times it is useful to keys to authenticate to your servers. This can dramatically improve security and is a great way to manage servers in bulk as well. You just need to keep track of your keys rather than having to remember a large number of passwords. The steps to get PEM key authentication are fairly straight forward but it never hurts to walk through the process of getting them set up correctly.

Side Note: I’d like to also mention briefly, that I have these steps set up to work with Chef, so every server that gets deployed using Chef will use PEM keys out of the box, which works out very nicely. If you’re interested I can expound on that topic a little more, just let me know.

The first step in the process is to generate some keys using openssl. If you don’t have openssl go download and install it. If you do have openssl but haven’t updated in ahwile, please update to avoid the heartbleed vulnerability that was recently exploited (nearly all distributors have released the patched version at this point so it should be trivial).

We want to generate our key and create a PEM file out of it. Here are the steps:

cd ~/.ssh
ssh-keygen -t dsa -b 1024
openssl dsa -in id_dsa -outform pem > test.pem
cat ida_dsa >> authorized_keys

You can leave the values blank (default) in the ssh-keygen.

Now you should have similar listings in your ~/.ssh directory:

authorized_keys – This is the public key that the pem file gets authenticated against
id_dsa – This is the private portion of the key that we generated in the steps above
id_dsa.pub – This is the public key section that is used when authenticating
test.pem – this is the file that will be used to authenticate. Essentially the private key minus the pass phrase

Now you just need to copy the test.pem file that was just generated to a different host in order to log in with your PEM key using scp or rsync. Once that is done, the command to connect to the remote host using your key should look similar to the following:

ssh -i /path/to/pem user@server-name

Next steps. At this point you should have a working pem authentication on your server. It is probably a good idea at this point to start looking at hardening the security as well as the SSH configuration on the host. Small things can go a long way. For example disabling root login, disabling password authentication, etc. will stop a very large amount of attacks from hitting your server now that you are authenticating with pem keys.

Using a self signed cert with Nginx

May 5, 2014August 31, 2015 Josh Reichardt 7 Comments

After the recent heart bleed incident (which I’m sure many of you well remember) I had to reassign some certificates. It turns out that this was a great opportunity to create a blog post. Since I do not create and assign certs very frequently it is a good opportunity to take some notes and hopefully ease the process for others. After patching the vulnerable version of Openssl, there are really only a few steps needed to accomplish this. Assuming you already have nginx installed, which is trivial to do on Ubuntu, the first step is to create the necessary crt and key files.

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/cert.key -out /etc/nginx/cert.crt

Next you will need to tell nginx to load up you new certs in its config. Here is an example of what the server block in you /etc/nginx/site-available config might look like. Notice the ssl_certificate and ssl_certificate_key files correspond to the cert files we created above, which we stuck in the /etc/nginx directory. If you decide to place these certs in a different location you will need to modify your config file to reflect the location.

server {

listen *:443; 
ssl on; 
ssl_certificate cert.crt; 
ssl_certificate_key cert.key; 
ssl_session_timeout 5m; 
ssl_protocols SSLv3 TLSv1; 
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP; 
ssl_prefer_server_ciphers on;

}

Just to cover all our bases here we will also redirect any requests that come in to port 80 (default web) back to 443 for ssl. The is a simple addition and will add an additional layer of security.

server { 
listen 80; 
return 301 https://$host$request_uri; 
}

The final step is to reload your configuration and test to make sure everything works.

sudo service nginx reload

If your nginx fails to reload, more than likely there is some sort of configuration or syntax error in your config file. Comb through it for any potential errors or mistakes. Once your config is loaded properly you can check your handy work by attempting to hit your site using http://. If your config is working properly it should automatically redirect you to https://.

That’s all it takes. I think it might be a good exercise to try something like this with Chef but for now this process works okay by hand. Let me know what you think or if this can be improved.

Setting up a private git repo in Chef

April 22, 2014August 31, 2015 Josh Reichardt 28 Comments

It turns out that cloning and managing private git repo’s in Chef is not as easy as it looks. That said, I have a method that works. I have no idea if this is the preferred method or if there are any easier ways but this worked for me, so let me know if there is an easier way and I will be glad to update this post.

First, I’d like to give credit where it is due. I used this post as a template as well as the SSH wrapper section in the deploy documentation on the Chef website.

The first issue is that when you connect to github via SSH it wants the Chef client to accept its public fingerprint. By default, if you don’t modify anything SSH will just sit there waiting for the fingerprint to be accepted. That is why the SSH Git wrapper is used, it tells SSH on the Chef client that we don’t care about the authentication to the github server, just accept the key. Here’s what my ssh git wrapper looks like:

 #!/bin/bash 
 exec ssh -o "StrictHostKeyChecking=no" -i "/home/vagrant/.ssh/id_rsa" $1 $2

You just need to tell your Chef recipe to use this wrapper script:

# Set up github to use SSH authentication 
cookbook_file "/home/vagrant/.ssh/wrap-ssh4git.sh" do 
  source "wrap-ssh4git.sh" 
  owner "vagrant" 
  mode 00700 
end

The next problem is that when using key authentication, you must specify both a public and a private key. This isn’t an issue if you are running the server and configs by hand because you can just generate a key on the fly and hand that to github to tell it who you are. When you are spinning instances up and down you don’t have this luxury.

To get around this, we create a couple of templates in our cookbook to allow our Chef client to connect to github with an already established public and private key, the id_rsa and id_rsa.pub files that are shown. Here’s what the configs look like in Chef:

# Public key 
template "/home/vagrant/.ssh/id_rsa.pub" do 
  source "id_rsa.pub" 
  owner "vagrant" 
  mode 0600 
end 
 
# Private key 
template "/home/vagrant/.ssh/id_rsa" do 
  source "id_rsa" 
  owner "vagrant" 
  mode 0600 
end

After that is taken care of, the only other minor caveat is that if you are cloning a huge repo then it might timeout unless you override the default timeout value, which is set to 600 seconds (10 mins). I had some trouble finding this information on the docs but thanks to Seth Vargo I was able to find what I was looking for. This is easy enough to accomplish, just use the following snippet to override the default value

timeout 9999

That should be it. There are probably other, easier ways to accomplish this and so I definitely think the adage “there’s more than one way to skin a cat” applies here. If you happen to know another way I’d love to hear it.

Podcasts for DevOps admins

April 9, 2014August 31, 2015 Josh Reichardt 1 Comment

Getting up to speed in a fast moving environment forces you to think about things in a different way, which for me was/is an interesting sort of paradigm shift. Moving from enterprise to start up I have found things to be much different and so embracing the DevOps philosophy and culture has been a main focus of mine through this transition, in a good way of course. Today I’d like to share some interesting resources that I have found to be immensely helpful in my journey thus far into the land of DevOps. Hopefully readers are in the same position that I am in and can use this information in their own DevOps journey.

In my experience I have found that podcasts are one of the absolute best ways to consume information, whether it be on a morning commute or viewing the show live, good podcasts are one of the best learning tools around. So for today’s post, I have compiled a list of some good shows related to DevOps that I hope others find to be useful.

If you’re interested, I wrote a post awhile back focusing on some my favorite podcasts relating to system administration. You can find the list and original Podcasts for System Administrators post here.

The Food Fight Show

From their website: “Food Fight is a bi-weekly podcast for the Chef community. We bring together the smartest people in the Chef community and the broader DevOps world to discuss the thorniest issues in system administration.” This show offers some great conversation in topics around DevOps, a lot of really in depth technical discussion from industry experts as well as some great interviews with various contributors to the DevOps community. This right now is my favorite DevOps podcast and there are a large number of episodes to choose from, so you can hand pick a few episodes to try out if you are skeptical.

DevOps Cafe

This show takes a similar round table format similar to the style of The Food Fight Show. This show is co-hosted by Damon Edwards and John Willis which covers a lot of cool news and interesting topics on the bleeding edge of the DevOps world. There is is a nice variety of interesting guests as well as relevant topics of discussion. I like this show because for me, it does a great job of focusing in on the more relevant aspects of DevOps, rather than the abstract concepts and ideas behind DevOps. To me, it is more practice than theory. That might be a horrible description so you’ll just have to go check out the podcast to find out for yourself.

Arrested DevOps

This podcast is in much the same vein as The Food Fight Show, where DevOps pro’s sit down and discuss issues related to what is going on in the DevOps world. I just started listening to this podcast as it is one of the newer additions to my DevOps podcast scene. This show definitely has a lot of potential; the hosts are knowledgeable, the guests are smart and the topics of conversation are interesting. Trevor and Matt do a good job of mixing technical discussion with some of the more DevOps type topics and ideas, I would definitely taking a look at this podcast.

Ops All The Things

Another new kid on the block, this show is hosted by Chris Webber and Steven Murawski of Stack Exchange fame. The focus of the show is geared towards system administration, operations and DevOps. I like this podcast because it does a good job of blending DevOps with system administration, which is the track that I followed into the world of DevOps. Much of the show is geared towards nuts and bolts administration which is nice. Topics are often in depth and technical, with discussions revolving around things like configuration management, monitoring, revision control, etc. One other nice feature is that it covers some administration topics related to Windows which I think gives listeners a good perspective.

The Ship Show

I haven’t had a chance to dive too deep into this one much yet but judging from the few episodes I’ve been able to listen to, this show definitely captures a lot of relevant and interesting issues in the community. Once I get more episodes under my belt and get a better feel for the show I will update the post. But just to give readers an idea, this is from their bio: “The Ship Show is a twice-monthly podcast, featuring discussion on everything from build engineering to DevOps to release management, plus interviews, new tools and techniques, and reviews.”

If I missed any or if you’re interested in starting a DevOps oriented podcast let me know and I will be sure to add you to the list and help spread the word. I think it’s important for people in the community to help out and share their knowledge.

Dev vs. Ops and DevOps

Setting up a GitHub webhook in Jenkins

March 27, 2014August 31, 2015 Josh Reichardt 9 Comments

This post will detail the steps to have Jenkins automatically create a build if it detects changes to a GitHub repository. This can be a very useful improvement to your continuous integration setup with Jenkins because this method is only telling Jenkins to attempt a new build when a change is detected rather than polling on an interval, which can be a little bit inefficient.

There are a few steps necessary to get this process working correctly that I would like to highlight in case I have to do this again or if anybody else would like to set this up. Most of the guides that I found were very out of date so their instructions were a little bit unclear and misleading.

The first step is to configure Jenkins to talk to GitHub. You will need to download and install the GitHub plugin (I am using version 1.8 as of this writing). Manage Jenkins -> Manage Plugins -> Available -> GitHub plugin

After this is installed you can either create a new build or configure an existing build job. Since I already have one set up I will just modify it to use the GitHub hook. There are a few things that need to be changed.

First, you will need to add your github repo:

Then you will then have to tick the box indicated below – “Build when a change is pushed to GitHub”

Also note that Jenkins should have an SSH key already associated with the desired GitHub project.

You’re pretty close to being done. The final step is to head over to GitHub and adjust the settings for the project by creating a webhook for your Jenkins server. Select the repo you’re interested in and click Settings. If you aren’t an admin of the repo you will not be able to modify the settings, so talk to an owner to either finish this step for you or have them grant you admin to make the change yourself.

The GitHub steps are pretty straight forward. Open the “Webhooks & Services” tab -> choose “Configure Services” -> find the Jenkins (GitHub plugin option) and fill it in with a similar URL to the following:

http://<Name of Jenkins server>:8080/github-webhook/

Make sure to tick the active box and ensure it works by running the “Test Hook”. If it comes back with a payload deployed message you should be good to go.

UPDATE

I found an issue that was causing us issues. There is a check box near the bottom of the authentication section labeled “Prevent Cross Site Request Forgery exploits” that needs to be unchecked in order for this particular method to work.

Let me know if you have any issues, I haven’t found a good way to debug or test outside of the message returned from the GitHub configuration page. I did find another alternative method that may work but didn’t need to use it so I can update this if necessary.

If you want more details about web hooks you can check out these resources:

developer documentation.
You can also check out the GitHub Jenkins plugin here.
And another nice guide.

Related articles