Set up PagerDuty alerts in Sensu

I am currently in the midst of rolling a monitoring solution using Sensu and a handful of other tools, which I will be covering sporadically in the future.  Onee important facet of any good monitoring solution is a reliable alerting method.  Sensu uses a distributed approach to monitoring so all of the components are spread out rather than run as one monolithic system.  So following this principle, Sensu integrates nicely with the awesome PagerDuty tools for alerting.  You can find more information about Sensu and its architecture over at the docs page of their website.

“The Sensu way” involves using what is called a handler (for the uninitiated) to trigger an alert.  So for example, my setup involves a number of checks, which are run on each of my clients.  These checks have associated subscribers and handlers that report back to the Sensu server.  From there the Sensu server will run the handler(s) specified and do something with results of the check that was run on the Sensu client.

For my project I am using PagerDuty to generate alerts if disk space gets low or a process dies.  I will briefly run through the steps of how to set the PagerDuty integration up because there were a few roadblocks that I encountered when I set this up the first time.

This set of instructions assumes that you already have a PagerDuty account created and configured.  So the first step is to create a Service API check for Sensu.  Pick a suitable name and choose Use our API directly.  It should look similar to the following:

pagerduty api key

Now that we have an API key set up in PagerDuty we should be able to jump on the Sensu server and add in the apporpriate json to configure the Sensu handler to communicate with PagerDuty.  Place the following contents in /etc/sensu/conf.d/handlers/pagerduty.json.

 "pagerduty": {
   "api_key": "xxxxxx"
 "handlers": {
   "pagerduty": {
     "type": "pipe",
     "command": "/etc/sensu/plugins/pagerduty.rb",
     "severities": [

I learned (the hard way) that the pagerduty.rb script won’t work out of the box.  It relies on a ruby gem called redphone.  It is easy enough to install and get working, just do a gem install redphone and you should be all set.

Next, go ahead and download the pagerduty.rb script to the appropriate location on the Sensu server:

cd /etc/sensu/plugins
wget -O /etc/sensu/plugins/pagerduy.rb

That should be it.  One good way to check if things are working and that the checks and handler are actually firing correctly is to tail the log file on both the client and server. On the server the log is located at /var/log/sensu/sensu-server.log and on the client machine at /var/log/sensu/sensu-client.log.

Bonus:  Chef integration

Of course all of this can be automated using Chef, which is ultimately what I ended up doing, so I will share some of the things that I learned in the process.  For starters, I am using the Sensu Chef cookbook, created by the maintainer of the Sensu project.  This cookbook exposes a few useful options for configuration Sensu.  You will need to clone the cookbook directly from the github repository to get the newest features that we need, as the Opscode version has not yet been updated to incorporate them.

Just add this line to your recipe before you call any of the Sensu resources/providers.

include_recipe "sensu::default"

The Sensu coobook exposes a number of nice resources that we can use in our recipes to deploy Sensu.  As an example if you wanted to clone the PagerDuty handler to the Chef server you would use something like the following in your recipe:

sensu_plugin ""

Which will place the pagerduty.rb script into the appropriate directory automaitcally.  There are other options as well, but this should do the trick.  You can find some more examples here.

Define your pagerduty handler:

sensu_handler "pagerduty" do 
  type "pipe" 
  command "/etc/sensu/plugins/pagerduty.rb" 
  severities ["ok", "critical"] 

You will need to add this handler to each check that you want to receive an alert on, and you will also need to subscribe your host to that check as well.  Here is what an example check might look like:

sensu_check "check_ntp" do 
   command "/etc/sensu/plugins/check-procs.rb -p ntpd -C 1" 
   handlers ["pagerduty"] 
   subscribers ["core"] 
   interval 60 
   additional(:notification => "NTP is not running", :occurrences => 5) 

That’s all I have for now.  So far Sensu has been amazing, it is very flexible and the IRC channel an excellent resource.  The docs are nice as well.  Again, props to Sean Porter for creating an awesome new way to do monitoring.  I am still just flirting with the very top of the iceburg as far as the capabilites of Sensu go and will be revisiting this subject in the future.

Read More

Setting up a private git repo in Chef

It turns out that cloning and managing private git repo’s in Chef is not as easy as it looks.  That said, I have a method that works.  I have no idea if this is the preferred method or if there are any easier ways but this worked for me, so let me know if there is an easier way and I will be glad to update this post.

First, I’d like to give credit where it is due.  I used this post as a template as well as the SSH wrapper section in the deploy documentation on the Chef website.

The first issue is that when you connect to github via SSH it wants the Chef client to accept its public fingerprint.  By default, if you don’t modify anything SSH will just sit there waiting for the fingerprint to be accepted.  That is why the SSH Git wrapper is used, it tells SSH on the Chef client that we don’t care about the authentication to the github server, just accept the key.  Here’s what my ssh git wrapper looks like:

 exec ssh -o "StrictHostKeyChecking=no" -i "/home/vagrant/.ssh/id_rsa" $1 $2

You just need to tell your Chef recipe to use this wrapper script:

# Set up github to use SSH authentication 
cookbook_file "/home/vagrant/.ssh/" do 
  source "" 
  owner "vagrant" 
  mode 00700 

The next problem is that when using key authentication, you must specify both a public and a private key.  This isn’t an issue if you are running the server and configs by hand because you can just generate a key on the fly and hand that to github to tell it who you are.  When you are spinning instances up and down you don’t have this luxury.

To get around this, we create a couple of templates in our cookbook to allow our Chef client to connect to github with an already established public and private key, the id_rsa and files that are shown.  Here’s what the configs look like in Chef:

# Public key 
template "/home/vagrant/.ssh/" do 
  source "" 
  owner "vagrant" 
  mode 0600 
# Private key 
template "/home/vagrant/.ssh/id_rsa" do 
  source "id_rsa" 
  owner "vagrant" 
  mode 0600 

After that is taken care of, the only other minor caveat is that if you are cloning a huge repo then it might timeout unless you override the default timeout value, which is set to 600 seconds (10 mins).  I had some trouble finding this information on the docs but thanks to Seth Vargo I was able to find what I was looking for. This is easy enough to accomplish, just use the following snippet to override the default value

timeout 9999

That should be it.  There are probably other, easier ways to accomplish this and so I definitely think the adage “there’s more than one way to skin a cat” applies here.  If you happen to know another way I’d love to hear it.

Read More

Podcasts for DevOps admins

podcastGetting up to speed in a fast moving environment forces you to think about things in a different way, which for me was/is an interesting sort of paradigm shift.  Moving from enterprise to start up I have found things to be much different and so embracing the DevOps philosophy and culture has been a main focus of mine through this transition, in a good way of course.  Today I’d like to share some interesting resources that I have found to be immensely helpful in my journey thus far into the land of DevOps.  Hopefully readers are in the same position that I am in and can use this information in their own DevOps journey.

In my experience I have found that podcasts are one of the absolute best ways to consume information, whether it be on a morning commute or viewing the show live, good podcasts are one of the best learning tools around.  So for today’s post, I have compiled a list of some good shows related to DevOps that I hope others find to be useful.

If you’re interested, I wrote a post awhile back focusing on some my favorite podcasts relating to system administration.   You can find the list and original Podcasts for System Administrators post here.

The Food Fight Show

From their website: “Food Fight is a bi-weekly podcast for the Chef community. We bring together the smartest people in the Chef community and the broader DevOps world to discuss the thorniest issues in system administration.”  This show offers some great conversation in topics around DevOps, a lot of really in depth technical discussion from industry experts as well as some great interviews with various contributors to the DevOps community.  This right now is my favorite DevOps podcast and there are a large number of episodes to choose from, so you can hand pick a few episodes to try out if you are skeptical.

DevOps Cafe

This show takes a similar round table format similar to the style of The Food Fight Show.  This show is co-hosted by Damon Edwards and John Willis which covers a lot of cool news and interesting topics on the bleeding edge of the DevOps world.  There is is a nice variety of interesting guests as well as relevant topics of discussion.  I like this show because for me, it does a great job of focusing in on the more relevant aspects of DevOps, rather than the abstract concepts and ideas behind DevOps.  To me, it is more practice than theory.  That might be a horrible description so you’ll just have to go check out the podcast to find out for yourself.

Arrested DevOps

This podcast is in much the same vein as The Food Fight Show, where DevOps pro’s sit down and discuss issues related to what is going on in the DevOps world.  I just started listening to this podcast as it is one of the newer additions to my DevOps podcast scene.  This show definitely has a lot of potential; the hosts are knowledgeable, the guests are smart and the topics of conversation are interesting.  Trevor and Matt do a good job of mixing technical discussion with some of the more DevOps type topics and ideas, I would definitely taking a look at this podcast.

Ops All The Things

Another new kid on the block, this show is hosted by Chris Webber and Steven Murawski of Stack Exchange fame.  The focus of the show is geared towards system administration, operations and DevOps.  I like this podcast because it does a good job of blending DevOps with system administration, which is the track that I followed into the world of DevOps.  Much of the show is geared towards nuts and bolts administration which is nice.  Topics are often in depth and technical, with discussions revolving around things like configuration management, monitoring, revision control, etc.  One other nice feature is that it covers some administration topics related to Windows which I think gives listeners a good perspective.

The Ship Show

I haven’t had a chance to dive too deep into this one much yet but judging from the few episodes I’ve been able to listen to, this show definitely captures a lot of relevant and interesting issues in the community.   Once I get more episodes under my belt and get a better feel for the show I will update the post.  But just to give readers an idea, this is from their bio:  “The Ship Show is a twice-monthly podcast, featuring discussion on everything from build engineering to DevOps to release management, plus interviews, new tools and techniques, and reviews.”

If I missed any or if you’re interested in starting a DevOps oriented podcast let me know and I will be sure to add you to the list and help spread the word.  I think it’s important for people in the community to help out and share their knowledge.

Read More