Set up PagerDuty alerts in Sensu

I am currently in the midst of rolling a monitoring solution using Sensu and a handful of other tools, which I will be covering sporadically in the future.  Onee important facet of any good monitoring solution is a reliable alerting method.  Sensu uses a distributed approach to monitoring so all of the components are spread out rather than run as one monolithic system.  So following this principle, Sensu integrates nicely with the awesome PagerDuty tools for alerting.  You can find more information about Sensu and its architecture over at the docs page of their website.

“The Sensu way” involves using what is called a handler (for the uninitiated) to trigger an alert.  So for example, my setup involves a number of checks, which are run on each of my clients.  These checks have associated subscribers and handlers that report back to the Sensu server.  From there the Sensu server will run the handler(s) specified and do something with results of the check that was run on the Sensu client.

For my project I am using PagerDuty to generate alerts if disk space gets low or a process dies.  I will briefly run through the steps of how to set the PagerDuty integration up because there were a few roadblocks that I encountered when I set this up the first time.

This set of instructions assumes that you already have a PagerDuty account created and configured.  So the first step is to create a Service API check for Sensu.  Pick a suitable name and choose Use our API directly.  It should look similar to the following:

Now that we have an API key set up in PagerDuty we should be able to jump on the Sensu server and add in the apporpriate json to configure the Sensu handler to communicate with PagerDuty.  Place the following contents in /etc/sensu/conf.d/handlers/pagerduty.json.

{
 "pagerduty": {
   "api_key": "xxxxxx"
 },
 "handlers": {
   "pagerduty": {
     "type": "pipe",
     "command": "/etc/sensu/plugins/pagerduty.rb",
     "severities": [
       "critical",
       "ok"
       ]
     }
   }
}

I learned (the hard way) that the pagerduty.rb script won’t work out of the box.  It relies on a ruby gem called redphone.  It is easy enough to install and get working, just do a gem install redphone and you should be all set.

Next, go ahead and download the pagerduty.rb script to the appropriate location on the Sensu server:

cd /etc/sensu/plugins
wget -O /etc/sensu/plugins/pagerduy.rb https://raw.github.com/sensu/sensu-community-plugins/master/handlers/notification/pagerduty.rb

That should be it.  One good way to check if things are working and that the checks and handler are actually firing correctly is to tail the log file on both the client and server. On the server the log is located at /var/log/sensu/sensu-server.log and on the client machine at /var/log/sensu/sensu-client.log.

Bonus:  Chef integration

Of course all of this can be automated using Chef, which is ultimately what I ended up doing, so I will share some of the things that I learned in the process.  For starters, I am using the Sensu Chef cookbook, created by the maintainer of the Sensu project.  This cookbook exposes a few useful options for configuration Sensu.  You will need to clone the cookbook directly from the github repository to get the newest features that we need, as the Opscode version has not yet been updated to incorporate them.

Just add this line to your recipe before you call any of the Sensu resources/providers.

include_recipe "sensu::default"

The Sensu coobook exposes a number of nice resources that we can use in our recipes to deploy Sensu.  As an example if you wanted to clone the PagerDuty handler to the Chef server you would use something like the following in your recipe:

sensu_plugin "https://raw.githubusercontent.com/sensu/sensu-community-plugins/master/handlers/notification/pagerduty.rb"

Which will place the pagerduty.rb script into the appropriate directory automaitcally.  There are other options as well, but this should do the trick.  You can find some more examples here.

Define your pagerduty handler:

sensu_handler "pagerduty" do 
  type "pipe" 
  command "/etc/sensu/plugins/pagerduty.rb" 
  severities ["ok", "critical"] 
end

You will need to add this handler to each check that you want to receive an alert on, and you will also need to subscribe your host to that check as well.  Here is what an example check might look like:

sensu_check "check_ntp" do 
   command "/etc/sensu/plugins/check-procs.rb -p ntpd -C 1" 
   handlers ["pagerduty"] 
   subscribers ["core"] 
   interval 60 
   additional(:notification => "NTP is not running", :occurrences => 5) 
end

That’s all I have for now.  So far Sensu has been amazing, it is very flexible and the IRC channel an excellent resource.  The docs are nice as well.  Again, props to Sean Porter for creating an awesome new way to do monitoring.  I am still just flirting with the very top of the iceburg as far as the capabilites of Sensu go and will be revisiting this subject in the future.