Introduction to Grafana

June 4, 2014August 31, 2015 Josh Reichardt 1 Comment

Grafana is a beautiful dashboard for displaying various Graphite metrics through a web browser. Grafana is nice because it is simple to set up and maintain and is easy to use and displays metrics in a very nice Kibana like display style. I would like to walk readers through the basics of this tool because although it is a very new project (beginning of 2014) it has an enormous amount of potential and I honestly believe that there is enough functionality to put it into production.

The guys over at Hosted Graphite have just recently released hosted Grafana as part of their standard plan. If you are interested you can check it out over at Hosted Graphite.

One very nice feature of this product offering is that you do not need to worry about any of the behind the scenes details or intricacies of how all of the Graphite components work together. You just click a few buttons and you are ready to go. Highly recommended if you just need something that works, go check it out.

Anyway, Grafana gives you the ability to bolt on all kinds of bells and whistles to your graphs. For example, there is a nice little node.js tool called statsd written by the guys over at etsy that sort of expands the capabilities of Graphite. And since it hooks right in with Graphite, it makes it very simple to represent and output the various metrics into Grafana.

If you do choose to roll your own Grafana solution, there are a few gotchas that I was not aware of that I’d like to cover. The first, which should seem easy enough is that you need to run but Graphite and Grafana simultaneously. So that means you either need to create two virtual hosts depending on which web server you choose to use or you need to to create to different port bindings. The Grafana documentation has a few examples of how to create new port binding, which uses Apache as the web server but it is pretty easy to find examples of configs on Github and other sites if you are interested in using nginx as your web server instead. The important thing to remember is that you will want to have two sites listed in your sites-enabled directory inside of the apache directory. One for Graphite (which I moved to port 8080 for simplicity) and one for Grafana (which I stuck on port 80).

If you choose to use authentication there are a few extra headers that need to be written as well so that Grafana is accessible correclty. This is easy to add to either your nginx or you apache config.

Here is the adapted Apache configuration necessary to add in authentication, where website name is the globally reachable DNS name of your webserver:

Header set Access-Control-Allow-Origin "http://WEBSITENAME"
Header set Access-Control-Allow-Methods "GET, OPTIONS"
Header set Access-Control-Allow-Headers "origin, authorization, accept"
Header set Access-Control-Allow-Credentials true

<Location />
  AuthName "graphs restricted"
  AuthType Basic
  AuthUserFile /etc/apache2/htpasswd
  <LimitExcept OPTIONS>
    require valid-user
  </LimitExcept>
</Location>

That is almost the entire configuration that I have for the Grafana webserver. You will need to download and install an Apache lib tool to generate a hash to use for you AuthUserFile, directions can be pieced together from this. The Graphite configuration is only a little bit more complex and was created by Chef, so I will go ahead and post that here as well:

<VirtualHost *:8080>
  ServerName SERVER
  ServerAdmin [email protected]
  DocumentRoot "/opt/graphite/webapp"
  ErrorLog /opt/graphite/storage/log/webapp/error.log
  CustomLog /opt/graphite/storage/log/webapp/access.log common

  Header set Access-Control-Allow-Origin "*"
  Header set Access-Control-Allow-Methods "GET, OPTIONS"
  Header set Access-Control-Allow-Headers "origin, authorization, accept"

  WSGIScriptAlias / /opt/graphite/conf/wsgi.py

  <Directory /opt/graphite>
  <Files wsgi.py>
    Require all granted
  </Files>
  </Directory>

  <Location "/content/">
    SetHandler None
  </Location>

  <Location "/media/">
    SetHandler None
  </Location>

 # NOTE: In order for the django admin site media to work you
 # must change @DJANGO_ROOT@ to be the path to your django
 # installation, which is probably something like:
 # /usr/lib/python2.6/site-packages/django
 Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"

</VirtualHost>

Again pretty straight forward. If you choose to add authentication, it is done in a similar way.

The only other step is to add your metrics into Graphite. I am using Sensu, which I have written a bit about and plan to write more about in the future. Reference some of those writings to get an idea for metric gather and if you have any question let me know. I will be writing a post in the near future about gathering and aggregating metrics with Sensu. For now I will just assume that you already have a way to collect metrics.

Once you have everything set up you should be able to start playing around with the Grafana GUI. After you have your configurations ironed out pretty much everything else is done through the GUI which is nice.

To illustrate the power of Grafana, here are a few example dashboards I have built recently:

If you want to check out the Grafana project you can find more information either on their Github page or their website. The docs page on the main website is a great resource as well as the IRC channel. In fact, the IRC channel is probably the most preferable place to go for help because it isn’t overcrowded and the creator of Grafana is in there quite a bit.

Set up PagerDuty alerts in Sensu

May 21, 2014August 31, 2015 Josh Reichardt 7 Comments

I am currently in the midst of rolling a monitoring solution using Sensu and a handful of other tools, which I will be covering sporadically in the future. Onee important facet of any good monitoring solution is a reliable alerting method. Sensu uses a distributed approach to monitoring so all of the components are spread out rather than run as one monolithic system. So following this principle, Sensu integrates nicely with the awesome PagerDuty tools for alerting. You can find more information about Sensu and its architecture over at the docs page of their website.

“The Sensu way” involves using what is called a handler (for the uninitiated) to trigger an alert. So for example, my setup involves a number of checks, which are run on each of my clients. These checks have associated subscribers and handlers that report back to the Sensu server. From there the Sensu server will run the handler(s) specified and do something with results of the check that was run on the Sensu client.

For my project I am using PagerDuty to generate alerts if disk space gets low or a process dies. I will briefly run through the steps of how to set the PagerDuty integration up because there were a few roadblocks that I encountered when I set this up the first time.

This set of instructions assumes that you already have a PagerDuty account created and configured. So the first step is to create a Service API check for Sensu. Pick a suitable name and choose Use our API directly. It should look similar to the following:

Now that we have an API key set up in PagerDuty we should be able to jump on the Sensu server and add in the apporpriate json to configure the Sensu handler to communicate with PagerDuty. Place the following contents in /etc/sensu/conf.d/handlers/pagerduty.json.

{
 "pagerduty": {
   "api_key": "xxxxxx"
 },
 "handlers": {
   "pagerduty": {
     "type": "pipe",
     "command": "/etc/sensu/plugins/pagerduty.rb",
     "severities": [
       "critical",
       "ok"
       ]
     }
   }
}

I learned (the hard way) that the pagerduty.rb script won’t work out of the box. It relies on a ruby gem called redphone. It is easy enough to install and get working, just do a gem install redphone and you should be all set.

Next, go ahead and download the pagerduty.rb script to the appropriate location on the Sensu server:

cd /etc/sensu/plugins
wget -O /etc/sensu/plugins/pagerduy.rb https://raw.github.com/sensu/sensu-community-plugins/master/handlers/notification/pagerduty.rb

That should be it. One good way to check if things are working and that the checks and handler are actually firing correctly is to tail the log file on both the client and server. On the server the log is located at /var/log/sensu/sensu-server.log and on the client machine at /var/log/sensu/sensu-client.log.

Bonus: Chef integration

Of course all of this can be automated using Chef, which is ultimately what I ended up doing, so I will share some of the things that I learned in the process. For starters, I am using the Sensu Chef cookbook, created by the maintainer of the Sensu project. This cookbook exposes a few useful options for configuration Sensu. You will need to clone the cookbook directly from the github repository to get the newest features that we need, as the Opscode version has not yet been updated to incorporate them.

Just add this line to your recipe before you call any of the Sensu resources/providers.

include_recipe "sensu::default"

The Sensu coobook exposes a number of nice resources that we can use in our recipes to deploy Sensu. As an example if you wanted to clone the PagerDuty handler to the Chef server you would use something like the following in your recipe:

sensu_plugin "https://raw.githubusercontent.com/sensu/sensu-community-plugins/master/handlers/notification/pagerduty.rb"

Which will place the pagerduty.rb script into the appropriate directory automaitcally. There are other options as well, but this should do the trick. You can find some more examples here.

Define your pagerduty handler:

sensu_handler "pagerduty" do 
  type "pipe" 
  command "/etc/sensu/plugins/pagerduty.rb" 
  severities ["ok", "critical"] 
end

You will need to add this handler to each check that you want to receive an alert on, and you will also need to subscribe your host to that check as well. Here is what an example check might look like:

sensu_check "check_ntp" do 
   command "/etc/sensu/plugins/check-procs.rb -p ntpd -C 1" 
   handlers ["pagerduty"] 
   subscribers ["core"] 
   interval 60 
   additional(:notification => "NTP is not running", :occurrences => 5) 
end

That’s all I have for now. So far Sensu has been amazing, it is very flexible and the IRC channel an excellent resource. The docs are nice as well. Again, props to Sean Porter for creating an awesome new way to do monitoring. I am still just flirting with the very top of the iceburg as far as the capabilites of Sensu go and will be revisiting this subject in the future.

Set up PEM key authentication

May 15, 2014August 31, 2015 Josh Reichardt 1 Comment

Many times it is useful to keys to authenticate to your servers. This can dramatically improve security and is a great way to manage servers in bulk as well. You just need to keep track of your keys rather than having to remember a large number of passwords. The steps to get PEM key authentication are fairly straight forward but it never hurts to walk through the process of getting them set up correctly.

Side Note: I’d like to also mention briefly, that I have these steps set up to work with Chef, so every server that gets deployed using Chef will use PEM keys out of the box, which works out very nicely. If you’re interested I can expound on that topic a little more, just let me know.

The first step in the process is to generate some keys using openssl. If you don’t have openssl go download and install it. If you do have openssl but haven’t updated in ahwile, please update to avoid the heartbleed vulnerability that was recently exploited (nearly all distributors have released the patched version at this point so it should be trivial).

We want to generate our key and create a PEM file out of it. Here are the steps:

cd ~/.ssh
ssh-keygen -t dsa -b 1024
openssl dsa -in id_dsa -outform pem > test.pem
cat ida_dsa >> authorized_keys

You can leave the values blank (default) in the ssh-keygen.

Now you should have similar listings in your ~/.ssh directory:

authorized_keys – This is the public key that the pem file gets authenticated against
id_dsa – This is the private portion of the key that we generated in the steps above
id_dsa.pub – This is the public key section that is used when authenticating
test.pem – this is the file that will be used to authenticate. Essentially the private key minus the pass phrase

Now you just need to copy the test.pem file that was just generated to a different host in order to log in with your PEM key using scp or rsync. Once that is done, the command to connect to the remote host using your key should look similar to the following:

ssh -i /path/to/pem user@server-name

Next steps. At this point you should have a working pem authentication on your server. It is probably a good idea at this point to start looking at hardening the security as well as the SSH configuration on the host. Small things can go a long way. For example disabling root login, disabling password authentication, etc. will stop a very large amount of attacks from hitting your server now that you are authenticating with pem keys.

Using a self signed cert with Nginx

May 5, 2014August 31, 2015 Josh Reichardt 7 Comments

After the recent heart bleed incident (which I’m sure many of you well remember) I had to reassign some certificates. It turns out that this was a great opportunity to create a blog post. Since I do not create and assign certs very frequently it is a good opportunity to take some notes and hopefully ease the process for others. After patching the vulnerable version of Openssl, there are really only a few steps needed to accomplish this. Assuming you already have nginx installed, which is trivial to do on Ubuntu, the first step is to create the necessary crt and key files.

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/cert.key -out /etc/nginx/cert.crt

Next you will need to tell nginx to load up you new certs in its config. Here is an example of what the server block in you /etc/nginx/site-available config might look like. Notice the ssl_certificate and ssl_certificate_key files correspond to the cert files we created above, which we stuck in the /etc/nginx directory. If you decide to place these certs in a different location you will need to modify your config file to reflect the location.

server {

listen *:443; 
ssl on; 
ssl_certificate cert.crt; 
ssl_certificate_key cert.key; 
ssl_session_timeout 5m; 
ssl_protocols SSLv3 TLSv1; 
ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP; 
ssl_prefer_server_ciphers on;

}

Just to cover all our bases here we will also redirect any requests that come in to port 80 (default web) back to 443 for ssl. The is a simple addition and will add an additional layer of security.

server { 
listen 80; 
return 301 https://$host$request_uri; 
}

The final step is to reload your configuration and test to make sure everything works.

sudo service nginx reload

If your nginx fails to reload, more than likely there is some sort of configuration or syntax error in your config file. Comb through it for any potential errors or mistakes. Once your config is loaded properly you can check your handy work by attempting to hit your site using http://. If your config is working properly it should automatically redirect you to https://.

That’s all it takes. I think it might be a good exercise to try something like this with Chef but for now this process works okay by hand. Let me know what you think or if this can be improved.

Setting up a private git repo in Chef

April 22, 2014August 31, 2015 Josh Reichardt 28 Comments

It turns out that cloning and managing private git repo’s in Chef is not as easy as it looks. That said, I have a method that works. I have no idea if this is the preferred method or if there are any easier ways but this worked for me, so let me know if there is an easier way and I will be glad to update this post.

First, I’d like to give credit where it is due. I used this post as a template as well as the SSH wrapper section in the deploy documentation on the Chef website.

The first issue is that when you connect to github via SSH it wants the Chef client to accept its public fingerprint. By default, if you don’t modify anything SSH will just sit there waiting for the fingerprint to be accepted. That is why the SSH Git wrapper is used, it tells SSH on the Chef client that we don’t care about the authentication to the github server, just accept the key. Here’s what my ssh git wrapper looks like:

 #!/bin/bash 
 exec ssh -o "StrictHostKeyChecking=no" -i "/home/vagrant/.ssh/id_rsa" $1 $2

You just need to tell your Chef recipe to use this wrapper script:

# Set up github to use SSH authentication 
cookbook_file "/home/vagrant/.ssh/wrap-ssh4git.sh" do 
  source "wrap-ssh4git.sh" 
  owner "vagrant" 
  mode 00700 
end

The next problem is that when using key authentication, you must specify both a public and a private key. This isn’t an issue if you are running the server and configs by hand because you can just generate a key on the fly and hand that to github to tell it who you are. When you are spinning instances up and down you don’t have this luxury.

To get around this, we create a couple of templates in our cookbook to allow our Chef client to connect to github with an already established public and private key, the id_rsa and id_rsa.pub files that are shown. Here’s what the configs look like in Chef:

# Public key 
template "/home/vagrant/.ssh/id_rsa.pub" do 
  source "id_rsa.pub" 
  owner "vagrant" 
  mode 0600 
end 
 
# Private key 
template "/home/vagrant/.ssh/id_rsa" do 
  source "id_rsa" 
  owner "vagrant" 
  mode 0600 
end

After that is taken care of, the only other minor caveat is that if you are cloning a huge repo then it might timeout unless you override the default timeout value, which is set to 600 seconds (10 mins). I had some trouble finding this information on the docs but thanks to Seth Vargo I was able to find what I was looking for. This is easy enough to accomplish, just use the following snippet to override the default value

timeout 9999

That should be it. There are probably other, easier ways to accomplish this and so I definitely think the adage “there’s more than one way to skin a cat” applies here. If you happen to know another way I’d love to hear it.