Transitioning from bash to zsh

August 16, 2014August 31, 2015 Josh Reichardt 1 Comment

I have know about zsh for a long time now but have never really had a compelling reason to switch my default shell from bash until just recently, I have been hearing more and more people talking about how powerful and awesome zsh is. So I thought I might as well take the dive and get started since that’s what all the cool kids seem to be doing these days. At first I thought that changing my shell was going to be a PITA with all the customizations and idiosyncrasies that I have grown accustomed to using bash but I didn’t find that to be the case at all when switching to zsh.

First and foremost, I used a tool called oh-my-zsh to help with the transition. If you haven’t heard about it yet, oh-my-zsh aims to be a sort of framework for zsh. This project is a nice clean way to get started with zsh because it give you a nice set of defaults out of the the box without having to do much configuration or tweaking and I found that many of my little tricks and shortcuts were already baked in to to oh-my-zsh, along with a ton of other settings and customizations that I did not have using bash.

From their github page:

oh-my-zsh is an open source, community-driven framework for managing your ZSH configuration. It comes bundled with a ton of helpful functions, helpers, plugins, themes, and few things that make you shout…

Here are just a few of the improvement that zsh/oh-my-zsh offer:

Improved tab completion
persistent history across all shells
Easy to use plugin system
Easy to use theme system
Autocorrect

The most obvious difference that I have noticed is the improved, out of the box tab completion, which I think should be enough on its own to convince you! On top of that, most of my tricks and customizations were already turned on with oh-my-zsh. Another nice touch is that themes and plugins come along as part of the package, which is really nice for easing the transition.

So after spending an afternoon with zsh I am convinced that it is the way to go (at least for my own workfolw). Of course there are always caveats and hiccups along the way as I’ve learned there are with pretty much everything.

Tuning up tmux

Out of the box, my tmux config uses the default shell, which happens to be bash. So I needed to modify my ~/.tmux.conf to reflect the switch over the zsh. It is a pretty straight forward change but is something that you will need to make note of kif you use tmux and are transitioning in to using zsh.

set-option -g default-shell /usr/bin/zsh

I am using Ubuntu 14.04, so my zsh is installed to /usr/bin/zsh. The other thing that you will need to do is make sure you kill any stale tmux processes after updating to zsh. I found one running in the background that was blocking me from using the new coonfig.

Goodies

There is a nice command cheat sheet for zsh. Take some time to learn these shortcuts and aliases, they are great time savers and are very usefull.

oh-my-zsh comes bundled up with a large number of goodies. At the time of this writing there were 135 plugins as well as a variety of themes. You can check the plugins wiki page for descriptions for the various plugins. To turn on a specific plugin you will need to add it to your ~/.zshrc config file. Find the following line in your config.

plugins=(git)

and add plugins separated by spaces

plugins (git vagrant chef)

You will need to reload the config for the changes to be picked up.

source ~/.zshrc

Most themes are hosted on the wiki, but there is also a web site dedicated to displaying the various themes, which is really cool. It does a much better job of showing differences between various themes. You can check it out here. Themes function in a similar way to plugins. If you want to change your theme, edit your ~/.zshrc file and select the desired them.

ZSH_THEME="clean"

You will need to reload your config for this option as well.

source ~/.zshrc

Conclusion

If you haven’t already made the switch to zsh I recommend that you at least experiment and play around with it before you make any final decisions. You may be set in your ways and happy with bash or any other shell that you are used to but for me, all the awesomeness changed my opinion and decide to reevaluate my biases. If you’re worried about making the switchin, using oh-my-zsh makes the transition so painless there is practically no reason not to try it out.

This post is really just the tip of the iceberg for the capabilities of this shell, I just wanted to expose readers to all of its glory. Zsh offers so much more power and customization than I have covered in this post and is an amazing productivity tool with little learning overhead.

Let me know if you have any awesome zsh tips or tweaks that folks should know about.

7 useful but hard to remember Linux commands

August 9, 2014August 31, 2015 Josh Reichardt 22 Comments

I have found myself using these commands over and over so I decided I’d take the time to go ahead and document them for future me as well as readers because I find these commands pretty useful. I just always manage to forget them, hence the title of the post. The smart thing to do would be to create aliases for these commands but I have just been too lazy and some of them are run across different servers so it isn’t always a convenient option.

Anyway, let’s go ahead and run through the commands before I forget…

1) du -ah / | sort -n -r | head -n 50

This one is really handy for debugging space issues. It will list the top 50 files according to file size, with the largest at the top of the list.Notice the “/” will specify the location to search so you can easily modify this one to search different locations, like “/var/log” for example if you are having trouble with growing log files.

1.5) du -sh /*

This will quickly give you an idea of how how your disk space has been allocated. Definitely handy when you are troubleshooting.

2) git checkout — .

I don’t use this one very often, which is probably why I manage to forget it so easily. But I really like it. Sometimes I will be working on a git repo across different machines at the same time and will run in to conflicts committing to the repo or more likely I committed changes on one machine and just need to pull down the newest changes but can’t since I have made modifications. For those scenarios you can run the above command quickly reset your git changes quickly and easily.

3) tmux kill-window -t 3

i use tmux for my terminal and window manager on all my workstations and love it. If you haven’t heard of it, take a look here. Sometimes the sessions can get stuck so it becomes necessary to close the window without destroying the tmux session. Again, this doesn’t happen very often so it is sometimes hard for me to remember the exact syntax but this one is a handy little trick for managing tmux windows and sessions.

4) grep -r “text”

I know, I should really have this one memorized by now. I am trying to remember but I don’t find myself using this one all that often even though it is really powerful and useful. This will essentially search through every file recursively and spit out the text pattern that you feed to it.

5) kill $(pgrep process)

This one is handy when there are a large number of stuck processes and you need to blow them all out with one command. For example if the chrome browser ever gets stuck with a million tabs open, there are likely a large number of processes all with the same – or similar names. If you pass all or part of the process name in to this command pgrep will find them and kill will destroy them

6) docker rm $(docker ps -a -q)

I have been using Docker more and more recently and every once in awhile I find myself with a large number of dead Docker processes that need to be cleaned up. This command will blow out all of these stale processes at once. This is nice because Docker processes take up a large amount of disk space and often times can fill up your drives without you being aware. I have been able to reclaim large amounts of disk space with this command.

7) watch -n 10 df -ah

This is another good one for checking disk space issues. It will update you every ten seconds with the disk utilization of the system. Pretty straight forward but a great tool to help troubleshooting space issues.

That’s all I have for now, there are lots more but these are the most useful ones that I find myself forgetting the most often, hopefully this post will serve as a nice reminder. If you have any cool or useful commands that you would like to share feel free to comment and I will update the post to include them.

Chef data bags with Test Kitchen

June 30, 2014August 31, 2015 Josh Reichardt 8 Comments

As a step towards integrating your Chef cookbooks with Jenkins CI and your testing/release pipeline it is important to make sure that local changes pass unit and integration tests before being accepted and committed into version control. For example, when running test kitchen it is important to fully simulate what data bags and encrypted data bags are doing on a local box for many tests to pass correctly. So, today I would like to focus on a stumbling block towards Jenkins and integration testing that I ran in to recently. There are a few lessons that I learned along the way that I would like to share to help clarify things a little bit because there wasn’t much good info out there on how to do this.

First, I need to give credit where it is due. This post was a great resource in my journey to find a solution to my test kitchen data bag issue.

The largest roadblock I found along the way was that the version of test kitchen I was using was being shipped with chef-solo as the primary driver. There has been a lot of discussion around this topic lately and (from what I understand) has pretty much become the general consensus within the Chef community that chef-solo should be replaced by chef-zero. There are a number of advantages to using chef-zero instead of chef-solo, including a lesson I learned the hard way, which is that chef-zero has the ability to act as a stand alone Chef server – unlocking the ability to store data bags and encrypted data bags without having to do any sort of wacky hacking to get Chef to compile and converge correctly.

There was a good post written recently that expounds more on the benefits of using chef-zero instead of chef-solo. It is here, and is definitely worth the read if you are interested in learning more about the benefits of chef-zero.

So with that knowledge in mind, here is what a newly updated sample .kitchen.yml file might look like:

--- 
driver: 
 name: vagrant 
 
provisioner: 
 name: chef_zero 
 
platforms: 
 - name: ubuntu-13.10-i386 
 - name: centos-6.4-i386 
 
suites: 
 - name: default 
 data_bags_path: "test/integration/data_bags" 
 run_list: 
 - recipe[recipe-to-test] 
 attributes:

It’s a pretty straight forward config. The biggest change that you will notice in this config is that instead of using chef-solo as the provisioner it has been changed to chef-zero – I now know that it makes all the difference in the world. The next big change to observe is the data_bags_path in the suites section. This bit of configuration basically tells the Chef provisioner to go look at the specified file path when chef-zero spins up and use that to store data bag, encrypted data bag or other information that potentially would live on the Chef server that client’s would use.

So in the test/integration/data_bags directory I have a directory and json file inside that directory for the specific data I am interested in, called sensu/ssl.json. This file essentially contains the same information that is stored on the Chef server about the ssl certificates used for live hosts in the production environment, just mirrored into a sandbox/integration testing environment.

If you’re interested, here is a sample of what the ssl.json file might look like:

{ 
 "id": "ssl", 
 "server": { 
 "key": "-----BEGIN RSA PRIVATE KEY-----gM
 "cert": "-----BEGIN CERTIFICATE-----gM
 "cacert": "-----BEGIN CERTIFICATE-----gM
 }, 
 "client": { 
 "key": "-----BEGIN RSA PRIVATE KEY-----gM
 "cert": "-----BEGIN CERTIFICATE-----gM
 } 
}

Note that the “id” is “ssl”. As far as I know the file name must match up to the id when you are creating this json file.

Now you should be able to create and converge your test recipe with test kitchen:

kitchen create ubuntu
kitchen converge ubuntu

If you have any difficulty, let me know. I tried to be thorough in this write up but could have accidentally skipped important information. The main keys or takeaways though should be 1) use chef-zero wherever possible and 2) make sure you have your data bag paths and files created correctly and referenced correctly in your .kitchen.yml file. Finally, if you are still having issues, make sure you have triple checked the spelling and json syntax of your paths and configs.

Leveraging Nagios Plugins with Chef and Sensu

June 20, 2014August 31, 2015 Josh Reichardt 7 Comments

Setting up Nagios plugins to run in a Sensu and Chef managed environment is straightforward and uncomplicated. For example, I recently have been interested in monitoring SSL certificate date expiration and it just so happens that the Nagios check_http plugin does exactly what I’m looking for.

The integration between Sensu and the Nagios plugins is very nice. For convenience in our Sensu environment, we like to put the additional Nagios plugins on to all of the systems we monitor because the footprint is negligible and it allows for some nice flexibility of services and checks to monitor should an additional service get added to a server in the future that we hadn’t anticipated. For the amount of effort it takes to get the checks onto the server and to get working, adding the Nagios plugins is totally worth the effort.

The first step is to add the Nagios plugins to your Chef recipe. I am using a generic Chef recipe for my Sensu clients that takes care of some of the more tedious tasks including downloading the appropriate scripts and checks for the clients to run as well as some other dependencies and items that Sensu likes to have. Luckily there is a public Debian package available for installing the Nagios plugins so it easy to add them. Just add this snippet into your Chef recipe for Sensu clients:

apt_package "nagios-plugins" do 
 action :install 
end

After you run your next chef-client job you will have access to a variety of checks provided by the Nagios plugins package as illustrated below.

There are a number of examples available but to run the check_http for cert expiration by hand you can run this command:

/usr/lib/nagios/plugins/check_http -H <sitename> -C 30,10

Where <sitename> is the URL of the website you would like to check. Now that we are able to run this check manually, go ahead and roll that in to your Chef recipe for Sensu. An example of this might look similar to the following:

sensu_check "check_web" do 
  command "/usr/lib/nagios/plugins/check_http -H localhost -C 30,10" 
  handlers ["pagerduty", "slack"] 
  subscribers ["core"] 
  interval 60
  standalone true
  additional(:notification => "Certificate will expire soon", :occurrences => 5) 
end

You may not want to run this check on every host so it may be a good idea to run this check as a stand alone check. It is simple enough to add this snippet in to any recipe and tack on the “standalone true” attribute to the sensu_check resource. I have an example of what this standalone attribute looks like in the example above for reference.

Adding in Nagios plugins gives you a very nice set of additional tools to add to your monitoring arsenal for not that much effort. You never know when something from the Nagios plugins might come in handy so I suggest you try them out. There are many other uses for the Nagios plugins so I suggest taking a look.

Set up PEM key authentication

May 15, 2014August 31, 2015 Josh Reichardt 1 Comment

Many times it is useful to keys to authenticate to your servers. This can dramatically improve security and is a great way to manage servers in bulk as well. You just need to keep track of your keys rather than having to remember a large number of passwords. The steps to get PEM key authentication are fairly straight forward but it never hurts to walk through the process of getting them set up correctly.

Side Note: I’d like to also mention briefly, that I have these steps set up to work with Chef, so every server that gets deployed using Chef will use PEM keys out of the box, which works out very nicely. If you’re interested I can expound on that topic a little more, just let me know.

The first step in the process is to generate some keys using openssl. If you don’t have openssl go download and install it. If you do have openssl but haven’t updated in ahwile, please update to avoid the heartbleed vulnerability that was recently exploited (nearly all distributors have released the patched version at this point so it should be trivial).

We want to generate our key and create a PEM file out of it. Here are the steps:

cd ~/.ssh
ssh-keygen -t dsa -b 1024
openssl dsa -in id_dsa -outform pem > test.pem
cat ida_dsa >> authorized_keys

You can leave the values blank (default) in the ssh-keygen.

Now you should have similar listings in your ~/.ssh directory:

authorized_keys – This is the public key that the pem file gets authenticated against
id_dsa – This is the private portion of the key that we generated in the steps above
id_dsa.pub – This is the public key section that is used when authenticating
test.pem – this is the file that will be used to authenticate. Essentially the private key minus the pass phrase

Now you just need to copy the test.pem file that was just generated to a different host in order to log in with your PEM key using scp or rsync. Once that is done, the command to connect to the remote host using your key should look similar to the following:

ssh -i /path/to/pem user@server-name

Next steps. At this point you should have a working pem authentication on your server. It is probably a good idea at this point to start looking at hardening the security as well as the SSH configuration on the host. Small things can go a long way. For example disabling root login, disabling password authentication, etc. will stop a very large amount of attacks from hitting your server now that you are authenticating with pem keys.