Lint your Dockerfiles with Hadolint

If you haven’t already gotten in to the habit of linting your Dockerfiles you should.  Code linting is a common practice in software development which helps find, identify and eliminate issues and bugs before they are ever able to become a problem.  One of the main benefits of linting your code is that it helps identify and eliminate nasty little bugs before they ever have a chance to become a problem.

Taking the concept of linting and appyling it to Dockerfiles gives you a quick and easy way to identify errors quickly before you ever build any Docker images.  By running your Dockerfile through a linter first, you can ensure that there aren’t any structural problems with the logic and instructions specified in your Dockerfiles.  Linting your files is fast and easy (as I will demonstrate) and getting in the habit of adding a linting step to your development workflow is often very useful because not only will the linter help identify hidden issues which you might not otherwise catch right away but it can potentially save hours of troubleshoot later on, so there is some pretty good effort to benefit ratio there.

There are serveral other Docker linting tools around:

But in my experience, these tools have either been overly complicated, don’t detect/catch as many errors and in general just don’t seem to work as well or have as much polish as Hadolint.  I may just have a skewed perspective of these tools but this was my experience when I tried them, so take my evaluation with a grain of salt.  Definitely let me know if you have experience with any of these tools, I may just need to revisit them to get a better perspective.

With that being said, Hadolint offers everything I need and is very easy to use and get started with and makes linting your Dockerfiles is trivially easy, which counts for the most points in my experience.  Another bonus of Hadolint is that the project is fairly active and the author is friendly, so if there are things you’d like to see get added, it shouldn’t be too hard to get some movement on.  You can check out the project on Github for details about how to install and run Hadolint.

Below, I will go over how to get setup and started as well as some basic usage.

Install

If you use Mac OS X there is a brew formula for installing Hadolint.

brew update
brew install hadolint

If you are a Windows user, for now you will have to run Hadolint from within a Docker conainer.

docker run --rm -i lukasmartinelli/hadolint < Dockerfile

If you feel comfortable with the source code you can try building the code locally.  I haven’t attempted that method, so I don’t have instructions here for how to do it.  Go check out the project if you are interested.

Usage

Hadolint helps you find syntax errors and other mistakes that you may not notice in your Dockerfiles otherwise.  It’s easy to get started.  To run Hadolint run the following.

hadolint Dockerfile

If there are any issues, Hadolint will print out the rule number as well as a blurb describing what could potentially be wrong.

DL4000 Specify a maintainer of the Dockerfile
L1 DL3007 Using latest is prone to errors if the image will ever update. Pin the version explicitly to a release tag.
L3 DL3013 Pin versions in pip. Instead of `pip install <package>` use `pip install <package>==<version>`

As with any linting tool, you will definitely get some false positives at some point so just be aware of items that can potentially be ignored.  There is an issue open on Github right now to allow Hadolint to ignore certain rules, which will help eliminate some of the false positives.  For example, in the above snippet, we don’t necessarily care about the maintainer missing so it will be nice to be able to ignore that line.

Here is a complete reference for the all of the linting rules.  The wiki gives examples of what is wrong and how to fix things, which is very helpful.  Additionally, the author is welcoming ideas for additional things to check, so if you have a good idea for a linting rule open up an issue.

That’s it for now.  Good luck and happy Docker linting!

Read More

Useful Vim Plugins

This post is mostly a reference for folks that are interested in adding a little bit of extra polish and functionality to the stock version of Vim.  The plugin system in Vim is a little bit confusing at first but is really powerful once you get past the initial learning curve.  I know this topic has been covered a million times but having a centralized reference for how to set up each plugin is a little bit harder to find.

Below I have highlighted a sample list of my favorite Vim plugins.  I suggest that you go try as many plugins that you can to figure out what suits your needs and workflow best.  The following plugins are the most useful to me, but certainly I don’t think will be the best for everybody so use this post as a reference to getting started with plugins and try some out to decide which ones are the best for your own environment.

Vundle

This is a package manager of sorts for Vim plugins.  Vundle allows you to download, install, search and otherwise manage plugins for Vim in an easy and straight forward way.

To get started with Vundle, put the following configuration at THE VERY TOP of your vimrc.

set nocompatible              " be iMproved, required
filetype off                  " required
set rtp+=~/.vim/bundle/Vundle.vim
call vundle#rc()
"" let Vundle manage Vundle
Bundle 'gmarik/Vundle.vim'
...

Then you need to clone the Vundle project in to the path specified in the vimrc from above.

git clone https://github.com/gmarik/Vundle.vim.git ~/.vim/bundle/Vundle.vim

Now you can install any other defined plugins from within Vim by  running :BundleInstall.  This should trigger Vundle to start downloading/updating its list of plugins based on your vimrc.

To install additional plugins, update your vimrc with the plugins you want to install, similar to how Vundle installs itself as shown below.

"" Example plugin
Bundle 'flazz/vim-colorschemes'

Color Schemes

Customizing the look and feel of Vim is a very personal experience.  Luckily there are a lot of options to choose from.

The vim-colorschemes plugin allows you to pick from a huge list of custom color schemes that users have put together and published.  As illustrated above you can simply add the repo to your vimrc to gain access to a large number of color options.  Then to pick one just add the following to your vimrc (after the Bundle command).

colorscheme xoria256

Next time you open up Vim you should see color output for the scheme you like.

Syntastic

Syntastic is a fantastic syntax highlighter and linting tool and is easily the best syntax checker I have found for Vim.  Syntastic offers support for tons of different languages and styles and even offers support for third party syntax checking plugins.

Here is how to install and configure Syntastic using Vundle.  The first step is to ddd Syntastic to your vimrc,

" Syntax highlighting
 Bundle 'scrooloose/syntastic'

There are a few basic settings that also need to get added to your vimrc to get Syntastic to work well.

" Syntastic statusline
 set statusline+=%#warningmsg#
 set statusline+=%{SyntasticStatuslineFlag()}
 set statusline+=%*
 " Sytnastic settings
 let g:syntastic_always_populate_loc_list = 1
 let g:syntastic_auto_loc_list = 1
 let g:syntastic_check_on_open = 1
 let g:syntastic_loc_list_height=5
 let g:syntastic_check_on_wq = 0
 " Better symbols
 let g:syntastic_error_symbol = 'XX'
 let g:syntastic_warning_symbol = '!!'

That’s pretty much it.  Having a syntax highlighter and automatic code linter has been a wonderful boon for productivity.  I have saved  myself so much time chasing down syntax errors and other bad code.  I definitely recommend this tool.

YouCompleteMe

This plugin is an autocompletion tool that adds tab completion to Vim, giving it a really nice IDE feel.  I’ve only tested YCM out for a few weeks now but have to say it doesn’t seem to slow anything down very much at all, which is nice.  An added bonus to using YCM with Syntastic is that they work together so if there are problems with the functions entered by YCM, Syntastic will pick them up.

Here are the installation instructions for Vundle.  The first thing you will need to do is add a Vundle reference to your vimrc.

"" Autocomplete
Bundle 'Valloric/YouCompleteMe'

Then, in Vim, run :BundleInstall – this will download the git repo for YouCompleteMe.  Once the repo is downloaded you will need a few other tools installed to get things working correctly.  Check the official documentation for installation instruction for your system.  On OS X you will need to have Python, cmake, MacVim and clang support.

xcode-select --install
brew install cmake

Then, to install YouCompleteMe.

cd ~/.vim/bundle/YouCompleteMe
git submodule update --init --recursive (not needed if you use Vundle)
./install.py --clang-completer

vim-better-whitespace

Highlights pesky whitespace automatically.  This one is really useful to just have on in the background to help you catch whitespace mistakes.  I know I make a lot of mistakes with regards to missing whitespace so having this is just really nice.

To install it.

"" Whitespace highlighting
Bundle 'ntpeters/vim-better-whitespace'

That’s it.  Vundle should handle the rest.

ctrlp / nerdtree

These tools are useful for file management and traversal.  These plugins become more powersful when you work with a lot of files and move around different directories a lot.  There is some debate about whether or not to use nerdtree in favor of the built in netrw.  Nonetheless, it is still worth checking out different file browsers and see how they work.

Check out Vim Unite for a sort of hybrid file manager for fuzzy finding like ctrlp with additional functionality, like the ability to grep files from within Vim using a mapped key.

Bonus – Shellcheck

This is a shell and bash linting tool that integrates with vim and is great.  Bash is notoriously difficult to read and debug and the shellcheck tools helps out with that a lot.

Install shellcheck on your system and syntastic will automatically pick up the installation and automatically do its linting whenever you save a file.  I have been writing a lot of bash lately and the shellcheck tool has been a godsend for catching mistakes, and especially useful in Vim since it runs all the time.

By combining the powers of a good syntax highlighter and a good solid understanding of Bash you should be able to be that much more productive once you get used to having a build in to syntax and style checker for your scripts.

Read More

What is DevOps?

Since landing myself in a new and unexplored terrain as a freshly minted DevOps admin, I have been thinking a lot about what exactly DevOps is and how I will translate my skills moving into the position.  I am very excited to have the opportunity to work in such a new and powerful area of IT (and at such a sweet company!) but really think I need to lay out some of the groundwork behind what DevOps is, to help strengthen my own understanding and hopefully to help others grasp some of the concepts and ideas behind it.

I have been hearing more and more about DevOps philosophy and its growing influence and adoption in the world of IT, especially in fast paced, cloud and start up companies.  From what I have seen so far, I think I people really need to start looking at the impact that DevOps is making in the realm of system administration and how to set themselves up to succeed in this profession moving forward.

Here is the official DevOps description on Wikipedia:

DevOps is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.  DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.

While this is a solid description, there still seems to be a large amount of confusion about what exactly DevOps is so I’d like to address some of the key ideas and views that go along with its mentality and application to system administration.  To me, DevOps can be thought of as a combination of the best practices that a career in operations has to offer with many of the concepts and ideas that are used in the world of development.  Especially those derived from Agile and Scrum.

The great thing about DevOps is that since it is so new, there is really no universally accepted definition of what it is limited to.  This means that those who are currently involved in the DevOps development and adoption are essentially creating a new discipline, adding to it as they go.  A current DevOps admin can be described in simple terms as a systems admin that works closely with developers to decrease the gap between operations and development.  But that is not the main strength that DevOps offers and really just hits the tip of the ice burg for what DevOps actually is and means.

For one, DevOps offers a sort of cultural shift in the IT environment.  Traditionally in IT landscapes, there has been somewhat of a divide between operations and development.  You can think of this divide as a wall built between the dev and the ops teams either due to siloing of job skills and responsibilities or how the organization at broader perspective operates.  Because of this dissection of duties, there is typically little to no overlap between the tool sets or thought process between the dev or ops teams, which can cause serious headaches trying to get products out the door.

So how do you fix this?

In practical application, the principals of DevOps can put into practice using things like Continuous Integration tools , configuration management, logging and monitoring, creating a standardized test, dev and QA environments, etc.  The DevOps mindset and culture has many of its roots in environments of rapid growth and change.  An example of this philosophy put in to practice is at start up companies that rely on getting their product to market as quickly as smoothly as possible.  The good news is that larger enterprise IT environments are beginning to look at some of the benefits of this approach and starting to tear down the walls of the silos.

Some of the benefits of DevOps include:

  • Increased stability in your environment (embracing config management and version control)
  • Faster resolution of problems (decrease MTT)
  • Continuous software delivery (increasing release frequency brings ideas to market faster)
  • Much faster software development life cycles
  • Quicker interaction and feedback loops for key business stakeholders
  • Automate otherwise cumbersome and tedious tasks to free up time for devs and ops teams

These are some powerful concepts.  And the benefits here cannot be underestimated because at the end of the day the company you work for is in the business of making money.  And the faster they can make changes to become more marketable and competitive in the market the better.

One final topic I’d like to cover is programming.  If you are even remotely interested in DevOps you should learn to program, if you don’t know already.  This is the general direction of the discipline and if you don’t have a solid foundation to work from you will not be putting yourself into the best position to progress your career.  This doesn’t mean you have to be a developer, but IMO you have have to at least know and understand what the developers are talking about.  It is also very useful to know programming for all of the various scripting and automation tasks that are involved in DevOps.  Not only will you be able to debug issues with other software, scripts and programs but you will be a much more valuable asset to your team if you can be trusted to get things done and help get product shipped out the door.

Read More

Why Computer Science degrees translate to System Administration

I run across a lot of articles and posts that talk about how a degree in Computer Science is usually irrelevant to system administration and that you are just as well off with another degree or no degree at all. I think that line of logic is very short sighted and today I am going (or at least attempt) to explain why. By no means am I criticizing these approaches, in fact I believe in the logic that there is more than one way to skin a cat, and I have found many other highly successful admins that have reached their positions by these alternate means. I just want to quickly clarify that I am not advising readers that taking the CS route to becoming a system admin is the only, right way to go, I am simply relating my own experiences in system administration to my background in CS and making a case of why pursuing a degree in Computer Science, or any other degree in engineering for that matter isn’t going to hurt your chances of becoming a sysadmin.

When you think of Computer Science you think of programming or maybe math, at least I do. Most CS programs these days have a heavy orientation towards programming and the scientific and mathematic applications of programming as it applies to the world around us. As an aside, I am beginning to see many more programs that are tailored to specific disciplines inside the realm of IT which looks promising. This is a great hybrid approach in my opinion because it gives students a chance to look at a few alternate options. Coding isn’t my passion so having an option to become a system administrator without the amount of intense coding from a CS program looks like an attractive approach.

It is true that many of the mundane daily tasks related to system administration don’t involve 8 hours a day of reading and writing code. Because of this I think it is important to characterize and distinguish a sysadmin as somebody who relies on software tools and programming to solve problems and technical challenges but doesn’t necessarily devote all of their time and energy to living in and interacting with code. The relationship of the sysadmin to programming is more of an indirect one, though still very important.

The farther along I wander on in my journey as a sysadmin the more I realize how the CS background is helping me.  I have a solid foundation in many of the core concepts that were taught through the CS program, which in turn  have indirectly influenced my abilities as a system administrator for the better. The first and most valuable asset my CS background has given me is the ability to write and understand code.  This is extremely useful in my daily slew of activities.  It allows me to approach problems with a programmatic methodology, it allows me to automate redundant and repeatable tasks with scripts, it gives me intuition into why databases or programs are slow, it allows me to debug issues systematically, and on and on.  Obviously these skills can be learned elsewhere but having them rolled up into your education when you learn about Computer Science as part of the package deal is very convenient.  I would much rather have this set of skills and have the ability to look at things from a different perspective than have to learn each of these techniques separately.  There is no way that somebody coming from a business or other similar background will know about silly things like big O notation or how different algorithms work at a fundamental level, it just isn’t part of their background so they don’t spend time thinking about these things.

This really parlays into other areas well and you are setting yourself up for a diversified and broad horizon for future employment prospects. For example, take a pure sysadmin that knows no programming or CS; at their core they know system administration. But what if they either get burnt out (which is common in this profession) or they don’t keep up the skills to match their position? There is nowhere in the industry for these individuals to turn, unless they want to go into management. That is why I believe individuals that choose not to further their careers are essentially crippling themselves and their future prospects by not knowing how or learning to program, or to at least understand how system administration and programming can relate to each other. With a diverse background the CS sysadmin could potentially move into a Devops role, a pure programming and development role or a management role. With the diverse IT ecosystem, programming and development skills are very much saught after and so the demand is high for these other types of positions and sets of skills.

Another well known fact in the IT industry, which I don’t necessarily agree with but nonetheless exists, is the fact that just having a CS degree will open doors that may not otherwise be open without a degree. I personally believe that a degree shouldn’t dictate your position but by having a degree you set yourself up for some unique opportunities and certainly are not hurting yourself. For example, all other things being equal, somebody scanning through resumes has to select an individual applicant that either has a degree in Computer Science or a degree in Philosophy. Which do you think will be picked? Like I said, I don’t think the hiring process is fair or even has anything to do with skill but can be used as a way to get ahead of the competition in the hiring process and can therefore a degree be valuable by itself as well as viewed as a strategic component in the hiring process if nothing else.

Here’s what I am saying. You don’t have to have a degree in Computer Science to be a great System Administrator. But the CS background definitely equips you with the tools to both understand some of the more abstract technical concepts and ideas and give you a robust framework working through and solving these difficult and complex problems. Ultimately the most important factors in being a good sysadmin (let alone anything else) is a combination of many different things, including a willingness to learn and the amount of experience an individual possesses. There is no cookie cutter way to build the perfect sysadmin and you will invariably find a very diverse group of people in this profession, but a head start with a CS degree is certainly one path that won’t hurt you and is a good attribute of many good sysadmins.

Read More

Using Find-String to grep in Powershell

For the longest time I have not been a fan of embracing the shell in the Windows world, but more and more I find that Powershell is able to do the things that I need.  I suppose my seething hatred was in part due to my negative bias towards Windows and the lack of useful tools from the command line in Windows.  Increasingly lately, I have been changing the way I think about and utilize the command line in Windows with Powershell.  And to be honest, I’m really beginning to fall in love with Powershell the more I get work with it and the more I get to see how to apply it in Windows based environments.  The good news is that Microsoft has put a lot of effort into this and are adding improvements and features all the time.  It still has a long way to go, but I can already see this as an alternative to GUI based administration in the Windows world and finally begin to feel like I can see the promise land on the horizon.  Using Powershell has been somewhat of a paradigm shift (in my opinion anyways) to how to do Windows administration recently and I feel like it will only get stronger and more common in the years to come.  So in this post I will try to show you some of the flexibility as well as some of the power that Powershell has to offer with some great tools from the community.

To highlight what I am talking about, let’s talk about grep, a well known and loved tool in the *nix universe.  I love grep.  For the longest time, I hadn’t known of a way to grep in the Windows world until just recently with a wonderful third party Powershell module called Find-String.  There are a few commands that you need to get this working.  To start, we need to get a module installed that is basically used for package management.  This tool is called PsGet. Installation is super simple, just run the following from a Powershell prompt, and ensure that your execution policy is set at least at remotesigned (Set-ExecutionPolicy RemoteSigned if you don’t have this turned on).

(new-object Net.WebClient).DownloadString(“http://psget.net/GetPsGet.ps1“) | iex

This should go out, download and install PsGet for you.  Once all that has completed you should be able to install Powershell modules that are contained in the PsGet repository.  With this installed we can just reach out to the PsGet repo and install our module.

Install-Module Find-String

Easy, simple, and clean.  That is the best part, there isn’t a ton of leg work to get this stuff working correctly and is why I’m enjoying Powershell so much these days.  Now we have a very functional grep clone!  Usage is quite a bit similar to grep, fore help you can do a Get-Help Find-String or look at the link I posted earlier to the author’s github page.  If you want to look for a word or substring in a file just use something like this:

Find-String word example.txt

This will output all occurrences of the word you are looking for in the file example.txt.  Here is a screen shot to show you.

Find-String in action

What other cool Powershell stuff are you doing?  I would love to hear about other cool uses that can be beneficial in every day use.

Read More