Category Archives: Productivity

What is DevOps?

Since landing myself in a new and unexplored terrain as a freshly minted DevOps admin, I have been thinking a lot about what exactly DevOps is and how I will translate my skills moving into the position.  I am very excited to have the opportunity to work in such a new and powerful area of IT (and at such a sweet company!) but really think I need to lay out some of the groundwork behind what DevOps is, to help strengthen my own understanding and hopefully to help others grasp some of the concepts and ideas behind it.

I have been hearing more and more about DevOps philosophy and its growing influence and adoption in the world of IT, especially in fast paced, cloud and start up companies.  From what I have seen so far, I think I people really need to start looking at the impact that DevOps is making in the realm of system administration and how to set themselves up to succeed in this profession moving forward.

Here is the official DevOps description on Wikipedia:

DevOps is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.  DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.

While this is a solid description, there still seems to be a large amount of confusion about what exactly DevOps is so I’d like to address some of the key ideas and views that go along with its mentality and application to system administration.  To me, DevOps can be thought of as a combination of the best practices that a career in operations has to offer with many of the concepts and ideas that are used in the world of development.  Especially those derived from Agile and Scrum.

The great thing about DevOps is that since it is so new, there is really no universally accepted definition of what it is limited to.  This means that those who are currently involved in the DevOps development and adoption are essentially creating a new discipline, adding to it as they go.  A current DevOps admin can be described in simple terms as a systems admin that works closely with developers to decrease the gap between operations and development.  But that is not the main strength that DevOps offers and really just hits the tip of the ice burg for what DevOps actually is and means.

For one, DevOps offers a sort of cultural shift in the IT environment.  Traditionally in IT landscapes, there has been somewhat of a divide between operations and development.  You can think of this divide as a wall built between the dev and the ops teams either due to siloing of job skills and responsibilities or how the organization at broader perspective operates.  Because of this dissection of duties, there is typically little to no overlap between the tool sets or thought process between the dev or ops teams, which can cause serious headaches trying to get products out the door.

So how do you fix this?

In practical application, the principals of DevOps can put into practice using things like Continuous Integration tools , configuration management, logging and monitoring, creating a standardized test, dev and QA environments, etc.  The DevOps mindset and culture has many of its roots in environments of rapid growth and change.  An example of this philosophy put in to practice is at start up companies that rely on getting their product to market as quickly as smoothly as possible.  The good news is that larger enterprise IT environments are beginning to look at some of the benefits of this approach and starting to tear down the walls of the silos.

Some of the benefits of DevOps include:

  • Increased stability in your environment (embracing config management and version control)
  • Faster resolution of problems (decrease MTT)
  • Continuous software delivery (increasing release frequency brings ideas to market faster)
  • Much faster software development life cycles
  • Quicker interaction and feedback loops for key business stakeholders
  • Automate otherwise cumbersome and tedious tasks to free up time for devs and ops teams

These are some powerful concepts.  And the benefits here cannot be underestimated because at the end of the day the company you work for is in the business of making money.  And the faster they can make changes to become more marketable and competitive in the market the better.

One final topic I’d like to cover is programming.  If you are even remotely interested in DevOps you should learn to program, if you don’t know already.  This is the general direction of the discipline and if you don’t have a solid foundation to work from you will not be putting yourself into the best position to progress your career.  This doesn’t mean you have to be a developer, but IMO you have have to at least know and understand what the developers are talking about.  It is also very useful to know programming for all of the various scripting and automation tasks that are involved in DevOps.  Not only will you be able to debug issues with other software, scripts and programs but you will be a much more valuable asset to your team if you can be trusted to get things done and help get product shipped out the door.

How I do internal documentation

Intro

Whether you are new to system administration or have spent some time in the role, getting a grip on documentation can be tricky and often times challenging.  I can speak from experience because when I started my career I had no idea how to do this stuff, but slowly over time I have been able to develop some useful techniques.  It also helps that I am obsessed with documentation.  Therefore the techniques I use are always changing and I am always looking for ways to improve my documentation skills.  I think the most important take away from what I have learned, is that there is no end all be all way to do documentation and everybody will probably find slightly different ways.  My method may or may not work for you but hopefully you can  look at it and use some of the underlying concepts to help with your own style.  I have been able to use my own documentation procedures over the past 3 years to really improve my job performance as well as other areas in my job so hopefully there is something in here you can apply to your own situation.

Once you get past the initial stage of how uncomfortable it is to document everything by your mindset to how useful it is to having a living reference for everything on the network, then documentation really becomes much more manageable.  The added bonus of solid documentation is that once you have a decent repository built up you don’t have to worry about constantly adding new items to your documentation.  I am at a point now that I primarily use my documentation as a reference and don’t find myself having to create or update things nearly as often as I used to.  The other main benefit of having solid documentation is to help others on your team, especially junior members so they can go out and learn about certain intricacies or specific details of systems in your environment.  It’s also good for avoiding situations like the hit by a bus scenarios.  The general idea is that by having a great piece of documentation, somebody can pick up where you left of with minimal downtime and impact.  Hence, if you were to get hit by a bus and/or become incapacitated then somebody else could get up to speed with the way things work (what, when, why, how) using your documentation as a reference without having to spend hours and hours of time to figure out what exact information that they needed to know.

Documentation Overview

There are really two main types of internal documentation, internal and external.  Public facing (external) documentation is an entirely separate topic and I will not be discussing it here.  Network and technical documentation (internal) are the two big ones that I use to reference typically.  Network documentation is essentially working with Visio to produce anything and everything related to the network.  If you want to be a good system/network administrator a great way to help is to get good at working with Visio.  The other type of documentation I like to refer to as “technical documentation” is typically anything and everything that I have found to be helpful in solving a problem.  As I mentioned earlier, the longer your work on your environment and building your documentation the less you will have to produce new documentation. This information comes from the documentation section in Limoncelli’s book Time Management for System Administrators which is a fantastic book for sysadmins if you haven’t already taken a look.

My own technical documentation has grown organically in the 3 years, from just a few pieces of information to essentially a living library of all of the different systems I and others on my team work with on a regular basis.  Everything I do is in there; from technical procedures, links to external blogs and websites, PDF’s, to pretty much anything else you can imagine that would be useful.  One of the hardest things about working on documentation in a collaborative environment is getting people to buy in to the system.  The best way I have found to handle this is to take the initiative, start the process and then let everyone know about it what tools you are using and how awesome this documentation is and hope that it sticks.  I am not a fan of forcing people to do things but by showing them an amazingly simple system that is also incredibly useful the goal is to get them excited and motivated to use your documentation system.  Sometimes it works, sometimes not, but I have found it to be the best approach when attempting to change people’s habits and get them used to working with something new.

In the remainder of this post I will go into some of the details of how I organize my technical documentation and use it to help quickly identify and fix problems quickly.  I don’t plan on covering network documentation in this post as it is a slightly different beast.

The Tool

I have used pretty much every documentation system you can think of, from SharePoint, EverNote, Dropbox, various Wiki’s, Office documents, etc. and through my time and experience with all of these various tools I have found one to be the best for syadmin documentation.  I really believe that many of these tools serve a certain purpose or function but in my time as a sysadmin I have found OneNote to be the hands down best tools for system administration documentation.  As a side note, Wiki’s are a great alternative if you don’t have access to OneNote but my preference lies with OneNote.

OneNote is flexible, you can set up a location on a file share inside your company and allow access by certain AD users or groups to the share to control who can see and write to and collaborate on your documentation system.  It is very handy to be able to work on some notes while in a meeting on my laptop then move back to my desktop and have these notes current and in sync between my machines.  It is also possible for multiple people to be in OneNote working live on different notes, which is also nice when you are working in a busy team environment.  I know there are many other products that allow this so even if you don’t choose OneNote I would highly recommend a tool that offers these features.

Organization

I have OneNote grouped in a specific way to help increase my productivity.  I have different notebooks for various items.  I have a notebook for my technical documentation, a task log, one for projects and meetings that eventually get merged into my tech docs if they grow and the project/meeting ever materializes into anything more than just a few notes and I also have a notebook for personal work items.  The personal notebook contains a few different notes for personal accomplishments, personal agenda and ideas for projects and improvement.

Personal Notebook

Within the tech docs notebook I have notes for all of the major technologies that I or members of my team work with on a regular basis.  Basically, anything that we deem noteworthy gets thrown into one of the categories somewhere inside the tech doc.  The structure of this document grows organically for me with a few minor pain points here and there along the way.  For example, when a new solution to a problem is found and there isn’t a specific category for the fix, sometimes it is necessary to either create a new category or just use your judgement when determining the best place to store the fix.  Luckily the search feature in OneNote works well so if you can’t remember where you put something then you can just search through everything to find it.  Coincidentally, the larger your docs grow to be the better the search function works.

Each category has a basic general structure guideline that I have found to improve the workflow and allows things to work more smoothly.  The structure applies to every page I can use it on and is a template that includes the  most common and hard to find information that I use as a template.  The items in each category each get their own note and are as follows:

  • Network Information – IP’s, DNS names, server and network roles, anything that plany a part in the network goes here.
  • Support/Contact – Direct software and product support numbers, vendor contacts, important email addresses, account ID’s and service agreements, licenses, contract numbers, etc.
  • Resources – Any links to relevant and important technical docs, deployment guides, implementation guide or admin docs that are either internally or online.
  • Commands – I work in a Windows environment so often times PowerShell is the go to resource.  I keep a table of all the commands that I deem to be useful here for that particular category.
  • Useful tools – This one is optional but for some categories is a nice little reference.  I use this section to help identify anything and everything that I can use as a tool to help with a particular category.

This information is the skeleton that I begin to model all of the other documentation around.  From there, the rest is easy and should essentially fall into place.  Most of the time for me, these other items are one off fixes for problems that I have found or have solved.  Sometimes a quick link to the blog post that helped will be a good enough reference and other times a painfully detailed set of steps is necessary for documenting the procedure for the fix.

Documentation template

Style

The actual documentation is more of an art form than anything else that gets better with practice.  Sometimes technical notes require an obscene amount of detail because of how complex they are and because the procedure  only needs to take place once or twice in a year.  Other notes just need a rough explanation and therefore don’t need any detail at all, it all depends on the situation.  As has been pointed out in the comments section, one good approach to documentation sometimes is to assume the reader has no prior knowledge about the topic and therefore painful explanation and detail are necessary to convey your materials.  The point is, there is no cookie cutter way to do all of your technical documentation and various tactics and techniques need to be developed for different types of issues, that is why documentation becomes an art form in my view.

Going back to my own technique, one of the most helpful tricks I have found is to create the commands section, which I throw all of the most useful and common commands into for the particular category I am referencing.  The more work you do from the command line, the more useful this note will be to you.  For example, being in a Windows environment, I work with Exchange pretty much daily and having a repository of all the useful commands that I need in one place is one of the best ways to save time rather then going through Google looking for the specific information I am looking for every time.  Here is a glimpse of what I am talking about.

PowerShell commands

Another thing that will help your documentation immensely is consistency.  I am talking broadly here but when you create a living document you will want to have some sort of consistent style across your documentation.  Items that come to mind here are things like having a consistent template as mentioned earlier, consistent naming conventions, fonts, naming conventions, established criteria for creating topics and notes, capitalization, fonts and organization, etc.  Having this general style guide established early on will help to make finding and reading information in your documents much easier and will consequently help to save time when you are looking for specific information.

Closing

That’s all I’ve got for now.  I just wanted to point out a few things and get people pointed in the right direction that are looking for ideas of how to get going on technical documentation.  As I mentioned, there’s more than one way to skin a cat – meaning there are a large number of ways to do documentation and there isn’t necessarily one best way to do things.  I have shown you my preferred way, but it may not be the best way for you.

Documentation is as much of a learning process as anything else and sometimes you just need to experiment with things and just spend some time wading around to get the best results.  One great way to check if your documentation is working or not is to get somebody from your team to attempt to use your documentation to fix something you have instructions for.  If you just want to get some practice writing clear and concise instructions, try this method on yourself.  Write out the doc and procedure and come back to the instructions a week or month later and see how hard it is to figure out how to fix the problem.  Chances are, if things are unclear or your teammate is unable to fix the issue then it is probably a good idea to take a look at reevaluating how you’re doing your documentation.

If you have any thoughts, suggestions, ideas or anything else just let me know.  Like I said earlier, I am obsessed with this kind of stuff and I am always looking for ways to improve my own processes and procedures.  I plan on coming back to this post and update it in the future if any of my documentation practices change but for the time being, I hope you find this information useful and applicable to your own documentation processes.

Some quick tips for getting ahead

Do you want to get ahead in this field?  In my experience in the industry, there are a lot of great ways to improve yourself and to put yourself in position to get ahead.  One of the best ways is to show that you are interested in your craft.  There are numerous ways to demonstrate this desire, and dedication; a simple yet great way to do so is by attending industry related conferences and networking events.  Pay for these events as well as certification courses out of pocket if your employer is unwilling to do so.  Paying for your own career betterment out of your own pocket shows your level of commitment to those around you that you may meet at these events as well as those that you work with.  I guarantee you people will notice this.  I would also like to mention that if you are interested in training, certification, or anything the will help you do your job better and employers aren’t willing to help pay for any of these opportunities then often times it is a sign that you have outgrown your current position or will be in the not so distant future.

Another great area to focus on is to never stop learning.  This applies not only to IT but to everything that you do in your life.  Taking the time to learn something new is a great way to help boost your career.  Interested in business?  Take a business class or at least start researching different aspects of business and find what interests you.  Then leverage that newly found knowledge in every day aspects of your life.  I am a firm believer in education overlapping and translating skills from one area to contribute to others.  If you learn a seemingly unrelated skill you never know when and where it will be applicable and it will all contribute to furthering your education and improving your knowledge.  It can be devotion to skills like Documentation, attention to detail, writing, programming, whatever it may be will help to improve and strengthen your IT skills, I promise.  Many of these skills are things that I work on improving outside of my life in IT but certainly they contribute to the success I have experienced in my career.

The final tip I’d like to mention is doing things that others either aren’t willing to do or are not aware of doing.  This is a more subtle point but I believe it is something that will separate people that want to be good at something from those that are content with where they are at in life.  For example, many individuals in IT don’t like to leave their comfort zone for many things.  Learn how to talk to people and network with others in the industry who have already discovered how to get ahead.  They will give you so many great pieces of information and can potentially help you get into new positions as well.  We all know that social skills are not a strong suit for many that are in the IT profession so learning how to talk to people is incredibly valuable.  Yes it will feel awkward and unnatural at first but the more you do it and the more you practice it (just like anything else) the better you will get at it.  And since it is so uncomfortable for many individuals they simply won’t attempt it because it is out of their comfort zone.  There are many other examples of simple ways to get ahead but just by knowing how to leverage and utilize things like networking and being social will help your career more than you might realize.

I am taking a different approach from the usual subject matter, I have just found it difficult recently to find anything technically interesting enough to write about.  I would love to cover more of these areas if there is any interest in the future, I think its an oft missed subject in the IT profession and I happen to like writing about it so hopefully I can get some positive feedback.  As always, if you have anything that you find interesting or think would be a positive contribution let me know, I’m always looking for guest posters and guest content and I welcome the fresh perspective and ideas.

A Git Primer for Sysadmins

I am writing this post with the intention of giving readers (and other sysadmins) that are unfamiliar with Git a brief introduction to get up and running as quickly as possible.  There are numerous guides out there but none I have found so far with the spin of getting things working quickly for system administrators.  I recently went through the process of submitting some work of mine to an Open Source project and went through all of the work myself and thought I would write up a little guide to getting started, just in case anyone is interested.  Because of this experience, I thought the process could be streamlined if I presented a simple set of steps and examples for other sysadmins so that they can get started and get comfortable using Git as quickly as possible.  By no means am I advocating that I am a Git expert, I simply know enough to “do stuff” so take this guide for what it is, a simple introduction.  Hopefully it is enough to get the ball rolling and get you hooked on revision control!

Step 1:  If you haven’t already, head over to github and set yourself up with a free account.  The steps are trivial, just like setting up any other account.  You will need to install Git on you machine locally so that you can do everything, sudo apt-get install git in Ubuntu, or, with a little more legwork if you are using Windows.

Step 2:  Once you are all set up and running you can either create a new repository or fork off of somebody else’s repo, which will essentially make a copy of their work to your user account in github.  Since I was contributing to a public project I forked their repo.  This is typically (for beginners at least) done through the website by browsing to the project you are interested in and then clicking the fork button at the top right.

Fork a repo

Once you have copied or forked the project you want, you can begin working on making your own changes and updates to the project.  This is where much of the initial confusion came in for me.  I should make a note here; there are a few things that must be done through the website and others that should be done with the git tool you downloaded and installed earlier.

Step 3:  Clone the repo down to your machine locally to begin making changes.  This is done via command line with the following:

git clone https://github.com/$username/repository.git
git clone https://github.com/jmreicha/curriculum.git

Once you have the repo cloned down to your machine, let’s create a branch which we will use to work on the specific piece of the project we are working on.  There are a few more common commands that are used for working with repos.  In some scenarios the public project may change or get updated while you are working on it so you might want to pull in the most recent changes to your own fork.  The following commands are used to work with public repos and forks.

Fetch new changes from original repo:

git fetch upstream

Merge the fetched changes into your working files:

git merge upstream/master

Pull newest updates and changes from a for or repo:

git pull

Okay, let’s assume that our repository is all up to date and we are ready to start working on the new changes that we want to make.  First, we need to create a local branch to make the changes on.  The command for creating a branch is:

git checkout -b $topic
git checkout -b my_topic

Where $topic is the specific change you will be working on.  Then go ahead and make your changes and/or updates with your favorite text editor.  If you have multiple changes to make it may be a good idea to create a few different branches using the command above.  To switch between branches use the following:

git checkout $branchname
git checkout my_topic

If you are working on a branch and either don’t want to commit the changes or the branch becomes obsolete, you can delete the branch with either of the following commands:

git reset --hard
git branch -d $branchname
git branch -d my_topic

Step 4:  Assuming you are all done with your branches and are finished making your changes the next step is to commit the changes.  This will basically record a snapshot of (the state) the files.  Typically for a commit you will want to do one thing at a time to keep track of things more easily.  So you will make one change and then commit and then repeat if there are numerous changes that need to be made.  The command is:

git commit -am "update goes in here"

Now that we have created a branch, made our changes, written our changes so that git knows about them and can keep track of them, we are ready to go ahead and merge the code that we have changed locally back up to github.  Start by pushing the branch up to the github repo.

git push $remote $branchname
git push origin my_topic

Next, merge your changes.  This is done by changing to the “master” branch and then issuing a merge command on the branch that was created to make changes into.  So here is what this process would look like:

git checkout master
git merge $branchname
git merge my_topic

alternatively if you just want to quickly push out a change to the master branch you can issue a git commit locally and then a git push to update the master branch without merging the items first.

git commit -am "updated message"
git push

I like to use this for local repositories where I just need to get things up quickly.

Step 5:  At this point everything should be accurate on your own personal github page.  You can double check your commit history or look at the last update time to make sure your changes worked their way into your public fork or repo.  If you are contributing to a public project the final step is to issue a pull request on the github site.  Once this is done, wait to hear back from a moderator of the project if your changes need to be fixed before they accept them or wait for a response saying that your changes were good and get merged to the project.

Git, to me, was a little bit confusing at first.  Like I said earlier, I still only have a generic and basic understanding but I think it is important for system administrators to know how to commit code to projects and how to keep track of their own code and projects as well.  Even if the code is only small scripting projects it is still beneficial to publish since github is a public site.  Potential colleagues and employers can browse through your work, which increases your visibility and can even lead to job offers.

If you’re interested in learning more, I found this site to be a great resource.

Setting up Git in PowerShell

It seems like everybody is using git these days.  And for most, not everybody is stuck using Windows in their day to day workflow.  Unfortunately, I am.  So that means it is much more painful to get up and running with a lot of the coolest and best open source projects that are offered by members of github and other online code repositories being shared via git.  However, there is hope and it is possible for Windows users to join the git party.  So in this post, I would like to describe just how to do that.  And it should only take a few minutes if done correctly.  I will mention beforehand that there are a few steps that need to be completed in order for this technique to work successfully that typically are taken care of in a Linux or OSX environment.

The goal of this post is to work through these steps as best I can to get users up and running as quickly as possible and as easily as possible, reducing the amount of confusion and fumbling around with settings.  This post is designed for beginners that are just getting their feet wet with git but hopefully others can use it as a resource if they are coming from a different environment and are confused by the Windows way of doing things.

First step – Download and install the git port for Windows.

This is pretty straight forward.  Download and run the executable to install git for Windows.  If you just want to get up and running or are lazy, you can leave all of the defaults when you run through the installation wizard.

Second step – Add the git binaries to your system path variable.

This is the most important step, because out of the box git won’t work in your ordinary PowerShell command prompt, it needs to be opened separately.  So to fix this and add all the necessary binaries open up your environmental variables (in Windows 8).

Computer -> Properties -> Advanced -> Environmental Variables

environmental variables

and add the following value to the PATH variable.

C:\Program Files\Git\bin

Here is what this should look like in Windows.

path variable

Third step (optional) – Download and install posh-git for better PowerShell and git integration.

I have highlighted part of this process before in an older post but will go through the steps again because it is pretty straight forward.  To be able to get posh-git you need to have a sort of PowerShell package management tool called PsGet (instructions here).  To get this tool run the following command from your PowerShell command prompt.

(new-object Net.WebClient).DownloadString("http://psget.net/GetPsGet.ps1") | iex

Once the command has completed you should be able to simply run this install command and be finished.

install-module posh-git

That should be it.  With these simple steps you should be able to utilize git from the command line like you are accustomed to on other operating systems.  As I said, there is a tad more leg work but you can really utilize the flexibility of PowerShell to get things working.  I hope it helps, and as always let me know if you have any tips or questions.