How I do internal documentation

Intro

Whether you are new to system administration or have spent some time in the role, getting a grip on documentation can be tricky and often times challenging.  I can speak from experience because when I started my career I had no idea how to do this stuff, but slowly over time I have been able to develop some useful techniques.  It also helps that I am obsessed with documentation.  Therefore the techniques I use are always changing and I am always looking for ways to improve my documentation skills.  I think the most important take away from what I have learned, is that there is no end all be all way to do documentation and everybody will probably find slightly different ways.  My method may or may not work for you but hopefully you can  look at it and use some of the underlying concepts to help with your own style.  I have been able to use my own documentation procedures over the past 3 years to really improve my job performance as well as other areas in my job so hopefully there is something in here you can apply to your own situation.

Once you get past the initial stage of how uncomfortable it is to document everything by your mindset to how useful it is to having a living reference for everything on the network, then documentation really becomes much more manageable.  The added bonus of solid documentation is that once you have a decent repository built up you don’t have to worry about constantly adding new items to your documentation.  I am at a point now that I primarily use my documentation as a reference and don’t find myself having to create or update things nearly as often as I used to.  The other main benefit of having solid documentation is to help others on your team, especially junior members so they can go out and learn about certain intricacies or specific details of systems in your environment.  It’s also good for avoiding situations like the hit by a bus scenarios.  The general idea is that by having a great piece of documentation, somebody can pick up where you left of with minimal downtime and impact.  Hence, if you were to get hit by a bus and/or become incapacitated then somebody else could get up to speed with the way things work (what, when, why, how) using your documentation as a reference without having to spend hours and hours of time to figure out what exact information that they needed to know.

Documentation Overview

There are really two main types of internal documentation, internal and external.  Public facing (external) documentation is an entirely separate topic and I will not be discussing it here.  Network and technical documentation (internal) are the two big ones that I use to reference typically.  Network documentation is essentially working with Visio to produce anything and everything related to the network.  If you want to be a good system/network administrator a great way to help is to get good at working with Visio.  The other type of documentation I like to refer to as “technical documentation” is typically anything and everything that I have found to be helpful in solving a problem.  As I mentioned earlier, the longer your work on your environment and building your documentation the less you will have to produce new documentation. This information comes from the documentation section in Limoncelli’s book Time Management for System Administrators which is a fantastic book for sysadmins if you haven’t already taken a look.

My own technical documentation has grown organically in the 3 years, from just a few pieces of information to essentially a living library of all of the different systems I and others on my team work with on a regular basis.  Everything I do is in there; from technical procedures, links to external blogs and websites, PDF’s, to pretty much anything else you can imagine that would be useful.  One of the hardest things about working on documentation in a collaborative environment is getting people to buy in to the system.  The best way I have found to handle this is to take the initiative, start the process and then let everyone know about it what tools you are using and how awesome this documentation is and hope that it sticks.  I am not a fan of forcing people to do things but by showing them an amazingly simple system that is also incredibly useful the goal is to get them excited and motivated to use your documentation system.  Sometimes it works, sometimes not, but I have found it to be the best approach when attempting to change people’s habits and get them used to working with something new.

In the remainder of this post I will go into some of the details of how I organize my technical documentation and use it to help quickly identify and fix problems quickly.  I don’t plan on covering network documentation in this post as it is a slightly different beast.

The Tool

I have used pretty much every documentation system you can think of, from SharePoint, EverNote, Dropbox, various Wiki’s, Office documents, etc. and through my time and experience with all of these various tools I have found one to be the best for syadmin documentation.  I really believe that many of these tools serve a certain purpose or function but in my time as a sysadmin I have found OneNote to be the hands down best tools for system administration documentation.  As a side note, Wiki’s are a great alternative if you don’t have access to OneNote but my preference lies with OneNote.

OneNote is flexible, you can set up a location on a file share inside your company and allow access by certain AD users or groups to the share to control who can see and write to and collaborate on your documentation system.  It is very handy to be able to work on some notes while in a meeting on my laptop then move back to my desktop and have these notes current and in sync between my machines.  It is also possible for multiple people to be in OneNote working live on different notes, which is also nice when you are working in a busy team environment.  I know there are many other products that allow this so even if you don’t choose OneNote I would highly recommend a tool that offers these features.

Organization

I have OneNote grouped in a specific way to help increase my productivity.  I have different notebooks for various items.  I have a notebook for my technical documentation, a task log, one for projects and meetings that eventually get merged into my tech docs if they grow and the project/meeting ever materializes into anything more than just a few notes and I also have a notebook for personal work items.  The personal notebook contains a few different notes for personal accomplishments, personal agenda and ideas for projects and improvement.

Personal Notebook

Within the tech docs notebook I have notes for all of the major technologies that I or members of my team work with on a regular basis.  Basically, anything that we deem noteworthy gets thrown into one of the categories somewhere inside the tech doc.  The structure of this document grows organically for me with a few minor pain points here and there along the way.  For example, when a new solution to a problem is found and there isn’t a specific category for the fix, sometimes it is necessary to either create a new category or just use your judgement when determining the best place to store the fix.  Luckily the search feature in OneNote works well so if you can’t remember where you put something then you can just search through everything to find it.  Coincidentally, the larger your docs grow to be the better the search function works.

Each category has a basic general structure guideline that I have found to improve the workflow and allows things to work more smoothly.  The structure applies to every page I can use it on and is a template that includes the  most common and hard to find information that I use as a template.  The items in each category each get their own note and are as follows:

  • Network Information – IP’s, DNS names, server and network roles, anything that plany a part in the network goes here.
  • Support/Contact – Direct software and product support numbers, vendor contacts, important email addresses, account ID’s and service agreements, licenses, contract numbers, etc.
  • Resources – Any links to relevant and important technical docs, deployment guides, implementation guide or admin docs that are either internally or online.
  • Commands – I work in a Windows environment so often times PowerShell is the go to resource.  I keep a table of all the commands that I deem to be useful here for that particular category.
  • Useful tools – This one is optional but for some categories is a nice little reference.  I use this section to help identify anything and everything that I can use as a tool to help with a particular category.

This information is the skeleton that I begin to model all of the other documentation around.  From there, the rest is easy and should essentially fall into place.  Most of the time for me, these other items are one off fixes for problems that I have found or have solved.  Sometimes a quick link to the blog post that helped will be a good enough reference and other times a painfully detailed set of steps is necessary for documenting the procedure for the fix.

Documentation template

Style

The actual documentation is more of an art form than anything else that gets better with practice.  Sometimes technical notes require an obscene amount of detail because of how complex they are and because the procedure  only needs to take place once or twice in a year.  Other notes just need a rough explanation and therefore don’t need any detail at all, it all depends on the situation.  As has been pointed out in the comments section, one good approach to documentation sometimes is to assume the reader has no prior knowledge about the topic and therefore painful explanation and detail are necessary to convey your materials.  The point is, there is no cookie cutter way to do all of your technical documentation and various tactics and techniques need to be developed for different types of issues, that is why documentation becomes an art form in my view.

Going back to my own technique, one of the most helpful tricks I have found is to create the commands section, which I throw all of the most useful and common commands into for the particular category I am referencing.  The more work you do from the command line, the more useful this note will be to you.  For example, being in a Windows environment, I work with Exchange pretty much daily and having a repository of all the useful commands that I need in one place is one of the best ways to save time rather then going through Google looking for the specific information I am looking for every time.  Here is a glimpse of what I am talking about.

PowerShell commands

Another thing that will help your documentation immensely is consistency.  I am talking broadly here but when you create a living document you will want to have some sort of consistent style across your documentation.  Items that come to mind here are things like having a consistent template as mentioned earlier, consistent naming conventions, fonts, naming conventions, established criteria for creating topics and notes, capitalization, fonts and organization, etc.  Having this general style guide established early on will help to make finding and reading information in your documents much easier and will consequently help to save time when you are looking for specific information.

Closing

That’s all I’ve got for now.  I just wanted to point out a few things and get people pointed in the right direction that are looking for ideas of how to get going on technical documentation.  As I mentioned, there’s more than one way to skin a cat – meaning there are a large number of ways to do documentation and there isn’t necessarily one best way to do things.  I have shown you my preferred way, but it may not be the best way for you.

Documentation is as much of a learning process as anything else and sometimes you just need to experiment with things and just spend some time wading around to get the best results.  One great way to check if your documentation is working or not is to get somebody from your team to attempt to use your documentation to fix something you have instructions for.  If you just want to get some practice writing clear and concise instructions, try this method on yourself.  Write out the doc and procedure and come back to the instructions a week or month later and see how hard it is to figure out how to fix the problem.  Chances are, if things are unclear or your teammate is unable to fix the issue then it is probably a good idea to take a look at reevaluating how you’re doing your documentation.

If you have any thoughts, suggestions, ideas or anything else just let me know.  Like I said earlier, I am obsessed with this kind of stuff and I am always looking for ways to improve my own processes and procedures.  I plan on coming back to this post and update it in the future if any of my documentation practices change but for the time being, I hope you find this information useful and applicable to your own documentation processes.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Some quick tips for getting ahead

Do you want to get ahead in this field?  In my experience in the industry, there are a lot of great ways to improve yourself and to put yourself in position to get ahead.  One of the best ways is to show that you are interested in your craft.  There are numerous ways to demonstrate this desire, and dedication; a simple yet great way to do so is by attending industry related conferences and networking events.  Pay for these events as well as certification courses out of pocket if your employer is unwilling to do so.  Paying for your own career betterment out of your own pocket shows your level of commitment to those around you that you may meet at these events as well as those that you work with.  I guarantee you people will notice this.  I would also like to mention that if you are interested in training, certification, or anything the will help you do your job better and employers aren’t willing to help pay for any of these opportunities then often times it is a sign that you have outgrown your current position or will be in the not so distant future.

Another great area to focus on is to never stop learning.  This applies not only to IT but to everything that you do in your life.  Taking the time to learn something new is a great way to help boost your career.  Interested in business?  Take a business class or at least start researching different aspects of business and find what interests you.  Then leverage that newly found knowledge in every day aspects of your life.  I am a firm believer in education overlapping and translating skills from one area to contribute to others.  If you learn a seemingly unrelated skill you never know when and where it will be applicable and it will all contribute to furthering your education and improving your knowledge.  It can be devotion to skills like Documentation, attention to detail, writing, programming, whatever it may be will help to improve and strengthen your IT skills, I promise.  Many of these skills are things that I work on improving outside of my life in IT but certainly they contribute to the success I have experienced in my career.

The final tip I’d like to mention is doing things that others either aren’t willing to do or are not aware of doing.  This is a more subtle point but I believe it is something that will separate people that want to be good at something from those that are content with where they are at in life.  For example, many individuals in IT don’t like to leave their comfort zone for many things.  Learn how to talk to people and network with others in the industry who have already discovered how to get ahead.  They will give you so many great pieces of information and can potentially help you get into new positions as well.  We all know that social skills are not a strong suit for many that are in the IT profession so learning how to talk to people is incredibly valuable.  Yes it will feel awkward and unnatural at first but the more you do it and the more you practice it (just like anything else) the better you will get at it.  And since it is so uncomfortable for many individuals they simply won’t attempt it because it is out of their comfort zone.  There are many other examples of simple ways to get ahead but just by knowing how to leverage and utilize things like networking and being social will help your career more than you might realize.

I am taking a different approach from the usual subject matter, I have just found it difficult recently to find anything technically interesting enough to write about.  I would love to cover more of these areas if there is any interest in the future, I think its an oft missed subject in the IT profession and I happen to like writing about it so hopefully I can get some positive feedback.  As always, if you have anything that you find interesting or think would be a positive contribution let me know, I’m always looking for guest posters and guest content and I welcome the fresh perspective and ideas.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Resetting SCCM Agent if patch installation fails

From time to time, a ticket will be created in regards to System Patches failing in an SCCM environment. To fix this, there are really only two major steps:

  1. Rename the C:\Windows\SoftwareDistribution folder to SoftwareDistribution.old (stop Windows Update service before renaming, then restart the service).
  2. Rename C:\Windows\System32\catroot2 to catroot2.old (stop the Cryptography service before renaming, then restart the service).

After this is done, run these actions from the configuration manager:

  1. Discovery Data Collection Cycle
  2. Software Updates Deployment Evaluation Cycle
  3. Software Updates Scan Cycle

The procedure above has taken care of the issue pretty reliably. If the updates still don’t install properly, you may have to download the specific updates and install them manually.

About the Author: Mike Erps

Michael Erps currently provides IT support and consulting to gov’t contractors in D.C. He is also interested in Internet Marketing and helping others build the online presence they are looking for. You can find him on about.me

OpenFlow and the Future of Networking

OpenFlow is all the rage right now and since I just got done doing a product overview of it and its relation to the HP product line we just recently purchased, I thought I would get in a quick post about all of it while the topics and ideas are still fresh in my mind.  So this post will be less of a technical post than usual and more of a detour about my thoughts on networking and the effect OpenFlow is having on it.

I am still trying to wrap my head around some of the key concepts and applications that OpenFlow has to offer but I think I am beginning to understand the core concepts behind it, and honestly I don’t understand all the OpenFlow hate and SDN bashing from other network professionals.

Even thought OpenFlow is a fresh concept for me I can already see potential benefits and possible use cases and I think that there is some great potential with SDN in general.  There must be some interesting value here, otherwise there wouldn’t be so much interest by all of the heavy hitting networking industry leaders like IBM, Cisco, HP, Google, etc. collaborating and working on projects like OpenDaylight and Floodlight. Since the concepts and ideas behind OpenFlow are so new and are largely unexplored there is a very mysterious and exciting quality behind the technology and because of this I believe that creativity can help drive its development and adoption.  The other nice part about OpenFlow is that it is an open standard so it can be developed and extended by whomever feels like participating or contributing (Cisco and its OnePK API and other vendor specific API’s are a different story) to the project and the code base.  I am a huge proponent of Open Source and I feel like having an open standard creates better code and more opportunities for everybody involved, it doesn’t benefit one but rather the collective.

I also want to touch briefly on the technical side of OpenFlow for all the IT pros.  Technology evolves and changes all the time, we’ve seen it time and again in our industry.  If you are stubborn to the point that you won’t dedicate the time to learn something new just because its not what you are familiar with then you probably won’t have much of a future in IT and ops or at least a future going forward in the networking world.  Sure you’ve built a career on your niche ability and skill set to solve complex and challenging networking problems, but that is not a unique quality.  All IT professionals build their careers on their ability to do this (at least the good ones I’ve seen so far), and every other area of IT is subject to these same types of issues that new technology brings.  In my opinion the haters just need to grow up and accept the fact that they will need to remodel their skills from time to time.  It’s not that big of a deal.  And besides, OpenFlow actually looks promising and looks like it will be a great tool for IT pros to utilize to solve interesting problems.

Rather than complain and find fault, embrace OpenFlow, because whether you like it or not, it will have its place in the networking world moving forward.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Wireshark Reference Guide

Based on a strange network problem recently I decided to put together some quick notes and a few tips on ways to improve your Wireshark experience based on my own experience with it.  There are many, many more features that Wireshark has to offer, these just happen to be the most apparent ones I have found so far.  Wireshark is extremely powerful and therefore extremely useful if used properly.  At first it takes a while to get used to everything Wireshark has to offer but once you start to get the hang of how things work then it can be a great network troubleshooting tool.  Basic knowledge of networking concepts should be assumed as well as familiarity of Wireshark for those who attempt to debug network problems using this tool.

Here is a list of some of the most common and handy features that you can utilize in Wireshark.  I am not going to dive into great detail with most of  these items because I honestly don’t have a ton of experience with all of them, I basically just wanted to point out the highlights.

  • Filtering in Wireshark is very handy.
  • Create custom profiles for different use cases (quickly select from bottom right hand corner).
  • Color filters are useful!  (Right click a field in the packet trace and selelct colorized rule)  The bottom left bar will tell you what variable you are looking at to make things easier when customizing.
  • Use Regex in wireshark using the “matches” clause to turn on regex patterns.
  • You can extract specific information from trace files on the command line using tshark.
  • Right click a packet and select “follow TCP/UDP stream” to debug a single network conversation.
  • Low delta times are good.  If you see high deltas you should probably investigate things.

Here are some more concrete examples and a few basics of how to put these tips into practice.  In Wireshark you can use either English or code like operators when filtering to help narrow down traffic and interesting networking patterns and issues.  So for example, “==” and “eq” will behave in the same manner when applying filters.  Other operators include <,>, !=, <=, >=, etc.  Just like you would see in a typical programming language.

Use custom configuration profiles.  If you look at packet traces often this will save you a tremendous amount of time if you are looking at specific types of traffic or are only interested in certain traffic patterns.  For example, you spend a lot of time looking at many traces that fit the same type of criteria; by using custom profiles you can quickly adjust and modify the view in Wireshark to help quickly identify patterns and potentially issues by cycling through different, specific views.  To begin creating custom profiles go to Edit -> Configuration Profiles and then either select the custom profile or create a new one to begin changing.

One handy trick is to disable TCP offload checks.  If your packet captures are getting clogged up with a bunch of red and black with offload errors, this is place you should go to look first.  There are a few places where this option can either be enabled or disabled.  The easiest way to check these options is under Edit -> Preferences and then under the Protocols tree for UDP, TCP and IPv4 protocols.  The example below shows what the options should look like for the TCP protocol.  The TCP and UDP offload checks are disabled by default but the IPv4 needs to be manually unchecked.  The specific option under the IPv4 protocol is labeled “Validate the IPv4 checksum if possible”, simply uncheck this and the red and black errors should disappear.

There is a capture option that allows you to resolve IP addresses to hostname, which I find can be very useful.  To enable this option open up the Show capture options screen, there should be an option in there under name resolution called “Resolve network-layer names”.  Simply check that box and you should have name resolution.

As mentioned in the bullets above, the “follow UDP/TCP stream” option can be extremely useful and is a very quick way to glean information.  It is so useful because it is so easy to use.  Simply find a traffic conversation you would like to debug and right click the packet number in the top Wireshark pane and choose the follow UDP/TCP stream option and you can get an idea of everything that happened during a particular conversation.  For example, using this technique you can follow FTP transactions.

Viewing a breakdown of the packet flow and traffic patterns can be a useful tool as well when diagnosing various network issues.  There is an option in Wireshark that shows in good detail the breakdown of various packets and protocols that can be used to troubelshoot the network.  This option is called Protocol Hierarchy Statistics and can be found un te Statistics -> Protocol Hierarchy Statistics page.

Only look at traffic for one IP address:

ip.addr==192.168.103.104

Likewise, filter out all traffic from an IP address:

!(ip.addr == x.x.x.x)
!(ip.addr == 1.2.3.4)

filter out all traffic for a specific port

!(tcp.port eq 2222)

Resources:

http://ask.wireshark.org/questions/
http://ask.wireshark.org/questions/
http://www.wireshark.org/docs/

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.