8 Golden Rules for Sysadmins

Getting the most out of your career can be rewarding.  Today I feel like taking a minute to slow down and reflect on a few of the things that I have observed in my time as a system administrator that I believe lead to success.  The following are some general rules that I have found to be true both in my work and more generally, many of these rules are just attitudes which can be applied to life as well.  Hopefully these come as common sense to you but it is always good to take time to reflect on good things.  I hope this isn’t too cliche or too much of a time waste for many of you but rather an opportunity to take a moment and analyze your current situation and potentially reevaluate anything you feel to be a weak area or area that could use improvement.

1. ) Always have a backup. Good backups are an invaluable asset to you as a system administrator, and can be a great bargaining piece if necessary in political battle.  Often times backups are overlooked by IT staff, so by ensuring you have good backups (you must always test them!) you are covering your own ass and are able to deflect blame if something out of your control occurs.   As a bonus, you look like a hero when the CEO or president of the company needs files from a month ago and have no idea where to turn, you will look like a magician and could potentially strengthen their view of IT.

2.)  Be likable. It can be a sad truth but many promotions hinge on whether or not people like you. You may be far and away the smartest, most technical or most talented person on your team but it is not going to get you very far if you are an asshole, and people don’t like you.  In this profession it is the case more often than not I see colleagues take the “holier than thou” approach which just perpetuates the stereotype that IT people are jerks.  If you can manage to be smart and not an asshole in IT you will go far.

3.)  Learn how to write. This doesn’t mean you have be able to produce enough volume for a novel, just use writing to develop your own voice, and use it as a way to communicate things effectively.  The great thing about writing is, the more you do it, the easier and more impactful it becomes.  Use your writing as an opportunity to help position yourself for success in the future.

4.)  Learn to program. Again, following up on the last point; this doesn’t mean that you need to become a software engineer, this is just the ability to quickly patch some code together to automate something that you are doing every day or having the ability to look at some process and say, “hey, I bet I could write a script to make this work better”.  It will make you more productive and efficient and will free up your time for other important tasks.

5.)  Patience. In this line of work the number one virtue any Sysadmins can have is patience. Being able to be pulled away from your work multiple times a day to help with completely unrelated issues can quickly become frustrating so having patience to deal with these things is incredibly helpful. And if you can deal with distractions well, people will like you more. Reference rule #2 for more on that.

6.)  Never stop learning.  System administration changes considerably quickly, which in my opinion is great, if you embrace it.  New technologies are always are always on the horizon, companies get bought and integrated into other companies all the time and technology strategies change all the time.  It is a never ending game of catch up for the sysadmins, so if you become content with where you are at and don’t keep up on your studies and on your technologies you will surely fall by the way side sooner than later.

7.) Attention to detail.  This one can be a real difference maker.  There is something to be said for tidiness and orderliness in system administration. Not only does it make things much easier to fix when everything is in a specific place, but it just makes you look better and in all reality doesn’t take much time to do things correctly. We all know the reckless admin who pays no attention to the mess they are making, and in turn it reflects poorly on their character.  Even if they are a genius and amazingly talented, it makes that person look sloppy and lazy to me.

8.) Balance your life.  This helps prevent stress and burnout.  Work and everything associated can get stressful at times so finding a balance becomes a great benefit when you learn how to manage it correctly.  Being able to leave work at work when you need to is crucial in keeping your sanity, but that’s not the only thing that’s important in balancing your life.  It is also helpful to find things outside of work that you enjoy doing.  Whether that be a hobby, interests outside of work, exercise, an interest group, vacations, whatever.  When you spend time focusing on the things that make you happy outside of work it will recharge your spirit more quickly and ultimately help keep you happy as well as productive at work.

Did I miss anything?  Have any other helpful tips that you’d like others to know about?  If so, let me know.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Deliberate Practice and System Administration (Part 3)

So far we have covered the following,

  • All the ideas, concepts and techniques needed for applying deliberate practice. (Part 1)
  • A framework for how to apply techniques of deliberate practice to system administration. (Part 2)

Now let’s wrap this all together and examine the last important piece in this series.  A real world example of deliberate practice in action and its association with expertise.  We can put some of these ideas and techniques into practice and build our own customized version for applying deliberate practice to system administration.

Practically everybody knows who Tiger Woods is and how incredibly skilled he is.  If you don’t I suggest you Google him.  However, few understand the level of dedication he has to his craft and how much he actually puts into the sport to compete at the level he does.  So let’s go ahead and take a look at what his daily routine consists and figure out what conclusions we can draw and how we can model a practice routine for system administration after this.

  • 6:30 a.m. – One hour of cardio. Choice between endurance runs, sprints or biking. 
  • 7:30 a.m. – One hour of lower weight training. 60-70 percent of normal lifting weight, high reps and multiple sets.
  • 8:30 a.m. – High protein/low-fat breakfast. Typically includes egg-white omelet with vegetables.
  • 9:00 a.m. – Two hours on the golf course. Hit on the range and work on swing.
  • 11:00 a.m. – Practice putting for 30 minutes to an hour.

Noon – Play nine holes.

  • 1:30 p.m. – High protein/low-fat lunch. Typically includes grilled chicken or fish, salad and vegetables.
  • 2:00 p.m. – Three-to-four hours on the golf course. Work on swing, short game and occasionally play another nine holes.
  • 6:30 p.m. – 30 minutes of upper weight training. High reps.
  • 7:00 p.m. – Dinner and rest.

That is pretty crazy.  So how do we model this to fit our purposes for system administration?

There are two obvious things that I think are crucial that we can borrow from this right away.  Exercise and healthy diet.  These are important facets because as I’ve talked about previously and has been proven many times, proper diet and exercise contribute to improved cognitive abilities.  I don’t suggest following Tiger’s workout or meal plan but I would suggest at least an hour of exercise as well as a healthy meal plan to help operate at optimal energy levels throughout the day.

Next, nearly all of the rest of his practice schedule revolves around improving very specific aspects of his game.  Thankfully we came up with some of these generalized aspects of improvement for system administration in Part 1 and Part 2 so we can put these to use in our own plan.  It is important that  we combine everything into one practice schedule that is challenging but is also realistic.  We also don’t want to go over 4-5 hours each day.  So here is the schedule I propose, feel free to adapt these any way you like:

  • 8:00 a.m – Breakfast.  Fresh made juice or several pieces of fruit + supplements (multivitamin, fish oil, vitamin D).
  • 9:00 a.m. – Check relevant news, new trends and tech, check mail, forums, etc.
  • 10:00 a.m. – Focused study on new or weak areas, 60 minutes on, 15 minutes off.  This can consist of reading, videos, audio.
  • 12:00 p.m. – Lunch.  Mixed salad.  The more vegetables the better!
  • 1:00 p.m. – Lab time.  Focus on strengthening and understanding of study topics.  This is where the most time and energy should be spent.
  • 4:00 – p.m. – Work on command line/programming skills and techniques.  This fulfills more of our hands on and practice time requirements.
  • 5:00 – p.m. – Gym.  Alternate days between cardio and strength training.
  • 7:00 – p.m. – Dinner and relaxation.
  • 9:00 – p.m. – Writing and reflections on the day.  Areas of improvement, etc.

Again, this is only a guideline.  I plan on updating this as I test these techniques and make refinements and adjustments to it.  As an example, I like to work out in the evening to help me relieve stress but many others (including Tiger like to take care of this in the morning), you just have to figure out what works best for your lifestyle, so I would definitely encourage you to experiment with what works and what doesn’t.  I’m very curious to know myself.  Since this is a first revision I think there will probably need to be a number of adjustments, but I look forward to trying it out and reporting back with some results!  If you have suggestions or have your own practice schedule let me know and I’ll definitely incorporate it into my routine.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Deliberate Practice and System Administration (Part 2)

Remember from the last post, deliberate practice can loosely be defined using the following:  Spending more time practicing and working on the most difficult tasks to gain the most improvement in the shortest amount of time for a given skill.  Okay, so the goal here is to come up with some seriously difficult but reproducible content to practice.  So how do we do that?  Well there’s a few things we can focus on doing to make this possible.

Set up a lab environment.  This is the main staple for working on your skills, and I believe without some sort of a lab environment one will not be able to completely and correctly apply the concepts that deliberate practice offers.  Hands on practice is where most of your time should be spent when learning, as well as time spent working on active recall. It is very important to build up some sort of environment for practicing different ideas, skills, content, etc. and it is an essential prerequisite for improvement.  The lab environment is not the only piece of the puzzle though.

I haven’t covered this much, but Active recall is an important concept to familiarize yourself with. It is a component of efficient learning and is a staple that is used to help develop long term memory.   Another main piece includes learning materials, whether it be audio, video or written format.  We need some type of reference material as well as a set of concepts and ideas to use as a guide for learning new subject matter.  A good example of this is Cisco and their CCIE certification.  They publish a list of topics and objectives covered by the exam, as well as a number of texts to use as study material.  So something like this would be useful for a network engineer that was looking to improve some aspects of their skill set.  They could use this topic list as a guide for what to study and the listed texts as reference material.

Once the objective has been defined, and the material has been clearly established for learning a particular skill or concept, the next step is to go out and attack it!  Remember, repetition (and especially spaced repetition) is an important element of practice, so when you are building these skills you will often find yourself going over things that you may have already covered.  Just remember that by repeating previously learned skills you are strengthening and improving your overall understanding, even if it can seem pointless and boring, it will help you.

Here are some techniques for quickly learning a new topic that I have found to be most useful for my own understanding and ability to grok ideas and concepts of a particular I am trying to learn more thoroughly.  This technique can be applied to practically anything, but I have found it especially efficient for my own purposes as a system administrator when I need to learn new subject material.

  • Read a chapter about the new topic or idea, taking notes as you go.
  • Watch related videos to further strengthen the concepts you are interested in learning about.
  • Set up and work through labs specific to the topic.  You may need to ask for help in getting some  ideas here but there are many great communities out there.   Remember that hands on practice is where most of your time should be spent!
  • Revisit the book/topic and notes, update anything that needs to be clarified, etc.
  • Revisit  the previous steps, focusing on weaker areas until you are comfortable with idea or topic.
  • Do it all over again with another topic.

Find yourself a mentor or a teacher in your journey to help with this process. I need to speak about this briefly, because I don’t think it can be understated.  If you get stuck or need ideas for labs, having somebody there to help and bounce ideas off of is incredibly important.  While you can learn all of this stuff on your own, having a mentor or teacher will increase the learning process by an order of magnitude.  So go out there and find some help, as I’ve mentioned before there are some great communities.  /r/sysadmin is one of my favorites, the people there are really smart and friendly and they love helping out others.

The final peace to this puzzle, in my opinion, is motivation.  It takes a certain level of grit to take the steps to become better at something through deliberate practice.  It all looks easy enough until you actually start putting these ideas into practice, and begin realizing what it means, and how much hard work and effort it takes to become an expert by grasping the amount of time and effort it takes to achieve.  The commitment to improvement means a full focus to improving your skills all of the time, not just every now and then and it becomes very difficult, unless of course you have a supreme level of motivation.

In the next part I will examine the work ethic and some specific examples of individuals in other disciplines that perform at an expert level so you can get a more complete understanding of just how much work it takes, and what it really means to become an expert at something, so stay tuned!

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Deliberate Practice and System Administration (Part 1)

I’m going to take a step back from the usual topics and go down a slightly different path than I usually do today and take some time to discuss something that has recently fascinated me.  I often wonder what it takes to excel and become great (ie expert) in particular areas, activities, et.  For example, how do athletes get to be so good at what they do?  I want to be able to generalize this concept and apply this type of thinking to my own profession to help answer the question of “what characterizes a great sysadmin?  What makes a great one stand out from an average, or even a good sysadmin?”  This is the sort of thing I have been researching and I’d like to share what I’ve come up with so far.

I stumbled across some interesting works recently on the 10,000 hour rule as well as a method known as Deliberate Practice.  Essentially deliberate practice can be broadly defined as spending more time practicing and working on the most difficult tasks to gain the most improvement in the shortest amount of time.  Look up Anders Ericcson for some more in depth examples, there is lots out there and they are great reads. These ideas have helped to shape my understanding and relationship of what is known as expert level knowledge.  I am still struggling to put all the pieces together for what this all means for my own career (namely, system administration) and how these ideas and practices can be applied generally to system administration but I’d like to carve out some general ideas and ways that deliberate practice can be utilized and put to use in system administration.

The problem most of you are surely familiar with is that system administration is such a broad field and applying such specific techniques of deliberate practice can be very difficult to generalize.  What I’m proposing are a number of generalized techniques for improving your performance as a system administrator by applying the techniques I have read about in a controlled and focused way to improve overall performance and skill as a system administrator.  My hope is that this type of generalization can be used in other areas as well.

To begin with, there are a number of maxims that can useful as a guideline or general rule of thumb for how to think about using deliberate practice in your own life and apply it to the way you practice and think about how to get better.

  • Deliberate practice should never extend to more than 4-5 hours per day.  It requires a high level of focus and anything beyond this point (studies have shown) begins to hurt performance.
  • There are two disticint times of day where indivduals have been show to be more productive and are the most focused.  Late morning and mid afternoon, these are optimal times to use for deliberate practice.
  • Also of note, are managing energy levels.  Practicing at a high level with such concentration can lead to burnout if not manager properly.  A good way to manage this is to take short breaks between semi long periods of deliberate practice.  90 minutes practice, 15 minutes break, and repeat.
  • Devote the most time practicing the most difficult tasks.  These activities are designed for the sole purpose of effectively improving specific aspects of an individual’s performance, therefore are the most beneficial but the most difficult.
  • Meticulous focus on the improvement of weak areas.  Spend large amounts of time analyzing and studying ones self, constantly looking for things to improve.

In my next post I will come back and revisit this idea and share some more specific examples of how to apply deliberate practice to specific topics and areas of interest in system administration.  In this post I will explore various ideas and techniques for ways to specifically apply deliberate practice to tasks in system administration.  As always, I’d love to hear any feedback you may have, especially on this and upcoming posts about becoming a better sysdamin.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

WTF Friday

Lately I have been building a Windows Hyper-V v3 clustered lab environment with all the bells and whistles.  It has been a great learning experience thus far and I can honestly say that I am enjoying Hyper-V overall thus far.  Recently I decided to take the plunge and begin experimenting with System Center Virtual Machine Manager (SCVMM), and managed to run across a bizarre issue last Friday.  The reason I am posting is because there were basically no real clues for this problem, so I would like to go over some of the various things that I looked and ultimately how this issue was resolved.  I feel this post may be useful to others because a lot of this stuff is relatively new and there wasn’t a ton of material out there on this specific problem to use as reference.

The installation process is relatively straight forward.  The environment I am using is Server 2012, so as a prerequisite you must use SCVMM 2012 w/SP1 in order for this to work.  If you are using 2008R2 you can use SCVMM 2012.  I used this guide as a reference for the installation instructions, which more or less go like this:

  1. Create your SCVMM accounts in AD.  scvmmadmin (admin account), scvmmsvc (service account), scvmmadmins (admin group).
  2. Install/point the SCVMM server to SQL 2012.  I won’t go over SQL installation because it is beyond the scope of this post.
  3. Install the prerequisites on your SCVMM server.  ADK for Windows 8, SQL 2012 native client, SQL 2012 command line utilities.
  4. Install SCVMM 2012 w/SP1.  VMM Management Server, VMM Console.
  5. Deploy agents to Hyper-V hosts.

This is easy enough to follow but I was getting suck on step 4 when I was attempting to install the Management Server and the Console.  The installation would choke about half way through with the following error:

A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local  .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command “winrm helpmsg hresult”.

WinRM error

Okay… WTF?  I knew that I already had WinRM installed and was on, but if you are not sure the quickest way to find out is to type winrm quickconfig from a command prompt.  You should get something similar to the following:

WinRM output

So we know WinRM is on and should be working.  Next, I checked the installation logs for clues.  They are located in C:\ProgramData\VMMLogs\SetupWizard.log.  I found the portion of the logs that indicated there were issues:

12:44:54:VMMPostinstallProcessor threw an exception: Threw Exception.Type: Microsoft.Carmine.WSManWrappers.WSManProviderException, Exception.Message: A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command "winrm helpmsg hresult".
12:44:54:StackTrace: at Microsoft.Carmine.WSManWrappers.ErrorContextParameterHelper.ThrowTranslatedCarmineException(WsmanSoapFault fault, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.RetrieveUnderlyingWMIErrorAndThrow(SessionCacheElement sessionElement, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, String url, Hashtable parameters, Type returnType, Boolean isCarmineMethod)
at Microsoft.Carmine.WSManWrappers.AgentManagement.AssociateLibrary(WsmanAPIWrapper wsmanObject, String CertificateSubjectName, String& ExportedCertificate, ErrorInfo& ErrorInfo)
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.AssociateDefaultLibraryServer()
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.SetupLibraryShare()
at Microsoft.VirtualManager.Setup.InstallItemCustomDelegates.PangaeaServerPostinstallProcessor()
12:44:54:InnerException.Type: System.Runtime.InteropServices.COMException, InnerException.Message: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol.
12:44:54:InnerException.StackTrace: at WSManAutomation.IWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.MyIWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
12:44:54:ProcessInstalls: Running the PostProcessDelegate returned false.
12:44:54:ProcessInstalls: Running the PostProcessDelegate for PangaeaServer failed.... This is a fatal item. Setting rollback.
12:44:54:SetProgressScreen: FinishMinorStep.
12:44:55:ProcessInstalls: Rollback is set and we are not doing an uninstall so we will stop processing installs

Incredibly useful, I know.  It is good to know where this stuff is located though just in case other issues arise that require troubleshooting like this.  So at this point I was dumbfounded and most of the stuff I found on Google was not helpful for my situation (I tried many different suggestions).

Finally I came across a post that mentioned disabling WinRM from Group Policy.  It just so happens that there is a policy in our test environment for enabling Powershell and remoting and all that jazz.  So I completely disabled the policy and was finally able to get SCVMM to install!  Here are the two policy settings you should take a look at first.

Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Service


Computer Configuration > Administrative Templates > Windows Components > Windows PowerShell

I need to go back and verify that the root issue was caused by the WinRM portion of the policy, which I’m suspecting it is.  But if you run across this error look at your Group Policy settings!

The moral of the story:  Windows Management Framework 3.0 and more specifically the WinRM components of WMF 3.0 are delicate, even on Server 2012 (there have been major compatibility issues with earlier versions of Windows).  In my scenario Group Policy was somehow getting in the way (if you can understand and decipher those logs and how they relate to Group Policy let me know) of allowing SCVMM to install.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.