Fix the “Can’t create highly Available VM” error in VMM

I was doing some lab testing with Virtual Machine Manager (VMM) for my Hyper-V environment the other day when I ran across this issue.  It was happening because I was attempting to save my disk image to a CSV location and I couldn’t figure out why it was blowing up.

It turns out that the answer is really simple actually but is annoying to fix.  So here is what the error looks like in VMM.  “Cannot create or update a non highly available virtual machine because the path cluster storage volume is a clustered resource error”.  Surprisingly Google didn’t turn up much for how to fix this so I thought I would make a note of it in case others happen to run across this.

VMM error

The fix is simple enough, I just wish VMM was smart enough to know when you select a CSV to use for storage.  When you go through the VM creation process and you get to the hardware selection tab, choose the “Make this virtual machine highly available” check box.

VM high availability

If you happen to be building a lot of identical (or near identical) VM’s I reccomend checking out Profiles through VMM, in this case a specific hardware profile for desktop VM’s.  Creating profiles simplifies the build and creation process and you don’t need to worry about picking all the right defaults without having your VM creation blow up.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Automatic Load Balancing Fails for New Exchange Mailboxes

I had a strange issue come up recently where mail administrators were unable to allow Exchange to pick which mailbox database to place users into (I’m not quite sure what this feature is called so we’re going with automatic load balancing) when they were creating new user mailbox profiles.  This feature essentially picks a database at random as a way to balance and spread out mailboxes among databases.  Makes, sense to me.  Anyway, I’m thinking we may have installed an update that changed the behavior because this feature had apparently been working up until we installed Rollup 5-v2.  So it appears that this update had broken this automatic mailbox load balancing functionality, although I’m not 100% sure.  I just wanted to mention it in case some of you are looking for clues as to why this is happening.

So here is what the error looks like when allowing Exchange to choose which database to use when a new user is initially created.

mailbox error

Error:  Load balancing failed to find a valid mailbox database.

Hmm, okay.  That is weird.  You can check to see if the database is set to be allowed with this command.

get-mailboxdatabase | ft name,isexcludedfromprovisioning

In my case, all of my mailbox databases came back true, so they are unable to be used for automatically selecting a database when the user created.  To fix this, go ahead and run the following.

get-mailboxdatabase | set-mailboxdatabase -isexcludedfromprovisioning $false

Now go ahead and run the first command again and you should see that all the mailboxes are now marked as false, meaning they are now included in the provisioning process.

Now you should be able to go through and create without selecing a specific mailbox.  My best guess is that the update must have set this flag to exclude these databases from provisioning but is weird and is worth mentioning because it happened to cause some seemingly unexplained issues for us.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Locking down your WordPress site

Since I don’t really want to get in trouble for this, I need to put in a disclaimer.  Some of these tools can be invasive and if you are running them against somebody then I take no responsibility for their actions against you.  I am testing these tools against my own site so the consequences are minimal.  Just be aware that there can be serious consequences for using these tools on sites and companies against their will.  I don’t want anybody going to jail.

The tools

Let’s take a poke around with WP Scan.  This tool is a WordPress vulnerability scanner, often packaged together with Backtrack or the newer Kali Linux pentesting distro.   WPScan helps find and eliminate security weaknesses in your WordPress site.  More information about this tool can be found here.

There are many other tools out there but for basic WordPress scanning this tool should suffice, because it offers a number of things that are of interest in a nice single tidy and clean interface.  Other tools that may be of interest include tools like Burp Suite, SQLmap, username enumeration through Metasploit and other reconnaissance tools.

The process

Most real world attacks will reach for the low hanging fruit when it comes to exploiting WordPress sites, typically gaining access to a site through password exploitation.  With so many WordPress sites going up it becomes easy to move from site to site trying different password brute forcing attacks, so that’s where you will see a large number of attacks.  There are others as well, such as vulnerability attacks, SQL injection attacks, XSS, etc.

To begin the process let’s start gathering some information about the WordPress site that will be the focus of this attack, my blog.  Here, I am running WPScan through Kali Linux, so the syntax may change depending on how you decide to use this tool.  Let’s see what basic information we can get about my blog.  This site scan will attempt to gather the basics of the site it is scanning.  For help just type ‘wpscan –help’.

wpscan --url

Let’s see how far we can get with the password brute forcing method.  To enumerate a list of user account names use the following,

wpscan --url --enumerate u

If you get any interesting results from this scan, for example the result returns the username admin, go ahead and see if you can brute force the account.

wpscan --url --wordlist /pentest/passwords/wordlists/darkc0de.lst --username admin

There are more features packed in this tool so take some time to explore what all it can do (preferably on a test box).  Odds are that on a site that hasn’t been properly locked down you can probably get in, one way or another.  I wouldn’t recommend running wpscan against this site though because I have already beefed up the security and temporarily block access if users run malicious scans against the site.

Locking it down

There are a number of techniques to help reduce the attack surface for your WordPress site as well as methods to increase the difficulty of breaching your site.  The first and foremost is the use of strong passwords.  That should be a given and I won’t get into the details here of how important strong passwords are.  Another (hopefully) obvious technique is to keep up to date with your patches.  Whether it be on the Operating System or your WordPress site/plugins you should try to be proactive about patching your systems.  The third and final obvious solution I will mention are getting good backups.  If your site does get compromised then it is incredibly helpful to have a point in time to go back to rather than starting over from square one.  There are plugins designed to help with this process and even doing it by hand isn’t that difficult.  You can get back on your feet even if you only have a database dump from your site at some point in the past.

I’d like to specifically mention some good tools to use if you have publicly facing SSH; one of which is fail2ban.  This tool can be used as a layer of defense to slow attackers down by detecting malicious activity and banning IP addresses.  Another great tool, a handy plugin for WordPress  sites is called Better WP Security.  This is an easy to use site hardening tool that can fill up weaknesses and security holes quickly for somebody that doesn’t necessarily have security in the foreground of their minds.

By utilizing  these basic techniques you will infinitely increase your WordPress site’s security and make it much more difficult to attack and exploit.  There are of course other techniques to improve security but at a certain point it can become a balancing act.  *Most* site admins aren’t overly conscious about security and so do not spend a lot of time on their security efforts, they are more concerned about the content and getting things up.  Likewise, some are probably more prone to lock things down more than they perhaps need to.  It is important to maximize your effort, and to cover the most important security aspects by implementing the basics.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

8 Golden Rules for Sysadmins

Getting the most out of your career can be rewarding.  Today I feel like taking a minute to slow down and reflect on a few of the things that I have observed in my time as a system administrator that I believe lead to success.  The following are some general rules that I have found to be true both in my work and more generally, many of these rules are just attitudes which can be applied to life as well.  Hopefully these come as common sense to you but it is always good to take time to reflect on good things.  I hope this isn’t too cliche or too much of a time waste for many of you but rather an opportunity to take a moment and analyze your current situation and potentially reevaluate anything you feel to be a weak area or area that could use improvement.

1. ) Always have a backup. Good backups are an invaluable asset to you as a system administrator, and can be a great bargaining piece if necessary in political battle.  Often times backups are overlooked by IT staff, so by ensuring you have good backups (you must always test them!) you are covering your own ass and are able to deflect blame if something out of your control occurs.   As a bonus, you look like a hero when the CEO or president of the company needs files from a month ago and have no idea where to turn, you will look like a magician and could potentially strengthen their view of IT.

2.)  Be likable. It can be a sad truth but many promotions hinge on whether or not people like you. You may be far and away the smartest, most technical or most talented person on your team but it is not going to get you very far if you are an asshole, and people don’t like you.  In this profession it is the case more often than not I see colleagues take the “holier than thou” approach which just perpetuates the stereotype that IT people are jerks.  If you can manage to be smart and not an asshole in IT you will go far.

3.)  Learn how to write. This doesn’t mean you have be able to produce enough volume for a novel, just use writing to develop your own voice, and use it as a way to communicate things effectively.  The great thing about writing is, the more you do it, the easier and more impactful it becomes.  Use your writing as an opportunity to help position yourself for success in the future.

4.)  Learn to program. Again, following up on the last point; this doesn’t mean that you need to become a software engineer, this is just the ability to quickly patch some code together to automate something that you are doing every day or having the ability to look at some process and say, “hey, I bet I could write a script to make this work better”.  It will make you more productive and efficient and will free up your time for other important tasks.

5.)  Patience. In this line of work the number one virtue any Sysadmins can have is patience. Being able to be pulled away from your work multiple times a day to help with completely unrelated issues can quickly become frustrating so having patience to deal with these things is incredibly helpful. And if you can deal with distractions well, people will like you more. Reference rule #2 for more on that.

6.)  Never stop learning.  System administration changes considerably quickly, which in my opinion is great, if you embrace it.  New technologies are always are always on the horizon, companies get bought and integrated into other companies all the time and technology strategies change all the time.  It is a never ending game of catch up for the sysadmins, so if you become content with where you are at and don’t keep up on your studies and on your technologies you will surely fall by the way side sooner than later.

7.) Attention to detail.  This one can be a real difference maker.  There is something to be said for tidiness and orderliness in system administration. Not only does it make things much easier to fix when everything is in a specific place, but it just makes you look better and in all reality doesn’t take much time to do things correctly. We all know the reckless admin who pays no attention to the mess they are making, and in turn it reflects poorly on their character.  Even if they are a genius and amazingly talented, it makes that person look sloppy and lazy to me.

8.) Balance your life.  This helps prevent stress and burnout.  Work and everything associated can get stressful at times so finding a balance becomes a great benefit when you learn how to manage it correctly.  Being able to leave work at work when you need to is crucial in keeping your sanity, but that’s not the only thing that’s important in balancing your life.  It is also helpful to find things outside of work that you enjoy doing.  Whether that be a hobby, interests outside of work, exercise, an interest group, vacations, whatever.  When you spend time focusing on the things that make you happy outside of work it will recharge your spirit more quickly and ultimately help keep you happy as well as productive at work.

Did I miss anything?  Have any other helpful tips that you’d like others to know about?  If so, let me know.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.

Deliberate Practice and System Administration (Part 3)

So far we have covered the following,

  • All the ideas, concepts and techniques needed for applying deliberate practice. (Part 1)
  • A framework for how to apply techniques of deliberate practice to system administration. (Part 2)

Now let’s wrap this all together and examine the last important piece in this series.  A real world example of deliberate practice in action and its association with expertise.  We can put some of these ideas and techniques into practice and build our own customized version for applying deliberate practice to system administration.

Practically everybody knows who Tiger Woods is and how incredibly skilled he is.  If you don’t I suggest you Google him.  However, few understand the level of dedication he has to his craft and how much he actually puts into the sport to compete at the level he does.  So let’s go ahead and take a look at what his daily routine consists and figure out what conclusions we can draw and how we can model a practice routine for system administration after this.

  • 6:30 a.m. – One hour of cardio. Choice between endurance runs, sprints or biking. 
  • 7:30 a.m. – One hour of lower weight training. 60-70 percent of normal lifting weight, high reps and multiple sets.
  • 8:30 a.m. – High protein/low-fat breakfast. Typically includes egg-white omelet with vegetables.
  • 9:00 a.m. – Two hours on the golf course. Hit on the range and work on swing.
  • 11:00 a.m. – Practice putting for 30 minutes to an hour.

Noon – Play nine holes.

  • 1:30 p.m. – High protein/low-fat lunch. Typically includes grilled chicken or fish, salad and vegetables.
  • 2:00 p.m. – Three-to-four hours on the golf course. Work on swing, short game and occasionally play another nine holes.
  • 6:30 p.m. – 30 minutes of upper weight training. High reps.
  • 7:00 p.m. – Dinner and rest.

That is pretty crazy.  So how do we model this to fit our purposes for system administration?

There are two obvious things that I think are crucial that we can borrow from this right away.  Exercise and healthy diet.  These are important facets because as I’ve talked about previously and has been proven many times, proper diet and exercise contribute to improved cognitive abilities.  I don’t suggest following Tiger’s workout or meal plan but I would suggest at least an hour of exercise as well as a healthy meal plan to help operate at optimal energy levels throughout the day.

Next, nearly all of the rest of his practice schedule revolves around improving very specific aspects of his game.  Thankfully we came up with some of these generalized aspects of improvement for system administration in Part 1 and Part 2 so we can put these to use in our own plan.  It is important that  we combine everything into one practice schedule that is challenging but is also realistic.  We also don’t want to go over 4-5 hours each day.  So here is the schedule I propose, feel free to adapt these any way you like:

  • 8:00 a.m – Breakfast.  Fresh made juice or several pieces of fruit + supplements (multivitamin, fish oil, vitamin D).
  • 9:00 a.m. – Check relevant news, new trends and tech, check mail, forums, etc.
  • 10:00 a.m. – Focused study on new or weak areas, 60 minutes on, 15 minutes off.  This can consist of reading, videos, audio.
  • 12:00 p.m. – Lunch.  Mixed salad.  The more vegetables the better!
  • 1:00 p.m. – Lab time.  Focus on strengthening and understanding of study topics.  This is where the most time and energy should be spent.
  • 4:00 – p.m. – Work on command line/programming skills and techniques.  This fulfills more of our hands on and practice time requirements.
  • 5:00 – p.m. – Gym.  Alternate days between cardio and strength training.
  • 7:00 – p.m. – Dinner and relaxation.
  • 9:00 – p.m. – Writing and reflections on the day.  Areas of improvement, etc.

Again, this is only a guideline.  I plan on updating this as I test these techniques and make refinements and adjustments to it.  As an example, I like to work out in the evening to help me relieve stress but many others (including Tiger like to take care of this in the morning), you just have to figure out what works best for your lifestyle, so I would definitely encourage you to experiment with what works and what doesn’t.  I’m very curious to know myself.  Since this is a first revision I think there will probably need to be a number of adjustments, but I look forward to trying it out and reporting back with some results!  If you have suggestions or have your own practice schedule let me know and I’ll definitely incorporate it into my routine.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Facebook.