WTF Friday

April 1, 2013August 31, 2015 Josh Reichardt 1 Comment

Lately I have been building a Windows Hyper-V v3 clustered lab environment with all the bells and whistles. It has been a great learning experience thus far and I can honestly say that I am enjoying Hyper-V overall thus far. Recently I decided to take the plunge and begin experimenting with System Center Virtual Machine Manager (SCVMM), and managed to run across a bizarre issue last Friday. The reason I am posting is because there were basically no real clues for this problem, so I would like to go over some of the various things that I looked and ultimately how this issue was resolved. I feel this post may be useful to others because a lot of this stuff is relatively new and there wasn’t a ton of material out there on this specific problem to use as reference.

The installation process is relatively straight forward. The environment I am using is Server 2012, so as a prerequisite you must use SCVMM 2012 w/SP1 in order for this to work. If you are using 2008R2 you can use SCVMM 2012. I used this guide as a reference for the installation instructions, which more or less go like this:

Create your SCVMM accounts in AD. scvmmadmin (admin account), scvmmsvc (service account), scvmmadmins (admin group).
Install/point the SCVMM server to SQL 2012. I won’t go over SQL installation because it is beyond the scope of this post.
Install the prerequisites on your SCVMM server. ADK for Windows 8, SQL 2012 native client, SQL 2012 command line utilities.
Install SCVMM 2012 w/SP1. VMM Management Server, VMM Console.
Deploy agents to Hyper-V hosts.

This is easy enough to follow but I was getting suck on step 4 when I was attempting to install the Management Server and the Console. The installation would choke about half way through with the following error:

A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command “winrm helpmsg hresult”.

Okay… WTF? I knew that I already had WinRM installed and was on, but if you are not sure the quickest way to find out is to type winrm quickconfig from a command prompt. You should get something similar to the following:

So we know WinRM is on and should be working. Next, I checked the installation logs for clues. They are located in C:\ProgramData\VMMLogs\SetupWizard.log. I found the portion of the logs that indicated there were issues:

12:44:54:VMMPostinstallProcessor threw an exception: Threw Exception.Type: Microsoft.Carmine.WSManWrappers.WSManProviderException, Exception.Message: A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command "winrm helpmsg hresult".
12:44:54:StackTrace: at Microsoft.Carmine.WSManWrappers.ErrorContextParameterHelper.ThrowTranslatedCarmineException(WsmanSoapFault fault, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.RetrieveUnderlyingWMIErrorAndThrow(SessionCacheElement sessionElement, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, String url, Hashtable parameters, Type returnType, Boolean isCarmineMethod)
at Microsoft.Carmine.WSManWrappers.AgentManagement.AssociateLibrary(WsmanAPIWrapper wsmanObject, String CertificateSubjectName, String& ExportedCertificate, ErrorInfo& ErrorInfo)
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.AssociateDefaultLibraryServer()
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.SetupLibraryShare()
at Microsoft.VirtualManager.Setup.InstallItemCustomDelegates.PangaeaServerPostinstallProcessor()
12:44:54:InnerException.Type: System.Runtime.InteropServices.COMException, InnerException.Message: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol.
12:44:54:InnerException.StackTrace: at WSManAutomation.IWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.MyIWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
12:44:54:ProcessInstalls: Running the PostProcessDelegate returned false.
12:44:54:ProcessInstalls: Running the PostProcessDelegate for PangaeaServer failed.... This is a fatal item. Setting rollback.
12:44:54:SetProgressScreen: FinishMinorStep.
12:44:55:ProcessInstalls: Rollback is set and we are not doing an uninstall so we will stop processing installs
12:44:55:****************************************************************
12:44:55:****Starting*RollBack*******************************************
12:44:55:****************************************************************

Incredibly useful, I know. It is good to know where this stuff is located though just in case other issues arise that require troubleshooting like this. So at this point I was dumbfounded and most of the stuff I found on Google was not helpful for my situation (I tried many different suggestions).

Finally I came across a post that mentioned disabling WinRM from Group Policy. It just so happens that there is a policy in our test environment for enabling Powershell and remoting and all that jazz. So I completely disabled the policy and was finally able to get SCVMM to install! Here are the two policy settings you should take a look at first.

Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Service

and

Computer Configuration > Administrative Templates > Windows Components > Windows PowerShell

I need to go back and verify that the root issue was caused by the WinRM portion of the policy, which I’m suspecting it is. But if you run across this error look at your Group Policy settings!

The moral of the story: Windows Management Framework 3.0 and more specifically the WinRM components of WMF 3.0 are delicate, even on Server 2012 (there have been major compatibility issues with earlier versions of Windows). In my scenario Group Policy was somehow getting in the way (if you can understand and decipher those logs and how they relate to Group Policy let me know) of allowing SCVMM to install.

IT Conference List

March 28, 2013August 31, 2015 Josh Reichardt 15 Comments

I figure now is the perfect time create this list because I am experiencing some serious writers block plus I keep hearing about all of these great conferences and I am beginning to lose track of them all. By no means is this list comprehensive or all encompassing, it is simply a collection of those that are 1.) located in the United Sates 2.) the most relevant to my career and therefore those which I find to be the most interesting, and 3.) well enough established and encompassing enough to be well known across the country.

My hope is that one day I will get to attend most, if not all of these conferences. It’s doubtful but at least it’s a dream. So to get started it will probably be helpful to break these up into different categories just to make things easier to read and understand. Initially (as of this writing) there is not really any method to the madness, this is more of a brain dump of all the interesting conferences I hear about. Also, some of these have multiple conferences and categories so I will just group them into the main ones for readability.

Please, if you have a suggestion or idea to add to the list let me know and I will be sure to add it.

Conference	Location	Category	Date

SCaLEX	Las Angeles, CA	Linux	February
Schmoocon	Washington DC	Hacker/Security	February

Cascadia	Seattle, WA	Sysadmin	March
PyCon	Location varies	Programming	March
Monitorama	Location varies	DevOps	March

Chef Conf	Location varies	DevOps	April

LOPSA-East	New Brunswick, NJ	Sysadmin	May
EMCWorld	Las Vegas, NV	Vendor (EMC)	May
Interop	Las Vegas, NV	Vendor	May

Redhat Summit	Boston, MA	Linux	June
HP Discover	Las Vegas, NV	Vendor (HP)	June
Cisco Live!	Location varies	Vendor (Cisco)	June

Blackhat	Las Vegas, NV	Hacker/Security	July
DEF CON	Las Vegas, NV	Hacker/Security	July/August

VMWorld	San Francisco, CA	Vendor (VMWare)	August

DerbyCon	Louisvill, KY	Hacker/Security	September
MEC	Location varies	Vendor (Microsoft)	September
Puppetconf	San Francisco, CA	Sysadmin	September

SkyDogCon	Nashville, TN	Hacker/Security	October
Spiceworld	Austin, TX	Vendor (Spicworks)	October
PhreakNIC	Nashville, TN	Hacker/Security	October/November

LISA	Location varies	Sysadmin	November

Toorcon	Location varies	Hacker/Security	Date varies
SANS	Location varies	Sysadmin	Date varies
B-Sides	Multiple locations	Hacker/Security	Multiple dates
RSA	Multiple locations	Security	Multiple dates
Usenix	Multiple locations	Sysadmin	Multiple dates
TechEd	Multiple locations	Vendor/Sysadmin	Multiple dates
Velocity	Multiple locations	Vendor/Sysadmin	Multiple dates

I’m sure there will be more to come but this is all I could come up with for the time being. As stated, I will be revisiting this post in the future to add and update the list. I hope you find it useful, who knows maybe I will see some of you at these conferences.

Using Vim as a word processor

March 5, 2013August 31, 2015 Josh Reichardt 64 Comments

Recently I have been asked to share some of my content on a site called Ops School, a very cool site, that bills itself as “a comprehensive program that will help you learn to be an operations engineer”. It is essentially an online guide covering topics geared towards a successful career in IT. If you haven’t checked the site out already I highly suggest you go take a look! Like right now. Even better if you have something to contribute! Either join the mailing list or get going by joining the community over on github. Contributing to this project is a fantastic way to get your name on an Open Source project and would also be a great learning experience if that type of things is interesting to you. At least it has been for me so far.

Anyway, the project has a set of guidelines and styles posted on their site for authors to adhere by. Thus far I have found Vim to be the best word processor for following these styles and also the best way to submit writing to this project, plus it is a good way to force myself to make use of Vim because I don’t get much practice using it otherwise.

I have taken bits and pieces from various other vimrc’s I’ve found and fit them into my own unique scenario, which I suggest you do as well. But the following section is a great example to use a starting point for adding in the word processor functionality to your vimrc.

func! WordProcessorMode()
  setlocal formatoptions=t1
  setlocal textwidth=80
  map j gj
  map k gk
  setlocal smartindent
  setlocal spell spelllang=en_us
  setlocal noexpandtab
endfu
com! WP call WordProcessorMode()

One gotcha that I encountered with this setup initially was that lines didn’t automatically re-balance for me if I went back to a previous paragraph and made a change that caused a line to spill over the 80 character word wrap limit. To do align paragraphs, select the text that has come out of line and type “gq” to balance out the text in the paragraph again.

If you have question let me know. Otherwise, if you have any other tricks or tips that you like to use to enhance your Vim word processing experience feel free to let me know!

Protip March: Quickly viewing logs with Powershell

March 4, 2013August 31, 2015 Josh Reichardt 1 Comment

Wow it feels like it’s been forever since I have posted. I have been crazy busy with work stuff and am just now getting caught up with everything and have enough room to poke my head above the water and breath again finally. We had a massive overhaul of our data center in mid February (among other things) and I am finally getting all the loose ends tied up from that project, including our brand-spanking new test environment which I am super excited about and which I will post about in the not so distant future.

Here is proof of some of our efforts just in case you don’t believe me 🙂

Anyway, getting back on track, I just discovered a slick way in Powershell to mimic the functionality of tail and tail -f in the Linux world. If you have ever used tail then you know it is a great tool for monitoring log files or quickly looking at the end of a piece of code for example.

With the trick I’m about to show you, the same can essentially be done in Windows. However, there are a few caveats. For one, the syntax is a little bit different (if you want to change this just set up an alias). The Powershell equivalent relies on the Get-Content cmdlet with the -Tail and -Wait flags to accomplish this task.

So in the following example I have instructed Powershell to look at the last 30 lines of the uploadpic.ps1 file and using the -Wait flag it will be updated as the file gets appended to.

Get-Content -path .\uploadpic.ps1 -Tail 30 -Wait

If you don’t care about viewing the file live then you can remove the -Wait flag and Powershell will simply grab the last N number of lines where N is 30 in our example. 30 seems like a good enough number in our example and can obviously be changed depending on your needs. Easy enough for what I need it for.

Get-Content -path .\uploadpic.ps1 -Tail 30

As I mentioned, I will be going into a little more detail about some of the things I learned from our data center rebuild that I feel were some great lessons and good things to know/be aware of. Standby for new contents as I get back to writing more blog posts and getting back up to speed on the writing side of things.

Are you sitting on an expensive disaster?

February 5, 2013August 31, 2015 James Cort 20 Comments

Those who have been following my posts will have spotted that occasionally I discuss something less technical. If that sort of thing bores you – look away now.

Everyone I’ve ever met who’s been in IT for any length of time – whether it’s as a technician, a sysadmin or a helpdesk operator – knows that this is a fast-moving industry and sometimes businesses get left behind.

Whether that’s the server that for some reason is still running Exchange 5.5, the PC with an IBM logo on the front that’s still running Windows 2000 or the sudden, urgent need to restore a backup from some obscure tape format that we thought had died out circa 2001.

And we get to pick up the pieces.

There’s a simple reason for this: as a profession, we’re fantastically good at spending money. We can easily spend half an hour on Dell’s website and our employer walks away £thousands lighter.

However, we’re fantastically bad at explaining why we’re spending the money or what benefit it’ll bring. Few of us buy a new car when the old one still meets our needs and it’s still economical to maintain, yet we provide equipment that’s more-or-less maintenance free and expect our employers to replace it while it still meets their needs just fine.

Upshot? We get to explain that yes, you can still buy Exchange. But no, you can’t easily upgrade the fifteen year old server in the corner to the latest version.

Solution? Explain what you want in terms the business will understand: it should either make money, save money or reduce risk. If you can’t think of at least one good reason based on one of these three, you probably shouldn’t be recommending the solution in the first place.