WTF Friday

Lately I have been building a Windows Hyper-V v3 clustered lab environment with all the bells and whistles.  It has been a great learning experience thus far and I can honestly say that I am enjoying Hyper-V overall thus far.  Recently I decided to take the plunge and begin experimenting with System Center Virtual Machine Manager (SCVMM), and managed to run across a bizarre issue last Friday.  The reason I am posting is because there were basically no real clues for this problem, so I would like to go over some of the various things that I looked and ultimately how this issue was resolved.  I feel this post may be useful to others because a lot of this stuff is relatively new and there wasn’t a ton of material out there on this specific problem to use as reference.

The installation process is relatively straight forward.  The environment I am using is Server 2012, so as a prerequisite you must use SCVMM 2012 w/SP1 in order for this to work.  If you are using 2008R2 you can use SCVMM 2012.  I used this guide as a reference for the installation instructions, which more or less go like this:

  1. Create your SCVMM accounts in AD.  scvmmadmin (admin account), scvmmsvc (service account), scvmmadmins (admin group).
  2. Install/point the SCVMM server to SQL 2012.  I won’t go over SQL installation because it is beyond the scope of this post.
  3. Install the prerequisites on your SCVMM server.  ADK for Windows 8, SQL 2012 native client, SQL 2012 command line utilities.
  4. Install SCVMM 2012 w/SP1.  VMM Management Server, VMM Console.
  5. Deploy agents to Hyper-V hosts.

This is easy enough to follow but I was getting suck on step 4 when I was attempting to install the Management Server and the Console.  The installation would choke about half way through with the following error:

A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local  .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command “winrm helpmsg hresult”.

WinRM error

Okay… WTF?  I knew that I already had WinRM installed and was on, but if you are not sure the quickest way to find out is to type winrm quickconfig from a command prompt.  You should get something similar to the following:

WinRM output

So we know WinRM is on and should be working.  Next, I checked the installation logs for clues.  They are located in C:\ProgramData\VMMLogs\SetupWizard.log.  I found the portion of the logs that indicated there were issues:

12:44:54:VMMPostinstallProcessor threw an exception: Threw Exception.Type: Microsoft.Carmine.WSManWrappers.WSManProviderException, Exception.Message: A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command "winrm helpmsg hresult".
12:44:54:StackTrace: at Microsoft.Carmine.WSManWrappers.ErrorContextParameterHelper.ThrowTranslatedCarmineException(WsmanSoapFault fault, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.RetrieveUnderlyingWMIErrorAndThrow(SessionCacheElement sessionElement, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, String url, Hashtable parameters, Type returnType, Boolean isCarmineMethod)
at Microsoft.Carmine.WSManWrappers.AgentManagement.AssociateLibrary(WsmanAPIWrapper wsmanObject, String CertificateSubjectName, String& ExportedCertificate, ErrorInfo& ErrorInfo)
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.AssociateDefaultLibraryServer()
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.SetupLibraryShare()
at Microsoft.VirtualManager.Setup.InstallItemCustomDelegates.PangaeaServerPostinstallProcessor()
12:44:54:InnerException.Type: System.Runtime.InteropServices.COMException, InnerException.Message: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol.
12:44:54:InnerException.StackTrace: at WSManAutomation.IWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.MyIWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
12:44:54:ProcessInstalls: Running the PostProcessDelegate returned false.
12:44:54:ProcessInstalls: Running the PostProcessDelegate for PangaeaServer failed.... This is a fatal item. Setting rollback.
12:44:54:SetProgressScreen: FinishMinorStep.
12:44:55:ProcessInstalls: Rollback is set and we are not doing an uninstall so we will stop processing installs
12:44:55:****************************************************************
12:44:55:****Starting*RollBack*******************************************
12:44:55:****************************************************************

Incredibly useful, I know.  It is good to know where this stuff is located though just in case other issues arise that require troubleshooting like this.  So at this point I was dumbfounded and most of the stuff I found on Google was not helpful for my situation (I tried many different suggestions).

Finally I came across a post that mentioned disabling WinRM from Group Policy.  It just so happens that there is a policy in our test environment for enabling Powershell and remoting and all that jazz.  So I completely disabled the policy and was finally able to get SCVMM to install!  Here are the two policy settings you should take a look at first.

Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Service

and

Computer Configuration > Administrative Templates > Windows Components > Windows PowerShell

I need to go back and verify that the root issue was caused by the WinRM portion of the policy, which I’m suspecting it is.  But if you run across this error look at your Group Policy settings!

The moral of the story:  Windows Management Framework 3.0 and more specifically the WinRM components of WMF 3.0 are delicate, even on Server 2012 (there have been major compatibility issues with earlier versions of Windows).  In my scenario Group Policy was somehow getting in the way (if you can understand and decipher those logs and how they relate to Group Policy let me know) of allowing SCVMM to install.

About the Author: Josh Reichardt

Josh is the creator of this blog, a system administrator and a contributor to other technology communities such as /r/sysadmin and Ops School. You can also find him on Twitter and Google+.
  1. Can you please tell me the policies you disabled.I am just facing the similar error

    • Hi Hashir,

      Here is where to look if you have WinRM turned on in GP. You might also check your Powershell settings, make sure either remote signed or unsigned are on.

      Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Service

      Computer Configuration > Administrative Templates > Windows Components > Windows PowerShell

  2. I am also facing this issue, and I cannot disable the group policies, I have tried to go into the permissions security tab of the policies in question and check deny on the apply group policy tabs, but it seems that the gpos are still being applied. When I run winrm enumerate … I see that the source of enabling credssp is still GPO, even though I have configured the 2 Credssp and 2 Winrm poilicies to deny apply group policy on the computer account of the server, as well as the user account of the VMM server, but it seems to be stuck. I am troubleshooting this for 3 days now, thanks for the post, I just wish there was more information on what exactly needs to be done before installing VMM for winrm to work correctly, especially if there are already gpos in place, Microsoft does not mention this anywhere.

    • I would try either moving the machine temporarily to an OU that doesn’t have the Group Policy applied to it or simply removing the specific policy as a test to see if it helps.

      Another thought is to leave the policy as “not configured” for Winrm and Credssp instead of “deny”. I don’t think MS has all the kinks worked out yet, but I definitely think it is something with the way GP’s are handled.

  3. Leaving IPv6 filter blank in GPO makes WinRM not listen on IPv6 addresses. Insert an asterix (*) in the filter. It worked for me.

  4. Awesome stuff. Ran into this on SCVMM2012R2 today. Thanks for the time saved.

    Greg

  5. SCVMM2012 Failed Installed | ittechabdulwaheedsk - pingback on January 28, 2014 at 10:55 am

Leave a Comment

Powered by sweetCaptcha


Trackbacks and Pingbacks: