Category Archives: Virtuallization

Nexus 1000v in a Hyper-V 2012 Environment (Part 1)

In the next few posts I will be going over some of the basics on how to get the Nexus 1000v setup and working in a Hyper-V environment.  I must warn readers ahead of time, this product was just released (as of a week or two ago) and the Cisco documentation is seriously lacking.  What documentation that does exist is thoroughly confusing so it may take some time to work through all of the issues.  Just as much if not more irritating, the Hyper-V way of doing things is just as confusing.  Taking on a project like this will surely improve your skills and abilities with virtualization, especially network virtualization.  I must admit, this stuff can get very confusing at first so it is important to realize that you might not understand everything at first, just be patient, it will eventually start making more sense.

First I need to lay some ground work.  I think it’s important not only in this example but a good habit in general to spec out a project and figure out all of the requirements in order to make sure you have everything lined up that you might need before tackling a project.  A few important considerations when working with the 1000v are to make sure the networking and NIC’s on the Hyper-V hosts are set correctly, Virtual Machine Manager (SCVMM) is installed and configured, the network is configured (LACP port channels, trunk ports, correct VLAN assignment, etc) and that configuring all of these pieces won’t cause any downtime or other issues with your production network.  Ideally, all of this would be thought of and set up ahead of time.  Luckily I have a test environment as well as SCVMM in my test environment to test this with and do not have to worry about any real world down time or production issues.

One of the most important things to get established is getting the underlying Hyper-V network stack configured properly.  I try to mimic a production type environment as much as possible so this configuration is a typical design you may see in the real world.  So let’s lay out the structure of the design.

  • Management VLAN(s)
  • DMZ VLAN(s)
  • Inside VLAN(s)
  • Live Migration VLAN(s)

It is common to break these out through different physical connections, so as an example you might see 4 different NIC’s on the Hyper-V host connecting to a switch that has 4 different VLAN’s configured.  If you want redundancy you can add NIC teaming into this scenario (which is native in Server 2012 now, which is nice).  I have limited resources so I am using a single NIC for management, DMZ and live migration traffic, and teaming the inside connection with 2 NIC’s.  Here is a crude example of how this is setup.

Hyper-V architecture

If you are setting this up in a clustered environment, you would want these settings to be identical across all Hyper-V hosts.  Once this is setup correctly make sure you have SCVMM installed and configured. That is a separate process and therefore is out of the scope of this post, I’d be happy to answer any questions you have, I’m just not discussing it here.  You will need to grab the Cisco Nexus 1000v for Hyper-V.  To download the files necessary for installation (let me know if you don’t have one) you will need a valid Cisco ID.  Cisco also provides some documentation as well as some installation videos links but I have found them to be less than helpful to be honest, there is some useful information to be sure, I just want to walk you through the process myself because there were a few caveats and the documentation creates a lot of unnecessary confusion.

There is some basic terminology to be familiar with when getting the 1000v up and going that helps to understand how and why different parts work the way that they do when running through the installation.

  • vsm - virtual superviser module.  This logically controls the virtual switch and can be thought of as a virtual line card to manage the different VEMs.
  • vem - virtual ethernet module.  This is the piece that actually replaces the virtual switch
  • nsm – network segmentation manager.

Once you have the 1000v downloaded you need to make sure you run the installation for it on the server that is hosting SCVMM.  The installer is hidden in the following location,

\Nexus1000v.5.2.1.SM1.5.1\VSM\Installer_App\Cisco.Nexus1000VInstaller.UI.exe

When you run this executable it should bring up a GUI to install and configure the virtual switch(es).  You will need to use an account that is a member of the SCVMMAdmins group in Active Directory, otherwise the installer will not be able to connect to SCVMM and will not be able to create and configure the VM for the new virtual switch.

Authenticate to SCVMM

The next portion of the installer is where things may get confusing if you don’t know what you are looking for.  I have linked to the sample configuration I used in my lab to help with this.  Since this is what I used in my test environment I know at least at one point this configuration worked.  It would be a good idea to deploy the VSM’s in high availability if you can, otherwise it isn’t a big deal.

  • Choose a meaningful name for VSM name, basically this is the same as the host name.  
  • The ISO linstall location is, \Nexus1000v.5.2.1.SM1.5.1\VSM\Install\nexus-1000v.5.2.1.SM1.5.1.iso.  
  • From the documentation I’ve read the VEM MSI location indicated is a little misleading because it points at the wrong installation file.  It should point at \Nexus1000v.5.2.1.SM1.5.1\VMM\Nexus1000V-VSEMProvider-5.2.1.SM1.5.1.0.msi.  
  • The VSM IP address should be an address in your management network, it can basically be thought of as the address to use to connect to the 1000v virtual switch.  
  • Subnet mask should be fine as 255.255.255.0.
  • Gateway IP should match up with the VSM IP address, essentially they just need to be on the same subnet.
  • Domain ID is an arbitrary number that is associated with the virtual network.  For most use cases you should be able to use one ID, 1000 in my example.
  • Use the VLAN ID that your VSM is on, in my case it is my management ID.
  • Since our management VLAN is that same as the VSM VLAN (typical in most deployments) simply choose “Yes” here.

1000v deployment config

At this point everything should be configured, the installer just needs to go out and create the VM’s and take care of getting everything up and running.  It may take awhile so take a break if needed and come back later.

Wait for the installation to finish

Everything should complete successfully, if not you will need to look at the log file and troubleshoot any errors you may have.

Installation summary

Almost done.  Everything should be out there and running but there is still one very important step left.  If you notice, about halfway down the installation summary page there is a username/password of admin and admin.  This obviously will change once the 1000v gets put into use but there is NOTHING in the documentation that tells you that this will break the configuration in SCVMM!

What you need to do is hop on the SCVMM server and manually configure the credentials that are used to connect to the 1000v switch.  To do this, drill down into the security settings in SCVMM by flipping open the Configuration pane -> Security -> Runas accounts -> Right click your 1000v admin account and select properties.

Updating the admin account in SCVMM

Then you will change the username and password to match the credentials that you have set on the 1000v. This will allow the switch to communicate with the SCVMM server so that 1000v network settings can be managed through Hyper-V.

In Part 2 I will discuss the intricacies of configuring the 1000v as well as how to reflect these settings in your Hyper-V virtual environment.  Since this is a brand new product, there are still some things yet  that need to get worked out, especially the documentation.  And as I mentioned earlier, the network settings in Hyper-V and SCVMM can be extremely confusing the first time you see them.   Working through and troubleshooting these issues will quickly help improve your knowledge and understanding of how Hyper-V and the Nexus 1000v work together to improve virtual networking.  If you have any questions or concerns about any of this I will try to help, but I am not promising anything at this point.

Fix the “Can’t create highly Available VM” error in VMM

I was doing some lab testing with Virtual Machine Manager (VMM) for my Hyper-V environment the other day when I ran across this issue.  It was happening because I was attempting to save my disk image to a CSV location and I couldn’t figure out why it was blowing up.

It turns out that the answer is really simple actually but is annoying to fix.  So here is what the error looks like in VMM.  “Cannot create or update a non highly available virtual machine because the path cluster storage volume is a clustered resource error”.  Surprisingly Google didn’t turn up much for how to fix this so I thought I would make a note of it in case others happen to run across this.

VMM error

The fix is simple enough, I just wish VMM was smart enough to know when you select a CSV to use for storage.  When you go through the VM creation process and you get to the hardware selection tab, choose the “Make this virtual machine highly available” check box.

VM high availability

If you happen to be building a lot of identical (or near identical) VM’s I reccomend checking out Profiles through VMM, in this case a specific hardware profile for desktop VM’s.  Creating profiles simplifies the build and creation process and you don’t need to worry about picking all the right defaults without having your VM creation blow up.

WTF Friday

Lately I have been building a Windows Hyper-V v3 clustered lab environment with all the bells and whistles.  It has been a great learning experience thus far and I can honestly say that I am enjoying Hyper-V overall thus far.  Recently I decided to take the plunge and begin experimenting with System Center Virtual Machine Manager (SCVMM), and managed to run across a bizarre issue last Friday.  The reason I am posting is because there were basically no real clues for this problem, so I would like to go over some of the various things that I looked and ultimately how this issue was resolved.  I feel this post may be useful to others because a lot of this stuff is relatively new and there wasn’t a ton of material out there on this specific problem to use as reference.

The installation process is relatively straight forward.  The environment I am using is Server 2012, so as a prerequisite you must use SCVMM 2012 w/SP1 in order for this to work.  If you are using 2008R2 you can use SCVMM 2012.  I used this guide as a reference for the installation instructions, which more or less go like this:

  1. Create your SCVMM accounts in AD.  scvmmadmin (admin account), scvmmsvc (service account), scvmmadmins (admin group).
  2. Install/point the SCVMM server to SQL 2012.  I won’t go over SQL installation because it is beyond the scope of this post.
  3. Install the prerequisites on your SCVMM server.  ADK for Windows 8, SQL 2012 native client, SQL 2012 command line utilities.
  4. Install SCVMM 2012 w/SP1.  VMM Management Server, VMM Console.
  5. Deploy agents to Hyper-V hosts.

This is easy enough to follow but I was getting suck on step 4 when I was attempting to install the Management Server and the Console.  The installation would choke about half way through with the following error:

A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local  .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command “winrm helpmsg hresult”.

WinRM error

Okay… WTF?  I knew that I already had WinRM installed and was on, but if you are not sure the quickest way to find out is to type winrm quickconfig from a command prompt.  You should get something similar to the following:

WinRM output

So we know WinRM is on and should be working.  Next, I checked the installation logs for clues.  They are located in C:\ProgramData\VMMLogs\SetupWizard.log.  I found the portion of the logs that indicated there were issues:

12:44:54:VMMPostinstallProcessor threw an exception: Threw Exception.Type: Microsoft.Carmine.WSManWrappers.WSManProviderException, Exception.Message: A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command "winrm helpmsg hresult".
12:44:54:StackTrace: at Microsoft.Carmine.WSManWrappers.ErrorContextParameterHelper.ThrowTranslatedCarmineException(WsmanSoapFault fault, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.RetrieveUnderlyingWMIErrorAndThrow(SessionCacheElement sessionElement, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, String url, Hashtable parameters, Type returnType, Boolean isCarmineMethod)
at Microsoft.Carmine.WSManWrappers.AgentManagement.AssociateLibrary(WsmanAPIWrapper wsmanObject, String CertificateSubjectName, String& ExportedCertificate, ErrorInfo& ErrorInfo)
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.AssociateDefaultLibraryServer()
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.SetupLibraryShare()
at Microsoft.VirtualManager.Setup.InstallItemCustomDelegates.PangaeaServerPostinstallProcessor()
12:44:54:InnerException.Type: System.Runtime.InteropServices.COMException, InnerException.Message: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol.
12:44:54:InnerException.StackTrace: at WSManAutomation.IWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.MyIWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
12:44:54:ProcessInstalls: Running the PostProcessDelegate returned false.
12:44:54:ProcessInstalls: Running the PostProcessDelegate for PangaeaServer failed.... This is a fatal item. Setting rollback.
12:44:54:SetProgressScreen: FinishMinorStep.
12:44:55:ProcessInstalls: Rollback is set and we are not doing an uninstall so we will stop processing installs
12:44:55:****************************************************************
12:44:55:****Starting*RollBack*******************************************
12:44:55:****************************************************************

Incredibly useful, I know.  It is good to know where this stuff is located though just in case other issues arise that require troubleshooting like this.  So at this point I was dumbfounded and most of the stuff I found on Google was not helpful for my situation (I tried many different suggestions).

Finally I came across a post that mentioned disabling WinRM from Group Policy.  It just so happens that there is a policy in our test environment for enabling Powershell and remoting and all that jazz.  So I completely disabled the policy and was finally able to get SCVMM to install!  Here are the two policy settings you should take a look at first.

Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Service

and

Computer Configuration > Administrative Templates > Windows Components > Windows PowerShell

I need to go back and verify that the root issue was caused by the WinRM portion of the policy, which I’m suspecting it is.  But if you run across this error look at your Group Policy settings!

The moral of the story:  Windows Management Framework 3.0 and more specifically the WinRM components of WMF 3.0 are delicate, even on Server 2012 (there have been major compatibility issues with earlier versions of Windows).  In my scenario Group Policy was somehow getting in the way (if you can understand and decipher those logs and how they relate to Group Policy let me know) of allowing SCVMM to install.