Getting Python Fabric setup in Windows

This has really turned into a wild goose chase.  Initially my goal when I set out on this project was simply to get Fabric up and running so I could test out some different features on some network gear.  It seems like the Python integration in Windows is very different than it is in the Linux world where everything is all bundled up nice and neatly.  There are several separate, seemingly unrelated pieces that all need to fit together to get Python and Fabric working correctly in a Windows environment, which can be very perplexing at first, hence my need to write a post so I don’t have to remember all this complexity for next time.  I thought I might as well show people how I got this to work instead of picking and choosing different bits of information from the internet.

The following is a list of links that I have found to be helpful in getting everything up and going, flip back to here for the different resources and components:

There’s a few steps for getting up and running.  For basic Python functionality it should be enough to download and install Python via the basic installer in your Windows environment.  Accepting the defaults should be enough.  Also, I recommend going with Python 2.7, rather than 3.3 because it has much better backwards compatibility.  You will also want to double check to make sure you download the correct version for you OS as well, either 32-bit or 64-bit.

Once you have your Python install up and going you will want to get pip installed. You will use this tool to get Python modules because it aids tremendously with downloading, managing and installing useful Python code.

So to get up and running with pip, first make sure that you have the correctly matched version of Python and the pip installed for your environment.  For example the 2.7 pip installer will not work with a 3.3 Python installation.  Second, you will need to make sure you have the Distribute package installed in your Python environment as well.  This is the tool that will allow pip to work.  Once you have these modules installed you will need to switch to the directory where pip is installed (or add it to your ENV path variable).  For me it was located in the following location:

C:\Python27\Scripts\pip.exe

So the command to install Fabric would be as follows:

pip.exe install fabric

You would think that’s all you need to get fabric working right?  Well it turns out that using this method we do not have the correct version of Pycrypto installed.

pycrypto error

So using the link posted above go ahead and get the correct version of Pycrypto downloaded and installed (version 2.1.0).  That still doesn’t fix it though!  It just gets us to a different error.  I used this post and this post as a guide for getting the correct version of Pycrypto installed on the Windows machine.

Okay, so now we should have a fully functioning Python environment with Fabric installed.  The only main issue that remains at this point (to my knowledge at least) is that pip still doesn’t work quite right when attempting to install various Python packages.  To get that part working you will need MinGW32 installed (reference above for links).  But that is basically out of the scope of this post, I will write another post about it if there is any interest or you can ask me if you have issues as always.

The only other piece left then is to get Fabric up and going with our Cisco gear.  Take a look at the docs for basic usage on getting acquainted with Fabric, it is fairly straight forward for the most part.

One thing I was not aware of was the way Cisco CLI and devices would behave when using Fabric to control them remotely.  I was having issues with Fabric flaking out whenever I went into config mode on a Cisco switch.  It turns out that when you enter into config mode you are essentially dropped into a new shell and Fabric doesn’t have a nice way to deal with that.  So something like this will bomb out,

def test():
	run("conf t", shell=False)
	run("int 1/0/1", shell=False)
	run("no shut", shell=False)
	run("exit", shell=False)

The “conf t” command opens your new shell and the Cisco gear freaks out because it doesn’t know what to do with the next command.  I should also mention the shell=False is somewhat unrelated to this issue but it gets around Fabric trying to use bash as its default shell.  The workaround?  Use the open_shell command in Fabric and escape each command by using \n to escape to a new line.  So a sample command using this format would be something like the following,

def test():
	open_shell("conf t \n"
		   "ip name-server 1.1.1.1 \n"
		   "exit \n"
		   "exit \n"
		   )

Yeah this is sort of hacky, and I’m not sure if it will be able to do everything I am looking for but hey at least it kind of works.  I am currently looking for a more robust and easier way around this limitation so if you have any suggestions let me know.

Credit goes to markmm on reddit for letting me know about this workaround as well as the people who hang out on the #fabric irc channel on freenode.

Read More

Conversation history not saving in Outlook 2010

Recently I ran into an issue with a Lync environment (2010) where Lync conversations were not being saved to the “Conversation History” folder in Outlook (2010).  Luckily there is a quick way to fix this issue, through Exchange.  From the reading that I have done it seems like the most common reason this occurs is when a user in your Exchange environment reaches or surpasses 1,000 combined folders and sub folders in their mailbox.  The easiest way to check if a user has reached this threshold is to use the Exchange Management Shell to quickly take a look at their total combined mailbox folders using the following command.

(Get-MailboxFolderStatistics “user”).Count

Easy enough, often times this is enough to determine the cause.  But I have taken this command one step further and wrapped it into a little script that will go through your Exchange environment and record all users that have reached this threshold and place their display name as well as the number of folders/subfolders into a csv file for an easier to reference.  Here is the logic of the script.

$mailboxes = Get-Mailbox
$overlimit = @()

ForEach ($mailbox in $mailboxes) {

$mbxmember = New-Object PSObject
$folders = (Get-MailboxFolderStatistics $mailbox).Count
$mbxmember | Add-Member -MemberType NoteProperty -Name "Display Name" -Value $mailbox.DisplayName
$mbxmember | Add-Member -MemberType NoteProperty –Name “Folder Count” –Value $folders

	If ($folders -gt 1000) {
		$overlimit += $mbxmember
	}
}

$overlimit

That logic right there is very basic but will iterate over all mailboxes in the Exchange environment, grab those with over 1,000 folders/sub folders, place them into an array and output the array.  This will take a while depending on the size of your environment, so feel free to let it run in the background.  It is not a super intensive process, it just takes forever.  To get this into a CSV file use the following Powershell command, I have this script name Get-Folders.ps1 in this example.

.\Get-Folders.ps1 | Export-CSV users.csv

That should be it.  Not everybody will need this obviously but I found that it came in handy.

Read More

WTF Friday

Lately I have been building a Windows Hyper-V v3 clustered lab environment with all the bells and whistles.  It has been a great learning experience thus far and I can honestly say that I am enjoying Hyper-V overall thus far.  Recently I decided to take the plunge and begin experimenting with System Center Virtual Machine Manager (SCVMM), and managed to run across a bizarre issue last Friday.  The reason I am posting is because there were basically no real clues for this problem, so I would like to go over some of the various things that I looked and ultimately how this issue was resolved.  I feel this post may be useful to others because a lot of this stuff is relatively new and there wasn’t a ton of material out there on this specific problem to use as reference.

The installation process is relatively straight forward.  The environment I am using is Server 2012, so as a prerequisite you must use SCVMM 2012 w/SP1 in order for this to work.  If you are using 2008R2 you can use SCVMM 2012.  I used this guide as a reference for the installation instructions, which more or less go like this:

  1. Create your SCVMM accounts in AD.  scvmmadmin (admin account), scvmmsvc (service account), scvmmadmins (admin group).
  2. Install/point the SCVMM server to SQL 2012.  I won’t go over SQL installation because it is beyond the scope of this post.
  3. Install the prerequisites on your SCVMM server.  ADK for Windows 8, SQL 2012 native client, SQL 2012 command line utilities.
  4. Install SCVMM 2012 w/SP1.  VMM Management Server, VMM Console.
  5. Deploy agents to Hyper-V hosts.

This is easy enough to follow but I was getting suck on step 4 when I was attempting to install the Management Server and the Console.  The installation would choke about half way through with the following error:

A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local  .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command “winrm helpmsg hresult”.

WinRM error

Okay… WTF?  I knew that I already had WinRM installed and was on, but if you are not sure the quickest way to find out is to type winrm quickconfig from a command prompt.  You should get something similar to the following:

WinRM output

So we know WinRM is on and should be working.  Next, I checked the installation logs for clues.  They are located in C:\ProgramData\VMMLogs\SetupWizard.log.  I found the portion of the logs that indicated there were issues:

12:44:54:VMMPostinstallProcessor threw an exception: Threw Exception.Type: Microsoft.Carmine.WSManWrappers.WSManProviderException, Exception.Message: A Hardware Management error has occurred trying to contact server GMVM-TEST-04.gmrcnet.local .

WinRM: URL: [http://gmvm-test-04.gmrcnet.local:5985], Verb: [INVOKE], Method: [AssociateLibrary], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/AgentManagement]

Check that WinRM is installed and running on server GMVM-TEST-04.gmrcnet.local. For more information use the command "winrm helpmsg hresult".
12:44:54:StackTrace: at Microsoft.Carmine.WSManWrappers.ErrorContextParameterHelper.ThrowTranslatedCarmineException(WsmanSoapFault fault, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.RetrieveUnderlyingWMIErrorAndThrow(SessionCacheElement sessionElement, COMException ce)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, String url, Hashtable parameters, Type returnType, Boolean isCarmineMethod)
at Microsoft.Carmine.WSManWrappers.AgentManagement.AssociateLibrary(WsmanAPIWrapper wsmanObject, String CertificateSubjectName, String& ExportedCertificate, ErrorInfo& ErrorInfo)
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.AssociateDefaultLibraryServer()
at Microsoft.VirtualManager.Setup.VirtualMachineManagerHelpers.SetupLibraryShare()
at Microsoft.VirtualManager.Setup.InstallItemCustomDelegates.PangaeaServerPostinstallProcessor()
12:44:54:InnerException.Type: System.Runtime.InteropServices.COMException, InnerException.Message: The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol.
12:44:54:InnerException.StackTrace: at WSManAutomation.IWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.MyIWSManSession.Invoke(String actionUri, Object resourceUri, String parameters, Int32 flags)
at Microsoft.Carmine.WSManWrappers.WsmanAPIWrapper.Invoke(String actionUri, WSManUri targetUri, Hashtable parameters, Type returnType, Boolean isCarmineMethod, Boolean forceResponseCast)
12:44:54:ProcessInstalls: Running the PostProcessDelegate returned false.
12:44:54:ProcessInstalls: Running the PostProcessDelegate for PangaeaServer failed.... This is a fatal item. Setting rollback.
12:44:54:SetProgressScreen: FinishMinorStep.
12:44:55:ProcessInstalls: Rollback is set and we are not doing an uninstall so we will stop processing installs
12:44:55:****************************************************************
12:44:55:****Starting*RollBack*******************************************
12:44:55:****************************************************************

Incredibly useful, I know.  It is good to know where this stuff is located though just in case other issues arise that require troubleshooting like this.  So at this point I was dumbfounded and most of the stuff I found on Google was not helpful for my situation (I tried many different suggestions).

Finally I came across a post that mentioned disabling WinRM from Group Policy.  It just so happens that there is a policy in our test environment for enabling Powershell and remoting and all that jazz.  So I completely disabled the policy and was finally able to get SCVMM to install!  Here are the two policy settings you should take a look at first.

Computer Configuration > Administrative Templates > Windows Components > Windows Remote Management (WinRM) > WinRM Service

and

Computer Configuration > Administrative Templates > Windows Components > Windows PowerShell

I need to go back and verify that the root issue was caused by the WinRM portion of the policy, which I’m suspecting it is.  But if you run across this error look at your Group Policy settings!

The moral of the story:  Windows Management Framework 3.0 and more specifically the WinRM components of WMF 3.0 are delicate, even on Server 2012 (there have been major compatibility issues with earlier versions of Windows).  In my scenario Group Policy was somehow getting in the way (if you can understand and decipher those logs and how they relate to Group Policy let me know) of allowing SCVMM to install.

Read More

IT Conference List

I figure now is the perfect time create this list because I am experiencing some serious writers block plus I keep hearing about all of these great conferences and I am beginning to lose track of them all.  By no means is this list comprehensive or all encompassing, it is simply a collection of those that are 1.) located in the United Sates  2.) the most relevant to my career and therefore those which I find to be the most interesting, and 3.) well enough established and encompassing enough to be well known across the country.

My hope is that one day I will get to attend most, if not all of these conferences.  It’s doubtful but at least it’s a dream.  So to get started it will probably be helpful to break these up into different categories just to make things easier to read and understand. Initially (as of this writing) there is not really any method to the madness, this is more of a brain dump of all the interesting conferences I hear about.  Also, some of these have multiple conferences and categories so I will just group them into the main ones for readability.

Please, if you have a suggestion or idea to add to the list let me know and I will be sure to add it.

Conference Location Category Date
SCaLEX Las Angeles, CA Linux February
Schmoocon Washington DC Hacker/Security February
Cascadia Seattle, WA Sysadmin March
PyCon Location varies Programming March
Monitorama Location varies DevOps March
Chef Conf Location varies DevOps April
LOPSA-East New Brunswick, NJ Sysadmin May
EMCWorld Las Vegas, NV Vendor (EMC) May
Interop Las Vegas, NV Vendor May
Redhat Summit Boston, MA Linux June
HP Discover Las Vegas, NV Vendor (HP) June
Cisco Live! Location varies Vendor (Cisco) June
Blackhat Las Vegas, NV Hacker/Security July
DEF CON Las Vegas, NV Hacker/Security July/August
VMWorld San Francisco, CA Vendor (VMWare) August
DerbyCon Louisvill, KY Hacker/Security September
MEC Location varies Vendor (Microsoft) September
Puppetconf San Francisco, CA Sysadmin September
SkyDogCon Nashville, TN Hacker/Security October
Spiceworld Austin, TX Vendor (Spicworks) October
PhreakNIC Nashville, TN Hacker/Security October/November
LISA Location varies Sysadmin November
Toorcon Location varies Hacker/Security Date varies
SANS Location varies Sysadmin Date varies
B-Sides Multiple locations Hacker/Security Multiple dates
RSA Multiple locations Security Multiple dates
Usenix Multiple locations Sysadmin Multiple dates
TechEd Multiple locations Vendor/Sysadmin Multiple dates
Velocity Multiple locations Vendor/Sysadmin Multiple dates

I’m sure there will be more to come but this is all I could come up with for the time being.  As stated, I will be revisiting this post in the future to add and update the list.  I hope you find it useful, who knows maybe I will see some of you at these conferences.

Read More

Are you sitting on an expensive disaster?

Those who have been following my posts will have spotted that occasionally I discuss something less technical. If that sort of thing bores you – look away now.

Everyone I’ve ever met who’s been in IT for any length of time – whether it’s as a technician, a sysadmin or a helpdesk operator – knows that this is a fast-moving industry and sometimes businesses get left behind.

Whether that’s the server that for some reason is still running Exchange 5.5, the PC with an IBM logo on the front that’s still running Windows 2000 or the sudden, urgent need to restore a backup from some obscure tape format that we thought had died out circa 2001.

And we get to pick up the pieces.

There’s a simple reason for this: as a profession, we’re fantastically good at spending money. We can easily spend half an hour on Dell’s website and our employer walks away £thousands lighter.

However, we’re fantastically bad at explaining why we’re spending the money or what benefit it’ll bring. Few of us buy a new car when the old one still meets our needs and it’s still economical to maintain, yet we provide equipment that’s more-or-less maintenance free and expect our employers to replace it while it still meets their needs just fine.

Upshot? We get to explain that yes, you can still buy Exchange. But no, you can’t easily upgrade the fifteen year old server in the corner to the latest version.

Solution? Explain what you want in terms the business will understand: it should either make money, save money or reduce risk. If you can’t think of at least one good reason based on one of these three, you probably shouldn’t be recommending the solution in the first place.

Read More