OpenFlow is all the rage right now and since I just got done doing a product overview of it and its relation to the HP product line we just recently purchased, I thought I would get in a quick post about all of it while the topics and ideas are still fresh in my mind. So this post will be less of a technical post than usual and more of a detour about my thoughts on networking and the effect OpenFlow is having on it.
I am still trying to wrap my head around some of the key concepts and applications that OpenFlow has to offer but I think I am beginning to understand the core concepts behind it, and honestly I don’t understand all the OpenFlow hate and SDN bashing from other network professionals.
Even thought OpenFlow is a fresh concept for me I can already see potential benefits and possible use cases and I think that there is some great potential with SDN in general. There must be some interesting value here, otherwise there wouldn’t be so much interest by all of the heavy hitting networking industry leaders like IBM, Cisco, HP, Google, etc. collaborating and working on projects like OpenDaylight and Floodlight. Since the concepts and ideas behind OpenFlow are so new and are largely unexplored there is a very mysterious and exciting quality behind the technology and because of this I believe that creativity can help drive its development and adoption. The other nice part about OpenFlow is that it is an open standard so it can be developed and extended by whomever feels like participating or contributing (Cisco and its OnePK API and other vendor specific API’s are a different story) to the project and the code base. I am a huge proponent of Open Source and I feel like having an open standard creates better code and more opportunities for everybody involved, it doesn’t benefit one but rather the collective.
I also want to touch briefly on the technical side of OpenFlow for all the IT pros. Technology evolves and changes all the time, we’ve seen it time and again in our industry. If you are stubborn to the point that you won’t dedicate the time to learn something new just because its not what you are familiar with then you probably won’t have much of a future in IT and ops or at least a future going forward in the networking world. Sure you’ve built a career on your niche ability and skill set to solve complex and challenging networking problems, but that is not a unique quality. All IT professionals build their careers on their ability to do this (at least the good ones I’ve seen so far), and every other area of IT is subject to these same types of issues that new technology brings. In my opinion the haters just need to grow up and accept the fact that they will need to remodel their skills from time to time. It’s not that big of a deal. And besides, OpenFlow actually looks promising and looks like it will be a great tool for IT pros to utilize to solve interesting problems.
Rather than complain and find fault, embrace OpenFlow, because whether you like it or not, it will have its place in the networking world moving forward.
Based on a strange network problem recently I decided to put together some quick notes and a few tips on ways to improve your Wireshark experience based on my own experience with it. There are many, many more features that Wireshark has to offer, these just happen to be the most apparent ones I have found so far. Wireshark is extremely powerful and therefore extremely useful if used properly. At first it takes a while to get used to everything Wireshark has to offer but once you start to get the hang of how things work then it can be a great network troubleshooting tool. Basic knowledge of networking concepts should be assumed as well as familiarity of Wireshark for those who attempt to debug network problems using this tool.
Here is a list of some of the most common and handy features that you can utilize in Wireshark. I am not going to dive into great detail with most of these items because I honestly don’t have a ton of experience with all of them, I basically just wanted to point out the highlights.
Filtering in Wireshark is very handy.
Create custom profiles for different use cases (quickly select from bottom right hand corner).
Color filters are useful! (Right click a field in the packet trace and selelct colorized rule) The bottom left bar will tell you what variable you are looking at to make things easier when customizing.
Use Regex in wireshark using the “matches” clause to turn on regex patterns.
You can extract specific information from trace files on the command line using tshark.
Right click a packet and select “follow TCP/UDP stream” to debug a single network conversation.
Low delta times are good. If you see high deltas you should probably investigate things.
Here are some more concrete examples and a few basics of how to put these tips into practice. In Wireshark you can use either English or code like operators when filtering to help narrow down traffic and interesting networking patterns and issues. So for example, “==” and “eq” will behave in the same manner when applying filters. Other operators include <,>, !=, <=, >=, etc. Just like you would see in a typical programming language.
Use custom configuration profiles. If you look at packet traces often this will save you a tremendous amount of time if you are looking at specific types of traffic or are only interested in certain traffic patterns. For example, you spend a lot of time looking at many traces that fit the same type of criteria; by using custom profiles you can quickly adjust and modify the view in Wireshark to help quickly identify patterns and potentially issues by cycling through different, specific views. To begin creating custom profiles go to Edit -> Configuration Profiles and then either select the custom profile or create a new one to begin changing.
One handy trick is to disable TCP offload checks. If your packet captures are getting clogged up with a bunch of red and black with offload errors, this is place you should go to look first. There are a few places where this option can either be enabled or disabled. The easiest way to check these options is under Edit -> Preferences and then under the Protocols tree for UDP, TCP and IPv4 protocols. The example below shows what the options should look like for the TCP protocol. The TCP and UDP offload checks are disabled by default but the IPv4 needs to be manually unchecked. The specific option under the IPv4 protocol is labeled “Validate the IPv4 checksum if possible”, simply uncheck this and the red and black errors should disappear.
There is a capture option that allows you to resolve IP addresses to hostname, which I find can be very useful. To enable this option open up the Show capture options screen, there should be an option in there under name resolution called “Resolve network-layer names”. Simply check that box and you should have name resolution.
As mentioned in the bullets above, the “follow UDP/TCP stream” option can be extremely useful and is a very quick way to glean information. It is so useful because it is so easy to use. Simply find a traffic conversation you would like to debug and right click the packet number in the top Wireshark pane and choose the follow UDP/TCP stream option and you can get an idea of everything that happened during a particular conversation. For example, using this technique you can follow FTP transactions.
Viewing a breakdown of the packet flow and traffic patterns can be a useful tool as well when diagnosing various network issues. There is an option in Wireshark that shows in good detail the breakdown of various packets and protocols that can be used to troubelshoot the network. This option is called Protocol Hierarchy Statistics and can be found un te Statistics -> Protocol Hierarchy Statistics page.
Only look at traffic for one IP address:
ip.addr==192.168.103.104
Likewise, filter out all traffic from an IP address:
Every once and awhile you will probably encounter a situation where you need to enable and then use telnet in a security focused environment. In certain situations telnet can be a great tool to test the functionality of firewall rule. Iif you aren’t certain whether or not a rule is working telnet can be a great way to help debug. The problem in Server 2008 and above is that telnet isn’t enabled by default. Luckily with PowerShell it is easy to enable the telnet functionality.
The following set of commands is a quick depiction of how you can enable telnet from a PowerShell prompt to ensure the ability of testing certain ports. Try it out.
Bam! As always, it is always easier to stay in command prompt and this is a great way to test port connectivity. I can understand why telnet is disabled by default on fresh server builds but sometimes it can become useful to have telnet as a tool to test connectivity. If you would like to debate the merits of disabling/enabling telnet on a server just drop me a line, I obviously will not be focusing on this aspect here. Anyway, just as easily as it is to enable telnet through PowerShell it can be disabled with the following command. If you already have the server manager module imported, skip to the second command.
I recently returned from my trip to Orlando and the Cisco Live! 2013 conference this year, so I thought I would take some time to reflect and go over the experience and report on some key highlights I was able to take away from this years conference. This year was my first Cisco Live! event and I have to say I was really impressed with the experience as a whole. There were maybe a few gripes here and there but overall the event in its entirety was pretty awesome. So in this post I just want to discuss some of the details of the event further. I don’t have a lot else to report on so I will go ahead and get get going, and begin by going over some of the Cisco specific trends that I noticed. Of course this is all a subjective experience and some may disagree but here is what I felt to be generally true throughout much of the event.
Technological takeaways (in my personal experience).
Cisco is hedging a huge bet that SDN is going to take off in the immediate future.
Mobility is going to continue to explode and increase the diversity of networks so we need to prepare and build our network infrastructures to handle the drive towards mobility.
Cisco really drove home the concept of the future connectedness of devices by pushing their idea of “the internet of everything”. This is the concept that technological experiences will converge and be tightly coupled. One example that was presented was a seamless experience at a hotel. The real chunk of info to take away is that as technology continues to evolve we will need to adapt networks to suit these needs.
In general, I felt these were the main drivers and ideas for a lot of what will be happening in the future of networking, at Cisco and abroad. Obviously Cisco was there to push their products so I will go ahead and cover a few of the key ideas and products that Cisco believes will help drive these future changes
The maturation of the ISE platform. This will be the convergence of a number of disparate technologies Cisco currently offers into a unified identity and access platform, this will correlate with the increase of mobile and the BYOD movement.
The SDN components. Essentially this is Cisco One line of products focused on the evolving SDN space. This includes the OnePK toolkit for OpenFlow development, the One controller for OpenFlow traffic control. There were more SDN components, I just can’t think of them right now.
New product introductions and evolutions. The Nexus 7710 and 7718 for scaling out the data center, the 6800 series to augment the capabilities of the 6500 series, improving performance, scale and speed.
There were many more announcements and products covered but to me, these aforementioned products were the main focus and effort. If I missed anything you thought was important let me know. Now that I have the big announcements covered I’d like to cover some of the other key highlights from the event.
The Good
Organization. Everything from hotel shuttles, information kiosks, to a very helpful event staff. I must say the event planners and organizers really thought things through (for the most part).
Deep dive sessions. The presenters were often the people who helped create the RFC’s or were responsible for writing the code. You can’t get much closer to the source than that. The few presenters I spoke with were all super nice people as well.
Free certification tests. This ranged all the way from CCNA all the way up to CCIE tests.
Universal Studios. Free food, amazing rides, it was just a great all around experience. Plus free booze, so you know, that was pretty awesome.
Journey. Do I even need to say more?
World of Solutions. This was their product and demo floor. Other than the fact that I sold my sole, I learned a lot here and was introduced to a ton of new products I otherwise would not have known about, plus I got about 20 t-shirts. Also free booze here as well.
Keynote speech by sir Richard Branson. It took on the format of a question/answer type interview, it was really cool to hear Branson talk and answer questions so candidly. No free booze but I gained some respect for him.
The Bad
The mobile apps. It almost seemed like this was an afterthought because much of the functionality either didn’t work at all or was crippled. It was a good idea but the execution was lacking, I’m sure this will get fixed next year.
The website was down the first day, due to a load balancer that broke. This caused a lot of confusion and problems, but I was able to print my schedule out at a kiosk so it wasn’t a huge issue for me.
The shuttle to and from Universal was a disaster for me. Many others didn’t experience this issue but it took about 45 minutes to get to the theme park and at least an hour to get back to my hotel. I can’t really complain looking back but it was frustrating at the time.
I would definitely recommend that anybody responsible for supporting their network at any capacity to attend this event at least one time. One nice thing about this event is that it doesn’t matter what skill level you are at, all ranges were covered and represented. I am lucky that I was able to attend this year and am very thankful. This was a great experience, it was incredibly eye opening and the positive effect it had on my own thought process can’t be overstated. I think that it will benefit me throughout my career and hopefully can be used to create opportunities for myself in the future.
In the next few posts I will be going over some of the basics on how to get the Nexus 1000v setup and working in a Hyper-V environment. I must warn readers ahead of time, this product was just released (as of a week or two ago) and the Cisco documentation is seriously lacking. What documentation that does exist is thoroughly confusing so it may take some time to work through all of the issues. Just as much if not more irritating, the Hyper-V way of doing things is just as confusing. Taking on a project like this will surely improve your skills and abilities with virtualization, especially network virtualization. I must admit, this stuff can get very confusing at first so it is important to realize that you might not understand everything at first, just be patient, it will eventually start making more sense.
First I need to lay some ground work. I think it’s important not only in this example but a good habit in general to spec out a project and figure out all of the requirements in order to make sure you have everything lined up that you might need before tackling a project. A few important considerations when working with the 1000v are to make sure the networking and NIC’s on the Hyper-V hosts are set correctly, Virtual Machine Manager (SCVMM) is installed and configured, the network is configured (LACP port channels, trunk ports, correct VLAN assignment, etc) and that configuring all of these pieces won’t cause any downtime or other issues with your production network. Ideally, all of this would be thought of and set up ahead of time. Luckily I have a test environment as well as SCVMM in my test environment to test this with and do not have to worry about any real world down time or production issues.
One of the most important things to get established is getting the underlying Hyper-V network stack configured properly. I try to mimic a production type environment as much as possible so this configuration is a typical design you may see in the real world. So let’s lay out the structure of the design.
Management VLAN(s)
DMZ VLAN(s)
Inside VLAN(s)
Live Migration VLAN(s)
It is common to break these out through different physical connections, so as an example you might see 4 different NIC’s on the Hyper-V host connecting to a switch that has 4 different VLAN’s configured. If you want redundancy you can add NIC teaming into this scenario (which is native in Server 2012 now, which is nice). I have limited resources so I am using a single NIC for management, DMZ and live migration traffic, and teaming the inside connection with 2 NIC’s. Here is a crude example of how this is setup.
If you are setting this up in a clustered environment, you would want these settings to be identical across all Hyper-V hosts. Once this is setup correctly make sure you have SCVMM installed and configured. That is a separate process and therefore is out of the scope of this post, I’d be happy to answer any questions you have, I’m just not discussing it here. You will need to grab the Cisco Nexus 1000v for Hyper-V. To download the files necessary for installation (let me know if you don’t have one) you will need a valid Cisco ID. Cisco also provides some documentation as well as some installation videos links but I have found them to be less than helpful to be honest, there is some useful information to be sure, I just want to walk you through the process myself because there were a few caveats and the documentation creates a lot of unnecessary confusion.
There is some basic terminology to be familiar with when getting the 1000v up and going that helps to understand how and why different parts work the way that they do when running through the installation.
vsm – virtual superviser module. This logically controls the virtual switch and can be thought of as a virtual line card to manage the different VEMs.
vem – virtual ethernet module. This is the piece that actually replaces the virtual switch
nsm – network segmentation manager.
Once you have the 1000v downloaded you need to make sure you run the installation for it on the server that is hosting SCVMM. The installer is hidden in the following location,
When you run this executable it should bring up a GUI to install and configure the virtual switch(es). You will need to use an account that is a member of the SCVMMAdmins group in Active Directory, otherwise the installer will not be able to connect to SCVMM and will not be able to create and configure the VM for the new virtual switch.
The next portion of the installer is where things may get confusing if you don’t know what you are looking for. I have linked to the sample configuration I used in my lab to help with this. Since this is what I used in my test environment I know at least at one point this configuration worked. It would be a good idea to deploy the VSM’s in high availability if you can, otherwise it isn’t a big deal.
Choose a meaningful name for VSM name, basically this is the same as the host name.
The ISO linstall location is, \Nexus1000v.5.2.1.SM1.5.1\VSM\Install\nexus-1000v.5.2.1.SM1.5.1.iso.
From the documentation I’ve read the VEM MSI location indicated is a little misleading because it points at the wrong installation file. It should point at \Nexus1000v.5.2.1.SM1.5.1\VMM\Nexus1000V-VSEMProvider-5.2.1.SM1.5.1.0.msi.
The VSM IP address should be an address in your management network, it can basically be thought of as the address to use to connect to the 1000v virtual switch.
Subnet mask should be fine as 255.255.255.0.
Gateway IP should match up with the VSM IP address, essentially they just need to be on the same subnet.
Domain ID is an arbitrary number that is associated with the virtual network. For most use cases you should be able to use one ID, 1000 in my example.
Use the VLAN ID that your VSM is on, in my case it is my management ID.
Since our management VLAN is that same as the VSM VLAN (typical in most deployments) simply choose “Yes” here.
At this point everything should be configured, the installer just needs to go out and create the VM’s and take care of getting everything up and running. It may take awhile so take a break if needed and come back later.
Everything should complete successfully, if not you will need to look at the log file and troubleshoot any errors you may have.
Almost done. Everything should be out there and running but there is still one very important step left. If you notice, about halfway down the installation summary page there is a username/password of admin and admin. This obviously will change once the 1000v gets put into use but there is NOTHING in the documentation that tells you that this will break the configuration in SCVMM!
What you need to do is hop on the SCVMM server and manually configure the credentials that are used to connect to the 1000v switch. To do this, drill down into the security settings in SCVMM by flipping open the Configuration pane -> Security -> Runas accounts -> Right click your 1000v admin account and select properties.
Then you will change the username and password to match the credentials that you have set on the 1000v. This will allow the switch to communicate with the SCVMM server so that 1000v network settings can be managed through Hyper-V.
In Part 2 I will discuss the intricacies of configuring the 1000v as well as how to reflect these settings in your Hyper-V virtual environment. Since this is a brand new product, there are still some things yet that need to get worked out, especially the documentation. And as I mentioned earlier, the network settings in Hyper-V and SCVMM can be extremely confusing the first time you see them. Working through and troubleshooting these issues will quickly help improve your knowledge and understanding of how Hyper-V and the Nexus 1000v work together to improve virtual networking. If you have any questions or concerns about any of this I will try to help, but I am not promising anything at this point.
.
