Category Archives: Networking

Setting up a pfSense NAT instance in AWS

One important aspect of cloud deployments that often get overlooked, especially at start ups is the aspect of security.  So I thought I would take some time to go through the process of setting up a NAT instance on AWS with full firewall capabilities.  There are instructions and documentation for this process which are very good but aren’t completely clear so I will attempt to fill in some of the gaps I ran in to when attempting to set this up myself.

There is one thing to take note of if you have used pfSense before.  This firewall isn’t free.  There is a slight hourly charge for this that ends up coming out to about $500/yr (which comes out to about $42/month).  If you look at other commercial solutions with similar functionality you are looking at thousands of dollars per month in costs.  Long story short, the cloud images of pfSense has a tiny tiny cost associated with it but is very much worth it.

Just for reference I put together a few comparison prices.

  • Barracuda web app firewall – ($1.04-1.76/hr) (up to ~$1300/month)
  • Vyatta  ($0.30-1.50/hr) (up to ~$1100/month)
  • Sophos UTM ($0.35-$2.80/hr) (up to ~$2000/month)
  • pfSense ($0.07/hr) ($42/month)

As you can see, pfSense is very reasonable compared to some of the other bigger players.  You can build an r3.8xlarge instance and the software price won’t change which doesn’t seem to be the case with others.  One bonus to choosing pfSense is that you automatically qualify for support by agreeing to the ToS when getting the pfSense AMI set up.

Finally, pfSense is rock solid being built on top BSD and is thoroughly tested.  I have been running pfSense on other projects outside of AWS for 5+ years and have never had an issue with it outside of a dead hard drive one time.  Other added benefits of choosing pfSense are that updates are frequent and thoroughly tested, tons of add-ons including IPS’s and VPN’s so additional functionality can be built on top and great community support as well.

Getting started

There are a few good resources that I found to be useful when working through this problem, which got me most of the way to a working setup.  They are listed below.

And here is the link to my question about how to do this on serverfault, there is some good detail in the post over there.

Setting up the NAT in pfSense

The first issue that was confusing was the issue of getting the network interfaces set up and configured.  For this setup you will need two interfaces, preferably with static IP addresses.  You will also need to make sure that you disable source/destination checks for the interface that will be acting as the LAN interface that the nat goes through.  Disabling source and destination checks is pretty straightforward and is detailed in pretty much all of the guides.

You should note that there will be tabs for firewalling for LAN as well as WAN, if you can keep these two straight it should be much easier to troubleshoot and configure your pfSense machine.  Out of the box, the firewall on pfSense will not be configured to allow your LAN interface to do any sort of NATing, you will need to manually create rules to get started.  If you check the WAN firewall tab you should notice some access rules but the LAN tab should be empty.  Most of the work we will be doing will be on the LAN firewall.

The first rule to set up to make things easier to troubleshoot is a ping rule.  There is a WAN rule for ping but not for LAN.  You can essentially copy the WAN rule into a new one and modify it to look similar to the following.

LAN firewall rule

 

 

 

 

 

 

 

 

 

 

 

 

 

This rule will work for the template for the other rules that need to be put in to place.  The other rules will be for outbound web access.   Just copy this rule in to a new rule and change the protcol to TCP and make one rule that allows port 80 and another that allows 443.  The resulting should look similar to what I have listed below.

firewall rules for nat

Just a quick note.  If at any point you are having trouble seeing traffic or are getting stuck in your troubleshooting, an excellent way to figure out what is going on is the logging that is provided by pfSense.  You can access all of the various logs to see what is happening by selecting Status -> System Logs and the highlighting the firewall tab.

Modifying your outbound nat

Here is what your outbound NAT rule should look like.

outbound nat

 

 

 

 

 

 

 

 

 

 

 

 

 

Notice the “Networks_to_NAT” value in the source section.  This is a pfSense alias that can be used as a sort of variable to help ease management.  You can either use this alias or specify the local subnet you want to use here.  To check the values in your alias you can go to Firewall -> Aliases.

Conclusion

This setup will provide you with a nice easy way to manage your network in AWS.  The guides for setting up a NAT are nice and are a good first step but with a Firewall in place you can do so many other things, especially auditing that just aren’t available or viable with a straight AWS nat instance or that are way out of your price range with some of the other commercial solutions available.

pfSense also provides the capability to add more advanced tools like IDS/IPS, VPN and high availability if you choose so there is nice room for expansion.  Even if you don’t take advantage of all of the additional components of pfSense you will still have a rock solid firewall and nat instance that is suitable for production workloads at a fraction of the cost of other commercial solutions.

Fix Xbox Strict NAT on PFSense

Out of the box, it turns out that PFSense is not configured to handle some connection settings for Xbox Live.  Unfortunately I couldn’t find much of an explanation as to what this message actually means as far as degraded online performance but noticed that I would randomly get kicked out of games, get disconnected from XBox Live and have communication issues every once in a awhile so decided to take a look at what was actually going on because the mentioned issues started to get annoying.

I figured it should be easy enough to fix, but I couldn’t find a definite guide on how to fix this issue so I figured I would make sure it is clear for those who find this post and are having the same issue.  I tried a few different combinations, including port forward combinations mentioned in some forums, firewall rule changes, various UPnP settings, etc. but none of these combo’s worked and were unclear not very clear either.

Eventually I found this guide, which works and is great but doesn’t depict how to set everything up.  There are a few steps to get this working correctly so I will briefly describe them below.

Verify the IP address of you Xbox 360.  There is documentation around for finding it, but essentially go to system -> network -> advanced and it should give you the information.  You may want to set a static IP for your Xbox but I won’t cover that here.  Ask me if you have issues.

Now you will need to modify your firewall settings (Firewall -> NAT).  Choose the “Outbound” tab and change the mode to Manual Outbound NAT rule generation.  After you have saved the settings, create an entry for your Xbox and give it the address of your Xbox, with a mask of /32.

Firewall rule

Once this rule has been created, move it up to the top of the rule list.  You should have something similar to the following when done.

Firewall rules

Next, modify UPnP settings (Services -> UPnP & NAT-PMP) and select the following settings.

  • Enable UPnP & NAT-PMP
  • Allow UPnP port mapping
  • External Interface -> WAN
  • Interfaces -> LAN
  • User specified permissions 1- > allow 88-65535 192.168.39.17/32 88-65535

It should look something like this.

UPnP settings

Go ahead and save the settings and restart your Xbox (just turn off and on) to make sure the settings get picked up and that should be it.  I’m not entirely sure the user permissions need to be this wide open but it works so it is there for now.  I will update the post if I find any evidence that the settings need to be modified.

OpenFlow and the Future of Networking

OpenFlow is all the rage right now and since I just got done doing a product overview of it and its relation to the HP product line we just recently purchased, I thought I would get in a quick post about all of it while the topics and ideas are still fresh in my mind.  So this post will be less of a technical post than usual and more of a detour about my thoughts on networking and the effect OpenFlow is having on it.

I am still trying to wrap my head around some of the key concepts and applications that OpenFlow has to offer but I think I am beginning to understand the core concepts behind it, and honestly I don’t understand all the OpenFlow hate and SDN bashing from other network professionals.

Even thought OpenFlow is a fresh concept for me I can already see potential benefits and possible use cases and I think that there is some great potential with SDN in general.  There must be some interesting value here, otherwise there wouldn’t be so much interest by all of the heavy hitting networking industry leaders like IBM, Cisco, HP, Google, etc. collaborating and working on projects like OpenDaylight and Floodlight. Since the concepts and ideas behind OpenFlow are so new and are largely unexplored there is a very mysterious and exciting quality behind the technology and because of this I believe that creativity can help drive its development and adoption.  The other nice part about OpenFlow is that it is an open standard so it can be developed and extended by whomever feels like participating or contributing (Cisco and its OnePK API and other vendor specific API’s are a different story) to the project and the code base.  I am a huge proponent of Open Source and I feel like having an open standard creates better code and more opportunities for everybody involved, it doesn’t benefit one but rather the collective.

I also want to touch briefly on the technical side of OpenFlow for all the IT pros.  Technology evolves and changes all the time, we’ve seen it time and again in our industry.  If you are stubborn to the point that you won’t dedicate the time to learn something new just because its not what you are familiar with then you probably won’t have much of a future in IT and ops or at least a future going forward in the networking world.  Sure you’ve built a career on your niche ability and skill set to solve complex and challenging networking problems, but that is not a unique quality.  All IT professionals build their careers on their ability to do this (at least the good ones I’ve seen so far), and every other area of IT is subject to these same types of issues that new technology brings.  In my opinion the haters just need to grow up and accept the fact that they will need to remodel their skills from time to time.  It’s not that big of a deal.  And besides, OpenFlow actually looks promising and looks like it will be a great tool for IT pros to utilize to solve interesting problems.

Rather than complain and find fault, embrace OpenFlow, because whether you like it or not, it will have its place in the networking world moving forward.

Wireshark Reference Guide

Based on a strange network problem recently I decided to put together some quick notes and a few tips on ways to improve your Wireshark experience based on my own experience with it.  There are many, many more features that Wireshark has to offer, these just happen to be the most apparent ones I have found so far.  Wireshark is extremely powerful and therefore extremely useful if used properly.  At first it takes a while to get used to everything Wireshark has to offer but once you start to get the hang of how things work then it can be a great network troubleshooting tool.  Basic knowledge of networking concepts should be assumed as well as familiarity of Wireshark for those who attempt to debug network problems using this tool.

Here is a list of some of the most common and handy features that you can utilize in Wireshark.  I am not going to dive into great detail with most of  these items because I honestly don’t have a ton of experience with all of them, I basically just wanted to point out the highlights.

  • Filtering in Wireshark is very handy.
  • Create custom profiles for different use cases (quickly select from bottom right hand corner).
  • Color filters are useful!  (Right click a field in the packet trace and selelct colorized rule)  The bottom left bar will tell you what variable you are looking at to make things easier when customizing.
  • Use Regex in wireshark using the “matches” clause to turn on regex patterns.
  • You can extract specific information from trace files on the command line using tshark.
  • Right click a packet and select “follow TCP/UDP stream” to debug a single network conversation.
  • Low delta times are good.  If you see high deltas you should probably investigate things.

Here are some more concrete examples and a few basics of how to put these tips into practice.  In Wireshark you can use either English or code like operators when filtering to help narrow down traffic and interesting networking patterns and issues.  So for example, “==” and “eq” will behave in the same manner when applying filters.  Other operators include <,>, !=, <=, >=, etc.  Just like you would see in a typical programming language.

Use custom configuration profiles.  If you look at packet traces often this will save you a tremendous amount of time if you are looking at specific types of traffic or are only interested in certain traffic patterns.  For example, you spend a lot of time looking at many traces that fit the same type of criteria; by using custom profiles you can quickly adjust and modify the view in Wireshark to help quickly identify patterns and potentially issues by cycling through different, specific views.  To begin creating custom profiles go to Edit -> Configuration Profiles and then either select the custom profile or create a new one to begin changing.

One handy trick is to disable TCP offload checks.  If your packet captures are getting clogged up with a bunch of red and black with offload errors, this is place you should go to look first.  There are a few places where this option can either be enabled or disabled.  The easiest way to check these options is under Edit -> Preferences and then under the Protocols tree for UDP, TCP and IPv4 protocols.  The example below shows what the options should look like for the TCP protocol.  The TCP and UDP offload checks are disabled by default but the IPv4 needs to be manually unchecked.  The specific option under the IPv4 protocol is labeled “Validate the IPv4 checksum if possible”, simply uncheck this and the red and black errors should disappear.

There is a capture option that allows you to resolve IP addresses to hostname, which I find can be very useful.  To enable this option open up the Show capture options screen, there should be an option in there under name resolution called “Resolve network-layer names”.  Simply check that box and you should have name resolution.

As mentioned in the bullets above, the “follow UDP/TCP stream” option can be extremely useful and is a very quick way to glean information.  It is so useful because it is so easy to use.  Simply find a traffic conversation you would like to debug and right click the packet number in the top Wireshark pane and choose the follow UDP/TCP stream option and you can get an idea of everything that happened during a particular conversation.  For example, using this technique you can follow FTP transactions.

Viewing a breakdown of the packet flow and traffic patterns can be a useful tool as well when diagnosing various network issues.  There is an option in Wireshark that shows in good detail the breakdown of various packets and protocols that can be used to troubelshoot the network.  This option is called Protocol Hierarchy Statistics and can be found un te Statistics -> Protocol Hierarchy Statistics page.

Only look at traffic for one IP address:

ip.addr==192.168.103.104

Likewise, filter out all traffic from an IP address:

!(ip.addr == x.x.x.x)
!(ip.addr == 1.2.3.4)

filter out all traffic for a specific port

!(tcp.port eq 2222)

Resources:

http://ask.wireshark.org/questions/
http://ask.wireshark.org/questions/
http://www.wireshark.org/docs/

Enable telnet with PowerShell

Every once and awhile you will probably encounter a situation where you need to enable and then use telnet in a security focused environment.  In certain situations telnet can be a great tool to test the functionality of firewall rule. Iif you aren’t certain whether or not a rule is working telnet can be a great way to help debug.  The problem in Server 2008 and above is that telnet isn’t enabled by default.  Luckily with PowerShell it is easy to enable the telnet functionality.

The following set of commands is a quick depiction of how you can enable telnet from a PowerShell prompt to ensure the ability of testing certain ports.  Try it out.

Import-Module servermanager
Add-WindowsFeature telnet-client

Bam!  As always, it is always easier to stay in command prompt and this is a great way to test port connectivity.  I can understand why telnet is disabled by default on fresh server builds but sometimes it can become useful to have telnet as a tool to test connectivity.  If you would like to debate the merits of disabling/enabling telnet on a server just drop me a line, I obviously will not be focusing on this aspect here.  Anyway, just as easily as it is to enable telnet through PowerShell it can be disabled with the following command.  If you already have the server manager module imported, skip to the second command.

Import-Module servermanager
Remove-WindowsFeature telnet-client

That’s all it takes.  Very simple and very straightforward.