Using Find-String to grep in Powershell

January 31, 2013August 31, 2015 Josh Reichardt 7 Comments

For the longest time I have not been a fan of embracing the shell in the Windows world, but more and more I find that Powershell is able to do the things that I need. I suppose my seething hatred was in part due to my negative bias towards Windows and the lack of useful tools from the command line in Windows. Increasingly lately, I have been changing the way I think about and utilize the command line in Windows with Powershell. And to be honest, I’m really beginning to fall in love with Powershell the more I get work with it and the more I get to see how to apply it in Windows based environments. The good news is that Microsoft has put a lot of effort into this and are adding improvements and features all the time. It still has a long way to go, but I can already see this as an alternative to GUI based administration in the Windows world and finally begin to feel like I can see the promise land on the horizon. Using Powershell has been somewhat of a paradigm shift (in my opinion anyways) to how to do Windows administration recently and I feel like it will only get stronger and more common in the years to come. So in this post I will try to show you some of the flexibility as well as some of the power that Powershell has to offer with some great tools from the community.

To highlight what I am talking about, let’s talk about grep, a well known and loved tool in the *nix universe. I love grep. For the longest time, I hadn’t known of a way to grep in the Windows world until just recently with a wonderful third party Powershell module called Find-String. There are a few commands that you need to get this working. To start, we need to get a module installed that is basically used for package management. This tool is called PsGet. Installation is super simple, just run the following from a Powershell prompt, and ensure that your execution policy is set at least at remotesigned (Set-ExecutionPolicy RemoteSigned if you don’t have this turned on).

(new-object Net.WebClient).DownloadString(“http://psget.net/GetPsGet.ps1“) | iex

This should go out, download and install PsGet for you. Once all that has completed you should be able to install Powershell modules that are contained in the PsGet repository. With this installed we can just reach out to the PsGet repo and install our module.

Install-Module Find-String

Easy, simple, and clean. That is the best part, there isn’t a ton of leg work to get this stuff working correctly and is why I’m enjoying Powershell so much these days. Now we have a very functional grep clone! Usage is quite a bit similar to grep, fore help you can do a Get-Help Find-String or look at the link I posted earlier to the author’s github page. If you want to look for a word or substring in a file just use something like this:

Find-String word example.txt

This will output all occurrences of the word you are looking for in the file example.txt. Here is a screen shot to show you.

What other cool Powershell stuff are you doing? I would love to hear about other cool uses that can be beneficial in every day use.

Bonding network cards: or “How I learned to stop worrying and love 2Gb/s ethernet”.

January 8, 2013August 31, 2015 James Cort 1 Comment

Most servers ship with at least two network ports, often more.

And yet so often we plug one of them into a switch and ignore the other one. We’ve paid for an expensive server with very capable networking, and now we’re going to ignore half of its capabilities. Meanwhile, we’re asking it to do more and more. Sooner or later, that gigabit network port is the bottleneck.

Why not use both network ports simultaneously?

There’s various ways you can set this up. Some require special configuration of managed switches; some don’t. For this blog post, I’m going to concentrate on methods that don’t require special switch configuration because they’re a little bit easier and they’re somewhat less fragile – you don’t risk your network collapsing in a big heap just because someone plugged a network port into the wrong socket. (We also get the added bonus that if one of the two network ports in our server fails, it’ll still work, albeit more slowly. But I can’t remember the last time I saw a network port fail…)

These instructions are written purely for Debian Squeeze. You may need minor tweaks to use them in Ubuntu; you’ll almost certainly need significant changes to use them in other distributions.

First, install the ifenslave package:

apt-get install ifenslave

Configuration is just a few lines in /etc/network/interfaces:

# The primary network interface
auto bond0
iface bond0 inet static
    address 192.168.42.43
    broadcast 192.168.42.255
    netmask 255.255.255.0
    gateway 192.168.42.254
    network 192.168.42.0
    bond_mode balance-tlb
    bond_miimon 100
    bond_downdelay 200
    bond_updelay 200
    slaves eth0 eth1

Whatever you do, make sure you test everything on a non-production network first!

More information: Linux Kernel bonding documentation

Properly wiring a network closet

December 17, 2012August 31, 2015 Josh Reichardt 17 Comments

I am in the middle of a network wiring closet makeover at work right now and thought that this would be the perfect time to go over some of the things that I have learned along the way. I feel like now that I have a few of these closet rewiring jobs under my belt I am confident enough in my techniques and methods to the point where I feel comfortable going over them and showing viewers how a wiring closet should be built out and should look when everything is said and done.

The only thing I will be covering in this post is the wiring portion of this process. The networking closets have been built out, the grounding racks and wires have all been rigged up and the wire management racks have all been installed. The switch chassis and PSU’s have all been mounted. The switch interfaces and other behind the scenes networking tasks have already been configured and taken care of. The point I’m trying to get at here is that there were a lot of hours spent taking care of all these small items and a ton of work done in the background to get to this point. I think the most important lesson to take away from this project was making sure all of the small things were done properly and with a certain expectation of quality, otherwise all of the other effort that went into one process will be wasted.

I am going to walk you through the process I went through to rewire a networking closet in the remainder of this post. Luckily I was able to take pictures for many of the things I went through along the way. I thought it would be useful to show rather than tell for the most part to make following things a little bit easier for readers.

Proper implements

Ethernet cables – Pretty obvious but I just wanted to mention this one quickly anyway. I have only seen a handful of bad cables but you can never be sure so having some extra cable to swap out is an easy way to test if a cable is bad or not. Also, we use a color coating scheme to help keep things organized, you will see later what I’m talking about. Just make sure you size out your cables to the appropriate length before hand.

White cables – Wireless internet.
Red cables – Generic printers.
Green cables – Special purpose, whether it be static computers or specialized printers.
Yellow – UPS

Velcro – Probably your most important tool and cannot be understated in my opinion. Effective use of velcro is really what ties everything together and keeps things organized and clean. Not sure how a set of cables should go? Velcro. It is your best friend if you have OCD and are working on one of these projects.

Label maker – Another great tool to help keep things organized. By no means do you need to label every cable in a wiring closet but you should be sure to highlight some of you landmark cables, so to speak. I’ve found it works out pretty slick to flag any static computer with a label, special printers, core uplinks, a basic rule of thumb I came up with is that any special case where you have a port you may easily forget later on should get labeled.

Wire cutters – Just about every closet I’ve had to reconfigure so far required me to use these for one thing or another. I like to have them handy just in case I need them.

Multitool/Razor – Handy for cutting the ends off of boots, lopping off pieces of velcro or just about any other odd job you might encounter in your wiring closet project. Another one of those nice to haves before hand so you don’t waste time later.

Music – It can get mighty boring doing this type of work. I suggest turning up some of your favorite tunes if you can, it will help you to keep your sanity. Listening to music may also work by keeping you distracted just a little bit as well as helping the time pass by.

The Buildout

Here is what the wiring closet looked like initially. As you can see it was not exactly in great shape. Although this is not nearly as bad as what some closets look like that I have come across (including some others within the scope of this project even), but I still don’t like it. I came up with a game plan before hand, which turned out to be really useful.

Since there were only 2 VLAN’s in this closet, each on their own switch, and each VLAN correlated to one side of the patch panel, it made doing the cut that much easier. This method should be somewhat foolproof and could easily be applied in any situation as long as the ports on the switch and patch panel and their correlating VLAN’s are known ahead of time. Again, because this wiring closet was smaller than the others it made things less messy as well as less time consuming.

There isn’t much to this method really. The one caveat that really helped me in this closet is that we were not reusing the old patch cables. Not reusing the old cables allowed for two things. First, it made pulling out all of the old cable SO much easier and secondly, it enabled me to use some of the clipped ends as markers, so I knew where to plug all of the new cable runs into. I unplugged (or cut the ends) all the cables from one switch at a time, and traced the cable back to the port it was plugged into on the patch panel. Once I knew which port the cable was plugged into I would simply cut the end and leave it there, in place until I had traced the cable from end to end. By doing it this way I was able to keep the cables from each switch separate and using the boots gave me a quick and easy way to know where I was at. Here is an example of what I’m talking about.

My new switch was setup in a particular way. the first 4 slots on each card were designated as wireless ports (white cable), and the last 4 ports were designated to be printer ports and static ports (green, red, yellow). So that left me with the middle 16/12 ports for the cards, therefore the bulk of my bundles were sized to fill all the ports. With the few extra I just sized my bundle according to how many ports were left over. If that doesn’t make much sense, take a look below and you can kind of see what I’m talking about. I screwed up on the yellow cable there, it ended up being at the far right by the end of things.

There were a few things that I found to be useful when I was putting the new cables in. Probably the most important thing to be mindful of when patching in the new stuff is to count out your bundles correctly. I was dealing with some really long patch cable, so having to go back because I miscounted was a real pain in the ass. Here I am laying out my bundles because they are so *^#@ing lengthy, it would have been a nightmare trying to manage them without some sort of organization beforehand.

I found that it was much easier to handle these bundles if I had them all laid out and untangled ahead of time, just be sure to double count how many ports you need first! I will also say that it also helped me tremendously to have the ends tied up before running the bundles as well.

You can’t really appreciate the length of these cables are from these pictures but it was really painful and time consuming pulling some of these bastards. Again, having a game plan and rechecking things will be your best friend if you are rewiring a network closet. Here is how everything turned out when I was done buttoning up the closet and finishing up with some final touches.

Not bad. But I think just as importantly, at least for myself, all of the other parts of the closet need to look as good as the switch. I think the best way is to give you some other shots to show you what I mean.

running to the patch panel — Out of patch panel

wall to cable guide — Wall to cable guide

running to cable management — Running in to to cable management

It really isn’t that difficult to dress up a closet, and make everything look nice and neat. Time consuming? Maybe. But as you can tell from this rewiring job, this networking closet looks way better than it did before, I actually found a sense of pride and enjoyment from the work done on dressing up these wiring closets. It may not be a lot, but I am strong believer in the small things counting. Let me know if you have any questions or would have done any of this differently. I am always experimenting with my wiring technique and would love to improve on this method if I can.

Document Storage: Part 5

December 11, 2012August 31, 2015 James Cort 14 Comments

Document Storage Project

This is Part 5: Uploading Scanned Images.

There’s two components to this part: configuring somewhere for the files to be uploaded to and setting up your MFD to upload to them. Most modern MFDs will upload to a CIFS share, which is what we’re going to use here. First thing’s first, we need to install Samba:

apt-get install samba

Now we need to set up Samba. We’ll have user-level security (it’ll be much easier to lock things down if we want to increase security at a later date, and besides share-level security went out with the Ark) and a single share called incoming. We also need a user for the MFD to log into Samba with; we’ll call this user “scanner”. We’ll also have a group called “scanner” so we can be a little more flexible over who can access this share should we wish.

Edit /etc/samba/smb.conf as follows:

......

# "security = user" is always a good idea. This will require a Unix account
# in this server for every user accessing the server. See
# /usr/share/doc/samba-doc/htmldocs/Samba3-HOWTO/ServerType.html
# in the samba-doc package for details.
   security = user

......

[incoming]
        path = /home/incoming
        guest ok = no
        browseable = no
        read only = no
        valid users = @scanner

Now, we need a new user for the MFD. Samba requires that users also have corresponding Unix accounts, so first we create a Unix account, then we set their Samba password. We also need to ensure the permissions on /home/incoming are correct – the folllowing commands deal with this:

  useradd scanner
  smbpasswd scanner
  chgrp scanner /home/incoming
  chmod g+rwx /home/incoming

Make sure you choose a password that is not only secure, but possible to type in on your MFD! Check this works by connecting to the following folder in Windows:

\\(hostname)\incoming

You’ll need to use the username/password for the scanner user you set up.

For the final part of this, you need to set up your MFD to scan to this directory.

I’ve chosen an Oki MB451 multifunction unit for a number of reasons:

It’s cheap.
It has a double-sided document feeder for scanning. More and more documents are being sent double-sided; it seems like a step back to have a document feeder that can’t deal with this.
It supports scanning directly to email and CIFS share without requiring extra software on the PC. (This is important; certainly a few years ago a lot of manufacturers claimed their products could do this but it wasn’t apparent until after you’d taken it out of the box that their product didn’t do any of it without additional software on your PC. Certain large photocopier-type units still have this restriction, though sometimes you can buy an optional bolt-on to overcome it. I prefer avoiding the need for extra bolt-ons because they’re usually extortionately priced and often difficult to source).
It has a nice big display. These units can be a pig to set up at the best of times; a large display often goes some way to alleviate this problem.
You can set up lots of profiles – preconfigured shortcuts that say “everything scanned under this profile should be stored under this name in this share accessed with this username and password; files should have this format”. Unfortunately you can’t nail a profile to say “everything scanned under this profile is double-sided” but you can’t have everything!
The printer supports Postscript, which means it’ll be pretty much guaranteed to work under any OS I can throw at it for a long time to come.

I won’t go into detail regarding MFD configuration – there’s simply too many on the market and they all vary. It’s enough to explain that I’ve set up a profile called “Correspondence” and I’ve pointed it at \\(hostname)\incoming.

With the profile I’ve set up, scanned documents will be stored under \\(hostname)\incoming\Correspondence-#####.pdf.

Test this all works by scanning a document and making sure it appears in the /home/incoming directory on your Linux box.

There’s only one thing left to do – tie all this together so incoming documents are automatically OCR’d, made available via Apache and OCR’d so they’re indexable in Sphider….

Becoming a better sysadmin

December 7, 2012August 31, 2015 Josh Reichardt 1 Comment

I typically don’t focus on philosophical topics or the more abstract subjects, but recently I have been reading up on the topic of self improvement and wanted to take some time today to lay out and develop some of the key concepts and ideas that I have found to be helpful so far. Hopefully some of these ideas can be used to help you improve as well in the world of system administration and other future career endeavors.

So this post is going to be more of a work in progress than anything else, since I really just wanted to get some of this stuff written down in order to clear it out of my head. There are literally books that have been written on self improvement and learning strategies so my goal with this isn’t to get every single detail, I just want to hit the high points and how their application to system administration. Here’s what I have so far, feel free to let me know what I’m missing or throw in anything else that might be particularly useful on this subject.

Explicit vs Tacit knowledge

Explicit knowledge can be defined as that gained from books or listening to a lecture. Basically some form of reading or auditory resource.

Tacit knowledge can be defined as that gained from experience, action and practice.

I’d like to start off by making a distinction between different types of knowledge. I believe that the practice of system administration relies heavily on both types and just one type of experience is not enough to be great in this field. They work hand in hand. So for example, reading a ton of books, while useful in its own right will not be nearly as effective as reading books and then applying the knowledge gained from hands on experience. Likewise, if somebody never bothers to pickup a book and relies entirely on hands experiences they will not be as knowledgeable as someone who incorporates both types of knowledge. Although I do feel that much more can be learned from hands on experience in the field of system administration than by books alone.

Types of learning

There has been a good deal of research done on this subject but for the purposes of this post I would like to boil this all down to what are considered the three primary or main styles of learning. The reason I want to focus on these is that they seem to work hand in hand with explicit and tacit knowledge and can be described a bit more easily. Each one of these different styles represents a different sort of idiom to the learning experience. So here they are:

Visual – Learning by watching or reading.
Auditory – Learning by listening.
Kinesthetic – Learning from experience, hands on.

I would argue that employing a good variety of learning and study methods would be the most appropriate way to develop your skills as a sysadmin. But even in my own experiences with learning styles I have realized that I tend to favor a kinesthetic learning approach, and I’m sure others have their own preferences as well. Instead of saying that one is better than another, I would suggest employing all of these types. Take a look at yourself and figure out how you learn best and then decide which method(s) are the most and least helpful and then decide how to make these styles work to your advantage. For example, I feel that I am a weak reader. While I know that reading is important I tend to spend the least amount of time doing just reading if at all possible. Having a piece of reading material as a reference or as an introduction is great. If I don’t quite understand things from reading the next step I like to take is internalizing things by listening to or watching. Finally, once I get a good enough idea about a topic I like to quickly put things into my own experiences. There is some quote about how experience sticks but I am too lazy to look it up. Suffice it say, I tend to remember things much more concretely when I am able to experience them for myself.

Again, this is just in my own experience and everybody is different. I just wanted to give a specific example of one way to utilize different styles of learning. There are many other possibilities and this just happens to be the way I prefer to learn things.

Learning strategies

Now that we have that out of the way, I want to highlight some of the major tactics that I use when attempting to learn a new subject. I definitely use some of these more than others but the point is that you should attempt to utilize as much as you can for your own benefit. Here are some different strategies I came up with that help me greatly when I encounter new and difficult to understand information. Many of these work together or in tandem so they may described more than once.

The Feynman technique – This is as close to the end all be all that there is when it comes to learning. Everybody is probably familiar with this one, but I am guessing they are not familiar with the name. This technique is used to explain or go through a topic as if you were teaching it to somebody else that was just learning about it for the first time. This basically forces you to know what you’re talking about. If you get stuck when trying to explain a particular concept or idea, make a note of what you are struggling with and research and relearn the material until you can confidently explain it. You should be able to explain the subject simply, if your explanations are wordy or convoluted you probably don’t understand it as well as you think.

Reading – I usually like to get an introduction to a topic by reading up on (and bookmarking) what information I feel to be the most informed, whether it be official documentation, RFC’s, books, magazines, respected blogs and authors, etc. As I mentioned before, I would consider myself a weak reader (something that I definitely need to improve on!) so I also like to take very brief notes when something I read seems like it would useful so I can try it out for myself.

Watching/Listening to others – After getting a good idea from reading about a subject I always like to reinforce this by either watching demonstrations, videos, listening to podcasts, lectures or anything else that will show me how to get a better idea of how to do something. When I’m on a long drive for example is a great time to put on a podcast. It kills time as well as improves knowledge at the cost of nothing. Very efficient! The same with videos and demonstrations, the only thing holding you back is the motivation.

Try things for yourself – Sometimes this can be the most difficult approach but definitely can also be the most rewarding, there is nothing better than learning things the hard way. Try things out for yourself in a lab or anywhere that you can practice the concepts that you are attempting to learn and understand.

Take notes – This is important for your own understanding of how things work in a way that you can internalize. I will take notes on simple things like commands I won’t remember, related topics and concepts or even just jotting down keywords quickly that to Google for later on. This goes hand in hand with the reading technique described above, just jotting down very simple, brief notes can be really useful.

Communicate with others – There are plenty of resources out there for getting help and for communicating and discussing what you learn with others. I would suggest looking a /r/sysadmin as a starting point. IRC channels are another great place to ask questions and get help, there are channels for pretty much any subject you can think of out there. There are good sysadmin related channels at irc.freenode.net, if you don’t already utilize IRC I highly suggest you take a look.

Come back later – Give your brain some time to start digesting some of the information and to take a step back and put the pieces together to begin creating a bigger picture. I can’t count how many times I have been working on learning a new concept or subject and felt overwhelmed and stuck until I took a break, did something completely different or thought about something else entirely and came back to the subject later on with a fresh perspective. Sometimes these difficult subjects just take time to fully understand so taking breaks and clearing your head can be very useful.

Sleep on it – Have you ever heard of the term before? This may sound crazy but sometimes if there is a particular problem that I can’t solve I will often times think about it before I go to sleep. I find that by blocking out all outside interference and noise I can much more easily think about it, come up with fresh perspectives and ideas and often times will wake up with an answer the next morning. I think meditation is comparable to this but I know nothing about meditation (I hope to at some point!) so I have to use this method for the time being.

Break stuff – One of the best ways to incorporate a number of these techniques is to intentionally break stuff in your own setups. Triple check to be sure that you aren’t breaking anything important first and then go ahead and give it a try. By forcing yourself to fix things that are broken you develop a much deeper and more intimate relationship with the way things work, why they work the way that they do and how things get broken to begin with. The great thing about using this method is that it is almost always useful for something in the future, whether it be the troubleshooting skills, the Googling skills or the specific knowledge in the particular area that needed to be fixed.

Practice, practice, practice – The more I read about becoming better at something the more I am convinced that you have to practice like an absolute maniac. I think for system administration this can partially come from practical job experience but it also comes from dedicated study and lab time. The hands on component is where most of your practice will come from and becoming better doesn’t just happen, it takes cultivation and time, just like with any other skill. Stick with it and never stop learning and improving on your skills through practice and experience.