Gather system details using BGInfo

I keep telling myself that I will write more blog posts but keep finding ways not to.  I keep getting more ideas to write about so I just need to kick myself into gear and get going on these.  The protip this February is a useful trick for getting quick and easy access to important server information using a tool written by Bryce Cogswell of the Sysinternals suite, called BGInfo.  This tool comes in handy when you begin to manage more than a handful of servers and need to keep your p’s and q’s straight.

So, to start things off I have made a quick guide for setting up a nice BGInfo background for Windows computers.

I found out that this script doesn’t update the background for users in Windows 7 unless you  explicitly tell it to write the background upon login.   So if you are interested, the color scheme  I have elected to use is R:29 G:95 B:122 (which happens to be the default Server 2008 background).

I have found it useful to gather a few extra   pieces of information through WMI as well as a few vb scripts to make my life as an administrator easier, plus these are kind of cool.  Adding to the basic information I have added free memory, number of processors, brand and  model.   I’m sure there are others but I haven’t had time to experiment with them yet.  Maybe you can come up with some suggestions?

Free Memory script:

winmgt = “winmgmts:{impersonationLevel=impersonate}!//”

Set oWMI_Qeury_Result = GetObject(winmgt).InstancesOf(“Win32_OperatingSystem”)

For Each oItem in oWMI_Qeury_Result
iFreeMemory   = oItem.FreePhysicalMemory
Next
iFreeMemory = Round(iFreeMemory/(1024))

Echo “” & iFreeMemory & ” MB”

Note: This will only check the amount of free memory when the script is run, either at logn or if the bginfotemplate is run manually.   It does not update itself otherwise.

Model:

SELECT Model FROM Win32_ComputerSystem

System Brand:

SELECT Manufacturer FROM Win32_Computer System

Processors:

SELECT NumberOfProcessors FROM Win32_ComputerSystem

We will need the following files for BGInfo to do its thing once we have adjusted our templates  to suit our needs.

To have the background populate when a   user logs in, we need to set up a group policy.  Call it BGInfo or something easy to remember.   Edit the policy to point at Users -> Windows Settings ->Scripts -> Logon

To create the script to run BGInfo when a user logs in, copy the following and create a file named bgscript.bat

%logonserver%\netlogon\bginfo\Bginfo.exe %logonserver%\netlogon\bginfo\servertemplate.bgi /Timer:0 /NoLicPrompt

I have applied this script to a user OU in active directory called ‘Admins’.   Members of this group are the only  set of users which this policy will apply to.   So for example, people that I have given   Admin rights  will all see this background when they log on.   Which, in my case is our sysadmin team.

That’s it!  Now we have a nice clean background on all of our servers (assuming we log on with admin priveliges) to quickly look up information that may be handy and to keep yourself from getting mixed up when working on multiple servers concurrently.

Resources:

http://jensolekragh.wordpress.com/2008/08/22/using-bginfo-exe-to-create-and-evaluate-wmi-queries/
http://www.zoutenbier.nl/ict-experience-kb/windows-servers/6-implement-bginfo-with-the-group-policys

Read More

What’s in Your Windows Toolboox?

I think the title explains what I will be talking about in this post pretty well. In my day to day work, as I have mentioned before, work primarily with Windows. I thought it would be a good idea to carve out a set of must have Windows administration tools, to have as a reference in the future. A good number of these tools are open source or freeware and some are people’s pet projects, so could become abandoned over time, that is why it will be good to come back and look at every now and then.  I would also suggest donating to the independent authors to help keep their efforts alive!

Since I will just be covering the essentials I don’t really feel a need to group or categorize them in any certain way. So let’s get Started.

Windirstat

This one is pretty handy for figuring out what is eating up all your disk space by organizing your drives visually.  But of course there are a number of really handy features, like organizing directories by largest size for a quick tree view of your disk, color coating based on file types.  This one kind of falls under the category, do one thing and do it well.  It also happens to be great for quickly analyzing disk and file sizes.


windirstat

RDTabs

I would like to shake the hand of the genius who created this piece of Windows goodness.  I honestly love this program.  It is an intuitive tool to help manage RDP sessions, which happens to work out very nicely since I am in Windows all day 😀  It has matured a great deal in its lifetime and offers things like tabbing (I hope that is obvious), favorite management, a handy dandy  built in  screenshot feature, detaching RDP sessions into separate windows, encrypted passwords, importing and exporting of favorites, a boatload of options for customization and many more I’m sure that I am forgetting.  Highly recommended.  You should seriously consider checking out this hidden gem.  I believe this one is freeware, so if you like you should hook the creator up!


BGInfo

What can I say, Mark Russinovich and Bryce Cogswell are kind of awesome.  This tool is really helpful for quickly looking up information and stats (I love stats) about the system you are working on.  Essentially it creates a custom bitmap image over top of your background desktop image based on the configuration information you feed it.  Fast, easy, clean.  This utility also gives you the ability to add custom queries to check for practically anything via WMI calls or registry entries. It also has command line options for scripting, so yeah.  Good stuff.  I can’t tell you how helpful something like this is if you have 5-10 remote connections open at a time to look at what server you are on quickly.


BGInfo configuration page

Wireshark

I don’t think I want to go into very much detail for this one at the risk of looking foolish, especially since I don’t use it that much and there is a vast amount of things that this program can do.  I mean, we’re talking about stuff like graphing TCP time/seq graphs or troubleshooting performance of certain types of network traffic, crazy stuff that I have no business looking at.  What I can say though, is that it has helped me a time or two when I have been otherwise clueless on network troubleshooting issues. It is a really powerful tool to have in your bag of tricks.


JavaRa

I just found this one today actually, which was sort of the inspiration for this blog post.  I don’t know about you but I absolutely hate dealing with Java, its updates, its previous versions, etc.  This tools is a quick and dirty way to purge old versions and update to the most current version.  That’s it.  And that is how it should be, I don’t know why Sun previously or Oracle now could have made a tool to do this a long time ago.  This one is all open source.


javara

mRemoteNG

I thought I would mention this tool as well.  Although it has fallen out of favor for me personally it was my go to remote administration tool when I had Mac’s, Linux and Windows to worry about.  This tool allows for administration through a number of remote protocols including SSH, VNC, RDP, ICA, telnet, etc.  So it really comes in handy for those admins that jump all over the board in terms of different platforms.  Completely open source, highly recommended.


mRemoteNG interface

OneNote

Now before you start to hate me for this one just hear me out.  I kind of felt the same way until I actually started using it.  I have searched a fair amount for a program that does what OneNote does and nothing comes even close.  To make my life and job easier, I love to take notes on things I do for projects for future reference.  In OneNote I can organize my thoughts and process easily.  As an example, we are in the middle of an Exchange migration and our setup will be fairly complex, so I have been keeping notes for everything I am doing.  This not only helps me to understand the process more clearly but gives me a reference if shit hits the fan later on as well.

Some nice features that OneNote provides for this type of note taking are things like the ability to copy in screenshots quickly for documenting my own steps via a built in snipping tool, pasting in website links instead of having to go to research Google later on down the road once I have already forgotten what I did  originally  saving time and energy, linking to network resources and scritps, exe’s, etc. from within OneNote.  This  program really is worth its weight in gold.  If you still turn your nose up at this product since it comes from Micro$oft you might check out Evernote, I have heard good things about it, though you won’t have nearly as much power with it.


Conclusion

I think this is just the tip of the iceburg.  As I get more comfortable in my current environment I’m sure I will continue to experiment many more tools for making my life as an admin easier.  I want to point out that this list only covers my favorite Windows tools for administration, as I know there are vastly more tools out there in both the Linux and Windows world.

What feedback do you have on these?  What sorts of tools do you like to make your life easier?  I would really like to hear your feedback.

Read More

Feeding your mail gateway a proper spam diet

In a previous post I described the process of how to get a Linux based mail filtering gateway set up on your network to check for viruses and do some basic filtering, eventually delivering messages to your Exchange server.

In this post I will expand on the various ways to “train” and customize your SpamAssassin mail filter to do more checks to weed out spam and generally lower the amount of junk that is making its way to your users’ inbox.

There are a number of things that aren’t enabled by default in SpamAssassin.  Obviously this isn’t as efficient as we would like, so there is a little bit of extra leg work getting everything set up the way it should be.

Tightening up Postfix:

This is the first step to improving the efficiency of your filtering process.  There are a number of checks that can be enabled in the configuration file (/etc/postfix/main.cf) here to fight the incoming spam.  I have appended these various checks to the end of my configuration posted previously to lower the amount of spam getting through by ensuring proper sending addresses, valid recipients, proper domains, etc.

smtpd_helo_required = yes
smtpd_sender_restrictions =
 reject_non_fqdn_sender,
 reject_unknown_sender_domain
smtpd_recipient_restrictions =
 permit_mynetworks,
 reject_unauth_destination,
 reject_invalid_hostname,
 reject_unknown_sender_domain,
 reject_unknown_recipient_domain,
 reject_non_fqdn_recipient

Configuring these properties will cause an immediate drop in the amount of spam that makes its way through the filter so the importance of getting this implemented cannot be overstated.

Make your spam filter happy, feed it spam:

The next technique that I will discuss took me FOREVER to figure out, so I hope that by sharing what I have learned I will help people save time in their own implementations.  It didn’t help that IMAP wasn’t enabled on our Exchange server, but I will save that story for another day.

Essentially you want to get a good chunk of SPAM and HAM emails messages to your mail filter for SpamAssassin to apply it Bayesian filtering techniques to learn how to classify incoming messages(statistical analysis stuff, I don’t know a lot about the specifics).

My first thought was to have users copy SPAM messages into a public folder on my Exchange server and pull the messages down directly to my mail gateway.  BUT that dream was shattered when I discovered that IMAP support for public folders had been dropped in the version of Exchange I am using (Exchange 2010).

So I dabbled with a few ideas that weren’t very graceful, the most notable of which was copying the Exchange public folder into Thunderbird then copying the mbox file from Thunderbird to the mail gateway, yuck.  I finally got some help from my friends over at ServerFault.  I basically had to install and configure fetchmail to go out and look for two specified mailboxes on my Exchange, one SPAM account (a spam collection account I created) and one HAM account (my personal inbox).

To install fetchmail issue the following command:

sudo aptitude install fetchmail

Next, we need to configure fetchmail to look at our specified IMAP acounts, so we need to edit the config file ~/.fetchmailrc

poll mail.domain.com protocol IMAP port 993:
auth password user "domain/spamacct" with password "password" ssl
auth password user "domain/hamacct" with password "password" ssl

Modify the permissions so that only the specified user can read/write the config file

chmod 600 .fetchmailrc

Finally you should be able to pull the emails onto your mail gateway by issuing the following command:

fetchmail -a -v -n -k --folder inbox

At this point the mail should be on your mail server in the directory /var/spool/mail/USER.  The final step is to feed the mail into the Bayesian filter provided by SpamAssassin.  To do this, issue the following command:

sa-learn --showdots --mbox --spam spam
sa-learn --showdots --mbox --ham ham

I had to fool around with the mail file names when I first copied them to the server to read as “spam” and “ham” but that should be easy enough to accomplish.

To check how the learning process is going we need to check the sa-learn database for the tokens, ham and spam it has received.  There are a few ways to check the database but the easiest I have found is to enter the following into the command line:

sa-learn --dump magic

This will output a number of results, the most important of which are the nham, nham and ntoken outputs.  Here is a sample from the initial training stages from my spam filter:

bruticus@bruticus:~$ sa-learn --dump magic
0.000          0          3          0  non-token data: bayes db version
0.000          0        341          0  non-token data: nspam
0.000          0        210          0  non-token data: nham
0.000          0      69078          0  non-token data: ntokens
0.000          0 1318421928          0  non-token data: oldest atime
0.000          0 1319205954          0  non-token data: newest atime
0.000          0 1319142287          0  non-token data: last journal sync atime
0.000          0 1319142287          0  non-token data: last expiry atime
0.000          0          0          0  non-token data: last expire atime delta
0.000          0          0          0  non-token data: last expire reduction count

Ideally you want the nham and nspam outputs around or above the 1000 message mark, but the filter can begin working with as little as 200 of each.

Also, I have read that the best way to train is to feed SpamAssassin the newest spam and ham messages that you have, so make sure to look for the newest messages to feed it.  I read that it has something to do with the Bayesian analysis.

NOTE:  Try to do the spam/ham learning step of the process in off hours or a slow time because it adds a tremendous amount of overhead to Postfix to process all the messages as well the machine itself taking up a large chunk of memory.

That’s it. The spam filter should be able to filter out even more messages now thanks to the bayesian filtering that we just enabled.

Final Step:

This one may or may not be overkill, I just implemented it yesterday and haven’t had a chance to get any feedback from it yet.  If you are in a multi-language  environment  this addition may not be feasible either.  With this step we are going to enable a SpamAssassin plugin to attempt to detect the email language and filter out everything that isn’t either English or Spanish.

To do this we need to enable the plugin so open up the SpamAssassin config, /etc/spamassassin/v310.pre and uncomment the following line,

loadplugin Mail::SpamAssassin::Plugin::TextCat

Then we need to edit the main SpamAssassin configuration file, /etc/spamassassin/local.cf to filter out all non English or Spanish languages, this line can be added anywhere, I chose to add it under the Bayesian filtering sections.

ok_languages en es
ok_locales en es

Conclusion:

That is pretty much it, at least for now. There are possibly a few other things to modify but I need to see how efficient the spam filter is at this point before I decide if I need to add any more layers.  I have a feeling that things are pretty good at this point and adding more filtering wouldn’t really add much value to the filter.

I am very satisfied with the results that I have attained with this project and hope to keep refining the process as I see fit.  Although, at some point I think I am just going to need to take a look at is and say “enough is enough”.  So, if you have any questions or ideas for improvement let me know, I would be glad to hear them.

Resources:

http://wiki.apache.org/spamassassin/SingleUserUnixInstall#Enable_IMAP_LearnAsSpam_folder
http://wiki.apache.org/spamassassin/RemoteImapFolder
http://www.byteplant.com/support/cleanmail/howtolearnexchange.html
http://faisal.com/docs/salearn
http://allaboutexchange.blogspot.com/2007/08/how-to-configure-spamassassin-bayesian.html
http://serverfault.com/questions/320594/bayesian-filtering-for-exchange-2010/320838#320838
http://www.howtoforge.com/debian_etch_fetchmail
http://spamassassinbook.packtpub.com/chapter9_preview.htm
http://www.linuxhomenetworking.com

Read More

Top to bottom troubleshooting: Part 2

In this post I will be going over the main methods that I use to remove infections from Windows based computers.  This technique is another best case, works on 9/10 machines I see type deal, so it covers the majority of the common infection issues users are likely to see.

I like to start at the lowest level I can when troubleshooting these types of issues and work my way up, using a similar approach to how I troubleshoot hardware problems.  That way it is much more difficult to miss things.  If you skip over issues, you almost always have inconveniences down the road.  That is especially true when there are rootkit infections present.

As much as we’d like to say that rootkits are going the way of the curb with the adoption of 64-bit operating systems, we are finding time and again that this simply is not true.  Malware creators are finding creative ways to bypass the 64-bit security mechanisms, making it important to check for their presence lest you get burnt later on.

Note: I almost always run these tools in safe mode first.  I know there are user level rootkits and the argument can be made to clean in normal mode but for me, cleaning in safemode gives a much better idea of what and where to look for things.

Infection Removal

Probably the best all around tool for infection removal out there, a handy piece of software called ComboFix by sUBs.  It is always the first thing I run and is the heavy hitter of my removal tools.  This tool has rootkit detection built into it, so it will alert you if it detects the presence of suspicious activity (this is actually the first thing to inspect for but we will get to it in a bit).  I don’t know how many times that this tool has saved me an excessive amount of time and effort.  It is pretty much automatic; download it, make sure it is up to date and then let it do its thing. It spits out a log file when it is finished giving clues to what exactly it was able to remove.

If ComboFix detects the presence of a rootkit and says it has successfully removed said rootkit, be sure to double check with another, different scanner to be sure.  CombFix is good at what it does but it is not designed to remove some of the more advanced rootkits so be sure to be thorough in your removal process.

Rootkit Inspection

There are a few really handy tools when dealing with rootkits, making your life easier when removing these bastards.  The first  is a nice little program called  TDSSKiller,  which can detect and remove a variety of different rootkits.  It runs a quick scan for a number of well known rootkits and attempts to remove them, afterwards producing a log file for later analysis.  If the program is unable to remove the rootkit(s) it will at least give you clues of where to look at with the more advanced tools mentioned in this section.

The one I always find myself coming back to when a machine is hosed is called  GMER.  This tool will give you a quick way to detect the presence of rootkits when you fire it up.  If it detects rootkit activity it will usually tell you and list the items in red within the Rootkil/Malware tab of the program. There are many advanced features and uses for this program which I may cover in other posts but are out scope for this topic.

Another solid rootkit scanner I have had success with is  RootRepeal.  If I am suspicious of rootkits after I have done a basic analysis with GMER I will usually run a full scan with RootRepeal, mainly because the scan doesn’t take nearly as long as the full scan in GMER.

Temp File Cleaning

Once we have cleaned out rootkits, it is a good idea to clean out temporary files.  Infections are commonly hidden inside of temp files and folders so be sure to check them just in case.   Another good reason to clean out temp files at this point is because it actually speeds up the process of malware scanning since these potentially malicious files and folders will have been cleaned already.  At the very least, temp file cleaning helps to improve the performance of the computer and free up space if there are an ungodly amount of temp files (believe me, I have seen some crazy %&#$).  I have had the best results with TFC by OldTimer for this type of thing.  It is fast and powerful.

Virus Cleaning

We’re almost done, so bear with me.  The final step in the process is to clean up all the loose ends left behind from the removal process.  Usually there are bad registry keys or trojan remnants or whatever else left over that still linger after the main cleaning process has been run.

At this stage there are a couple of handy tools.  The first is the  Malwarebytes free scanner.  This is another essential tool that I use on nearly every infection I work on.  If I believe a machine has be cleaned up enough at this point I will run a quick scan and if no malware traces are found I will call it a day.  If the quickscan reveals traces, I will remove them, reboot the machine and run a full scan to search for further traces.

Another good 2nd opinion scanner is Hitman Pro, a free cloud based scanner that does an excellent job of analyzing left over malware traces.

Once these scans are clean all we need to do is put a reliable anti-virus software on the computer and call it a day.  This however can be tricky.   Without starting a holy war here I have to say I have had luck recently with Microsoft Security Essentials (which is free up to 8 licenses or something).  I’ve heard good things about the full paid version of Malwarebytes for real time protection, as well as Kaspersky, NOD32 and Avira ant-virus products.  The reason I have been recommending MSE is due to its light installation, low background noise, freeness and its decent detection rates (flame me if you must).

As a side note, one thing I will say with full  certainty  though is that I absolutely abhor Symantec Anti-anything as well as McAfee-anything.  There are programs designed  specifically  to remove them because they are so bad.  They are expensive on memory and cpu overhead, take up tons of space and get in the way of everything a user does.

Conclusion

There are a ton of viruses and malware out there which are continually evolving and expanding into new areas.  Likewise, there are a ton of tools out there to combat the bad guys.  Some of these tools are better than others, but generally speaking the combination of tools I have outlined here will combat the majority of malicious code out there targeted towards the average user.  These tools have kept up with the malware writers and while they don’t offer a perfect solution they do a pretty good job.

So to reiterate, here is the general order of my cleaning process:

  • Combofix
    • TDSSKiller
    • GMER
    • RootRepeal
  • TFC
  • Malwarebytes
  • Hitman Pro
  • Microsoft Security Essentials (virus protection)

Read More

Top to bottom troubleshooting: Part 1

In this post I will be discussing the techniques that I have found to be the most tried and true methods for fixing broken Windows machines.  Granted there are a million and one ways that things can go wrong (as we all know) but using this approach I have found that it can cure 9/10 computers that make their way to me.  For the other 1/10, I haven’t really discovered a bulletproof technique for fixing as each of those issues is usually something entirely unique and may lead to future posts.

The first and absolute most important step in the troubleshooting process is ensuring that the hardware of a system is functioning properly.  The reason is simple;  often times a machine can display wacky symptoms due to bad hardware and it becomes a situation where you chasing to try to fix the symptom rather than the core problem.  I have seen it (and learned it painfully myself) too many times before, a piece of hardware is faulty and causes the computer to fail at different points, making it impossible to isolate problems effectively, causing many headaches in the troubleshooting process.

Physical Inspection

The easiest and often times most overlooked technique to fixing an issue quickly is to simply crack open the case and check for symptoms.  You would be surprised how filthy a neglectd case can become over time so vacuuming and spraying out the case with compressed air becomes just as important as any other step in the process, laptops included.

After cleaning out the case inspect for physical damage to internal components.  So many times I have seen leaky capacitors cause sporadic issues on otherwise perfectly running machines.  At this point you should also check to make sure the fans are running (especially for graphics cards), the CPU heat sink is clean, correct wiring to components, etc.

Power Supply

This step can be tricky, and is something that you just seem to get a feeling for over time (it certainly doesn’t hurt try at any stage but can sometimes be more work than is necessary).  I have been seeing fewer and fewer instances of bad power supplies recently so I don’t know if the manufacturing quality has gone up or if I have just had good luck.

There are power supply testers that can be purchased but I will usually just grab a known good power supply off the shelf and hook it up to test if the original power supply is bad or in the process failing.  Simple enough.  This is an issue that is pretty straight forward, either it works or it doesn’t work.  Green good.  If the power supply doesn’t work then you won’t be able to test anything else.

Hard Drive (doesn’t apply to SSD drives)

This is the stage where I see the majority of problems.  It is vital to ensure the hard drive is healthy and working properly.  The most battle tested and reliable tool in my bag for hardware troubleshooting is a tool called “Drive Fitness Test” from Hitachi.

Essentially this tool scans the hard drive for bad sectors as well as testing the features of S.M.A.R.T.  It is simple in function but so many times is overlooked in the process.  Another tool that for testing drives is “MHDD”.  This tool is VERY comprehensive in its analysis of the drive but  unfortunately  lacks good documentation (it was made by some mad scientist  Russian  dude) so there is somewhat of a learning curve to it.

Memory

Another important step to consider is testing out the memory modules of a misbehaving machines.  Although this is the least often cause of failures it is an important step the process because it may come back to bite you later, the same way a leaky cap or some other simple, overlooked step can be.  The go to tool for testing RAM is called “Memtest 86”, this is found on most Linux distributions these days so if you have an old disc laying around you are ready to rock.

Conclusion

These are some of the fundamentals that I have painstakingly learned the hard way over the past 5 years.  There are many, many other tools for testing out faulty pieces of hardware.  Even with so many options for testing tools, I keep coming back to these basics time and time again.  They have really become a foundation in my troubleshooting techniques and just seem to get the job done.

So to recap, here is the general order that things should be checked when testing for broken hardware:

  • Physical Inspection
  • Power Supply
  • Hard Drive
  • Memory

Next Up: Infection Removal

Read More