Month: October 2012

Restore an Exchange Mailbox Database using Data Protector

Josh Reichardt 13 Comments

Forgive the boring title for this post but I do think that this is a really important topic and one that I had to deal with recently at work.  Somehow one of our Exchange mailbox databases became corrupted and one of our users lost a ton of email, which, I’m almost 100% sure was related to the outage catastrophe we experienced 1.5 weeks ago.  This event made me thank the Flying Spaghetti Monster that I was getting good backups from our (sometimes shaky) backup solution, Data Protector.  Anyway, for this topic I will just assume that you are getting backups from whatever backup solution but it isn’t all that important because the majority of this post will cover specific instructions for the procedure within Exchange, so you can take bits and pieces and apply them where you need to.

Before I go any further, it is always worth mentioning;  make sure you are getting good backups! 

Ok, now that we have that out of the way I will show you the basic restore procedure within the Data Protector environment.  Select the Restore option from the drop down list -> MS Exchange 2010 Server.

Then select the source to backup up (Whichever database that needs to be restored).  With in Data Protector specify the restore options that you would like.

These are the options I used most recently.

  • Restore method: Restores files to temporary location
  • Backup version: Whichever data you decide you need to roll back to
  • Restore chain: Restore only this backup
  • Target client: Select the mailbox server that you want to restore to
  • Restore into location: This can be any location, just make sure there is enough disk space.
  • Select Restore databse file only

Once you have chosen your restore options, click the restore button to begin the restore procedure.

Once the database has been restored with Data Protector

Now for the fun stuff.  This is the part that I’m guessing most will probably be concerned with, but I didn’t want to leave out my Data Protector peeps.  Open up an Exchange Management Shell on the mailbox server that you restored your database to.  Technically it can be from any server as long as you connect to the correct mailbox server I guess.  Anyway, rename your restored database to something like “recoverydb.edb”.  Change directories into the restore folder, then check the status of the newly restored database with the following command:

eseutil /mh recoverydb.edb

You should see something similar to the following:

If it shows Clean Shutdown you can skip ahead.  Since we didn’t bring any log files down with us in this restore we will need to run the database hard repair on this database using the following command:

eseutil /p recoverydb.edb

After running the repair you should get a clean shutdown state if you check again (eseutil /mh).

Now we need to create a recovery database for Exchange to use in order to recover this data from.

New-MailboxDatabase -Recovery -Name “recoverydb” -Server Mailbox1 -EdbFilePath “M:\recovery\recoverydb.edb” -LogFolderPath “M:\recovery” -Verbose

It is important that when you create your recovery database it matches the renamed .edb file.  So since I renamed my recovery database to recoverydb.edb, I used recoverydb in the Powershell command.  If you want to check to make sure this step was done properly, use the following command to verify that the database is roughly the size you are expecting it to be:

Get-MailboxDatabase -status | select Servername,Name,DatabaseSize

After everything looks good we mount our database.

Mount-Database recoverydb

Just to verify that the database has stuff in it and we can find the person we’re looking for, we will take a quick look at the database contents, as shown below.

Get-MailboxStatistics -Database recoverydb

It looks like there are users there so all we need to do now is dump their emails into a temporary/recovery account in Exchange with the following command:

Restore-Mailbox -RecoveryMailbox “user_to_recover” -Identity “temporary_account” -RecoveryDatabase recoverydb -TargetFolder “RecoveredItems”

  • -RecoveryMailbox is the user mailbox that we are pulling data from, the source mailbox
  • -Identity is the user mailbox that we are putting data into, the destination mailbox
  • -RecoveryDatabase is our newly created recoverydb
  • -TargetFolder is the a folder that we will create on the target user to house the recovered items
  • -Verbose optional debugging information if there is a problem anywhere in the process

The wording and syntax of this command is a little bit tricky.  Just remember that the -RecoveryMailbox signifies the backup location and the -Identity signifies the restore location.  After this process completes (could take awhile depending on the mailbox size) you should be able to log in to the temporary account and take a look at the newly created “RecoveredItems” folder in which the mailbox contents of the user mailbox we are restoring have been copied in to.

Once this is done, just right click the target mailbox (temporary_account), click Manage Full Access Permission and give the restore mailbox (user_to_recover) full permissions through the Exchange Console so you can copy over messages, etc. in Outlook.

This step can be done any number of different ways but I chose this method because I was more concerned about the safest way to do this.  You could, for example, copy the contents directly into the user mailbox if you wanted.  Another option would be to export the contents of the temporary user out into a .pst file, with something like the following:

New-MailboxExportRequest –Mailbox mailboxserver –FilePath \recovery.pst

That should be it, after you are done and the emails have all been recovered , be sure to dismount the recovery database and delete the files to free space back up on your mailbox server.

Resources:

http://blogs.perficient.com/microsoft/2011/02/working-with-exchange-2010-recovery-databases-2/
http://www.mikepfeiffer.net/2011/07/restoring-mailbox-data-from-a-recovery-database
http://pmirmand.files.wordpress.com/2011/08/restore-database-on-exchange-2010.docx

Read More

Sending Test Emails with Telnet

Josh Reichardt 13 Comments

I’d like to talk quickly about a great and underutilized method for troubleshooting email flow problems.  Today I had to rebuild an Exchange Hub Transport server after a slight catastrophe from last week in which the VM the Hub lived on was completely unrecoverable.  That is another story but it brings up the need for using a great tool that is often skipped over, and that is sending test email via telnet.

The reason I say that this method is underutilized is because, well who uses telnet these days?  What’s great about using this is that you can test different aspects and essentially pinpoint where mail flow issues are occurring.  In my case I was have trouble relaying email from an internal account to outside mail servers.  So let’s jump into how to use this tool, its easy but I feel like not enough people know about it, so here we go.

First, since I was testing from inside, I need to connect to the local server name.

telnet hubserver.psa.local 25

Easy enough, we are using telnet to connect to the hub server, hubserver.psa.local on port 25 (SMTP).  Once we get in we run a simple,

ehlo

That gives us back a little bit of information, basically telling us that this is an email server and some of its capabilities.  Next, we will need to run through the following set of commands to send out the test email.  It is important that these commands are entered in exactly, with no backspaces, otherwise it will break the command and you will get an error message spit back out from your telnet session.

MAIL FROM: [email protected]
RCPT TO: [email protected]
DATA
SUBJECT:

message content.

.
QUIT
  • MAIL FROM: This is telling the mail server who this message is being sent from.
  • [email protected] is the internal mail sender I was using.
  • RCPT TO: Tells the mail server the email address that is being sent to.
  • [email protected] is the address we are sending to. It can be any of your internet based mail addresses (google, yahoo, etc.).
  • DATA signifies the start of the message body.
  • SUBJECT: This line is optional, probably a good idea to include a subject so the message doesn’t get blocked or sent to spam.  Hit enter twice after this to drop into the message content.
  • message content is whatever you want to include in your message.  Follow your message by hitting enter.
  • “.” (read dot) on a line by itself will tell the mail server to end the message and send it.  It is basically the equivalent of an escape character for emails.
  • QUIT leaves the telnet session from the mail server.

It is important that the previous set of commands is run the way that they look.  This whole string of commands should look something similar to the following inside of your shell when things are all said and done, assuming everything is working properly.

In my case, I was unable to enter an address for the RCPT TO: command.  To fix this, among with a few other steps in rebuilding the hub was to grant anonymous send permission on the Exchange side of things, then after that mail began flowing through the newly rebuilt Hub Transport server perfectly.

That should be it, I highly suggest going through the process of sending out a few test emails to get this method stuck in your brain for later on down the road if you ever have to do any mail flow type troubleshooting.  Good luck!

Read More

How to rescue your data on a failing hard drive with dd_rescue

Mike Erps 11 Comments

I just got done answering this question for someone at work:

What was that app that allowed you to duplicate the failing hard drive?

So I figured, why not put it out there for the Internets to see?

It’s the dd_rescue command within Knoppix. (I’ve always used 5.0 or 6.1 – I think dd_rescue is still included in version 7)

You need to boot to the Knoppix CD/DVD and run dd_rescue on the DISK, not the PARTITION.

Syntax:

dd_rescue /dev/sda /dev/sdb

Given that /dev/sda is your source drive, and /dev/sdb is the destination drive.

You can check your disk configuration like this:

fdisk -l

Read the output from there to see which disk is which. Your best bet is to use a sanitized drive as the destination so it’s easier to tell which disk has stuff on it and which doesn’t.

Since this is all command line, I HIGHLY recommend running a practice job using data that doesn’t matter before going forward with real data. This process can easily ruin good data if you have the command backwards.

Additional switches:

You can use these switches to “sandwich” a bad spot on the drive and pick up as much data as possible.

-s defines the start position.
Ex: (will start at 15G and go forwards)

dd_rescue –s 15G /dev/sda /dev/sdb

-r defines working from front to back
Ex: (will start at 15G and go in reverse, back to the beginning of the drive)
(-r does not use DMA buffer, which can increase error correcting abilities)

dd_rescue –r –s 15G /dev/sda /dev/sdb

Read More

Centralising logs for fun and profit

James Cort 14 Comments

It’s one of those things that usually gets pushed to the back burner because it seems like too much work for too little gain: setting up a central syslog server which all your other systems can report back to.

This is a shame, because there’s lots of benefits to having such a server:

  • You can analyse what’s going on in your network from a single, central location – saving you from having to log into a variety of devices for troubleshooting.
  • Improved security – if you have a security breach, the offender has to break into the logging server as well if they’re to cover their tracks properly. (I wouldn’t recommend re-purposing an existing server for precisely this reason – you want your syslog server to be as secure as possible, which means it needs to be running as few services as possible).
  • You only need to remember one set of tools to manage logs from a range of devices. Most routers will happily send logs back to a remote syslog server; there are also third-party products you can install on Windows.

It’s trivially easy to set this up in any reasonably modern Linux distribution. Once again, I’m going to use Debian for this example.

Out of the box, Debian uses rsyslog and stores the configuration file in /etc/rsyslog.conf. Fortunately, the default configuration only needs minor changes to two lines as shown in this excerpt:

# provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

Uncomment the lines beginning $ModLoad and $UDPServerRun by removing the # symbols:

# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

Restart rsyslogd (service rsyslog restart) and…. that’s it. Done.

Well, that’s not quite it. A remote syslog server isn’t much good unless you have equipment sending logs to it.  On any other Debian servers you may have, this is just a matter of adding a line to /etc/rsyslog.conf:

*.* @192.168.42.39:514

(substitute your own logging server’s IP address or hostname for 192.168.42.39).

Restart rsyslog on the server that will be sending logs to your remote syslog server. Now when you check your remote syslog server, you should find logs appearing from both itself and anything else that’s configured to send logs to it.

Advanced Tweaks

Once you’ve got this done, there’s all sorts of things you can add. You can separate logfiles according to the host that generated them, you can have new logfiles created every day with an appropriate filename… or you can just stick with the basic configuration which will put everything in the same set of log files and just use grep to separate the interesting information.

Whatever you do, keep a sharp eye on disk space on your logfile server. Logs can grow very large very quickly, and a syslog server with a full disk won’t log anything at all.

Read More

Quickly Add Users to Lync 2010 Environment

Josh Reichardt 16 Comments

Update (10/11):  new script posted

I had the chance to work on this script a little bit more and was able to add a few of the features and checks that I wanted originally such as the ability to check if a user is a member of Active Directory or not, if the user is already of member of the Lync environment and also a very simple logging mechanism to gather information on names that weren’t in AD, users already in Lync, etc. to make life easier if the script has been run but there were misspellings or whatever other anomalies in the CSV file that is read in.

The format for reading in users is exactly the same, so just look at the example CSV file posted at the bottom of the page to get an idea of how users should be entered and look in the file for getting read in properly.

The updated code is posted below.  I will be adding this onto my github account in the future as well so check out github if you aren’t already a member.

##======================================================================================
##Script:  EnableLyncUsers.ps1
##Name:    Josh Reichardt
##Email:   [email protected]
##Date:    10/9/12
##Purpose: Use this script to add users already in AD domain into Lync 2010 environment.
##Notes:   Reads in a CSV file with pre populated AD Display names.  Can be adjusted to
##	   work with alternate AD names (eg SIP address, UPN or AD log on name).
##======================================================================================

#Variables.
$File = "C:\Lync\test.csv"
$Log = New-Item -ItemType File -Path "C:\Lync\userlog.txt" -Force

#Import CSV File
$UserArray = Import-CSV -Path $File

#Check if user file is empty.
if ($UserArray -eq $null)
{
	 write-host "No Users Found in Input File"
	 exit 0
}

#Get total number of users in CSV file and begin proccessing.
$count = $UserArray | Measure-Object | Select-Object -expand count
Write-Host "Found " $count "Users to import."
Write-Host "Processing Users.....`n"
$index = 1

ForEach ($User in $UserArray) {

	Write-Host "Processing User " $index " of " $count
	$Fullname = $User.DisplayName
	$aduser = get-csaduser -Identity $Fullname

	#Check if user is in AD.  Log if they are NOT.
	if ($aduser -eq $null) {
		$notinad = $true
		Write-Host "User " $Fullname " is not in AD.  Double check spelling, etc." -Foregroundcolor Red
		Add-Content -Path $Log -Value "$($Fullname) is not in AD.  Double check spelling, etc."
	}

	else {
		$notinad = $false
	}

	#If user is in AD check if enabled in Lync and log if enabled.
	if ($aduser.Enabled) {
		Write-Host $User.DisplayName "is already enabled in Lync, skipping."  -Foregroundcolor Yellow
		Add-Content -Path $Log -Value "$($Fullname) is already enabled in Lync."
	}		

	#User not enabled.
	else {
		Write-Host "Adding user " $User.DisplayName -Foregroundcolor Green
		Enable-CsUser -Identity $User.DisplayName -Registrarpool "lyncpoolGMRC.gmrcnt.local" -SipAddressType Emailaddress

		#Check if last command failed.  If it does, log it.
		if(!$?) {
			Add-Content -Path $Log -Value "$($Fullname) not enabled.  $(Get-Date)$($error[0])"
			continue
		}

	}

	$index++	

}

Below I have posted the quick and dirty method to bulk import a list of users from a CSV file into your Lync environment using their first and last name using the Lync Management Shell (LMS).  I say quick and dirty because it lacks a way to tell if users are already in the Lync environment as well as not having the ability to cope with users that have misspelled or changed names in the CSV file.  But it works, for the most part in a jam.

I should mention that this can be modified to cope with different user views.  For example,

  • The user’s Active Directory display name (e.g., “John Doe”)
  • The user’s SIP address (e.g., “sip:[email protected]”)
  • The user’s User Principal Name (e.g., “[email protected]”)
  • The user’s domain name and logon name, in the format domain_name\logon_name (e.g., psa\jdoe)

And here is the original code to get the users imported into your Lync environment.  Again, very rough, check the updated version if you want more features.

#Variables
$File = "C:\Lync\lyncusers.csv"

#Import CSV File
$UserArray = Import-CSV -Path $File

if ($UserArray -eq $null)
{
write-host "No Users Found in Input File" -foregroundcolor red -backgroundcolor black
exit 0
}

ForEach ($User in $UserArray) {

#Check if user is in AD
$aduser = Get-CsAdUser -Identity $User.Name | Where-Object {$_.enabled -ne "true"}

#Enable user
if($aduser -ne $()) {
Enable-CsUser -Identity $User.DisplayName -Registrarpool "lyncpoolGMRC.gmrcnt.local" -SipAddressType Emailaddress
}
}

Write-Host "Users added successfully."

The CSV file will look similar to the following (in my case, DisplayName, or First and Last name):

I’ve found this to be helpful any time I need to add more than just a few users at a time into the current environment because the GUI is so cumbersome.

Read More