Category Archives: Backup

Recover a Grafana dashboard

Grafana uses Elasticsearch (optionally) to store its dashboards.  If you ever migrate your Graphite/Grafana servers or simply need to grab all of your dashboards from the old server then you will likely be looking for them in Elasticsearch.  Luckily, migrating to a new server and moving the dashboards is and uncomplicated and easy to do process.  In this post I will walk through the process of moving Grafana dashboards between servers.

This guide assumes that Elasticsearch has been installed on both old and new servers.  The first thing to look at is your current Grafana config.  This is the file that you probably used to set up your Grafana environment originally.  This file resides in the directory that you placed your Grafana server files in to, and is named config.js.  There is a block inside this config file that tells Elasticsearch where to save dashboards, which by default is called “grafana-dashboards” which should look something like this:

/**
 * Elasticsearch index for storing dashboards
 *
 */
 grafana_index: "grafana-dash-orig",

Now, if you still have access to the old server it is merely a matter of copying this Elasticsearch directory that houses your Grafana dashboard over to the new location. By default on an Ubuntu installation the Elasticsearch data files get placed in to the following path:

/var/lib/elasticsearch/elasticsearch/nodes/0/indices/grafana-dashboards

Replace “0″ with the node if this is a clustered Elasticsearch instance, otherwise you should see the grafana-dashboard directory.  Simply copy this directory over to the new server with rsync or scp and put it in a temporary location for the time being (like /tmp for example).  Rename the existing grafana-dashboards directory to something different, in case there are some newly created dashboards that you would like to retrieve.  Then move the original dashboards (from the old server) from the /tmp directory in to the above path, renaming it to grafana-dashboard.  The final step is to chown the directory and its contents.  The steps for accomplishing this task are similar to the following.

On the old host:

cd /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
rsync -avP -e ssh grafana-dashbaords/ user@remote_host:/tmp/

On the new host:

cd /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
mv grafana-dashboards grafana-dash-orig
mv /tmp/grafana-dashboards ./grafana-dashboards
chown -R elasticsearch:elasticsearch grafana-dashboards

You don’t even need to restart the webserver or Elasticsearch for the old dashboards to show up.  Just reload the page and bam.   Dashboards.

grafana dashboard

Cloud Backup Tutorial

I have been knee deep in backups for the past few weeks, but I think I can finally see light at the end of the tunnel.  What looked like a simple enough idea to implement turned out to be a much more complicated task to accomplish.  I don’t know why, but there seems to be practically no information at all out there covering this topic.  Maybe it’s just because backups suck?  Either way they are extremely important to the vitality of a company and without a workable set of data, you are screwed if something happens to your data.  So today I am going to write about managing cloud data and cloud backups and hopefully shine some light on this seemingly foreign topic.

Part of being a cloud based company means dealing with cloud based storage.  Some of the terms involved are slightly different than the standard backup and storage terminology.  Things like buckets, object based storage, S3, GCS, boto all come to mind when dealing with cloud based storage and backups.  It turns out that there are a handful of tools out there for dealing with our storage requirements which I will be discussing today.

The Google and Amazon API’s are nice because they allow for creating third party tools to manage the storage, outside of their official and standard tools.  In my journey to find a solution I ran across several, workable tools that I would like to mention.  The end goal of this project was to sync a massive amount of files and data from S3 storage to GCS.  I found that the following tools all provided at least some of my requirements and each has its own set of uses.  They are included here in no real order:

  • duplicity/duply – This tool works with S3 for small scale storage.
  • Rclone – This one looks very promising, supports S3 to GCS sync.
  • aws-cli – The official command line tool supported by AWS.

S3cmd – This was the first tool that came close to doing what I wanted.  It’s a really nice tool for smallish amounts of files and has some really nice and handy features and is capable of syncing S3 buckets.  It is equipped with a number of nice and handy options but unfortunately the way it is designed does not allow for reading and writing a large number of files.  It is a great tool for smaller sets of data.

s3s3mirror – This is an extremely fast copy tool written in Java and hosted on Github.  This thing is awesome at copying data quickly.  This tool was able to copy about 6 million files in a little over 5 hours the other day.  One extremely nice feature of this tool is that it has an intelligent sync built in so it knows which files have been copied over.  Even better, this tool is even faster when it is running reads only.  So once your initial sync has completed, additional syncs are blazing fast.

This is a jar file so you will need to have Java installed on your system to run it.

sudo apt-get install openjdk-jre-headless

Then you will need to grab the code from Github.

git clone git@github.com:cobbzilla/s3s3mirror.git

And to run it.

./s3s3mirror.sh first-bucket/ second-bucket/

That’s pretty much it.  There are some handy flags but this is the main command. There is an -r flag for changing the retry count, a -v flag for verbosity and troubleshooting as well as a –dry-run flag to see what will happen.

The only down side of this tool is that it only seems to be supported for S3 at this point – although the source is posted to Github so could easily be adapted to work for GCS, which is something I am actually looking at doing.

Gsutil – The Python command line tool that was created and developed by Google.  This is the most powerful tool that I have found so far.  It has a ton of command line options, the ability to communicate with other cloud providers, open source and is under active development and maintenance.  Gsutil is scriptable and has code for dealing with failures – it can retry failed copies as well as resumable transfers, and has intelligence for checking which files and directories already exist for scenarios where synchronizing buckets is important.

The first step to using gsutil after installation is to run through the configuration with the gsutil config command.  Follow the instructions to link gsutil with your account.  After the initial configuration has been run you can modify or update all the gsutil goodies by editing the config file – which lives in ~/.boto by default.  One config change that is worth mentioning is the parallel_process_count and parallel_thread_count.  These control how much data can get shoved through gsutil at once – so on really beefy boxes you can crank this number up quite a bit higher than its default.  To utilize the parallel processing you simply need to set the -m flag on your gsutil command.

gsutil -m sync/cp gs://bucket-name

One very nice feature of gsutil is that it has built in functionality to interact with AWS and S3 storage.  To enable  this functionality you need to copy your AWS access_id and your secret_access_key in to your ~/.boto config file.  After that, you can test out the updated config to look at your buckets that live on S3.

gsutil ls s3://

So your final command to sync an S3 bucket to Google Cloud would look similar to the following,

gsutil -m cp -R s3://bucket-name gs://bucket-name

Notice the -R flag, which sets the copy to be a recursive copy instead everything in one bucket to the other, instead of a single layer copy.

There is one final tool that I’d like to cover, which isn’t a command line tool but turns out to be incredibly useful for copying large sets of data from S3 in to GCS, which is the GCS Online Import tool.  Follow the link and go fill out the interest form listed and after a little while you should hear from somebody from Google about setting up and using your new account.  It is free to use and the support is very good. Once you have been approved for using this tool you will need to provide a little bit of information for setting up sync jobs, your AWS ID and key, as well as allowing your Google account to sync the data.  But it is all very straight forward and if you have any questions the support is excellent.  This tool saved me from having to manually sync my S3 storage to GCS manually, which would have taken at least 7 days (and that was even with a monster EC2 instance).

Ultimately, the tools you choose will depend on your specific requirements.  I ended up using a combination of s3s3mirror, AWS bucket versioning, the Google cloud import tool and gsutil.  But my requirements are probably different from the next person and each backup scenario is unique so a combination of these various tools allows for flexibility to accomplish pretty much all scenarios.  Let me know if you have any questions or know of some other tools that I have failed to mention here.  Cloud backups are an interesting and unique challenge that I am still mastering so I would love to hear any tips and tricks you may have.

Restore an Exchange Mailbox Database using Data Protector

Forgive the boring title for this post but I do think that this is a really important topic and one that I had to deal with recently at work.  Somehow one of our Exchange mailbox databases became corrupted and one of our users lost a ton of email, which, I’m almost 100% sure was related to the outage catastrophe we experienced 1.5 weeks ago.  This event made me thank the Flying Spaghetti Monster that I was getting good backups from our (sometimes shaky) backup solution, Data Protector.  Anyway, for this topic I will just assume that you are getting backups from whatever backup solution but it isn’t all that important because the majority of this post will cover specific instructions for the procedure within Exchange, so you can take bits and pieces and apply them where you need to.

Before I go any further, it is always worth mentioning;  make sure you are getting good backups! 

Ok, now that we have that out of the way I will show you the basic restore procedure within the Data Protector environment.  Select the Restore option from the drop down list -> MS Exchange 2010 Server.

Then select the source to backup up (Whichever database that needs to be restored).  With in Data Protector specify the restore options that you would like.

These are the options I used most recently.

  • Restore method: Restores files to temporary location
  • Backup version: Whichever data you decide you need to roll back to
  • Restore chain: Restore only this backup
  • Target client: Select the mailbox server that you want to restore to
  • Restore into location: This can be any location, just make sure there is enough disk space.
  • Select Restore databse file only

Once you have chosen your restore options, click the restore button to begin the restore procedure.

Once the database has been restored with Data Protector

Now for the fun stuff.  This is the part that I’m guessing most will probably be concerned with, but I didn’t want to leave out my Data Protector peeps.  Open up an Exchange Management Shell on the mailbox server that you restored your database to.  Technically it can be from any server as long as you connect to the correct mailbox server I guess.  Anyway, rename your restored database to something like “recoverydb.edb”.  Change directories into the restore folder, then check the status of the newly restored database with the following command:

eseutil /mh recoverydb.edb

You should see something similar to the following:

If it shows Clean Shutdown you can skip ahead.  Since we didn’t bring any log files down with us in this restore we will need to run the database hard repair on this database using the following command:

eseutil /p recoverydb.edb

After running the repair you should get a clean shutdown state if you check again (eseutil /mh).

Now we need to create a recovery database for Exchange to use in order to recover this data from.

New-MailboxDatabase -Recovery -Name “recoverydb” -Server Mailbox1 -EdbFilePath “M:\recovery\recoverydb.edb” -LogFolderPath “M:\recovery” -Verbose

It is important that when you create your recovery database it matches the renamed .edb file.  So since I renamed my recovery database to recoverydb.edb, I used recoverydb in the Powershell command.  If you want to check to make sure this step was done properly, use the following command to verify that the database is roughly the size you are expecting it to be:

Get-MailboxDatabase -status | select Servername,Name,DatabaseSize

After everything looks good we mount our database.

Mount-Database recoverydb

Just to verify that the database has stuff in it and we can find the person we’re looking for, we will take a quick look at the database contents, as shown below.

Get-MailboxStatistics -Database recoverydb

It looks like there are users there so all we need to do now is dump their emails into a temporary/recovery account in Exchange with the following command:

Restore-Mailbox -RecoveryMailbox “user_to_recover” -Identity “temporary_account” -RecoveryDatabase recoverydb -TargetFolder “RecoveredItems”

  • -RecoveryMailbox is the user mailbox that we are pulling data from, the source mailbox
  • -Identity is the user mailbox that we are putting data into, the destination mailbox
  • -RecoveryDatabase is our newly created recoverydb
  • -TargetFolder is the a folder that we will create on the target user to house the recovered items
  • -Verbose optional debugging information if there is a problem anywhere in the process

The wording and syntax of this command is a little bit tricky.  Just remember that the -RecoveryMailbox signifies the backup location and the -Identity signifies the restore location.  After this process completes (could take awhile depending on the mailbox size) you should be able to log in to the temporary account and take a look at the newly created “RecoveredItems” folder in which the mailbox contents of the user mailbox we are restoring have been copied in to.

Once this is done, just right click the target mailbox (temporary_account), click Manage Full Access Permission and give the restore mailbox (user_to_recover) full permissions through the Exchange Console so you can copy over messages, etc. in Outlook.

This step can be done any number of different ways but I chose this method because I was more concerned about the safest way to do this.  You could, for example, copy the contents directly into the user mailbox if you wanted.  Another option would be to export the contents of the temporary user out into a .pst file, with something like the following:

New-MailboxExportRequest –Mailbox mailboxserver –FilePath \recovery.pst

That should be it, after you are done and the emails have all been recovered , be sure to dismount the recovery database and delete the files to free space back up on your mailbox server.

Resources:

http://blogs.perficient.com/microsoft/2011/02/working-with-exchange-2010-recovery-databases-2/
http://www.mikepfeiffer.net/2011/07/restoring-mailbox-data-from-a-recovery-database
http://pmirmand.files.wordpress.com/2011/08/restore-database-on-exchange-2010.docx