We have all heard about DNS catastrophes. I just read about horror story on reddit the other day, where an Azure root DNS zone was accidentally deleted with no backup. I experienced a similar disaster a few years ago – a simple DNS change managed to knock out internal DNS for an entire domain, which contained hundreds of records. Reading the post hit close to home, uncovering some of my own past anxiety, so I began poking around for solutions. Immediately, I noticed that backing up DNS records is usually skipped over as part of the backup process. Folks just tend to never do it, for whatever reason.
I did discover, though, that backing up DNS is easy. So I decided to fix the problem.
I wrote a simple shell script that dumps out all Route53 zones for a given AWS account to a json file, and uploads the zones to an S3 bucket. The script is a handful lines, which is perfect because it doesn’t take much effort to potentially save your bacon.
If you don’t host DNS in AWS, the script can be modified to work for other DNS providers (assuming they have public API’s).
Here’s the script:
#!/usr/bin/env bash
set -e
# Dump route 53 zones to a text file and upload to S3.
BACKUP_DIR=/home/<user>/dns-backup
BACKUP_BUCKET=<bucket>
# Use full paths for cron
CLIPATH="/usr/local/bin"
# Dump all zones to a file and upload to s3
function backup_all_zones () {
local zones
# Enumerate all zones
zones=$($CLIPATH/aws route53 list-hosted-zones | jq -r '.HostedZones[].Id' | sed "s/\/hostedzone\///")
for zone in $zones; do
echo "Backing up zone $zone"
$CLIPATH/aws route53 list-resource-record-sets --hosted-zone-id $zone > $BACKUP_DIR/$zone.json
done
# Upload backups to s3
$CLIPATH/aws s3 cp $BACKUP_DIR s3://$BACKUP_BUCKET --recursive --sse
}
# Create backup directory if it doesn't exist
mkdir -p $BACKUP_DIR
# Backup up all the things
time backup_all_zones
Be sure to update the <user> and <bucket> in the script to match your own environment settings. Dumping the DNS records to json is nice because it allows for a more programmatic way of working with the data. This script can be run manually, but is much more useful if run automatically. Just add the script to a cronjob and schedule it to dump DNS periodically.
For this script to work, the aws cli and jq need to be installed. The installation is skipped in this post, but is trivial. Refer to the links for instructions.
The aws cli needs to be configured to use an API key with read access from Route53 and the ability to write to S3. Details are skipped for this step as well – be sure to consult the AWS documentation on setting up IAM permissions for help with setting up API keys. Another, simplified approach is to use a pre-existing key with admin credentials (not recommended).
If you have worked with Jenkins for any extended length of time you quickly realize that the Jenkins server configurations can become complicated. If the server ever breaks and you don’t have a good backup of all the configuration files, it can be extremely painful to recreate all of the jobs that you have configured. And most recently if you have started using the Jenkins workflow libraries, all of your custom scripts and coding will disappear if you don’t back it up.
Luckily, backing up your Jenkins job configurations is a fairly simple and straight forward process. Today I will cover one quick and dirty way to backup configs using a Jenkins job.
There are some AWS plugins that will backup your Jenkins configurations but I found that it was just as easy to write a little bit of bash to do the backup, especially since I wanted to backup to S3, which none of the plugins I looked at handle. In genereal, the plugins I looked at either felt a little bit too heavy for what I was trying to accomplish or didn’t offer the functionality I was looking for.
If you are still interested in using a plugin, here are a few to check out:
Keep reading if none of the above plugins look like a good fit.
The first step is to install the needed dependencies on your Jenkins server. For the backup method that I will be covering, the only tools that need to be installed are aws cli, tar and rsync. Tar and rsync should already be installed and to get the aws cli you can download and install it with pip, from the Jenkins server that has the configurations you want to back up.
pip install awscli
After the prerequisites have been installed, you will need to create your Jenkins job. Click New Item -> Freestyle and input a name for the new job.
Then you will need to configure the job.
The first step will be figuring out how often you want to run this backup. A simple strategy would be to backup once a day. The once per day strategy is illustrated below.
Note the ‘H’ above means to randomize when the job runs over the hour so that if other jobs were configured they would try to space out the load.
The next step is to backup the Jenkins files. The logic is all written in bash so if you are familiar it should be easy to follow along.
# Delete all files in the workspace
rm -rf *
# Create a directory for the job definitions
mkdir -p $BUILD_ID/jobs
# Copy global configuration files into the workspace
cp $JENKINS_HOME/*.xml $BUILD_ID/
# Copy keys and secrets into the workspace
cp $JENKINS_HOME/identity.key.enc $BUILD_ID/
cp $JENKINS_HOME/secret.key $BUILD_ID/
cp $JENKINS_HOME/secret.key.not-so-secret $BUILD_ID/
cp -r $JENKINS_HOME/secrets $BUILD_ID/
# Copy user configuration files into the workspace
cp -r $JENKINS_HOME/users $BUILD_ID/
# Copy custom Pipeline workflow libraries
cp -r $JENKINS_HOME/workflow-libs $BUILD_ID
# Copy job definitions into the workspace
rsync -am --include='config.xml' --include='*/' --prune-empty-dirs --exclude='*' $JENKINS_HOME/jobs/ $BUILD_ID/jobs/
# Create an archive from all copied files (since the S3 plugin cannot copy folders recursively)
tar czf jenkins-configuration.tar.gz $BUILD_ID/
# Remove the directory so only the tar.gz gets copied to S3
rm -rf $BUILD_ID
Note that I am not backing up the job history because the history isn’t important for my uses. If the history IS important, make sure to add a line to backup those locations. Likewise, feel free to modify and/or update anything else in the script if it suits your needs any better.
The last step is to copy the backup to another location. This is why we installed aws cli earlier. So here I am just uploading the tar file to an S3 bucket, which is versioned (look up how to configure bucket versioning if you’re not familiar).
export AWS_DEFAULT_REGION="xxx"
export AWS_ACCESS_KEY_ID="xxx"
export AWS_SECRET_ACCESS_KEY="xxx"
# Upload archive to S3
echo "Uploading archive to S3"
aws s3 cp jenkins-configuration.tar.gz s3://<bucket>/jenkins-backup/
# Remove tar.gz after it gets uploaded to S3
rm -rf jenkins-configuration.tar.gz
Replace the AWS_DEFAULT_REGION with the region where the bucket lives (typically us-east-1), make sure to update the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use an account with access to write to AWS S3 (not covered here). The final thing to note, <bucket> should be replaced to use your own bucket.
The backup process itself is usually pretty fast unless the Jenkins server has a massive amount of jobs and configurations. Once you have configured the job, feel free to run it once to test if it works. If the job worked and returns as completed, go check your S3 bucket and make sure the tar.gz file was uploaded. If you are using versioning there should just be one file, and if you choose the “show versions” option you will see something similar to the following.
If everything went okay with your backup and upload to s3 you are done. Common issues configuring this backup method are choosing the correct AWS bucket, region and credentials. Also, double check where all of your Jenkins configurations live in case there aren’t in a standard location.
Grafana uses Elasticsearch (optionally) to store its dashboards. If you ever migrate your Graphite/Grafana servers or simply need to grab all of your dashboards from the old server then you will likely be looking for them in Elasticsearch. Luckily, migrating to a new server and moving the dashboards is and uncomplicated and easy to do process. In this post I will walk through the process of moving Grafana dashboards between servers.
This guide assumes that Elasticsearch has been installed on both old and new servers. The first thing to look at is your current Grafana config. This is the file that you probably used to set up your Grafana environment originally. This file resides in the directory that you placed your Grafana server files in to, and is named config.js. There is a block inside this config file that tells Elasticsearch where to save dashboards, which by default is called “grafana-dashboards” which should look something like this:
/**
* Elasticsearch index for storing dashboards
*
*/
grafana_index: "grafana-dash-orig",
Now, if you still have access to the old server it is merely a matter of copying this Elasticsearch directory that houses your Grafana dashboard over to the new location. By default on an Ubuntu installation the Elasticsearch data files get placed in to the following path:
Replace “0” with the node if this is a clustered Elasticsearch instance, otherwise you should see the grafana-dashboard directory. Simply copy this directory over to the new server with rsync or scp and put it in a temporary location for the time being (like /tmp for example). Rename the existing grafana-dashboards directory to something different, in case there are some newly created dashboards that you would like to retrieve. Then move the original dashboards (from the old server) from the /tmp directory in to the above path, renaming it to grafana-dashboard. The final step is to chown the directory and its contents. The steps for accomplishing this task are similar to the following.
On the old host:
cd /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
rsync -avP -e ssh grafana-dashbaords/ user@remote_host:/tmp/
I have been knee deep in backups for the past few weeks, but I think I can finally see light at the end of the tunnel. What looked like a simple enough idea to implement turned out to be a much more complicated task to accomplish. I don’t know why, but there seems to be practically no information at all out there covering this topic. Maybe it’s just because backups suck? Either way they are extremely important to the vitality of a company and without a workable set of data, you are screwed if something happens to your data. So today I am going to write about managing cloud data and cloud backups and hopefully shine some light on this seemingly foreign topic.
Part of being a cloud based company means dealing with cloud based storage. Some of the terms involved are slightly different than the standard backup and storage terminology. Things like buckets, object based storage, S3, GCS, boto all come to mind when dealing with cloud based storage and backups. It turns out that there are a handful of tools out there for dealing with our storage requirements which I will be discussing today.
The Google and Amazon API’s are nice because they allow for creating third party tools to manage the storage, outside of their official and standard tools. In my journey to find a solution I ran across several, workable tools that I would like to mention. The end goal of this project was to sync a massive amount of files and data from S3 storage to GCS. I found that the following tools all provided at least some of my requirements and each has its own set of uses. They are included here in no real order:
duplicity/duply – This tool works with S3 for small scale storage.
Rclone – This one looks very promising, supports S3 to GCS sync.
aws-cli – The official command line tool supported by AWS.
S3cmd – This was the first tool that came close to doing what I wanted. It’s a really nice tool for smallish amounts of files and has some really nice and handy features and is capable of syncing S3 buckets. It is equipped with a number of nice and handy options but unfortunately the way it is designed does not allow for reading and writing a large number of files. It is a great tool for smaller sets of data.
s3s3mirror – This is an extremely fast copy tool written in Java and hosted on Github. This thing is awesome at copying data quickly. This tool was able to copy about 6 million files in a little over 5 hours the other day. One extremely nice feature of this tool is that it has an intelligent sync built in so it knows which files have been copied over. Even better, this tool is even faster when it is running reads only. So once your initial sync has completed, additional syncs are blazing fast.
This is a jar file so you will need to have Java installed on your system to run it.
That’s pretty much it. There are some handy flags but this is the main command. There is an -r flag for changing the retry count, a -v flag for verbosity and troubleshooting as well as a –dry-run flag to see what will happen.
The only down side of this tool is that it only seems to be supported for S3 at this point – although the source is posted to Github so could easily be adapted to work for GCS, which is something I am actually looking at doing.
Gsutil – The Python command line tool that was created and developed by Google. This is the most powerful tool that I have found so far. It has a ton of command line options, the ability to communicate with other cloud providers, open source and is under active development and maintenance. Gsutil is scriptable and has code for dealing with failures – it can retry failed copies as well as resumable transfers, and has intelligence for checking which files and directories already exist for scenarios where synchronizing buckets is important.
The first step to using gsutil after installation is to run through the configuration with the gsutil config command. Follow the instructions to link gsutil with your account. After the initial configuration has been run you can modify or update all the gsutil goodies by editing the config file – which lives in ~/.boto by default. One config change that is worth mentioning is the parallel_process_count and parallel_thread_count. These control how much data can get shoved through gsutil at once – so on really beefy boxes you can crank this number up quite a bit higher than its default. To utilize the parallel processing you simply need to set the -m flag on your gsutil command.
gsutil -m sync/cp gs://bucket-name
One very nice feature of gsutil is that it has built in functionality to interact with AWS and S3 storage. To enable this functionality you need to copy your AWS access_id and your secret_access_key in to your ~/.boto config file. After that, you can test out the updated config to look at your buckets that live on S3.
gsutil ls s3://
So your final command to sync an S3 bucket to Google Cloud would look similar to the following,
gsutil -m cp -R s3://bucket-name gs://bucket-name
Notice the -R flag, which sets the copy to be a recursive copy instead everything in one bucket to the other, instead of a single layer copy.
There is one final tool that I’d like to cover, which isn’t a command line tool but turns out to be incredibly useful for copying large sets of data from S3 in to GCS, which is the GCS Online Import tool. Follow the link and go fill out the interest form listed and after a little while you should hear from somebody from Google about setting up and using your new account. It is free to use and the support is very good. Once you have been approved for using this tool you will need to provide a little bit of information for setting up sync jobs, your AWS ID and key, as well as allowing your Google account to sync the data. But it is all very straight forward and if you have any questions the support is excellent. This tool saved me from having to manually sync my S3 storage to GCS manually, which would have taken at least 7 days (and that was even with a monster EC2 instance).
Ultimately, the tools you choose will depend on your specific requirements. I ended up using a combination of s3s3mirror, AWS bucket versioning, the Google cloud import tool and gsutil. But my requirements are probably different from the next person and each backup scenario is unique so a combination of these various tools allows for flexibility to accomplish pretty much all scenarios. Let me know if you have any questions or know of some other tools that I have failed to mention here. Cloud backups are an interesting and unique challenge that I am still mastering so I would love to hear any tips and tricks you may have.
Forgive the boring title for this post but I do think that this is a really important topic and one that I had to deal with recently at work. Somehow one of our Exchange mailbox databases became corrupted and one of our users lost a ton of email, which, I’m almost 100% sure was related to the outage catastrophe we experienced 1.5 weeks ago. This event made me thank the Flying Spaghetti Monster that I was getting good backups from our (sometimes shaky) backup solution, Data Protector. Anyway, for this topic I will just assume that you are getting backups from whatever backup solution but it isn’t all that important because the majority of this post will cover specific instructions for the procedure within Exchange, so you can take bits and pieces and apply them where you need to.
Before I go any further, it is always worth mentioning; make sure you are getting good backups!
Ok, now that we have that out of the way I will show you the basic restore procedure within the Data Protector environment. Select the Restore option from the drop down list -> MS Exchange 2010 Server.
Then select the source to backup up (Whichever database that needs to be restored). With in Data Protector specify the restore options that you would like.
These are the options I used most recently.
Restore method: Restores files to temporary location
Backup version: Whichever data you decide you need to roll back to
Restore chain: Restore only this backup
Target client: Select the mailbox server that you want to restore to
Restore into location: This can be any location, just make sure there is enough disk space.
Select Restore databse file only
Once you have chosen your restore options, click the restore button to begin the restore procedure.
Once the database has been restored with Data Protector
Now for the fun stuff. This is the part that I’m guessing most will probably be concerned with, but I didn’t want to leave out my Data Protector peeps. Open up an Exchange Management Shell on the mailbox server that you restored your database to. Technically it can be from any server as long as you connect to the correct mailbox server I guess. Anyway, rename your restored database to something like “recoverydb.edb”. Change directories into the restore folder, then check the status of the newly restored database with the following command:
eseutil /mh recoverydb.edb
You should see something similar to the following:
If it shows Clean Shutdown you can skip ahead. Since we didn’t bring any log files down with us in this restore we will need to run the database hard repair on this database using the following command:
eseutil /p recoverydb.edb
After running the repair you should get a clean shutdown state if you check again (eseutil /mh).
Now we need to create a recovery database for Exchange to use in order to recover this data from.
It is important that when you create your recovery database it matches the renamed .edb file. So since I renamed my recovery database to recoverydb.edb, I used recoverydb in the Powershell command. If you want to check to make sure this step was done properly, use the following command to verify that the database is roughly the size you are expecting it to be:
After everything looks good we mount our database.
Mount-Database recoverydb
Just to verify that the database has stuff in it and we can find the person we’re looking for, we will take a quick look at the database contents, as shown below.
Get-MailboxStatistics -Database recoverydb
It looks like there are users there so all we need to do now is dump their emails into a temporary/recovery account in Exchange with the following command:
-RecoveryMailbox is the user mailbox that we are pulling data from, the source mailbox
-Identity is the user mailbox that we are putting data into, the destination mailbox
-RecoveryDatabase is our newly created recoverydb
-TargetFolder is the a folder that we will create on the target user to house the recovered items
-Verbose optional debugging information if there is a problem anywhere in the process
The wording and syntax of this command is a little bit tricky. Just remember that the -RecoveryMailbox signifies the backup location and the -Identity signifies the restore location. After this process completes (could take awhile depending on the mailbox size) you should be able to log in to the temporary account and take a look at the newly created “RecoveredItems” folder in which the mailbox contents of the user mailbox we are restoring have been copied in to.
Once this is done, just right click the target mailbox (temporary_account), click Manage Full Access Permission and give the restore mailbox (user_to_recover) full permissions through the Exchange Console so you can copy over messages, etc. in Outlook.
This step can be done any number of different ways but I chose this method because I was more concerned about the safest way to do this. You could, for example, copy the contents directly into the user mailbox if you wanted. Another option would be to export the contents of the temporary user out into a .pst file, with something like the following:
That should be it, after you are done and the emails have all been recovered , be sure to dismount the recovery database and delete the files to free space back up on your mailbox server.
