BKaF – Brad Kovach and Friends

Every night, I make a backup of bradkovach.com and some other websites.  The script begins at 11:00 MST, gathers all the information that my website needs to survive, bundles it up, and sends it to a remote server.  By “remote” I mean, “in my Mom’s laundry room, connected to sub-standard broadband.”

The process takes about eight hours–due to the slow connection.

Also, the server’s hard drive fills up quickly, so it can barely hold a week’s worth of backups–not enough.  But still, having the backup has been beneficial.  Just a few days ago, I ruined a client’s website (while the client was there) with some trigger-happy deletions.  The backup saved my day.

But when I needed to restore the backup, it took HOURS.  I thought–there has to be a better way.  And there was…

Enter Amazon S3

Everyone who has the Internet is familiar with Amazon.com, the Wal-Mart of the online world.  They’ve taken shopping to a whole new level by making it affordable and easy.

Amazon has amazing infrastructure that keeps their site running.  The same infrastructure has been made available to end-users who want simple, solid, affordable storage.

At its core, S3 is a very simple file system.  S3’s structure includes buckets and objects.  Buckets are like folders.  Within a bucket, you can place objects.

Each S3 account allows 100 buckets.  Each bucket can hold unlimited objects.  Within buckets, you may not create folders, rename, move, copy or modify files.  Within buckets, you may store data–lots of it–and cheap, too!

Backup your *nix server with Amazon S3

If you have a *nix server that needs regular backups, S3 is a perfect solution.

The only downside?  It’s kinda tricky to maneuver.  But I’ve figured it out!  You can too.

There are a number of approaches.  Most involve programs, compiling, and administrative access to the machine they run on.  I don’t have administrative access to my webserver (thank goodness)–so this solution is shared-hosting friendly.  You only need shell access to make this work.

The Ingredients

1. s3-bash – a collection of scripts that can interface with Amazon S3 to get, put and delete objects from your buckets.
2. *nix-based system to back up
3. An Amazon S3 account.  Set one up at Amazon’s website.  (you need a credit card)
   1. Have your Access Key ID and Secret Access Key ready.  You can find them on this page.

Let’s Backup!

1. Create a bucket to store some objects
   1. Visit http://jets3t.s3.amazonaws.com/applets-jets3t-0.6.1/jets3t-cockpit.html
   2. Click the “Direct Login” tab
   3. Enter your Access Key ID and Secret Access Key
   4. Click Login
   5. Click the gear positioned at the top right of the left-hand pane
   6. Click “Create New Bucket…”
   7. Complete the wizard.  Make note of what you named the bucket.
2. Download and extract s3-bash to an empty folder on your *nix-based system:
   mkdir ~/s3/
   cd ~/s3/
   wget http://s3-bash.googlecode.com/files/s3-bash.0.02.tar.gz
   tar -xzvf s3-bash.0.02.tar.gz
3. Create a key file for your Secret Access Key.  The key goes between the quotes with NO ADDITIONAL CHARACTERS BEFORE OR AFTER THE QUOTES.
   echo -n "-SECRETACCESSKEYHERE-" > ~/s3/aws_key
4. Copy some files to your account:
   ~/s3/s3-put -S -k ACCESSKEYID -s ~/s3/aws_key -T FILEtoCOPY /BUCKETNAME/NAMEDOBJECT
   ACCESSKEYID is your Access Key.
   FILEtoCOPY is a file to copy.  In my circumstances it would be backup.tar.gz
   BUCKETNAME comes from the bucket in step 1-7
   NAMEDOBJECT is what the file will be named in S3.  Usually, keep it the same as FILEtoCOPY.  You can create pseudo-folders by using slashes.
5. Wait for the script to exit.  It shouldn’t take long if your server is on a high-bandwidth backbone.  On my setup, I can transfer 1 gigabyte in about 1 minute 30 seconds.

That should do it.  You can easily work this into scripts.  My script compiles a folder, tar/gzips it and then uploads it to Amazon S3 and then (for some odd reason), I still upload to my little server that could.

Restoring from Amazon S3

Using s3-bash, we can retrieve files from Amazon S3 as well.  Assuming you have followed the above steps to upload to Amazon S3, the following command will retrieve from S3.

~/s3/s3-get.sh -S -k ACCESSKEYID -s ~/s3/aws_key /BUCKET/NAMEDOBJECT > FILEtoSAVE

S3 Benefits

Fast - I can backup in MINUTES instead of HOURS.  Inversely, I can restore a backup in MINUTES, too.

Reliable - Amazon has multiple copies of all of my data–in multiple geographic locations.

Cheap - I estimate that I’ll be paying three dollars a month to maintain a month’s worth of backups.

---

---

No Responses to “Nerd Alert: Awesome *nix backups with Amazon S3”

Leave a Comment

Name (required)

Mail (will not be published) (required)

Website

Authenticate this comment using OpenID.