Backing up a Server to Amazon S3

Motivation

When deploy­ing a serv­er on the inter­net you always have to deal with secu­ri­ty issues. You hard­en your serv­er by set­ting up encrypt­ed con­nec­tions, con­fig­ur­ing a tight fire­wall and putting crit­i­cal ser­vices in a chroot jail. How­ev­er what hap­pens if an intrud­er hacks into your serv­er and deletes your con­tent? Or if you make a mis­take and erase some data? The last line of defense is hav­ing a good back­up strat­e­gy.

The ques­tion aris­es what do back­up and where to back it up. For me some­thing like to holy grail would be hav­ing a fast, reli­able, file-sys­tem-based back­up solu­tion like the snap­shot fea­ture in ZFS. It should then be pos­si­ble to sync these snap­shots in a band­width-effi­cient man­ner to a remote loca­tion.

Using Duplicity for Backup

There are a lot of dif­fer­ent solu­tions around, how­ev­er they dif­fer in secu­ri­ty, price and reli­a­bil­i­ty. Duplic­i­ty is one of them. It is rel­a­tive­ly easy to set up if you have a sim­ple back­up prob­lem, such as back­ing up the web serv­er and its cor­re­spond­ing data­base. The neat thing about Duplic­i­ty is that it can make encrypt­ed, incre­men­tal back­ups using stan­dard file for­mats. For the incre­men­tal part of this oper­a­tion it relies on rdiff to do the heavy lift­ing and it is using GPG to encrypt the back­up with a public/private key pair. One of the oth­er ben­e­fits is that Duplic­i­ty offers out of the box Ama­zon S3 sup­port. This means that you are able to store your back­ups up in the cloud in a save man­ner. By per­form­ing incre­men­tal back­ups only, the costs for traf­fic and stor­age are min­i­mized. In my case which is per­form­ing a dai­ly back­up of the con­fig­u­ra­tion and the blog of the serv­er I nev­er paid any­thing because there is a min­i­mum billing amount per month.

Using a file serv­er in the cloud as the back­up des­ti­na­tion has its ben­e­fits espe­cial­ly in the restore case. Then you can rely on Ama­zon’s band­width to per­form a fast restore instead of using your home cable or DSL con­nec­tion.

What is happening during a backup?

In the begin­ning, when there are no pre­vi­ous back­ups to be used for incre­men­tal back­ups, Duplic­i­ty is per­form­ing a full back­up. For its back­up files it uses stan­dard tar file for­mat which then gets encrypt­ed using GPG and your pub­lic key. It is then uploaded to a remote serv­er.

The next back­up is an incre­men­tal back­up. This means that duplic­i­ty now first checks if its local cache of pre­vi­ous diffs is up to date with the remote repos­i­to­ry. If that is not the case it down­loads all the pre­vi­ous diffs because it needs them to gen­er­ate the new diff of the most recent changes. It then cal­cu­lates the diff, encrypts it and uploads it togeth­er with a hash of the diff to the remote serv­er.

Backup a Database

When using a web appli­ca­tion often there are not only files, but also data­bas­es to be backed up to be able to ful­ly restore your web page from the back­up. In my case I am using a MySQL serv­er. The way I am doing it is first per­form­ing a mysql­dump and then back­ing this file up. I have writ­ten a small helper script in perl that is kicked off by a cron job.

I use for quite a while now and it is work­ing very nice­ly. As a start­ing point I pub­lished the back­up wrap­per script on github.

DTerm — A Terminal at Your Fingertips for Mac OS X

Some­times it would be great if one could just enter a com­mand in the ter­mi­nal which is relat­ed to the direc­to­ry of the file you are work­ing on: send it to a serv­er via scp, build a tar.gz archive, com­pile it or open anoth­er file. These are all oper­a­tions which have also a graph­i­cal “nice” way of exe­cut­ing it, but leav­ing your hands on the key­board can be so much faster.

The tool, which pro­vides a sys­tem-wide access to the ter­mi­nal is DTerm. It has been around for some time, as it has been released in 2008, but it was men­tioned recent­ly in the very nice mobileMac pod­cast (ger­man) I like a lot. The appli­ca­tion is free.

So what is it all about? With DTerm you define a hotkey (stan­dard is Com­mand-Shift-Return), which will over­lay a win­dow with a text field which accepts ter­mi­nal com­mands. They will be exe­cut­ed in the same direc­to­ry as your front­most appli­ca­tion. DTerm is use­ful if you for exam­ple would like to open anoth­er doc­u­ment in the same fold­er: just press DTer­m’s hotkey, enter open then the begin­ning of the file and press the tab­u­la­tor key to show a com­ple­tion list of all the files in the direc­to­ry. Choose the file and press enter and it is opened with its asso­ci­at­ed appli­ca­tion. Anoth­er use­ful short­cut is open . which opens the cur­rent direc­to­ry in the Find­er. If you are famil­iar with a ter­mi­nal on a *NIX based sys­tem now is the time to use all of your ter­mi­nal hacks every­where in your work­flow with­out touch­ing the Ter­mi­nal appli­ca­tion.

Screenshot

Links

DTerm home­page

New Highlighting Support in PasteBinIt

There is a new beta of Paste­BinIt avail­able with a nice fea­ture: high­light­ing sup­port. This means that you can click in the line num­ber gut­ter of the code view to set a mark­er that will show up in the paste­bin as a yel­low marked line

In addi­tion some UI tweaks have been made such as list­ing the 5 most recent syn­tax high­light­ings at the top of the menu.

Get the update here or use the inte­grat­ed updater if you have already down­loaded ver­sion 0.2–2.