Rsync backups to Amazon S3

Having recently got married I wanted to make sure all the photos taken at the event are safely stored for the posterity. I thought I’d take the opportunity of making sure that all the rest of my photos are safely backed up, and that any new ones are also backed up without me needing to do anything.

One of the simplest places to keep your backups is Amazon S3. There is essentially an unlimited amount of space available, and it’s pretty cheap. rsync is a great tool to use when backing up because it only copies files, and parts of files that have changed so it will reduce the amount of data transferred to the lowest amount of possible. With S3 you not only pay for the data stored, but also for the data transferred so rsync is perfect. So, how do we use rsync to transfer data to S3?

I won’t go through setting up an Amazon S3 or creating a bucket, the Amazon documentation does that just fine.

Using Python Logging Effectively

If you’re writing a Python program that doesn’t have a text-based user interface (either it’s a GUI or runs as part of another program, e.g. a webserver) then you should avoid using the print statement. It’s tempting to use print to fill the console with information about what your program is up to. For code of any size though, this quickly devolves into a hard to navigate mess.

Python’s standard library contains a module, logging, that lets you write code to log as much information as you like and configure what you bits you are interested in at runtime.

There are two concepts that you need to understand with logging. Firstly there is the logging level. This is how you determine how important the message is. The levels range from debug as the least important, through info, warning, error, critical to exception, the most important. Secondly there is the logger. This allows you divide your messages into groups depending on the part of your code they relate to. For example, you might have a gui logger and a data logger.

Evolving The Blind Watchmaker

Recently I’ve been reading the classic book by Richard Dawkins, The Blind Watchmaker. In it he begins by discussing how evolution can produce complex systems from only a few simple rules. He demonstrates this using a simple tree drawing algorithm in which a few ‘genes’ control aspects such as the number of branches and the branch angle. The trees are evolved solely through mutation of an initial tree, rather combing the ‘genes’ of two trees to produce a child, and introducing mutations in those children.

In reality evolution is driven by pressures from the environment on the genes and those that produce the fittest host will survive. As this is early in the book though Dawkins uses himself as the environment and manually picks the most visually appealing trees.

Although the book is essentially timeless as although new evidence is continually being found in favour of evolution, the general thrust remains true. The passages where he talks about his computer, however, have dated horribly (which is not surprising given it was first published in 1986!). In this post I’ll describe how to recreate the section where he describes evolving trees in Python so you can create your own trees on your pc.

Simplifying Django dependencies with virtualenv

virtualenv is a tool for simplifying dependency management in Python applications. As the name suggests, virtualenv creates a virtual environment which makes it easy to install Python packages without needing root privileges to do so.

To use the packages installed in a virtual environment you run the activate script in the bin directory of the virtual environment. This is fine when you’re working on the command line, but you don’t want to have to remember this step when running the debug server, and it’s hard to get that to work when the site is deployed under mod_wsgi.

To make things easier you can add the appropriate directory from the virtual environment to Python’s path as part of manage.py, or the appropriate fcgi or wsgi control script.

import os
import sys
import siten
root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
site.addsitedir(os.path.join(root_dir, 've/lib/python%i.%i/site-packages'
                % (sys.version_info[0], sys.version_info[1])))

Just add the code above to the top of your manage.py file and the ve virtual environment will always be activated when you run the script.

Making History by Stephen Fry Book Review

This book poses an interesting question, what would happen if you could stop Adolf Hitler from being born? Ask anyone if they had the choice, would they take it and I imagine that almost everyone would say yes. When Michael, the lead character in this story has that choice he takes it with both hands. Unfortunately though, things do not go according to plan.

Stephen Fry is a British institution and a well known upper class intellectual. In this book he is clearly writing about what he knows as it focuses on a post-graduate student at Cambridge. The book is full of colour and detail and has an air of authenticity that draws you into the world effortlessly. There seems to be a significant amount of himself in the lead character and he writes as he talks. At first I found the overly intellectual mode of writing to be annoying and distracting from what the author was trying to say. Persevere though, and you’re rewarded with a charming, warm and funny story that will also make you think about history and the choices we make.

This book can easily be described a genre-defying as the time-travel premise means it clearly fits into science fiction. Don’t worry though, this book is not about spaceships or lasers as the two moments of time-travel are tiny compared to their consequences, and combined with flashbacks into the two world wars, the book fits into historical fiction as well. Is historical science fiction a genre? It is now.

This book was first published in 1996, and if you haven’t read in the last fifteen years then you should definitely go and hunt down copy. You’ll read it and imagine that it was written yesterday.