Eventual Consistency

Automatic Host Discovery with Fabric and Boto

Over at Swayy our server infrastructure is hosted on Amazon's AWS.

We use EC2 instances that can roughly be divided into 3 roles - DB, Web and Worker servers. We generally use Fabric to manage and deploy to these servers, which means we must maintain a roledef for each of our environments. A roledef, in Fabric, is a simple mapping between a role (say "db") and an array of hostnames.

It's a clever mechanism for deployment, but since it is essentially a directory - it also means that we need to update it when we add or tear down nodes. Even though this is not necessarily a bad idea, and it does follow Python's "Explicit is better than implicit" approach, dealing with a large number of nodes gets tedious.

While looking for more of a "convention over configuration" approach we came up with a little helper class of our own. It uses the Boto library (which is a Python wrapper around the AWS API) to dynamically fetch the list of EC2 servers and generate a roledef using a set of regular expressions. This is perhaps better explained with an example.

Here is what our roledefs look like now:

production_roledef = {
    "db": "swayy-prod-db\d+",
    "web": "swayy-prod-web\d+",
    "worker": "swayy-prod-worker\d+",
    "python": "swayy-prod-(web|worker)\d+",  # Will match all web and worker servers
    "all": "swayy-prod-.*"  # Will match ALL production servers

Our helper class takes this roledef as input and dynamically generates a "real" roledef usable by Fabric by querying the AWS API and retrieving the public DNS record for hosts whose logical name matches the regular expression.

This approach is definitely not for everyone, since it is somewhat implicit (and requires you to double check yourself when naming a new EC2 host). But for us it has proved rather useful since it means we don't have to change actual code (fabfile.py) whenever we add a new node.

The code is available in gist form here - feel free to improve and change it to your liking - as it is very basic at the moment.

As always, I'd love to keep the discussion going. Please leave a comment bellow,
Or Follow me on Twitter where I talk a lot about Backbone, Django, Ops and other web development related things.

comments powered by Disqus

Hi, I'm Oz Katz

I am a co-founder and CTO over at Swayy.

I usually write about software development using Python, JavaScript and other awesome, open source tools.

Feel free to reach out on Twitter, or contact me using the links at the bottom of the page.