Eventual Consistency

Improving Your Python Productivity

I've been programming in python for a few years now, and I'm still often amazed by how clear and DRY well written Python code can be. There are many small tips and tricks I've learned over time, mostly by reading the code of popular open source projects, such as Django, Flask, Requests and others.

Here are a few things I've picked up that are sometimes overlooked, but can really help with everyday work.

1. Dictionary and Set comprehensions

Most Python developers know and use list comprehensions. For those of you unfamiliar with the concept, a list comprehension is a shorter, more concise way of creating lists.

>>> some_list = [1, 2, 3, 4, 5]
>>> another_list = [ x + 1 for x in some_list ]
>>> another_list
[2, 3, 4, 5, 6]

Since python 3.1 (and also back-ported to Python 2.7), we can now use the same idiom to create sets and dictionaries:

>>> # Set Comprehensions
>>> some_list = [1, 2, 3, 4, 5, 2, 5, 1, 4, 8]
>>> even_set = { x for x in some_list if x % 2 == 0 }
>>> even_set
set([8, 2, 4])

>>> # Dict Comprehensions
>>> d = { x: x % 2 == 0 for x in range(1, 11) }
>>> d
{1: False, 2: True, 3: False, 4: True, 5: False, 6: True, 7: False, 8: True, 9: False, 10: True}

In the first example we create a set of unique items from some_list, but only ones that are even numbers. The Dictionary example shows the creation of a dictionary, where the keys are integers from 1 to 10 (inclusive), and the values are booleans, indicating whether or not the key is an even number.

Another feature worth mentioning here is the availability of set literals. We can create a set simply by doing:

>>> my_set = {1, 2, 1, 2, 3, 4}
>>> my_set
set([1, 2, 3, 4])

Instead of using the set() built-in function.

2. Using a Counter object for, well, counting.

This is a pretty obvious one, but it's often forgotten. Counting stuff is a pretty common task for most developers, and while it isn't a very challenging thing to do (In most cases) - there are ways to make it even easier.

Python's collections library includes a subclass of the built-in dict, created specifically for this task:

>>> from collections import Counter
>>> c = Counter('hello world')
>>> c
Counter({'l': 3, 'o': 2, ' ': 1, 'e': 1, 'd': 1, 'h': 1, 'r': 1, 'w': 1})
>>> c.most_common(2)
[('l', 3), ('o', 2)]

3. Pretty printing JSON

JSON is a great serialization format, heavily used today by APIs and web-services. While it's somewhat human-readable, when working with the built-in json module, It's very hard to make sense of larger data structures with the naked eye, as they appear as one long, continuous line.

To make things a bit easier we can use the indent argument to "pretty print" our JSON data. this is especially useful when working interactively from the REPL or for logging:

>>> import json
>>> print(json.dumps(data))  # No indention
{"status": "OK", "count": 2, "results": [{"age": 27, "name": "Oz", "lactose_intolerant": true}, {"age": 29, "name": "Joe", "lactose_intolerant": false}]}
>>> print(json.dumps(data, indent=2))  # With indention
{
  "status": "OK",
  "count": 2,
  "results": [
    {
      "age": 27,
      "name": "Oz",
      "lactose_intolerant": true
    },
    {
      "age": 29,
      "name": "Joe",
      "lactose_intolerant": false
    }
  ]
}

Also, be sure to checkout the built-in pprint module for pretty-printing anything else.

4. Create a one-off, quick and dirty web service

Sometimes we need a very basic and quick way of doing some sort of RPC between services or machines. All we want is a simple way to call a method of program A by program B, possibly on another machine - for internal use.

While I wouldn't advocate this technique for creating anything other than internal one-off hacks, we can use a protocol called XML-RPC (with its corresponding Python libraries) for this sort of work.

Here is an example of a quick and dirty file-reading server, using the SimpleXMLRPCServer module:

from SimpleXMLRPCServer import SimpleXMLRPCServer

def file_reader(file_name):
    with open(file_name, 'r') as f:
        return f.read()

server = SimpleXMLRPCServer(('localhost', 8000))
server.register_introspection_functions()

server.register_function(file_reader)

server.serve_forever()

With the corresponding client:

import xmlrpclib
proxy = xmlrpclib.ServerProxy('http://localhost:8000/')
proxy.file_reader('/tmp/secret.txt')

And we now have some sort of a remote file reader with no external dependencies and only a little code (And, of course, no security of any kind, so try this only at home).

5. Python's awesome open source community has you covered

Well, I've only covered things available in Python's standard library, which, if you have Python installed, is already included with your distribution. for most other tasks, there are a ton of community maintained 3rd party libraries at your disposal.

Here is my checklist for choosing a great reliable open source library:

A Great open source library must...
  • Contain a clear license that suites your particular case.
  • Be Actively developed and maintained (Or, your willingness to step in and maintain it).
  • Be installable using pip for easy, repeatable deployment.
  • have a test suite with at least decent coverage.

If you do find a good library that fits your needs, don't be shy - most open source projects welcome contributions and assistance, even if you are not a Python veteran. Helping hands (and heads) are almost always welcome.


As always, I'd be happy to learn from your experience and of course discuss the tips I presented! Hit me up in the comments here below, or on Twitter at ozkatz100.

comments powered by Disqus

Hi, I'm Oz Katz

I am a co-founder and CTO over at Swayy.

I usually write about software development using Python, JavaScript and other awesome, open source tools.

Feel free to reach out on Twitter, or contact me using the links at the bottom of the page.