Python Decorators: Syntactic Sugar

Python decorators are syntactic sugar—you’ll hear that a lot. Decorators are an extremely powerful tool that, at first, don’t seem to offer any real use. Take, for example:

def decorator_test(function):
  print function.__name__

def foobar():
  print "Hello, world!"

decorator_test(foobar)

That will output foobar—quite clear, and a simple example. To convert this into decorator syntax:

def decorator_test(function):
  print function.__name__

@decorator_test
def foobar():
  print "Hello, world!"

Again, you’ll see the output foobar. So what happened? You’ll notice the @decorator_test line above the function definition of foobar(). This is the syntax for applying a decorator to a function. Comparing our two examples, you’ll see applying a decorator to an arbitrary function and then calling function, is the equivalent of calling decorator(function).

Why bother doing this at all? There is plenty of real-world examples for decorators, and I would consider it to be “modern” Python—it applies well to object-oriented programming, and properly, in a Pythonic way, exposes Python’s functions as first-class objects. Here’s a more practical application:

class safe:
  def __init__(self, function):
    self.function = function

  def __call__(self, *args):
    try:
      return self.function(*args)
    except Exception, e:
      print "Error: %s" % (e)

There’s some major changes over the last example function. Firstly, I’ve encapsulated the functionality into the class; notice how it doesn’t affect the decorator. Rather than decorating a function with another function, I’d do it with a class here. To make things work, I’m using the class’ __call__ method, which is going to be how I pass the functionality to my target function. I also need to __init__ my class so that I can take the target function as a first-class object into my class. The functionality is very simple: I’m going to receive a function (self.function, created at __init__), and test it’s execution safely. I use *args to receive all arguments from the target function so that functionality is preserved and completely generalized. Here’s a sample on how to use this:

@safe
def unsafe(x):
  return 1 / x

print "unsafe(1): ", unsafe(1)
print "unsafe(0): ", unsafe(0)

This outputs:

unsafe(1):  1
unsafe(0):  Error: integer division or modulo by zero

Python doesn’t like when you divide by zero1, and so safe catches that and cleanly lets us know without killing the application.

This class can be used almost as a template for handling a large proportion of decorator functions; the combination of __init__ and __call__ is a lot more powerful and Pythonic—at least in my opinion—than declaring a wrapper function with another one inside it to achieve the same functionality.

Outside of Django, I haven’t really used decorators a whole lot, but spending a lot of time on Project Euler meant I needed to speed up a lot of my recursive algorithms. Decorators really came to the rescue in the form of memoization.

Let’s take a very simple Fibonacci number generator:

def fibonacci(n):
  if n in (0, 1): return n
  return fibonacci(n - 1) + fibonacci(n - 2)

It’s clear this is a very inefficient algorithm: the amount of function calls increases exponentially for increasing values of n—this is because the function calls values that it has already calculated again and again. The easy way to optimize this would be to cache the values in a dictionary and check to see if that value of n has been called previously. If it has, return it’s value in the dictionary, if not, proceed to call the function. This is memoization. Let’s look at our memoize class:

class memoize:
  def __init__(self, function):
    self.function = function
    self.memoized = {}

  def __call__(self, *args):
    try:
      return self.memoized[args]
    except KeyError:
      self.memoized[args] = self.function(*args)
      return self.memoized[args]

This is very similar to the safe class structurally. There is now a dictionary, self.memoized, that acts as our cache, and a change in the exception handling that looks for KeyError, which throws an error if a key doesn’t exist in a dictionary. Again, this class is generalized, and will work for any recursive function that could benefit from memoization.

Let’s run a few comparisons. First, the setup:

def fibonacci(n):
  if n in (0, 1): return n
  return fibonacci(n - 1) + fibonacci(n - 2)

@memoize
def fibonacci_memoized(n):
  if n in (0, 1): return n
  return fibonacci_memoized(n - 1) + fibonacci_memoized(n - 2)

Notice how fibonacci_memoized is extremely clean—it’s the exact same function. We don’t have any extraneous cache = {} calls outside the function, and there is nothing in the algorithm that detracts from the natural flow of the process. That is what I think is the biggest benefit of decorators: it abstracts away functionality that isn’t relevant to the core of the function.

Using a simple home-brewed timer function:

Beginning trial for fibonacci_memoized(30).
fibonacci_memoized(30) = 832040 in 0.000516s.

Beginning trial for fibonacci(30).
fibonacci(30) = 832040 in 1.147118s.

The memoized function is over 2223 times faster. Even better, in this case, it scales very well.

Beginning trial for fibonacci_memoized(40).
fibonacci_memoized(40) = 102334155 in 0.000699s.

Beginning trial for fibonacci(40).
fibonacci(40) = 102334155 in 145.366141s.

The memoized function went up about 35% (an increase of 0.000183s) whereas the vanilla version went up almost 126% (an increase of 144.219023s). While the percentage values might not show a great deal of improvement, take a look at the actual values: this is effective. In fact, you can easily reach the maximum value Python will accept before you hit maximum recursion depth:

>>> fibonacci_memoized(332)
1082459262056433063877940200966638133809015267665311237542082678938909
0.009884s

I’m not going to give you a comparison—I guess I’m just not too big on leaving my laptop on for a few hours ever with the CPU working on overload. While this essentially just became a post on the glory and wonders of memoization, note how easy it was to get was to get speed improvements of several orders of magnitude by using decorator functions. I’ve already created memoize, you just have to use it. No hassle.

Here’s a bit of homework to practice your decorator-fu: write a decorator function that rounds off the output of another function down to an arbitrary precision. Email it to me (I have a contact form). Let me know if you have some cool uses for decorators, again, email me.

  1. Then, who does?

—★—

« Python and Twitter | Quicksilver vs. Spotlight »