Deferreds

Motivation

Dealing with Blocking Code

When coding I/O based programs - networking code, databases, file access - there are many APIs that are blocking, and many methods where the common idiom is to block until a result is gotten.

class Getter:

    def getData(self, x):
        self.blockUntilResult(x)
        return result

g = Getter()
print g.getData(3)

Don't Call Us, We'll Call You

Unfortunately, Twisted can not support blocking calls in most of its code, since it is single threaded, and event based. The solution for this issue is to refactor the code, so that instead of blocking until data is available, we return immediately, and use a callback to notify the requester once the data eventuall arrives. However, this is harder to use, doesn't deal with error raised while waiting for the data, and makes connecting such classes together rather difficult. Nonetheless, looking at how this is implemented will help us understand Deferreds.

class Getter:

    def getData(self, x, callback):
        self.callback = callback
        # this call does not block, it ensure self.gotResult is called
        # when we have the result
        self.onResult(x, self.gotResult)
    
    def gotResult(self, result):
        self.callback(result)

def gotData(d):
    print d

g = Getter()
g.getData(3, gotData)
# run the program's event loop here

Deferreds

A twisted.python.defer.Deferred is a promise that a function will at some point have a result. We can attach callback functions to a Deferred, and once it gets a result these callbacks will be called. In addition Deferreds allow the developer to register a callback for an error, with the default behavior of logging the error. This is an asynchronous equivalent of the common idiom of blocking until a result is returned or an exception it raised.

As we said, multiple callbacks can be added to a Deferred. The first callback in the Deferred's callback chain will be called with the result, the second with the result of the first callback, and so on. Why do we need this? Well, consider a Deferred returned by twisted.enterprise.adbapi - the result of a SQL query. A web widget might add a callback that converts this result into HTML, and pass the Deferred onwards, where the callback will be used by twisted to return the result to the HTTP client.

In order for a Deferred to pass its results to a register callback function, it needs to be armed. Frameworks that support Deferreds - twisted.web.widgets, twisted.web.xmlrpc, twisted.spread.pb server-side objects - will arm the Deferred automatically for you.

import sys
from twisted.python import defer

class Getter:

    def getResult(self, x):
        self.d = defer.Deferred()
        self.doNonblockingStuff(x)
        return self.d
    
    def gotResult(self, result):
        """Called when we get some info from somewhere via the event loop.

        E.g. this may be called because we got a chunk of data off a socket.
        """
        if self.goodResult(result):
            # tell the Deferred that we have a result for it
            self.d.callback(result)
        else:
            # tell the Deferred that we have an error
            self.d.errback("An error has occured.")

def printData(d): sys.stdout.write(d)
def printError(e): sys.stderr.write(e)

g = Getter()
d = g.getResult(3) # notice how this is similar to the blocking version
d.addCallback(printData) # printData will be called when a result is available
d.addErrback(printError) # printError will be called on an error
d.arm()

# run main event loop here