Writing Servers in Twisted

Concepts

Twisted is an async framework, so remember that none of your code can block. You'll get notified about new data coming in, you'll write small functions which deal with it and end so something else can do its thing. A Twisted server consists of several parts:
twisted.main.Application
This class is Twisted's connection to the world. Usually, most writers do not deal with it (see below, on deployment considerations).
twisted.protocols.protocol.Factory
90% of the time, this class will be good enough. This class seats inside an application, listens to a port, and builds a protocol for each new connection. The default buildProtocol calls the attribute "protocol" (which is usually a class), and then initializes the "factory" attribute. Overriding buildProtocol is easy: just return something which follows the protocol interface.
twisted.protocols.protocol.Protocol
While there is no need to derive from this class to write a protocol handler, it usually makes your work easier. The next section will deal with writing a new protocol, since this is where most of your code will be.

Writing protocols

The protocol interface has several methods:
def makeConnection(self, transport, server = None)
This method is already implemented inside twisted.protocols.protocol.Protocol. It will initialize your protocol with the "connected", "transport" and "server" attributes, and call "connectionMade". If you want to do connection-specific initializing, the right way to do it is to implement "connectionMade".
def connectionMade(self)
Called when a connection to the server has been established. Here you should set up attributes, send greetings and similar things.

Note that for "clientish" connections, you will probably want to send the initial request to the server here, unless the server sends a greeting, in which case you would set up a callback to the greeting.

def dataReceived(self, data)
The protocol's raison d'etre: some data was sent by the other side. You will probably want to handle it in some way.

There are a couple of Protocol derived classes which override this method to provide a higher level view of the data - netstrings, lines and length-prefixed messages.

def connectionLost(self)
The low-level connection is lost. Clean up after yourself.
def connectionFailed(self)
The connection cannot be made. This will only be called on client protocols.

Transports

One of the most important instance variables of the Protocol class is "transport", which must follow the transport interface. This is what lets protocol classes talk to the world.

Transports have the following methods:

.write(data)
Write the data when next the connection is available for writing. This method will never block, but it might not send the data right away.
.loseConnection()
When there is no more pending data to write, close the connection down.
.getPeer()
Returns a tuple. The first element of the tuple is a string describing the kind of the transport, with 'INET' being the identifier of good old TCP sockets.

Writing New Transport Types

I will concentrate here on server transports.

Writing a transport type is usually a two-step procedure. First, you will need to write the port-like class. The class's constructor should take, among other transport-specific administrative data (for example, for TCP ports this could be the numeric port and the addresses to listen on), a factory. The only thing that should be assumed by the class is that the given factory has a .buildProtocol method which creates protocol-interface class.

This class should be pickle-safe, which means any process specific information (such as open files) should be thrown out by the __getstate__ method.

The class should have a .startListening() method, which should take whatever action it needs to open a port for listening. It should also have a .fileno method, which will return a valid file descriptor. When this file descriptor is valid for reading, the .doRead method will be called. Usually, this method will create a transport, call the factory's .buildProtocol and call .makeConnection on the protocol with the transport as an argument.

The transports written for every connection and passed to protocols' .makeConnection must follow the transport interface:

.write(data)
Queue some data to write. Do not block.
.loseConnection()
When there will be no more data to write, close down the connection.
.getPeer()
Return a tuple describing your peer. This should be a tuple - (TYPE, ...), where the ... depend on your connection's semantics. Use common sense. So far, for files, it is ('file', filename), for TCP sockets it is ('INET', address, port) and for UNIX-domain sockets it is ('UNIX', path). You can probably count on 'INET6' being IPv6 TCP sockets.

In addition, the select loop will call the following methods:

.doWrite()
Flush the data you have, in a non-blocking manner.
.doRead()
Read data, and call the protocol's .dataReceived method with this chunk.
.connectionLost()
If either .doRead() or .doWrite() return CONNECTION_LOST, this method will be called by the select loop. It should clean up any OS-level resources, and call the protocol's .connectionLost()

Deployment

Most protocols are deployed using the "tap" mechanism, which hides many none interesting details. Tap-based deployment works by writing a module in the twisted.tap package which is compatible to the tap interface:
The Options class
This should inherit from twised.python.usage.Options and handle valid options. It must be called Options. The next section will deal with writing Option classes.
The usage_message string
This should be a helpful multiline message which would be displayed when the user asks for help.
The getPorts(app, config) function
The function gets a twisted.main.Application and an instance of the Options class defined, and should return an array of two-tuples (port, Factory). See next section about how to get the command-line options from an Option instance.

Writing Option Class

optStrings
This is a list of 3-lists, each should be [long_name, flag, default].

This will set the "long_name" attribute to the value of that option, or to default, if it was not be given on the command line.

optFlags
This is a list of 2-lists, each should be [long_name, flag].

This will set the "long_name" attribute to 1 if it was given on the command line, else to 0.

opt_* methods
If the method takes no arguments (except from self), it will be called once for every time the part after the "opt_" is given.

If it takes one argument, then it will be the value for the option given on the command line.

opt_* methods are called in the order the arguments are given on the command line.

Finishing Touches

OK.

You have written a protocol, a factory, a twisted.tap submodule which parses command line options and returns a valid value from getPorts. What now?

  1. Run "mktap <your twisted.tap submodule> <valid options>". This will create a file named "<your twisted.tap submodule>.tap", which is a pickle containing a twisted.main.Application.
  2. Run "twistd -f <your twisted.tap.submodule>.tap". twistd knows to turn pickled twisted.main.Application into a living, breathing application. Some common arguments are "-l <logfile>" and "-n" (no daemonize).

That is it. You have written a twisted server.

Post-finishing Touches

So, you are happily running a twisted server via twistd, when suddenly, you wish you could add another HTTP server running on port 833. Or maybe you want your living HTTP server to become virtual-domains based. What are you going to do?

Well, you have several options.

Warning: some of this stuff assumes a 0.9.6 or later version of twisted. If you are running an older version, you should upgrade

To add another HTTP server to your tap, just use mktap's --append option. Note how that does not require support in the twisted.tap.whatever module, so it will just work even for your newly written protocol.

Changing the HTTP is a bit more complicated. What you will certainly want to do is add a telnet server to your tap. Just use "mktap --append telnet". It is recommended to give the username/password arguments and not use the defaults, for obvious reasons.

Now that you have the telnet server, you can just telnet into your application and change things there on the run.

When you telnet, you are given, after the usual name/password login box, absolutely no prompt. Hopefully, this will change. The first thing you will want to do is import __main__. In __main__, there is a global named "application". This is your application. If you want, you can add servers to it on the fly:

import __main__
from twisted.web import static, server
file = static.File("/var/www")
factory = server.Site(file)
__main__.application.listenOn(8443, factory)

That's it! It really is this easy.

Now, let us say you want to configure the server to serve another directory instead.

import __main__
__main__.application.ports
-> [<twisted.protocols.telnet.ShellFactory on 4040>, <twisted.web.server.Site on 8443>]

So it is the second port. Great!

__main__.application.ports[1].factory.resource.path = 'static'

In similar ways, you can configure your factory's resource to be a vhost.NamedVirtualHost, configure the name servers, and leave it running.

If you want to save known good configurations, just use

__main__.application.save("known-good")

Which will save the application to <original tap file>+"known-good.tap" On shutdown, it will behave as if __main__.application.save("shutdown") has been issued.