Applications of the Twisted Framework

Jp Calderone


Two projects developed using the Twisted framework are described; one, Twisted.names, which is included as part of the Twisted distribution, a domain name server and client API, and one, Pynfo, which is packaged separately, a network information robot.

Twisted (dot) Names


The field of domain name servers is well explored and numerous strong, widely-deployed implementations of the protocol exist. DNSSEC, IPv6, service location, geographical location, and many of the other DNS extension proposals all have high quality support in BIND, djbdns, maradns, and others. From a client's perspective, though, the landscape looks a little different. APIs to perform arbitrary domain name lookups are sparse. In contrast, Twisted.names presents a richly featured, asynchronous client API.

Names Server

Names is capable of operating as a fully functional domain name server. It implements caching, recursive lookups, and can act as the authority for an arbitrary number of domains. It is not, however, a finely tuned performance machine. Responding to queries can take about twice the time other domain name servers might need. It has not been investigated whether this is a design limitation or merely the result of an unoptimized implementation.

Names Client

As a client, Names provides an easy interface to every type of record supported by. Looking up the MX records for a host, for example, might look like this:

    def _cbMailExchange(results):
        # Callback for MX query
        answers = results[0]
        print 'Mail Exchange is: ', answers

    def _ebMailExchange(failure):
        # Error callback for MX query
        print 'Lookup failed:'

    from twisted.names import client
    d = client.lookupMailExchange('')
    d.addCallbacks(_cbMailExchange, _ebMailExchange)

Looking up other record types is as simple as calling a different lookup* function.


As with most network software written using Twisted, the first step in developing Names was to write the protocol support. In this case, the protocol was DNS, and support was partially implemented. However, it attempted to merge support for both UDP and TCP, and ended up with less than optimal results. Much of this code was discarded, though some of the lowest level encoding and decoding code worked well and was re-used.

With the two protocol classes, DNSDatagramProtocol and DNSProtocol (the TCP version) implemented, the next step was to write classes which created the proper behavior for a domain name server. This logic was put in the twisted.names.server.DNSServerFactory class, which in turn relies on several different kind of Resolvers to find the appropriate response to queries it receives from the protocol instance.

The chain of execution, then, is this: a packet is received by the protocol object (a DNSDatagramProtocol or DNSProtocol instance); the packet is decoded by twisted.protocols.dns.RRHeader in cooperation with one of the record classes (twisted.protocols.dns.Record_A for example); the decoded twisted.protocols.dns.Query object is passed up to the twisted.names.server.DNSServerFactory, which determines the query type and invokes the appropriate lookup method on each of its resolver objects in turn; if an answer is found, it is passed back down to the protocol instance (otherwise the appropriate bit for an error condition is set), where it is encoded and transmitted back to the client.

There are four kinds of resolvers in the current implementation. The first three are authorities, caches, and recursive resolvers. They are generally queried, in this order, using the fourth resolver, the "chain" resolver, which simply queries the resolvers it knows about, moving on to the next when any given resolver fails to produce a response, and generating the proper exception when the last resolver has failed.


There are several aspects of Twisted.Names that might preclude its use in "production" software. These issues stem mainly from its immaturity, it being less than six months old at the writing of this paper.



Pynfo was originally begun as a learning project to become acquainted with the Twisted framework. After a brief initial development period and an extended period of non-development, Pynfo was picked up again to serve as a replacement for several existing robots, each with fragile code bases and with designs not intended for future integration with other services. After it subsumed the functions of network relaying and Google searches, other desired features, which enhanced the IRC medium and had not previously been considered due to the difficulty of extending existing robots, were added to Pynfo, prompting the development of an elementary plug-in system to further facilitate the integration process.


Pynfo performs such simple tasks as noting the last time an individual spoke and querying the Google search engine, as well as several more complex operations like relaying traffic between different IRC networks and publishing channel logs through an HTTP interface.

Toward these ends, it is useful to abstract the functionality into several different layers:

Employing Components

Twisted provides a component system which Pynfo relies on to split up useful functionality used in different areas of the code. The Interface class is the primary element in the component system, and is used as a location for a semi-format definition of an API, as well as documentation. Classes declare that they implement an Interface by including it in their __implements__ tuple attribute. Interfaces can also be added to classes by third parties using the registerAdapter() function. This takes an Adapter type in addition to the interface being registered and the type it is being registered for. Adapters are a objects which can store additional state information and implement functionality without being part of the classes that are "actually" being operated upon. They, as their name suggests, adapt components to conform to interfaces.

Components can implement interfaces themselves, or maintain a cache of adapter objects for each interfaces that is requested of them. These persist like any other attribute, and so state stored in adapters remains associated with the component as long as that component exists, or until the adapter is explicitly removed.

Pynfo's Factory class uses two adapters to implement two basic Interfaces that many plugins find useful. The first is the IStorage interface.

    class IStorage(components.Interface):

        def store(self, key, version, value):
            Store any pickleable object

        def retrieve(self, key, version):
            Retrieve the previously stored object associated with key and
An example usage of this interface is the PyPI plugin, which polls the Python Package Index and reports updates to a configurable list of outputs:
    def init(factory):
        global notifyChannels
        store = factory.getComponent(interfaces.IStorage)
            notifyChannels = store.retrieve('pypi', __version__)
        except error.RetrievalError:
            notifyChannels = []


The module requests the component of factory which implements IStorage, then attempts to load any previously stored version of "notifyChannels". If none is found, it defaults to none. In the finalizer below, this global is stored, using the same interfaced, to be retrieved when the module is next initialized.

    def fini(factory):
        s = factory.getComponent(interfaces.IStorage)'pypi', __version__, notifyChannels)
The second interface allows low granularity scheduling of events:
    class IScheduler(components.Interface):
        MINUTELY = 60
        HOURLY = 60 * 60
        DAILY = 60 * 60 * 24
        WEEKLY = 60 * 60 * 24 * 7

        def schedule(self, period, fn, *args, **kw):
            Cause a function to be invoked at regular intervals with the given
The Adapter which implements this interface is just as simple:
    class SchedulerAdapter(components.Adapter):
        __implements__ = (interfaces.IScheduler,)

        def schedule(self, period, fn, *args, **kw):
            from twisted.internet import reactor
            def cycle():
                fn(*args, **kw)
                reactor.callLater(period, cycle)
            reactor.callLater(period, cycle)

Implementing these interfaces as adapters using the component system has two primary advantages over a simple inheritance or mixins approach. First, it allows plugins to add completely new behavior to the system without complex and fragile manipulation of the factory's __class__ attribute. This is a big win when it comes to plugins that want to share new functionality with other plugins. For example, the "ignore" plugin adds an IDiscriminating interface and an adapter which implements it. Once this plugin is loaded, any other plugin can request the component for IDiscriminating and add users to or remove users from the ignore list.

The Plugin Framework

Before a module can be loaded and initialized as a plugin, it must be located. This could be done with a simple use of os.listdir(), or __all__ could be set to include each new plugin added. Twisted provides another way, though.

The twisted.python.plugin provides the most high-level interface to the plugin system, a function called getPlugIns. It usually takes one argument, a plugin type, which is an arbitrary string used to categorize the different kinds of plugins available on a system. Twisted's own "mktap" tool uses the "tap" plugin type. For Pynfo, I have elected to use the "infobot" string. getPlugIns("infobot") searches the system (by way of PYTHONPATH) for files named "plugins.tml". These files contain python source, and are run as such; a function, "register" is placed in their namespace, and the most common action for them is to invoke this function one or more times, providing information about a plugin. Here is a snippet from one which Pynfo uses:

        description="Commands to check the weather at "
                    "various places around the world.",

Any number of plugin.tml files may exist in the filesystem, allowing per-user and even per-robot plugins to be installed, all without modifying the Pynfo installation itself. The second argument indicates the module which may be imported to get this plugin. Pynfo traverses the resulting list, importing these modules, and initializing them if necessary.