Applications of the Twisted Framework

Jp Calderone

exarkun@twistedmatrix.com

ABSTRACT

Two projects developed using the Twisted framework are described; one, Twisted.names, which is included as part of the Twisted distribution, a domain name server and client API, and one, Pynfo, which is packaged separately, a network information robot.

Twisted (dot) Names

Motivation

The field of domain name servers is well explored and numerous strong, widely-deployed implementations of the protocol exist. DNSSEC, IPv6, service location, geographical location, and many of the other DNS extension proposals all have high quality support in BIND, djbdns, maradns, and others. From a client's perspective, though, the landscape looks a little different. APIs to perform arbitrary domain name lookups are sparse. In contrast, Twisted.names presents a richly featured, asynchronous client API.

Names Server

Names is capable of operating as a fully functional domain name server. It implements caching, recursive lookups, and can act as the authority for an arbitrary number of domains. It is not, however, a finely tuned performance machine. Responding to queries can take about twice the time other domain name servers might need. It has not been investigated whether this is a design limitation or merely the result of an unoptimized implementation.

Names Client

As a client, Names provides an easy interface to every type of record supported by. Looking up the MX records for a host, for example, might look like this:

    def _cbMailExchange(results):
        # Callback for MX query
        answers = results[0]
        print 'Mail Exchange is: ', answers

    def _ebMailExchange(failure):
        # Error callback for MX query
        print 'Lookup failed:'
        failure.printTraceback()

    from twisted.names import client
    d = client.lookupMailExchange('example-domain.com')
    d.addCallbacks(_cbMailExchange, _ebMailExchange)

Looking up other record types is as simple as calling a different lookup* function.

Implementation

As with most network software written using Twisted, the first step in developing Names was to write the protocol support. In this case, the protocol was DNS, and support was partially implemented. However, it attempted to merge support for both UDP and TCP, and ended up with less than optimal results. Much of this code was discarded, though some of the lowest level encoding and decoding code worked well and was re-used.

With the two protocol classes, DNSDatagramProtocol and DNSProtocol (the TCP version) implemented, the next step was to write classes which created the proper behavior for a domain name server. This logic was put in the twisted.names.server.DNSServerFactory class, which in turn relies on several different kind of Resolvers to find the appropriate response to queries it receives from the protocol instance.

The chain of execution, then, is this: a packet is received by the protocol object (a DNSDatagramProtocol or DNSProtocol instance); the packet is decoded by twisted.protocols.dns.RRHeader in cooperation with one of the record classes (twisted.protocols.dns.Record_A for example); the decoded twisted.protocols.dns.Query object is passed up to the twisted.names.server.DNSServerFactory, which determines the query type and invokes the appropriate lookup method on each of its resolver objects in turn; if an answer is found, it is passed back down to the protocol instance (otherwise the appropriate bit for an error condition is set), where it is encoded and transmitted back to the client.

There are four kinds of resolvers in the current implementation. The first three are authorities, caches, and recursive resolvers. They are generally queried, in this order, using the fourth resolver, the "chain" resolver, which simply queries the resolvers it knows about, moving on to the next when any given resolver fails to produce a response, and generating the proper exception when the last resolver has failed.

Shortcomings

There are several aspects of Twisted.Names that might preclude its use in "production" software. These issues stem mainly from its immaturity, it being less than six months old at the writing of this paper.

Possibly of foremost interest to those who might use it in a high-load environment, it has somewhat poor runtime performance characteristics. One potential reason for this is the extensive use of exceptions to signal the relatively common case of a resolver lookup failing. Solutions to this problem are apparent, but an implementation change has not been attempted. Until this area of its development is more fully examined, it will likely not be of use in anything other than for low- to mid-load tasks, or with more hardware available to it than might seem reasonable.
No attempt has been made to implement DNSSEC.
Certain areas of the server remain out of compliance with the standardized RFCs, occasionally causing undesirable behavior when interacting with clients. This most frequently manifests itself as a lookup which fails the first time and succeeds on subsequent attempts. It is not believed that these represent architectural flaws, only small oversights in areas such as the "additional processing" sections of the current authority resolver implementations.

Pynfo

Motivation

Pynfo was originally begun as a learning project to become acquainted with the Twisted framework. After a brief initial development period and an extended period of non-development, Pynfo was picked up again to serve as a replacement for several existing robots, each with fragile code bases and with designs not intended for future integration with other services. After it subsumed the functions of network relaying and Google searches, other desired features, which enhanced the IRC medium and had not previously been considered due to the difficulty of extending existing robots, were added to Pynfo, prompting the development of an elementary plug-in system to further facilitate the integration process.

Architecture

Pynfo performs such simple tasks as noting the last time an individual spoke and querying the Google search engine, as well as several more complex operations like relaying traffic between different IRC networks and publishing channel logs through an HTTP interface.

Toward these ends, it is useful to abstract the functionality into several different layers:

The factory: All shared data, such as the channels a given user is known to be in, the plugins currently loaded, and the addresses of servers to connect to, is aggregated here. When it is necessary to make a connection, the factory creates an instance of the appropriate Protocol subclass, in a manner similar to this:
```
    def buildProtocol(self, address):
        for net in self.data['networks'].values():
            if net.address == address:
                break

        proto = IRCProtocol(net)
        self.allBots[net.alias] = proto
        proto.factory = self
        return proto
    
```
The factory instance is created only once, and that instance persists through the entire time a particular Pynfo bot operates.
The protocol: Each kind of service Pynfo can connect to has a Protocol class associated with it, a class which handles the specifics of communicating over this protocol. Unlike the factory, protocols instances can be short lived and are created and destroyed as many times as network connectivity demands. When a Pynfo robot shuts down and is serialized to disk, all Protocol instances are destroyed and discarded, to be created anew when the robot is restarted.
Plugins: These give Pynfo most of its functionality. From the very simple logging module, which does no more than write strings to disk, to the esoteric lookup module, which translates hostnames into dotted-quads, to the informative dictionary module, which queries an online dictionary, plugins come in all shapes and sizes, and can be written to fill almost any niche.

Employing Components

Twisted provides a component system which Pynfo relies on to split up useful functionality used in different areas of the code. The Interface class is the primary element in the component system, and is used as a location for a semi-format definition of an API, as well as documentation. Classes declare that they implement an Interface by including it in their __implements__ tuple attribute. Interfaces can also be added to classes by third parties using the registerAdapter() function. This takes an Adapter type in addition to the interface being registered and the type it is being registered for. Adapters are a objects which can store additional state information and implement functionality without being part of the classes that are "actually" being operated upon. They, as their name suggests, adapt components to conform to interfaces.

Components can implement interfaces themselves, or maintain a cache of adapter objects for each interfaces that is requested of them. These persist like any other attribute, and so state stored in adapters remains associated with the component as long as that component exists, or until the adapter is explicitly removed.

Pynfo's Factory class uses two adapters to implement two basic Interfaces that many plugins find useful. The first is the IStorage interface.

    class IStorage(components.Interface):

        def store(self, key, version, value):
            """
            Store any pickleable object
            """

        def retrieve(self, key, version):
            """
            Retrieve the previously stored object associated with key and
            version
            """

An example usage of this interface is the PyPI plugin, which polls the Python Package Index and reports updates to a configurable list of outputs:

    def init(factory):
        global notifyChannels
        store = factory.getComponent(interfaces.IStorage)
        try:
            notifyChannels = store.retrieve('pypi', __version__)
        except error.RetrievalError:
            notifyChannels = []

The module requests the component of factory which implements IStorage, then attempts to load any previously stored version of "notifyChannels". If none is found, it defaults to none. In the finalizer below, this global is stored, using the same interfaced, to be retrieved when the module is next initialized.

    def fini(factory):
        s = factory.getComponent(interfaces.IStorage)
        s.store('pypi', __version__, notifyChannels)

The second interface allows low granularity scheduling of events:

    class IScheduler(components.Interface):
        MINUTELY = 60
        HOURLY = 60 * 60
        DAILY = 60 * 60 * 24
        WEEKLY = 60 * 60 * 24 * 7


        def schedule(self, period, fn, *args, **kw):
            """
            Cause a function to be invoked at regular intervals with the given
            arguments.
            """

The Adapter which implements this interface is just as simple:

    class SchedulerAdapter(components.Adapter):
        __implements__ = (interfaces.IScheduler,)

        def schedule(self, period, fn, *args, **kw):
            from twisted.internet import reactor
            def cycle():
                fn(*args, **kw)
                reactor.callLater(period, cycle)
            reactor.callLater(period, cycle)

Implementing these interfaces as adapters using the component system has two primary advantages over a simple inheritance or mixins approach. First, it allows plugins to add completely new behavior to the system without complex and fragile manipulation of the factory's __class__ attribute. This is a big win when it comes to plugins that want to share new functionality with other plugins. For example, the "ignore" plugin adds an IDiscriminating interface and an adapter which implements it. Once this plugin is loaded, any other plugin can request the component for IDiscriminating and add users to or remove users from the ignore list.

The Plugin Framework

Before a module can be loaded and initialized as a plugin, it must be located. This could be done with a simple use of os.listdir(), or __all__ could be set to include each new plugin added. Twisted provides another way, though.

The twisted.python.plugin provides the most high-level interface to the plugin system, a function called getPlugIns. It usually takes one argument, a plugin type, which is an arbitrary string used to categorize the different kinds of plugins available on a system. Twisted's own "mktap" tool uses the "tap" plugin type. For Pynfo, I have elected to use the "infobot" string. getPlugIns("infobot") searches the system (by way of PYTHONPATH) for files named "plugins.tml". These files contain python source, and are run as such; a function, "register" is placed in their namespace, and the most common action for them is to invoke this function one or more times, providing information about a plugin. Here is a snippet from one which Pynfo uses:

    register(
        "Weather",
        "Pynfo.plugins.weather",
        description="Commands to check the weather at "
                    "various places around the world.",
        type="infobot"
    )

Any number of plugin.tml files may exist in the filesystem, allowing per-user and even per-robot plugins to be installed, all without modifying the Pynfo installation itself. The second argument indicates the module which may be imported to get this plugin. Pynfo traverses the resulting list, importing these modules, and initializing them if necessary.