Perspective Broker: Translucent Remote Method calls in Twisted

• Brian Warner: <warner@lothar.com>

Abstract

One of the core services provided by the Twisted networking framework is Perspective Broker, which provides a clean, secure, easy-to-use Remote Procedure Call (RPC) mechanism. This paper explains the novel features of PB, describes the security model and its implementation, and provides brief examples of usage.

PB is used as a foundation for many other services in Twisted, as well as projects built upon the Twisted framework. twisted.web servers can delegate responsibility for different portions of URL-space by distributing PB messages to the object that owns that subspace. twisted.im is an instant-messaging protocol that runs over PB. Applications like CVSToys and the BuildBot use PB to distribute notices every time a CVS commit has occurred. Using Perspective Broker as the RPC layer allows these projects to stay focused on the interesting parts.

The PB protocol is not limited to Python. There is a working Java implementation available from the Twisted web site, as is an Emacs-Lisp version (which can be used to control a PB-enabled application from within your editing session, or effectively embed a Python interpreter in Emacs). Python's dynamic and introspective nature makes Perspective Broker easier to implement (and very convenient to use), but neither are strictly necessary. With a set of callback tables and a good dictionary implementation, it would be possible to implement the same protocol in C, C++, Perl, or other languages.

Overview

Features

Perspective Broker provides the following basic RPC features.

• remotely-invokable methods: certain methods (those with names that start with remote_) of pb.Referenceable objects can be invoked by remote clients who hold matching pb.RemoteReference objects.
• transparent, controllable object serialization: other objects sent through those remote method invocations (either as arguments or in the return value) will be automatically serialized. The data that is serialized, and the way they are represented on the remote side, depends upon which twisted.pb.flavor class they inherit from, and upon overridable methods to get and set state.
• per-connection object ids: certain objects that are passed by reference are tracked when they are sent over a wire. If the receiver sends back the reference it received, the sender will see their original object come back to them.
• twisted.cred authentication layer: provides common username/password verification functions. pb.Viewable objects keep a user reference with them, so remotely-invokable methods can find out who invoked them.
• remote exception reporting: exceptions that occur in remote methods are wrapped in Failure objects and serialized so they can be provided to the caller. All the usual traceback information is available on the invoking side.
• runs over arbitrary byte-pipe transports: including TCP, UNIX-domain sockets, and SSL connections. UDP support (in the form of Airhook) is being developed.
• numerous sandwich-related puns: PB, Jelly, Banana, twisted.spread, Marmalade, Tasters, and Flavors. By contrast, CORBA and XML-RPC have few, if any, puns in their naming conventions.

Example

Here is a simple example of PB in action. The server code creates an object that can respond to a few remote method calls, and makes it available on a TCP port. The client code connects and runs two methods.

from twisted.spread import pb
import twisted.internet.app

class ServerObject(pb.Root):
    def remote_add(self, one, two):
        answer = one + two
        print "returning result:", answer
        return answer
    def remote_subtract(self, one, two):
        return one - two

app = twisted.internet.app.Application("server1")
app.listenTCP(8800, pb.BrokerFactory(ServerObject()))
app.run(save=0)

from twisted.spread import pb
from twisted.internet import reactor

class Client:
    def connect(self):
        deferred = pb.getObjectAt("localhost", 8800, 30)
        deferred.addCallbacks(self.got_obj, self.err_obj)
        # when the Deferred fires (i.e. when the connection is established and
        # we receive a reference to the remote object), the 'got_obj' callback
        # will be run

    def got_obj(self, obj):
        print "got object:", obj
        self.server = obj
        print "asking it to add"
        def2 = self.server.callRemote("add", 1, 2)
        def2.addCallbacks(self.add_done, self.err)
        # this Deferred fires when the method call is complete

    def err_obj(self, reason):
        print "error getting object", reason
        self.quit()

    def add_done(self, result):
        print "addition complete, result is", result
        print "now trying subtract"
        d = self.server.callRemote("subtract", 5, 12)
        d.addCallbacks(self.sub_done, self.err)

    def err(self, reason):
        print "Error running remote method", reason
        self.quit()

    def sub_done(self, result):
        print "subtraction result is", result
        self.quit()

    def quit(self):
        print "shutting down"
        reactor.stop()

c = Client()
c.connect()
reactor.run()

When this is run, the client emits the following progress messages:

% ./pb-client1.py
got object: <twisted.spread.pb.RemoteReference instance at 0x817cab4>
asking it to add
addition complete, result is 3
now trying subtract
subtraction result is -7
shutting down

This example doesn't demonstrate instance serialization, exception reporting, authentication, or other features of PB. For more details and examples, look at the PB howto docs at twistedmatrix.com.

Why Translucent References?

Remote function calls are not the same as local function calls. Remote calls are asynchronous. Data exchanged with a remote system may be interpreted differently depending upon version skew between the two systems. Method signatures (number and types of parameters) may differ. More failure modes are possible with RPC calls than local ones.

Transparent RPC systems attempt to hide these differences, to make remote calls look the same as local ones (with the noble intention of making life easier for programmers), but the differences are real, and hiding them simply makes them more difficult to deal with. PB therefore provides translucent method calls: it exposes these differences, but offers convenient mechanisms to handle them. Python's flexible object model and exception handling take care of part of the problem, while Twisted's Deferred class provides a clean way to deal with the asynchronous nature of RPC.

Asynchronous Invocation

A fundamental difference between local function calls and remote ones is that remote ones are always performed asynchronously. Local function calls are generally synchronous (at least in most programming languages): the caller is blocked until the callee finishes running and possibly returns a value. Local functions which might block (loosely defined as those which would take non-zero or indefinite time to run on infinitely fast hardware) are usually marked as such, and frequently provide alternative APIs to run in an asynchronous manner. Examples of blocking functions are select() and its less-generalized cousins: sleep(), read() (when buffers are empty), and write() (when buffers are full).

Remote function calls are generally assumed to take a long time. In addition to the network delays involved in sending arguments and receiving return values, the remote function might itself be blocking.

Transparent RPC systems, which pretend that the remote system is really local, usually offer only synchronous calls. This prevents the program from getting other work done while the call is running, and causes integration problems with GUI toolkits and other event-driven frameworks.

Failure Modes

In addition to the usual exceptions that might be raised in the course of running a function, remotely invoked code can cause other errors. The network might be down, the remote host might refuse the connection (due to authorization failures or resource-exhaustion issues), the remote end might have a different version of the code and thus misinterpret serialized arguments or return a corrupt response. Python's flexible exception mechanism makes these errors easy to report: they are just more exceptions that could be raised by the remote call. In other languages, this requires a special API to report failures via a different path than the normal response.

Deferreds to the rescue

In PB, Deferreds are used to handle both the asynchronous nature of the method calls and the various kinds of remote failures that might occur. When the method is invoked, PB returns a Deferred object that will be fired later, when the response (success or failure) is received from the remote end. The caller (the one who invoked callRemote) is free to attach callback and errback handlers to the Deferred. If an exception is raised (either by the remote code or a network failure during processing), the errback will be run with the wrapped exception. If the function completes normally, the callback is run.

By using Deferreds, the invoking program can get other work done while it is waiting for the results. Failure is handled just as cleanly as success.

In addition, the remote method can itself return a Deferred instead of an actual return value. When that Deferreds fires, the data given to the callback will be serialized and returned to the original caller. This allows the remote server to perform other work as well, putting off the answer until one is available.

Calling Remote Methods

Perspective Broker is first and foremost a mechanism for remote method calls: doing something to a local object which causes a method to get run on a distant one. The process making the request is usually called the client, and the process which hosts the object that actually runs the method is called the server. Note, however, that method requests can go in either direction: instead of distinguishing client and server, it makes more sense to talk about the sender and receiver for any individual method call. PB is symmetric, and the only real difference between the two ends is that one initiated the original TCP connection and the other accepted it.

With PB, the local object is an instance of twisted.spread.pb.RemoteReference, and you do something to it by calling its .callRemote method. This call accepts a method name and an argument list (including keyword arguments). Both are serialized and sent to the receiving process, and the call returns a Deferred, to which you can add callbacks. Those callbacks will be fired later, when the response returns from the remote end.

That local RemoteReference points at a twisted.spread.pb.Referenceable object living in the other program (or one of the related callable flavors). When the request comes over the wire, PB constructs a method name by prepending remote_ to the name requested by the remote caller. This method is looked up in the pb.Referenceable and invoked. If an exception is raised (including the AttributeError that results from a bad method name), the error is wrapped in a Failure object and sent back to the caller. If it succeeds, the result is serialized and sent back.

The caller's Deferred will either have the callback run (if the method completed normally) or the errback run (if an exception was raised). The Failure object given to the errback handler allows a full stack trace to be displayed on the calling end.

For example, if the holder of the RemoteReference does rr.callRemote("foo", 1, 3), the corresponding Referenceable will be invoked with r.remote_foo(1, 3). A callRemote of bar would invoke remote_bar, etc.

Obtaining other references

Each pb.RemoteReference object points to a pb.Referenceable instance in some other program. The first such reference must be acquired with a bootstrapping function like pb.getObjectAt, but all subsequent ones are created when a pb.Referenceable is sent as an argument to (or a return value from) a remote method call.

When the arguments or return values contain references to other objects, the object that appears on the other side of the wire depends upon the type of the referred object. Basic types are simply copied: a dictionary of lists will appear as a dictionary of lists, with internal references preserved on a per-method-call basis (just as Pickle will preserve internal references for everything pickled at the same time). Class instances are restricted, both to avoid confusion and for security reasons.

Transferring Instances

PB only allows certain kinds of objects to be transferred to and from remote processes. Most of these restrictions are implemented in the Jelly serialization layer, described below. In general, to send an object over the wire, it must either be a basic python type (list, dictionary, etc), or an instance of a class which is derived from one of the four basic PB Flavors: Referenceable, Viewable, Copyable, and Cacheable. Each flavor has methods which define how the object should be treated when it needs to be serialized to go over the wire, and all have related classes that are created on the remote end to represent them.

There are a few kinds of callable classes. All are represented on the remote system with RemoteReference instances. callRemote can be used on these RemoteReferences, causing methods with various prefixes to be invoked.

Viewable (and the related Perspective class) are described later (in Authorization). They provide a secure way to let methods know who is calling them. Any time a Referenceable (or Viewable) is sent over the wire, it will appear on the other end as a RemoteReference. If any of these references are sent back to the system they came from, they emerge from the round trip in their original form.

Note that RemoteReferences cannot be sent to anyone else (there are no third-party references): they are scoped to the connection between the holder of the Referenceable and the holder of the RemoteReference. (In fact, the RemoteReference is really just an index into a table maintained by the owner of the original Referenceable).

There are also two data classes. To send an instance over the wire, it must belong to a class which inherits from one of these.

pb.Copyable

Copyable is used to allow class instances to be sent over the wire. Copyables are copy-by-value, unlike Referenceables which are copy-by-reference. Copyable objects have a method called getStateToCopy which gets to decide how much of the object should be sent to the remote system: the default simply copies the whole __dict__. The receiver must register a RemoteCopy class for each kind of Copyable that will be sent to it: this registration (described later in Representing Instances) maps class names to actual classes. Apart from being a security measure (it emphasizes the fact that the process is receiving data from an untrusted remote entity and must decide how to interpret it safely), it is also frequently useful to distinguish a copy of an object from the original by holding them in different classes.

getStateToCopy is frequently used to remove attributes that would not be meaningful outside the process that hosts the object, like file descriptors. It also allows shared objects to hold state that is only available to the local process, including passwords or other private information. Because the default serialization process recursively follows all references to other objects, it is easy to accidentally send your entire program to the remote side. Explicitly creating the state object (creating an empty dictionary, then populating it with only the desired instance attributes) is a good way to avoid this.

The fact that PB will refuse to serialize objects that are neither basic types nor explicitly marked as being transferable (by subclassing one of the pb.flavors) is another way to avoid the don't tug on that, you never know what it might be attached to problem. If the object you are sending includes a reference to something that isn't marked as transferable, PB will raise an InsecureJelly exception rather than blindly sending it anyway (and everything else it references).

Finally, note that getStateToCopy is distinct from the __getstate__ method used by Pickle, and they can return different values. This allows objects to be persisted (across time) differently than they are transmitted (across [memory]space).

pb.Cacheable

Cacheable is a variant of Copyable which is used to implement remote caches. When a Cacheable is sent across a wire, a method named getStateToCacheAndObserveFor is used to simultaneously get the object's current state and to register an Observer which lives next to the Cacheable. The Observer is effectively a RemoteReference that points at the remote cache. Each time the cached object changes, it uses its Observers to tell all the remote caches about the change. The setter methods can just call observer.callRemote("setFoo", newvalue) for all their observers.

On the remote end, a RemoteCache object is created, which populates the original object's state just as RemoteCopy does. When changes are made, the Observers remotely invoke methods like observe_setFoo in the RemoteCache to perform the updates.

As RemoteCache objects go away, their Observers go away too, and call stoppedObserving so they can be removed from the list.

The PB howto docs have more information and complete examples of both pb.Copyable and pb.Cacheable.

Authorization

As a framework, Perspective Broker (indeed, all of Twisted) was built from the ground up. As multiple use cases became apparent, common requirements were identified, code was refactored, and layers were developed to cleanly serve the needs of all customers. The twisted.cred layer was created to provide authorization services for PB as well as other Twisted services, like the HTTP server and the various instant messaging protocols. The abstract notions of identity and authority it uses are intended to match the common needs of these various protocols: specific applications can always use subclasses that are more appropriate for their needs.

Identity and Perspectives

In twisted.cred, Identities are usernames (with passwords), represented by Identity objects. Each identity has a keyring which authorizes it to access a set of objects called Perspectives. These perspectives represent accounts or other capabilities; each belongs to a single Service. There may be multiple Services in a single application; in fact the flexible nature of Twisted makes this easy. An HTTP server would be a Service, and an IRC server would be another one.

As an example, a login service might have perspectives for Alice, Bob, and Charlie, and there might also be an Admin perspective. Alice has admin capabilities. In addition, let us say the same application has a chat service with accounts for each person (but no special administrator account).

So, in this example, Alice's keyring gives her access to three perspectives: login/Alice, login/Admin, and chat/Alice. Bob only gets two: login/Bob and chat/Bob. Perspective objects have names and belong to Service objects, but the Identity.keyring is a dictionary indexed by (serviceName, perspectiveName) pairs. It uses names instead of object references because the Perspective object might be created on demand. The keys include the service name because Perspective names are scoped to a single service.

pb.Perspective

The PB-specific subclass of the generic Perspective class is also capable of remote execution. The login process results in the authorized client holding a special kind of RemoteReference that will allow it to invoke perspective_ methods on the matching pb.Perspective object. In PB applications that use the twisted.cred authorization layer, clients get this reference first. The client is then dependent upon the Perspective to provide everything else, so the Perspective can enforce whatever security policy it likes.

(Note that the pb.Perspective class is not actually one of the serializable PB flavors, and that instances of it cannot be sent directly over the wire. This is a security feature intended to prevent users from getting access to somebody else's Perspective by mistake, perhaps when a list all users command sends back an object which includes references to other Perspectives.)

PB provides functions to perform a challenge-response exchange in which the remote client proves their identity to get that Perspective reference. The Identity object holds a password and uses an MD5 hash to verify that the remote user knows the password without sending it in cleartext over the wire. Once the remote user has proved their identity, they can request a reference to any Perspective permitted by their Identity's keyring.

There are twisted.cred functions (twisted.enterprise.dbcred) which can pull user information out of a database, and it is easy to create modules that could check /etc/passwd or LDAP instead. Authorization can then be centralized through the Perspective object: each object that is accessible remotely can be created with a pointer to the local Perspective, and objects can ask that Perspective whether the operation is allowed before performing method calls.

Most clients use a helper function called pb.connect() to get the first Perspective reference: it takes all the necessary identifying information (host, port, username, password, service name, and perspective name) and returns a Deferred that will be fired when the RemoteReference is available. (This may change in the future: there are plans afoot to use a URL-like scheme to identify the Perspective, which will probably mean a new helper function).

Viewable

There is a special kind of Referenceable called pb.Viewable. Its remote methods (all named view_) are called with an extra argument that points at the Perspective the client is using. This allows the same Referenceable to be shared among multiple clients while retaining the ability to treat those clients differently. The methods can check with the Perspective to see if the request should be allowed, and can use per-client information in processing the request.

PB Design: Object Serialization

Fundamental to any calling convention, whether ABI or RPC, is how arguments and return values are passed from caller to callee and back. RPC systems require data to be turned into a form which can be delivered through a network, a process usually known as serialization. Sharing complex types (references and class instances) with a remote system requires more care: references should all point to the same thing (even though the object being referenced might live on either end of the connection), and allowing a remote user to create arbitrary class instances in your memory space is a security risk that must be controlled.

PB uses its own serialization scheme called Jelly. At the bottom end, it uses s-expressions (lists of numbers and strings) to represent the state of basic types (lists, dictionaries, etc). These s-expressions are turned into a bytestream by the Banana layer, which has an optional C implementation for speed. Unserialization for higher-level objects is driven by per-class jellyier objects: this flexibility allows PB to offer inheritable classes for common operations. pb.Referenceable is a class which is serialized by sending a reference to the remote end that can be used to invoke remote methods. pb.Copyable is a class which creates a new object on the remote end, with methods that the developer can override to control how much state is sent or accepted. pb.Cacheable sends a full copy the first time it is exchanged, but then sends deltas as the object is modified later.

Objects passed over the wire get to decide for themselves how much information is actually passed to the remote system. Copy-by-reference objects are given a per-connection ID number and stashed in a local dictionary. Copy-by-value objects may send their entire __dict__, or some subset thereof. If the remote method returns a referenceable object that was given to it earlier (either in the same RPC call or an earlier one), PB sends the ID number over the wire, which is looked up and turned into a proper object reference upon receipt. This provides one-sided reference transparency: one end sees objects coming and going through remote method calls in exactly the same fashion as through local calls. Those references are only capable of very specific operations; PB does not attempt to provide full object transparency. As discussed later, this is instrumental to security.

Banana and s-expressions

The Banana low-level serialization layer converts s-expressions which represent basic types (numbers, strings, and lists of numbers, strings, or other lists) to and from a bytestream. S-expressions are easy to encode and decode, and are flexible enough (when used with a set of tokens) to represent arbitrary objects. cBanana is a C extension module which performs the encode/decode step faster than the native python implementation.

Each s-expression element is converted into a message with two or three components: a header, a type marker, and an optional body (used only for strings). The header is a number expressed in base 128. The type marker is a single byte with the high bit set, that both terminates the header and indicate the type of element this message describes (number, list-start, string, or tokenized string).

When a connection is first established, a list of strings is sent to negotiate the dialect of Banana being spoken. The first dialect known to both sides is selected. Currently, the dialect is only used to select a list of string tokens that should be specially encoded (for performance), but subclasses of Banana could use self.currentDialect to influence the encoding process in other ways.

When Banana is used for PB (by negotiating the pb dialect), it has a list of 30ish strings that are encoded into two-byte sequences instead of being sent as generalized string messages. These string tokens are used to mark complex types (beyond the simple lists, strings, and numbers provided natively by Banana) and other objects Jelly needs to do its job.

Jelly

Jelly handles object serialization. It fills a similar role to the standard Pickle module, but has design goals of security and portability (especially to other languages) where Pickle favors efficiency of representation. In addition, Jelly serializes objects into s-expressions (lists of tokens, strings, numbers, and other lists), and lets Banana do the rest, whereas Pickle goes all the way down to a bytestream by itself.

Basic python types (apart from strings and numbers, which Banana can handle directly) are generally turned into lists with a type token as the first element. For example, a python dictionary is turned into a list that starts with the string token dictionary and continues with elements that are lists of [key, value] pairs. Modules, classes, and methods are all transformed into s-expressions that refer to the relevant names. Instances are represented by combining the class name (a string) with an arbitrary state object (which is usually a dictionary).

Much of the rest of Jelly has to do with safely handling class instances (as opposed to basic Python types) and dealing with references to shared objects.

Tracking shared references

Mutable types are serialized in a way that preserves the identity between the same object referenced multiple times. As an example, a list with four elements that all point to the same object must look the same on the remote end: if it showed up as a list pointing to four independent objects (even if all the objects had identical states), the resulting list would not behave in the same way as the original. Changing newlist[0] would not modify newlist[1] as it ought to.

Consequently, when objects which reference mutable types are serialized, those references must be examined to see if they point to objects which have already been serialized in the same session. If so, an object id tag of some sort is put into the bytestream instead of the complete object, indicating that the deserializer should use a reference to a previously-created object. This also solves the issue of recursive or circular references: the first appearance of an object gets the full state, and all subsequent ones get a reference to it.

Jelly manages this reference tracking through an internal _Jellier object (in particular through the .cooked dictionary). As objects are serialized, their id values are stashed. References to those objects that occur after jellying has started can be replaced with a dereference marker and the object id.

The scope of this _Jellier object is limited to a single call of the jelly function, which in general corresponds to a single remote method call. The argument tuple is jellied as a single object (a tuple), so different arguments to the same method will share referenced objects1, but arguments of separate methods will not share them. To do more complex caching and reference tracking, certain PB flavors (see below) override their jellyFor method to do more interesting things. In particular, pb.Referenceable objects have code to insure that one which makes a round trip will come back as a reference to the same object that was originally sent.

An exception to this one-call scope is provided: if the Jellier is created with a persistentStore object, all class instances will be passed through it first, and it has the opportunity to return a persistent id. If available, this id is serialized instead of the object's state. This would allow object references to be shared between different invocations of jelly. However, PB itself does not use this technique: it uses overridden jellyFor methods to provide per-connection shared references.

Representing Instances

Each class gets to decide how it should be represented on a remote system. Sending and receiving are separate actions, performed in separate programs on different machines. So, to be precise, each class gets to decide two things. First, they get to specify how they should be sent to a remote client: what should happen when an instance is serialized (or jellied in PB lingo), what state should be recorded, what class name should be sent, etc. Second, the receiving program gets to specify how an incoming object that claims to be an instance of some class should be treated: whether it should be accepted at all, if so what class should be used to create the new object, and how the received state should be used to populate that object.

A word about notation: in Perspective Broker parlance, to jelly is used to describe the act of turning an object into an s-expression representation (serialization, or at least most of it). Therefore the reverse process, which takes an s-expression and turns it into a real python object, is described with the verb to unjelly.

Jellying Instances

Serializing instances is fairly straightforward. Classes which inherit from Jellyable provide a jellyFor method, which acts like __getstate__ in that it should return a serializable representation of the object (usually a dictionary). Other classes are checked with a SecurityOptions instance, to verify that they are safe to be sent over the wire, then serialized by using their __getstate__ method (or their __dict__ if no such method exists). User-level classes always inherit from one of the PB flavors like pb.Copyable (all of which inherit from Jellyable) and use jellyFor; the __getstate__ option is only for internal use.

Secure Unjellying

Unjellying (for instances) is triggered by the receipt of an s-expression with the instance tag. The s-expression has two elements: the name of the class, and an object (probably a dictionary) which holds the instance's state. At that point in time, the receiving program does not know what class should be used: it is certainly not safe to simply do an import of the classname requested by the sender. That effectively allows a remote entity to run arbitrary code on your system.

There are two techniques used to control how instances are unjellied. The first is a SecurityOptions instance which gets to decide whether the incoming object should accepted or not. It is said to taste the incoming type before really trying to unserialize it. The default taster accepts all basic types but no classes or instances.

If the taster decides that the type is acceptable, Jelly then turns to the unjellyableRegistry to determine exactly how to deserialize the state. This is a table that maps received class names names to unserialization routines or classes.

The receiving program must register the classes it is willing to accept. Any attempts to send instances of unregistered classes to the program will be rejected, and an InsecureJelly exception will be sent back to the sender. If objects should be represented by the same class in both the sender and receiver, and if the class is defined by code which is imported into both programs (an assumption that results in many security problems when it is violated), then the shared module can simply claim responsibility as the classes are defined:

class Foo(pb.RemoteCopy):
  def __init__(self):
    # note: __init__ will *not* be called when creating RemoteCopy objects
    pass
  def __getstate__(self):
    return foo
  def __setstate__(self, state):
    self.stuff = state.stuff
setUnjellyableForClass(Foo, Foo)

In this example, the first argument to setUnjellyableForClass is used to get the fully-qualified class name, while the second defines which class will be used for unjellying. setUnjellyableForClass has two functions: it informs the taster that instances of the given class are safe to receive, and it registers the local class that should be used for unjellying.

Broker

The Broker class manages the actual connection to a remote system. Broker is a Protocol (in Twisted terminology), and there is an instance for each socket over which PB is being spoken. Proxy objects like pb.RemoteReference, which are associated with another object on the other end of the wire, all know which Broker they must use to get to their remote counterpart. pb.Broker objects implement distributed reference counts, manage per-connection object IDs, and provide notification when references are lost (due to lost connections, either from network problems or program termination).

PB over Jelly

Perspective Broker is implemented by sending Jellied commands over the connection. These commands are always lists, and the first element of the list is always a command name. The commands are turned into proto_-prefixed method names and executed in the Broker object. There are currently 9 such commands. Two (proto_version and proto_didNotUnderstand) are used for connection negotiation. proto_message is used to implement remote method calls, and is answered by either proto_answer or proto_error.

proto_cachemessage is used by Observers (see pb.Copyable) to notify their RemoteCache about state updates, and behaves like proto_message. pb.Cacheable also uses proto_decache and proto_uncache to manage reference counts of cached objects.

Finally, proto_decref is used to manage reference counts on RemoteReference objects. It is sent when the RemoteReference goes away, so that the holder of the original Referenceable can free that object.

Per-Connection ID Numbers

Each time a Referenceable is sent across the wire, its jellyFor method obtains a new unique local ID (luid) for it, which is a simple integer that refers to the original object. The Broker's .localObjects{} and .luids{} tables maintain the luid-to-object mapping. Only this ID number is sent to the remote system. On the other end, the object is unjellied into a RemoteReference object which remembers its Broker and the luid it refers to on the other end of the wire. Whenever callRemote() is used, it tells the Broker to send a message to the other end, including the luid value. Back in the original process, the luid is looked up in the table, turned into an object, and the named method is invoked.

A similar system is used with Cacheables: the first time one is sent, an ID number is allocated and recorded in the .remotelyCachedObjects{} table. The object's state (as returned by getStateToCacheAndObserveFor()) and this ID number are sent to the far end. That side uses .cachedLocallyAs() to find the local CachedCopy object, and tracks it in the Broker's .locallyCachedObjects{} table. (Note that to route state updates to the right place, the Broker on the CachedCopy side needs to know where it is. The same is not true of RemoteReferences: nothing is ever sent to a RemoteReference, so its Broker doesn't need to keep track of it).

Each remote method call gets a new requestID number. This number is used to link the request with the response. All pending requests are stored in the Broker's .waitingForAnswers{} table until they are completed by the receipt of a proto_answer or proto_error message.

The Broker also provides hooks to be run when the connection is lost. Holders of a RemoteReference can register a callback with .notifyOnDisconnect() to be run when the process which holds the original object goes away. Trying to invoke a remote method on a disconnected broker results in an immediate DeadReferenceError exception.

Reference Counting

The Broker on the Referenceable end of the connection needs to implement distributed reference counting. The fact that a remote end holds a RemoteReference should prevent the Referenceable from being freed. To accomplish this, The .localObjects{} table actually points at a wrapper object called pb.Local. This object holds a reference count in it that is incremented by one for each RemoteReference that points to the wrapped object. Each time a Broker serializes a Referenceable, that count goes up. Each time the distant RemoteReference goes away, the remote Broker sends a proto_decref message to the local Broker, and the count goes down. When the count hits zero, the Local is deleted, allowing the original Referenceable object to be released.

Security

Insecurity in network applications comes from many places. Most can be summarized as trusting the remote end to behave in a certain way. Applications or protocols that do not have a way to verify their assumptions may act unpredictably when the other end misbehaves; this may result in a crash or a remote compromise. One fundamental assumption that most RPC libraries make when unserializing data is that the same library is being used at the other end of the wire to generate that data. Developers put so much time into making their RPC libraries work at all that they usually assume their own code is the only thing that could possibly provide the input. A safer design is to assume that the input will almost always be corrupt, and to make sure that the program survives anyway.

Controlled Object serialization

Security is a primary design goal of PB. The receiver gets final say as to what they will and will not accept. The lowest-level serialization protocol (Banana) is simple enough to validate by inspection, and there are size limits imposed on the actual data received to prevent excessive memory consumption. Jelly is willing to accept basic data types (numbers, strings, lists and dictionaries of basic types) without question, as there is no dangerous code triggered by their creation, but Class instances are rigidly controlled. Only subclasses of the basic PB flavors (pb.Copyable, etc) can be passed over the wire, and these all provide the developer with ways to control what state is sent and accepted. Objects can keep private data on one end of the connection by simply not including it in the copied state.

Jelly's refusal to serialize objects that haven't been explicitly marked as copyable helps stop accidental security leaks. Seeing the pb.Copyable tag in the class definition is a flag to the developer that they need to be aware of what parts of the class will be available to a remote system and which parts are private. Classes without those tags are not an issue: the mere act of trying to export them will cause an exception. If Jelly tried to copy arbitrary classes, the security audit would have to look into every class in the system.

Controlled Object Unserialization

On the receiving side, the fact that Unjellying insists upon a user-registered class for each potential incoming instance reduces the risk that arbitrary code will be executed on behalf of remote clients. Only the classes that are added to the unjellyableRegistry need to be examined. Half of the security issues in RPC systems will boil down to the fact that these potential unserializing classes will have their setCopyableState methods called with a potentially hostile state argument. (the other half are that remote_ methods can be called with arbitrary arguments, including instances that have been sent to that client at some point since the current connection was established). If the system is prepared to handle that, it should be in good shape security-wise.

RPC systems which allow remote clients to create arbitrary objects in the local namespace are liable to be abused. Code gets run when objects are created, and generally the more interesting and useful the object, the more powerful the code that gets run during its creation. Such systems also have more assumptions that must be validated: code that expects to be given an object of class A so it can call A.foo could be given an object of class B instead, for which the foo method might do something drastically different. Validating the object is of the required type is much easier when the number of potential types is smaller.

Controlled Method Invocation

Objects which allow remote method invocation do not provide remote access to their attributes (pb.Referenceable and pb.Copyable are mutually exclusive). Remote users can only invoke a well-defined and clearly-marked subset of their methods: those with names that start with remote_ (or other specific prefixes depending upon the variant of Referenceable in use). This insures that they can have local methods which cannot be invoked remotely. Complete object transparency would make this very difficult: the translucent reference scheme allows objects some measure of privacy which can be used to implement a security model. The remote_ prefix makes all remotely-invokable methods easy to locate, improving the focus of a security audit.

Restricted Object Access

Objects sent by reference are indexed by a per-connection ID number, which is the only way for the remote end to refer back to that same object. This list means that the remote end can not touch objects that were not explicitly given to them, nor can they send back references to objects outside that list. This protects the program's memory space against the remote end: they cannot find other local objects to play with.

This philosophy of using simple, easy to validate identifiers (integers in the case of PB) that are scoped to a well-defined trust boundary (in this case the Broker and the one remote system it is connected to) leads to better security. Imagine a C system which sent pointers to the remote end and hoped it would receive back valid ones, and the kind of damage a malicious client could do. PB's .localObjects{} table insures that any given client can only refer to things that were given to them. It isn't even a question of validating the identifier they send: if it isn't a value of the .localObjects{} dictionary, they have no physical way to get at it. The worst they can do with a corrupt ObjectID is to cause a KeyError when it is not found, which will be trapped and reported back.

Size Limits

Banana limits string objects to 640k (because, as the source says, 640k is all you'll ever need). There is a helper class called pb.util.StringPager that uses a producer/consumer interface to break up the string into separate pages and send them one piece at a time. This also serves to reduce memory consumption: rather than serializing the entire string and holding it in RAM while waiting for the transmit buffers to drain, the pages are only serialized as there is space for them.

Future Directions

PB can currently be carried over TCP and SSL connections, and through UNIX-domain sockets. It is being extended to run over UDP datagrams and a work-in-progress reliable datagram protocol called airhook. (clearly this requires changes to the authorization sequence, as it must all be done in a single packet: it might require some kind of public-key signature).

At present, two functions are used to obtain the initial reference to a remote object: pb.getObjectAt and pb.connect. They take a variety of parameters to indicate where the remote process is listening, what kind of username/password should be used, and which exact object should be retrieved. This will be simplified into a PB URL syntax, making it possible to identify a remote object with a descriptive URL instead of a list of parameters.

Another research direction is to implement typed arguments: a way to annotate the method signature to indicate that certain arguments may only be instances of a certain class. Reminiscent of the E language, this would help remote methods improve their security, as the common code could take care of class verification.

Twisted provides a componentization mechanism to allow functionality to be split among multiple classes. A class can declare that all methods in a given list (the interface) are actually implemented by a companion class. Perspective Broker will be cleaned up to use this mechanism, making it easier to swap out parts of the protocol with different implementations.

Finally, a comprehensive security audit and some performance improvements to the Jelly design are also in the works.

Footnotes

1. Actually, PB currently jellies the list arguments in a separate tuple from the keyword arguments. This issue is currently being examined and may be changed in the future