I've spent some days writing something I call 'RemoteCall', a package of modules
for writing client/server applications in Python for a project I'm doing. The
package is still pretty early in the development phase, but I thought I might
ventilate my ideas and hear your objections to them.
The problem I am trying to solve:
I have a server process in python with some extension modules for in inhouse
library. I want to communicate with the process with a number of other python
scripts, possibly from a remote node. The scripts will call objects, as well as
create new objects in the server.
What I did so far:
RemoteCall a class for sending python calls across a socket
RemoteCallServer a class for receiving calls and execute them
Agent a class acting as a local representative of a remote object
To transfer objects over the wire I am using the new pickle module present
in the 1.2-beta release of python (excellent module!). However, some type of
objects are not really meant to be transferred over the wire, say a user
interface button object. That's what agents are for.
Agents are initialized with an object name, the node in which the object resides,
and at what port you can reach it. The agent then sends requests to the remote
node inquiring what methods the object supports and create local 'method objects'
correspondingly. The user can then use the remote object through the agent as if
it were a local object.
When a server returns a value, it can decide whether to return the object or an
agent over the net. This is done either on a per object or a per class basis.
If the return value is a structured value (tuple, list or dict), it is traversed
to find values that should be transferred as agents.
There are some problems with the way things are at the moment:
RemoteCall has a method called Exec, which allows arbitrary Python code to be
sent to the server. Probably not a very good idea if you are the slightest
nervous about security issues ;^).
When sending a tuple over the wire, there is quite a lot of copying behind the
scenes on the server side. When data is comming in, it is scanned for agents to
be substituted. Since tuples are immutable, they are always copied in the process
regardless of whether they contain agents or not. The same is true when returning
results. I see no easy way out here.
For those of you with a minute or two to spare :^), I appended the source code as
a shar archive. Any comments gratefully accepted.
------------------------------------------------------------------
Daniel Larsson email: dlarsson@sw.seisy.abb.se
ABB Industrial Systems AB phone: +46-21-34 30 29
S-721 67 Vaesteraas, Sweden fax : +46-21-34 25 55
--- CUT HERE ------------------------------------------------------------
# This is a shell archive. Remove anything before this line,
# then unpack it by saving it in a file and typing "sh file".
#
# Wrapped by Daniel Larsson <dlarsson@ws210> on Wed Mar 1 00:25:04 1995
#
# This archive contains:
# Agent.py AgentTest.py
# RemCallServTest.py RemCallServTestMain.py
# RemoteCall.py RemoteCallServer.py
# RemoteCalldefs.py
#
LANG=""; export LANG
PATH=/bin:/usr/bin:$PATH; export PATH
echo x - Agent.py
cat >Agent.py <<'@EOF'
"""Agents provide a simple mechanism to implement distributed applications.
An agent is a client side representative of a server object. Through the
agent it is possible to access attributes and execute methods on the server.
Communication with servers are done via sockets.
The server object can be any kind of python object having attributes (e.g.
instance, class, module).
Server side exceptions are reraised on the client side with the original
value, unless the exception value contains unsupported values (see below).
Normally, agents are created by clients to represent a known server object.
Sometimes, however, server methods return objects that should not be moved
to the client process, but rather stay in the server. In such cases it is
possible to ask the client to create an agent for the returned object instead.
The server then keeps a reference to the object to avoid garbage collection.
This reference is removed when the client removes his agent. The value to
return from the server in this case is an agent identification string which
contains a reference to the object in string form (the reference is prefixed
by a "magic" string to make it distinguishable from normal return values).
EXAMPLE
To implement an agent for the server object
class Server:
def __init__(self):
self.anAttr = 47
def method(sel, arg1, arg2):
....
server = Server()
you must create an agent for it:
from Agent import Agent
server = Agent('host.name.here', port)
# get an attribute
print server.anAttr
# Call a method
server.method('To be or not to be...', 'do be do be do')
CAVEAT
The types of values that can be transferred over the net is limited to:
None, integers, long integers, floating point numbers, strings, tuples,
lists, dictionaries and code objects. (the elements of tuples, lists and
dictionaries must of course also fall into above categories).
"""
from RemoteCall import RemoteCall
from types import *
error = 'Agent.error'
class Agent:
def __init__(self, remote=0, host=0, port=0, do_config=1):
"""Arguments:
remote name of the remote object
host name of host where remote object resides
port socket port of application holding remote object
byServer Agent created by server (only used internally)
"""
# Since this class has a '__setattr__' method, we must go
# via the '__dict__' attribute to fool python.
if remote:
if not (host and port):
raise error, 'You must give host and port of remote object'
self.__dict__['_remote_'] = remote
self.__dict__['_remcall_'] = RemoteCall(host, port)
self.__dict__['_byServer_'] = 0
if do_config:
self._config_()
else:
self.__dict__['_remote_'] = None
self.__dict__['_remcall_'] = None
def __repr__(self):
return self._remote_+' at %s:%d' % (self._remcall_._host_, self._remcall_._port_)
__str__ = __repr__
def __getinitargs__(self):
return ()
def __getstate__(self):
return self._remote_, self._remcall_
def __setstate__(self, state):
self.__dict__['_remote_'] = state[0]
self.__dict__['_remcall_'] = state[1]
self.__dict__['_byServer_'] = 1
def __getattr__(self, attr):
"""All variable read accesses (i.e. agent.attr) result in
a call to this function, unless 'attr' is found in
the agent object itself. This function will get 'attr'
from the remote server object."""
result = self._remcall_.Eval(self._remote_+'.'+attr)
self._config_agents_(result)
return result
def __setattr__(self, attr, value):
"""All variable write accesses (i.e. agent.attr = 4) result in
a call to this function, unless 'attr' is found in
the agent object itself. This function will set 'attr'
in the remote server object."""
return self._remcall_.Setattr(self._remote_, attr, value)
def remoteObject(self):
return self._remote_
def Remove(self):
"""Remove agent (i.e. remove all method objects and if the
agent was created by server, also remove the book keeping
reference to the server object from the server).
The reason we have an explicit remove service is because all
method objects have references to the agent so no garbage
collection will occur if we simply delete the reference to
the agent."""
if self._byServer_:
self._remcall_.Exec('del '+self._remote_)
for name in self.__dict__.keys():
object = self.__dict__[name]
if type(object) == InstanceType and object.__class__ == self.Method:
del self.__dict__[name]
class Method:
# This class represents a method on the agent.
def __init__(self, agent, method):
"""Arguments:
agent the agent instance the method shall use
method name of the method (string)"""
self._agent_ = agent
self._method_ = method
def __call__(self, *args):
"""This 'magic' method enables calling instances as if
they were functions. It will perform a remote procedure
call on the server object the agent is connected to."""
result = apply(self._agent_._remcall_.Call,
(self._agent_._remote_+'.'+self._method_,)+args)
self._agent_._config_agents_(result)
return result
# Private methods:
def _config_(self):
"""Tries to create Method objects (see below) to match the
connected server object's interface. This method checks
the server's __class__ attribute (if it has any) and the
object itself."""
try:
class_obj = self._remote_+'.__class__'
# Pick up the object's class' methods, if any
self._configobj_(class_obj)
# Pick up the object's class' bases' methods
self._config_bases_(class_obj)
except: pass
self._configobj_(self._remote_)
def _configobj_(self, object):
"""Tries to create Method objects (see below) to match "object"'s
interface. This method assumes the object has a '__dict__'
attribute."""
# Find all functions in 'object'
f_type = 'type('+object+'.__dict__[f])'
filter_fun = 'lambda f:'+f_type+' == FunctionType or '+ \
f_type+' == BuiltinFunctionType or '+ \
f_type+' == UnboundMethodType'
funs_cmd = 'filter('+filter_fun+','+object+'.__dict__.keys())'
# Send the filter function to server, but do not record
# the request (the last 0)
funs = self._remcall_.Eval(funs_cmd, 0)
# Create method objects for all attributes not starting with '_'
for fun in funs:
if fun[0] != '_':
self.__dict__[fun] = Agent.Method(self, fun)
def _config_bases_(self, class_obj):
"""Traverse the inheritance hierarchy of the class object
and pick up inherited methods."""
bases = self._remcall_.Eval('len('+class_obj+'.__bases__)', 0)
for base in range(bases):
# Add methods from base
base_obj = class_obj+'.__bases__['+`base`+']'
self._configobj_(base_obj)
# Add methods from base's bases
self._config_bases_(base_obj)
def _config_agents_(self, value):
"If 'value' contains agents, configure them."
valueType = type(value)
if valueType == ListType:
for item in value:
self._config_agents_(item)
elif valueType == DictionaryType:
for item in value.values():
self._config_agents_(item)
elif valueType == TupleType:
for item in value:
self._config_agents_(item)
elif hasattr(value, '__class__') and value.__class__ == Agent:
value._config_()
@EOF
chmod 644 Agent.py
echo x - AgentTest.py
cat >AgentTest.py <<'@EOF'
from Agent import Agent
import sys
port = eval(sys.argv[1])
print "Create agent"
server = Agent('a_server', '138.221.22.200', port)
obj = server.get()
print obj, obj.a, obj.b
#obj.a = 2
#obj.b = tuple(range(1000))
server.set(obj)
obj = server.get()
print obj.a, obj.b
@EOF
chmod 644 AgentTest.py
echo x - RemCallServTest.py
cat >RemCallServTest.py <<'@EOF'
from types import *
from RemoteCallServer import RemoteCallServer
class SmallClass:
def __init__(self):
self.a = 10
self.b = 20
class A_Server:
def __init__(self):
self.obj = SmallClass()
def get(self):
return self.obj
def set(self, obj):
self.obj = obj
a_server = A_Server()
class TestServer(RemoteCallServer):
def __init__(self, port):
RemoteCallServer.__init__(self, port)
self.EnableAgents()
self.SendAsAgent(a_server.obj)
def Apply(self, func, args):
# Called by RemoteCallServer.
# Derived to let remote client use objects
# in this namespace
return apply(func, args)
def Eval(self, expr):
# Called by RemoteCallServer.
# Derived to let remote client use objects
# in this namespace
return eval(expr)
def main():
RemoteCallServer.VERBOSE = 2
# By default, use system generated port
port = 0
# Create socket at port 'port' (or at system generated port
# if port == 0).
server = TestServer(port)
# Register server in Xt dispatcher
server.AddInput()
print 'OK, waiting on port', server.Port(), '...'
import Xt
Xt.MainLoop()
if __name__ == '__main__':
main()
@EOF
chmod 644 RemCallServTest.py
echo x - RemCallServTestMain.py
cat >RemCallServTestMain.py <<'@EOF'
#!/usr/local/bin/xpython
import RemCallServTest
RemCallServTest.main()
@EOF
chmod 644 RemCallServTestMain.py
echo x - RemoteCall.py
cat >RemoteCall.py <<'@EOF'
"""RemoteCall - Execute python statements in remote application.
DESCRIPTION
The RemoteCall is a class for sending calls to other Python applications.
Remote calls are one of
* arbitrary expression
* function call (i.e. anything you can 'apply' arguments on)
* attribute assignment (setattr(obj, attr, value))
Values transferred across the wire are marshalled using the 'pickle' module,
which means not only builtin data types (integers, lists, tuples, etc) can
be transferred, but also user defined classes.
Server side exceptions are reraised on the client side with the original
value, unless the exception value contains unsupported values (see below).
SEE ALSO
Agent.py - A framework for building agent-based applications.
"""
__version__ = '0.0'
from socket import socket, AF_INET, SOCK_STREAM
from RemoteCalldefs import *
error = 'RemoteCall.error'
class RemoteCall:
"See module documentation."
def __init__(self, host=0, port=0):
# Arguments:
# host name of remote host
# port socket port of remote application
self._host_ = host
self._port_ = port
def StartRecording(self, filename=None):
"""Tell server to start recording requests. The server will
record requests in a single file for all clients. If no
filename is given, store in file 'record<server-pid>.<seqno>.rec'.
If the server is already recording, no new recording will start
and 0 is returned (1 otherwise)."""
return self._rpc_(`(RECON, filename)`)
def StopRecording(self):
"Tell server to stop recording requests."
return self._rpc_(`(RECOFF, None)`)
def Exec(self, call, rec=RECORD):
"""Executes 'call', which is a string of python commands,
on a remote application. Exceptions in the remote application
due to the call are propagated to the caller.
'rec' controls whether this request should be recorded if the
server is recording requests. By default it is recorded, but
if you for some reason do not want it to be set 'rec' to 0.
"""
return self._rpc_(`(EXEC | rec, call)`)
def Call(self, call, *args):
"""Executes 'call', which is a string of python commands,
on a remote application. Exceptions in the remote application
due to the call are propagated to the caller.
'rec' controls whether this request should be recorded if the
server is recording requests. By default it is recorded, but
if you for some reason do not want it to be set 'rec' to 0.
"""
return unpicklify(self._rpc_(`(CALL | RECORD, (call, picklify(args)))`))
def Setattr(self, obj, attr, value, rec=RECORD):
"""Sets the attribute 'attr', in object 'obj' (which is a string
referring to a remote object), to 'value'. Exceptions in the remote
application due to the assignment are propagated to the caller.
'rec' controls whether this request should be recorded if the
server is recording requests. By default it is recorded, but
if you for some reason do not want it to be, set 'rec' to 0.
"""
return self._rpc_(`(SETATTR | rec, (obj, attr, picklify(value)))`)
def Eval(self, expr, rec=RECORD):
"""Executes 'expr', which is a python expression in string form,
in a remote application. The value of the expression is
returned to the caller. Exceptions in the remote application
due to the call are propagated to the caller.
'rec' controls whether this request should be recorded if the
server is recording requests. By default it is recorded, but
if you for some reason do not want it to be set 'rec' to 0.
"""
return unpicklify(self._rpc_(`(EVAL | rec, expr)`))
# Private methods:
def _rpc_(self, rpc):
sock = socket(AF_INET, SOCK_STREAM)
sock.connect(self._host_, self._port_)
sock.send(rpc)
try:
value = eval(sock.recv(10240))
except:
# Could not parse response
raise error, rpc+': Invalid return value'
# The result is returned as a 2-tuple, where
# the first element is None if everything
# went ok. In case of an error the first
# element is the exception name and the second
# the exception value.
if value[0]:
raise value[0], value[1]
else:
return value[1]
@EOF
chmod 644 RemoteCall.py
echo x - RemoteCallServer.py
cat >RemoteCallServer.py <<'@EOF'
"""RemoteCallServer - Handle remote calls from other python applications
DESCRIPTION
The RemoteCallServer is a class for handling remote calls from other Python
applications. Remote calls are one of
* arbitrary expression
* function call (i.e. anything you can 'apply' arguments on)
* attribute assignment (setattr(obj, attr, value))
Values transferred across the wire are marshalled using the 'pickle' module,
which means not only builtin data types (integers, lists, tuples, etc) can
be transferred, but also user defined classes.
After an incomming value is unpickled, but before it is further processed
(e.g. sent as arguments to a function) one can optionally process it with
a 'before' method. Likewise, before pickling the results and sending them back
an after method may massage it. The main reason for this facility is to
implement 'Agents'.
AGENTS
Agents are objects that acts as local representatives of remote objects.
Agents are also known in other systems as proxys, ambassadors, and other
names. Agents are useful for objects that use facilities local to the server
that might not be available at the client (e.g. they might interface to
an application with python embedded).
By installing agent support, and telling the RemoteCallServer object what
objects and/or classes not to send over the wire, all such objects appearing
in results from a RemoteCall will automatically be converted to agents.
An agent sent as an argument is likewise translated automatically to the
corresponding server object.
RECORDING
The RemoteCallServer can be instructed to record all incomming requests.
The recorded requests can later be replayed to repeat a sequence of calls.
Caveat
The recording mechanism will unfortunately not work with automatically
created agents, since their identity will most likely change from one time
to another.
PROTOCOL
A remote request is a 2-tuple where the first element indicates whether
the second element is to be considered an expression, a call, an assignment
or a record control request.
If the request is an expression or a call, it is evaluated and if the evaluation
succeeds the tuple '(None, result)' is returned. In the case an exception is
raised the tuple '(exc_type, exc_value)' is returned instead.
If the request is an assignment it is executed and if successfull the tuple
'(None, None)' is returned. In the case an exception is raised the tuple
'(exc_type, exc_value)' is returned instead.
NOTES
When enabling agent support, currently all transferred tuples will be copied,
whether they contain agents or not.
"""
__version__ = '$Revision: 1.10 $'
import posix
from types import *
from RemoteCalldefs import *
from Agent import Agent
class RemoteCallServer:
"""RemoteCallServer - Handle remote calls from other python applications.
Further documentation on module."""
_agent_refs_ = {}
# Verbose mode for printing trace and debug messages.
VERBOSE = 0
def __init__(self, port=0):
"""Arguments:
port port to listen for requests. If 0 is given
the system will generate a suitable port.
"""
from socket import socket, SOCK_STREAM, AF_INET, gethostname, gethostbyname
self._sock_ = socket(AF_INET, SOCK_STREAM)
self._addr_ = gethostbyname(gethostname())
self._sock_.bind(self._addr_, port)
self._sock_.listen(1)
# Get the actual port number (in case port arg was 0)
if port == 0:
port = self._sock_.getsockname()[1]
# Initialize members
self._port_ = port # socket port nr
self._recseqno_ = 1 # Sequence nr of record files
self._recfile_ = None # Current record file
self._makeagents_= [] # List of objects/classes which
# shouldn't be transferred over
# the wire, but send agents instead.
self._after_ = self._default_after_
self._before_ = self._default_before_
# --- Public methods ---
def Port(self):
"""Return the port number I am using."""
return self._port_
def SetBefore(self, before):
"Set user defined 'before' method"
self._before_ = before
def SetAfter(self, after):
"Set user defined 'after' method"
self._after_ = after
def EnableAgents(self):
self.SetAfter(self.MakeAgents)
self.SetBefore(self.ReplaceAgents)
def AddInput(self):
"""Register myself in Xt dispatcher."""
import Xt
from Xtdefs import XtInputReadMask
Xt.AddInput(self._sock_.fileno(), XtInputReadMask, self.ReceiveRequestCB, None)
def SendAsAgent(self, obj):
"""'obj' or instances of 'obj' should not be sent over the wire
"as is", but rather as agents (i.e. the object stays at the
server and the client gets a proxy object, here called agent)"""
self._makeagents_.append(obj)
def MakeAgents(self, result):
"""Replaces objects that should be sent as agents with a corresponding
agent.
This method is normally used internally when agent support
is enabled. You can use this method directly if you for some
reason do not want general agent support but only for some
selected cases."""
resultType = type(result)
if resultType == ListType:
for ix in range(len(result)):
result[ix] = self.MakeAgents(result[ix])
elif resultType == DictionaryType:
for ix in result.keys():
result[ix] = self.MakeAgents(result[ix])
elif resultType == TupleType:
new_tuple = ()
for ix in range(len(result)):
new_tuple = new_tuple + (self.MakeAgents(result[ix]),)
result = new_tuple
elif result in self._makeagents_ \
or (hasattr(result, '__class__') and result.__class__ in self._makeagents_):
RemoteCallServer._agent_refs_[id(result)] = result
agent = Agent('RemoteCallServer._agent_refs_['+`id(result)`+']', self._addr_, self._port_, 0)
result = agent
if self.VERBOSE > 1:
print 'Agent created :', result
return result
def ReplaceAgents(self, value):
"""Replaces agents in 'value' with the actual objects they
refer to.
This method is normally used internally when agent support
is enabled. You can use this method directly if you for some
reason do not want general agent support but only for some
selected cases."""
valueType = type(value)
if valueType == ListType:
for ix in range(len(value)):
value[ix] = self.ReplaceAgents(value[ix])
elif valueType == DictionaryType:
for ix in value.keys():
value[ix] = self.ReplaceAgents(value[ix])
elif valueType == TupleType:
new_tuple = ()
for ix in range(len(value)):
new_tuple = new_tuple + (self.ReplaceAgents(value[ix]),)
value = new_tuple
elif hasattr(value, '__class__') and value.__class__ == Agent:
value = self.Eval(value.remoteObject())
return value
# --- Evaluating and executing python code ---
def Eval(self, expr):
"""This is the function which must evaluate the expression sent
from a remote node. To be able to call application specific
functions it is necessary to inherit from this class and
override the eval method so that it executes in the right name
space. By default they execute here, which means almost nothing
is visible."""
return eval(expr)
def Exec(self, code):
"""This is the function which executes the code sent
from a remote node. To be able to call application specific
functions it is necessary to inherit from this class and
override the eval method so that it executes in the right name
space. By default they execute here, which means almost nothing
is visible."""
exec expr
def Apply(self, func, args):
"""This is the function which calls a function in the server
from a remote node. To be able to call application specific
functions it is necessary to inherit from this class and
override the apply method so that it executes in the right
name space. By default they execute here, which means almost
nothing is visible."""
return apply(func, args)
# --- Process incomming requests ---
def ReceiveRequestCB(self, summy, fd, id):
"""Callback to call when a message is received on the socket.
If 'AddInput' is used you don't have to bother with this
function. If, however you use some other synchronization
mechanism, this is the function to call to handle requests."""
conn, addr = self._sock_.accept()
if self.VERBOSE:
print 'Connected by ', addr
try:
mode, request = eval(conn.recv(10240, 0200000))
except:
import sys
result = sys.exc_type, sys.exc_value
else:
if self.VERBOSE > 1:
print 'Received :', request
if 1:
# Default return value
result = (None, None)
if mode & 0xFF == CALL:
method, args = request
result = None, self._call_(method, args)
elif mode & 0xFF == EVAL:
result = None, self._eval_expr_(request)
elif mode & 0xFF == EXEC:
result = None, self.Exec(request)
elif mode & 0xFF == SETATTR:
obj, attr, value = request
result = None, self._setattr_(obj, attr, value)
elif mode & 0xFF == RECON:
result = None, self._startRecording_(request)
elif mode & 0xFF == RECOFF:
self._stopRecording_()
else:
raise 'Mode error', mode
# If I am currently recording, and this request
# is recordable (i.e. the RECORD bit is set) save
# request to record file.
if self._recfile_ and mode & RECORD:
if self.VERBOSE:
print 'Record request'
self._recfile_.write(`mode, request`+'\n')
else:
import sys
result = sys.exc_type, sys.exc_value
if self.VERBOSE > 1:
print 'Result :', result
conn.send(`result`)
conn.close()
if self.VERBOSE:
print 'OK, wait for next...'
# --- Private methods ---
def _call_(self, method, args):
m = self.Eval(method)
return picklify(self._after_(self.Apply(m, self._before_(unpicklify(args)))))
def _eval_expr_(self, expr):
return picklify(self._after_(self.Eval(expr)))
def _setattr_(self, obj, attr, value):
return picklify(setattr(self.Eval(obj), attr, self._before_(unpicklify(value))))
def _startRecording_(self, filename=None):
"""Start recording incomming requests to file 'filename'. If no
filename is given, store in file 'record<server-pid>.<seqno>.rec'.
If the server is already recording, no new recording will start.
"""
# Am I already recording? If so don't start a new one.
if self._recfile_:
return 0
if not filename:
pid = posix.getpid()
filename='record'+`pid`+'.'+`self._recseqno_`+'.rec'
self._recseqno_ = self._recseqno_ + 1
self._recfile_ = open(filename, 'w')
self._recfile_.write(`self._addr_, self._port_`+'\n')
if self.VERBOSE:
print 'Start recording in', filename
return 1
def _stopRecording_(self):
"""Stop recording incomming requests."""
self._recfile_.close()
self._recfile_ = None
def _default_before_(self, value):
return value
_default_after_ = _default_before_
@EOF
chmod 644 RemoteCallServer.py
echo x - RemoteCalldefs.py
cat >RemoteCalldefs.py <<'@EOF'
"""Definitions common for both clients and servers"""
# Mode to execute request in on server side
EXEC = 0
EVAL = 1
CALL = 2
SETATTR= 3
# Special modes to turn on and off recording
RECON = 2
RECOFF = 3
# Magic bit in modes to indicate whether to record this
# request or not when recording is enabled (not all requests
# are recorded even if recording is enabled, notably the
# agent configuration requests)
RECORD = (1 << 8)
def picklify(objs):
from StringIO import StringIO
from pickle import Pickler
sio = StringIO()
pickler = Pickler(sio)
pickler.dump(objs)
return sio.getvalue()
def unpicklify(pickled_objs):
from StringIO import StringIO
from pickle import Unpickler
sio = StringIO(pickled_objs)
unp = Unpickler(sio)
return unp.load()
@EOF
chmod 644 RemoteCalldefs.py
exit 0