Changes since the original Python
2.2 version of this tutorial:
- Don't scare people by suggesting classmethod may go away. (4-Apr-2002)
Python 2.2 introduces the first phase of "type/class unification".
This is a series of changes to Python intended to remove most of the
differences between built-in types and user-defined classes. Perhaps
the most obvious one is the restriction against using built-in types
(such as the type of lists and dictionaries) as a base class in a
class statement. Andrew Kuchling's Python Warts
paper devotes one of its longest sections to this language deficiency.
Incidentally, Andrew's paper What's New in Python 2.2 gives a broad overview of what else is
new in Python 2.2.
This is one of the biggest changes to Python ever, and yet it can
be done with very few backwards incompatibilities. The changes are
described in minute detail in a series of PEPs (Python Enhancement Proposals). PEPs are not designed to be
tutorials, and the PEPs describing the type/class unification are
sometimes hard to read. They also aren't finished yet. That's where
this paper comes in: it introduces the key elements of the type/class
unification for the average Python programmer.
A bit of terminology: "classic Python" refers to Python 2.1 (and
its patch releases such as 2.1.1) or
earlier versions, while "classic classes" refer to classes defined
with a class statement that does not have a built-in object amongst
its bases: either because it has no bases, or because all of its bases
are classic classes themselves - applying the definition recursively.
Classic classes are still a special category in Python 2.2.
Eventually they will be totally unified with types, but because of
additional backwards incompatibilities, this will be done after 2.2 is
released (maybe not before Python 3.0). I'll try to say "type" when I
mean a built-in type, and "class" when I'm referring to a classic
class or something that could be either; if it wouldn't be clear from
the context which interpretation is meant, I'll try to be explicit,
using "classic class" or "class or type".
Let's start with the juiciest bit: you can subtype built-in types
like dictionaries and lists. All you need is a name for a base class
that is a built-in type and you're in business.
There's a new built-in name, "dict", for the type of dictionaries.
(In version 2.2b1 and before, this was called "dictionary"; while in
general I don't like abbreviations, "dictionary" was just too long to
type, and we've been saying "dict" for years.)
This is really just sugar, since there are already two other ways to
name this type: type({}) and (after importing the types
module) types.DictType (and a third, types.DictionaryType). But now
that types play a more central role, it seems appropriate to have
built-in names for the types that you're likely to encounter.
Here's an example of a simple dict subclass, which provides a
"default value" that is returned when a missing key is requested:
class defaultdict(dict):
def __init__(self, default=None):
dict.__init__(self)
self.default = default
def __getitem__(self, key):
try:
return dict.__getitem__(self, key)
except KeyError:
return self.default
This example shows a few things. The __init__() method extends the
dict.__init__() method. Like __init__() methods are wont to do,
it has a different argument list than the base class __init__()
method. Likewise, the __getitem__() method extends the base class
__getitem__() method.
The __getitem__() method could also be written as follows, using
the new "key in dict" test introduced in Python 2.2:
def __getitem__(self, key):
if key in self:
return dict.__getitem__(self, key)
else:
return self.default
I believe that this version is less efficient, because it does the
key lookup twice. The exception would be when we expect that the
requested key is almost never in the dictionary: then setting up the
try/except statement is more expensive than the failing "key in self"
test.
To be complete, the get() method should probably also be
extended, to make it use the same default as __getitem__():
def get(self, key, *args):
if not args:
args = (self.default,)
return dict.get(self, key, *args)
(Although this method is declared with a variable-length argument
list, it really should only be called with one or two arguments; if
more are passed, the base class method call will raise a TypeError
exception.)
We're not restricted to extending methods defined on the
base class. Here's a useful method that does something similar to
update(), but keeps existing values rather than overwriting them with
new values if a key exists in both dictionaries:
def merge(self, other):
for key in other:
if key not in self:
self[key] = other[key]
This uses the new "key not in dict" test as well as the new "for
key in dict:" to iterate efficiently (without making a copy of the
list of keys) over all keys in a dictionary. It doesn't require the
other argument to be a defaultdict or even a dictionary: any mapping
object that supports "for key in other" and other[key] will do.
Here's the new type at work:
>>> print defaultdict # show our type
<class '__main__.defaultdict'>
>>> print type(defaultdict) # its metatype
<type 'type'>
>>> a = defaultdict(default=0.0) # create an instance
>>> print a # show the instance
{}
>>> print type(a) # show its type
<class '__main__.defaultdict'>
>>> print a.__class__ # show its class
<class '__main__.defaultdict'>
>>> print type(a) is a.__class__ # its type is its class
1
>>> a[1] = 3.25 # modify the instance
>>> print a # show the new value
{1: 3.25}
>>> print a[1] # show the new item
3.25
>>> print a[0] # a non-existant item
0.0
>>> a.merge({1:100, 2:200}) # use a dictionary method
>>> print a # show the result
{1: 3.25, 2: 200}
>>>
We can also use the new type in contexts where classic only allows
"real" dictionaries, such as the locals/globals dictionaries for the
exec statement or the built-in function eval():
>>> print a.keys()
[1, 2]
>>> exec "x = 3; print x" in a
3
>>> print a.keys()
['__builtins__', 1, 2, 'x']
>>> print a['x']
3
>>>
However, our __getitem__() method is not used for variable access
by the interpreter:
>>> exec "print foo" in a
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<string>", line 1, in ?
NameError: name 'foo' is not defined
>>>
Why doesn't this print 0.0? The interpreter uses an internal
function to access the dictionary, which bypasses our __getitem__()
override. I admit that this can be a problem (although it is
only a problem in this context, when a dict subclass is
used as a locals/globals dictionary); it remains to be seen if I can
fix this without compromising performance in the common case.
Now we'll see that defaultdict instances have dynamic instance
variables, just like classic classes:
>>> a.default = -1
>>> print a["noway"]
-1
>>> a.default = -1000
>>> print a["noway"]
-1000
>>> print a.__dict__.keys()
['default']
>>> a.x1 = 100
>>> a.x2 = 200
>>> print a.x1
100
>>> print a.__dict__.keys()
['default', 'x2', 'x1']
>>> print a.__dict__
{'default': -1000, 'x2': 200, 'x1': 100}
>>>
This is not always what you want; in particular, using a separate
dictionary to hold a single instance variable doubles the memory used
by a defaultdict instance compared to using a regular dictionary!
There's a way to avoid this:
class defaultdict2(dict):
__slots__ = ['default']
def __init__(self, default=None):
...(like before)...
The __slots__ declaration takes a list of instance variables, and
reserves space in the instance for exactly these in the instance.
When __slots__ is used, other instance variables cannot be assigned
to:
>>> a = defaultdict2(default=0.0)
>>> a[1]
0.0
>>> a.default = -1
>>> a[1]
-1
>>> a.x1 = 1
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'defaultdict2' object has no attribute 'x1'
>>>
Some noteworthy tidbits and warnings about __slots__:
- An undefined slot variable will raise AttributeError as
expected. (Note that in Python 2.2b2 and earlier, slot variables had
the value None by default, and "deleting" them restores this default
value.)
- You cannot use a class attribute to define a default value for
an instance variable defined by __slots__. The __slots__ declaration
creates a class attribute containing a descriptor for each slot,
and setting a class attribute to a default value would overwrite this
descriptor.
- There's no check to prevent name conflicts between the slots
defined in a class and the slots defined in its base classes. If a
class defines a slot that's also defined in a base class, the instance
variable defined by the base class slot is inaccessible (except by
retrieving its descriptor directly from the base class; this could be
used to rename it). Doing this renders the meaning of your program
undefined; a check to prevent this may be added in the future.
- Instances of a class that uses __slots__ don't have a __dict__
(unless a base class defines a __dict__); but instances of derived
classes of it do have a __dict__, unless their class also uses
__slots__.
- You can define an object with no instance variables and no
__dict__ by using __slots__ = [].
- You cannot use slots with "variable-length" built-in types as
base class. Variable-length built-in types are long, str and tuple.
- A class using __slots__ does not support weak references to its
instances, unless one of the strings in the __slots__ list equals
"__weakref__". (In Python 2.3, this feature has been extended to
"__dict__")
- The __slots__ variable doesn't have to be a list; any non-string that
can be iterated over will do, and the values returned by the iteration
are used as the slot names. In particular, a dictionary can be used.
You can also use a single string, to declare a single slot.
However, in the future, an additional meaning may be assigned to using
a dictionary, for example, the dictionary values may be used to
restrict the type of an instance variable or provide a doc string;
the effect of using something that's not a list renders the meaning of
your program undefined.
Note that while in general operator overloading works just as for
classic classes, there are some differences. (The biggest one is the
lack of support for __coerce__; new-style classes should always use
the new-style numeric API, which passes the other operand uncoerced to
the __add__ and __radd__ methods, etc.)
There's a new way of overriding attribute access. The __getattr__
hook, if defined, works the same way as it does for classic classes:
it is only called if the regular way of searching for the
attribute doesn't find it. But you can now also override
__getattribute__, a new operation that is called for all
attribute references.
When overriding __getattribute__, bear in mind that it is easy to
cause infinite recursion: whenever __getattribute__ references an
attribute of self (even self.__dict__!), it is called recursively.
(This is similar to __setattr__, which gets called for all attribute
assignments; __getattr__ can also suffer from this when it is
carelessly written and references a non-existent attribute of self.)
The correct way to get any attribute from self inside
__getattribute__ is to call the base class's __getattribute__ method,
in the same way any method that overrides a base class method can call
the base class method: Base.__getattribute__(self, name). (See also
the discussion of super() below if you want
to be correct in a multiple inheritance world.)
Here's an example of overriding __getattribute__ (really extending
it, since the overriding method calls the base class method):
class C(object):
def __getattribute__(self, name):
print "accessing %r.%s" % (self, name)
return object.__getattribute__(self, name)
A note about __setattr__: sometimes attributes are not stored in
self.__dict__ (for example when using __slots__ or properties, or when
using a built-in base class). The
same pattern as for __getattribute__ applies, where you call the base
class __setattr__ to do the actual work. Here's an example:
class C(object):
def __setattr__(self, name, value):
if hasattr(self, name):
raise AttributeError, "attributes are write-once"
object.__setattr__(self, name, value)
C++ programmers may find it useful to realize that this form of
subtyping in Python is implemented very similarly to
single-inheritance subclassing in C++, with __class__ in the role of
the vtable.
There's much more that could be explained (like the __metaclass__
declaration, and the __new__ method), but most of that is pretty
esoteric. See below if you're interested.
I'll end with a list of caveats:
- You can use multiple inheritance, but you can't multiply
inherit from different built-in types (for example, you can't create a
type that inherits from both the built-in dict and list
types). This is a permanent restriction; it would require too many
changes to Python's object implementation to lift
it. However, you can create mix-in classes by inheriting from
"object". This is a new built-in, naming the featureless base type of
all built-in types under the new system.
- When using multiple inheritance, you can mix classic classes
and built-in types (or types derived from built-in types) in the list
of base classes. (This is new in Python 2.2b2; in earlier versions
you couldn't.)
- See also the general bugs in 2.2 list.
The previous section showed that an instance of the built-in subtype
defaultdict can be created by calling defaultdict(). This is
expected, because this also works for classic classes. But here's a
new feature: built-in base types themselves can also be instantiated
by calling the type directly.
For several built-in types, there are already factory functions
named after the type in classic Python, for example str() and int().
I've changed these built-ins so that they are now names for the
corresponding types. While this changes the type of these names from
built-in function to built-in type, I don't expect that this will create
backward compatibility problems: I've made sure that the types can be
called with exactly the same argument lists as the former functions.
(They can also generally be called without arguments, producing an
object with a suitable default value, such as zero or empty; this is
new.)
These are the affected built-ins:
- int([number_or_string[, base_number]])
- long([number_or_string])
- float([number_or_string])
- complex([number_or_string[, imag_number]])
- str([object])
- unicode([string[, encoding_string]])
- tuple([iterable])
- list([iterable])
- type(object) or type(name_string, bases_tuple, methods_dict)
The signature of type() requires an explanation: traditionally,
type(x) returns the type of object x, and this usage is still
supported. However, type(name, bases, methods) is a new usage that
creates a brand new type object. (This gets into metaclass programming, and I won't go into
this further here except to note that this signature is the same as
that used by the Don Beaudry hook of metaclass fame.)
There are also a few new built-ins that follow the same pattern.
These have been described above or will be described below:
- dict([mapping_or_iterable]) - return a new dictionary; the
optional argument must be either a mapping whose items are copied, or
a sequence of 2-tuples (or of sequences of length 2) giving the (key,
value) pairs to be inserted into the new dictionary
- object([...]) - return a new featureless object; arguments are
ignored
- classmethod(function) - see below
- staticmethod(function) - see below
- super(class_or_type[, instance]) - see below
- property([fget[, fset[, fdel[, doc]]]]) - see below
The purpose of this change is twofold. First, this makes it
convenient to use any of these types as a base class in a class
statement. Second, it makes testing for a specific type a little
easier: rather than writing type(x) is type(0), you can now write
isinstance(x, int).
Which reminds me. The second argument of isinstance() may now be a
tuple of classes or types. For example, isinstance(x, (int, long))
returns true when x is an int or a long (or an instance of a subclass
of either of those types), and similarly isinstance(x, (str, unicode))
tests for a string of either variety. We didn't do this to issubclass().
(Yet. It was done to issubclass() in Python 2.3.)
For instances of built-in types (and for new-style classes in
general), x.__class__ is now the same as type(x):
>>> type([])
<type 'list'>
>>> [].__class__
<type 'list'>
>>> list
<type 'list'>
>>> isinstance([], list)
1
>>> isinstance([], dict)
0
>>> isinstance([], object)
1
>>>
In classic Python, the method names of lists were available as the
__methods__ attribute of list objects, with the same effect as using
the built-in dir() function:
Python 2.1 (#30, Apr 18 2001, 00:47:18)
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> [].__methods__
['append', 'count', 'extend', 'index', 'insert', 'pop',
'remove', 'reverse', 'sort']
>>>
>>> dir([])
['append', 'count', 'extend', 'index', 'insert', 'pop',
'remove', 'reverse', 'sort']
Under the new proposal, the __methods__ attribute no longer exists:
Python 2.2c1 (#803, Dec 13 2001, 23:06:05)
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> [].__methods__
Traceback (most recent call last):
File "<stdin>", line 1, in ?
AttributeError: 'list' object has no attribute '__methods__'
>>>
Instead, you can get the same information from the dir() function,
which gives more information:
>>> dir([])
['__add__', '__class__', '__contains__', '__delattr__',
'__delitem__', '__eq__', '__ge__', '__getattribute__',
'__getitem__', '__getslice__', '__gt__', '__hash__', '__iadd__',
'__imul__', '__init__', '__le__', '__len__', '__lt__', '__mul__',
'__ne__', '__new__', '__reduce__', '__repr__', '__rmul__',
'__setattr__', '__setitem__', '__setslice__', '__str__', 'append',
'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse',
'sort']
>>>
The new dir() gives more information than the old one: in addition
to the names of instance variables and regular methods, it also shows
the methods that are normally invoked through special notations, like
__iadd__ (+=), __len__ (len), __ne__ (!=).
More about the new dir() function:
- dir() on an instance (classic or new-style) shows the instance
variables as well as the methods and class attributes defined by the
instance's class and all its base classes.
- dir() on a class (classic or new-style) shows the contents of
the __dict__ of the class and all its base classes. It does not show
class attributes that are defined by a metaclass.
- dir() on a module shows the contents of the module's __dict__.
(This is unchanged.)
- dir() without arguments shows the caller's local variables.
(Again, unchanged.)
- There's a new C API that implements the dir() function:
PyObject_Dir().
- There are more details; in particular, for objects that
override __dict__ or __class__, these are honored, and for backwards
compatibility, __members__ and __methods__ are honored if they are
defined.
You can use a method of a built-in type as an "unbound method":
>>> a = ['tic', 'tac']
>>> list.__len__(a) # same as len(a)
2
>>> list.append(a, 'toe') # same as a.append('toe')
>>> a
['tic', 'tac', 'toe']
>>>
This is just like using an unbound method of a user-defined class
- and similarly, it's mostly useful from inside a subclass
method, to call the corresponding base class method.
Unlike user-defined classes, you cannot change built-in types:
attempts to assign an attribute of a built-in type raises a TypeError,
and their __dict__ is a read-only proxy object. The restriction on
attribute assignment is lifted for new-style user-defined classes,
including subclasses of built-in types; however even those have a
read-only __dict__ proxy, and you must use attribute assignment to
replace or add a method of a new-style class. Example session:
>>> list.append
<method 'append' of 'list' objects>
>>> list.append = list.append
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: can't set attributes of built-in/extension type 'list'
>>> list.answer = 42
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: can't set attributes of built-in/extension type 'list'
>>> list.__dict__['append']
<method 'append' of 'list' objects>
>>> list.__dict__['answer'] = 42
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: object does not support item assignment
>>> class L(list):
... pass
...
>>> L.append = list.append
>>> L.answer = 42
>>> L.__dict__['answer']
42
>>> L.__dict__['answer'] = 42
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: object does not support item assignment
>>>
For the curious: there are two reasons why changing built-in
classes is disallowed.
First, it would be too easy to break an invariant of a
built-in type that is relied upon elsewhere, either by the standard
library, or by the run-time code. Second, when Python is embedded in
another application that creates multiple Python interpreters, the
built-in class objects (being statically allocated data structures)
are shared between all interpreters; thus, code running in one
interpreter might wreak havoc on another interpreter, which is a
no-no.
The new descriptor API makes it possible to add static methods
and class methods. Static methods are easy to describe: they behave
pretty much like static methods in C++ or Java. Here's an example:
class C:
def foo(x, y):
print "staticmethod", x, y
foo = staticmethod(foo)
C.foo(1, 2)
c = C()
c.foo(1, 2)
Both the call C.foo(1, 2) and the call c.foo(1, 2) call foo() with
two arguments, and print "staticmethod 1 2". No "self" is declared in
the definition of foo(), and no instance is required in the call. If
an instance is used, it is only used to find the class that defines
the static method. This works for classic and new classes!
The line "foo = staticmethod(foo)" in the class statement is the
crucial element: this makes foo() a static method. The built-in
staticmethod() wraps its function argument in a special kind of
descriptor whose __get__() method returns the original function
unchanged.
More on __get__ methods: in Python 2.2, the magic of binding
methods to instances (even for classic classes!) is done through the
__get__ method of the object found in the class. The __get__ method
for regular function objects returns a bound method object; the
__get__ method for staticmethod objects returns the underlying
function. If a class attribute has no __get__ method, it is never
bound to an instance, or in other words there's a default __get__
operation that returns the object unchanged; this is how simple class
variables (for example numerical values) are handled.
Class methods use a similar pattern to declare methods that receive
an implicit first argument that is the class for which they are
invoked. This has no C++ or Java equivalent, and is not quite the
same as what class methods are in Smalltalk, but may serve a similar
purpose. (Python also has real metaclasses, and perhaps methods defined in a metaclass have more
right to the name "class method"; but I expect that most programmers
won't be using metaclasses.) Here's an example:
class C:
def foo(cls, y):
print "classmethod", cls, y
foo = classmethod(foo)
C.foo(1)
c = C()
c.foo(1)
Both the call C.foo(1) and the call c.foo(1) end up calling foo()
with two arguments, and print "classmethod __main__.C 1". The first
argument of foo() is implied, and it is the class, even if the method
was invoked via an instance. Now let's continue the example:
class D(C):
pass
D.foo(1)
d = D()
d.foo(1)
This prints "classmethod __main__.D 1" both times; in other words,
the class passed as the first argument of foo() is the class involved
in the call, not the class involved in the definition of foo().
But notice this:
class E(C):
def foo(cls, y): # override C.foo
print "E.foo() called"
C.foo(y)
foo = classmethod(foo)
E.foo(1)
e = E()
e.foo(1)
In this example, the call to C.foo() from E.foo() will see class C
as its first argument, not class E. This is to be expected, since the
call specifies the class C. But it stresses the difference between
these class methods and methods defined in metaclasses, where an upcall to a metamethod would pass the
target class as an explicit first argument. (If you don't understand
this, don't worry, you're not alone. :-)
Properties are a neat way to implement attributes whose
usage resembles attribute access, but whose
implementation uses method calls. These are sometimes known
as "managed attributes". In prior Python versions, you could only do
this by overriding __getattr__ and __setattr__; but overriding
__setattr__ slows down all attribute assignments considerably,
and overriding __getattr__ is always a bit tricky to get right.
Properties let you do this painlessly, without having to override
__getattr__ or __setattr__.
I'll show an example first. Let's define a class with an attribute
x defined by a pair of methods, getx() and setx():
class C(object):
def __init__(self):
self.__x = 0
def getx(self):
return self.__x
def setx(self, x):
if x < 0: x = 0
self.__x = x
x = property(getx, setx)
Here's a small demonstration:
>>> a = C()
>>> a.x = 10
>>> print a.x
10
>>> a.x = -10
>>> print a.x
0
>>> a.setx(12)
>>> print a.getx()
12
>>>
The full signature is property(fget=None, fset=None, fdel=None,
doc=None). The fget, fset and fdel arguments are the methods called
when the attribute is get, set or deleted. If any of these three is
unspecified or None, the corresponding operation will raise an
AttributeError exception. The fourth argument is the doc string for
the attribute; it can be retrieved from the class as the following
example shows:
>>> class C(object):
... def getx(self): return 42
... x = property(getx, doc="hello")
...
>>> C.x.__doc__
'hello'
>>>
Things to notice about property() (all advanced material except the
first one):
- Properties do not work for classic classes, but you
don't get a clear error when you try this. Your get method will be
called, so it appears to work, but upon attribute assignment, a
classic class instance will simply set the value in its __dict__
without calling the property's set method, and after that, the
property's get method won't be called either. (You could override
__setattr__ to fix this, but it would be prohibitively expensive.)
- As far as property() is concerned, its fget, fset and fdel
arguments are functions, not methods - they are passed an explicit
reference to the object as their first argument. Since property() is
typically used in a class statement, this is correct (the methods
really are function objects at the time when property() is
called) but you can still think of them as methods - as long as you
aren't using a metaclass that does special
things to methods.
- The get method won't be called when the property is accessed as
a class attribute (C.x) instead of as an instance attribute (C().x).
If you want to override the __get__ operation for properties when used
as a class attribute, you can subclass property - it is a new-style
type itself - to extend its __get__ method, or you can define a
descriptor type from scratch by creating a new-style class that
defines __get__, __set__ and __delete__ methods.
With multiple inheritance comes the question of method resolution
order: the order in which a class and its bases are searched
looking for a method of a given name.
In classic Python, the rule is given by the following recursive
function, also known as the left-to-right depth-first rule:
def classic_lookup(cls, name):
"Look up name in cls and its base classes."
if cls.__dict__.has_key(name):
return cls.__dict__[name]
for base in cls.__bases__:
try:
return classic_lookup(base, name)
except AttributeError:
pass
raise AttributeError, name
In Python 2.2, I've decided to adopt a different lookup rule for
new-style classes. (The rule for classic classes remains unchanged
for backwards compatibility considerations; eventually all classes
will be new-style classes and then the distinction will go away.) I'll
try to explain what's wrong with the classic rule first.
The problem with the classic rule becomes apparent when we consider
a "diamond diagram". In code:
class A:
def save(self): ...
class B(A):
...
class C(A):
def save(self): ...
class D(B, C):
...
Or as a diagram with arrows representing subclassing relationships
(explaining the name):
class A:
^ ^ def save(self): ...
/ \
/ \
/ \
/ \
class B class C:
^ ^ def save(self): ...
\ /
\ /
\ /
\ /
class D
Arrows point from a subtype to its base type(s). This particular
diagram means B and C derive from A, and D derives from B and C (and
hence also, indirectly, from A).
Assume that C overrides the method save(), which is defined in the
base A. (C.save() probably calls A.save() and then saves some of its
own state.) B and D don't override save(). When we invoke save() on
a D instance, which method is called? According to the classic lookup
rule, A.save() is called, ignoring C.save()!
This is not good. It probably breaks C (its state doesn't get
saved), defeating the whole purpose of inheriting from C in the first
place.
Why wasn't this a problem in classic Python? Diamond diagrams are
rarely found in classic Python class hierarchies. Most class
hierarchies use single inheritance, and multiple inheritance is
usually limited to mix-in classes. In fact, the problem shown here
is probably the reason why multiple inheritance is unpopular in
classic Python!
Why will this be a problem in the new system? The 'object' type at
the top of the type hierarchy defines a number of methods that can
usefully be extended by subtypes, for example __getattribute__() and
__setattr__().
(Aside: the __getattr__() method is not really the implementation
for the get-attribute operation; it is a hook that only gets invoked
when an attribute cannot be found by normal means. This has often
been cited as a shortcoming - some class designs have a legitimate
need for a get-attribute method that gets called for all
attribute references, and this problem is solved now by making
__getattribute__() available. But then this method has to
be able to invoke the default implementation somehow. The most
natural way is to make the default implementation available as
object.__getattribute__(self, name).)
Thus, a classic class hierarchy like this:
class B class C:
^ ^ __setattr__()
\ /
\ /
\ /
\ /
class D
will change into a diamond diagram under the new system:
object:
^ ^ __setattr__()
/ \
/ \
/ \
/ \
class B class C:
^ ^ __setattr__()
\ /
\ /
\ /
\ /
class D
and while in the original diagram C.__setattr__() is invoked, under
the new system with the classic lookup rule, object.__setattr__()
would be invoked!
Fortunately, there's a lookup rule that's better. It's a bit
difficult to explain, but it does the right thing in the diamond
diagram, and it is the same as the classic lookup rule when there are
no diamonds in the inheritance graph (when it is a tree).
The new lookup rule constructs a list of all classes in the
inheritance diagram in the order in which they will be searched. This
construction is done when the class is defined, to save time. To
explain the new lookup rule, let's first consider what such a list
would look like for the classic lookup rule. Note that in the
presence of diamonds the classic lookup visits some classes multiple
times. For example, in the ABCD diamond diagram above, the classic
lookup rule visits the classes in this order:
D, B, A, C, A
Note how A occurs twice in the list. The second occurrence is
redundant, since anything that could be found there would already have
been found when searching the first occurrence. But it is visited
nonetheless (the recursive implementation of the classic rule doesn't
remember which classes it has already visited).
Under the new rule, the list will be
D, B, C, A
Searching for methods in this order will do the right thing for the
diamond diagram. Because of the way the list is constructed, it
never changes the search order in situations where no diamond is
involved.
The exact rule used will be explained in the next section (which
refers to a separate paper for the most subtle details). I note
here only the important property of monotonicity in the lookup
rule: if class X precedes class Y in the lookup order for any of the
base classes of class D, then class X will also precede class Y in the
lookup order for class D. For example, since B precedes A in the
lookup list for B, it also precedes A in the lookup list for D; and
ditto for C preceding A. Exception: if, amongst the bases of class D,
there is one where X precedes Y and another where Y precedes X, the
algorithm has to break a tie. In this case, all bets are off; in the
future, this condition may cause a warning or an error.
(A rule previously described at this place was proven not to have
the monotonicity property. See a thread on python-dev started by Samuele Pedroni.)
Isn't this backwards incompatible? Won't it break existing code?
It would, if we changed the method resolution order for all classes.
However, in Python 2.2, the new lookup rule will only be applied to
types derived from built-in types, which is a new feature. Class
statements without a base class create "classic classes", and so do
class statements whose base classes are themselves classic classes.
For classic classes the classic lookup rule will be used.
We may also provide
a tool that analyzes a class hierarchy looking for methods that would
be affected by a change in method resolution order.
(This section is for advanced readers only.)
Any algorithm for deciding the method resolution order may be
confronted with contradicting requirements. This shows up for
example when two given base classes occur in a different order in the
inheritance list of two different derived classes, and those derived
classes are both inherited by yet another class. Here's an example:
class A(object):
def meth(self): return "A"
class B(object):
def meth(self): return "B"
class X(A, B): pass
class Y(B, A): pass
class Z(X, Y): pass
If you try this, (using Z.__mro__, see below), you get [Z, X, Y, A, B, object], which
does not maintain the monotonicity requirement mentioned above: the
MRO for Y is [Y, B, A, object], and this is not a subsequence of the
above list! In fact, there is no solution that satisfies the
monotonicity requirement for both X and Y here. This is called an
order disagreement. In a future version, we may decide to
outlaw such order disagreements under certain circumstances, or issue
warnings for them.
The book "Putting Metaclasses to Work",
which inspired me to change the MRO, defines the MRO algorithm that's
currently implemented, but its description of the algorithm is pretty
hard to grasp - I had originally documented a different, naive,
algorithm and didn't even realize that it didn't always compute the
same MRO until Tim Peters
found a counterexample. More recently, Samuele Pedroni has found a
counterexample showing that the naive algorithm fails to maintain
monotonicity, so I won't even describe it any more. Samuele has
convinced me to use a newer MRO algorithm named C3, described in the
paper "A Monotonic Superclass Linearization for
Dylan". This algorithm will be used in Python 2.3. C3 is
monotonic just like the book's algorithm, but in addition maintains
the order of the immediate base classes, which the book's algorithm
doesn't always do. A very accessible description of C3 for Python is
The Python 2.3 Method Resolution Order
by Michele Simionato.
The book outlaws classes containing such order disagreements, if
the order disagreement is "serious". An order disagreement between
two classes is serious when the two classes define at least one method
with the same name. In the example above, the order disagreement is
serious. In Python 2.2, I chose not to check for serious order
disagreements; but the meaning of a program containing a serious order
disagreement is undefined, and its effect may change in the future.
But since Samuele's counterexample, we know that outlawing order
disagreements isn't enough to avoid different outcomes between the
Python 2.2 algorithm (from the book) and the Python 2.3 algorithm (C3,
from the Dylan paper).
One of the coolest, but perhaps also one of the most unusual
features of the new classes is the possibility to write "cooperative" classes.
Cooperative classes are written with multiple inheritance in mind,
using a pattern that I call a "cooperative super call". This is known
in some other multiple-inheritance languages as "call-next-method",
and is more powerful than the super call found in single-inheritance
languages like Java or Smalltalk. C++ has neither form of super call,
relying instead on an explicit mechanism similar to that used in
classic Python. (The term "cooperative method" comes from
"Putting Metaclasses to Work".)
As a refresher, let's first review the traditional,
non-cooperative super call. When a class C derives from a base class
B, C often wants to override a method m defined in B. A "super call"
occurs when C's definition of m calls B's definition of m to do some of its
work. In Java, the body of m in C can write super(a, b, c) to call
B's definition of m with argument list (a, b, c). In Python, C.m writes
B.m(self, a, b, c) to accomplish the same effect. For example:
class B:
def m(self):
print "B here"
class C(B):
def m(self):
print "C here"
B.m(self)
We say that C's method m "extends" B's method m. The pattern here
works well as long as we're using single inheritance, but it breaks
down with multiple inheritance. Let's look at four classes whose
inheritance diagram forms a "diamond" (the same diagram was
shown graphically in the previous section):
class A(object): ..
class B(A): ...
class C(A): ...
class D(B, C): ...
Suppose A defines a method m, which is extended by both B and C.
Now what is D to do? It inherits two implementations of m, one from B
and one from C. Traditionally, Python simply picks the first one
found, in this case the definition from B. This is not ideal, because
this completely ignores C's definition. To see what's wrong with
ignoring C's m, assume that these classes represent some kind of
persistent container hierarchy, and consider a method that implements
the operation "save your data to disk". Presumably, a D instance has
both B's data and C's data, as well as A's data (a single copy of the
latter). Ignoring C's definition of the save method would mean that a D
instance, when requested to save itself, only saves the A and B parts
of its data, but not the part of its data defined by class C!
C++ notices that D inherits two conflicting definitions of method m,
and issues an error message. The author of D is then supposed to
override m to resolve the conflict. But what is D's definition of m
supposed to do? It can call B's m followed by C's m, but because both
definitions call the definition of m inherited from A, A's m ends up being
called twice! Depending on the details of the operation, this is at
best an inefficiency (when m is idempotent), at worst an error.
Classic Python has the same problem, except it doesn't even consider it an
error to inherit two conflicting definitions of a method: it simply picks
the first one.
The traditional solution to this dilemma is to split each derived
definition of m into two parts: a partial implementation _m, which only
saves the data that is unique to one class, and a full implementation
m, which calls its own _m and that of the base class(es). For
example:
class A(object):
def m(self): "save A's data"
class B(A):
def _m(self): "save B's data"
def m(self): self._m(); A.m(self)
class C(A):
def _m(self): "save C's data"
def m(self): self._m(); A.m(self)
class D(B, C):
def _m(self): "save D's data"
def m(self): self._m(); B._m(self); C._m(self); A.m(self)
There are several problems with this pattern. First of all, there
is the proliferation of extra methods and calls. But perhaps more
importantly, it creates an undesirable dependency in the derived classes
on details of the dependency graph of their base classes: the
existence of A can no longer be considered an implementation detail of
B and C, since class D needs to know about it. If, in a future
version of the program, we want to remove the dependency on A from B
and C, this will affect derived classes like D as well; likewise, if we
want to add another base class AA to B and C, all their derived
classes must be updated as well.
The "call-next-method" pattern solves this problem nicely, in
combination with the new method resolution order. Here's how:
class A(object):
def m(self): "save A's data"
class B(A):
def m(self): "save B's data"; super(B, self).m()
class C(A):
def m(self): "save C's data"; super(C, self).m()
class D(B, C):
def m(self): "save D's data"; super(D, self).m()
Note that the first argument to super is always the class in which
it occurs; the second argument is always self. Also note that self is
not repeated in the argument list for m.
Now, in order to explain how super works, consider the MRO for each
of these classes. The MRO is given by the __mro__ class attribute:
A.__mro__ == (A, object)
B.__mro__ == (B, A, object)
C.__mro__ == (C, A, object)
D.__mro__ == (D, B, C, A, object)
The expression super(C, self).m should only be used inside the
implementation of method m in class C. Bear in mind that while self
is an instance of C, self.__class__ may not be C: it may be a class
derived from C (for example, D). The expression super(C, self).m,
then, searches self.__class__.__mro__ (the MRO of the class that was
used to create the instance in self) for the occurrence of C, and then
starts looking for an implementation of method m following that
point.
For example, if self is a C instance, super(C, self).m will find
A's implementation of m, as will super(B, self).m if self is a B
instance. But now consider a D instance. In D's m, super(D,
self).m() will find and call B.m(self), since B is the first base
class following D in D.__mro__ that defines m. Now in B.m, super(B,
self).m() is called. Since self is a D instance, the MRO is (D, B, C,
A, object) and the class following B is C. This is where the search
for a definition of m continues. This finds C.m, which is called, and
in turn calls super(C, self).m(). Still using the same MRO, we see
that the class following C is A, and thus A.m is called. This is the
original definition of m, so no super call is made at this point.
Note how the same super expression finds a different class
implementing a method depending on the class of self! This is the
crux of the cooperative super mechanism.
The super call as shown above is
somewhat prone to errors: it is easy to copy and paste a super call
from one class to another while forgetting to change the class name to
that of the target class, and this mistake won't be detected if both
classes are part of the same inheritance graph. (You can even cause
infinite recursion by mistakenly passing in the name of a derived
class of the class containing the super call.) It would be nice if we
didn't have to name the class explicitly, but this would require more
help from Python's parser than we can currently get. I hope to fix
this in a future Python release by making the parser recognize super.
In the mean time, here's a trick you can apply. We can create a
class variable named __super that has "binding" behavior. (Binding
behavior is a new concept in Python 2.2, but it formalizes a
well-known concept from classic Python: the transformation from an
unbound method to a bound method when it is accessed via the getattr
operation on an instance. It is implemented by the __get__ method
discussed above.) Here's a simple
example:
class A:
def m(self): "save A's data"
class B(A):
def m(self): "save B's data"; self.__super.m()
B._B__super = super(B)
class C(A):
def m(self): "save C's data"; self.__super.m()
C._C__super = super(C)
class D(B, C):
def m(self): "save D's data"; self.__super.m()
D._D__super = super(D)
Part of the trick is in the use of the name __super, which (through
the name mangling transformation) contains the class name. This
ensures that self.__super means something different in each class (as
long as the class names differ; unfortunately, it is possible
in Python to reuse the name of a base class for a derived class).
Another part of the trick is that the super built-in can be called
with a single argument, and then creates an unbound version that can be
bound by a later instance getattr operation.
Unfortunately, this example is still rather ugly for a number of
reasons: super requires that the class is passed in, but the class is
not available until after execution of the class statement is
completed, so the __super class attribute must be assigned outside the
class. Outside the class, name mangling doesn't work (after all it is
intended to be a privacy feature) so the assignment must use the
unmangled name. Fortunately, it's possible to write a metaclass that automatically adds a __super
attribute to its classes; see the autosuper metaclass example below.
Note that super(class, subclass) also works; this is needed for
__new__ and other static methods.
As an illustration of the power of the new system, here's a fully
functional implementation of the super() built-in class in pure
Python. This may also help clarify the semantics of super() by
spelling out the search in ample detail. The print statement at the
bottom of the following code prints "DCBA".
class Super(object):
def __init__(self, type, obj=None):
self.__type__ = type
self.__obj__ = obj
def __get__(self, obj, type=None):
if self.__obj__ is None and obj is not None:
return Super(self.__type__, obj)
else:
return self
def __getattr__(self, attr):
if isinstance(self.__obj__, self.__type__):
starttype = self.__obj__.__class__
else:
starttype = self.__obj__
mro = iter(starttype.__mro__)
for cls in mro:
if cls is self.__type__:
break
# Note: mro is an iterator, so the second loop
# picks up where the first one left off!
for cls in mro:
if attr in cls.__dict__:
x = cls.__dict__[attr]
if hasattr(x, "__get__"):
x = x.__get__(self.__obj__)
return x
raise AttributeError, attr
class A(object):
def m(self):
return "A"
class B(A):
def m(self):
return "B" + Super(B, self).m()
class C(A):
def m(self):
return "C" + Super(C, self).m()
class D(C, B):
def m(self):
return "D" + Super(D, self).m()
print D().m() # "DCBA"
When subclassing immutable built-in types like numbers and strings,
and occasionally in other situations, the static method __new__ comes
in handy. __new__ is the first step in instance construction, invoked
before __init__. The __new__ method is called with the class
as its first argument; its responsibility is to return a new instance
of that class. Compare this to __init__: __init__ is called with an
instance as its first argument, and it doesn't return anything; its
responsibility is to initialize the instance. There are situations
where a new instance is created without calling __init__ (for example when
the instance is loaded from a pickle). There is no way to create a
new instance without calling __new__ (although in some cases you can
get away with calling a base class's __new__).
Recall that you create class instances by calling the class. When
the class is a new-style class, the following happens when it is
called. First, the class's __new__ method is called, passing the class
itself as first argument, followed by any (positional as well as
keyword) arguments received by the original call. This returns a new
instance. Then that instance's __init__ method is called to further
initialize it. (This is all controlled by the __call__ method of the
metaclass, by the way.)
Here is an example of a subclass that overrides __new__ - this
is how you would normally use it.
>>> class inch(float):
... "Convert from inch to meter"
... def __new__(cls, arg=0.0):
... return float.__new__(cls, arg*0.0254)
...
>>> print inch(12)
0.3048
>>>
This class isn't very useful (it's not even the right way to go
about unit conversions) but it shows how to extend the constructor of
an immutable type. If instead of __new__ we had tried to override
__init__, it wouldn't have worked:
>>> class inch(float):
... "THIS DOESN'T WORK!!!"
... def __init__(self, arg=0.0):
... float.__init__(self, arg*0.0254)
...
>>> print inch(12)
12.0
>>>
The version overriding __init__ doesn't work because the float
type's __init__ is a no-op: it returns immediately, ignoring its
arguments.
All this is done so that immutable types can preserve their
immutability while allowing subclassing. If the value of a float
object were initialized by its __init__ method, you could change the
value of an existing float object! For example, this would work:
>>> # THIS DOESN'T WORK!!!
>>> import math
>>> math.pi.__init__(3.0)
>>> print math.pi
3.0
>>>
I could have fixed this problem in other ways, for example by adding an
"already initialized" flag or only allowing __init__ to be called on
subclass instances, but those solutions are inelegant.
Instead, I added __new__, which is a perfectly general mechanism that
can be used by built-in and user-defined classes, for immutable and
mutable objects.
Here are some rules for __new__:
- __new__ is a static method. When defining it, you don't need
to (but may!) use the phrase "__new__ = staticmethod(__new__)",
because this is implied by its name (it is special-cased by the class
constructor).
- The first argument to __new__ must be a class; the remaining
arguments are the arguments as seen by the constructor call.
- A __new__ method that overrides a base class's __new__ method
may call that base class's __new__ method. The first argument to the
base class's __new__ method call should be the class argument to the
overriding __new__ method, not the base class; if you were to pass in
the base class, you would get an instance of the base class. (This is
really just analogous to passing self to an overridden __init__ call.)
- Unless you want to play games like those described in the next
two bullets, a __new__ method must call its base class's
__new__ method; that's the only way to create an instance of your
object. The subclass __new__ can do two things to affect the
resulting object: pass different arguments to the base class __new__,
and modify the resulting object after it's been created
(for example to initialize essential instance variables).
- __new__ must return an object. There's nothing that requires
that it return a new object that is an instance of its class argument,
although that is the convention. If you return an existing object,
the constructor call will still call its __init__ method. If you
return an object of a different class, its __init__ method will
be called. If you forget to return something, Python will unhelpfully
return None, and your caller will probably be very confused.
- For immutable classes, your __new__ may return a cached
reference to an existing object with the same value; this is what the
int, str and tuple types do for small values. This is one of the
reasons why their __init__ does nothing: cached objects would be
re-initialized over and over. (The other reason is that there's
nothing left for __init__ to initialize: __new__ returns a fully
initialized object.)
- If you subclass a built-in immutable type and want to add some
mutable state (maybe you add a default conversion to a string type),
it's best to initialize the mutable state in the __init__ method and
leave __new__ alone.
- If you want to change the constructor's signature, you often have
to override both __new__ and __init__ to accept the new signature.
However, most built-in types ignore the arguments to the method they
don't use; in particular, the immutable types (int, long, float,
complex, str, unicode, and tuple) have a dummy __init__, while the
mutable types (dict, list, file, and also super, classmethod,
staticmethod, and property) have a dummy __new__. The built-in
type 'object' has a dummy __new__ and a dummy __init__ (which the others
inherit). The built-in type 'type' is special in many respects; see
the section on metaclasses.
- (This has nothing to do to __new__, but is handy to know
anyway.) If you subclass a built-in type, extra space is
automatically added to the instances to accomodate __dict__ and
__weakrefs__. (The __dict__ is not initialized until you use it
though, so you shouldn't worry about the space occupied by an empty
dictionary for each instance you create.) If you don't need this
extra space, you can add the phrase "__slots__ = []" to your class.
(See above for more about __slots__.)
- Factoid: __new__ is a static method, not a class method. I
initially thought it would have to be a class method, and that's why I
added the classmethod primitive. Unfortunately, with class methods,
upcalls don't work right in this case, so I had to make it a static
method with an explicit class as its first argument. Ironically,
there are now no known uses for class methods in the Python
distribution (other than in the test suite). However, class methods
are still useful in other places, for example, to program inheritable
alternate constructors.
As another example of __new__, here's a way to implement the
singleton pattern.
class Singleton(object):
def __new__(cls, *args, **kwds):
it = cls.__dict__.get("__it__")
if it is not None:
return it
cls.__it__ = it = object.__new__(cls)
it.init(*args, **kwds)
return it
def init(self, *args, **kwds):
pass
To create a singleton class, you subclass from Singleton; each
subclass will have a single instance, no matter how many times its
constructor is called. To further initialize the subclass instance,
subclasses should override 'init' instead of __init__ - the __init__
method is called each time the constructor is called. For example:
>>> class MySingleton(Singleton):
... def init(self):
... print "calling init"
... def __init__(self):
... print "calling __init__"
...
>>> x = MySingleton()
calling init
calling __init__
>>> assert x.__class__ is MySingleton
>>> y = MySingleton()
calling __init__
>>> assert x is y
>>>
In the past, the subject of metaclasses in Python has caused hairs
to raise and even brains to explode (see, for example Metaclasses in Python 1.5). Fortunately, in
Python 2.2, metaclasses are more accessible and less dangerous.
Terminology-wise, a metaclass is simply "the class of a class".
Any class whose instances are themselves classes, is a metaclass.
When we talk about an instance that's not a class, the instance's
metaclass is the class of its class: by definition, x's metaclass is
x.__class__.__class__. But when we talk about a class C, we often
refer to its metaclass when we mean C.__class__ (not
C.__class__.__class__, which would be a meta-metaclass; there's not
much use for those although we don't rule them out).
The built-in 'type' is the most common metaclass; it is the
metaclass of all built-in types. Classic classes use a different
metaclass: the type known as types.ClassType. The latter is
relatively uninteresting; it's a historical artefact that's needed
to give classic classes their classic behavior. You can't get to the
metaclass of a classic instance using x.__class__.__class__; you have
to use type(x.__class__), because classic classes don't support the
__class__ attribute on classes (only on instances).
When a class statement is executed, the interpreter first
determines the appropriate metaclass M, and then calls M(name, bases,
dict). All this happens at the end of the class statement,
after the body of the class (where methods and class variables are
defined) has already been executed. The arguments to M are the class
name (a string taken from the class statement), a tuple of base
classes (expressions evaluated at the start of the class statement;
this is () if no bases are specified in the class statement), and a
dictionary containing the methods and class variables defined by the
class statement. Whatever this call M(name, bases, dict) returns is
then assigned to the variable corresponding to the class name, and
that's all there is to the class statement.
How is M determined?
- If dict['__metaclass__'] exists, it is used.
- Otherwise, if there is at least one base class, its metaclass is
used (this looks for a __class__ attribute first and if that's not
found, uses its type). (In classic Python, this step existed too, but
was only executed when the metaclass was callable. This was called
the Don Beaudry hook - may it rest in peace.)
- Otherwise, if there's a global variable named __metaclass__, it is
used.
- Otherwise, the classic metaclass (types.ClassType) is used.
The most common outcomes here are that M is either types.ClassType
(creating a classic class), or 'type' (creating a new-style class).
Other common outcomes are a custom extension type (like Jim Fulton's
ExtensionClass), or a subtype of 'type' (when we're using new-style
metaclasses). But it's possible to have something completely
outlandish here: if we specify a base class that has a custom
__class__ attribute, we can use anything as a "metaclass". That was
the brain-exploding topic of my original metaclass paper, and I won't repeat it here.
There's always an additional wrinkle. When you mix
classic classes and new-style classes in the list of bases, the
metaclass of the first new-style base class is used instead of
types.ClassType (assuming dict['__metaclass__'] is undefined). The
effect is that when you cross a classic class and a new-style class,
the offspring is a new-style class.
And another one (I promise this is the last wrinkle in the
metaclass determination). For new-style metaclasses, there is a
constraint that the chosen metaclass is equal to, or a subclass of, each
of the metaclasses of the bases. Consider a class C with two base
classes, B1 and B2. Let's say M = C.__class__, M1 = B1.__class__, M2
= B2.__class__. Then we require issubclass(M, M1) and issubclass(M,
M2). (This is because a method of B1 should be able to call a
meta-method defined in M1 on self.__class__, even when self is an
instance of a subclass of B1.)
The metaclasses book describes a mechanism
whereby a suitable metaclass is automatically created, when necessary,
through multiple inheritance from M1 and M2. In Python 2.2, I have
chosen a simpler approach which raises an exception if the
metaclass constraint is not satisfied; it is up to the programmer to
provide a suitable metaclass through the __metaclass__ class variable.
However, if one of the base metaclasses satisfies the constraint
(including the explicitly given __metaclass__, if any), the first base
metaclass found satisfying the constraint will be used as the
metaclass.
In practice, this means that if you have a degenerate metaclass
hierarchy that has the shape of a tower (meaning that for two
metaclasses M1 and M2, at least one of issubclass(M1, M2) or
issubclass(M2, M1) is always true), you don't have to worry about the
metaclass constraint. For example:
# Metaclasses
class M1(type): ...
class M2(M1): ...
class M3(M2): ...
class M4(type): ...
# Regular classes
class C1:
__metaclass__ = M1
class C2(C1):
__metaclass__ = M2
class C3(C1, C2):
__metaclass__ = M3
class D(C2, C3):
__metaclass__ = M1
class C4:
__metaclass__ = M4
class E(C3, C4):
pass
For class C2, the constraint is satisfied because M2 is a subclass
of M1. For class C3, it is satisfied because M3 is a subclass of both
M1 and M2. For class D, the explicit metaclass M1 is not a subclass
of the base metaclasses (M2, M3), but choosing M3 satisfies the
constraint, so D.__class__ is M3. However, class E is an error: the
two metaclasses involved are M3 and M4, and neither is a subclass of
the other. We can fix this latter case as follows:
# A new metaclass
class M5(M3, M4): pass
# Fixed class E
class E(C3, C4):
__metaclass__ = M5
(The approach from the metaclasses book would automatically supply
the class definition for M5 given the original definition of class E.)
Let's refresh some theory first. Remember that a class statement
causes a call to M(name, bases, dict) where M is the metaclass. Now,
a metaclass is a class, and we've already established that when a
class is called, its __new__ and __init__ methods are called in
sequence. Therefore, something like this will happen:
cls = M.__new__(M, name, bases, dict)
assert cls.__class__ is M
M.__init__(cls, name, bases, dict)
I'm writing the __init__ call as an unbound method call here. This
clarifies that we're calling the __init__ defined by M, not the
__init__ defined in cls (which would be the initialization for
instances of cls). But it really calls the __init__ method of object
cls; cls just happens to be a class.
Our first example is a metaclass that looks through the methods of
a class for methods named _get_<something> and _set_<something>,
and automatically adds property descriptors named <something>. It
turns out that it's sufficient to override __init__ to do what we
want. The algorithm makes two passes: first it collects names of
properties, then it adds them to the class. The collection pass looks
through dict, which is the dictionary representing the class variables
and methods (excluding base class variables and methods). But the
second pass, the property construction pass, looks up
_get_<something> and _set_<something> as class attributes. This
means that if a base class defines _get_x and a subclass defines
_set_x, the subclass will have a property x created from both methods,
even though only _set_x occurs in the subclass's dictionary. Thus,
you can extend properties in a subclass. Note that we use the
three-argument form of getattr(), so a missing _get_x or _set_x will
be translated into None, not raise an AttributeError. We also call
the base class __init__ method, in cooperative fashion using super().
class autoprop(type):
def __init__(cls, name, bases, dict):
super(autoprop, cls).__init__(name, bases, dict)
props = {}
for name in dict.keys():
if name.startswith("_get_") or name.startswith("_set_"):
props[name[5:]] = 1
for name in props.keys():
fget = getattr(cls, "_get_%s" % name, None)
fset = getattr(cls, "_set_%s" % name, None)
setattr(cls, name, property(fget, fset))
Let's test autoprop with a silly example. Here's a class that
stores an attribute x as its inverted value under self.__x:
class InvertedX:
__metaclass__ = autoprop
def _get_x(self):
return -self.__x
def _set_x(self, x):
self.__x = -x
a = InvertedX()
assert not hasattr(a, "x")
a.x = 12
assert a.x == 12
assert a._InvertedX__x == -12
Our second example creates a class, 'autosuper', which will add a
private class variable named __super, set to the value super(cls).
(Recall the discussion of self.__super above.) Now, __super is a private name
(starts with double underscore) but we want it to be a private name of
the class to be created, not a private name of autosuper. Thus, we
must do the name mangling ourselves, and use setattr() to set the
class variable. For the purpose of this example, I'm simplifying the
name mangling to "prepend an underscore and the class name". Again,
it's sufficient to override __init__ to do what we want, and again, we
call the base class __init__ cooperatively.
class autosuper(type):
def __init__(cls, name, bases, dict):
super(autosuper, cls).__init__(name, bases, dict)
setattr(cls, "_%s__super" % name, super(cls))
Now let's test autosuper with the classic diamond diagram:
class A:
__metaclass__ = autosuper
def meth(self):
return "A"
class B(A):
def meth(self):
return "B" + self.__super.meth()
class C(A):
def meth(self):
return "C" + self.__super.meth()
class D(C, B):
def meth(self):
return "D" + self.__super.meth()
assert D().meth() == "DCBA"
(Our autosuper metaclass is easily fooled if you define a subclass
with the same name as a base class; it should really check for that
condition and raise an error if it occurs. But that's more code than
feels right for an example, so I'll leave it as an exercise for the
reader.)
Now we have two independently developed metaclasses, we can combine
the two into a third metaclass that inherits from them both:
class autosuprop(autosuper, autoprop):
pass
Simple eh? Because we wrote both metaclasses cooperatively
(meaning their methods use super() to call the base class method),
that's all we need. Let's test it:
class A:
__metaclass__ = autosuprop
def _get_x(self):
return "A"
class B(A):
def _get_x(self):
return "B" + self.__super._get_x()
class C(A):
def _get_x(self):
return "C" + self.__super._get_x()
class D(C, B):
def _get_x(self):
return "D" + self.__super._get_x()
assert D().x == "DCBA"
That's all for today. I hope your brain doesn't hurt too much!
Relax! Most features described above are only invoked when
you use a class statement with a built-in object as a base class (or
when you use an explicit __metaclass__ assignment).
Some things that might affect old code:
Additional Topics
These topics should also be discussed:
- descriptors: __get__, __set__, __delete__
- The specs of the built-in types that are subclassable
- The 'object' type and its methods
- <type 'foo'> vs. <type 'mod.foo'> vs. <class 'mod.foo'>
- What else?
