How to help with the development of Python

In an attempt to make it easy as possible for people to find out how they can help contribute to Python's development, I wrote an essay on the topic (mentioned last month, but some revisions have been done). It covers how Python is developed and how anyone can contribute to the development process. The latest version can be found at http://mail.python.org/pipermail/python-dev/2003-October/039473.html .

Any comments on the essay are appreciated.

Contributing threads:

• Draft of an essay on Python development
• 2nd draft of "How Py is Developed" essay

Generator Expressions: list comp's older brother

If you ever wanted to have the power of list comprehensions but without the overhead of generating the entire list you have Peter Norvig to thank initially, and then what seems like the rest of the world, for generator expressions.

PEP 289 covers all the details, but here is a quick intro. You can think of generator expressions as list comprehensions that return an iterator for each item instead a list items. The syntax is practically the same as list comprehensions as well; just substitute parentheses for square brackets (most of the time; generator expressions just need parentheses around them, so being the only argument to a method takes care of the parentheses requirement).

A quick example is:

(x for x in range(10) if x%2)

returns an iterator that returns the odd numbers from 0 to 10. This make list comprehensions just syntactic sugar for passing a generator expression to list() (note how extra parentheses are not needed):

list(x for x in range(10) is x%2)

Having list comprehensions defined this way also takes away the dangling item variable for the 'for' loop. Using that dangling variable is now discouraged and will be made illegal at some point.

For complete details, read the PEP.

Contributing threads:

• decorate-sort-undecorate
• accumulator display syntax
• listcomps vs. for loops
• PEP 289: Generator Expressions (second draft)

list.sorted() is born

After the addition of the 'key' argument to list.sort(), people began to clamor for list.sort() to return self. Guido refused to do give in, so a compromise was reached. 'list' now has a class method named 'sorted'; pass it an iterable and it will return a sorted list.

Contributing threads:

• decorate-sort-undecorate
• inline sort option
• sort() return value
• copysort patch

Recursion limit in re is now history

Thanks to Gustavo Niemeyer the recursion limit in the re module has now be removed!

Contributing threads:

• SRE recursion
• SRE recursion removed

Copying iterators one day at a time

Reiteration for iterators came up as part of the immense discussion on generator expressions. The difficulty of doing it generally came up. This lead to Alex Martelli proposing magic method support for __copy__ in iterators that have want to allow making a copy of itself. This was written down as PEP 323.

As an interim solution, itertools grew a new function: tee. It takes in an iterable and returns two or more iterators which independently iterate over the iterable.

Contributing threads:

• Reiterability
• cloning iterators again
• ... python/nondist/peps pep-0323.txt, NONE ...
• Guido's Magic Code

Returning Py_(None, True, False) now easier than ever!

Py_RETURN_NONE, Py_RETURN_TRUE, and Py_RETURN_FALSE have been added to Python 2.4. They are macros for returning the singleton mentioned in the name. Documentation has yet to be written (my fault).

Contributing threads:

• How to spell Py_return_None and friends
• python/dist/src/Include object.h, 2.121, ...

'String substitutions'/'dict interpolation'/'replace %(blah)s with $blah'

The idea of introducing string substitutions using '$' came up. Guido said that if this was made a built-in feature it would have to wait until Python 3. He was receptive to moving the functionality to a module, though.

Barry Warsaw pasted code into http://mail.python.org/pipermail/python-dev/2003-October/039369.html that handles string substitutions.

Contributing threads:

• Can we please have a better dict interpolation syntax?

"reduce() just doesn't get enough mileage"

That quote comes from Guido during the discussion over whether 'product' should be added as an accumulator function built-in like 'sum'. The idea was shot down and conversation quickly turned to whether 'reduce' should stay in the language (the consensus was "no" since the function does not read well and its functionality can easily be done with a 'for' loop).

A larger discussion on what built-ins should eventually disappear will be covered in the next Summary.

Contributing threads:

• product()

PyPy update

The PyPy development group sent an update on their happenings to the list. Turns out they are trying to get funding from the European Union. They are also fairly close to getting a working version (albeit with some bootstrapping from CPython, but it will still be damn cool what they have pulled off even with this caveat).

They also announced a sprint they are holding in Amsterdam from Dec. 14-21. More info can be found at http://codespeak.net/moin/pypy/moin.cgi/AmsterdamSprint .

Contributing threads:

• PyPy: sprint and news

Never say Python is finished

I asked python-dev for masters thesis ideas. I great number of possibilities were put out. If anyone out there is curious to see what some people would like to see done for Python in terms of a large project check the thread out.

Contributing threads:

• Looking for master thesis ideas involving Python

Rough draft of Decimal object PEP

Facundo Batista has posted a rough draft of a PEP for a decimal object that is being worked on in the sandbox. Comment on it on comp.lang.python if this interests you.

Contributing threads:

• prePEP: Decimal data type

Relations of basestring and bye-bye operator.isMappingType

The idea of introducing relatives of basestring for numbers came from Alex Martelli. That idea was shot down for not being needed once the merger of int and long occurs.

The point that operator.isMappingType is kind of broken came up. Both Alex and Raymond Hettinger would not mind seeing it disappear. No one objected. It is still in CVS at the moment, but I would not count on it necessarily sticking around (although there are rumblings that there might be a way to fix it so it is partially useful).

Contributed threads:

• reflections on basestring -- and other abstract basetypes
• operator.isMappingType

Why one checks into the trunk before a maintenance branch

The question of whether checking a change into a maintenance branch before applying it to the main trunk was acceptable came up. The short answer is "no". Basically the trunk gets more testing than the maintenance branches and thus the patch should have to prove its stability first. Only then should it go into a maintenance branch.

The same goes for changes to code that will eventually disappear in the trunk. Someone might be planning on removing some code, but if that person falls off the face of the earth the code will still be there. That means applying the patch to the code that is scheduled to disappear is still a good idea.

Contributing threads:

• check-in policy, trunk vs maintenance branch

New reversed() built-in

There was a new built-in named reversed(), and all rejoiced after giving the creator a thorough beating.

Straight from the function's doc string: "reverse iterator over values of the sequence". PEP 322 has the relevant details on this new built-in.

Contributing threads:

• PEP 322: Reverse Iteration

Cleaning the built-in house

Guido asked what built-ins should be considered for deprecation. Instantly intern, coerce, and apply came up. apply already had a PendingDeprecationWarning and that will stay for the next release or two. intern and coerce, though, did not have any major champions (intern had some fans, but just for the functionality).

Guido did state that none of these built-in will be removed any time soon. If they do get deprecated it does not lead to immediate removal. Python 3, though, takes the gloves off and that can see them just completely disappear.

Contributing threads:

• Deprecating obsolete builtins

Passing arguments to str.(encode|decode)

The idea of allowing keyword arguments be passed to any specified encoder/decoder was brought up by Raymond Hettinger. It seemed like an idea that was supported. The idea of specifying the encoder or decoder based on the actual object instead of the current way of specifying a string that is passed to the registered codec search functions was suggested.

Nothing has been finalized on this idea as of now.

Contributing threads:

• Optional arguments for str.encode /.decode

Where, oh where, to move the good stuff out of string?

It looks like ascii_* and possibly maketrans from the string module will be tacked on to the str type so that the string module can finally be removed from the language. It has not been pronounced upon, but it looks like that is what the BDFL is leaning towards.

Issues of using the methods of str as unbound methods did come up. As it stands you cannot pass a unicode object to str.upper and thus there is no one uppercasing function as there is in the string module. This issue brought up the problem of Unicode and locale ties and collation (how to sort things).

Contributing threads:

• other "magic strings" issues

Supported versions of Sleepycat for bsddb

The basic answer is 3.2 - 4.2 should work when you compile from source.

Contributing threads:

• which sleepycat versions do we support in 2.3.* ?

Sets now at blazing C speeds!

Raymond Hettinger implemented the sets API in C! The new built-ins are set (which replaces sets.Set) and frozenset (which replaces sets.ImmutableSet). The APIs are the same as the sets module sans the name change from ImmutableSet to frozenset.

Contributing threads:

• set() and frozenset()