How Python is Developed

Guido, Some Guys, and a Mailing List: How Python is Developed

by Brett Cannon (brett at python.org)

Introduction

Software does not make itself. Code does not spontaneously come from the ether of the universe. Python is no exception to this rule. Since Python made its public debut back in 1991 many people beyond the BDFL (Benevolent Dictator For Life, Guido van Rossum) have helped contribute time and energy to making Python what it is today; a powerful, simple programming language available to all.

But it has not been a random process of people doing whatever they wanted to Python. Over the years a process to the development of Python has emerged by the group that heads Python's growth and maintenance; python-dev. This document is an attempt to write this process down in hopes of lowering any barriers possibly preventing people from contributing to the development of Python.

Tools Used

To help facilitate the development of Python, certain tools are used. Beyond the obvious ones such as a text editor and email client, two tools are very pervasive in the development process.

SourceForge is used by python-dev to keep track of feature requests, reported bugs, and contributed patches. A detailed explanation on how to use SourceForge is covered later in General SourceForge Guidelines.

Subversion (also called svn) is a networked file versioning system that stores all of files that make up Python that is currently hosted on svn.python.org . It allows the developers to have a single repository for the files along with being able to keep track of any and all changes to every file. The basic commands and uses can be found in the dev FAQ along with a multitude of tutorials spread across the web.

Communicating

Python development is not just programming. It requires a great deal of communication between people. This communication is not just between the members of python-dev; communication within the greater Python community also helps with development. Several mailing lists and newsgroups are used to help organize all of these discussions.

In terms of Python development, the primary location for communication is the python-dev mailing list. This is where the members of python-dev hash out ideas and iron out issues. It is an open list; anyone can subscribe to the mailing list. While the discussion can get quite technical, it is not all out of the reach for even a novice and thus should not discourage anyone from joining the list. Please realize, though, this list is meant for the discussion of the development of Python; all other questions (including those about development in Python) should be directed somewhere else, such as python-list. Along with this, a level of etiquette is expected to be maintained. A lack of manners will not be tolerated.

When the greater Python community is involved in a discussion, it always ends up on python-list. This mailing list is a gateway to the newsgroup comp.lang.python. This is also a good place to go when you have a question about Python that does not pertain to the actual development of the language.

Using svn allows the development team to know who made a change to a file and when they made their change. But unless one wants to continuously update their local checkout of the repository, the best way to stay on top of changes to the repository is to subscribe to Python-checkins. This list sends out an email for each and every change to a file in Python. This list can generate a large amount of traffic since even changing a typo in some text will trigger an email to be sent out. But if you wish to be kept abreast of all changes to Python then this is a good way to do so.

The Patches mailing list sends out an email for all changes to patch items on SourceForge. This list, just like Python-checkins, can generate a large amount of email traffic. It is in general useful to people who wish to help out with the development of Python by knowing about all new submitted patches as well as any new developments on preexisting ones.

Python-bugs-list functions much like the Patches mailing list except it is for bug items on SourceForge. If you find yourself wanting to help to close and remove bugs in Python this is the right list to subscribe to if you can handle the volume of email.

The Actual Development

Developing Python is not all just conversations about neat new language features (although those neat conversations do come up and there is a process to it). Developing Python also involves maintaining it by eliminating discovered bugs, adding and changing features, and various other jobs that are not necessarily glamorous but are just as important to the language as anything else.

General SourceForge Guidelines

Since a good amount of Python development involves using SourceForge, it is important to follow some guidelines when handling a tracker item (bug, patch, etc.). Probably one of the most important things you can do is make sure to set the various options in a new tracker item properly. The submitter should make sure that the Data Type, Category, and Group are all set to reasonable values. The remaining values (Assigned To, Status, and Resolution) should in general be left to Python developers to set. The exception to this rule is when you want to retract a patch; then "close" the patch by setting Status to "closed" and Resolution to whatever is appropriate.

Make sure you do a cursory check to make sure what ever you are submitting was not previously submitted by someone else. Duplication just uses up valuable time. If you must create a new item (for instance, you are posting a patch for a bug report), please place cross-references in both tracker items.

And please do not post feature requests, bug reports, or patches to the python-dev mailing list. If you do you will be instructed to create an appropriate SourceForge tracker item. When in doubt as to whether you should bring something to python-dev's attention, you can always ask on comp.lang.python; Python developers actively participate there and move the conversation over if it is deemed reasonable.

Feature Requests

A feature request means you have no plans on actually implementing your idea by writing a patch. On occasion people do go through the features requests (also called RFEs on SourceForge) to see if there is anything there that they think should be implemented and actually do the implementation. But in general do not expect something put here to be implemented without some participation on your part.

The best way to get something implemented is to campaign for it in the greater Python community. comp.lang.python is the best place to accomplish this. Post to the newsgroup with your idea and see if you can either get support or convince someone to implement it. It might even end up being added to PEP 42 so that the idea does not get lost in the noise as time passes.

Bug Reports

Think you found a bug? Then submit a bug report on SourceForge. Make sure you clearly specify what version of Python you are using, what OS, and under what conditions the bug was triggered. The more information you can give the faster the bug can be fixed since time will not be wasted requesting more information from you.

Patches

Create a patch tracker item on SourceForge for any code you think should be applied to the Python svn tree. For practically any change to Python's functionality the documentation and testing suite will need to be changed as well. Doing this in the first place speeds things up considerably.

Please make sure your patch is against the svn repository. If you don't know how to use it (basics are covered in the dev FAQ), then make sure you specify what version of Python you made your patch against.

In terms of coding standards, PEP 8 specifies for Python while PEP 7 specifies for C. Always try to maximize your code reuse; it makes maintenance much easier.

For C code make sure to limit yourself to ANSI C code as much as possible. If you must use non-ANSI C code then see if what you need is checked for by looking in pyconfig.h . You can also look in Include/pyport.h for more helpful C code. If what you need is still not there but it is in general available, then add a check in configure.in for it (don't forget to run autoreconf to make the changes to take effect). And if that still doesn't fit your needs then code up a solution yourself. The reason for all of this is to limit the dependence on external code that might not be available for all OSs that Python runs on.

Be aware of intellectual property when handling patches. Any code with no copyright will fall under the copyright of the Python Software Foundation. If you have no qualms with that, wonderful; this is the best solution for Python. But if you feel the need to include a copyright then make sure that it is compatible with copyright used on Python (i.e., BSD-style). The best solution, though, is to sign the copyright over to the Python Software Foundation.

When you think you're ready, review your own patch (see the instructions in Helping Out). Then try to review a few other patches as well; many patches are never applied because no one has the time to review them. If you review enough patches from other people, someone with checkin privileges is more likely to spend some time on your patch.

Changing the Language

You understand how to file a patch. You think you have a great idea on how Python should change. You are ready to write code for your change. Great, but you need to realize that certain things must be done for a change to be accepted. Changes fall into two categories; changes to the standard library (referred to as the "stdlib") and changes to the language proper.

Changes to the stdlib

Changes to the stdlib can consist of adding functionality or changing existing functionality.

Adding minor functionality (such as a new function or method) requires convincing a member of python-dev that the addition of code caused by implementing the feature is worth it. A big addition such as a module tends to require more support than just a single member of python-dev. As always, getting community support for your addition is a good idea.

With all additions, make sure to write up documentation for your new functionality. Also make sure that proper tests are added to the testing suite.

If you want to add a module, be prepared to be called upon for any bug fixes or feature requests for that module. Getting a module added to the stdlib makes you by default its maintainer. If you can't take that level of responsibility and commitment and cannot get someone else to take it on for you then your battle will be very difficult; when there is not a specific maintainer of code python-dev takes responsibility and thus your code must be useful to them or else they will reject the module. There is also the possibility of having to write a PEP (read about PEPs in Changing the Language Proper).

Guido has also stated that for any new module it has to have been accepted and widely used by the greater Python community before it can be considered for addition to the stdlib. This is to prevent bloat in the stdlib by making sure that any new modules will actually be used and not just by the person who wrote the module and a few people who supported its addition.

Changing existing functionality can be difficult to do if it breaks backwards-compatibility. If your code will break existing code, you must provide a legitimate reason on why making the code act in a non- compatible way is better than the status quo. This requires python- dev as a whole to agree to the change.

Changing the Language Proper

Changing Python the language is taken very seriously. Python is often heralded for its simplicity and cleanliness. Any additions to the language must continue this tradition and view. Thus any changes must go through a long process.

First, you must write a PEP (Python Enhancement Proposal). This is basically just a document that explains what you want, why you want it, what could be bad about the change, and how you plan on implementing the change. It is best to get feedback on PEPs on comp.lang.python and from python-dev. Once you feel the document is ready you can request a PEP number and to have it added to the official list of PEPs in PEP 0.

Once you have a PEP, you must then convince python-dev and the BDFL that your change is worth it. Be expected to be bombarded with questions and counter-arguments. It can drag on for over a month, easy. If you are not up for that level of discussion then do not bother with trying to get your change in. If you manage to convince a majority of python-dev and the BDFL (or most of python-dev; that can lead to the BDFL changing his mind) then your change can be applied.

As with all new code make sure you also have appropriate documentation patches along with tests for the new functionality.

Changes to the documentation

The Python documentation is maintained using LaTeX with many additional macros specific for use in documenting Python. However, you don't need to know LaTeX to contribute to the documentation. Documentation contributions can be made as plain text or ReST, and one of the Python TeXnicians will be glad to convert the text to LaTeX for you. (It does help to indication where in the documentation the contribution is expected to go, to allow the TeXnician to test it in place.)

If you're working on an issue in the SourceForge trackers that contains non-LaTeX documentation, and you don't feel you can convert the text to LaTeX yourself, you can assign the issue to one of the TeXnicians as well.

Helping Out

Many people say they wish they could help out with the development of Python but feel they are not up to writing code. There are plenty of things one can do, though, that does not necessarily require you to write code. Regardless of your coding abilities, there is something for everyone to help with. Below is an attempt to discuss all possible options one has in helping out with development.

For feature requests, adding a comment about what you think is helpful. State whether or not you would like to see the feature. You can also volunteer to write the code to implement the feature if you feel up to it.

For bugs, stating whether or not you can reproduce the bug yourself can be extremely helpful. (Do give details about your system; if the bug shows up only on Linux, or only on Windows XP, that makes it easier to track down.) If you can write a fix for the bug that is very helpful as well; start a patch item and reference it in a comment in the bug item.

There is always a need for more people to test and review patches! Apply the patch and run the testing suite. You can do a code review on the patch to make sure that it is good, clean code. If the patch adds a new feature, comment on whether you think it is worth adding. If it changes functionality then comment on whether you think it might break code; if it does, say whether you think it is worth the cost of breaking existing code. Help add to the patch if it is missing documentation patches or needed regression tests. (Please check for existing tests both in the code itself, and in the test package.)

A special mention about adding a file to a tracker item is needed. Only official developers and the creator of the tracker item can add a file. This means that if you want to add a file and you are neither of the types of people just mentioned you have to do an extra step or two. One thing you can do is post the file you want added somewhere else online and reference the URL in a comment. You can also create a new patch item if you feel the change is thorough enough and cross- reference between both patches in the comments. Be wary of this last option, though, since some people might be offended since it might come off as if you think their code is bad and yours is better. The best solution of all is to work with the original poster if they are receptive to help. But if they do not respond or are not friendly then do go ahead and do one of the other two suggestions.

For language changes, make your voice be heard. Comment about any PEPs on comp.lang.python so that the general opinion of the community can be assessed.

If there is nothing specific you find you want to work on but still feel like contributing nonetheless, there are several things you can do. The documentation can always use fleshing out. Adding more tests to the testing suite is always useful. Running the test suite itself (see the documentation for the 'test' package in the standard library on how) and reporting any failed test is always a great help. Contribute to discussions on python-dev, comp.lang.python, or one of the SIGs (Special Interest Groups). Just helping out in the community by spreading the word about Python or helping someone with a question is helpful.

If you really want to get knee-deep in all of this, join python-dev. Once you have been actively participating for a while and are generally known on python-dev you can request to have checkin rights on the svn tree. It is a great way to learn how to work in a large, distributed group along with how to write great code.

And if all else fails give money; the Python Software Foundation is a non-profit organization that accepts donations that are tax- deductible in the United States. The funds are used for various thing such as lawyers for handling the intellectual property of Python to funding PyCon. But the PSF could do a lot more if they had the funds. One goal is to have enough money to fund having Guido work on Python for a full year full-time; this would bring about Python 3. Every dollar does help, so please contribute if you can.

Conclusion

If you get any message from this document, it should be that anyone can help with the development of Python. All help is greatly appreciated and keeps the language the wonderful piece of software that it is.