Python Distutils-SIG

Proposed User Interface

In addition to identifying the common tasks and division of labour involved in developing, distributing, and installing Python modules, the "Extension Building Considered Painful" Developer's Day Session also came up with a proposed user interface. The core idea of the interface is that the module developer would provide a small Python script, called setup.py for purposes of illustration (although it is hoped that this convention catches on!). This script would have two components: meta-data about the module distribution (name, version number, description, etc.), written as Python code of some sort; and optional code to inspect the target system and carry out any needed pre-build configuration operations. Only the meta-data will be required, and indeed it is anticipated that most module distributions (especially those written purely in Python) will have only the meta-data component. Some ideas for how to represent the meta-data will be presented below.

This document describes an interface based on what we agreed upon at the Developer's Day session, but with a lot more detail. (Some of that detail could be construed as implementation rather than interface, but it's important to specify it somewhere.)

Once the setup.py script is written, developers, packagers, and installers alike will use it to carry out all of their tasks that can be automated (i.e. everything except actually writing the modules, documentation, and test suite). This will be done by running setup.py with a mandatory "command" argument corresponding to the task to be accomplished. (By sheer coincidence, many of these commands will look a lot like traditional makefile targets.)

For example, the command to initiate a build is build. The developer, packager, and installer (at least an installer working from a source distribution) will all have to build the module(s), using the following command:

    ./setup.py build

    ./setup.py test

    ./setup.py dist

    ./setup.py bdist

    ./setup.py bdist -rpm

By now, you're probably wondering what's really happening behind each of those commands. It could be argued that that is an implementation detail that doesn't belong in an interface proposal, but I think that most developers, many packagers, and a few curious users will be inclined to look "under the hood" and see what's really going on as they build and install module distributions. (And the unlucky ones will have to understand the process if things go wrong!) So, here is a list of commands to be supported by setup.py (through cooperation with the distutils modules) and the actions corresponding to each command:

make_blib

If it doesn't already exist, create a mockup installation tree, blib/, under the current directory. blib/ would contain directories for pure Python code (non-architecture-specific, or shared) and compiled code (architecture-specific), modelled after the directories in the system Python library on the current machine (which are determined when Python itself is built). For instance, make_blib might create blib/share/ (shared files) and blib/plat-i86-linux/ (architecture-specific files). There might also be directories for documentation, e.g. blib/man/ for Unix-style man pages, blib/info/ for GNU info documentation, and/or blib/html/ for (you guessed it) HTML documenation.

build_py

Do make_blib; copy .py files into the "shared" blib directory (or subdirectories of it, if they belong to packages), and compile them to .pyc and .pyo form.

build_extensions

Do make_blib; compile ancillary C files (those that don't provide extension modules themselves, but are needed for the extension(s) being built to work); compile extension C files; link ancillary and extension C files (along with any external libraries needed) to create dynamic libraries (e.g. .so files for Unix, DLLs for Windows, etc.) in the architecture-specific blib directory

build_doc

process documentation to an installable form somewhere under ./blib e.g. *roff-formatted man pages, GNU Info, HTML, Windows help, etc. Perhaps the desired documentation format(s) could be determined when Python itself is built (and, naturally, the Python documentation would be available in that format, in the same place that module documentation accumulates)?

build

Do build_py, build_extensions, and build_doc. (Of course, any one of these operations does nothing if the appropriate inputs are not there.)

dist

Create a source distribution ...

bdist

Create a built distribution ...

test

Look for the test suite and run it. One possible way to define a test suite: put a bunch of scripts in a predefined subdirectory of the source distribution; they will then be executed with the shared and architecture-specific blib directories added to Python's library search path. For instance, the test command might search for test/*.t, executing each script in turn and interpreting its output to determine whether that series of tests succeeded or failed.

install

Copy .py, .pyc, and .pyo files into the shared installation directory (which would default to the site-specific area of the Python system library tree); copy dynamic libraries (shared objects or DLLs) to the architecture-specific installation directory; copy processed documentation (man pages, Info files, whatever) to the documentation installation directory. (Perhaps this should be split up into install_py, install_extensions, and install_doc?)