[Python Home] [PEP Index] [PEP Source] |
PEP: | 301 |
---|---|
Title: | Package Index and Metadata for Distutils |
Version: | 1978 |
Last-Modified: | 2005-03-19 07:23:53 -0800 (Sat, 19 Mar 2005) |
Author: | Richard Jones <rjones at ekit-inc.com> |
Status: | Final |
Type: | Standards Track |
Content-Type: | text/x-rst |
Created: | 24-Oct-2002 |
Python-Version: | 2.3 |
Post-History: | 8-Nov-2002 |
This PEP proposes several extensions to the Distutils packaging system [1]. These enhancements include a central package index server, tools for submitting package information to the index and extensions to the package metadata to include Trove [2] information.
This PEP does not address issues of package dependency. It also does not address storage and download of packages as described in PEP 243 [6]. Nor is it proposing a local database of packages as described in PEP 262 [7].
Existing package repositories such as the Vaults of Parnassus [3], CPAN [4] and PAUSE [5] will be investigated as prior art in this field.
Python programmers have long needed a simple method of discovering existing modules and systems available for their use. It is arguable that the existence of these systems for other languages have been a significant contribution to their popularity. The existence of the Catalog-SIG, and the many discussions there indicate that there is a large population of users who recognise this need.
The introduction of the Distutils packaging system to Python simplified the process of distributing shareable code, and included mechanisms for the capture of package metadata, but did little with the metadata save ship it with the package.
An interface to the index should be hosted in the python.org domain, giving it an air of legitimacy that existing catalog efforts do not have.
The interface for submitting information to the catalog should be as simple as possible - hopefully just a one-line command for most users.
Issues of package dependency are not addressed due to the complexity of such a system. PEP 262 proposes such a system, but as of this writing the PEP is still unfinished.
Issues of package dissemination (storage on a central server) are not addressed because they require assumptions about availability of storage and bandwidth that I am not in a position to make. PEP 243, which is still being developed, is tackling these issues and many more. This proposal is considered compatible with, and adjunct to the proposal in PEP 243.
The specification takes three parts, the web interface, the Distutils register command and the Distutils Trove classification.
A web interface is implemented over a simple store. The interface is available through the python.org domain, either directly or as packages.python.org.
The store has columns for all metadata fields. The (name, version) double is used as a uniqueness key. Additional submissions for an existing (name, version) will result in an update operation.
The web interface implements the following commands/interfaces:
Accepts a POST submission of metadata about a package. The "name" and "version" fields are mandatory, as they uniquely identify an entry in the index. Submit will automatically determine whether to create a new entry or update an existing entry. The metadata is checked for correctness where appropriate - specifically the Trove discriminators are compared with the allowed set. An update will update all information about the package based on the new submitted information.
There will also be a submit/edit form that will allow manual submission and updating for those who do not use Distutils.
Registers a new user with the index. Requires username, password and email address. Passwords will be stored in the index database as SHA hashes. If the username already exists in the database:
Registration will be a three-step process, involving:
The submit command will require HTTP Basic authentication, preferably over an HTTPS connection.
The server interface will indicate success or failure of the commands through a subset of the standard HTTP response codes:
Code | Meaning | Register command implications |
---|---|---|
200 | OK | Everything worked just fine |
400 | Bad request | Data provided for submission was malformed |
401 | Unauthorised | The username or password supplied were incorrect |
403 | Forbidden | User does not have permission to update the package information (not Owner or Maintainer) |
Three user Roles will be assignable to users:
The index is stored in a set of relational database tables:
An additional table, rego_otk holds the One Time Keys generated during registration and is not interesting in the scope of the index itself.
An additional Distutils command, register, is implemented which posts the package metadata to the central index. The register command automatically handles user registration; the user is presented with three options:
On systems where the $HOME environment variable is set, the user will be prompted at exit to save their username/password to a file in their $HOME directory in the file .pypirc.
Notification of changes to a package entry will be sent to all users who have submitted information about the package. That is, the original submitter and any subsequent updaters.
The register command will include a --verify option which performs a test submission to the index without actually committing the data. The index will perform its submission verification checks as usual and report any errors it would have reported during a normal submission. This is useful for verifying correctness of Trove discriminators.
The Trove concept of discrimination will be added to the metadata set available to package authors through the new attribute "classifiers". The list of classifiers will be available through the web, and added to the package like so:
setup( name = "roundup", version = __version__, classifiers = [ 'Development Status :: 4 - Beta', 'Environment :: Console', 'Environment :: Web Environment', 'Intended Audience :: End Users/Desktop', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'License :: OSI Approved :: Python Software Foundation License', 'Operating System :: MacOS :: MacOS X', 'Operating System :: Microsoft :: Windows', 'Operating System :: POSIX', 'Programming Language :: Python', 'Topic :: Communications :: Email', 'Topic :: Office/Business', 'Topic :: Software Development :: Bug Tracking', ], url = 'http://sourceforge.net/projects/roundup/', ... )
It was decided that strings would be used for the classification entries due to the deep nesting that would be involved in a more formal Python structure.
The original Trove specification that classification namespaces be separated by slashes ("/") unfortunately collides with many of the names having slashes in them (e.g. "OS/2"). The double-colon solution (" :: ") implemented by SourceForge and FreshMeat gets around this limitation.
The list of classification values on the module index has been merged from FreshMeat and SourceForge (with their permission). This list will be made available both through the web interface and through the register command's --list-classifiers option as a text list which may then be copied to the setup.py file. The register command's --verify option will check classifiers values against the server's list.
Unfortunately, the addition of the "classifiers" property is not backwards-compatible. A setup.py file using it will not work under Python 2.1.3. It is hoped that a bug-fix release of Python 2.2 (most likely 2.2.3) will relax the argument checking of the setup() command to allow new keywords, even if they're not actually used. It is preferable that a warning be produced, rather than a show-stopping error. The use of the new keyword should be discouraged in situations where the package is advertised as being compatible with python versions earlier than 2.2.3 or 2.3.
In the PKG-INFO, the classifiers list items will appear as individual Classifier: entries:
Name: roundup Version: 0.5.2 Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) . . Classifier: Topic :: Software Development :: Bug Tracking Url: http://sourceforge.net/projects/roundup/
The server is available at:
http://www.python.org/pypi
The code is available from the SourceForge project:
http://sourceforge.net/projects/pypi/
The register command has been integrated into Python 2.3.
Originally, the index server was to return custom headers (inspired by PEP 243):
However, it has been pointed out [8] that this is a bad scheme to use.
[1] | Distutils packaging system (http://www.python.org/doc/current/lib/module-distutils.html) |
[2] | Trove (http://tuxedo.org/~esr/trove/) |
[3] | Vaults of Parnassus (http://www.vex.net/parnassus/) |
[4] | CPAN (http://www.cpan.org/) |
[5] | PAUSE (http://pause.cpan.org/) |
[6] | PEP 243, Module Repository Upload Mechanism (http://www.python.org/peps/pep-0243.html) |
[7] | PEP 262, A Database of Installed Python Packages (http://www.python.org/peps/pep-0262.html) |
[8] | [PEP243] upload status is bogus (http://mail.python.org/pipermail/distutils-sig/2001-March/002262.html) |
This document has been placed in the public domain.
Anthony Baxter, Martin v. Loewis and David Goodger for encouragement and feedback during initial drafting.
A.M. Kuchling for support including hosting the second prototype.
Greg Stein for recommending that the register command interpret the HTTP response codes rather than custom X-PyPI-* headers.
The many participants of the Distutils and Catalog SIGs for their ideas over the years.