Python Library Reference

Next: 3.20 pyclbr Up: 3. Python Services Previous: 3.18 keyword

---

3.19 tokenize -- Tokenizer for Python source

The tokenize module provides a lexical scanner for Python source code, implemented in Python. The scanner in this module returns comments as tokens as well, making it useful for implementing ``pretty-printers,'' including colorizers for on-screen displays.

The scanner is exposed by a single function:

tokenize (readline[, tokeneater])

The tokenize() function accepts two parameters: one representing the input stream, and one providing an output mechanism for tokenize().

The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects (see section 2.1.7). Each call to the function should return one line of input as a string.

The second parameter, tokeneater, must also be a callable object. It is called with five parameters: the token type, the token string, a tuple (srow, scol) specifying the row and column where the token begins in the source, a tuple (erow, ecol) giving the ending position of the token, and the line on which the token was found. The line passed is the logical line; continuation lines are included.

All constants from the token module are also exported from tokenize, as is one additional token type value that might be passed to the tokeneater function by tokenize():

COMMENT

Token value used to indicate a comment.

---

Python Library Reference

Next: 3.20 pyclbr Up: 3. Python Services Previous: 3.18 keyword