 |
 |
 |
Python Library Reference |  |
 |
 |
|
|
|
Python Library Reference | |
|
|
The tokenize module provides a lexical scanner for Python source code, implemented in Python. The scanner in this module returns comments as tokens as well, making it useful for implementing ``pretty-printers,'' including colorizers for on-screen displays.
The scanner is exposed by a single function:
The first parameter, readline, must be a callable object which provides the same interface as the readline() method of built-in file objects (see section 2.1.7). Each call to the function should return one line of input as a string.
The second parameter, tokeneater, must also be a callable
object. It is called with five parameters: the token type, the
token string, a tuple (srow, scol) specifying the
row and column where the token begins in the source, a tuple
(erow, ecol) giving the ending position of the
token, and the line on which the token was found. The line passed
is the logical line; continuation lines are included.
All constants from the token module are also exported from tokenize, as are two additional token type values that might be passed to the tokeneater function by tokenize():
|
|
|
Python Library Reference | |
|
|