|
|
|
Python/C API Reference Manual |
|
|
|
7.3.2.1 Built-in Codecs
Python provides a set of builtin codecs which are written in C
for speed. All of these codecs are directly usable via the
following functions.
Many of the following APIs take two arguments encoding and
errors. These parameters encoding and errors have the same semantics
as the ones of the builtin unicode() Unicode object constructor.
Setting encoding to NULL causes the default encoding to be used
which is ASCII. The file system calls should use
Py_FileSystemDefaultEncoding as the encoding for file
names. This variable should be treated as read-only: On some systems,
it will be a pointer to a static string, on others, it will change at
run-time (such as when the application invokes setlocale).
Error handling is set by errors which may also be set to NULL
meaning to use the default handling defined for the codec. Default
error handling for all builtin codecs is ``strict''
(ValueError is raised).
The codecs all use a similar interface. Only deviation from the
following generic ones are documented for simplicity.
These are the generic codec APIs:
PyObject* PyUnicode_Decode( | const char *s,
int size,
const char *encoding,
const char *errors) |
-
Return value:
New reference.
Create a Unicode object by decoding size bytes of the encoded
string s. encoding and errors have the same
meaning as the parameters of the same name in the
unicode() builtin function. The codec to be used is
looked up using the Python codec registry. Return NULL if an
exception was raised by the codec.
PyObject* PyUnicode_Encode( | const Py_UNICODE *s,
int size,
const char *encoding,
const char *errors) |
-
Return value:
New reference.
Encode the Py_UNICODE buffer of the given size and return
a Python string object. encoding and errors have the
same meaning as the parameters of the same name in the Unicode
encode() method. The codec to be used is looked up using
the Python codec registry. Return NULL if an exception was
raised by the codec.
PyObject* PyUnicode_AsEncodedString( | PyObject *unicode,
const char *encoding,
const char *errors) |
-
Return value:
New reference.
Encode a Unicode object and return the result as Python string
object. encoding and errors have the same meaning as the
parameters of the same name in the Unicode encode() method.
The codec to be used is looked up using the Python codec registry.
Return NULL if an exception was raised by the codec.
These are the UTF-8 codec APIs:
PyObject* PyUnicode_DecodeUTF8( | const char *s,
int size,
const char *errors) |
-
Return value:
New reference.
Create a Unicode object by decoding size bytes of the UTF-8
encoded string s. Return NULL if an exception was raised
by the codec.
PyObject* PyUnicode_DecodeUTF8Stateful( | const char *s,
int size,
const char *errors,
int *consumed) |
-
If consumed is NULL, behave like PyUnicode_DecodeUTF8().
If consumed is not NULL, trailing incomplete UTF-8 byte sequences
will not be treated as an error. Those bytes will not be decoded and the
number of bytes that have been decoded will be stored in consumed.
New in version 2.4.
PyObject* PyUnicode_EncodeUTF8( | const Py_UNICODE *s,
int size,
const char *errors) |
-
Return value:
New reference.
Encode the Py_UNICODE buffer of the given size using UTF-8
and return a Python string object. Return NULL if an exception
was raised by the codec.
PyObject* PyUnicode_AsUTF8String( | PyObject *unicode) |
-
Return value:
New reference.
Encode a Unicode objects using UTF-8 and return the result as
Python string object. Error handling is ``strict''. Return
NULL if an exception was raised by the codec.
These are the UTF-16 codec APIs:
PyObject* PyUnicode_DecodeUTF16( | const char *s,
int size,
const char *errors,
int *byteorder) |
-
Return value:
New reference.
Decode length bytes from a UTF-16 encoded buffer string and
return the corresponding Unicode object. errors (if
non-NULL) defines the error handling. It defaults to ``strict''.
If byteorder is non-NULL, the decoder starts decoding using
the given byte order:
*byteorder == -1: little endian
*byteorder == 0: native order
*byteorder == 1: big endian
and then switches according to all byte order marks (BOM) it finds
in the input data. BOMs are not copied into the resulting Unicode
string. After completion, *byteorder is set to the current
byte order at the end of input data.
If byteorder is NULL, the codec starts in native order mode.
Return NULL if an exception was raised by the codec.
PyObject* PyUnicode_DecodeUTF16Stateful( | const char *s,
int size,
const char *errors,
int *byteorder,
int *consumed) |
-
If consumed is NULL, behave like
PyUnicode_DecodeUTF16(). If consumed is not NULL,
PyUnicode_DecodeUTF16Stateful() will not treat trailing incomplete
UTF-16 byte sequences (i.e. an odd number of bytes or a split surrogate pair)
as an error. Those bytes will not be decoded and the number of bytes that
have been decoded will be stored in consumed.
New in version 2.4.
PyObject* PyUnicode_EncodeUTF16( | const Py_UNICODE *s,
int size,
const char *errors,
int byteorder) |
-
Return value:
New reference.
Return a Python string object holding the UTF-16 encoded value of
the Unicode data in s. If byteorder is not 0
,
output is written according to the following byte order:
byteorder == -1: little endian
byteorder == 0: native byte order (writes a BOM mark)
byteorder == 1: big endian
If byteorder is 0
, the output string will always start with
the Unicode BOM mark (U+FEFF). In the other two modes, no BOM mark
is prepended.
If Py_UNICODE_WIDE is defined, a single Py_UNICODE
value may get represented as a surrogate pair. If it is not
defined, each Py_UNICODE values is interpreted as an
UCS-2 character.
Return NULL if an exception was raised by the codec.
PyObject* PyUnicode_AsUTF16String( | PyObject *unicode) |
-
Return value:
New reference.
Return a Python string using the UTF-16 encoding in native byte
order. The string always starts with a BOM mark. Error handling is
``strict''. Return NULL if an exception was raised by the
codec.
These are the ``Unicode Escape'' codec APIs:
PyObject* PyUnicode_DecodeUnicodeEscape( | const char *s,
int size,
const char *errors) |
-
Return value:
New reference.
Create a Unicode object by decoding size bytes of the
Unicode-Escape encoded string s. Return NULL if an
exception was raised by the codec.
PyObject* PyUnicode_EncodeUnicodeEscape( | const Py_UNICODE *s,
int size,
const char *errors) |
-
Return value:
New reference.
Encode the Py_UNICODE buffer of the given size using
Unicode-Escape and return a Python string object. Return NULL
if an exception was raised by the codec.
PyObject* PyUnicode_AsUnicodeEscapeString( | PyObject *unicode) |
-
Return value:
New reference.
Encode a Unicode objects using Unicode-Escape and return the
result as Python string object. Error handling is ``strict''.
Return NULL if an exception was raised by the codec.
These are the ``Raw Unicode Escape'' codec APIs:
PyObject* PyUnicode_DecodeRawUnicodeEscape( | const char *s,
int size,
const char *errors) |
-
Return value:
New reference.
Create a Unicode object by decoding size bytes of the
Raw-Unicode-Escape encoded string s. Return NULL if an
exception was raised by the codec.
PyObject* PyUnicode_EncodeRawUnicodeEscape( | const Py_UNICODE *s,
int size,
const char *errors) |
-
Return value:
New reference.
Encode the Py_UNICODE buffer of the given size using
Raw-Unicode-Escape and return a Python string object. Return
NULL if an exception was raised by the codec.
PyObject* PyUnicode_AsRawUnicodeEscapeString( | PyObject *unicode) |
-
Return value:
New reference.
Encode a Unicode objects using Raw-Unicode-Escape and return the
result as Python string object. Error handling is ``strict''.
Return NULL if an exception was raised by the codec.
These are the Latin-1 codec APIs:
Latin-1 corresponds to the first 256 Unicode ordinals and only these
are accepted by the codecs during encoding.
PyObject* PyUnicode_DecodeLatin1( | const char *s,
int size,
const char *errors) |
-
Return value:
New reference.
Create a Unicode object by decoding size bytes of the Latin-1
encoded string s. Return NULL if an exception was raised
by the codec.
PyObject* PyUnicode_EncodeLatin1( | const Py_UNICODE *s,
int size,
const char *errors) |
-
Return value:
New reference.
Encode the Py_UNICODE buffer of the given size using
Latin-1 and return a Python string object. Return NULL if an
exception was raised by the codec.
PyObject* PyUnicode_AsLatin1String( | PyObject *unicode) |
-
Return value:
New reference.
Encode a Unicode objects using Latin-1 and return the result as
Python string object. Error handling is ``strict''. Return
NULL if an exception was raised by the codec.
These are the ASCII codec APIs. Only 7-bit ASCII data is
accepted. All other codes generate errors.
PyObject* PyUnicode_DecodeASCII( | const char *s,
int size,
const char *errors) |
-
Return value:
New reference.
Create a Unicode object by decoding size bytes of the
ASCII encoded string s. Return NULL if an exception
was raised by the codec.
PyObject* PyUnicode_EncodeASCII( | const Py_UNICODE *s,
int size,
const char *errors) |
-
Return value:
New reference.
Encode the Py_UNICODE buffer of the given size using
ASCII and return a Python string object. Return NULL if an
exception was raised by the codec.
PyObject* PyUnicode_AsASCIIString( | PyObject *unicode) |
-
Return value:
New reference.
Encode a Unicode objects using ASCII and return the result as
Python string object. Error handling is ``strict''. Return
NULL if an exception was raised by the codec.
These are the mapping codec APIs:
This codec is special in that it can be used to implement many
different codecs (and this is in fact what was done to obtain most of
the standard codecs included in the encodings package). The
codec uses mapping to encode and decode characters.
Decoding mappings must map single string characters to single Unicode
characters, integers (which are then interpreted as Unicode ordinals)
or None (meaning "undefined mapping" and causing an error).
Encoding mappings must map single Unicode characters to single string
characters, integers (which are then interpreted as Latin-1 ordinals)
or None (meaning "undefined mapping" and causing an error).
The mapping objects provided must only support the __getitem__ mapping
interface.
If a character lookup fails with a LookupError, the character is
copied as-is meaning that its ordinal value will be interpreted as
Unicode or Latin-1 ordinal resp. Because of this, mappings only need
to contain those mappings which map characters to different code
points.
PyObject* PyUnicode_DecodeCharmap( | const char *s,
int size,
PyObject *mapping,
const char *errors) |
-
Return value:
New reference.
Create a Unicode object by decoding size bytes of the encoded
string s using the given mapping object. Return
NULL if an exception was raised by the codec.
PyObject* PyUnicode_EncodeCharmap( | const Py_UNICODE *s,
int size,
PyObject *mapping,
const char *errors) |
-
Return value:
New reference.
Encode the Py_UNICODE buffer of the given size using the
given mapping object and return a Python string object.
Return NULL if an exception was raised by the codec.
PyObject* PyUnicode_AsCharmapString( | PyObject *unicode,
PyObject *mapping) |
-
Return value:
New reference.
Encode a Unicode objects using the given mapping object and
return the result as Python string object. Error handling is
``strict''. Return NULL if an exception was raised by the
codec.
The following codec API is special in that maps Unicode to Unicode.
PyObject* PyUnicode_TranslateCharmap( | const Py_UNICODE *s,
int size,
PyObject *table,
const char *errors) |
-
Return value:
New reference.
Translate a Py_UNICODE buffer of the given length by
applying a character mapping table to it and return the
resulting Unicode object. Return NULL when an exception was
raised by the codec.
The mapping table must map Unicode ordinal integers to Unicode
ordinal integers or None (causing deletion of the character).
Mapping tables need only provide the method__getitem__()
interface; dictionaries and sequences work well. Unmapped character
ordinals (ones which cause a LookupError) are left
untouched and are copied as-is.
These are the MBCS codec APIs. They are currently only available on
Windows and use the Win32 MBCS converters to implement the
conversions. Note that MBCS (or DBCS) is a class of encodings, not
just one. The target encoding is defined by the user settings on the
machine running the codec.
PyObject* PyUnicode_DecodeMBCS( | const char *s,
int size,
const char *errors) |
-
Return value:
New reference.
Create a Unicode object by decoding size bytes of the MBCS
encoded string s. Return NULL if an exception was
raised by the codec.
PyObject* PyUnicode_EncodeMBCS( | const Py_UNICODE *s,
int size,
const char *errors) |
-
Return value:
New reference.
Encode the Py_UNICODE buffer of the given size using MBCS
and return a Python string object. Return NULL if an exception
was raised by the codec.
PyObject* PyUnicode_AsMBCSString( | PyObject *unicode) |
-
Return value:
New reference.
Encode a Unicode objects using MBCS and return the result as
Python string object. Error handling is ``strict''. Return
NULL if an exception was raised by the codec.
Release 2.4.2, documentation updated on 28 September 2005.
See About this document... for information on suggesting changes.