Lore is a documentation generation system which uses a limited subset of XHTML, together with some class attributes, as its source format. This allows for lower barrier of entry than many other similar systems, since HTML authoring tools are plentiful as is knowledge of HTML writing. As an added advantage, the source format is viewable directly, so that even if Lore is not available the documentation is useful. It currently outputs LaTeX and HTML, which allows for most use-cases.
Lore is currently in use by the Twisted project to generate its documentation for versions 1.0.1 and above.
At the beginning of Twisted's life cycle, as with any self-respecting free software project, it came completely devoid of documentation. As Twisted progressed in maturity, the Twisted development team realized that documentation is necessary.
Since at that time the Twisted development
team did not want the overhead of integrating
a full-scale document generation framework into its build infrastructure,
documents were written for the least common denominator -- plain HTML.
When the Twisted team wanted the documentation to be
featured on the web site, it was desirable to have them integrated with
the web site's look and feel. Thus, generate-domdocs
was born as a simple XML-based command line hack which improved the look of the
documents so they would share the look and feel of the other pages in the web
site, including a standard header and footer. As
generate-domdocs
slowly grew more and more features, it gradually became too large to maintain.
The authors, members of the Twisted development team, decided that in order to
make it more maintainable, it should be refactored into a
library and by the way also add alternate output formats. Some of the documents
which were reluctant to be transformed into alternate formats were fixed,
and guidelines for making compatible documents were drafted. Those documents,
together with the conversion code, are the Lore documentation generation
system.
Lore is documentation generation system which is a part of the
Twisted framework. It uses
the Twisted XML parsing framework
(microdom) to parse compliant XHTML
and generate the various output formats from it.
Lore consists of a Python package, twisted.lore,
and a command-line program: lore, which
generates HTML output (which is more presentation-oriented than the source
format), LaTeX or runs an linter, depending on command-line arguments.
In the case where the default output of Lore is not exactly suited to a
Lore user,
it is possible to subclass the output generators and customize their behavior.
This could be done for many purposes, from straight-forward additions like
adding a new span or div class to advanced tweaking
such as changing the way Lore does image conversion on LaTeX output.
Lore uses reflection intensively to make adding new features as simple
as adding a new method, without the need for awkward registration schemes.
Thus, adding another check to the linter or letting
Lore handle the link element in some way require only the addition
of one method.
Lore was written when the Twisted team felt it needed to write documentation
and looked for a documentation format. Looking through alternatives, the
best one seemed to be the Python way, using LaTeX format and
latex2html. However, the Python way has its share
of problems, not the least of which is latex2html
being a long and crufty Perl program whose Perl APIs, which are the
only way to add support for custom markup, change every version.
Since documentation writing is important, a documentation system with minimal impact on the writer would be desirable. While LaTeX certainly has very little impact in terms of markup overhead, it has a very big impact both in terms of installed base (installing LaTeX on UNIX systems or Windows is non-trivial at best) and in terms of familiarity.
HTML has the benefit of being directly readable on every post-1995 computer, so the installed base is as big as could be hoped for. It also has the benefit of being easily parsed, at least in its new XHTML guise.
The goals of Lore were taken to be:
Lore's source format is a subset of XHTML; all Lore source documents are
valid XHTML documents. The XHTML tags that Lore allows are:
html, title, head, body,
h1, h2, h3, ol,
ul, dl, li, dt,
dd, p, code, img,
blockquote, a, cite, div,
span, strong, em, pre,
q, table, tr, td,
th and style.
We would like to stress the omission of the font tag (which is
deprecated in HTML 4.01 anyway). Instead of using font, Lore
mandates the use of stylesheets
and the class attribute, and in particular Lore defines several
classes, such as footnote, API,
py-listing. The use of classes on div and
span elements effectively allows XHTML to be arbitrarily
extensible without needing to define custom tags.
Further discouraging explicit style decision, Lore deprecates the
style attribute which allowing HTML (and XHTML) authors to embed
pieces of the stylesheet in the document. Though Lore properly processes
such documents, they are against the specification of Lore -- and
the Lore lint-like problem finder will complain.
Requiring XHTML rather than just HTML greatly simplifies the code to manipulate Lore source, because we can use standard XML libraries. For documentation authors, the difference is negligible -- and any mistakes made in balancing tags can be easily found using the linter. Since tag balancing problems, in many cases, cause a discrepancy between author intention and the result, it is better to balance the tags anyway.
Like LaTeX, Lore encourages authors to focus on content, letting the presentation take care of itself. This is an inherently restrictive approach, but results in much more consistent and higher-quality output.
The Lore source format is quite usable (if somewhat plain) as an end-format. Any web browser can read it, and it does not require special stylesheet support, JavaScript or any other modern HTML additions. It is also, as intended, straightforward to create and edit documents in this format.
However, reading the source format directly has some major limitations, which are inherent in the combination of the facilities which render HTML and the requirement that the format will be easily writable, and easy to modify, using any standard text editor. The limitations include:
The two most important formats, for the end-user, are the computer screen and pages of print outs. Any other format should be first and foremost be thought of as a prelude to these final formats.
The easiest computer-screen oriented format is HTML. However, the HTML which is most comfortable and useful to the end-user is not necessarily easy to write and modify. For example, it is painful to manually write a table of contents, and even more painful to keep it updated as sections are added, removed or changed. However, when reading a long document having a table of contents, with hyperlinks into the sections, is a boon. Thus, even though both Lore's source and one output format are HTML, an HTML to HTML conversion is still necessary, paradoxical though it may sound.
For printable output, the most widely supported formats are PostScript and Portable Document Format. On UNIX systems PostScript is often preferred, since there are many tools for manipulating it and printing it (and PostScript printers are more common in the UNIX world). On Windows and Apple computers, Portable Document Format (PDF) is preferred because of the ease of installation of the necessary tools. Mac OS X, though being technically a UNIX, supports PDF natively.
Directly generating PostScript or PDF, however, is hard. Since these formats
are very low-level, the application generating them must do the hard work
of calculating line breaks, guessing hyphenation points and deciding on fonts.
Since these tasks are already implemented by LaTeX, Lore just generates LaTeX
code and lets the user run LaTeX to generate PostScript and
ps2pdf to generate PDF. Granted, this still causes
the problems with the difficulties of installing LaTeX. It is
possible to implement direct Lore to PDF converter, though this hasn't been
done yet, by using pdflib.
The HTML to HTML converter works by running a series of transformations on
the Document Object Model (DOM) tree of the parsed document, and then
writing it out. The most important transformation is that of throwing
away anything outside the body element, and putting the
body element inside a template file. This allows large
parts of the common layout code to be customized without modifying or writing
any Python code.
Each step is implemented as a separate function, to allow Lore-using Python programmers to customize which tree transformations to do in their own code, without forcing them to rewrite functionality in Lore. In addition, other output generators might perform a subset of these transformations on the input tree before processing it -- and indeed, this is being used even in Lore itself.
One of the steps taken is caused by a need which is common in large Python frameworks: many of the class or module names are deeply nested, but are commonly referred to by just their last one or two components in writing. However, the user would like to know the full name of the class or module name, and where to look up the API documentation -- but without having the complete name thrust upon him during the flow of text each time the module is mentioned.
Lore makes sure that each class or module name which is mentioned will appear at least once using its full name, and afterwards use a common short name, regardless of how the author wrote it up. This frees authors from needing to observe, manually, this useful rule in their documents.
The HTML Lore outputs aims to be the poster boy of graceful degradation.
Thus, for example, while footnotes always appear as hyper-links to the footnote
text, browsers which respect the title attribute (which is usually
rendered as a tooltip) will also show the beginning of the footnote while
hovering above the hyper-link.
Lore avoids using the font
or color
tags and attributes,
preferring to use HTML classes and using a stylesheet to specify graphical
design decisions. This allows the Lore user to customize the presentation of
the output without touching Python code. Since most often the stylesheet
link is found in the head element, this is determined by
the by the template.
Lore uses the same approach even for syntax-highlighting Python code,
generating such elements as
<span class="keyword">if</span>.
The LaTeX home page describes LaTeX as a high-quality typesetting system,
with features designed for the production of technical and scientific
documentation.
LaTeX is very popular for generating printable content,
building on Donald Knuth's TeX system to generate nearly optimal output
by putting together much of the typesetting industry's experience in the
form of a program and adding sophisticated algorithms for line-breaking and
hyphenation.
It is very common for document generation systems to avoid generating printable output themselves, instead letting LaTeX do the hard work, and Lore is no exception.
Lore can output LaTeX in two modes: article mode, in which it generates
a complete article ready to be be processed, and a section mode in which
it generates a LaTeX file whose top-level element is a section. Such a file
is usually included in some other LaTeX file via the include mechanism.
Twisted itself uses mainly the section mode, and includes everything in the
file book.tex, which is later processed to generate
the Twisted book.
While, conceivably, other modes could be done (a chapter mode or a subsection mode) there has not been any demand for those. In the case of demand, supplying these would be very few lines of Python code (less than 10), which can even be done by subclassing existing classes and avoiding the modification of Lore itself.
Docbook output is currently experimental. Its chief use to Lore would be in generating Texinfo, which is the source for the GNU info documentation format.
Very early in the Lore development life-cycle it was found that a good Lint-like tool is necessary to find errors without necessitating a full compilation to all formats and sometimes even browsing the results. Because Lore was written to accommodate a large set of already existing documents (which were not previously checked for potential problems), such a tool was very useful so that finding a problem in one document would not mean this problem needs to be manually searched, and corrected, in all the other documents.
Lore's linter tries to find problems in documents that would either stop the conversion to other formats by Lore completely (for example, by being not well-formed XML), or that would make it less useful (for example, by warning about tags or classes that are not supported by Lore).
The linter even detects more exotic problems, including:
pre elements containing lines over 80 characters. Long lines
can be ugly to render in some output formats, and even impossible to
render in others." character in a non-pre or non-code
environment. This makes a big difference for high-quality typographical
output targets like LaTeX, which
have distinct left- and right-quote characters.
for x in sequence:
...
This check caught a surprisingly large number of errors in the Twisted
documentation!h1 contents being equal to title contents.
HTML is somewhat unique in that it has two places to specify the logical
idea of title. Since other output formats do not support that, in Lore papers, the contents of both must be the same.
Since many of the incremental improvements done to Lore found a problem
in the existing documentation files, the linter has been
an important part of the Lore development effort. One may even argue that
part of the reason other documentation generation systems produce suboptimal
output for their non-native
application is the lack of a linting
tool.
Finally, if the linter gives a false positive, that is
it emits a warning for something that isn't a problem in a particular situation,
the user can add an hlint="off" attribute to the offending tag, and
the linter will ignore it. This is necessary only very rarely.
The chief design decision made in the linter, after
painful experience when running tidy, is that
it must never change the document. Thus, while the linter
will be as pedantic as possible finding
errors, it never changes the contents. This is particularly important
when dealing with version control systems, where spurious changes can
render diff listings useless.
All existing syntax highlighters for Python used pre-tokenize
techniques to analyse the Python code. As a result, they were cumbersome
and non-standard. The Lore developers decided that writing a Python
HTML syntax-highlighter would be easier than modifying one of the existing
ones. A syntax-highlighter was built on top of a null-tokenizer: that is,
a tokenizer which emits the exact same characters as the input.
This allowed easy debugging of the parsing code.
The only non-trivial code in the syntax highlighter is when dealing with whitespace which is not significant syntactically, since the tokenizer does not report it. However, since the tokenizer does report row and column, when the code sees a discrepancy between where the previous token ended and the current token starts, it adds whitespace to make up for the discrepancy.
When writing out the HTML, the only difference between that and the
null-tokenizer is the wrapping of each token by a span
tag with the appropriate class and escaping.
Note that the basic Python tokenizer does not distinguish between the
various roles of the production NAME
(that is, a string of alphanumeric
and
underscore characters starting with an underscore or a letter) in Python.
The tokenizer Lore uses adds that information by having a simple state machine:
if the word is a keyword, there is nothing to be determined; otherwise, it
depends on the last detected name -- class or
def mean it is a function or class names, and after a
class/def and until a :, everything
is a parameter
or a superclass.
The Python syntax highlighter Lore uses can be found in the
twisted.python.htmlizer.
Often, when writing detailed documents, the author wishes to test his
examples or even use examples from a working project. Pasting such examples
directly into the HTML has both the usual problems of pasting code -- the
version in the document will not benefit from bug fixes or enhancement to
the original version -- and the problem that the HTML needs proper escaping,
which is a tedious and error-prone procedure if done manually.
Both problems are solved by Lore's listing
mechanism. The
listing
mechanism converts HTML such as
<a href="foo.py" class="py-listing>foo.py</a>
into inclusion of the foo.py file. It will always
be properly escaped for whatever output format. It will
also be syntax-highlighted, just as if it had been included verbatim.
A similar class, html-listing is available for inclusion
of HTML files.
Twisted's documentation frequently references API documentation. In Lore,
the name of an API such as
twisted.internet.defer.Deferred is marked up as
<code class="API" base="twisted.internet.defer">Deferred</code>
This will unambiguously link to
twisted.internet.defer.Deferred, even though it is
displayed as
. Lore
produces API links that work with
epydoc,
but could easily be adapted for another API documentation generator; in fact,
Lore originally worked with happydoc.
In addition, in the HTML output, Lore will add a Deferredtitle
attribute to the API reference, containing the full name of the link.
A collection of documents will typically refer to each other, for instance to avoid re-explaining some central concept. In HTML, cross-referencing is implemented as linking:
See <a href="defer.html">Deferring Execution</a>.
As a collection of HTML documents, this works with no changes. Other output
formats do linking in other ways. When Lore is used to convert a collection of
source HTML files into a single LaTeX book, each file is its own section, and
the links are automatically converted into cross-references. Thus the example
above might be rendered as See Deferring Execution (page 163).
Lore also recognizes fragment identifiers in links, so that a link
to glossary.html#psu will be cross-referenced to that part of the
glossary named psu
, not just the whole glossary. This ensures that the
page the reader is referred to is the correct one.
Man pages are a fact of life on UNIX, and every self-respecting command line program is expected to come with one. The man format, implemented as troff macros, is somewhat arcane. Since, when Lore was written, we already had written man pages, the decision was to convert them to HTML rather than try to rewrite them in HTML and design a man output format.
A limited parser for man pages is available in the
twisted.lore.man2lore module. It is not yet
exposed via any public command line program.
Earlier attempts, using groff -Thtml to
generate HTML and then post-process it into Lore-compatible HTML
were crufty and unmaintainable. It seems the man format shares some
of LaTeX's problem: being written as a macro package over a powerful
processor, it is too flexible for its own good. Fortunately, the subset
normally used in man pages is quite small, so heuristically parsing man pages is
much easier than the same task with LaTeX.
HTML, when invented by Tim Berners-Lee, was meant to be a simple language for writing and sharing documents. With the explosion of the web, HTML has grown to a confusing jumble of logical and presentation features, with more layers, such as CSS, dumped on top of it. As a result, a modern browser is a complicated beast. That given, it is perhaps understandable that today's browsers do a sub-standard job at printing. Thus, while being extremely well suited to the world wide web, HTML is significantly lacking, at least in today's application market, when it comes to paper output. It might be possible to write an application to properly convert HTML with CSS to PostScript or PDF -- however, it would probably be much more complicated than Lore. Moreover, the portability of such an application would be worse of the portability of Lore itself, which currently only depends on Python 2.1 or higher and the Twisted framework.
Limiting HTML to a small subset of features enables Lore to be small
and readable while remaining useful. By including the class
attribute among those features, Lore is also extensible.
When it comes to paper output, LaTeX cannot be out done except by a skilled typesetter designing and implementing. However, the architecture of LaTeX presents significant problems when trying to view LaTeX online. LaTeX is written as a macro layer above TeX rather than a preprocessor. Thus, all of TeX's power is available, and sometimes used, in LaTeX. TeX is non-trivial to parse and format by anyone short of Donald Knuth -- it contains such commands as to change the tokenizer by modifying which characters are considered word characters or even which character is the command character. In fact, the authors are not aware of any application which handles the full power of TeX without being based on the original TeX code.
All this makes LaTeX extremely difficult to parse, and even partial attempts
to parse LaTeX are big and cumbersome -- for example,
latex2html. It is thus difficult to convert
LaTeX to something appropriate to online viewing.
LyX's internal source format is not well documented, and the only supported way to write it is using the LyX GUI. Thus it is inherently limiting to documentation authors. In addition, it is not trivial to write LyX preprocessors to save documentation authors tedious work.
Docbook is a big standard, with non-trivial to install tool-set. Writing Docbook is different than most other document generation formats, so it takes significant training to write. In addition, using Docbook for a specific project usually requires writing custom DSSSL stylesheets in a scheme-like language, and additional XML DTD snippets. Writing these was quite possibly comparable to writing Lore, and Lore has the advantage of being written in Python.
Texinfo imposes a significant effort on authors. Many things need to be written twice, and the error messages leave a lot to be desired. After starting to work on the Lore texinfo output format the authors are grateful they have never had to write Texinfo by hand.
When generating LaTeX, Lore does it via a visitor pattern while visiting
the nodes. A node which does not have a specific visitor is visited by
first writing the start_ attribute, then visiting its
children and then writing the end_ attribute. If the attributes
do not exist, they are treated as though they were empty strings.
That code allows most of the HTML elements to LaTeX converters to have no code -- only a pair of strings -- while the elements converters which need more sophisticated programming can do it via defining a method, which can still call the default processor if it needs this functionality.
This pattern is also friendly to subclassing: all a subclass needs to do in order to change how an element is handled is to define either a pair of class attributes or a method.
In the above example of the visitor pattern, registration of the methods
and attributes is avoided thanks to using the crudest form of reflection
in Python -- the getattr() function.
In the Lint support tool, more sophisticated reflection is needed when
it needs to find all methods whose name begins with check_.
This is done via the Twisted reflection code, built on top of the native
Python facilities, in the module
twisted.python.reflect.
In the HTML output code, the most common operation is that of getting
a list of elements which satisfy some property. This is done by one
primary work-horse function:
twisted.web.domhelpers.findNodes. This function
accepts a DOM tree and a function, and returns a list of all elements
for which this function returns true. Using this, and the fact that Python makes
it easy to combine functions into boolean combinations, makes analysis
and modification and of the DOM tree a breeze.
Probably the trickiest thing about non-HTML output formats is escaping. The problem comes from two annoying problems which are not really hard to solve, but do represent annoyances in the code:
\
is escaped, in TeX, as $\backslash$ while most other
characters are escaped as \<char>.pre, </> should not be
escaped inside code and should be escaped as
$<$/$>$ outside it.In Docbook, the sections are nested, so there is only need for
a title element. However, in HTML only the headers care
at which level they are. This requires the Docbook converter to keep
the last header level and when it reaches a new header, to close and open
enough sections so the header will get to the correct level. While Docbook's
way may be more correct
, it is unfortunate it chose to diverge from
all other systems here.
Texinfo requires all the sections in a document will have unique names. This makes it very inconvenient as both an input and an output format.
Also, differing significance of whitespace in different formats requires that all whitespace emitted by lore must be normalized for the particular output format being used. Blank lines which have no impact on HTML will trigger paragraph breaks in LaTeX.
The first version of the LaTeX output generator was using an event-based
XML parsing engine. It quickly turned out one needs to keep a lot of
information in stacks and manage many instance variables. For example,
though XML gets the name of the closing element (even that is arguably
too much information), it does not get the attributes. In span
elements, for example, the interesting information is the class
attribute. Since a-priory, spans might be nested, the class
needs to keep a stack of attribute collections.
Quite soon, stacks were needed for proper handling of div
tags and for determining proper quoting formats. Moreover, getting the
code to function correctly in the face of edge cases, such as cross-references
inside pre tags, proved to be quite a challenge.
The code was shortened, simplified and became more maintainable when
it was moved to microdom.
We feel that unless there is an inherent reason to do XML event-based parsing, then it is much easier to read the whole thing into a DOM and then process it. The code is both shorter and clearer, and features are much easier to add.
Lore, out of the box, does not attempt to be all things to all people. Particularly in the LaTeX output format, there is a lot of room for interpretation and personal preferences. Lore chose one specific way, without trying to add half a dozen options to tweak it. However, thanks to the way it is coded, it is easy to add or modify features to suit individual preferences. Many customizations only involve adding or overriding simple data attributes to a subclass; more advanced changes require adding or overriding methods.
Likewise, the HTML output is built by running several tree-modification functions which are independent. Completely different HTML output could be build by adding more functions, or not running some of those which are being run.
We already know of multiple users that have extended Lore for custom LaTeX generation. In each case it was a simple matter of subclassing Lore's LaTeX code.
Documentation generation systems were already a solved problem before Lore was written. However, we know of no system with Lore's unique combination of features -- in particular, portability, having a directly readable source format which is also directly writable in text editors. The common wisdom that a documentation generation system is a hard sell because it requires people to learn a new language was refuted by using an existing language.
Wheel reinvention also occurred in a nearby area -- Twisted's XML support,
for which Lore is one of the biggest users. Again, the common wisdom was that
this was a solved problem, with many existing DOM and SAX implementations.
However, implementation of some features, no implementation of other features
and API instability have lead the Twisted team to write its own, highly
pythonic, DOM-like implementation. In
microdom, the aim is to be
as thin a wrapper over the basic Python wrappers as possible. This feature
has been used to the full in Lore, where many of the tree manipulations
would have been much more cumbersome had a standard opaque
DOM
implementation been used. In addition, using
microdom frees Lore from the
dependence on both Python version and whether PyXML is installed.
For example, microdom
exposes the list of child nodes as a plain Python lists. This means that
not only all the list operations can be done of it, which could possibly
be simulated by a list-like object, but that it is possible to
replace it by our own list. As another example,
microdom allows us to freely
copy nodes from one DOM tree into another.
Python, as a language well suited to rapid application development, acts as a way to make wheel reinvention far from the horrible mistake which is portrayed in the common software engineering folklore. Indeed, Python makes it easy enough to reinvent wheels that only the best, and easy to use, wheels, get reused at all.
Lore can be found in Twisted 1.0.1 and higher, in the
twisted.lore package. When you install the package,
the relevant script, lore,
should be installed in a sane directory,
as determined by distutils.
For usage examples, see admin/release-twisted
in the Twisted source distribution. It runs the various Lore scripts
as part of the package build.
It would be nice to have the Docbook output fully working. It would also
be nice to have Texinfo in full working order so that GNU info aficionados could
read the documents with the info browser. As suggested above, it might
also be useful to have a way to directly generate PDF output via
pdflib in order to skip LaTeX.
In addition, another potential output format is to have high-quality
text output. This is non-trivial, but possibly useful: browsers'
Save as text
feature is usually implemented as an afterthought,
and hardly uses the flexibility available in the text format to its
full power. The authors are unaware, for example, of an HTML to text
converter which uses the underlining with =
sign or -
to indicate a header, or which uses the /slant/ or
*asterisk* conventions to indicate emphasis.
Another output format we are considering is a split-page HTML with interlinks, so that long documents can be converted into something which is web-friendly. One nice use for that would be in web-based presentations.
Currently all images are converted to EPS format. It would be nice to have the LaTeX converter try to see if there is already an EPS version, via some naming convention, and use that. This would allow better scaling of things like Dia diagrams. The versions in bitmap-based formats (such as PNG) are impossible to scale, because the text would become unreadable.
Currently, the only interface to Lore is through the command-line, and even that is somewhat spotty: for example, the man page parser is not directly available via the command line. We hope to remedy that, having at least a full suite of command-line tools and possibly graphical wrappers, particularly EMACS modes.
When starting with a historical note, it is only fitting to end with a historical note. Since the writing of Lore, Twisted documentation is successfully generated by it and distributed in the tarball. It contains generated HTML from the HOWTO documents, specifications and man pages. It also contains all these documents inside a LaTeX-generated PostScript file and PDF file in an easy to print format, suitable for reading on those long plane flights or train rides.
Lore is also used to generate pages with consistent headers and footers for the twistedmatrix.com web site -- not just the Twisted documentation. This is shows the inherent flexibility in Lore's model of being easily configurable via an HTML template, a feature which none of the major document generation systems support for their HTML output.
lore(1) man page