QMTest: A Software Testing Tool

2.1. Domain

A test class contains machinery for executing a test, deciding whether it passed or failed, and reporting the results of the test. Every test class takes arguments that parameterize its behavior. A test is an instance of a test class, i.e., a test class together with arguments to that test class.

QMTest comes with a number of test classes to handle typical testing situations. For example, QMTest comes with a test class (called ExecTest) that spawns a process and checks its exit status. If the command exits with a zero exit status, the test passes; otherwise, the test fails. Every test class takes arguments that parameterize its behavior; for example, ExecTest takes parameters that indicates the name of the program to spawn and the command-line arguments that should be provided to it.

In theory, ExecTest is capable of testing any domain. Any test can be written as an executable that contains all the testing logic and then exits successfully if (and only if) the test passes. Of course, trying to perform testing using this crude method is painful in a variety of respects. First and foremost, it is not necessarily convenient to create separate executables that contain testing logic. In addition, there is no good way for the executable to report information about why a test failed. Nor is there a good way for QMTest to assist users in creating new tests, since creating a new test requires creating an entire new executable.

To solve these problems, an extender may create a custom test class. A test class is a Python class that implements a particular interface. The interface consists of one method and one attribute:

It is important to note that this interface does not specify a storage format for the test. In QMTest, test storage is orthogonal to the domain tested. Tests written using new test classes can still be stored by QMTest; the default storage mechanism simply records the values of all arguments provided to the test class.

class ExecTest(Test):
    """Check that a Python expression evaluates to true.

    An 'ExecTest' test consists of Python source code together with
    an (optional) Python expression.  First the Python code is
    executed.  If it throws an (uncaught) exception the test fails.
    If the optional expression is present, it is then evaluated.  If it
    evaluates to false, the test fails.  Otherwise, the test passes."""

    arguments = [
        qm.fields.TextField(
            name="source",
            title="Python Source Code",
            description="""The source code.

            This code may contain class definitions, function
            definitions, statements, and so forth.  If this code
            throws an uncaught exception, the test will fail.""",
            verbatim="true",
            multiline="true",
            default_value="pass"
            ),

        qm.fields.TextField(
            name="expression",
            title="Python Expression",
            description="""The expression to evaluate.

            If the expression evaluates to true, the test will pass,
            unless the source code above throws an uncaught exception.

            If this field is left blank, it is treated as an expression
            that is always true.""",
            verbatim="true",
            multiline="true",
            default_value="1"
            )
        ]


    def __init__(self, **properties):

        apply(Test.__init__, (self,), properties)
        
        # Store stuff for later.
        if self.source is None:
            self.source = ""
        else:
            # Make sure the source ends with a newline.  A user is
            # likely to be confused by the error message if it's
            # missing. 
            if self.source[-1] != "\n":
                self.source = self.source + "\n" 


    def Run(self, context, result):
        global_namespace, local_namespace = make_namespaces(context)
        # Execute the source code.
        try:
            exec self.source in global_namespace, local_namespace
        except:
            # The source raised an unhandled exception, so the test
            # fails
            result.NoteException(cause="Exception executing source.")
        else:
            # The source code executed OK.  Was an additional expression
            # provided? 
            if self.expression is not None:
                # Yes; evaluate it.
                try:
                    value = eval(self.expression,
                                 global_namespace, local_namespace)
                except:
                    # Oops, an exception while evaluating the
                    # expression.  The test fails.
                    result.NoteException(cause=
                                         "Exception evaluating expression.")
                else:
                    # We evaluated the expression.  The test passes iff
                    # the expression's value is boolean true.
                    if not value:
                        result.Fail("Expression evaluates to false.",
                                    { "ExecTest.expr" : self.expression,
                                      "ExecTest.value" : repr(value) })
            else:
                # No expression provided; if we got this far, the test
                # passes. 
                pass
   

2.2. Storage

By default, QMTest represents each test as an Extensible Markup Language (XML) [XML] file. For each test, QMTest stores the name of the Python test class and representations of each of the arguments to the class. This representation is fully general. XML is a good format to use in that it is rigorously defined, standardized, and well supported in most programming languages. (For example, it is easy to mechanically generate QMTest files from a tool written in C++ by emitting XML.) In addition, the fact that XML is a textual format makes it easy to store in any revision control system.

While XML is a textual format, it is not always easy for people to read and edit. XML is also somewhat bulky; in some cases, the XML tags can consume large amounts of space. Encoding binary data in XML requires converting it to a textual format. Therefore, the efficiency of QMTest can be adversely impacted by the cost of encoding and decoding binary data in the test files. For some applications, a different storage format can overcome these problems.

For example, it might be more efficient to store large testsuites in a relational database, rather than as files in the filesystem. Users could then perform operations on the tests that QMTest does not support directly, such as the retrieval of all tests created after a particular date.

In situations such as these, extenders can create a customized test database class. A test database is an instance of such a class. The test database is responsible for storing and retrieving tests; it maps test identifiers to the tests themselves. How it performs its task is entirely up to the implementor of the test database class.

Converting an existing testsuite to QMTest can also be accomplished by means of a customized test database class. The test database class can parse the existing test format and create test instances that QMTest can execute. When using this approach, there is no need to convert each individual test to QMTest's XML format.

Test databases can be used in ways that do not directly relate to storage. For example, QMTest comes with a database class that can combine multiple test databases into a single database. This database class is useful for testing of a number of components at once, each of which has its own test database. For example, an operating system distributor must integrate several hundred packages to create a GNU/Linux distribution. If these packages had their own QMTest test databases, the distributor could easily run all of the tests for all of the packages and get a single report on the status of the complete system.

The user of QMTest does not need to understand how tests are stored. For example, QMTest's graphical user interface will appear exactly the same to a user who uses the default test database and to a user who uses a test database built atop a relational database. From the user's point of view, creating a new test is simply a matter of choosing a test class and then providing arguments to that class. [1]

2.3. Schedule

The default QMTest execution engine executes tests sequentially on a single machine. However, QMTest also comes with an execution engine that is capable of spreading the test load across a large testing farm. This feature is particularly useful to organizations that have access to supercomputers, or large clusters of machines, but it is also useful to developers who have a symmetric multiprocessor on their desks.

Scheduling jobs, especially on machines with varying loads and varying resource characteristics, is a notoriously difficult problem. In addition, the means used to connect to a remote machine varies from environment to environment; while rsh is standard on many systems, it is not available under Microsoft Windows, and even on UNIX systems it may be more efficient to use the Message Passing Interface (MPI), threads, or some other means of obtaining parallelism. Therefore, QMTest allows extenders to override the default parallel execution engine with a customized version.

The execution engine need not be aware of the test class implementation or storage format; it is simply responsible for allocating test instances to machines. Therefore, it is possible for the administrators of a high-performance supercomputer to provide a test execution engine that can then be used by all users on the machine.

2.4. Display

QMTest can produce reports about test execution in a number of different formats. For example, QMTest can simply produce a report that indicates which tests passed and which tests failed. Alternatively, QMTest can indicate how the tests behaved relative to some previous test run. This mode of output is useful in determining whether some recent changes have caused new problems; if the tests that failed also failed the last time the tests were run, the new changes are likely not the cause of the problems.

In many situations, just as interesting as knowing that a test failed is knowing why it failed. For example, when testing a compiler, did the test fail because the compiler crashed, because it produced an incorrect error message, or because the compiler generated incorrect code? Even if the test did not fail, there may be other interesting information associated with the test execution, such as the amount of time the test took to execute.

As discussed above, the test class can embed this information in the result object that it returns to QMTest. Then, QMTest will display this information for the user. If the test class wishes to control not only the information displayed, but also its formatting, the test class can provide a method that formats a result object for output. In that way, the test class can, for example, provide hyperlinks from information embedded in the failure description to a manual that explains the likely causes of the failure.

By default, QMTest stores results as XML, with the goal that other applications may be written to process the results and display them in interesting ways. For example, it is likely that a future version of QMTest will contain a script that takes a series of results files and produces a Microsoft Excel spreadsheet showing test failures over time. However, users can replace the module that performs results storage in the same way that they can replace display modules, so that, for example, results can be stored directly in a relational database.