QMTest: A Software Testing Tool

Mark Mitchell
CodeSourcery, LLC

Jeffrey Oldham
CodeSourcery, LLC

Greg Wilson
Baltimore Technologies, Inc.

QMTest is an open-source, general-purpose, cross-platform software testing tool written entirely in Python [Python]. The problem of testing can be divided into a domain-dependent problem and a domain-independent problem. QMTest is intended to solve the domain-independent part of the problem and to offer a convenient, powerful, and flexible interface for solving the domain-dependent problem. The choice of Python as an implementation language has improved the user experience and simplified the development of QMTest; the use of Python is fundamental to the success of QMTest.

1. Introduction

QMTest is an open-source, general-purpose, cross-platform software testing tool written entirely in Python. QMTest can be used to test compilers, databases, graphical user interfaces, or embedded systems. QMTest provides a convenient graphical user interface for creating, managing, and executing tests, provides support for parallel test execution, and can be extended in a variety of ways.

The central principle underlying the design of QMTest is that the problem of testing can be divided into a domain-dependent problem and a domain-independent problem. The domain-dependent problem is deciding what to test and how to test it. For example, should a database be tested by performing unit tests on the C code that makes up the database, or by performing integration tests using SQL queries? How should the output of a query asking for a set of records be compared to expected output? Does the order in which the records presented in matter? These are questions that only someone who understands the application domain can answer.

The domain-independent part of the problem is managing the creation of tests, executing the tests, and displaying the results for users. For example, how does a user create a new test? How are tests stored? Should failing tests be reported to the user, even if the failure was expected? These questions are independent of the application domain; they are just as relevant for compiler tests as they are for database tests.

QMTest is intended to solve the domain-independent part of the problem and to offer a convenient, powerful, and flexible interface for solving the domain-dependent problem. QMTest is both a complete application, in that it can be used "out of the box" to handle many testing domains, and infrastructure, in that it can be extended to handle other domains.

Users of QMTest can use either a command-line interface, or a web-based graphical interface. Using either interface, users can, with a single command, run some or all of the tests in a test database, and obtain a summary report indicating which tests passed and which tests failed. Optionally, users can obtain more detailed reports containing domain-dependent data about the test results. When using the graphical interface, users can also create new tests by filling in a form.

The development of QMTest is part of the second phase of the Software Carpentry [Wilson99] project, sponsored by the Los Alamos National Laboratory (LANL). That project originally set out to design and implement Python-based tools for testing, issue tracking, build management, configuration. The design phase was accomplished via a contest that accepted submissions and awarded prizes for winning submissions.

The first section of this paper discusses the design of QMTest, with particular focus on the extension facilities that it provides, and the implementation of those facilities in Python. The second section discusses practical experience with QMTest to date. The third section discusses some of the ways in which the choice of Python as an implementation language has improved the user experience and simplified the development of QMTest. The fourth and final section discusses some possible future directions for QMTest.