Abstract: This paper describes an empirical study of the relative usability of Perl, Python, and Tcl for object-oriented (OO) development. The paper's first author specified and designed two representative programs: a paint program and a chat server. He then implemented these programs in Perl, Python, and Tcl, using PSP to collect data on the work of program development. This data, when analyzed, showed that Python was the easiest of the three languages to use, with Perl the most difficult, and Tcl somewhere in between. In particular,
Areas where Tcl and Perl proved superior to Python included Tcl's built-in support for an event loop mechanism, and the Perl community's support for the CPAN database. Recommendations for improvements that emerge from this study include revising the Tcl megawidget library to support OO interfaces for raw Tk widgets; adding event loop mechanisms to Tcl and Perl; implementing CPAN-like support for Tcl and Perl; and creating new dialects of Perl and Tcl that support OO programming as an integral part of these languages.
Keywords: Perl; Python; Tcl/Tk; Tk widgets; TCL megawidgets; Personal Software Process (PSP); object-oriented software development; sockets.
This paper describes an empirical study of the relative usability of Perl, Python, and Tcl for object-oriented (OO) program development. Perl, Python, and Tcl were originally developed as scripting languages: languages that support rapid application development through the "gluing" of smaller applications [Oust98]. Perl, Python, and Tcl have since evolved beyond mere glue languages, adding extensive runtime libraries and—for Perl and Tcl—constructs for OO programming.
The key question addressed by this study is the extent to which the choice of Perl, Python, or Tcl could influence programmer productivity. Our interest in this question stemmed from the following five considerations. One was the current emphasis on OO strategies for software development, particularly with regard to large applications. A second was the claim that large increases in programmer productivity can be obtained from using scripting languages in place of 3GLs (e.g., C, C++) for software development [Oust98]. A third was the current popularity of Perl, Python, and Tcl. A fourth was the presence of mechanisms in Perl, Python, and Tcl for supporting OO development. A final consideration was a perception that inherent differences in language design could influence how effectively these languages were used. Specifically:
We chose, in this investigation, to equate a language's support for programmer productivity with that language's usability. The International Standards Organization defines usability as “the effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments” [DFAB98]. Here, usability was assessed by determining how well Perl, Python, and Tcl supported the implementation of a paint program and a chat program. These applications were deemed complex enough to test the languages' overall usability; general enough to avoid favoring any one language; comprehensive enough to test the usability of the languages as well as their run-time libraries; and distinct enough to reduce biases that might result from implementing one application alone.
This study was conducted as follows. The first author developed specifications and designs for the paint and chat programs, following an OO approach to program design (cf. §2.1). He implemented these programs in the current version of each platform (cf. §2.2), using Watts Humphrey's Personal Software Process (PSP) to track development time and defect data (cf. §2.4). Finally, the data gathered from PSP (cf. §3) were analyzed, relative to that author's prior programming experience (cf. §2.3), to assess the relative usability of Perl, Python, and Tcl (cf. §4).
The first of the two sample applications that formed the basis of our study—the paint program—was an interactive graphical editor similar to the Windows Paint utility. This paint program can be used to create drawings composed of lines, rectangles, circles, and ovals. Users may select a color for each drawing object, and an interior (fill) color for rectangles, circles, and ovals with colors. Lines and shapes are created by pressing the left mouse button, which defines a start point in the drawing box; dragging the mouse cursor, which displays a tentative shape for the object being drawn; and releasing the button, which completes the draw operation. Users can also select objects in the drawing edit box and move and delete them.
Internally, the paint program is organized as three packages: a GUI library package, which includes all classes contained in the implementation language's native GUI library (e.g., menu, frame, canvas); a drawing functionality package, which includes all classes that implement the program's drawing functions; and a program interface package, which includes all classes built specifically to manage the program's interface. The drawing functionality package uses the GUI library package to implement the draw commands. The program interface package obtains user inputs from the drawing functionality package (e.g., drawing command and selected color).
The paint program's class diagram is shown below. The design assumes that the implementation language provides a well-defined GUI library.
The
second sample application, a chat application, consisted of a
server and a client program. The server program supports concurrent
communications among multiple connected clients. It implements six client commands: help; show current users;
shutdown this connection, shutdown the
server; broadcast message to all users, and send
message to one client. The chat client transmits requests to the
chat server, and receives and
displays responses and other server messages. The chat client accepts user inputs at any time,
independently of the receiving and displaying of server messages.
Internally, client and server are organized as three packages: a socket library package, consisting of classes from the implementation language's native socket library; an I/O Package, which includes classes from the language's native I/O library, and a main package, which implements the chat application's basic functions. The main package accesses classes in the socket and I/O packages.
The chat program's class diagram is shown below. The design assumes that the implementation language provides well-defined I/O and socket libraries.
The paint and chat applications were implemented using those versions of Perl, Python, and Tcl/Tk that were current and stable in December 2000: ActivePerl 5.6, ActivePython 2.0, and Tcl/Tk 8.2 (iTcl/iTk 3.1). All programs were implemented under Microsoft Windows 98. Data on the work of program implementation were gathered in accordance with Watts Humphrey's' Personal Software Process (PSP). PSP is a measurement-driven strategy for self-assessment that uses data on coding times, defect types, and defect removal times to identify areas for programmer self-improvement. PSP data is typically collected using tools for tracking program development time by task, and for logging defects by type. The first of the two PSP-related tools used in this study, PPLog Control, is a GUI application for recording time and defect data. The second, evalpsp, is a Perl script for summarizing PPLog-Control-generated log files. Both were downloaded from the University of Karlsruhe (cf. http://wwwipd.ira.uka.de/PSP/).
This study's conclusions are based on a sample population of one. At the start of this study, our subject, the first author, was a second-year graduate student in computer science with an undergraduate degree in computer science. His prior programming experience included the development of GUI and socket applications; procedural and OO program design and implementation; and limited professional experience with C and C++. His prior knowledge of scripting languages was limited to bash, csh, and a classroom exercise involving a short Perl script.
The following is a detailed synopsis of how the study was conducted:
Standard
PSP defect types, as identified by Chillarge [4], were
not used to classify defects. Some PSP defect
types, such as syntax and interface defects, were too general to identify
specific problem areas in programming languages.
Our strategy for defect classification used three distinct categories of defects: defects arising from misuse of the language’s basic, non-OO constructs (LBC); misuse of the language’s OO constructs (OO); and misuse of the language’s libraries (LIB). Defects arising from OO constructs were given their own category because of the distinctiveness of OO syntax, and the belated incorporation of OO into Perl and Tcl. Library-related defects were treated separately because libraries are typically used as sources of finished components, rather than as codes whose internals must be mastered and revised.
Within these three categories of defects, fifteen specific types of defects were tracked:
The OO defect types used here are based on work by Chidamber and Kemerer [4]. The UAC type shows the difficulties created by coupling between classes. The INH type shows the difficulties of inheritance between classes. The CAC type shows the difficulties created by using a language's OO syntax to build classes, objects, methods and data members.
The three categories of defects are largely independent. Defects in one pair of overlapping categories, those arising from the misuse of OO libraries, were counted as library-related (LIB) defects. The possibility that LIB defects may be caused by an imperfectly designed OO library or an inconsistent interface between the library and the core language is addressed in the recommendations.
Requirement defects and design defects were not tracked. Requirement and design defects were usually common to all three implementations, and the first implementation usually exposed these defects. Discounting these defects also reduced order-related bias.
This section presents implementation times, defect counts, and fix times for the paint and chat programs. These data are analyzed relative to the effort needed to implement each application; the change in implementation effort over the course of the study; and the types and relative severity of the problems encountered. All times below are given in minutes. All implementation and fix times exclude time spent on the largely language-independent work of specification and design.
The balance of this section is divided into four subsections.
Section 3.1 summarizes the time-to-implement data from the paint and chat programs. Section 3.1's key point—i.e., that fix times for language-related defects were largely responsible for the discrepancies in effort among the three implementations—suggests that ease of use was strongly correlated with ease of defect repair. This observation, in turn, motivates the focus on defect data in Sections 3.2, 3.3, and 3.4.
Section 3.2 presents defect data and conclusions drawn from the paint program part of the study.
Section 3.3 presents defect data and conclusions drawn from the chat program part of the study.
Section 3.4 generalizes the findings in Sections 3.2 and 3.3 by analyzing the combined defect data from the paint and chat program studies.
The
total development time for the Perl and Tcl paint
implementations were close. Both were far more than Python’s.
The total development time of Python’s and Tcl’s chat implementations were close. Both were far less than Perl’s. The Perl, Python, and Tcl versions of the chat programs required about 850, 800, and 850 minutes to develop, respectively, exclusive of fix times (cf. §3.3).
This
section describes data obtained from implementing the paint program.
Fix-times and number of defects are given thrice: first by implementation, then
by defect category, and finally by defect type within category.
The Python paint program took a little less than half the time to repair as the Perl program and about 2/3 the time of the Tcl program.
The number of defects in each program was roughly proportional to time to fix each program. Tcl generated more defects than Perl and Python. Subsequent figures show that Tcl’s Tk libraries were responsible for the increased number of defects.
Python defects were the easiest (i.e., fastest) to fix. Perl had the highest fix times for basic language and OO-related defects. Tcl had the highest fix times for Tk library defects. This last result is surprising because the Tk library is native to Tcl.
Perl’s basic syntax and operators accounted for most of the disparity between Perl and Python defect counts. Perl yielded more than three times as many basic defects as Python. Tcl yielded more OO-related defects than Perl, even though Tcl has the most complete support for OO (in [incr Tcl]).
Because Perl lacks a megawidgets library, the Perl Paint program was built from the PerlTk library, which provides Perl Tk basic widgets. Therefore, the total fix time in the LIB_TK category was counted only on the TK Basic Widgets category.
Tcl’s Tk, which provides the basic Widgets support for Tcl, yielded the fewest defects of the graphical support libraries. However, defects involving Tcl's [incr Widgets] library, Tcl's library for supporting megawidgets, took far more time to repair than defects involving Python’s megawidgets library, Pmw. Problems with Tcl’s [incr Widgets] included the following:
Reasons that Perl's fix times were longer than Python’s were as follows:
Reasons for Python’s good fix times are as follows:
Total defects are roughly proportional to the time consumed by defect fixes. Again, Tcl megawidgets were the major source of problems. Note that megawidget defects cost more time to fix than basic widget defects.
Defects related to class construction were the most expensive of all OO defects. Class construction and inheritance defects were much more expensive to fix in Perl than Python and Tcl. One reason for Perl's poorer usability is Perl’s limited support for OO. Perl programmers must build most basic OO constructs from scratch. A second is that programmers have to handle object references carefully to build class methods.
Reasons that [incr Tcl] performs worse than Python are as follows:
Python’s good performance is attributable to the language's emphasis on OO development. Python’s OO support is internal to the interpreter. The basic language constructs, OO constructs and libraries exhibit a consistency that helped to prevent defects.
Although the number of defects was approximately the same across all three programs, Perl OO-related defects typically required more time to fix. Perl yielded more OO inheritance-related defects due to Perl’s weak support for inheritance: e.g., Perl's failure to support attribute inheritance, and its requirement that explicit references to parent classes be used when accessing inherited methods.
Of the 10 subcategories of defects in the BASICS category, five—strings (STR), basic data structures (BDS), expressions (EXP), code block delimiters (CBD), and variables (VAR)—accounted for 4/5 of the basic defect repair time for each language.
Perl’s basic data structures were its major source of trouble. Arrays of references and hashed references created significant problems. The many ways of constructing a Perl array or a hash were often confusing. Passing array and hash arguments to classes, methods, or functions yielded multiple defects.
Perl and Tcl identifier syntax caused more problems than Python’s. Perl, the worst, was complicated by the excess of symbols like @, $, / and %. Perl's use of init symbols to define and reference data types was a second source of trouble. Tcl's use of $ to reference, but not to define, variables was also confusing.
Expression manipulation was a problem in all three languages.
Block delimiters were less of a problem in Perl, due to their similarity to delimiters in languages that the author already knew. Python's use of tabs—and non-use of blanks—as block delimiters leads to codes that appear OK, but may actually be erroneous. Tcl’s block delimiters were the worst: the need to use ‘\’ as line connectors, semicolons as line terminators, and # as comments accounted for most of these Tcl defects.
Tcl yielded far more EXP and CBD defects than Perl and Python. Perl yielded far more VAR defects than Tcl and Python. These defects, however, were easy to detect and fix.
This section describes data obtained from implementing the chat program. Fix-times and number of defects are given thrice: first by implementation, then by defect category, and finally by defect type within category.
The Perl and Tcl chat projects required about 1 1/2 times and 2 1/2 times as much time to repair as the Python project, respectively.
The differences among the numbers of defects in these three languages
were not
as great, implying that an average Perl defect was
harder (i.e., slower) to fix than an average Python or Tcl defect.
Perl’s I/O library- and Socket library-related defects accounted for most
of the time on defect repair. Each program incurred a small amount of
repair time on OO-related defects, although the chat program was fully OO. Tcl’s OO-related defects took more time than
Perl’s and Python’s. Again, Perl’s basic language defects took far more
time to repair than Python’s and Tcl’s.
For all three languages, I/O-related and
Socket-related defects took far more time to fix than defects in other two
categories: OO and BASICS. Tcl yielded almost twice as many OO-related defects as
Python and Perl.
Perl’s I/O- and socket-related defects took approximately four times as long to fix as Python’s and three times as Tcl’s.
The two factors that made Perl's I/O library difficult to use were its lack of support for non-blocking I/O, and its weak support for managing line delimiters. The ReadKey module, which provides non-blocking IO support for Perl, is not part of the Perl standard library: programmers must download and install ReadKey for themselves. ReadKey was also not well maintained: some functions were not working on Windows platforms at the time when this study was conducted.
Perl’s I/O functions fail to manipulate line delimiters (EOLs) on a program's behalf. Perl programmers must explicitly use chomp() or chop() to delete EOLs and resolve cross-platform differences in line delimiters. Perl’s weak support for line delimiter manipulation yields a significant number of defects, and these defects are usually hard to detect. Both Python and Tcl automatically delete EOLs in their implementations of their I/O functions.
Perl’s socket library, like Perl’s I/O library, lacks built-in mechanisms for managing EOLs. The mixed use of non-OO socket methods and OO socket objects in Perl’s IO::Socket module was a second source of trouble.
The previous figure groups Tcl’s IO- and Socket-related defects by underlying software component. Tcl uses a single, built-in event loop mechanism to handle all I/O-related tasks, including socket I/O. The event loop component yielded multiple defects that took some time to fix. However, Tcl’s event loop mechanism reduced I/O-related and Socket-related defects significantly by providing an easy and consistent event-driven model for non-blocking IO and Sockets.
The number of I/O-related and Socket-related defects was almost the same among these three languages, which means that Perl’s defects took far more time to repair than Python’s and Tcl’s.
Tcl yielded more OO-related defects, which took more time to repair than Python’s and Perl’s. These defects were due to Tcl’s idioms for supporting calls to base-class constructors and destructors, which were challenging to master. The passing of arguments from child class constructor to parent class constructor yielded an especially high number of hard-to-fix defects.
Tcl’s string manipulation routines took the least time to fix. Perl’s basic data structure defects needed far more fix time than Python’s and Tcl’s. References to hashes and lists, though flexible, accounted for many of Perl's defects. Objects containing hash data structures required the manipulation of hashes of hashes, which was still more difficult and yielded more defects. Finally, constructing a table of sockets proved difficult in Perl. Perl uses a special data structure to represent sockets and file handles. This data structure cannot act as a key for Perl’s hash data. Accordingly, a more complex structure with related functions had to be implemented, which yielded many defects.
For both programs, Perl yielded more defects than Python. Perl defects took 2 to 3 times as long to fix as Python’s defects. Tcl was somewhere in-between.
For all three languages, the number and fix times of OO-related defects decreased significantly between the first and second programs. This result suggests that programmers can adjust to each language's style of OO development. Still, stylistic inconsistencies between a language's libraries and its idioms for supporting OO may yield defects with each new attempt to use an unfamiliar library. Such problems can be viewed as hidden OO defects that may be attributed to the libraries rather than the language's internal inconsistencies. Examples of problem libraries include Tcl’s [incr Tcl] library and Perl’s PerlTk library. Python’s consistent and thorough approach to supporting OO significantly reduces these defects.
The data suggest the following conclusions about defects resulting from basic syntax:
Perl. Perl is poorly suited for OO programming, especially for new users. Perl's support for OO suffers from four basic deficiencies. Perl lacks a class data type: Perl objects must be constructed explicitly by using hashes. Perl lacks support for inherited attributes. Perl lacks support for object address management: users must specify object addresses in the course of creating class methods. Finally, the internals of at least some of Perl's object-oriented libraries—here, IO::Socket and Perl/Tk—must be mastered as a precondition for their effective use.
Perl's run-time libraries suffer from various problems. Perl/Tk lacks clear documentation; fails to support megawidgets like statusbar and menubar; and lacks support for other useful graphic objects, like balloons. Perl's I/O library lacks support for non-blocking I/O. Perl's I/O and socket libraries fail to support the use of file handles in hashes, and provide weak support for managing line delimiters.
Perl's usability is further compromised by its use of special symbols to represent different data types. Perl's references are flexible and convenient, but difficult for novices to use in conjunction with hashes and lists.
Positive aspects of the Perl environment included PPM and CPAN. PPM is Active Perl's Perl Package Manager, a command-line tool that downloads and installs Perl modules from the Internet. CPAN, the Comprehensive Perl Archive Network, provides users with many well-organized Perl modules.
Tcl. [incr Tcl] offers the most complete support for OO, including the use of Private, Protected, and Public attributes for data members, and the automatic, hierarchically based invocation of constructors and destructors. Other positive aspects of Tcl include Tcl's minimal syntax, and Tcl’s event loop mechanism, which provides an easy and consistent event-driven model for non-blocking IO and Sockets.
[incr Tcl] and [incr Widgets], however, suffer from deficiencies. [incr Tcl] uses non-standard strategies for supporting OO, including unusual idioms for supporting calls to base-class constructors and destructors. Tcl's Array (Hash) construct is awkward to use from within [incr Tcl] classes. [incr Tcl]'s approach to memory management, which requires the explicit allocation and release of memory, differs from the standard Tcl approach to memory management. [incr Widgets] is missing one important megawidget: balloon. And coding is complicated by stylistic inconsistencies between raw Tk widgets and [incr Widgets] objects.
Tcl’s standard library does not support the UDP protocol—a final area of weakness.
Python. Python performed best in this study, in terms of the number of defects incurred during program development and the time needed to fix these defects. Python's strengths include its support for standard OO constructs and its extensive collection of OO supporting libraries. Problems with Python included the language's failure to treat tabs and blanks in a uniform way, and failure to support a platform-independent module for non-blocking I/O.
Perl. Perl should provide better support for OO programming, including a Class data type and constructs for hiding object implementation and inheritance relationships. This improved support could be obtained by splitting the Perl standard: i.e., by defining a pure OO version of Perl for large software development, and a totally non-OO Perl for classic hacking.
Perl/Tk's use as a de facto standard for creating GUIs implies a need to include Perl/Tk in the standard Perl distribution; to provide complete support for Tk functionality; and to improve PerlTk's documentation.
Perl should provide a mechanism to handle line delimiters automatically for I/O or Socket programming, or at least provide programmers with an option to do so.
Perl should provide a mechanism like Tcl’s event loop to simplify event-driven programming.
Tcl. Tcl, like Perl, was retrofitted with OO extensions. OO development in Tcl, like OO development in Perl, would benefit from a split standard that produced a second, pure OO version of Tcl. Tk basic widgets should be reimplemented in [Incr Widgets], using [Incr Tk] as a starting point for this implementation. Finally, the Tcl user community should study the CPAN model of module distribution as a basis for software development.
Python. Python should provide a mechanism like Tcl’s event loop to simplify event-driven programming. The Python user community should study the CPAN model of module distribution as a basis for software development.
This study is apparently one of a very few systematic attempts to assess the usability of Perl, Python, and Tcl. Other comparisons of scripting languages that we have seen are much less formal. For example, Udell, writing for byte.com, describes some of the concerns raised here: Perl's support for multiple ways of doing a common task, which Udall regards as a plus; Perl's unsuitability as a first language for new programmers; and the need for a CPAN-like model of software distribution for the Python language community [Udell00]. Udell, however, offers only anecdotal data to support his claims. Stajano, in a similar vein, cites reasons for preferring Python over Tcl, including Python's libraries and support for OO development [Staj97].
A partial analysis of the relative usability of scripting languages has been done in the context of function point measurement. A function point is an abstract metric that represents a level of observable program functionality [Jones97]. One approach to assessing a language's usability is to equate usability with expressiveness, and to measure that language's expressiveness by the average number of source lines of code (SLOC) required to implement a "typical" function point. In one SLOC-based study of expressiveness, Jones reported average SLOCs per function point of 21 and 64 for Perl and Tcl, respectively [Jones96]. By way of comparison, VB 5.0, Java, and macro-less assembly languages required an average of 39, 53, and 320 SLOCs per function point, respectively. Jones's study, however, provides no data on Python, and fails to address specific areas of language design—including the suitability of Perl, Python, and Tcl for OO development.
A second contribution of this work lies in is its approach to assessing usability. Traditionally, PSP has been used as a tool for improving programmer productivity through self-assessment. Here, PSP was used in an apparently novel and practical way: i.e., to assess the relative usability of programming languages.
The study presented here could be improved in at least three obvious ways. The study should be repeated with a larger population of programmers; expanded to include additional scripting languages, including Ruby (cf. http://www.ruby-lang.org) and VBscript; and enhanced with at least one sample program that highlights Tcl's support for the automatic, hierarchical invocation of constructors and destructors.
The authors gratefully acknowledge the help of John Chenoweth and Don Gotterbarn with this research.
[DFAB98] Alan Dix, Janet Finlay, Gregory Abowd, Russell Beale. Human Computer Interaction. Prentice Hall, 1998.
[GiLu98] David Gillibrand, Kecheng Liu. Quality Metrics for Object-Oriented Design. JOOP, January 1998.
[Hump01] Humphrey, W. S., The Personal Software Process: Overview, Practice, and Results. Carnegie Mellon Univ. Software Eng. Institute, Pittsburgh, PA.Last modified 21 March 2001. cf. http://www.sei.cmu.edu/pub/documents/articles/pdf/psp.over.prac.res.pdf [ref. 26 Sept. 2001]
[Hump96] Humphrey, W. S., Introduction to the Personal Software Process. Reading, MA. Addison-Wesley-Longman, 1996
[Jones96] Jones, C., Programming Languages Table, Release 8.2, March 1996. cf.http://www.spr.com/library/0langtbl.htm [ref. 24 Sept. 2001]
[Jones97] Jones, C., What are Function Points?, 1997. cf.http://www.spr.com/library/0funcmet.htm [ref. 24 Sept. 2001]
[Oust98] Ousterhout, J., Scripting: Higher Level Programming for the 21st Century. March 1998, IEEE Computer.
[Perl5] Perl5 Manuals. http://www.perl.com/pub/q/documentation [ref. 24 Sept. 2001]
[Py2.0] Python2.0 Manuals. http://www.python.org/doc/
[Scriptics] Tcl Manuals. http://www.scriptics.com/man/
[Staj97] Stajano, F., Implementing the SMS server, or why I switched from Tcl to Python, 7th International Python Conference, Nov. 10-13, 1998. cf. http://www.foretec.com/python/workshops/1998-11/proceedings/papers/stajano/stajano.html [ref. 26 Sept. 2001]
[Udell00] Udell, J, A Perl Hacker in the Land of Python, BYTE.com, 1 February 2000. cf. http://www.byte.com/documents/s=284/byt20000201s0001/index.htm [ref. 26 September 2001]
[Welch99] Brent Welch. Practical Programming in Tcl and Tk. Prentice Hall, 1999. 3rd Ed.