Using Python in Commercial Projects

[Editor's note: The growth of the Open Software movement is both a product of and a force in free software's increasing commercial acceptance. See www.opensource.org for background on, examination of, and advocacy for the phenomenon.]

1) Developing a computer system for a multi-million dollar factory using 'shareware' is just too risky. What happens if there's a bug. Who do we call and how quickly will it be fixed?

We have only discovered one bug so far, and it was fixed in a couple days. Internal bugs in the Python interpreter are even more rare. Currently if you have a problem you think is internal to Python (rather than with code you wrote in Python) you can email python-help@python.org. This is read by Guido himself as well as other knowledgeable people. Alternatively, there is a very active USENET group comp.lang.python which can be a fountain of ideas and solutions. In the future there may be a company offering support contracts for Python.

The likelihood of you needing this is, as I said, is quite rare. Python is a language. And like any language you must provide correct instructions. If what you are worried about is faulty calculations, then the fact that thousands of programs around the world calculate correctly should allay that fear. And as with all software you write, it should be tested thoroughly.

The quality of software is proportional to the number of people using it and examining its source. Python, being freeware, (not shareware by the way), is used and scrutinized in its source code form by many people worldwide. Bugs are flushed out early on and there are many people with understanding of the internals who can and do help if problems arise. Python has been out in the world for about five years now and has been 'Purified'. In contrast, commercial software is never shipped with source. Very few people are available to address the quality of the code. You rely on the company's limited resources which are usually taxed by tight schedules. (I've reported bugs to Sun years ago which still haven't been fixed.) You are taking the quality of the code on pure faith when you buy a commercial package. You may be familiar with the SEI software quality model. No commercial software company has reached higher than level 1 on their scale. Software you develop internally is usually of higher quality if you have a decent staff and don't rush things too much. I'm sure your software can be of higher quality than software you purchase. I don't see any additional risk introduced by writing code in a higher level language like Python.

<soapbox> Ah, risk; an often misunderstood four letter word. Risk is a relative concept. When one says something is too risky they are making a relative judgment against some alternative they feel is less so. Often however that less risky alternative is not in reality any less prone to adverse events. What they really are reacting to is newness. It is natural to conclude that anything that is less known is riskier. This misses the point though and can do a disservice to the new approach/tool. It's important to be realistic about the inherent risks involved in doing things the old way. What's the MTBF and MRTR figures for software you have written in lower level languages? </soapbox>

In our experience with mission critical software systems, the likelihood of adverse events is more a function of complexity of implementation than anything else. For example, the likelihood of a bug existing in 1000 lines of newly written C code is far greater than the equivalent functionality written in Python. As software systems written in lower level languages grow, this hidden error rate is such that large investments in testing energy must be expended to give even a mild assurance that nothing major will malfunction during operations. The cost of this development / testing / bugfix / retesting / certification cycle is the major problem we see in software systems today. Very few organizations have the resources and discipline to make sure things are bulletproof. What happens all too often is corners are cut to meet schedule and subtle defects are left for future engineers (and users) to deal with.

An effective way of dealing with this problem is to use a higher level of programming abstraction to isolate the details in smaller packages. Some call this middleware. Python can be thought of as a middleware product when it is used as a binding language providing directive logic to objects which may have very specific operations. Overall control logic can be easily altered at this higher level without breaking the working components. It's best to isolate the complexities of device operations in specific control objects rather than architecting a large complicated system where everything knows, (or is assumed to know), about everything else. This also allows better unit testing. You can implement the detailed logic in whatever language is appropriate and provide a clean type interface to expose up through Python. The coordination logic is a much different animal and can be nicely expressed directly in Python. What's important then, is to have a powerful extension mechanism which allows easy integration of the two levels. Fortunately Python has one. Python has been designed from the beginning to support extended data types with a C API. This allows control of complicated C systems/libraries from the Python layer and promotes reuse while preserving flexibility. An often overlooked bonus of using Python to control applications is the added ease of testing. Using Python gives an excellent scripting language to write your test drivers. Writing C test cases for C code can be very time consuming as you only have low level data types to work with.

We took all this into consideration when NASA decided to standardize on Python as its scripting language of choice for the Integrated Planning System last year. I lobbied hard to get NASA to realize how much money was being wasted with all these engineers going off and programming in whatever language they knew. This had created a hodge-podge of C shell, Bourne and Korn shell, PERL, awk, sed scripts as well as Fortran and C programs, used in inappropriate ways. Code was understood only by its author and reuse was made impossible. With Python we are able to build libraries of reusable modules and use the power of object oriented programming to rapidly write applications. In the case where heavy number crunching is required we rely on Python's excellent C/C++ extension facility to call compiled code. We are also placing Python in the new Mission Control Center here in Houston where it will be used to augment the analytical GUI's we've written. Obviously with manned space flight, people are very sensitive to code reliability. Python has demonstrated good reliability as full exception handling is build into the language (and we use it). Other people have requested PERL, tcl, even Java to be made available for MCC applications, but only Python has been approved.

2) What about training? How do I get dozens of people in various countries in the Americas, Asia and Europe up to speed with Python?

One of the things which first impressed me about Python was the simple straightforward syntax. No convoluted rules and specials cases like PERL. I think you'll find people can pick up Python quite quickly. In the courses I've taught so far the student's level of confusion when learning Python is more a function of pre-existing biases. Fortran people will have a little more trouble than C people and so on.

Python is more wide than deep. By that I mean, the basic concepts of Python can be understood without delving deep into books for months. The challenge is getting familiar with all the wide variety of Python add on modules which make your tasks quicker to finish. The standard docs are good at documenting this. They aren't as good for initial training material. For that I would recommend purchase of the Python book written by Mark Lutz to be published by O'Reilly and Associates next month. I would wager that a typical programmer can be highly proficient in Python in a couple weeks. Non programmer types like engineers may take a little longer but the point is that the learning curve is not steep.

3) How do I know Python will still be around in 5 years? What structures are in place to insure the language is maintained and expanded. Isn't a commitment by a major player like Microsoft, IBM, HP or SUN required?

The Python community has banded together to form the Python Software Activity under the auspices of the Center for National Research Initiatives (CNRI). Its expressed purpose is the promotion of Python. www.python.org provides a central focal point for information, software, and coordination of joint efforts in Python. User base is expanding greatly and should accelerate once the book(s) hit the shelves. During the time I've been involved with Python I have heard of few defectors while many newcomers to the language express joy in their newfound tool. I see no indication that people will loose interest in Python (either in its use or enhancement).

Languages do just fine without commercial marketing. PERL and LISP come to mind. When a language is very new it needs constant attention from the author as new ideas are contributed, but once the language stabilizes around a feature set the job turns to providing libraries of modules. This is the phase Python is in. Once the syntax and feature set is largely fixed, people spend most of there time using it to get the job done. These applications and supporting modules can be provided by commercial ventures as well as freeware. There are no restrictions. Judging by the amount of software being developed in Python I see no indication that it won't be much more popular 5 years from now than it is today. We asked the same questions in our technology insertion process and came to this conclusion. Having a big brother commercial concern controlling a language is not always a good thing. They may very well react to changing business needs by changing the language or APIs forcing you to rework code to stay current. How many times has Microsoft done that? Plenty. Python is driven purely by the needs of its user base. There are no hidden agendas to worry about.

As I don't know all that much about the project you are embarking on, I can't say for sure that Python is the right choice. But I can say for sure that the issues raised in your email are not reasons to reject Python. I hope this has been helpful.

-Robin Friedrich
Houston, Texas
Created: August 6, 1996
Last update: August 13, 1996 (Editor's note, 08/06/1998)