Notez-Bien: This is NOT the full text of the paper.

Note to Reviwers: This is the "final" proposal based on the feedback I've gotten from the reviewers thus far (thanks for the helpful feedback!). I've attempted to expand the Role of Python section to address the comments of 2005-01-09. In the end this is basically a Python success story as the reviewer states. On the other hand, the fact that we chose to deploy this system-level code in Python without a lot of soul-searching basically says that Python is as standard as C, C++, or Java in the modern developer's arsenal and that one does not really need an excuse to deploy using Python. I hope that you find this proposal to be an interesting non-traditional use of Python. I am admittedly as excited about the CFS design concepts and accomplishments as I am about its deployment in Python - and I hope that PyCon participants are as interested in "how did you do that" talks as much as they are about the language in which "that" is done.
Thanks for your consideration,
--CCG 2005-01-13

One Last Note: There is a lot of material in this presentation, and I could see it going beyond 30 minutes. If the proposal is accepted should the content be trimmed down to place more focus on Python, less on details, and fit it into 30 minutes, or should it be expanded to the longer format? I am happy to do whatever conference organizers feel is best.

Abstract

The Compete File System (CFS) is a file management and file system virtualization software layer and Unix command set. CFS provides a framework for distributing files across multiple file systems while allowing the end-user or software developer the convenience of viewing the multiple systems as though they were a single large file system. Internally, CFS uses load balancing and server scheduling algorithms to provide maximum file system space availability while maintaining file server performance. CFS provides this functionality while still remaining in user application space - modifications to the operating system kernel are not necessary, and CFS never needs to operate in kernel or protected mode. In fact, to the extent that the underlying physical file systems can map to the local host's file system, CFS is largely system and platform independent.

CFS is written entirely in Python. The ability to develop high quality code extremely rapidly using Python combined with the depth and sophistication of the Python runtime system allowed a very small team to create this powerful system-level software package very quickly.

Terms/Keywords: virtualization, file systems, load-balancing, resource allocation, scheduling

Rough Outline for Proposed Paper & Presentation

• Problem Statement
• Company Overview - Technical Perspective
• Large data processing needs
• Tera-scale storage requirements
• Parallel processing environment
• Architecture prior to CFS
• Need for a low/no cost solution deployed quickly
• Capability to manage lots of files on many devices
• Tiny development staff
• Small budget
• CFS Architectural Overview
• End-user perspective: CFS command set & API
• Using CFS at the command level
• Coding with CFS
• Scheduling and Load Balancing
• Server allocation strategies
• Device allocation strategies
• CFS Version 1.0 Allocator & Scheduler
• Current Work-in-Progress around Allocation & Scheduling
• CFS Internal State Model & Failure Prevention
• Caching CFS Information
• Implementing Redundant Stores
• Recovery from Failure
• Role of Python in Implementing CFS

We chose to develop CFS exclusively in Python for a variety of reasons. We needed to build this software fast. The written code needed to be concise, clear, and easy for less-experienced developers to understand, maintain, and extend. We needed a system that was extremely flexible so that we could try new approaches and new algorithms without extensive refactoring. Python gave us all those things.

• The Usual Explanations
• Our success in building and deploying something like this is good evidence that Python is not just a "scripting language"
• Consideration given to other languages for deployment - C, C++ (Java considered evil by mgmt), but rejected for the "usual" reasons
• Robust class libraries and modules made it easy to think about algorithms, functionality, without worrying about whether a particular bit of functionlaity was available or supported
• Object-oriented nature of Python made it easy for developers to work on code without stepping on each other
• Whining and Ranting
• Python VM footprint and speed are good - but a compiled language would be better
• When this project was started there were no compilers at all
• System-level work in Python really requires stand-alone executables with no VM interaction - unclear if this is really possible
• A better interface to foreign code would have allowed us to do more in more efficient ways
• What is up with I/O in Python?
• Not possible to write code to deal with hundreds of gigabytes of data
• Even SWIGging in high performance file manipulation code runs orders of magnitude slower
• Bottom line is that most CFS usage is from the command set and drives C/C++ code
• So then is Python Bad?
• Emphatically no - it does surprisingly well in this very "not a script" application
• Ability to develop massive functionality quickly, and to debug and test easily allows a very small organization to do big work fast
• We don't think in terms of language-based religious wars: this stuff just works well for us so we use it
• Bottom Line: I have budget, time, and headcount to implement CFS however I want, and we are still using Python
• Results of Building and Deploying CFS
• Multiple terabytes of data under coherent structure
• Hundreds of thousands of files across multiple devices
• Effective storage resource management
• Zero worries
• CFS Future Direction
• Improved Security Model
• User Quotas
• File Versioning

About the Author: Christopher Gillett is the Chief Software Architect for Compete, Inc. Over the past 18 years he has developed or contributed to a variety of software systems, ranging from compilers and runtime systems for languages from FORTRAN to Java, to behavioral simulation systems used in microprocessor design, to complex application development. Most recently he has been designing and developing software solutions to facilitate storage, management, querying, and mining of massive amounts of data.

About Compete, Inc.: As a predictive analytics company, Compete is pioneering the analysis of consumer behavior online to predict consumer purchasing — both offline and online. Compete's unique vantage point on the consumer consideration process gives clients the edge needed to create significant value for their organizations.