The Compete File System: File System Virtualization using Python

Notez-Bien: This is NOT the full text of the paper.

Note to Reviwers: This is the "final" proposal based on the feedback I've gotten from the reviewers thus far (thanks for the helpful feedback!). I've attempted to expand the Role of Python section to address the comments of 2005-01-09. In the end this is basically a Python success story as the reviewer states. On the other hand, the fact that we chose to deploy this system-level code in Python without a lot of soul-searching basically says that Python is as standard as C, C++, or Java in the modern developer's arsenal and that one does not really need an excuse to deploy using Python. I hope that you find this proposal to be an interesting non-traditional use of Python. I am admittedly as excited about the CFS design concepts and accomplishments as I am about its deployment in Python - and I hope that PyCon participants are as interested in "how did you do that" talks as much as they are about the language in which "that" is done.
Thanks for your consideration,
--CCG 2005-01-13

One Last Note: There is a lot of material in this presentation, and I could see it going beyond 30 minutes. If the proposal is accepted should the content be trimmed down to place more focus on Python, less on details, and fit it into 30 minutes, or should it be expanded to the longer format? I am happy to do whatever conference organizers feel is best.

Abstract

The Compete File System (CFS) is a file management and file system virtualization software layer and Unix command set. CFS provides a framework for distributing files across multiple file systems while allowing the end-user or software developer the convenience of viewing the multiple systems as though they were a single large file system. Internally, CFS uses load balancing and server scheduling algorithms to provide maximum file system space availability while maintaining file server performance. CFS provides this functionality while still remaining in user application space - modifications to the operating system kernel are not necessary, and CFS never needs to operate in kernel or protected mode. In fact, to the extent that the underlying physical file systems can map to the local host's file system, CFS is largely system and platform independent.

CFS is written entirely in Python. The ability to develop high quality code extremely rapidly using Python combined with the depth and sophistication of the Python runtime system allowed a very small team to create this powerful system-level software package very quickly.

Terms/Keywords: virtualization, file systems, load-balancing, resource allocation, scheduling

Rough Outline for Proposed Paper & Presentation

About the Author: Christopher Gillett is the Chief Software Architect for Compete, Inc. Over the past 18 years he has developed or contributed to a variety of software systems, ranging from compilers and runtime systems for languages from FORTRAN to Java, to behavioral simulation systems used in microprocessor design, to complex application development. Most recently he has been designing and developing software solutions to facilitate storage, management, querying, and mining of massive amounts of data.

About Compete, Inc.: As a predictive analytics company, Compete is pioneering the analysis of consumer behavior online to predict consumer purchasing — both offline and online. Compete's unique vantage point on the consumer consideration process gives clients the edge needed to create significant value for their organizations.