Creating Web-based Classroom Tools with Zope

Nathan R. Yergler, Vern Ceder

Canterbury School

Introduction

Student Teacher Online Applications (STOA) is a web-based K-12 environment built on Zope/CMF, Python and MySQL. In its second year of deployment, STOA is now used by over 800 students, faculty and administrators to manage student schedules and class rosters, to report attendance, to post assignments, notes and handouts, to turn in work, and to write and print student progress reports. This case study will discuss the process of developing and maintaining a Zope/CMF web application, along with the advantages and pitfalls of Zope/CMF we encountered in implementing STOA.

Background

STOA was developed at Canterbury, a K-12 independent school in Fort Wayne, IN with an enrollment of over 800 students. The decision to develop STOA in-house was not taken lightly. However, at the time it seemed to be the only way to satisfy the school's growing needs for web-based communication in a managable and affordable fashion.

In the 2000-2001 academic year Vern had used the (then) free version of ?BlackBoard in his C++ programming courses and had found it useful. Since ?BlackBoard's services were either slow or costly, Vern searched for open source alternatives. In the 2001-2002 school year, as the school was beginning a 3 year technology initiative, the Manhattan Project was deployed in the high school on what was intended to be an experimental basis. Given the increased focus on technology and Manhattan's simple user interface it was a runaway success, with over 50% of the high school courses using Manhattan by the end of the year.

In fact, the Manhattan experiment was almost too successful, and as the number of courses and users grew, we encountered several problems. First of all, Manhattan was designed so that each course had a separate user accounts and passwords. This meant that both teachers and students had to log out of one course and then log into another. Teachers with multiple sections of the same course found this particularly irritating.

In addition, the course-by-course organization of Manhattan made administration a chore, requiring repetitious manual intervention to create courses, maintain the list of entry point links, handle student section changes, handle user account problems, etc. Some of these chores were eased by various scripts and hacks, but the chore of managing 100 Manhattan courses was getting onerous.

Finally, as we considered the features we wanted to add, we found that Manhattan's structure as a collection of separate but tightly coupled small C programs was a barrier to easy modification. Manhattan's login code was hacked to check passwords off of a WinNT domain controller, but it was clear that more sweeping modifications would require a complete rewrite of Manhattan's modules.

Our experience with Manhattan combined with user comments and feature requests led us to decide upon the following features as goals for STOA:

Zope was selected as the basic platform since it seemed to offer excellent support for those goals. In addition, both developers were quite comfortable with Python, since Canterbury School had used Python in other problem areas, and had also integrated the language into our mandatory introductory curriculum. After some test coding it became apparent that CMF was also needed because of its implementation of user roles, workflows, and easier content management. Further exploration confirmed that CMF's use of skins would make customization easier as well.

At this point we considered adding Plone to the stack but decided not to. First of all, CMF seemed to supply the key features we needed without any addition, and we were unable to locate any documentation that articulated why Plone might be useful. Secondly, by this time we had less than two months before the start of the school year, and only one developer able to work actively on the project, so we didn't have the time to spend integrating another layer of software. Finally, the quality and availability of documentation discouraged us. We had discovered that Zope documentation available was contradictory and even misleading, CMF's documentation was extremely sparse, and Plone appeared to be even less documented.

Initial Development

Vern did most of the initial prototyping and development, since Nathan was not available for development until after the beginning of the school year. While both developers were familiar with Python, Zope had a steep learning curve. The concept of aquisition, the details of TALES and Python integration, and external procedures all took time to learn and understand. In particular, a very imperfect understanding of acquisition led us to hard code paths to objects when we shouldn't have. When the decision was made to begin developing Stoa for Zope, both Vern and Nathan began reading through the available print documentation (see list at end). While it may have been a case of poor timing, we found these to be of limited use, often out-dated or just plain wrong. This meant that many questions had to be answered by experimentation and/or browsing the source code. While The Zope Book's online instance did eventually prove useful as a reference, the lack of high quality documentation was definitly a hindrance to early development efforts.

Since our time was fast running out, we concentrated on a core set of features for our first iteration of STOA. First of all, we needed to verify users against their accounts on the main school network, a WinNT domain. Second, we wanted to present users with actions and links that were appropriate to them, whether they were students, teachers or administrators. Third, we wanted teachers to be able to post and manage static content of any type in their online course areas through simple form based interfaces.

The first hurdle was authenticating user logins from the NT domain. A search of the Zope and CMF sites revealed several products that seemed likely candidates. In fact, most of these products turned out to be inactive, broken on Zope 2.5 and higher, and undocumented. After trying several, one (LoginManager) was found to work in our environment, and implemented. Implementation required adding pam_smb to the server, recompiling Zope's instance of Python to support pam_smb, and then writing the external and Zope Python code for LoginManager. While none of these steps is particularly difficult for a seasoned administrator, any one of them could easily frustrate and discourage someone with less technical experience. Unfortunately, this experience seemed to be the template for our experiences with Zope and CMF products: a fairly straightforward feature request would lead to a brute force search of Zope and CMF sites, then to a painful sifting out of broken and/or inactive projects, followed by extensive hacking and tinkering to get possible candidates installed and running. We soon reached the point where we were reluctant to even investigate adding functionality unless we were confident we had the time to code it ourselves.

In comparison, tailoring the actions and links available to each user was quite straightforward, thanks to CMF's rich mechanisms for managing actions and content types. In addition to adding "tools" and "course" action categories we also created a "My Courses" list of links to course folders and derived a course folder type from the CMFCore ?PortalFolder type. The course folders were generated programmatically from our academic records database, giving teachers ownership of their course folders. Dropboxes for assignment submission were then created with a different script which gave the students enrolled in a course permission to submit assignments to that course's dropbox. This proved to be a hassle since the dropbox script had to be run on every drop/add. This led us to make the dropbox script available to teachers as a course action. While this strategy worked, requiring an extra administrative activity from teachers was far from ideal.

After the correct course folders were created it was a simple matter to script the generation of a list of links to the current user's courses based on the academic records database and then to include that list in the actions panel. That meant that course changes were instantly reflected in a student's or teacher's "My Courses" links. This feature has been so popular that any bugs in the "My Courses" list are showstoppers - many users are literally unable to navigate the site in any other way.

The ease with Python/Zope/CMF handled connecting to the academic records database made adding several other tools possible, including "live" versions of student and teacher schedules and rosters for classes, gradelevels, workcrews, etc. One of the big headaches in running a school is making sure that the rosters and schedules in "the office" match with reality. By making much of the official information available to the appropriate people via STOA, we were much more able to manage that challenge. This ease and flexibility of presenting content appropriate to the user easily justified the choice of Zope and CMF, far outweighing the other hassles.

Finally, in order to make adding content for teachers as simple as possible we created simplified new item forms and scripts to make posting and publishing of common content types a one or two click operation. Here again, instead of the documentation we had to rely on the source of the CMF defaults as our guide. However, following those models this modification was fairly easy to implement.

STOA went live in mid-August of 2002, providing students and teachers with access to the online folders for the courses they were currently taking and live schedule information. In addition, teachers got live roster information and a way to publish content without knowing HTML.

Due to our time contraints we did not attempt to support features like attendance, online discussion groups, or calendaring, and support for many administrative functions was rudimentary. We did add online report card writing and printing in time to process grades for the first term, and over the course of the year the capability to track activities, such as Community Service and Speech, was also added. The relatively painless nature of such additions demonstrated the easy extensibility of the platform.

So, in its first year STOA was used by all teachers and administrators K-12 for grade card writing and printing, and was used by many Middle School teachers and virtually all High School teachers on a regular basis for publishing content and accepting online submission of assignments. While we endured our share of minor problems and bugs in getting the system up and running, STOA was judged a success by faculty, administration, students, and parents (who used their children's accounts to log on and verify assignments, etc.) In particular, faculty who visited other schools came back impressed with how much more they liked STOA than the systems (if any) they saw in use elsewhere.

A Product Is Born

In the spring of 2003, it was becoming clear that TAL and Python Scripts alone would not be sufficient to grow Stoa in the manner which users were demanding. Teachers were asking for increasingly complex features, and the administrators were spending large portions of time writing and managing small scripts to keep the system in line. Scripts such as the previously mentioned "Enable Dropbox" action were inelegant hacks, and such scripts seemed to be multiplying. This proliferation of scripts also led to another problem: total disorganization. Our inexperience with Zope led us to create a scripts folder and a sql folder; all Python scripts went into scripts, all ZSQL methods went into sql and all page templates when into portal_skins/custom. This practice led to an amazing array of camel-case, ambiguous names, making building on existing components difficult.

As we researched what it would take to extend the CMF to fit our users' needs, we also realized that these three areas were desperately in need of clean-up. The plan that emerged consisted of two major development efforts. First, completely reorganize the layout of our scripts, SQL methods and page templates with a standard naming convention and location convention. Second, extend the CMF with a product to handle advanced functionality.

The first task was time consuming, but not difficult. We used a methodical approach: rename, replace references, test, fix the areas we missed. The task was simplified somewhat by the development of a simple script dubbed zgrep which has become a staple in our refactoring efforts. Zgrep is implemented as a simple Python script which takes a starting point and search string. It then returns a list of scripts, sql methods and page templates which contain the search string, along with an occurence count. There were very few situations where we actually had to rewrite code to accomodate the change; these were special cases where the orignal code was not well designed and as such did not easily fit into our new classification schema.

The second task, implementing the StoaCmf product, was more difficult. The lack of documentation mentioned earlier applies as much if not more to CMF. As a relative newbie exploring the world of Zope, CMF was the ultimate black box: capable of great power and amazing feats, but the way it happened was truly magical. Nathan led the development of the initial product, StoaCmf, which was intended to provide advanced content services to Stoa users. In particular, users wanted more control over presentation of their content, the ability to "share" folders in more than one place (common content across sections), and improved drop box support. Nathan had been some what careless during faculty presentations during the school year, promising "all this and more" in the next school year. In late June 2003, we began the second major round of development on Stoa.

In the process of implementing Garbo, a grade card composition system, Nathan had previously implemented a light-weight XML traversal object. While ?ObjDom worked, it was somewhat of a sham: it had just enough security assertions to be instantiated from within a Python script (it's actual domain), but tended to break in spectacular ways when created through the ZMI. With this experience under our belts, we felt we had at least some concept of how to proceed with development.

As is common when approaching a new development area, we began by searching for tutorials or examples on how to write CMF products. The Zope Book, our trusted reference through Python scripts and Page templates, was of no assistance. Most of the examples we found applied to ZClasses, and we had just enough experience with ZClasses to know they weren't for us.

The key to developing the StoaCmf product turned out to be reading the source, which had been our greatest ally from the start. As Nathan read the source for CMF, he realized that all that was really necessary was some basic presentation wrapping around CMF's provided Portal Folder. Thus an incremental approach to developing a product was born.

This approach had a very basic model: create a sub-class of an existing CMF object, write some initialization code (lots of find and replace), try to install it into Zope, curse, repeat. We repeat the cursing step a lot. The goal for the first few repetitions is to create a sub-class with no additional functionality. While seemingly counter-intuitive, this allowed us to figure out the registration mechanics, and then promptly forget them while we focused on actual functionality. After registration was taken care of we focused on the functionality. The result was StoaFolder. StoaFolder, in addition to having support for custom ordering and "flattened" views, allowed us to check for a specific type when trying to determine if something was Stoa-ish, or Course-ish. This provided a passable stand-in for actual objects and instance checking.

After developing the StoaFolder, we went on to develop the DropBox (a sub-class of StoaFolder, naturally) and Assignment (also a sub-class of StoaFolder). The DropBox provided more machinery around content creation and viewing. A DropBox has the unique property that fewer people can see content than can create content. A user with the Student role should be able to create content, but only users with the Teacher role should be able to actually view said content. The development of the DropBox also introduced an issue not encountered with StoaFolder: default permissions. Teachers were used to creating documents in Stoa and publishing them, but weren't familiar with local roles or advanced permissions. When we initially deployed DropBox, teachers could create and publish them as expected, but students did not have permission to drop their content. This was eventually solved with a call to self.manage_permission, but not before several abortive attempts to use self.rolesOfPermission to determine if the user held the correct permissions. An additional feature that was added to DropBox after deployment was the ability to retrieve the entire contents as a Zip file.

A feature of Manhattan that was lost with the initial move to Stoa was the ability to post multi-document assignments. These "bundles" of files were often used by teachers who wanted to combine the text of an assignment with images or supporting documents students would need to complete the assignment. Nathan decided to address this while developing the StoaCmf product with the introduction of an Assignment type.

The Assignment type followed the pattern of StoaCmf development closely: a container with additional simple behavior and more involved presentation bits. In this case, an Assignment object has a "main document" stored as an attribute which is displayed as part of the document's default view (instead of it's contents like a normal folder). Every assignment also has it's own DropBox automatically created upon instantiation. Finally, each assignment has some additional actions defined which map to the actions of the contained dropbox.

A final, late addition to the StoaCmf product was the ?FolderLink type. During the initial deployment of Stoa, many teachers requested the ability to share content between multiple sections of a course without needing to copy and paste it. Due to Stoa's section-centric design, this was difficult to overcome. The solution, an admitted kludge, was to provide a "link" type, which would allow teachers to create a single folder in their Member folder, and then link to it from each section's individual course folder. Additional machinery in the selection of the link target would restrict the link to only target StoaFolder s, thus limiting the targets to objects we were familiar with how to handle. We also decided to immediately redirect to the linked page instead of showing the interstitial with the URL that CMF's default Link type used.

This sounded great in theory: a simple solution which delivered 95% of what the user wanted. In practice it exposed several more thorny issues. First, the target selection and traversal machinery, while simple, was a little troublesome. Stoa can be accessed through a few different URL's, some of them exclusively internal to the Canterbury network. The initial iteration of the ?FolderLink just stored an absolute link to the target. However, if the user had accessed Stoa at the school through the school-only address, the link broke for all users outside of the school.

The second issue was even more frustrating. Member folders in our Stoa installation had very weak security by default: registered users could view their contents, and only the member herself could add content. However, when students attempted to click through a ?FolderLink, they received security errors stating they did not have permission to access the resource. The problem turned out to be in the breadcrumbs script which placed links to parent folders above the content area of the portal. The supplied script visits every parent folder when building the breadcrumbs. However, if a student doesn't have access to a parent folder, breadcrumbs breaks.

This problem was discovered after the start of school, and after teachers began expecting ?FolderLink to work as promised. Therefore the solution was a stupid one: place a breadcrumbs script in the Members folder which returns nothing. Through the magic of acquisition, pages in Member folders find the new breadcrumbs script which doesn't cause security violations by traversing the heirarchy. This solution, while a complete hack, also helped mask what was really going on behind the scenes with the ?FolderLink. During early demonstrations teachers complained (rightfully?) that the breadcrumbs changed when you clicked on the ?FolderLink. The response at the time was somewhere along the lines of "yes, it does change; next question." With the introduction of the breadcrumbs hack, teachers were actually relieved that the breadcrumbs just went away.

Lessons Learned: Product Development

In looking at the current state of the Zope3 project, it's heartening to see that documentation is apparently an important part of the project. The lack of quality, concise, accurate documentation is a huge problem in developing for Zope 2.x, and while the situation has improved somewhat in the past 12 to 6 months, it still exists. Knowing what we know from the development of the StoaCmf product, there are three things we would do differently.

First, read the source early and read it often. A huge asset to Zope developers is the source code, but often we don't consider it as a source of documentation. It may not be the most efficient way of learning about a product, but it is reliable. Another boost in this area came from the ?DocFinder product, which adds docstring extraction to the ZMI.

Second, steal what works. When Nathan began working on StoaCmf, he approached it with a clean slate. It wasn't until he shamelessly cribbed code from CMFDefault that things started moving forward. This goes hand in hand with the previous point.

Finally, plan small milestones. This lesson from extreme programming teaches that small, incremental improvment is important. This goes double for Zope development: there are enough variables involved that for developers new to Zope, just getting things registered with no new functionality is a huge step (not to mention confidence builder).

Products are the most powerful way to add functionality to Zope. It is no coincidence they are also the most complex. However, learning from what works enables you to create and easily distribute your additions to Zope.

Lessons Learned: Development Practices

The initial development of Stoa happened on what was eventually our live Zope instance. This had as much to do with resources as with difficulty of implementing a source control system. Our initial attempts to deploy products which purported to syncronize the ZODB with CVS, such as ZCVSFolder, were fraught with difficulties and instabilities. In retrospect, it would have been worth the effort to develop our current practices early on, rather than in response to breaking the live instance one too many times.

Our current development cycle includes a test, or staging, server, the live server and ad hoc development instances. The staging and live servers are supposed to be relatively stable: code is developed on the staging server, tested, and then syncronized to the live server using ZSyncer. It is assumed that all code developed on the staging server is syncronized as it stablizes. This has caused some conflicts with long running development tasks. In those cases, not all dependencies have been able to be syncronized at the same time. For this reason we are currently developing ways of packaging Stoa on the filesystem so that CVS, or other version control, can be used. This, along with ad hoc development instances, would allow us to use the full power of branching, merging and change syncronization. In general, however, ZSyncer has worked very well for us and we anticipate continuing to use it indefinitly.

Current Development

The Stoa CMF product developed in 2003 has undergone minor updates throughout the school year to fix bugs discovered during heavy usage. We have recently begun working to package Stoa outside of the ZODB. This has led to the development of the StoaSkin product, which provides the necessary page templates, Python scripts and ZSQL methods for converting a CMF portal into a Stoa instance. Due in large part to the massive reorganization undertaken earlier in the project, this has required only minimal effort so far.

An unexpected advantage of this packaging effort has been the ability to explore database abstraction more easily. Stoa was developed with MySQL as it's sole backend, but does not take advantage of any MySQL-specific features. This, combined with the installation process of the StoaSkin product, in theory allows users to simply swap out the database connection. We are currently in the process of testing with PostrgreSQL in order to verify this theory.

The Future of Stoa

Our future development efforts are shifting away from the classroom and towards the administrative functions of Stoa. We have recently integrated student schedule management with Stoa, and are working on porting other back-office functions to the Stoa platform. Our goal at Canterbury is to drive our infrastructure from a single unified database. In addition to integrating administrative functions, we are working on more fully integrating workflow into Stoa through the use of DCWorkflow. Our goal is to provide an easy to use interface for administrators to create simple workflows for document routing and management.

Scalability is also an issue under discussion. Stoa's performance, while generally good, tends to decline rapidly under a heavy load. In a school environment this is a common occurance, especially as faculty members use Stoa to compose grades and comments at the end of each term. While we believe ZEO will solve this problem, we have not configured or tested that yet. Once the pending upgrade to Zope 2.7 and CMF 1.4.2 this will become a more pressing issue.

Finally, there has been some discussion about the possibility of "objectifying" everything; that is, move to a more object oriented approach, such that courses, students, etc are actual Python objects. However this is a major architectural shift, and as such probably won't be undertaken until there is a compelling argument for it. It is possible such a task will be undertaken if and when Stoa is ported to Zope 3.

Conclusion

Stoa is presently a stable, successfully implemented Zope application. Through the course of developing Stoa, Zope has proven to be a powerful platform for development of web applications. The lessons learned from the development of Stoa show that Zope is a powerful, flexible, extensible platform for developing web applications. The power and flexibility of Zope means that developing for Zope is like nothing else: concepts such as acquisition and products are very different conceptually than many other programming tasks. Zope 2.x suffers from a lack of tutorials, examples and correct documentation. This, along with the difficulty in implementing basic revision control using standard tools make learning Zope a challenge for newcomers.

Resources

at Canterbury

other products

online documentation & websites

print documentation

CategoryStoa CategoryZope