Node Subsystem

The node subsystem of the Build Engine is responsible for managing the knowledge in SCons of the relationships among the external objects (files) it is responsible for updating. The most important of these relationships is the dependency relationship between various Node objects, which SCons uses to determine the order in which builds should be performed.

The scons script (or other user interface) tells the Build Engine about dependencies through its construction environment API. The Build Engine also discovers dependencies automatically through the use of Scanner objects.

Subclasses of the Node class maintain additional relationships that reflect the real-world existence of these objects. For example, the Node.FS subclass is responsible for managing a representation of the directory hierarchy of a file system.

A Walker class is used by other subsystems to walk the dependency tree maintained by the Node class. The Walker class maintains a stack of Node objects visited during its depth-first traversal of the dependency tree, and uses an intermediate node Wrapper class to maintain state information about a Node object's dependencies.

Scanner Subsystem

The scanner subsystem is responsible for maintaining objects that can scan the contents of a Node's for implicit dependencies.

In practice, a given Scanner subclass object functions as a prototype, returning clones of itself depending on the construction environment values governing how the Node should be scanned.

Signature Subsystem

The signature subsystem is responsible for computing signature information for Node objects. The signature subsystem in SCons supports multiple ways to determine whether a Node is up-to-date by using an abstract Sig class as a strategy wrapper:

By default, SCons tracks dependencies by computing and maintaining MD5 signatures for the contents of each source file (or other object). The signature of a derived file consists of the aggregate of the signatures of all the source files plus the command-line string used to build the file. These signatures are stored in a .sconsign file in each directory.

If the contents of any of the source files changes, the change to its MD5 signature is propogated to the signature of the derived file(s). The simple fact that the new signature does not match the stored signature indicates that the derived file is not up to date and must be rebuilt.

A separate TimeStamp subclass of the Sig class supports the use of traditional file timestamps for deciding whether files are up-to-date.

Builder Subsystem

The SCons Build Engine records how out-of-date files (or other objects) should be rebuilt in Builder objects, maintained by the builder subsystem:

The actual underlying class name is BuilderBase, and there are subclasses that can encapsulate multiple Builder objects for special purposes. One subclass (CompositeBuilder) selects an appropriate encapsulated Builder based on the file suffix of the target object. The other (MultiStepBuilder). can chain together multiple Builder objects, for example, to build an executable program from a source file through an implicit intermediate object file.

A BuilderBase object has an associated ActionBase object responsible for actually executing the appropriate steps to update the target file. There are three subclasses, one for externally executable commands (CommandAction), one for Python functions (FunctionAction), and one for lists of multiple Action objects (ListAction).

Job/Task Subsystem

SCons supports parallel builds with a thread-based tasking model, managed by the job/task subsystem.

Instead of performing an outer-loop recursive descent of the dependency tree and then forking a task when it finds a file that needs updating, SCons starts as many threads as are requested, each thread managed by the Jobs class. As a performance optimization, the Jobs class maintains an internal distinction between Serial and Parallel build jobs, so that serial builds don't pay any performance penalty by using a multi-threaded implementation written for Parallel builds.

Each Jobs object, running in its own thread, then requests a Task from a central Taskmaster, which is responsible for handing out available Task objects for (re-)building out-of-date nodes. A condition variable makes sure that the Jobs objects query the Taskmaster one at a time.

The Taskmaster uses the node subsystem's Walker class to walk the dependency tree, and the Sig class to use the appropriate method of deciding if a Node is up-to-date.

This scheme has many advantages over the standard Make implementation of -j. Effective use of -j is difficult with the usual recursive use of Make, because the number of jobs started by -j multiply at each level of the source tree. This makes the actual number of jobs executed at any moment very dependent on the size and layout of the tree. SCons, in contrast, starts only as many jobs as are requested, and keeps them constantly busy (excepting jobs that block waiting for their dependency files to finish building).

Construction Variables

In SCons, a construction environment is an object through which an external interface (such as the scons script) communicates dependency information to the SCons Build Engine.

A construction environment is implemented as a dictionary containing:

• construction variables, string values that are substituted into command lines or used by builder functions;

• one or more Builder objects that can be invoked to update a file or other object;

• one or more Scanner objects that can be used to scan a file automatically for dependencies (such as files specified on #include lines).

Construction environments are instantiated as follows:

      env = Environment()
      env_debug = Environment(CCFLAGS = '-g')
    

Builder Objects

An SCons Builder object encapsulates information about how to build a specific type of file: an executable program, an object file, a library, etc. A Builder object is associated with a file through an associated construction environment method and later invoked to actually build the file. The Builder object will typically use construction variables (such as CCFLAGS, LIBPATH) to influence the specific build execution.

Builder objects are instantiated as follows:

      bld = Builder(name = 'Program', action = "$CC -o $TARGET $SOURCES")
    

In the above example, the action is a command-line string in which the Build Engine will interpolate the values of construction variables before execution. The actual action specified, though, may be a function:

      def update(dest):
          # [code to update the object]
          return 0

      bld = Builder(name = 'Program', function = update)
    

Or a callable Python object (or class):

      class class_a:
          def __call__(self, kw):
              # build the desired object
          return 0

      builder = SCons.Builder.Builder(action = class_a())
    

A Builder object may have the prefix and suffix of its target file type specified as keyword arguments at instantiation. Additionally, the suffix of the source files used by this Builder to build its target files may be specified using the src_suffix keyword argument:

      bld_lib = Builder(name = 'Library', action = "$AR r $TARGET $SOURCES",
                        prefix = 'lib', suffix = '.a', src_suffix = '.o')
    

The specified prefix and suffix will be appended to the name of any target file built by this Builder object, if they are not already part of the file name. The src_suffix is used by the SCons Build Engine to chain together multiple Builder objects to create, for example, a library from the original source files without having to specify the intermediate .o files.

Builder objects are associated with a construction environment through a construction variable named BUILDERS, a list of the Builder objects that will be available for execution through the construction environment:

      env = Environment(BUILDERS = [ Object, Library, WebPage, Program ])
    

Scanner Objects

Scanner objects perform automatic checking for dependencies by scanning the contents of files. The canonical example is scanning a C source file or header file for files specified on #include lines.

A Scanner object is instantiated as follows:

      def c_scan(contents):
           # scan contents of file
           return # list of files found

      c_scanner = Scanner(name = 'CScan', function = c_scan,
                          argument = None,
                          skeys = ['.c', '.C', '.h', '.H')
    

The skeys argument specifies a list of file suffixes for file types that this Scanner knows how to scan.

Scanner objects are associated with a construction environment through a construction variable named SCANNERS, a list of the Scanner objects that will be available through the construction environment:

      env = Environment(SCANNERS = [ CScan, M4Scan ])
    

For utilities that will build files with a variety of file suffixes, or which require unusual scanning rules, a Scanner object may be associated explicitly with a Builder object as follows:

      def tool_scan(contents):
          # scan contents of file
          return # list of files found

      tool_scanner = Scanner(name = 'TScan', function = tool_scan)

      bld = Builder(name = 'Tool', scanner = tool_scanner)
    

BuildDir

SCons supports a flexible mechanism for building target files in a separate build directory from the source files. The BuildDir syntax is straightforward:

      BuildDir(source = 'src', build = 'bld')
    

By default, source files are linked or copied into the build directory, because exactly replicating the source directory is sometimes necessary for certain combinations of use of #include "..." and -I search paths. An option exists to specify that only output files should be placed in the build directory:

      BuildDir(source = 'src', build = 'bld', no_sources = 1)
    

Repository

SCons supports the ability to search a list of code repositories for source files and derived files. This works much like Make's VPATH feature, as implemented in recent versions of GNU Make. (The POSIX standard for Make specifies slightly different behavior for VPATH.) The syntax is:

      Repository('/home/source/1.1', '/home/source/1.0')
    

A command-line -Y option exists to allow repositories to be specified on the command line, or in the SCONSFLAGS environment variable (not construction variable!). This avoids a chicken-and-egg situation and allows the top-level SConstruct file to be found in a repository as well.

Cache

SCons supports a way for developers to share derived files. Again, the syntax is straightforward:

      Cache('/var/build.cache/i386')
    

Copies of any derived files built will be placed in the specified directory with their MD5 signature. If another build results in an out-of-date derived file with the same signature, the derived file will be copied from the cache instead of being rebuilt.

Testing Criteria

The underlying criteria for testing changes to the SCons code are taken from Aegis:

• Every change must have one or more new or modified tests checked in along with the code.

• The new code being checked in must pass all of the new and/or modified tests.

• The old, already checked-in code in must fail all of the new and/or modified tests.

• The new code being checked in must pass all unmodified, already checked-in tests.

In practice, these restrictions can be overridden as necessary­for example, when changing comments or documentation.

The criterion that surprises many people is having the old code fail the tests in the change. This makes sure that the new tests or modified tests really do exercise the bug fix or feature being added by the change.

Together, these criteria ensure that every newly checked-in version SCons conforms to defined behavior, as defined by the tests. Whenever a bug is found, its fix is checked in with a new or modified test that guarantees the bug will not recur in the future. We have already built up a regression test base of almost 90 tests that cover the vast majority of SCons' functionality.

Testing Infrastructure

Testing standards are no good if they're too much of a burden for developers, who will at best work around or ignore the testing requirements, or at worst stop contributing code and go join a project that's more fun. To this end, good testing infrastructure that makes it easy to write tests is crucial.

SCons development uses two development methodologies, one for the individual modules in the build engine, and the other for end-to-end tests of the SCons script.

For the build engine modules, we use PyUnit. Every change to a build engine module must have a change to its corresponding unit tests, which live side-by-side in a separate file that imports module. As we build up a large body of unit tests, this ensures that the build engine will perform correctly whenever someone uses it in some application other than the SCons script itself.

For end-to-end script tests, we have developed two modules to make writing tests easy. The first, TestCmd.py, is a generic module for testing commands or scripts (in any language, not just Python). The second module, TestScons.py, is a subclass of the generic TestCmd.py module. TestScons.py takes care of initialization and displaying error conditions specific to testing SCons.

In practice, simple tests only need to initialize a test object, use the object to write some input files, run SCons, and then check whatever criteria determine whether the test passed or failed. A complete test of the Program method, for example, looks like this:

      test = TestSCons.TestSCons()

      test.write('SConstruct',
      """env = Environment()
      env.Program(target = 'foo', source = 'foo.c')
      """)

      test.write('foo.c',
      """
      int
      main(int argc, char *argv[])
      {
          argv[argc++] = "-"; /* dummy use of args */
          printf("foo.c successfully compiled\\n");
          exit (0);
      }
      """)

      test.run(arguments = 'foo') # runs SCons

      test.run(program = test.workpath('foo'))

      test.fail_test(test.stdout() != "foo.c successfully compiled\n")

      test.pass_test()