File Cache

Introduction

JamPlus provides an MD5sum based file caching system, provided by Alen Ladavac. Using all relevant dependency information specified during the Parsing Phase to guarantee absolute uniqueness within the file cache, the JamPlus file caching system is able to dramatically speed builds by retrieving precached target contents from a (usually) shared file cache, most often located on a network share. This can be used as a poor man's distributed build system, as one machine can populate the file cache while the other machines do a minimum of work and just merely retrieve the prebuilt targets.

The most important thing to note is JamPlus will do a deep dependency scan to calculate the proper file cache entry. If you find yourself safely using incremental builds in Jam, then the file cache feature could be for you. If, on the other hand, you haven't specified your dependencies properly in Jam and are often doing full rebuilds "just in case", steer clear of using the file cache. It will just cause you pain.

Note

The file cache will only be available if the dependency cache is turned on.

Creating and Using File Caches

Creation of a File Cache

A single file cache is defined as follows:

FILECACHE.cachename.PATH = \\\\uncshare\\networkcache\\cachename ;
FILECACHE.cachename.GENERATE = 1 ;
FILECACHE.cachename.USE = 1 ;

The default file cache's name is generic. When rule UseFileCache TARGETS [ : CACHE_NAME ] ; is specified without a CACHE_NAME, the default is generic. Given this is the most common case, you'll most always want to declare the generic file cache similar to:

FILECACHE.generic.PATH = \\\\mynetworkcacheserver\\generic ;
# or perhaps:
#   FILECACHE.generic.PATH = c:/netcache/generic ;
FILECACHE.generic.GENERATE = 1 ;
FILECACHE.generic.USE = 1 ;

It is not necessary to manually create the directory populated by the file cache.

When FILECACHE.cachename.GENERATE is set to 1, any built targets not existing in the file cache are copied to the cache and made available for the next build to retrieve. If FILECACHE.cachename.USE is set to 0 and the built target does not exist in the file cache, no transfer to the cache is made.

When FILECACHE.cachename.USE is set to 1, any available targets in the file cache are retrieved, if they match the appropriate md5sum. If the target is not in the file cache, it is built locally. If FILECACHE.cachename.USE is set to 0, the target is always built locally. No cache access is made.

Additional (generally categoric) file caches may be specified in the same way as the generic file cache. Simply assign a different cachename, and you'll be able to access the new cache.

Using the File Cache

The JamPlus file cache is turned on for a given target by using the UseFileCache or OptionalFileCache rules.

In the example below, the only additional line over the standard Jam rule definition is the usage of the UseFileCache rule. Nothing further must be specified, although there are things you can do that will break shared access to the cache. Those will be discussed below.

rule ConvertImage SOURCE
{
    local target = $(SOURCE:S=.image) ;
    MakeLocate $(target) : $(LOCATE_TARGET) ;
    Clean clean : $(target) ;
    UseFileCache $(target) ;
 
    Depends all : $(target) : $(SOURCE) ;
    ConvertImageHelper $(target) : $(SOURCE) ;
}

Technical Details

To use the JamPlus file cache facility properly, the basics of how the target's buildmd5sum is generated is worth knowing.

For any given target or dependent target, the name of the target should generally be a "shareable" gristed name, not the physical location of the target, which could vary from machine to machine. That is, if the name of the target is c:/content/file.image on one machine and on another is d:/morecontent/file.image, the cache will never be able to share content. A name like <content>file.image or just simply file.image is considered a shared name and is safe to use with the file cache.

The following items are added to a target's buildmd5sum.

1. The target's name.
2. If UseCommandLine was specified on the target, each command line entry string is summed.
3. For each dependency, sorted by name:
   1. The name of the dependency.
   2. The md5sum of the physical content of the dependency is summed and added to the current buildmd5sum.
   3. If UseCommandLine was specified on the dependency, each command line entry string is summed.
   4. If there are includes specified through the rule Includes, the string #includes is summed.
      1. For every includes dependency, the physical content is summed and added to the current buildmd5sum.
   5. Any dependents of this dependency are added.

Content MD5sums

Without any additional guidance, the md5sum for a physical file will include all its contents. This is always the most accurate, although the price to calculate md5sums for a large amount of content may be high.

JamPlus provides an additional method of calculating md5sums by calling into a Lua script. Any target's MD5 calculating functionality may be replaced by using the rule UseMD5Callback on the target.

Example: A PNG file is made of a small number of chunks, and each chunk has a size and a CRC. Instead of calculating the entire file contents, we'll just collect the CRCs and sum them.

require 'md5'
 
function md5png(filename)
    print("md5png: Calculating " .. filename .. "...")
    local file = io.open(filename, 'rb')
    if not file then return nil end
 
    file:seek('cur', 8)
 
    local md5sum = md5.new()
    local offset
    while true do
        local length = select(2, string.unpack(file:read(4), '>I'))
        local chunkType = file:read(4)
        if chunkType == 'IEND' then break end
        file:seek('cur', length)
        local crc = file:read(4)
        md5sum:update(crc)
    end
 
    return md5sum:digest(true)
end