Tutorial¶

Let’s kick off with a few examples, getting more into details as we move along.

First example¶

Our first example is a classic. Copy/paste the code below into a file called first_example.jolt.

from jolt import *

class HelloWorld(Task):
    def run(self, deps, tools):
        print("Hello world!")

This task will simply print “Hello world!” on the console when executed. The task name is automatically derived from the class name, but can be overridden by setting the name class attribute.

Now try to execute the task:

$ jolt build helloworld

Once the task has been executed, executing it again won’t have any effect. This is because the task produced an empty artifact which is now stored in the local artifact cache. When Jolt determines if a task should be executed or not, it first calculates a task identity by hashing different attributes that would influence the output of the task, such as:

Source code
Name
Parameters
Dependencies

When the task identity is known, Jolt searches its artifact cache for an artifact with the same identity. If one is found, no action is taken. You can try this out by changing the Hello world! string to something else and executing the task again. If the string is reverted back to Hello world!, the task will regain its first identity and no action will be taken because that artifact is still present in the cache.

To clean the local cache and remove all artifacts, run:

$ jolt clean

To selectively remove a specific task’s artifacts, run:

$ jolt clean helloworld

Publishing Files¶

Tasks that don’t produce output are not very useful. Let’s rework our task to instead produce a file with the Hello world! message. We also shorten its name to hello.

from jolt import *

class HelloWorld(Task):
    """ Creates a text file with cheerful message """

    name = "hello"

    def run(self, deps, tools):
        with tools.cwd(tools.builddir()):
            tools.write_file("message.txt", "Hello world!")

    def publish(self, artifact, tools):
        with tools.cwd(tools.builddir()):
            artifact.collect("*.txt")

The implementation of the task is now split into two methods, run and publish.

The run method performs the main work of the task. It creates a file called message.txt containing our greeting from the first example. The file is written into a temporary build directory that will persist for the duration of the task’s execution. The directory is removed afterwards.

The publish method collects the output from the work performed by run. It does so by instructing the artifact to collect all textfiles from the build directory.

$ jolt build hello

After executing the task an artifact will be present in the local cache. Let’s investigate its contents, but first we need to know the identity of the task in order to know what artifact to look for. Run:

$ jolt inspect -a hello

The inspect command displays information about the task, including the documentation written in its Python class implementation. We’re looking for the identity:

Identity          50a215905eb28a0911ff83828ac56b542525bce4

With this identity digest at hand, we can dive into the artifact cache. By default, the cache is located in $HOME/.cache/jolt. To list the content of the current hello artifact, run:

$ ls $HOME/.cache/jolt/hello/50a215905eb28a0911ff83828ac56b542525bce4

You will see the message.txt file just created.

Parameters¶

Next, we’re going to use a task parameter to alter the Hello world! message. Instead of greeting the world, we’ll allow the executor to specify an alternative recipient. We rename the class to reflect this change and we also add a parameter class attribute. The run method is changed to use the new parameter’s value when writing the message.txt file.

class Hello(Task):
    """ Creates a text file with a cheerful message """

    recipient = Parameter(default="world", help="Name of greeting recipient.")

    def run(self, deps, tools):
        with tools.cwd(tools.builddir()):
            tools.write_file("message.txt", "Hello {recipient}!")

    def publish(self, artifact, tools):
        with tools.cwd(tools.builddir()):
            artifact.collect("*.txt")

By default, the produced message will still read Hello world! because the default value of the recipient parameter is world. To produce a different message, try this:

$ jolt build hello:recipient=John

Dependencies¶

To better illustrate the flexibility of the new parameterized task, let’s add another task class, Print, which prints the contents of the message.txt file to the console. Print will declare a dependency on Hello.

class Print(Task):
    """ Prints a cheerful message """

    recipient = Parameter(default="world", help="Name of greeting recipient.")
    requires = ["hello:recipient={recipient}"]
    cacheable = False

    def run(self, deps, tools):
        hello = deps["hello:recipient={recipient}"]
        with tools.cwd(hello.path):
            print(tools.read_file("message.txt"))

The output from this task is not cacheable, forcing the task to be executed every time. It’s dependency hello however, will only be re-executed if its influence changes, for example by passing new values to the recipient parameter. Try it out:

$ jolt build print:recipient=John
$ jolt build print:recipient=Lisa
$ jolt build print:recipient=Kelly

Tools¶

The run and publish methods take a tools argument as their last parameter. This toolbox provides a large set of tools useful for many different types of tasks. See the reference documentation for more information.

However, Jolt was originally created with compilation tasks in mind. Below is a real world example of a task compiling the e2fsprogs package containing EXT2/3/4 filesystem utility programs. It uses AutoTools to configure and build its sources into different binary applications. Luckily, the tools object provides utilities for building autotools projects as seen below. In addition to AutoTools, there is also support for CMake and Meson as well as generic support for running any third-party build tool.

from jolt import *
from jolt.plugins import git


class E2fsprogs(Task):
    """ Ext 2/3/4 filesystem utilities """

    requires = ["git:url=git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git"]

    def run(self, deps, tools):
        ac = tools.autotools()
        ac.configure("e2fsprogs")
        ac.build()
        ac.install()

    def publish(self, artifact, tools):
        ac = tools.autotools()
        ac.publish(artifact)
        artifact.environ.PATH.append("bin")

The autotools ac object automatically creates temporary build and install (–prefix) directories which are used when configuring, building and installing the project. All files installed in the installation directory will be published. Both directories are removed when execution has finished, i.e. the project will be completely rebuilt if the task’s influence changes.

The task also extends the environment of consumer tasks by adding the artifact’s bin directory to the PATH. That way, any task that depends on e2fsprogs will be able to run its published programs directly without explicitly referencing the artifact where they reside. Use this method to package tools required by other tasks.

Also, note that the task requires a git repository hosted at kernel.org. This git task, implemented by a builtin plugin, is actually not a task but a resource. You can read more about resources next.

Resources¶

Resources are a special kind of task only executed in the context of other tasks. They are invoked to acquire and release a resource before and after the execution of a task. No artifact is produced by a resource.

A common use-case for resources is to allocate and reserve equipment required during the execution of a task. Such equipment could be a build server or a mobile device on which to run tests.

Below is a skeleton example providing mutual exclusion:

from jolt import *

class Exclusivity(Resource):
    """ Resource providing mutual exclusion to an object """

    to = Parameter(help="Name of shared object")

    def acquire(self, artifact, deps, tools):
        # TODO: Implement locking
        self.info("{to} is now locked")

    def release(self, artifact, deps, tools):
        # TODO: Implement unlocking
        self.info("{to} is now unlocked")


class RebootDevice(Task):
    """ Reboots the specified test device """

    device = Parameter(help="Name of device to reboot")
    requires = ["exclusivity:to={device}"]
    cacheable = False

    def run(self, deps, tools):
        tools.run("ssh {device} reboot")

Tests¶

After implementing the e2fsprogs task above, the next logical step is to write a few test-cases for the utility programs it builds. Luckily, Jolt has integrated test support.

Test tasks are derived from the Test base class instead of Task and they are implemented like a regular Python unittest.TestCase. You can use all assertions and decorators like you normally would. In all other respects, a Test task behaves just like a regular Task.

Below is an example:

from jolt import *

class E2fsTest(Test):
    requires = ["e2fsprogs"]

    def setup(self, deps, tools):
        self.tools = tools

    def test_mke2fs(self):
        self.assertTrue(self.tools.run("mke2fs"))

    def test_badblocks(self):
        self.assertTrue(self.tools.run("badblocks"))

    def test_tune2fs(self):
        self.assertTrue(self.tools.run("tune2fs"))

Influence¶

It is important that all attributes that define the output of a task are known and registered to avoid false cache hits. For example, in a compilation task all compiled source files should influence the task’s identity and trigger re-execution of the task if changed, otherwise binary compatibility will be lost quickly.

When using an external third-party build tool such as make, Jolt has no way of knowing what source files to monitor. This information must be explicitly provided by the task’s implementor. Luckily, Jolt provides a few builtin class decorators to make it easier.

Let’s revisit the e2fsprogs task from earlier, but this time we assume that the repository is already cloned and managed by external tools and not through the builtin Jolt git resource. We can no longer rely on the resource to automatically influence the hash of the task. We instead use the git.influence decorator:

from jolt import *
from jolt.plugins import git

@git.influence(path="path/to/e2fsprogs")
class E2fsprogs(Task):
    def run(self, deps, tools):
        ac = tools.autotools()
        ac.configure("path/to/e2fsprogs")
        ac.build()
        ac.install()

The decorator adds the git repository’s tree hash as hash influence. It will also add the git diff output as influence to simplify iterative local development.

There are a number of other useful influence decorators as well:

from jolt import *
from jolt import influence
from jolt.plugins import git

@influence.files("path/to/e2fsprogs/*.c")
@influence.environ("CFLAGS")
@influence.weekly
@influence.attribute("webstatus")
class E2fsprogs(Task):
    @property
    def webstatus(self):
        r = requests.get("http://statusindicator/")
        return r.text

    def run(self, deps, tools):
        ac = tools.autotools()
        ac.configure("path/to/e2fsprogs")
        ac.build()
        ac.install()
        self.report()

Above, the git.influence decorator has been replaced by influence.files. The result is virtually the same, the content of all files matched by the provided pattern will influence the hash of the task. However, the Git tree hash implementation is more effecient and faster, but it obviously doesn’t work if sources reside in a different type of repository.

The influence.environ decorator is used to influence the hash of the task based on the value of the CFLAGS environment variable. If the value of the variable changes the task will be re-executed.

The influence.weekly decorator adds the week number as hash influence. If nothing else changes, the task will be re-executed once every week. This can be useful to verify that external resources, such as files downloaded from the Internet, are still available. Other time-based decorators include:

influence.yearly
influence.montly
influence.daily
influence.hourly

The influence.attribute decorator adds the value of an attribute or property as hash influence. Above, the webstatus property is registered to influence the task with data obtained from a web service. The source code of the property itself is monitored automatically.

Ninja¶

Ninja is a fast third-party build system. Where other build systems, such as Jolt, are high-level languages, Ninja aims to be an assembler. Together they form a powerful couple. Jolt has builtin Ninja tasks which automatically generate Ninja build files and build your projects for you. All you have to do is to tell Jolt which source files to compile. You can also define custom build rules for file types not recognized by Jolt, see the Rule class.

Basics¶

Below is an example of a library and a program. The library contains a function returning a message. The program calls this function and prints the message.

// lib/message.cpp

#include "message.h"

const char *message() {
  return "Hello " RECIPIENT "!";
}

// program/main.cpp

#include <cstdlib>
#include <iostream>
#include "lib/message.h"

int main() {
  std::cout << message() << std::endl;
  return EXIT_SUCCESS;
}

To build the library and the program we use this Jolt recipe:

from jolt import *
from jolt.plugins.ninja import *


class Message(CXXLibrary):
    recipient = Parameter(default="world", help="Name of greeting recipient.")
    headers = ["lib/message.h"]
    sources = ["lib/message.cpp"]
    macros = ['RECIPIENT="{recipient}"']


class HelloWorld(CXXExecutable):
    requires = ["message"]
    sources = ["program/main.cpp"]

Metadata¶

Jolt automatically configures include paths, link libraries, and other build attributes for the HelloWorld program based on metadata found in the artifact of the Message library task. In the example, the Message library task relies upon CXXLibrary.publish to collect public headers and to export the required metadata such as include paths and linking information. Customization is possible by overriding the publish method as illustrated below. These two implementations of Message are equivalent.

class Message(CXXLibrary):
    recipient = Parameter(default="world", help="Name of greeting recipient.")
    sources = ["lib/message.*"]
    macros = ['RECIPIENT="{recipient}"']

    def publish(self, artifact, tools):
        with tools.cwd("{outdir}"):
            artifact.collect("*.a", "lib/")
            artifact.cxxinfo.libpaths.append("lib")
        artifact.collect("lib/*.h", "include/")
        artifact.cxxinfo.incpaths.append("include")

The cxxinfo artifact metadata can be used with other build systems too, such as CMake, Meson and Autotools. It enables your Ninja tasks to stay oblivious to whatever build system their dependencies use as long as binary compatibility is guaranteed.

Parameterization¶

To support build customization based on parameters, several class decorators can be used to extend a task with conditional build attributes.

The first example uses a boolean debug parameter to disable optimizations and set a preprocessor macro. The decorators enable Ninja to consider alternative attributes, in addition to the default cxxflags and macros. The names of alternatives are expanded with the values of parameters. When the debug parameter is assigned the value true, the cxxflags_debug_true and macros_debug_true attributes will be matched and included in the build. If the debug parameter value is false, no extra flags or macros will be included because there are no cxxflags_debug_false and macros_debug_false attributes in the class.

@ninja.attributes.cxxflags("cxxflags_debug_{debug}")
@ninja.attributes.macros("macros_debug_{debug}")
class Message(ninja.CXXLibrary):
    debug = BooleanParameter()
    cxxflags_debug_true = ["-g", "-Og"]
    macros_debug_true = ["DEBUG"]
    sources = ["lib/message.*"]

The next example includes source files conditionally.

@ninja.attributes.sources("sources_{os}")
class Message(ninja.CXXLibrary):
    os = Parameter(values=["linux", "windows"])
    sources = ["lib/*.cpp"]
    sources_linux = ["lib/posix/*.cpp"]
    sources_windows = ["lib/win32/*.cpp"]

Influence¶

The Ninja tasks automatically let the content of the listed header and source files influence the task identity. However, sometimes source files may #include headers which are not listed. This is an error which may result in objects not being correctly recompiled when the header changes. To protect against such errors, Jolt uses output from the compiler to ensure that files included during a compilation are properly influencing the task.

In the example below, the message.h header is no longer listed in headers, nor in sources.

from jolt import *
from jolt.plugins.ninja import *

class Message(CXXLibrary):
    sources = ["lib/message.cpp"]

Assuming message.cpp includes message.h, this would be an error because Jolt no longer tracks the content of the message.h header and message.cpp would not be properly recompiled. However, thanks to the builtin sanity checks, trying to build this library would fail:

$ jolt build message
[  ERROR] Execution started (message b9961000)
[ STDOUT] [1/2] [CXX] message.cpp
[ STDOUT] [1/2] [AR] libmessage.a
[WARNING] Missing influence: message.h
[  ERROR] Execution failed after 00s (message b9961000)
[  ERROR] task is missing source influence (message)

The solution is to ensure that the header is covered by influence, either by listing it in headers or sources, or by using an influence decorator such as @influence.files.

class Message(CXXLibrary):
    sources = ["lib/message.h", "lib/message.cpp"]

Headers from artifacts of dependencies are exempt from the sanity checks. They already influence the consuming task implicitly. This is also true for files in build directories.

Custom Rules¶

Rules are used to transform files from one type to another. An example is the rule that compiles a C/C++ file to an object file. Ninja tasks can be extended with additional rules beyond those already builtin and the builtin rules may also be overridden.

To define a new rule for a type of file, assign a Rule object to an arbitrary attribute of the compilation task being defined. Below is an example where a rule has been added to generate Qt moc source files from headers.

class MyQtProject(CXXExecutable):
    sources = ["myqtproject.h", "myqtproject.cpp"]

    moc_rule = Rule(
        command="moc -o $out $in",
        infiles=[".h"],
        outfiles=["{outdir}/{in_path}/{in_base}_moc.cpp"])

The moc rule applies to all .h header files listed as sources, i.e. myqtproject.h. It takes the input header file and generates a corresponding moc source file, myqtproject_moc.cpp. The moc source file will then automatically be fed to the builtin compiler rule from which the output is an object file, myqtproject_moc.o.

Below, another example illustrates how to override one of the builtin compilation rules. The example also defines an environment variable that will be accessible to the rule.

class MyQtProject(CXXExecutable):
    sources = ["myqtproject.h", "myqtproject.cpp"]

    custom_cxxflags = EnvironmentVariable()

    cxx_rule = Rule(
        command="g++ $custom_cxxflags -o $out -c $in",
        infiles=[".cpp"],
        outfiles=["{outdir}/{in_path}/{in_base}{in_ext}.o"])

$ CUSTOM_CXXFLAGS=-DDEBUG jolt build myqtproject

Toolchains¶

Maintaining binary compatibility between libraries can be a pain. To ensure that a chain of dependencies stay compatible you could inject a synthetic toolchain task at the bottom of your dependency tree and use it to control all compiler options. This methods also enables easy cross-compilation.

First, define a toolchain task:

class Toolchain(Task):
    arch = Parameter("i386", values=["i386", "arm"])
    host = Parameter(platform.system())
    debug = BooleanParameter(False)

    def publish(self, artifact, tools):
        if self.arch.get_value() == "arm":
            artifact.environ.CC = "arm-linux-gnueabi-gcc"
        if self.arch.get_value() == "i386":
            artifact.environ.CC = "x86_64-linux-gnu-gcc -m32"
        if self.debug.is_true:
            artifact.cxxinfo.cflags.append("-g")
            artifact.cxxinfo.cflags.append("-Og")
        else:
            artifact.cxxinfo.cflags.append("-O2")

Flags can also be exported as environment variables, CFLAGS, CXXFLAGS, etc.

Secondly, declare the toolchain as a dependency of all your compilation tasks:

class HelloWorld(CXXExecutable):
    requires = ["toolchain"]
    sources = ["src/main.cpp"]

Default toolchain parameter values can be overridden from the command line when you need to. For example, to build the HelloWorld task for the ARM architecture, run:

$ jolt build helloworld -d toolchain:arch=arm

The -d toolchain:arch=arm command line argument instructs Jolt to overide the default value of the arch parameter of the toolchain task. The new value changes the identity of the toolchain artifact which triggers a rebuild of all depending tasks.

To build the HelloWorld task without optimizations and with debug information:

$ jolt build helloworld -d toolchain:debug=true

This approach with default valued parameters can also be used to enable other use-cases where you temporarily may want:

code coverage builds

builds with custom cflags

etc

Conan Package Manager¶

The Conan package manager is an excellent way to quickly obtain prebuilt binaries of third-party libraries. It has been integrated into Jolt allowing you to seemlessly use Conan packages with your Jolt Ninja tasks.

In the example below, Conan is used to collect the Boost C++ libraries. Boost is then used in our example application. All build metadata is automatically configured.

from jolt.plugins.conan import Conan

class Boost(Conan):
    requires = ["toolchain"]
    packages = ["boost/1.74.0"]

class HelloWorld(CXXExecutable):
    requires = ["toolchain", "boost"]
    sources = ["src/main.cpp"]

With the toolchain as a dependency also for Boost, Conan will be able to fetch the appropriate binaries that match your toolchain. If no such binaries are available, Conan will build them for you.