Reference

Alias

class jolt.Alias(*args, **kwargs)

An alias task.

Aliases are a special kind of task which can be used to introduce new names for other tasks or groups of tasks. They are useful as milestones when building continuous integration pipelines since they won’t be executed, thus saving time compared to a regular task.

name

Name of the alias. Derived from class name if not set.

requires = []

List of dependencies to other tasks.

Artifact

class jolt.Artifact(cache, node, name=None, identity=None, tools=None, session=False)

An artifact is a collection of files and metadata produced by a task.

Task implementors call artifact methods to collect files to be published. In addition to files, other metadata can be provided as well, such as variables that should be set in the environment of consumer tasks.

collect(files, dest=None, flatten=False, symlinks=False, cwd=None)

Collect files to be included in the artifact.

Parameters:
  • files (str) – A filename pattern matching the files to be included in the artifact. The pattern may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

  • dest (str, optional) – Destination path within the artifact. If the string ends with a path separator a new directory will be created and all matched source files will be copied into the new directory. A destination without trailing path separator can be used to rename single files, one at a time.

  • flatten (boolean, optional) – If True, the directory tree structure of matched source files will flattened, i.e. all files will be copied into the root of the destination path. The default is False, which retains the directory tree structure relative to the current working directory.

  • symlinks (boolean, optional) – If True, symlinks are copied. The default is False, i.e. the symlink target is copied.

  • cwd (str, optional) – Change current working directory before starting collection.

copy(files, dest, flatten=False, symlinks=False, cwd=None)

Copy files from the artifact.

Parameters:
  • files (str) – A filename pattern matching the files to be copied from the artifact. The filepath is relative to the artifact root and may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

  • dest (str, optional) – Destination path, relative to the current working directory. If the string ends with a path separator a new directory will be created and all matched source files will be copied into the new directory. A destination without trailing path separator can be used to rename single files, one at a time.

  • flatten (boolean, optional) – If True, the directory tree structure of matched source files will flattened, i.e. all files will be copied into the root of the destination path. The default is False, which retains the directory tree structure.

  • symlinks (boolean, optional) – If True, symlinks are copied. The default is False, i.e. the symlink target is copied.

  • cwd (str, optional) – Change destination working directory before starting copy.

cxxinfo = {}

Artifact C/C++ build metadata.

A task can add compilation metadata to an artifact. Such metadata will be automatically applied when consumer compilation tasks are executed. A common use-case is to add preprocessor definitions, link libraries, etc. These string fields are supported:

  • asflags - assembler flags (string)

  • cflags - compiler flags (string)

  • cxxflags - compiler flags (string)

  • ldflags - linker flags (string)

  • libraries - libraries to link with (list, use append())

  • macros - preprocessor macros to set (list, use append())

  • sources - source files to compile (list, use append())

Values appended to PATH-type metadata fields are relative to the artifact root. They will be automatically expanded to absolute paths. These PATH-type fields are supported:

  • incpaths - preprocessor include paths (list, use append())

  • libpaths - linker library search paths (list, use append())

Example

def publish(self, artifact, tools):
    artifact.collect("*.h", "include/")
    artifact.cxxinfo.incpaths.append("include")
    artifact.cxxinfo.macros.append("PACKAGE_VERSION=1.0")
environ = {}

Artifact environment variables.

A task can add environment variables to an artifact. Such a variable will automatically be set in the environment when consumer tasks are executed. A common use-case is to add programs to the PATH.

Values appended to PATH variables are relative to the artifact root. They will be automatically expanded to absolute paths. This applies to all variables with PATH in the name.

Example

def publish(self, artifact, tools):
    artifact.environ.PATH.append("bin")
    artifact.environ.JAVA_HOME = artifact.final_path
property final_path

The final location of the artifact in the local cache.

Type:

str

property path

The current location of the artifact in the local cache.

Type:

str

paths = {}

Artifact paths.

A task can add paths to files and directories inside an artifact. Paths are relative to the root of the artifact when created, but are expanded to absolute paths when the artifact is consumed by a task.

This is useful as an abstraction when directories or filenames have varying names.

Example

def publish(self, artifact, tools):
    artifact.paths.file = "{date}.txt"

The file path is then expanded to a full path for consumers:

requires = ["dep"]

def run(self, deps, tools):
    filedata = tools.read_file(deps["dep"].paths.file)
python = {}

Artifact Python configuration.

A task can add Python configuration to an artifact. Such configuration will automatically be set in the environment when consumer tasks are executed. A common use-case is to add Python modules to the PATH so that they can be easily imported by a consumer.

Values appended to PATH-type variables are relative to the artifact root. They will be automatically expanded to absolute paths.

Example

def publish(self, artifact, tools):
    artifact.python.PATH.append("my_module")
strings = {}

Artifact strings.

A task can add arbitrary string values to an artifact. Such a string will be available for consumer tasks to read.

Example

def publish(self, artifact, tools):
    artifact.strings.version = "1.2"

Chroot

class jolt.Chroot(*args, **kwargs)

Resource to use task artifact or directory path as chroot in consumers.

Example

from jolt import Chroot, Task
from jolt.plugins.podman import ContainerImage

class SdkImage(ContainerImage):
    dockerfile = """
    FROM debian:sid-slim
    ARG DEBIAN_FRONTEND=noninteractive
    RUN apt-get update && apt-get install -y --no-install-recommends gcc g++ && apt-get clean
    """
    output = "directory"

class Sdk(Chroot):
    chroot = "sdkimage"

class Compile(Task):
    requires = ["sdk"]

    def run(self, deps, tools):
        tools.run("gcc -v")
chroot = None

Task name or directory path to use as chroot

Context

class jolt.Context(cache, node)

Execution context and dependency wrapper.

A Context gathers dependencies and initializes the environment for an executing task. It is passed as an argument to the Task’s run() method.

A task implementor can use the context as a dictionary of dependencies where the key is the name of a dependency and the value is the dependency’s Artifact.

__getitem__(key)

Get artifact for a task listed as a requirement.

Parameters:

key (str) – Name of the task listed as a requirement.

Returns:

The Artifact associated with the task.

Example

requires = "dependency"

def run(self, deps, tools):
    dependency_artifact = deps["dependency"]
items()

List all requirements and their artifacts.

Returns:

Requirement dictionary items. Each item is a tuple with the requirement name and the artifact.

Decorators

class jolt.attributes
static artifact(name, session=False)

Decorator adding an additional artifact to a task.

Jolt calls the new publish method publish_<name> with the new artifact as argument. Non-alphanumeric characters in the name are replaced with underscores (_). The new artifact is consumable from another task by using the string format <artifact-name>@<task-name> to index dependencies. The standard artifact is named main. See the example below.

If session is True, the new artifact will be a session artifact that is only valid during a single Jolt invokation. Session artifacts are published even if the task fails and may be used to save logs and data for post-mortem analysis.

Parameters:
  • name (str) – Name of artifact. Used as reference from consuming tasks.

  • session (boolean, False) – Session artifact.

Example

examples/artifacts/build.jolt
from jolt import Task, attributes


@attributes.artifact("logs", session=True)
class Producer(Task):
    def run(self, deps, tools):
        with tools.cwd(tools.builddir()):
            tools.write_file("file.txt", "a text file")
            tools.write_file("file.log", "a log file")

    def publish(self, artifact, tools):
        # Publish text files into main artifact
        with tools.cwd(tools.builddir()):
            artifact.collect("*.txt")

    def publish_logs(self, artifact, tools):
        # Publish logs into session artifact.
        # The artifact is created and shared
        # even if the task fails.
        with tools.cwd(tools.builddir()):
            artifact.collect("*.log")


class Consumer(Task):
    requires = ["producer"]

    def run(self, deps, tools):
        # Display file in main artifact
        with tools.cwd(deps["main@producer"].path):
            tools.run("cat *.txt")

        # Display logs
        try:
            with tools.cwd(deps["logs@producer"].path):
                tools.run("cat *.log")
        except KeyError:
            # KeyError is raised if the producer task
            # didn't have to be executed due to caching.
            # In that case, no session artifact exists.
            print("No logs available")
static attribute(alias, target, influence=True, default=False)

Decorates a task with an alias for another attribute.

Parameters:
  • attrib (str) – Name of alias attribute.

  • target (str) – Name of target attribute. Keywords are expanded.

  • influence (boolean) – Add value of target attribute as influence of the task.

  • default (boolean) – Return alias attribute if target attribute does not exist. Value is accessed through the alias attribute name with a leading underscore, e.g. ‘_alias’.

static environ(envname, influence=True)

Decorator marking the task as dependent on an environment variable.

The value of the environment variable will be automatically transferred to workers in distributed network builds.

Parameters:
  • envname (str) – Name of the environment variable.

  • influence (boolean) – Add value of environment variable as influence of the task. Default: True.

static load(filepath)

Decorator which loads task class attributes from a file.

The loaded file is Python source file declaring a dictionary with keys and values to be assigned to the task instance.

The file is automatically registered as task hash influence.

Example:

@attributes.load("attributes-{os}.py")
class Print(Task):
    os = Parameter()

    def run(self, deps, tools):
        print("OS Author: ", self.os_author)
# attributes-linux.py
{
    "os_author": "Torvalds",
}
$ jolt build print:os=linux
static requires(attrib)

Decorates a task with an alternative requires attribute.

The new attribute will be concatenated with the regular requires attribute.

Parameters:

attrib (str) – Name of alternative attribute. Keywords are expanded.

static system(cls)

Decorates a task with a property returning the operating system name.

Examples: “linux”, “windows”

Download

class jolt.tasks.Download(parameters=None, **kwargs)

Downloads file(s) over HTTP(S).

Once downloaded, archives are extracted and all of their files are published. If the file is not an archive it is published as is. Recognized archive extensions are:

  • .tar

  • .tar.bz2

  • .tar.gz

  • .tar.xz

  • .tgz

  • .zip

Example

class NodeJS(Download):
    """ Downloads and publishes Node.js. Adds binaries to PATH. """

    version = Parameter("14.16.1")
    url = "https://nodejs.org/dist/v{version}/node-v{version}-win-x64.zip"

    def publish(self, artifact, tools):
        super(publish).publish(artifact, tools)
        artifact.environ.PATH.append("node-v{version}-win-x64")
collect = ['*']

A list of file publication instructions.

Items in the list are passed directly to Artifact.collect() and can be either strings, tuples or dictionaries.

Example

collect = [
    "*",                           # Collect all files
    ("*", "src/"),                 # Collect all files into the artifact's src/ directory
    {"files": "*", cwd="subdir"},  # Collect all files from the archive's subdir/ directory
]
extract = True

Automatically extract archives.

url = None

URL(s) of file(s) to download.

A single URL string is accepted, as well as a list of URL strings.

Influence

jolt.influence.always(cls)

Always execute the task.

Example

from jolt import influence

@influence.always
class Example(Task):
jolt.influence.attribute(name, type=None, sort=False)

Add task attribute value as hash influence.

Parameters:
  • name (str) – Name of task class attribute/property.

  • type (InfluenceProvider, optional) – Alternative HashInfluenceProvider implementation used to interpret the value of the attribute. If the attribute is a list, the provider will be instantiated once for each item in the list. For example, if the attribute is a list of filesystem paths, the FileInfluence class could be passed as argument to automatically monitor files at those paths.

  • sort (boolean, optional) – Optionally sort the value. Always sort values that are unstable, such as dictionaries.

Example:

from jolt import influence

@influence.attribute("attribute")
class Example(Task):
    attribute = False
from jolt import influence

@influence.attribute("sources", type=influence.FileInfluence)
class Example(Task):
    sources = [
        "main.cpp",
        "utils.cpp",
    ]
jolt.influence.daily(cls)

Add daily hash influence.

If nothing else changes, the task is re-executed once every day.

Example

from jolt import influence

@influence.daily
class Example(Task):
jolt.influence.environ(variable)

Add environment variable hash influence.

Parameters:

variable (str) – Name of an environment variable that will influence the hash of the task.

Example:

from jolt import influence

@influence.environ("CFLAGS")
class Example(Task):
jolt.influence.files(pathname)

Add file content hash influence.

Parameters:

pathname (str) – A pathname pattern used to find files that will influence the hash of the task The pattern may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

Example:

from jolt import influence

@influence.files("*.cpp")
class Example(Task):
jolt.influence.hourly(cls)

Add hourly hash influence.

If nothing else changes, the task is re-executed once every hour.

Example

from jolt import influence

@influence.hourly
class Example(Task):
jolt.influence.monthly(cls)

Add monthly hash influence.

If nothing else changes, the task is re-executed once every month.

Example

from jolt import influence

@influence.monthly
class Example(Task):
jolt.influence.weekly(cls)

Add weekly hash influence.

If nothing else changes, the task is re-executed once every week.

Example

from jolt import influence

@influence.weekly
class Example(Task):
jolt.influence.yearly(cls)

Add yearly hash influence.

If nothing else changes, the task is re-executed once every year.

Example

from jolt import influence

@influence.yearly
class Example(Task):

Parameter

class jolt.Parameter(default=None, values=None, required=True, const=False, influence=True, help=None)

Generic task parameter type.

__init__(default=None, values=None, required=True, const=False, influence=True, help=None)

Creates a new parameter.

Parameters:
  • default (str, optional) – An optional default value.

  • values (list, optional) – A list of accepted values. An assertion is raised if an unlisted value is assigned to the parameter.

  • required (boolean, optional) – If required, the parameter must be assigned a value before the task can be executed. The default is True.

  • const (boolean, optional) – If const is True, the parameter is immutable and cannot be assigned a non-default value. This is useful in a class hierarchy where a subclass may want to impose restrictions on a parent class parameter. The default is False.

  • influence (boolean, optional) – If influence is False, the parameter value will not influence the identity of the task artifact. The default is True.

  • help (str, optional) – Documentation for the parameter. This text is displayed when running the info command on the associated task.

Raises:

ParameterValueError – If the parameter is assigned an illegal value.

__str__()

Returns the parameter value as a string

get_default()

Get the default value of the parameter.

Returns:

The default value or None if no default was given.

get_value()

Get the parameter value.

is_default()

Check if parameter is set to its default value.

Returns:

True if the assigned value is the default value.

is_set()

Check if the parameter is set to a non-default value.

Returns:

True if the assigned value is not the default value.

is_unset()

Check if the parameter is unset.

Returns:

True if the assigned value is None.

set_value(value)

Set the parameter value.

Parameters:

value (str) – The new parameter value.

Raises:

ParameterValueError – If the parameter is assigned an illegal value.

class jolt.BooleanParameter(default=None, required=True, const=False, influence=True, help=None)

Boolean task parameter.

Accepted values are:

  • False

  • True

  • “false”

  • “true”

  • “no”

  • “yes”

  • “0”

  • “1”

__bool__()

Returns the boolean parameter value

__getitem__(key)

Returns a substitution string depending on the parameter value.

Parameters:
  • key (str) – A special key syntax, enabled,disabled, where

  • returned (and either enabled or disabled would be) –

  • below. (depending on the paramter value. See the example) –

Returns:

Substitution string.

Example

class Example(Task):
   debug = BooleanParameter()

   def run(self, deps, tools):
       self.info("debug is {debug[enabled,disabled]}")
$ jolt build example:debug=true
[INFO] debug is enabled (example)

$ jolt build example:debug=false
[INFO] debug is disabled (example)
__init__(default=None, required=True, const=False, influence=True, help=None)

Creates a new parameter.

Parameters:
  • default (boolean, optional) – An optional default boolean value.

  • required (boolean, optional) – If required, the parameter must be assigned a value before the task can be executed. The default is True.

  • const (boolean, optional) – If const is True, the parameter is immutable and cannot be assigned a non-default value. This is useful in a class hierarchy where a subclass may want to impose restrictions on a parent class parameter. The default is False.

  • influence (boolean, optional) – If influence is False, the parameter value will not influence the identity of the task artifact. The default is True.

  • help (str, optional) – Documentation for the parameter. This text is displayed when running the inspect command on the associated task.

Raises:

ParameterValueError – If the parameter is assigned an illegal value.

__str__()

Returns the parameter value as a string

get_default()

Get the default value of the parameter.

Returns:

The default value or None if no default was given.

get_value()

Get the parameter value.

is_default()

Check if parameter is set to its default value.

Returns:

True if the assigned value is the default value.

property is_false

The parameter value is False.

is_set()

Check if the parameter is set to a non-default value.

Returns:

True if the assigned value is not the default value.

property is_true

The parameter value is True.

is_unset()

Check if the parameter is unset.

Returns:

True if the assigned value is None.

class jolt.IntParameter(default=None, min=None, max=None, values=None, required=True, const=False, influence=True, help=None)

Integer task parameter.

Implements all regular unary and binary integer operators.

__bool__()

Evaluates to False if the value is 0, True otherwise

__init__(default=None, min=None, max=None, values=None, required=True, const=False, influence=True, help=None)

Creates a new parameter.

Parameters:
  • default (int, optional) – An optional default integer value.

  • min (int, optional) – Minimum allowed value.

  • max (int, optional) – Maximum allowed value.

  • values (list, optional) – A list of accepted values. An assertion is raised if an unlisted value is assigned to the parameter.

  • required (boolean, optional) – If required, the parameter must be assigned a value before the task can be executed. The default is True.

  • const (boolean, optional) – If const is True, the parameter is immutable and cannot be assigned a non-default value. This is useful in a class hierarchy where a subclass may want to impose restrictions on a parent class parameter. The default is False.

  • influence (boolean, optional) – If influence is False, the parameter value will not influence the identity of the task artifact. The default is True.

  • help (str, optional) – Documentation for the parameter. This text is displayed when running the inspect command on the associated task.

Raises:

ParameterValueError – If the parameter is assigned an illegal value.

__int__()

Returns the integer parameter value

__str__()

Returns the parameter value as a string

get_default()

Get the default value of the parameter.

Returns:

The default value or None if no default was given.

get_value()

Get the parameter value.

is_default()

Check if parameter is set to its default value.

Returns:

True if the assigned value is the default value.

is_set()

Check if the parameter is set to a non-default value.

Returns:

True if the assigned value is not the default value.

is_unset()

Check if the parameter is unset.

Returns:

True if the assigned value is None.

class jolt.ListParameter(*args, **kwargs)

List parameter type.

A list parameter allows multiple values to be assigned to it. Values are separated by the ‘+’ character in qualified task names. Each assigned value is validated against the list of accepted values. They are sorted in alphabetical order before the task is executed.

Example

class Example(Task):
    arg = ListParameter(default=["c"], values=["a", "b", "c"], help="A list parameter")

    def run(self, deps, tools):
        for item in self.arg:
            print(item)
$ jolt build example example:arg=a example:arg=a+b
__getitem__(key)

Returns an element or a slice from the list.

Parameters:

key (int, slice, str) –

Element index or slice. A key string can be used to check for the existence of that value in the list. If the key is present the same value is returned, otherwise None.

A special key syntax is also available to request an alternate return value depending on the presence of the key. Instead of a list value you pass value,present,absent and either present or absent will be returned. See the example below.

Returns:

Element value, or substitution.

Example

class Example(Task):
   features = ListParameter(values=["optimize", "strip"], required=False)

   def run(self, deps, tools):
       if len(self.features) > 0:
           self.info("first feature is {features[0]}")
           self.info("optimize == {features[optimize]}")
           self.info("optimization is {features[optimize,enabled,disabled]}")
$ jolt build example:features=optimize+strip
[INFO] first feature is optimize (example)
[INFO] optimize = optimize (example)
[INFO] optimization is enabled (example)

$ jolt build example:features=strip
[INFO] first feature is debug (example)
[INFO] optimize = None (example)
[INFO] optimization is disabled (example)
__init__(*args, **kwargs)

Creates a new list parameter.

Parameters:
  • default (boolean, optional) – An optional list of default values.

  • values (list, optional) – A list of accepted values. An assertion is raised if an unlisted value is assigned to the parameter.

  • required (boolean, optional) – If required, the parameter must be assigned a value before the task can be executed. The default is True.

  • const (boolean, optional) – If const is True, the parameter is immutable and cannot be assigned a non-default value. This is useful in a class hierarchy where a subclass may want to impose restrictions on a parent class parameter. The default is False.

  • influence (boolean, optional) – If influence is False, the parameter value will not influence the identity of the task artifact. The default is True.

  • help (str, optional) – Documentation for the parameter. This text is displayed when running the inspect command on the associated task.

Raises:

ParameterValueError – If the parameter is assigned an illegal value.

__iter__()

Returns a sequence iterator.

__len__()

Returns the length of the list.

__str__()

Returns the parameter value as a string

get_default()

Get the default value of the parameter.

Returns:

The default value or None if no default was given.

get_value()

Get the parameter value.

is_default()

Check if parameter is set to its default value.

Returns:

True if the assigned value is the default value.

is_set()

Check if the parameter is set to a non-default value.

Returns:

True if the assigned value is not the default value.

is_unset()

Check if the parameter is unset.

Returns:

True if the assigned value is None.

Resource

class jolt.tasks.Resource(*args, **kwargs)

Bases: Task

A resource task.

Resources are special tasks executed in the Context of other tasks. They are invoked to acquire and release a resource, such as hardware equipment, before and after the execution of a task. No artifact is produced by a resource.

Implementors should override acquire() and release().

acquire(artifact, deps, tools, owner)

Called to acquire the resource.

An implementor overrides this method in a subclass. The acquired resource must be released manually if an exception occurs before the method has returned. If this method returns successfully, the release() method is guaranteed to be called in the future upon completion of the consuming task (unless the process is forcibly interrupted or killed).

Parameters:
  • artifact (Artifact) – The artifact associated with the resource. It is not possible to publish files from a resource, but the implementor can still use the resource to pass information to consuming tasks.

  • deps (Context) – Task execution context used to access the artifacts of dependencies.

  • tools (Tools) – A task specific toolbox.

  • owner (Task) – The owner task for which the resource is acquired.

release(artifact, deps, tools, owner)

Called to release the resource.

An implementor overrides this method in a subclass.

Parameters:
  • artifact (Artifact) – The artifact associated with the resource. It is not possible to publish files from a resource, but the implementor can still use the resource to pass information to consuming tasks.

  • deps (Context) – Task execution context used to access the artifacts of dependencies.

  • tools (Tools) – A task specific toolbox.

  • owner (Task) – The owner task for which the resource is released.

Runner

class jolt.tasks.Runner(parameters=None, **kwargs)

Bases: Task

A Runner task executes applications packaged by other tasks.

It is typically used to run test applications compiled and linked by other tasks. The Runner finds the executable through the artifact metadata string artifact.strings.executable which must be exported by the consumed task artifact.

Example

from jolt import Runner, Task

class Exe(Task):
    """ Publish a script printing 'Hello world' to stdout """
    def publish(self, artifact, tools):
        with tools.cwd(tools.builddir()):
            # Create Hello world script
            tools.write_file("hello.sh", "#!/bin/sh\necho Hello world")

            # Make it executable
            tools.chmod("hello.sh", 0o555)

            # Publish script in artifact
            artifact.collect("hello.sh")

            # Inform consuming Runner task about executable's name
            artifact.strings.executable = "hello.sh"

class Run(Runner):
    """ Runs the 'Hello world' script """
    requires = ["exe"]

The Ninja CXXExecutable task class automatically sets the required artifact metadata.

Example

from jolt import Task
from jolt.plugins.ninja import CXXExecutable

class Exe(CXXExecutable):
    """ Compiles and links the test application """
    sources = ["test.cpp"]

class Run(Runner):
    """ Runs the test application """
    requires = ["exe"]
args = []

List of arguments to pass to executables.

The arguments are passed the same way to all executables if there are multiple task requirements.

requires = []

List of tasks packaging executables to run.

shell = True

Launch the executables through a shell.

Script

class jolt.tasks.Script(parameters=None, **kwargs)

Bases: Task

A simple shell script task.

The script source is extracted directly from the task class documentation. All text following a --- separator will be executed.

A temporary build directory is automatically created and can be accessed with {builddir}. All other task attributes are also expanded as usual. Dependency artifacts are accessible through the deps dictionary.

echo {deps[task].path}

By default, all files in the build directory are published in the task artifact. The collect attribute can be used to customize file collection. Alternatively, the publish method may be overridden.

Keep in mind that shell scripts are not necessarily portable between host operating systems. Implement your tasks in Python code if portability is a concern.

Example

class Hello(Script):
    """
    Classic Hello World!
    ---
    # Script source

    echo "Hello world!" > {builddir}/hello.txt
    """
collect = [{'files': '*', 'cwd': '{builddir}'}]

A list of file publication instructions.

Items in the list are passed directly to Artifact.collect() and can be either strings, tuples or dictionaries.

By default, all files in the build directory are published.

Example

collect = [
    "*",                              # Collect all files
    ("*", "src/"),                    # Collect all files into the artifact's src/ directory
    {"files": "*", "cwd": "subdir"},  # Collect all files from the archive's subdir/ directory
]

Task

class jolt.tasks.Task(parameters=None, **kwargs)
abstract = True

An abstract task class indended to be subclassed.

Abstract tasks can’t be executed and won’t be listed.

cacheable = True

Whether the task produces an artifact or not.

clean(tools)

Cleans up resources and intermediate files created by the task.

The method is invoked in response to the user running clean on the command line. It should restore the environment to its original state. The next execution of the task should behave as if the task is executed for the first time.

An implementation must not clean any local or remote artifact cache.

debugshell(deps, tools)

Invoked to start a debug shell.

The method prepares the environment with attributes exported by task requirement artifacts. The shell is entered by passing the -g flag to the build command.

Task execution resumes normally when exiting the shell.

error(fmt, *args, **kwargs)

Log an error concerning the task

expand(string_or_list, *args, **kwargs)

Expands keyword arguments/macros in a format string.

See jolt.Tools.expand() for details.

expires = <jolt.expires.Immediately object>

An expiration strategy, defining when the artifact may be evicted from the cache.

When the size of the artifact cache exceeds the configured limit an attempt will be made to evict artifacts from the cache. The eviction algorithm processes artifacts in least recently used (LRU) order until an expired artifact is found.

By default, an artifact expires immediately and may be evicted at any time (in LRU order). An exception to this rule is if the artifact is required by a task in the active task set. For example, if a task A requires the output of task B, B will never be evicted by A while A is being executed.

There are several expiration strategies to choose from:

  • jolt.expires.WhenUnusedFor

  • jolt.expires.After

  • jolt.expires.Never

Examples

# May be evicted if it hasn't been used for 15 days
expires = WhenUnusedFor(days=15)
# May be evicted 1h after creation
expires = After(hours=1)
# Never evicted
expires = Never()
extends = ''

Name of extended task.

A task with this attribute set is called an extension. An extension is executed in the context of the extended task, immediately after the extended task has executed.

A common use-case for extensions is to produce additional artifacts from the output of another task. Also, for long-running tasks, it is sometimes beneficial to utilize the intermediate output from an extended task. The extension artifact can then be acquired more cheaply than if the extension had performed all of the work from scratch.

An extension can only extend one other task.

fast = False

Indication of task speed.

The information is used by the distributed execution strategy to optimize how tasks are scheduled. Scheduling tasks remotely is always associated with some overhead and sometimes it’s beneficial to instead schedule fast tasks locally if possible.

An extended task is only considered fast if all extensions are fast.

influence = []

List of influence provider objects

info(fmt, *args, **kwargs)

Log information about the task.

joltdir = '.'

Path to the directory of the .jolt file where the task was defined.

joltproject = None

Name of project this task belongs to.

name = None

Name of the task. Derived from class name if not set.

platform = {}

Dictionary of task platform requirements.

Platform requirements control where tasks are allowed to execute. Multiple requirement key/values may be specified in which case all must be fulfilled in order for the task to be schedulable on a node.

The following builtin requirement labels exist:

  • node.arch [“arm”, “amd64”]

  • node.os [“linux”, “windows”]

Example

class Hello(Task):
    # This task must run on Linux.
    platform = {
        "node.os": "linux",
    }
publish(artifact, tools)

Publishes files produced by run().

Files can be collected in to the artifact by calling artifact.collect().

Additional metadata can be provided, such as environment variables that should be set whenever the task artifact is consumed. Example:

# Append <artifact-path>/bin to the PATH
artifact.environ.PATH.append("bin")

# Pass an arbitrary string to a consumer
artifact.strings.foo = "bar"
report()

Provide error analysis for task.

Intentionally undocumented. Use at own risk.

requires = []

List of dependencies to other tasks.

run(deps, tools)

Performs the work of the task.

Dependencies specified with “requires” are passed as the deps dictionary. The tools argument provides a set of low level tool functions that may be useful.

with tools.cwd("path/to/subdir"):
    tools.run("make {target}")

When using methods from the toolbox, task parameters, such as target above, are automatically expanded to their values.

selfsustained = False

Consume this task independently of its requirements.

Requirements of a self-sustained task will be pruned if the task artifact is present in a cache. In other words, if the task is not executed its requirements are considered unnecessary.

For example, consider the task graph A -> B -> C. If B is self-sustained and present in a cache, C will never be executed. C will also never be a transitive requirement of A. If A requires C, it should be listed as an explicit requirement.

Using this attribute speeds up execution and reduces network traffic by allowing the task graph to be reduced.

taint = None

An arbitrary value used to change the identity of the task.

It may be hard to remove bad artifacts in a distributed build environment. A better method is to taint the task and let the artifact be recreated with a different identity.

unpack(artifact, tools)

Unpacks files published by publish() .

The intention of this hook is to make necessary adjustments to artifact files and directories once they have been downloaded into the local cache on a different machine. For example, paths may have to be adjusted or an installer executed.

This hook is executed in the context of a consuming task.

unstable = False

An unstable task is allowed to fail without stopping or failing the entire build.

The unstable task is still reported as a failure at the end of the build.

verbose(fmt, *args, **kwargs)

Log verbose information about the task.

warning(fmt, *args, **kwargs)

Log a warning concerning the task

weight = 0

Indication of task execution time.

The weight is used to optimize the order in which tasks are executed using a heuristic scheduling algorithm where ready tasks along the critical path are favored.

MultiTask

class jolt.tasks.MultiTask(*args, **kwargs)

A task with subtasks that are executed in parallel with intermediate caching.

A MultiTask is useful for tasks with many subtasks that benefit from intermediate caching, such as compilation tasks where multiple source files are compiled into object files and then either linked into an executable or archived into a library.

Subtasks are executed in parallel and their output is cached locally in a build directory. The output is not automatically shared with other Jolt clients, only the files published by the MultiTask is shared. A subtask is only re-executed if the influence one of its dependencies change.

Subtasks are defined in the MultiTask method generate() and they can be either a shell command or a python function. Helper methods in the class allow implementors to define outputs and inter-subtask dependencies.

Example

flags = ["-DDEBUG"]

def generate(self, deps, tools):
    sources = ["a.cpp", "b.cpp", "c.cpp"]
    objects = []

    # Create compilation subtasks for each source file
    for source in sources:
        object = self.command(
            "g++ {flags} -c {inputs} -o {outputs} ",
            inputs=[source],
            outputs=[source  +".o"])
        objects.append(object)

    # Create linker subtask
    executable = self.command(
        "g++ {inputs} -o {output}",
        inputs=objects,
        outputs=["executable"])
call(fn, outputs, **kwargs)

Create a Python function call subtask.

The subtask executes the specified Python function, passing the subtask as argument.

Parameters:
  • fn (func) – Python function to execute.

  • outputs (str, list) – list of files that the subtasks produces.

  • kwargs – additional keyword values used to format the output file paths.

Returns:

Subtask object.

Example

def mkdir(subtask):
    for output in subtask.outputs:
        self.tools.mkdir(output)

dirtask = self.call(mkdir, outputs=["newly/created/directory"])
command(command, inputs=None, outputs=None, message=None, mkdir=True, **kwargs)

Create shell command subtask.

The subtask executes the specified command. String format specifiers may be used and are resolved primarily by kwargs and secondarily by task attributes.

Parameters:
  • inputs (str, list) – files or subtasks that the command depends on.

  • outputs (str, list) – list of files that the subtasks produces.

  • message (str) – custom message that the subtask will print when executed.

  • mkdir (boolean) – automatically create directories for outputs. If False, the caller must ensure that the directories exist before the the subtask is executed.

  • kwargs – additional keyword values used to format the command line string.

Returns:

Subtask object.

Example

executable = self.command(
    "g++ {inputs} -o {output}",
    inputs=["main.cpp"],
    outputs=["executable"])
generate(deps, tools)

Called to generate subtasks.

An implementer can override this method in order to create subtasks that will later be executed during the run() stage of the task.

Subtasks can be defined using either of these helper methods:

mkdir(path, *args, **kwargs)

Create a subtask that creates a directory.

Parameters:
  • path (str) – Path to directory.

  • kwargs – additional keyword values used to format the directory path string.

Returns:

Subtask object.

Example

dirtask = self.mkdir("{outdir}/directory", outdir=tools.builddir())
mkdirname(path, *args, **kwargs)

Create a subtask that creates a parent directory.

Parameters:
  • path (str) – Path for which the parent directory shall be created.

  • kwargs – additional keyword values used to format the directory path string.

Returns:

Subtask object.

Example

# Creates {outdir}/directory
dirtask = self.mkdir("{outdir}/directory/object.o", outdir=tools.builddir())
render(template, outputs, **kwargs)

Create a subtask that renders a Jinja template string to file.

Parameters:
  • template (str) – Jinja template string.

  • outputs (str, list) – list of files that the subtasks produces.

  • kwargs – additional keyword values used to render the template and output file paths.

Returns:

Subtask object.

Example

# Creates file.list with two lines containing "a.o" and "b.o"

template_task = self.render(
    "{% for line in lines %}{{ line }}\n{% endfor %}",
    outputs=["file.list"],
    lines=["a.o", "b.o"])
render_file(template, outputs, **kwargs)

Create a subtask that renders a Jinja template file to file.

Parameters:
  • template (str) – Jinja template file path.

  • outputs (str, list) – list of files that the subtasks produces.

  • kwargs – additional keyword values used to format the output file paths.

Returns:

Subtask object.

Example

# Render file.list.template into file.list
template_task = self.render_file("file.list.template", outputs=["file.list"])

Test

class jolt.tasks.Test(*args, **kwargs)

A test task.

The test task combines a regular Jolt task with a Python unittest.TestCase. As such, a test task is a collection of similar test-cases where each test-case is implemented as an instancemethod named with a test_ prefix. When executed, the task runs all test-case methods and summarizes the result.

All regular unittest assertions and decorators can be used in the test methods. For details about inherited task attributes, see jolt.tasks.Task and Python unittest.TestCase.

Example:

class OperatorTest(Test):

    def test_add(self):
        self.assertEqual(1+1, 2)

    def test_sub(self):
        self.assertEqual(2-1, 1)
cleanup()

Implement this method to clean up after a test

static parameterized(args)

Parameterizes a test method.

The test method is instantiated and called once with each argument tuple in the list.

Example

class Example(Test):
   @Test.parameterized([
       (1, 1, 1),  # 1*1 == 1
       (1, 2, 2),  # 1*2 == 2
       (2, 1, 2),  # 2*1 == 2
       (2, 2, 4),  # 2*2 == 4
   ])
   def test_multiply(self, factor1, factor2, product):
       self.assertEqual(factor1*factor2, product)
setup(deps, tools)

Implement this method to make preparations before a test

Tools

class jolt.Tools(task=None, cwd=None, env=None)

A collection of useful tools.

Any {keyword} arguments, or macros, found in strings passed to tool functions are automatically expanded to the value of the associated task’s parameters and properties. Relative paths are made absolute by prepending the current working directory.

append_file(pathname, content, expand=True)

Appends data at the end of a file.

Parameters:
  • pathname (str) – Path to file. The file must exist.

  • content (str) – Data to be appended at the end of the file.

archive(pathname, filename)

Creates a (compressed) archive.

The type of archive to create is determined by the filename extension. Supported formats are:

  • tar

  • tar.bz2

  • tar.gz

  • tar.xz

  • zip

Parameters:
  • pathname (str) – Directory path of files to be archived.

  • filename (str) – Name/path of created archive.

autotools(deps=None)

Creates an AutoTools invokation helper

builddir(name=None, incremental=False, unique=True)

Creates a temporary build directory.

The build directory will persist for the duration of a task’s execution. It is automatically removed afterwards.

Parameters:
  • name (str) – Name prefix for the directory. A unique autogenerated suffix will also be appended to the final name.

  • incremental (boolean) – If false, the created directory is deleted upon completion of the task.

Returns:

Path to the created directory.

property buildroot

Return the root path of all build directories

checksum_file(filelist, concat=False, hashfn=<built-in function openssl_sha1>, filterfn=None)

Calculate a checksum of one or multiple files.

Parameters:
  • filelist (str,list) – One or multiple files.

  • concat (boolean) – Concatenate files and return a single digest. If False, a list with one digest for each file is returned. Default: False.

  • hashfn – The hash algorithm used. Any type which provides an update() and hexdigest() method is accepted. Default: hashlib.sha1

  • filterfn – An optional data filter function. It is called repeatedly with each block of data read from files as its only argument. It should return the data to be included in the checksum. Default: None

Returns:

A list of checksum digests, or a single digest if files where concatenated.

chmod(pathname, mode)

Changes permissions of files and directories.

Parameters:
  • pathname (str) – Path to a file or directory to change permissions for.

  • mode (int) – Requested permission bits.

chroot(chroot, *args, **kwargs)

Experimental: Use chroot as root filesystem when running commands.

Mounts the specified chroot as the root filesystem in a new Linux namespace, which is used when calling Tools.run().

Requires a Linux host.

Parameters:

chroot (str, artifact) – Path to rootfs directory, or an artifact with a ‘rootfs’ metadata path (artifact.paths.rootfs).

Example

with tools.choot("path/to/rootfs"):
    tools.run("ls")
cmake(deps=None, incremental=False)

Creates a CMake invokation helper

compress(src, dst)

Compress a file.

Supported formats are:

  • .bz2

  • .gz

  • .xz

Parameters:
  • src (str) – Source file to be compressed.

  • dst (str) – Destination path for compressed file. The filename extension determines the compression algorithm used.

copy(src, dst, symlinks=False)

Copies file and directories (recursively).

The directory tree structure is retained when copying directories.

Parameters:
  • src (str) – Path to a file or directory to be copied.

  • dest (str) – Destination path. If the string ends with a path separator a new directory will be created and source files/directories will be copied into the new directory. A destination without trailing path separator can be used to rename single files, one at a time.

  • symlinks (boolean, optional) – If True, symlinks are copied. The default is False, i.e. the symlink target is copied.

cpu_count()

The number of CPUs on the host.

Returns:

The number of CPUs on the host.

Return type:

int

cwd(pathname, *args)

Change the current working directory to the specified path.

This function doesn’t change the working directory of the Jolt process. It only changes the working directory for tools within the tools object.

Parameters:

pathname (str) – Path to change to.

Example

with tools.cwd("subdir") as cwd:
    print(cwd)
download(url, pathname, exceptions=True, **kwargs)

Downloads a file using HTTP.

Throws a JoltError exception on failure.

Parameters:
  • url (str) – URL to the file to be downloaded.

  • pathname (str) – Name/path of destination file.

  • kwargs (optional) – Addidional keyword arguments passed on directly requests.get().

environ(**kwargs)

Set/get environment variables.

Only child processes spawned by the same tools object will be affected by the changed environment.

The changed environment is only valid within a context and it is restored immediately upon leaving the context.

Parameters:

kwargs (optinal) – A list of keyword arguments assigning values to environment variable.

Example

with tools.environ(CC="clang"):
    tools.run("make all")
expand(string, *args, **kwargs)

Expands keyword arguments/macros in a format string.

This function is identical to str.format() but it automatically collects keyword arguments from a task’s parameters and properties.

It also supports three additional conversion specifiers:

  • c: call method without arguments

  • l: convert string to lower case

  • u: convert string to upper case

Parameters:
  • string (str) – The string to be expanded.

  • args (str, optional) – Additional positional values required by the format string.

  • kwargs (str, optional) – Additional keyword values required by the format string.

Returns:

Expanded string.

Return type:

str

Example

target = Parameter(default="all")
verbose = "yes"

def run(self, deps, tools):
    print(tools.expand("make {target} VERBOSE={verbose}"))  # "make all VERBOSE=yes"
expand_path(pathname, *args, **kwargs)

Expands keyword arguments/macros in a pathname format string.

This function is identical to str.format() but it automatically collects keyword arguments from a task’s parameters and properties.

The function also makes relative paths absolute by prepending the current working directory.

Parameters:
  • pathname (str) – The pathname to be expanded.

  • args (str, optional) – Additional positional values required by the format pathname.

  • kwargs (str, optional) – Additional keyword values required by the format pathname.

Return

str: Expanded string.

expand_relpath(pathname, relpath=None, *args, **kwargs)

Expands keyword arguments/macros in a pathname format string.

This function is identical to str.format() but it automatically collects keyword arguments from a task’s parameters and properties.

The function also makes absolute paths relative to a specified directory.

Parameters:
  • pathname (str) – The pathname to be expanded.

  • relpath (str, optional) – Directory to which the returned path will be relative. If not provided, the joltdir attribute is used.

  • args (str, optional) – Additional positional values required by the format pathname.

  • kwargs (str, optional) – Additional keyword values required by the format pathname.

Return

str: Expanded string.

extract(filename, pathname, files=None, ignore_owner=False)

Extracts files in an archive.

Supported formats are:

  • tar

  • tar.bz2

  • tar.gz

  • tar.xz

  • zip

Parameters:
  • filename (str) – Name/path of archive file to be extracted.

  • pathname (str) – Destination path for extracted files.

  • files (list, optional) – List of files the be extracted from the archive. If not provided, all files are extracted.

file_size(pathname)

Determines the size of a file.

Parameters:

pathname (str) – Name/path of file for which the size is requested.

Returns:

The size of the file in bytes.

Return type:

int

getcwd()

Returns the current working directory.

getenv(key, default='')

Returns the value of an environment variable.

Only child processes spawned by the same tools object can see the environment variables and their values returned by this method. Don’t assume the same variables are set in the Jolt process’ environment.

glob(pathname, expand=False)

Enumerates files and directories.

Parameters:
  • pathname (str) – A pathname pattern used to match files to be included in the returned list of files and directories. The pattern may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

  • expand (boolean) – Expand matches to absolute paths. Default: false.

Returns:

A list of file and directory pathnames. The pathnames are relative to the current working directory unless the pathname argument was absolute.

Example

textfiles = tools.glob("*.txt")
map_concurrent(callable, iterable, max_workers=None)

Concurrent ~map().

Parameters:
  • callable – A callable object to be executed for each item in the collection.

  • iterable – An iterable collection of items.

  • max_workers (optional) – The maximum number of worker threads allowed to be spawned when performing the work. The default is the value returned by jolt.Tools.cpu_count().

Returns:

List of return values.

Return type:

list

Example

def compile(self, srcfile):
    objfile = srcfile + ".o"
    tools.run("gcc -c {0} -o {1}", srcfile, objfile)
    return objfile

srcfiles = ["test.c", "main.c"]
objfiles = tools.map_concurrent(compile, srcfiles)
map_consecutive(callable, iterable)

Same as map().

meson(deps=None)

Creates a Meson invokation helper

mkdir(pathname, recursively=True)

Create directory.

mkdirname(pathname, recursively=True)

Create parent directory.

move(src, dst)

Move/rename file.

Parameters:
  • src (str) – Path to a file or directory to be moved.

  • dest (str) – Destination path. If the destination is an existing directory, then src is moved inside that directory. If the destination already exists but is not a directory, it may be overwritten. If the destination is not on the same filesystem, the source file or directory is copied to the destination and then removed.

nixpkgs(nixfile=None, packages=None, pure=False, path=None, options=None)

Creates a Nix environment with the specified packages.

Parameters:
  • nixfile (str) – Path to a Nix expression file.

  • packages (list) – List of Nix packages to include in environment.

  • pure (boolean) – Create a pure environment.

  • path (list) – List of Nix expression paths.

  • options (dict) – Nix configuration options.

Example

def run(self, deps, tools):
    with tools.nixpkgs(packages=["gcc13"]):
        tools.run("gcc --version")
read_depfile(pathname)

Reads a Make dependency file.

Returns:

Dictionary of files and their dependencies.

Return type:

dict

read_file(pathname, binary=False)

Reads a file.

render(template, **kwargs)

Render a Jinja template string.

Parameters:
  • template (str) – Jinja template string.

  • kwargs (dict) – Keywords made available to the template context. Task attributes are automatically available.

Returns:

Renderered template data.

Return type:

str

render_file(template, **kwargs)

Render a Jinja template file.

Parameters:
  • template (str) – Filesystem path to template file.

  • kwargs (dict) – Keywords made available to the template context. Task attributes are automatically available.

Returns:

Renderered template data.

Return type:

str

replace_in_file(pathname, search, replace, regex=False)

Replaces all occurrences of a substring in a file.

Parameters:
  • pathname (str) – Name/path of file to modify.

  • search (str) – Substring to be replaced.

  • replace (str) – Replacement substring.

  • regex (boolean) – Interpret search parameter as a regular expression matching the string to be replaced.

Example

version = Parameter(default="1.0")

def run(self, deps, tools):
    tools.replace_in_file("Makefile", "VERSION := 0.9", "VERSION := {version}")
rmtree(pathname, *args, **kwargs)

Removes a directory tree from disk.

Parameters:
  • pathname (str) – Path to the file or directory to be removed.

  • ignore_errors (boolean, optional) – Ignore files that can’t be deleted. The default is False.

rsync(srcpath, dstpath, *args, **kwargs)

Synchronizes files from one directory to another.

The function performs a smart copy of files from the srcpath directory to the dstpath directory in such a way that dstpath will mirror srcpath.

If dstpath is empty, the files are copied normally.

If dstpath already contains a sub or superset of the files in srcpath, files are either copied or deleted depending on their presence in the source directory. Common files are only copied if the file content differs, thereby retaining metadata (such as timestamps) of identical files already present in dstpath.

Parameters:
  • srcpath (str) – Path to source directory. The directory must exist.

  • dstpath (str) – Path to destination directory.

run(cmd, *args, **kwargs)

Runs a command in a shell interpreter.

These additional environment variables will be set when the command is run:

  • JOLTDIR - Set to Task.joltdir

  • JOLTCACHEDIR - Set to the location of the Jolt cache

A JoltCommandError exception is raised on failure.

Parameters:
  • cmd (str) – Command format string.

  • args – Positional arguments for the command format string.

  • kwargs – Keyword arguments for the command format string.

  • output (boolean, optional) – By default, the executed command’s output will be written to the console. Set to False to disable all output.

  • output_on_error (boolean, optional) – If True, no output is written to the console unless the command fails. The default is False.

  • output_rstrip (boolean, optional) – By default, output written to stdout is stripped from whitespace at the end of the string. This can be disabled by setting this argument to False.

  • shell (boolean, optional) – Use a shell to run the command. Default: True.

  • timeout (int, optional) – Timeout in seconds. The command will first be terminated if the timeout expires. If the command refuses to terminate, it will be killed after an additional 10 seconds have passed. Default: None.

Example

target = Parameter(default="all")
verbose = "yes"

def run(self, deps, tools):
    tools.run("make {target} VERBOSE={verbose} JOBS={0}", tools.cpu_count())
runprefix(cmdprefix, *args, **kwargs)

Adds a command prefix to all commands executed by run().

A new prefix is appended to any existing prefix.

Parameters:
  • cmdprefix (str, list) – The command prefix. The string, or list, is expanded with expand().

  • args (str, optional) – Additional positional values passed to expand().

  • kwargs (str, optional) – Additional keyword values passed to expand().

Example

with tools.runprefix("docker exec container"):
    tools.run("ls")

The above code is equivalent to:

tools.run("docker exec container ls")
sandbox(artifact, incremental=False, reflect=False)

Creates a temporary build directory populated with the contents of an artifact.

Files are copied using rsync.

Parameters:
  • artifact (cache.Artifact) – A task artifact to be copied into the sandbox.

  • incremental (boolean) – If false, the created directory is deleted upon completion of the task.

  • reflect (boolean) – If true, a virtual sandbox is constructed from artifact metadata only. Files are not copied, but instead symlinks are created pointing at the origin of each file contained within the artifact. The sandbox reflects the artifact with a live view of the the current workspace.

Returns:

Path to the build directory..

Return type:

str

Example

def run(self, deps, tools):
   sandbox = tools.sandbox(deps["boost"], incremental=True)
setenv(key, value=None)

Sets or unset an environment variable.

Only child processes spawned by the same tools object can see the set environment variable and its value. Don’t assume the same variable is set in the Jolt process’ environment.

Parameters:
  • key (str) – Name of variable to set.

  • value (str) – Value of the variable or None to unset it.

Creates a symbolic link.

Parameters:
  • src (str) – Path to target file or directory.

  • dst (str) – Name/path of symbolic link.

  • replace (boolean) – Replace existing file or link. Default: false.

  • relative (boolean) – Create link using relative path to target. Default: false (absolute path).

thread_count()

Number of threads to use for a task.

Returns:

number of threads to use.

Return type:

int

tmpdir(name)

Creates a temporary directory.

The directory is only valid within a context and it is removed immediately upon leaving the context.

Parameters:

name (str) – Name prefix for the directory. A unique autogenerated suffix will also be appended to the final name.

Example

with tools.tmpdir("temp") as tmp, tools.cwd(tmp.path):
    tools.write_file("tempfile", "tempdata")

Removes a file from disk.

To remove directories, use rmtree().

Parameters:

pathname (str) – Path to the file to be removed.

unshare(uid=0, gid=0, groups=[0], uidmap=None, gidmap=None)

Experimental: Create a Linux namespace.

This method yields a new Linux namespace in which Python code may be executed. By default, the current user is mapped to root inside the namespace and all other users and groups are automatically mapped to the other user’s configured subuids and subgids.

The main use-case for namespaces is to fake the root user which may be useful in a number of situations:

  • to allow chroot() without beeing root

  • to allow mount() without beeing root

  • to preserve file ownership after tar file extraction

  • etc…

Requires a Linux host.

Note that the fork() system call is used. Changes made to variables will not persist after leaving the namespace.

Parameters:
  • uid (int) – Requested uid inside the namespace. This is always mapped to the uid of the caller.

  • gid (int) – Requested gid inside the namespace. This is always mapped to the gid of the caller.

  • uidmap (list) – List of uids to map in the namespace. A list of tuples is expected: (inner uid, outer id, number of ids to map).

  • gidmap (list) – List of gids to map in the namespace. A list of tuples is expected: (inner gid, outer id, number of ids to map).

Example

with tools.unshare() as ns, ns:
    # Inside namespace
    tools.run("whoami")  # "root"
# Back outside namespace, namespace destructed
upload(pathname, url, exceptions=True, auth=None, **kwargs)

Uploads a file using HTTP (PUT).

Throws a JoltError exception on failure.

Parameters:
  • pathname (str) – Name/path of file to be uploaded.

  • url (str) – Destination URL.

  • auth (requests.auth.AuthBase, optional) – Authentication helper. See requests.auth for details.

  • kwargs (optional) – Addidional keyword arguments passed on directly to ~requests.put().

which(executable)

Find executable in PATH.

Parameters:

executable (str) – Name of executable to be found.

Returns:

Full path to the executable.

Return type:

str

write_file(pathname, content=None, expand=True, **kwargs)

Creates a file.

Note

Existing files are overwritten.

Parameters:
  • pathname (str) – Name/path of file to be created.

  • content (str, optional) – Data to be written to the file.

  • expand (boolean, optional) – Expand macros in file content. Default: True.

  • **kwargs (dict, optional) – Additional key value dictionary used in macro expansion.

property wsroot

Return the root path of all build directories

CMake

CMake

class jolt.plugins.cmake.CMake(parameters=None, **kwargs)

Builds and publishes a CMake project

cmakelists = 'CMakeLists.txt'

Path to CMakeLists.txt or directory containing CMakelists.txt

options = []

List of options and their values (option[:type]=value)

Conan

Conan

class jolt.plugins.conan.Conan(*args, **kwargs)

Conan package installer task.

This task base class can be used to fetch, build and publish Conan packages as Jolt artifacts. All package metadata is transfered from the Conan package manifest to the Jolt artifact so that no manual configuration of include paths, library paths, macros, etc is required.

An existing installation of Conan is required. Please visit https://conan.io/ for installation instructions and documentation.

A minimal task to download and publish the Boost C++ libraries can look like this:

from jolt.plugins.conan import Conan

class Boost(Conan):
    packages = ["boost/1.74.0"]

Boost may then be used from Ninja tasks by declaring a requirement:

from jolt.plugins.ninja import CXXExecutable

class BoostApplication(CXXExecutable):
    requires = ["boost"]
    sources = ["main.cpp"]

The task supports using an existing conanfile.txt, but it is not required. Packages are installed into and collected from Jolt build directories. The user’s regular Conan cache will not be affected.

conanfile = None

An existing conanfile.txt file to use.

Instead of generating the conanfile.txt file on-demand, an external file may be used. If this attribute is set, the generators, options and packages attributes must not be set.

See Conan documentation for further details.

generators = []

A list of Conan generators to use.

See Conan documentation for details about supported generators. The json generator is always used.

Example:

generators = ["cmake"]
incremental = True

Keep installed packages in the Conan cache between Jolt invokations.

If incremental build is disabled, the Jolt Conan cache is removed before execution begins.

options = []

A list of Conan package options to apply

The option format is PkgName:Option=Value. See Conan documentation for further details.

Any {keyword} arguments, or macros, found in the strings are automatically expanded to the value of the associated task’s parameters and properties.

Example:

options = [
    "boost:shared=True",
    "zlib:shared=True",
]
packages = []

A list of Conan package references to collect and publish.

The reference format is PkgName/<version>@user/channel. See Conan documentation for further details.

Any {keyword} arguments, or macros, found in the strings are automatically expanded to the value of the associated task’s parameters and properties.

Example:

sdl_version = Parameter("2.0.12")

packages = [
    "boost/1.74.0",
    "sdl2/{sdl_version}@bincrafters/stable",
]
remotes = {}

A dictionary with Conan remotes to use when fetching packages.

The dictionary key is the name of remote and its value is the URL.

Example:

remotes = {
    "bincrafters": "https://api.bintray.com/conan/bincrafters/public-conan"
}

Docker

DockerClient

class jolt.plugins.docker.DockerClient(parameters=None, **kwargs)

Task: Downloads and publishes the Docker command line client.

The task will be automatically made available after importing jolt.plugins.docker.

arch = <jolt.tasks.Parameter object>

Host architecture [x86_64]

host = <jolt.tasks.Parameter object>

Host operating system [autodetected]

name = 'docker/cli'

Name of the task

url = 'https://download.docker.com/{host}/static/stable/{arch}/docker-{version}.tgz'

URL of binaries

version = <jolt.tasks.Parameter object>

Docker version [20.10.13]

DockerContainer

class jolt.plugins.docker.DockerContainer(*args, **kwargs)

Resource: Starts and stops a Docker container.

arguments = []

Container argument list

cap_adds = []

A list of capabilities to add to the container

cap_drops = []

A list of capabilities to remove from the container

entrypoint = None

Container entrypoint

environment = []

Environment variables

image = None

Image tag or Jolt task.

If a Jolt task is specified, its artifact must export a metadata string named tag with the name of the image tag.

labels = []

A list of container metadata labels

ports = []

A list of container ports to publish.

Example:

ports = [
    "80",
    "443:443",
]

Alternatively, assign True to publish all exposed ports to random ports.

privileged = False

Start container with elevated privileges.

user = None

Username or UID.

Defaults to the current user.

volumes = []

A list of volumes to mount.

By default, the cache directory and joltdir are automatically mounted in the container. See volumes_default.

volumes_default = ['{joltdir}:{joltdir}', '{joltcachedir}:{joltcachedir}']

A list of default volumes to mount.

By default, the cache directory and joltdir are automatically mounted in the container. Override to disable.

DockerImage

class jolt.plugins.docker.DockerImage(*args, **kwargs)

Abstract Task: Builds and publishes a Docker image.

Builds the selected Dockerfile and optionally tags and pushes the image to a registry. The image can also be saved to file and published in the task artifact. Compression formats supported are bzip2, gzip and lzma.

By default, base images referenced in the Dockerfile will be pulled during the build. Note that Jolt has no way of knowing beforehand if images have been updated in the registry. Use time-based influence to trigger rebuilds if it’s important that base images are kept up-to-date.

No automatic influence for Dockerfile or context is collected. Make sure to use an appropriate influence decorator.

Optionally add requirements to:

  • docker/cli to provision the Docker client, if none is available on the host.

  • docker/login to automatically login to the Docker registry.

This class must be subclassed.

Example:

# Dockerfile

FROM busybox:latest
CMD ["busybox"]
# build.jolt

from jolt.plugins.docker import DockerImage

class Busybox(DockerImage):
    """ Publishes Busybox image as gzip-compressed tarball """
    compression = "gz"
    requires = ["docker/cli"]
    tags = ["busybox:{identity}"]
autoload = True

Automatically load image file into local registry when the artifact is consumed by another task.

If the built image is saved to a file (i.e. imagefile is set), the image file is automatically loaded into the local Docker registry when the task artifact is consumed by another task. The image is also automatically removed from the registry upon completion of the consumer task.

Default: True.

buildargs = []

List of build arguments and their values (“ARG=VALUE”).

The arguments are passed to Docker using --build-arg.

cleanup = True

Remove image from Docker daemon upon completion [True]

compression = None

Optional image compression “bz2”, “gz”, or “xz”.

context = '.'

Path to build context, relative to joltdir (directory).

dockerfile = 'Dockerfile'

Path to the Dockerfile to build, or the full source code of such a file.

extract = False

Extract image and publish rootfs tree.

This option is useful when building a chroot to be used with jolt.Tools.chroot(). It disables saving of the image to a tarball.

imagefile = '{canonical_name}.tar'

Name of the image tarball published by the task.

If set to None, no image file will be saved and published.

Defaults to the task’s canonical name.

labels = []

A list of image metadata labels

platform = {}

Dictionary of task platform requirements.

Platform requirements control where tasks are allowed to execute. Multiple requirement key/values may be specified in which case all must be fulfilled in order for the task to be schedulable on a node.

The following builtin requirement labels exist:

  • node.arch [“arm”, “amd64”]

  • node.os [“linux”, “windows”]

Example

class Hello(Task):
    # This task must run on Linux.
    platform = {
        "node.os": "linux",
    }
pull = True

Always pull images when building.

Passes –pull to the Docker client.

push = False

Optionally push image to registry [False]

To be able to push images, the current user must login to the Docker Registry. The docker/login Jolt resource can be used for that purpose.

squash = False

Squash image layers

tags = ['{canonical_name}:{identity}']

Optional list of image tags. Defaults to task’s canonical name.

DockerLogin

class jolt.plugins.docker.DockerLogin(*args, **kwargs)

Resource: Logs in and out of a Docker Registry.

If the user and password parameters are unset, credentials are fetched from the environment variables:

  • DOCKER_USER

  • DOCKER_PASSWD

The resource will be automatically made available after importing jolt.plugins.docker.

name = 'docker/login'

Name of the resource

passwd = <jolt.tasks.Parameter object>

Docker Registry password.

If not set, the environment variable DOCKER_PASSWD is read instead.

user = <jolt.tasks.Parameter object>

Docker Registry username.

If not set, the environment variable DOCKER_USER is read instead.

Metadata

The Docker module registers and makes available these artifact metadata attributes:

  • artifact.docker.load - List of image files to be loaded from the artifact into the local Docker registry when the artifact is consumed.

    Example:

    def publish(self, artifact, tools):
        artifact.docker.load.append("image.tar")
    
  • artifact.docker.pull - List of image tags to be pulled from a remote registry to the local Docker registry when the artifact is consumed.

    Example:

    def publish(self, artifact, tools):
        artifact.docker.pull.append("busybox:latest")
    
  • artifact.docker.rmi - List of image tags to remove from the local Docker registry when a consuming task has finished.

    Example:

    def publish(self, artifact, tools):
        artifact.docker.rmi.append("busybox:latest")
    

Google Test

This module provides the GTestRunner task base class, a derivation of jolt.tasks.Runner for running Google Test applications. Several parameters can be assigned or overidden to control the behavior.

Example

examples/googletest/runner.jolt
from jolt.plugins.conan import Conan
from jolt.plugins.googletest import GTestRunner
from jolt.plugins.ninja import CXXExecutable

class GTest(Conan):
    packages = ["gtest/1.14.0"]
    settings = ["compiler.libcxx=libstdc++11"]

class Test(CXXExecutable):
    requires = ["gtest"]
    sources = ["test.cpp"]

class Run(GTestRunner):
    requires = ["test"]
    shuffle = False

The module also provides a set of task decorators for use with standalone task classes. They set different GTEST_* environment variables to control the behavior of Google Test executables in the same manner as the parameters in GTestRunner do.

Example

from jolt import Task
from jolt.plugins import googletest

# ....

@googletest.fail_fast(default=True)
@googletest.repeat(default=10)
@googletest.filter()
@googletest.junit_report()
class TestRunner(Task):
    requires = ["test"]

    def run(self, deps, tools):
        tools.run("unittest")
$ jolt build testrunner:repeat=1,filter=TestModule.*
jolt.plugins.googletest.break_on_failure(default: bool = False, param: bool = True, attr: str = 'break_on_failure')

Task class decorator controlling the GTEST_BREAK_ON_FAILURE environment variable.

When the variable is set, test applications failures trigger breakpoints. On Linux, this typically results in a core dump that can be loaded into the debugger.

The value is taken from the parameter break_on_failure which is created by default. If no parameter is created, the value is instead taken from the class attribute with the same name. The name of the paramter/attribute can be changed.

Parameters:
  • default (boolean) – Default value of the parameter and env. variable. Default: False.

  • param (boolean) – Create a task parameter. Default: True.

  • attr (str) – Name of parameter or class attribute.

jolt.plugins.googletest.brief(default: bool = False, param: bool = True, attr: str = 'brief')

Task class decorator controlling the GTEST_BRIEF environment variable.

When the variable is set, only failed test-cases are logged.

The value is taken from the parameter brief which is created by default. If no parameter is created, the value is instead taken from the class attribute with the same name. The name of the paramter/attribute can be changed.

GoogleTest version >= 1.11 is required.

Parameters:
  • default (boolean) – Default value of the parameter and env. variable. Default: False.

  • param (boolean) – Create a task parameter. Default: True.

  • attr (str) – Name of parameter or class attribute.

jolt.plugins.googletest.disabled(default: bool = False, param: bool = True, attr: str = 'disabled')

Task class decorator controlling the GTEST_ALSO_RUN_DISABLED_TESTS environment variable.

When the variable is set, disabled test-cases are also run.

The value is taken from the parameter disabled which is created by default. If no parameter is created, the value is instead taken from the class attribute with the same name. The name of the paramter/attribute can be changed.

Parameters:
  • default (boolean) – Default value of the parameter and env. variable. Default: False.

  • param (boolean) – Create a task parameter. Default: True.

  • attr (str) – Name of parameter or class attribute.

jolt.plugins.googletest.fail_fast(default: bool = False, param: bool = True, attr: str = 'fail_fast')

Task class decorator controlling the GTEST_FAIL_FAST environment variable.

When the variable is set, test applications will abort when the first failure is found.

The value is taken from the parameter fail_fast which is created by default. If no parameter is created, the value is instead taken from the class attribute with the same name. The name of the paramter/attribute can be changed.

Parameters:
  • default (boolean) – Default value of the parameter and env. variable. Default: False.

  • param (boolean) – Create a task parameter. Default: True.

  • attr (str) – Name of parameter or class attribute.

jolt.plugins.googletest.filter(default: str = '*', param: bool = True, attr: str = 'filter')

Task class decorator controlling the GTEST_FILTER environment variable.

The variable instructs test applications to only run test-cases that matches a wildcard filter string (default: *).

The value is taken from the parameter filter which is created by default. If no parameter is created, the value is instead taken from the class attribute with the same name. The name of the paramter/attribute can be changed.

Parameters:
  • default (boolean) – Default value of the parameter and env. variable. Default: False.

  • param (boolean) – Create a task parameter. Default: True.

  • attr (str) – Name of parameter or class attribute.

jolt.plugins.googletest.json_report()

Decorator enabling JSON test reporting in Google Test applications.

The decorator sets the GTEST_OUTPUT environment, thereby instructing test applications to write a JSON report file upon test completion. The task publishes the report in the task artifact under report/json/.

Any errors found in the report are parsed and attached to the task. They are included in email reports if the email plugin is enabled.

jolt.plugins.googletest.junit_report()

Decorator enabling JUnit test reporting in Google Test applications.

The decorator sets the GTEST_OUTPUT environment, thereby instructing test applications to write a JUnit report file upon test completion. The task publishes the report in the task artifact under report/junit/.

Any errors found in the report are parsed and attached to the task. They are included in email reports if the email plugin is enabled.

jolt.plugins.googletest.repeat(default: int = 1, param: bool = True, attr: str = 'repeat')

Task class decorator controlling the GTEST_REPEAT environment variable.

The variable instructs test applications to repeat test-cases the specified number of times.

The value is taken from the parameter repeat which is created by default. If no parameter is created, the value is instead taken from the class attribute with the same name. The name of the paramter/attribute can be changed.

Parameters:
  • default (boolean) – Default value of the parameter and env. variable. Default: False.

  • param (boolean) – Create a task parameter. Default: True.

  • attr (str) – Name of parameter or class attribute.

jolt.plugins.googletest.seed(default: int = 0, param: bool = True, attr: str = 'seed')

Task class decorator controlling the GTEST_RANDOM_SEED environment variable.

The variable sets an initial value for the random number generator used to shuffle test-cases. In order to reproduce ordering issues, a user may assign a specific seed to get the same test-case order. The default value is 0 which causes Google Test to use time as the seed.

The value is taken from the parameter seed which is created by default. If no parameter is created, the value is instead taken from the class attribute with the same name. The name of the paramter/attribute can be changed.

Parameters:
  • default (boolean) – Default value of the parameter and env. variable. Default: False.

  • param (boolean) – Create a task parameter. Default: True.

  • attr (str) – Name of parameter or class attribute.

jolt.plugins.googletest.shuffle(default: bool = False, param: bool = True, attr: str = 'shuffle')

Task class decorator controlling the GTEST_SHUFFLE environment variable.

When set, the test application runs its test-cases in random order. In order to reproduce test-case ordering issues, a user may assign a specific random number generator seed to get the same test-case execution order. See seed().

The value is taken from the parameter shuffle which is created by default. If no parameter is created, the value is instead taken from the class attribute with the same name. The name of the paramter/attribute can be changed.

Parameters:
  • default (boolean) – Default value of the parameter and env. variable. Default: False.

  • param (boolean) – Create a task parameter. Default: True.

  • attr (str) – Name of parameter or class attribute.

class jolt.plugins.googletest.GTestRunner(parameters=None, **kwargs)

Abstract task base class that runs Google Test applications.

Test executables are consumed from requirement artifacts. The artifacts must export metadata indicating the name of the binary, using artifact.strings.executable.

A number of parameters can be assigned from the command line to control the behavior of the task. Parameters can also be overridden using regular Python class attributes in subclasses.

A JUnit test report is generated and published in the task artifact. The results of the report are also imported into Jolt and distributed with report emails if the email plugin has been enabled.

Example:

examples/googletest/runner.jolt
from jolt.plugins.conan import Conan
from jolt.plugins.googletest import GTestRunner
from jolt.plugins.ninja import CXXExecutable

class GTest(Conan):
    packages = ["gtest/1.14.0"]
    settings = ["compiler.libcxx=libstdc++11"]

class Test(CXXExecutable):
    requires = ["gtest"]
    sources = ["test.cpp"]

class Run(GTestRunner):
    requires = ["test"]
    shuffle = False
break_on_failure = <jolt.tasks.BooleanParameter object>

Trigger breakpoint on test-case failure.

brief = <jolt.tasks.BooleanParameter object>

Only print failed test-cases.

disabled = <jolt.tasks.BooleanParameter object>

Also run disabled test-cases.

fail_fast = <jolt.tasks.BooleanParameter object>

Stop when the first failure is found.

filter = <jolt.tasks.Parameter object>

Test-case filter.

repeat = <jolt.tasks.IntParameter object>

Test-case repetitions.

seed = <jolt.tasks.IntParameter object>

Random number generator seed.

shuffle = <jolt.tasks.BooleanParameter object>

Randomize test-case execution order.

Ninja

CXXExecutable

CXXLibrary

Decorators

Rule