Command Line

jolt build

Build task artifact.

TASK is the name of the task to execute. It is optionally followed by a colon and parameter value assignments. Assignments are separated by commas. Example:

taskname:param1=value1,param2=value2

Default parameter values can be overridden for any task in the dependency tree with –default. DEFAULT is a qualified task name, just like TASK, but parameter assignments change default values.

By default, a task is executed locally and the resulting artifact is stored in the local artifact cache. If an artifact is already available in the cache, no execution takes place. Artifacts are identified with a hash digest, constructed from hashing task attributes.

When remote cache providers are configured, artifacts may be downloaded from and/or uploaded to the remote cache as execution progresses. Several options exist to control the behavior, such as –local which disables all remote caches.

Distributed task execution is enabled by passing the –network option. Tasks are then distributed to and executed by a pool of workers, if one has been configured.

Rebuilds can be forced with either –force or –salt. –force rebuilds the requested task, but not its dependencies. –salt affects the entire dependency tree. Both add an extra attribute to the task hash calculation in order to taint the identity and induce a cache miss. In both cases, existing intermediate files in build directories are removed before execution starts.

jolt build [OPTIONS] [TASK]...

Options

-c, --copy <copy>

Copy artifact content to directory PATH.

-d, --default <DEFAULT>

Override default parameter values.

-f, --force

Force rebuild of TASK artifact.

-g, --debug

Start debug shell before executing TASK.

-j, --jobs <JOBS>

Number of tasks allowed to execute in parallel [1].

-k, --keep-going

Build as many task artifacts as possible.

-l, --local

Disable remote cache access.

-n, --network

Distribute tasks to network workers.

-s, --salt <SALT>

Add salt as hash influence for all tasks in dependency tree.

--no-download

Don’t download artifacts from remote storage

--no-upload

Don’t upload artifacts to remote storage

--download

Do download artifacts from remote storage

--upload

Do upload artifacts to remote storage

Arguments

TASK

Optional argument(s)

jolt clean

Delete task artifacts and intermediate files.

When run without arguments, this command removes all task artifacts from the local cache, but no intermediate files are removed.

When TASK is specified, the task clean() method is invoked to remove any intermediate files still present in persistent build directories. Secondly, the task artifact will be removed from the local cache. Global caches are not affected. The –deps parameter can be used to also clean all dependencies of the specified TASK.

By default, task artifacts are removed without considering any artifact expiration metadata. To only remove artifact which have expired, use the –expired parameter. Artifacts typically expire immediately after creation unless explicitly configured not to.

jolt clean [OPTIONS] [TASK]...

Options

-d, --deps

Clean all task dependencies.

-e, --expired

Only clean expired tasks.

Arguments

TASK

Optional argument(s)

jolt config

Configure Jolt.

You can query/set/replace/unset configuration keys with this command. Key strings are constructed from the configuration section and the option separated by a dot.

There are tree different configuration sources:

  • A global configuration file

  • A user configuration file

  • Temporary configuration passed on the command line.

When reading, the values are read from all configuration sources. The options –global and –user can be used to tell the command to read from only one of the sources. If a configuration key is available from multiple sources, temporary CLI configuration has priority followed by the user configuration file and lastly the global configuration file.

When writing, the new values are written to the user configuration by default. The options –global and –user can be used to tell the command to write to only one of the sources.

When removing keys, the values are removed from all sources. The options –global and –user can be used to restrict removal to one of the sources.

To assign a value to a key:

$ jolt config jolt.default all # Change name of the default task

To list existing keys:

$ jolt config -l # List all existing keys

$ jolt config -l -g # List keys in the global config file

$ jolt config jolt.colors # Display the value of a key.

To delete an existing key:

$ jolt config -d jolt.colors

To pass temporary configuration:

$ jolt -c jolt.colors=true config -l

jolt config [OPTIONS] [KEY] [VALUE]

Options

-l, --list

List all configuration keys and values.

-d, --delete

Delete configuration key.

-g, --global

List, set or get configuration keys in the global config.

-u, --user

List, set or get configuration keys in the user config.

Arguments

KEY

Optional argument

VALUE

Optional argument

jolt display

Display a task and its dependencies visually.

jolt display [OPTIONS] [TASK]...

Options

-r, --reverse <reverse>

Display consumers of REVERSE if TASK is executed.

-c, --cached

Highlight cache presence with colors.

Arguments

TASK

Optional argument(s)

jolt docs

Opens the Jolt documentation in the default webbrowser.

jolt docs [OPTIONS]

jolt export

Export task artifact metadata into shell environment.

When running the command, a shell script is printed to stdout. It can be redirected to a file and executed separately, or sourced directly in the shell by running:

source <(jolt export <task>)

The script creates a virtual environment that is identical to the environment that would be setup when the specified task artifact is consumed by another task. This enables packaged applications, such as compilers, to be run directly from the shell.

The command will fail if the task artifact or any dependency artifact is missing in the local cache. Build the task to populate the cache and then try again.

Run deactivate-jolt to leave the virtual environment. All environment variables will be restored to their original values.

jolt export [OPTIONS] TASK...

Arguments

TASK

Required argument(s)

jolt inspect

View information about a task.

This command displays information about a task, such as its class documentation, parameters and their accepted values, requirements, task class origin (file/line), influence attributes, artifact identity, cache status, and more. Default parameter values, if any, are highlighted.

jolt inspect [OPTIONS] TASK

Options

-i, --influence

Print influence attributes and values.

-a, --artifact

Print artifact cache status.

-s, --salt <SALT>

Add salt as task influence.

Arguments

TASK

Required argument

jolt list

List all tasks, or dependencies of a task.

By default, when no TASK is specified, all known task names are listed in alphabetical order.

When a TASK is specified, only direct dependencies of that task are listed. Use -a to also list its indirect dependencies.

Multiple TASK names are allowed.

jolt list [OPTIONS] [TASK]...

Options

-a, --all

List all direct and indirect dependencies of TASK.

-r, --reverse <REVERSE>

Only list dependencies of TASK that are also reverse dependencies of REVERSE.

Arguments

TASK

Optional argument(s)

jolt log

Display the Jolt log file.

jolt log [OPTIONS]

Options

-f, --follow

Display log output as it appears

-d, --delete

Delete the log file

Reference

Alias

class jolt.Alias(*args, **kwargs)

An alias task.

Aliases are a special kind of task which can be used to introduce new names for other tasks or groups of tasks. They are useful as milestones when building continuous integration pipelines since they won’t be executed, thus saving time compared to a regular task.

name

Name of the alias. Derived from class name if not set.

requires = []

List of dependencies to other tasks.

Artifact

class jolt.Artifact(cache, node, tools=None)

An artifact is a collection of files and metadata produced by a task.

Task implementors call artifact methods to collect files to be published. In addition to files, other metadata can be provided as well, such as variables that should be set in the environment of consumer tasks.

collect(files, dest=None, flatten=False, symlinks=False, cwd=None)

Collect files to be included in the artifact.

Parameters
  • files (str) – A filename pattern matching the files to be included in the artifact. The pattern may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

  • dest (str, optional) – Destination path within the artifact. If the string ends with a path separator a new directory will be created and all matched source files will be copied into the new directory. A destination without trailing path separator can be used to rename single files, one at a time.

  • flatten (boolean, optional) – If True, the directory tree structure of matched source files will flattened, i.e. all files will be copied into the root of the destination path. The default is False, which retains the directory tree structure relative to the current working directory.

  • symlinks (boolean, optional) – If True, symlinks are copied. The default is False, i.e. the symlink target is copied.

  • cwd (str, optional) – Change current working directory before starting collection.

copy(files, dest, flatten=False, symlinks=False, cwd=None)

Copy files from the artifact.

Parameters
  • files (str) – A filename pattern matching the files to be copied from the artifact. The filepath is relative to the artifact root and may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

  • dest (str, optional) – Destination path, relative to the current working directory. If the string ends with a path separator a new directory will be created and all matched source files will be copied into the new directory. A destination without trailing path separator can be used to rename single files, one at a time.

  • flatten (boolean, optional) – If True, the directory tree structure of matched source files will flattened, i.e. all files will be copied into the root of the destination path. The default is False, which retains the directory tree structure.

  • symlinks (boolean, optional) – If True, symlinks are copied. The default is False, i.e. the symlink target is copied.

  • cwd (str, optional) – Change destination working directory before starting copy.

cxxinfo = {}

Artifact C/C++ build metadata.

A task can add compilation metadata to an artifact. Such metadata will be automatically applied when consumer compilation tasks are executed. A common use-case is to add preprocessor definitions, link libraries, etc. These string fields are supported:

  • asflags - assembler flags (string)

  • cflags - compiler flags (string)

  • cxxflags - compiler flags (string)

  • ldflags - linker flags (string)

  • libraries - libraries to link with (list, use append())

  • macros - preprocessor macros to set (list, use append())

  • sources - source files to compile (list, use append())

Values appended to PATH-type metadata fields are relative to the artifact root. They will be automatically expanded to absolute paths. These PATH-type fields are supported:

  • incpaths - preprocessor include paths (list, use append())

  • libpaths - linker library search paths (list, use append())

Example

def publish(self, artifact, tools):
    artifact.collect("*.h", "include/")
    artifact.cxxinfo.incpaths.append("include")
    artifact.cxxinfo.macros.append("PACKAGE_VERSION=1.0")
environ = {}

Artifact environment variables.

A task can add environment variables to an artifact. Such a variable will automatically be set in the environment when consumer tasks are executed. A common use-case is to add programs to the PATH.

Values appended to PATH variables are relative to the artifact root. They will be automatically expanded to absolute paths. This applies to all variables with PATH in the name.

Example

def publish(self, artifact, tools):
    artifact.environ.PATH.append("bin")
    artifact.environ.JAVA_HOME = artifact.final_path
property final_path

The final location of the artifact in the local cache.

Type

str

property path

The current location of the artifact in the local cache.

Type

str

paths = {}

Artifact paths.

A task can add paths to files and directories inside an artifact. Paths are relative to the root of the artifact when created, but are expanded to absolute paths when the artifact is consumed by a task.

This is useful as an abstraction when directories or filenames have varying names.

Example

def publish(self, artifact, tools):
    artifact.paths.file = "{date}.txt"

The file path is then expanded to a full path for consumers:

requires = ["dep"]

def run(self, deps, tools):
    filedata = tools.read_file(deps["dep"].paths.file)
python = {}

Artifact Python configuration.

A task can add Python configuration to an artifact. Such configuration will automatically be set in the environment when consumer tasks are executed. A common use-case is to add Python modules to the PATH so that they can be easily imported by a consumer.

Values appended to PATH-type variables are relative to the artifact root. They will be automatically expanded to absolute paths.

Example

def publish(self, artifact, tools):
    artifact.python.PATH.append("my_module")
strings = {}

Artifact strings.

A task can add arbitrary string values to an artifact. Such a string will be available for consumer tasks to read.

Example

def publish(self, artifact, tools):
    artifact.strings.version = "1.2"

Context

class jolt.Context(cache, node)

Execution context and dependency wrapper.

A Context gathers dependencies and initializes the environment for an executing task. It is passed as an argument to the Task’s run() method.

A task implementor can use the context as a dictionary of dependencies where the key is the name of a dependency and the value is the dependency’s Artifact.

__getitem__(key)

Get artifact for a task listed as a requirement.

Parameters

key (str) – Name of the task listed as a requirement.

Returns

The Artifact associated with the task.

Example

requires = "dependency"

def run(self, deps, tools):
    dependency_artifact = deps["dependency"]
items()

List all requirements and their artifacts.

Returns

Requirement dictionary items. Each item is a tuple with the requirement name and the artifact.

Decorators

class jolt.attributes
static attribute(alias, target, influence=True)

Decorates a task with an alias for another attribute.

Parameters
  • attrib (str) – Name of alias attribute.

  • target (str) – Name of target attribute. Keywords are expanded.

  • influence (boolean) – Add value of target attribute as influence of the task.

static environ(envname, influence=True)

Decorator marking the task as dependent on an environment variable.

The value of the environment variable will be automatically transferred to workers in distributed network builds.

Parameters
  • envname (str) – Name of the environment variable.

  • influence (boolean) – Add value of environment variable as influence of the task. Default: True.

static requires(attrib)

Decorates a task with an alternative requires attribute.

The new attribute will be concatenated with the regular requires attribute.

Parameters

attrib (str) – Name of alternative attribute. Keywords are expanded.

static system(cls)

Decorates a task with a property returning the operating system name.

Examples: “linux”, “windows”

Download

class jolt.tasks.Download(parameters=None, **kwargs)

Downloads file(s) over HTTP(S).

Once downloaded, archives are extracted and all of their files are published. If the file is not an archive it is published as is. Recognized archive extensions are:

  • .tar

  • .tar.bz2

  • .tar.gz

  • .tar.xz

  • .tgz

  • .zip

Example

class NodeJS(Download):
    """ Downloads and publishes Node.js. Adds binaries to PATH. """

    version = Parameter("14.16.1")
    url = "https://nodejs.org/dist/v{version}/node-v{version}-win-x64.zip"

    def publish(self, artifact, tools):
        super(publish).publish(artifact, tools)
        artifact.environ.PATH.append("node-v{version}-win-x64")
collect = ['*']

A list of file publication instructions.

Items in the list are passed directly to Artifact.collect() and can be either strings, tuples or dictionaries.

Example

collect = [
    "*",                           # Collect all files
    ("*", "src/"),                 # Collect all files into the artifact's src/ directory
    {"files": "*", cwd="subdir"},  # Collect all files from the archive's subdir/ directory
]
extract = True

Automatically extract archives.

url = None

URL(s) of file(s) to download.

A single URL string is accepted, as well as a list of URL strings.

Influence

jolt.influence.always(cls)

Always execute the task.

Example

from jolt import influence

@influence.always
class Example(Task):
jolt.influence.attribute(name, type=None, sort=False)

Add task attribute value as hash influence.

Parameters
  • name (str) – Name of task class attribute/property.

  • type (InfluenceProvider, optional) – Alternative HashInfluenceProvider implementation used to interpret the value of the attribute. If the attribute is a list, the provider will be instantiated once for each item in the list. For example, if the attribute is a list of filesystem paths, the FileInfluence class could be passed as argument to automatically monitor files at those paths.

  • sort (boolean, optional) – Optionally sort the value. Always sort values that are unstable, such as dictionaries.

Example:

from jolt import influence

@influence.attribute("attribute")
class Example(Task):
    attribute = False
from jolt import influence

@influence.attribute("sources", type=influence.FileInfluence)
class Example(Task):
    sources = [
        "main.cpp",
        "utils.cpp",
    ]
jolt.influence.daily(cls)

Add daily hash influence.

If nothing else changes, the task is re-executed once every day.

Example

from jolt import influence

@influence.daily
class Example(Task):
jolt.influence.environ(variable)

Add environment variable hash influence.

Parameters

variable (str) – Name of an environment variable that will influence the hash of the task.

Example:

from jolt import influence

@influence.environ("CFLAGS")
class Example(Task):
jolt.influence.files(pathname)

Add file content hash influence.

Parameters

pathname (str) – A pathname pattern used to find files that will influence the hash of the task The pattern may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

Example:

from jolt import influence

@influence.files("*.cpp")
class Example(Task):
jolt.influence.hourly(cls)

Add hourly hash influence.

If nothing else changes, the task is re-executed once every hour.

Example

from jolt import influence

@influence.hourly
class Example(Task):
jolt.influence.monthly(cls)

Add monthly hash influence.

If nothing else changes, the task is re-executed once every month.

Example

from jolt import influence

@influence.monthly
class Example(Task):
jolt.influence.weekly(cls)

Add weekly hash influence.

If nothing else changes, the task is re-executed once every week.

Example

from jolt import influence

@influence.weekly
class Example(Task):
jolt.influence.yearly(cls)

Add yearly hash influence.

If nothing else changes, the task is re-executed once every year.

Example

from jolt import influence

@influence.yearly
class Example(Task):

Parameter

class jolt.Parameter(default=None, values=None, required=True, const=False, influence=True, help=None)

Generic task parameter type.

__init__(default=None, values=None, required=True, const=False, influence=True, help=None)

Creates a new parameter.

Parameters
  • default (str, optional) – An optional default value.

  • values (list, optional) – A list of accepted values. An assertion is raised if an unlisted value is assigned to the parameter.

  • required (boolean, optional) – If required, the parameter must be assigned a value before the task can be executed. The default is True.

  • const (boolean, optional) – If const is True, the parameter is immutable and cannot be assigned a non-default value. This is useful in a class hierarchy where a subclass may want to impose restrictions on a parent class parameter. The default is False.

  • influence (boolean, optional) – If influence is False, the parameter value will not influence the identity of the task artifact. The default is True.

  • help (str, optional) – Documentation for the parameter. This text is displayed when running the info command on the associated task.

Raises

ValueError – If the parameter is assigned an illegal value.

__str__()

Returns the parameter value as a string

get_default()

Get the default value of the parameter.

Returns

The default value or None if no default was given.

get_value()

Get the parameter value.

is_default()

Check if parameter is set to its default value.

Returns

True if the assigned value is the default value.

is_set()

Check if the parameter is set to a non-default value.

Returns

True if the assigned value is not the default value.

is_unset()

Check if the parameter is unset.

Returns

True if the assigned value is None.

set_value(value)

Set the parameter value.

Parameters

value (str) – The new parameter value.

Raises

ValueError – If the parameter is assigned an illegal value.

BooleanParameter

class jolt.BooleanParameter(default=None, required=True, const=False, influence=True, help=None)

Boolean task parameter.

Accepted values are:

  • False

  • True

  • “false”

  • “true”

  • “no”

  • “yes”

  • “0”

  • “1”

__bool__()

Returns the boolean parameter value

__getitem__(key)

Returns a substitution string depending on the parameter value.

Parameters
  • key (str) – A special key syntax, enabled,disabled, where

  • returned (and either enabled or disabled would be) –

  • below. (depending on the paramter value. See the example) –

Returns

Substitution string.

Example

class Example(Task):
   debug = BooleanParameter()

   def run(self, deps, tools):
       self.info("debug is {debug[enabled,disabled]}")
$ jolt build example:debug=true
[INFO] debug is enabled (example)

$ jolt build example:debug=false
[INFO] debug is disabled (example)
__init__(default=None, required=True, const=False, influence=True, help=None)

Creates a new parameter.

Parameters
  • default (boolean, optional) – An optional default boolean value.

  • required (boolean, optional) – If required, the parameter must be assigned a value before the task can be executed. The default is True.

  • const (boolean, optional) – If const is True, the parameter is immutable and cannot be assigned a non-default value. This is useful in a class hierarchy where a subclass may want to impose restrictions on a parent class parameter. The default is False.

  • influence (boolean, optional) – If influence is False, the parameter value will not influence the identity of the task artifact. The default is True.

  • help (str, optional) – Documentation for the parameter. This text is displayed when running the info command on the associated task.

Raises

ValueError – If the parameter is assigned an illegal value.

__str__()

Returns the parameter value as a string

get_default()

Get the default value of the parameter.

Returns

The default value or None if no default was given.

get_value()

Get the parameter value.

is_default()

Check if parameter is set to its default value.

Returns

True if the assigned value is the default value.

property is_false

The parameter value is False.

is_set()

Check if the parameter is set to a non-default value.

Returns

True if the assigned value is not the default value.

property is_true

The parameter value is True.

is_unset()

Check if the parameter is unset.

Returns

True if the assigned value is None.

ListParameter

class jolt.ListParameter(*args, **kwargs)

List parameter type.

A list parameter allows multiple values to be assigned to it. Values are separated by the ‘+’ character in qualified task names. Each assigned value is validated against the list of accepted values. They are sorted in alphabetical order before the task is executed.

Example

class Example(Task):
    arg = ListParameter(default=["c"], values=["a", "b", "c"], help="A list parameter")

    def run(self, deps, tools):
        for item in self.arg:
            print(item)
$ jolt build example example:arg=a example:arg=a+b
__getitem__(key)

Returns an element or a slice from the list.

Parameters

key (int, slice, str) –

Element index or slice. A key string can be used to check for the existence of that value in the list. If the key is present the same value is returned, otherwise None.

A special key syntax is also available to request an alternate return value depending on the presence of the key. Instead of a list value you pass value,present,absent and either present or absent will be returned. See the example below.

Returns

Element value, or substitution.

Example

class Example(Task):
   features = ListParameter(values=["optimize", "strip"], required=False)

   def run(self, deps, tools):
       if len(self.features) > 0:
           self.info("first feature is {features[0]}")
           self.info("optimize == {features[optimize]}")
           self.info("optimization is {features[optimize,enabled,disabled]}")
$ jolt build example:features=optimize+strip
[INFO] first feature is optimize (example)
[INFO] optimize = optimize (example)
[INFO] optimization is enabled (example)

$ jolt build example:features=strip
[INFO] first feature is debug (example)
[INFO] optimize = None (example)
[INFO] optimization is disabled (example)
__init__(*args, **kwargs)

Creates a new list parameter.

Parameters
  • default (boolean, optional) – An optional list of default values.

  • values (list, optional) – A list of accepted values. An assertion is raised if an unlisted value is assigned to the parameter.

  • required (boolean, optional) – If required, the parameter must be assigned a value before the task can be executed. The default is True.

  • const (boolean, optional) – If const is True, the parameter is immutable and cannot be assigned a non-default value. This is useful in a class hierarchy where a subclass may want to impose restrictions on a parent class parameter. The default is False.

  • influence (boolean, optional) – If influence is False, the parameter value will not influence the identity of the task artifact. The default is True.

  • help (str, optional) – Documentation for the parameter. This text is displayed when running the info command on the associated task.

Raises

ValueError – If the parameter is assigned an illegal value.

__iter__()

Returns a sequence iterator.

__len__()

Returns the length of the list.

__str__()

Returns the parameter value as a string

get_default()

Get the default value of the parameter.

Returns

The default value or None if no default was given.

get_value()

Get the parameter value.

is_default()

Check if parameter is set to its default value.

Returns

True if the assigned value is the default value.

is_set()

Check if the parameter is set to a non-default value.

Returns

True if the assigned value is not the default value.

is_unset()

Check if the parameter is unset.

Returns

True if the assigned value is None.

Resource

class jolt.tasks.Resource(*args, **kwargs)

Bases: Task

A resource task.

Resources are special tasks executed in the Context of other tasks. They are invoked to acquire and release a resource, such as hardware equipment, before and after the execution of a task. No artifact is produced by a resource.

Implementors should override acquire() and release().

acquire(artifact, deps, tools, owner)

Called to acquire the resource.

An implementor overrides this method in a subclass. The acquired resource must be released manually if an exception occurs before the method has returned. If this method returns successfully, the release() method is guaranteed to be called in the future upon completion of the consuming task (unless the process is forcibly interrupted or killed).

Parameters
  • artifact (Artifact) – The artifact associated with the resource. It is not possible to publish files from a resource, but the implementor can still use the resource to pass information to consuming tasks.

  • deps (Context) – Task execution context used to access the artifacts of dependencies.

  • tools (Tools) – A task specific toolbox.

  • owner (Task) – The owner task for which the resource is acquired.

release(artifact, deps, tools, owner)

Called to release the resource.

An implementor overrides this method in a subclass.

Parameters
  • artifact (Artifact) – The artifact associated with the resource. It is not possible to publish files from a resource, but the implementor can still use the resource to pass information to consuming tasks.

  • deps (Context) – Task execution context used to access the artifacts of dependencies.

  • tools (Tools) – A task specific toolbox.

  • owner (Task) – The owner task for which the resource is released.

Script

class jolt.tasks.Script(parameters=None, **kwargs)

Bases: Task

A simple shell script task.

The script source is extracted directly from the task class documentation. All text following a --- separator will be executed.

A temporary build directory is automatically created and can be accessed with {builddir}. All other task attributes are also expanded as usual. Dependency artifacts are accessible through the deps dictionary.

echo {deps[task].path}

By default, all files in the build directory are published in the task artifact. The collect attribute can be used to customize file collection. Alternatively, the publish method may be overridden.

Keep in mind that shell scripts are not necessarily portable between host operating systems. Implement your tasks in Python code if portability is a concern.

Example

class Hello(Script):
    """
    Classic Hello World!
    ---
    # Script source

    echo "Hello world!" > {builddir}/hello.txt
    """
collect = [{'files': '*', 'cwd': '{builddir}'}]

A list of file publication instructions.

Items in the list are passed directly to Artifact.collect() and can be either strings, tuples or dictionaries.

By default, all files in the build directory are published.

Example

collect = [
    "*",                              # Collect all files
    ("*", "src/"),                    # Collect all files into the artifact's src/ directory
    {"files": "*", "cwd": "subdir"},  # Collect all files from the archive's subdir/ directory
]

Task

class jolt.tasks.Task(parameters=None, **kwargs)
cacheable = True

Whether the task produces an artifact or not.

expires = Immediately()

An expiration strategy, defining when the artifact may be evicted from the cache.

When the size of the artifact cache exceeds the configured limit an attempt will be made to evict artifacts from the cache. The eviction algorithm processes artifacts in least recently used (LRU) order until an expired artifact is found.

By default, an artifact expires immediately and may be evicted at any time (in LRU order). An exception to this rule is if the artifact is required by a task in the active task set. For example, if a task A requires the output of task B, B will never be evicted by A while A is being executed.

There are several expiration strategies to choose from:

  • jolt.expires.WhenUnusedFor

  • jolt.expires.After

  • jolt.expires.Never

Examples:

# May be evicted if it hasn't been used for 15 days
expires = WhenUnusedFor(days=15)
# May be evicted 1h after creation
expires = After(hours=1)
# Never evicted
expires = Never()
extends = ""

Name of extended task.

A task with this attribute set is called an extension. An extension is executed in the context of the extended task, immediately after the extended task has executed.

A common use-case for extensions is to produce additional artifacts from the output of another task. Also, for long-running tasks, it is sometimes beneficial to utilize the intermediate output from an extended task. The extension artifact can then be acquired more cheaply than if the extension had performed all of the work from scratch.

An extension can only extend one other task.

fast = False

Indication of task speed.

The information is used by the distributed execution strategy to optimize how tasks are scheduled. Scheduling tasks remotely is always associated with some overhead and sometimes it’s beneficial to instead schedule fast tasks locally if possible.

An extended task is only considered fast if all extensions are fast.

influence = []

List of influence provider objects.

name

Name of the task. Derived from class name if not set.

requires = []

List of dependencies to other tasks.

selfsustained = False

Consume this task independently of its requirements.

Requirements of a self-sustained task will be pruned if the task artifact is present in a cache. In other words, if the task is not executed its requirements are considered unnecessary.

For example, consider the task graph A -> B -> C. If B is self-sustained and present in a cache, C will never be executed. C will also never be a transitive requirement of A. If A requires C, it should be listed as an explicit requirement.

Using this attribute speeds up execution and reduces network traffic by allowing the task graph to be reduced.

error(fmt, *args, **kwargs)

Log an error concerning the task

info(fmt, *args, **kwargs)

Log information about the task.

joltdir = '.'

Path to the directory of the .jolt file where the task was defined.

publish(artifact, tools)

Publishes files produced by run().

Files can be collected in to the artifact by calling artifact.collect().

Additional metadata can be provided, such as environment variables that should be set whenever the task artifact is consumed. Example:

# Append <artifact-path>/bin to the PATH
artifact.environ.PATH.append("bin")

# Pass an arbitrary string to a consumer
artifact.strings.foo = "bar"
run(deps, tools)

Performs the work of the task.

Dependencies specified with “requires” are passed as the deps dictionary. The tools argument provides a set of low level tool functions that may be useful.

with tools.cwd("path/to/subdir"):
    tools.run("make {target}")

When using methods from the toolbox, task parameters, such as target above, are automatically expanded to their values.

unpack(artifact, tools)

Unpacks files published by publish() .

The intention of this hook is to make necessary adjustments to artifact files and directories once they have been downloaded into the local cache on a different machine. For example, paths may have to be adjusted or an installer executed.

This hook is executed in the context of a consuming task.

warning(fmt, *args, **kwargs)

Log a warning concerning the task

Test

class jolt.tasks.Test(*args, **kwargs)

A test task.

The test task combines a regular Jolt task with a Python unittest.TestCase. As such, a test task is a collection of similar test-cases where each test-case is implemented as an instancemethod named with a test_ prefix. When executed, the task runs all test-case methods and summarizes the result.

All regular unittest assertions and decorators can be used in the test methods. For details about inherited task attributes, see jolt.tasks.Task and Python unittest.TestCase.

Example:

class OperatorTest(Test):

    def test_add(self):
        self.assertEqual(1+1, 2)

    def test_sub(self):
        self.assertEqual(2-1, 1)
cleanup()

Implement this method to clean up after a test

static parameterized(args)

Parameterizes a test method.

The test method is instantiated and called once with each argument tuple in the list.

Example

class Example(Test):
   @Test.parameterized([
       (1, 1, 1),  # 1*1 == 1
       (1, 2, 2),  # 1*2 == 2
       (2, 1, 2),  # 2*1 == 2
       (2, 2, 4),  # 2*2 == 4
   ])
   def test_multiply(self, factor1, factor2, product):
       self.assertEqual(factor1*factor2, product)
setup(deps, tools)

Implement this method to make preparations before a test

Tools

class jolt.Tools(task=None, cwd=None, env=None)

A collection of useful tools.

Any {keyword} arguments, or macros, found in strings passed to tool functions are automatically expanded to the value of the associated task’s parameters and properties. Relative paths are made absolute by prepending the current working directory.

append_file(pathname, content, expand=True)

Appends data at the end of a file.

Parameters
  • pathname (str) – Path to file. The file must exist.

  • content (str) – Data to be appended at the end of the file.

archive(pathname, filename)

Creates a (compressed) archive.

The type of archive to create is determined by the filename extension. Supported formats are:

  • tar

  • tar.bz2

  • tar.gz

  • tar.xz

  • zip

Parameters
  • pathname (str) – Directory path of files to be archived.

  • filename (str) – Name/path of created archive.

autotools(deps=None)

Creates an AutoTools invokation helper

builddir(name=None, incremental=False, unique=True)

Creates a temporary build directory.

The build directory will persist for the duration of a task’s execution. It is automatically removed afterwards.

Parameters
  • name (str) – Name prefix for the directory. A unique autogenerated suffix will also be appended to the final name.

  • incremental (boolean) – If false, the created directory is deleted upon completion of the task.

Returns

Path to the created directory.

property buildroot

Return the root path of all build directories

checksum_file(filelist, concat=False, hashfn=<built-in function openssl_sha1>, filterfn=None)

Calculate a checksum of one or multiple files.

Parameters
  • filelist (str,list) – One or multiple files.

  • concat (boolean) – Concatenate files and return a single digest. If False, a list with one digest for each file is returned. Default: False.

  • hashfn – The hash algorithm used. Any type which provides an update() and hexdigest() method is accepted. Default: hashlib.sha1

  • filterfn – An optional data filter function. It is called repeatedly with each block of data read from files as its only argument. It should return the data to be included in the checksum. Default: None

Returns

A list of checksum digests, or a single digest if files where concatenated.

chmod(pathname, mode)

Changes permissions of files and directories.

Parameters
  • pathname (str) – Path to a file or directory to change permissions for.

  • mode (int) – Requested permission bits.

chroot(chroot, *args, **kwargs)

Experimental: Use chroot as root filesystem when running commands.

Mounts the specified chroot as the root filesystem in a new Linux namespace, which is used when calling Tools.run().

Requires a Linux host.

Parameters

chroot (str, artifact) – Path to rootfs directory, or an artifact with a ‘rootfs’ metadata path (artifact.paths.rootfs).

Example

with tools.choot("path/to/rootfs"):
    tools.run("ls")
cmake(deps=None, incremental=False)

Creates a CMake invokation helper

compress(src, dst)

Compress a file.

Supported formats are:

  • .bz2

  • .gz

  • .xz

Parameters
  • src (str) – Source file to be compressed.

  • dst (str) – Destination path for compressed file. The filename extension determines the compression algorithm used.

copy(src, dst, symlinks=False)

Copies file and directories (recursively).

The directory tree structure is retained when copying directories.

Parameters
  • src (str) – Path to a file or directory to be copied.

  • dest (str) – Destination path. If the string ends with a path separator a new directory will be created and source files/directories will be copied into the new directory. A destination without trailing path separator can be used to rename single files, one at a time.

  • symlinks (boolean, optional) – If True, symlinks are copied. The default is False, i.e. the symlink target is copied.

cpu_count()

The number of CPUs on the host.

Returns

The number of CPUs on the host.

Return type

int

cwd(pathname, *args)

Change the current working directory to the specified path.

This function doesn’t change the working directory of the Jolt process. It only changes the working directory for tools within the tools object.

Parameters

pathname (str) – Path to change to.

Example

with tools.cwd("subdir") as cwd:
    print(cwd)
download(url, pathname, exceptions=True, **kwargs)

Downloads a file using HTTP.

Throws a JoltError exception on failure.

Parameters
  • url (str) – URL to the file to be downloaded.

  • pathname (str) – Name/path of destination file.

  • kwargs (optional) – Addidional keyword arguments passed on directly requests.get().

environ(**kwargs)

Set/get environment variables.

Only child processes spawned by the same tools object will be affected by the changed environment.

The changed environment is only valid within a context and it is restored immediately upon leaving the context.

Parameters

kwargs (optinal) – A list of keyword arguments assigning values to environment variable.

Example

with tools.environ(CC="clang"):
    tools.run("make all")
expand(string, *args, **kwargs)

Expands keyword arguments/macros in a format string.

This function is identical to str.format() but it automatically collects keyword arguments from a task’s parameters and properties.

It also supports three additional conversion specifiers:

  • c: call method without arguments

  • l: convert string to lower case

  • u: convert string to upper case

Parameters
  • string (str) – The string to be expanded.

  • args (str, optional) – Additional positional values required by the format string.

  • kwargs (str, optional) – Additional keyword values required by the format string.

Returns

Expanded string.

Return type

str

Example

target = Parameter(default="all")
verbose = "yes"

def run(self, deps, tools):
    print(tools.expand("make {target} VERBOSE={verbose}"))  # "make all VERBOSE=yes"
expand_path(pathname, *args, **kwargs)

Expands keyword arguments/macros in a pathname format string.

This function is identical to str.format() but it automatically collects keyword arguments from a task’s parameters and properties.

The function also makes relative paths absolute by prepending the current working directory.

Parameters
  • pathname (str) – The pathname to be expanded.

  • args (str, optional) – Additional positional values required by the format pathname.

  • kwargs (str, optional) – Additional keyword values required by the format pathname.

Return

str: Expanded string.

expand_relpath(pathname, relpath=None, *args, **kwargs)

Expands keyword arguments/macros in a pathname format string.

This function is identical to str.format() but it automatically collects keyword arguments from a task’s parameters and properties.

The function also makes absolute paths relative to a specified directory.

Parameters
  • pathname (str) – The pathname to be expanded.

  • relpath (str, optional) – Directory to which the returned path will be relative. If not provided, the joltdir attribute is used.

  • args (str, optional) – Additional positional values required by the format pathname.

  • kwargs (str, optional) – Additional keyword values required by the format pathname.

Return

str: Expanded string.

extract(filename, pathname, files=None, ignore_owner=False)

Extracts files in an archive.

Supported formats are:

  • tar

  • tar.bz2

  • tar.gz

  • tar.xz

  • zip

Parameters
  • filename (str) – Name/path of archive file to be extracted.

  • pathname (str) – Destination path for extracted files.

  • files (list, optional) – List of files the be extracted from the archive. If not provided, all files are extracted.

file_size(pathname)

Determines the size of a file.

Parameters

pathname (str) – Name/path of file for which the size is requested.

Returns

The size of the file in bytes.

Return type

int

getcwd()

Returns the current working directory.

getenv(key, default='')

Returns the value of an environment variable.

Only child processes spawned by the same tools object can see the environment variables and their values returned by this method. Don’t assume the same variables are set in the Jolt process’ environment.

glob(pathname, expand=False)

Enumerates files and directories.

Parameters
  • pathname (str) – A pathname pattern used to match files to be included in the returned list of files and directories. The pattern may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

  • expand (boolean) – Expand matches to absolute paths. Default: false.

Returns

A list of file and directory pathnames. The pathnames are relative to the current working directory unless the pathname argument was absolute.

Example

textfiles = tools.glob("*.txt")
map_concurrent(callable, iterable, max_workers=None)

Concurrent ~map().

Parameters
  • callable – A callable object to be executed for each item in the collection.

  • iterable – An iterable collection of items.

  • max_workers (optional) – The maximum number of worker threads allowed to be spawned when performing the work. The default is the value returned by jolt.Tools.cpu_count().

Returns

List of return values.

Return type

list

Example

def compile(self, srcfile):
    objfile = srcfile + ".o"
    tools.run("gcc -c {0} -o {1}", srcfile, objfile)
    return objfile

srcfiles = ["test.c", "main.c"]
objfiles = tools.map_concurrent(compile, srcfiles)
map_consecutive(callable, iterable)

Same as map().

meson(deps=None)

Creates a Meson invokation helper

mkdir(pathname, recursively=True)

Create directory.

read_file(pathname, binary=False)

Reads a file.

render(template, **kwargs)

Render a Jinja template string.

Parameters
  • template (str) – Jinja template string.

  • kwargs (dict) – Keywords made available to the template context. Task attributes are automatically available.

Returns

Renderered template data.

Return type

str

render_file(template, **kwargs)

Render a Jinja template file.

Parameters
  • template (str) – Filesystem path to template file.

  • kwargs (dict) – Keywords made available to the template context. Task attributes are automatically available.

Returns

Renderered template data.

Return type

str

replace_in_file(pathname, search, replace)

Replaces all occurrences of a substring in a file.

Parameters
  • pathname (str) – Name/path of file to modify.

  • search (str) – Substring to be replaced.

  • replace (str) – Replacement substring.

Example

version = Parameter(default="1.0")

def run(self, deps, tools):
    tools.replace_in_file("Makefile", "VERSION := 0.9", "VERSION := {version}")
rmtree(pathname, *args, **kwargs)

Removes a directory tree from disk.

Parameters
  • pathname (str) – Path to the file or directory to be removed.

  • ignore_errors (boolean, optional) – Ignore files that can’t be deleted. The default is False.

rsync(srcpath, dstpath, *args, **kwargs)

Synchronizes files from one directory to another.

The function performs a smart copy of files from the srcpath directory to the dstpath directory in such a way that dstpath will mirror srcpath.

If dstpath is empty, the files are copied normally.

If dstpath already contains a sub or superset of the files in srcpath, files are either copied or deleted depending on their presence in the source directory. Common files are only copied if the file content differs, thereby retaining metadata (such as timestamps) of identical files already present in dstpath.

Parameters
  • srcpath (str) – Path to source directory. The directory must exist.

  • dstpath (str) – Path to destination directory.

run(cmd, *args, **kwargs)

Runs a command in a shell interpreter.

These additional environment variables will be set when the command is run:

  • JOLTDIR - Set to Task.joltdir

  • JOLTCACHEDIR - Set to the location of the Jolt cache

A JoltCommandError exception is raised on failure.

Parameters
  • cmd (str) – Command format string.

  • args – Positional arguments for the command format string.

  • kwargs – Keyword arguments for the command format string.

  • output (boolean, optional) – By default, the executed command’s output will be written to the console. Set to False to disable all output.

  • output_on_error (boolean, optional) – If True, no output is written to the console unless the command fails. The default is False.

  • output_rstrip (boolean, optional) – By default, output written to stdout is stripped from whitespace at the end of the string. This can be disabled by setting this argument to False.

  • shell (boolean, optional) – Use a shell to run the command. Default: True.

  • timeout (int, optional) – Timeout in seconds. The command will first be terminated if the timeout expires. If the command refuses to terminate, it will be killed after an additional 10 seconds have passed. Default: None.

Example

target = Parameter(default="all")
verbose = "yes"

def run(self, deps, tools):
    tools.run("make {target} VERBOSE={verbose} JOBS={0}", tools.cpu_count())
runprefix(cmdprefix, *args, **kwargs)

Adds a command prefix to all commands executed by run().

A new prefix is appended to any existing prefix.

Parameters
  • cmdprefix (str, list) – The command prefix. The string, or list, is expanded with expand().

  • args (str, optional) – Additional positional values passed to expand().

  • kwargs (str, optional) – Additional keyword values passed to expand().

Example

with tools.runprefix("docker exec container"):
    tools.run("ls")

The above code is equivalent to:

tools.run("docker exec container ls")
sandbox(artifact, incremental=False, reflect=False)

Creates a temporary build directory populated with the contents of an artifact.

Files are copied using rsync.

Parameters
  • artifact (cache.Artifact) – A task artifact to be copied into the sandbox.

  • incremental (boolean) – If false, the created directory is deleted upon completion of the task.

  • reflect (boolean) – If true, a virtual sandbox is constructed from artifact metadata only. Files are not copied, but instead symlinks are created pointing at the origin of each file contained within the artifact. The sandbox reflects the artifact with a live view of the the current workspace.

Returns

Path to the build directory..

Return type

str

Example

def run(self, deps, tools):
   sandbox = tools.sandbox(deps["boost"], incremental=True)
setenv(key, value=None)

Sets or unset an environment variable.

Only child processes spawned by the same tools object can see the set environment variable and its value. Don’t assume the same variable is set in the Jolt process’ environment.

Parameters
  • key (str) – Name of variable to set.

  • value (str) – Value of the variable or None to unset it.

Creates a symbolic link.

Parameters
  • src (str) – Path to target file or directory.

  • dst (str) – Name/path of symbolic link.

  • replace (boolean) – Replace existing file or link. Default: false.

  • relative (boolean) – Create link using relative path to target. Default: false (absolute path).

thread_count()

Number of threads to use for a task.

Returns

number of threads to use.

Return type

int

tmpdir(name)

Creates a temporary directory.

The directory is only valid within a context and it is removed immediately upon leaving the context.

Parameters

name (str) – Name prefix for the directory. A unique autogenerated suffix will also be appended to the final name.

Example

with tools.tmpdir("temp") as tmp, tools.cwd(tmp.path):
    tools.write_file("tempfile", "tempdata")

Removes a file from disk.

To remove directories, use rmtree().

Parameters

pathname (str) – Path to the file to be removed.

unshare(uid=0, gid=0, groups=[0], uidmap=None, gidmap=None)

Experimental: Create a Linux namespace.

This method yields a new Linux namespace in which Python code may be executed. By default, the current user is mapped to root inside the namespace and all other users and groups are automatically mapped to the other user’s configured subuids and subgids.

The main use-case for namespaces is to fake the root user which may be useful in a number of situations:

  • to allow chroot() without beeing root

  • to allow mount() without beeing root

  • to preserve file ownership after tar file extraction

  • etc…

Requires a Linux host.

Note that the fork() system call is used. Changes made to variables will not persist after leaving the namespace.

Parameters
  • uid (int) – Requested uid inside the namespace. This is always mapped to the uid of the caller.

  • gid (int) – Requested gid inside the namespace. This is always mapped to the gid of the caller.

  • uidmap (list) – List of uids to map in the namespace. A list of tuples is expected: (inner uid, outer id, number of ids to map).

  • gidmap (list) – List of gids to map in the namespace. A list of tuples is expected: (inner gid, outer id, number of ids to map).

Example

with tools.unshare() as ns, ns:
    # Inside namespace
    tools.run("whoami")  # "root"
# Back outside namespace, namespace destructed
upload(pathname, url, exceptions=True, auth=None, **kwargs)

Uploads a file using HTTP (PUT).

Throws a JoltError exception on failure.

Parameters
  • pathname (str) – Name/path of file to be uploaded.

  • url (str) – Destination URL.

  • auth (requests.auth.AuthBase, optional) – Authentication helper. See requests.auth for details.

  • kwargs (optional) – Addidional keyword arguments passed on directly to ~requests.put().

which(executable)

Find executable in PATH.

Parameters

executable (str) – Name of executable to be found.

Returns

Full path to the executable.

Return type

str

write_file(pathname, content=None, expand=True, **kwargs)

Creates a file.

Note

Existing files are overwritten.

Parameters
  • pathname (str) – Name/path of file to be created.

  • content (str, optional) – Data to be written to the file.

  • expand (boolean, optional) – Expand macros in file content. Default: True.

  • **kwargs (dict, optional) – Additional key value dictionary used in macro expansion.

CMake

CMake

class jolt.plugins.cmake.CMake(parameters=None, **kwargs)

Builds and publishes a CMake project

cmakelists = 'CMakeLists.txt'

Path to CMakeLists.txt or directory containing CMakelists.txt

options = []

List of options and their values (option[:type]=value)

Conan

Conan

class jolt.plugins.conan.Conan(*args, **kwargs)

Conan package installer task.

This task base class can be used to fetch, build and publish Conan packages as Jolt artifacts. All package metadata is transfered from the Conan package manifest to the Jolt artifact so that no manual configuration of include paths, library paths, macros, etc is required.

An existing installation of Conan is required. Please visit https://conan.io/ for installation instructions and documentation.

A minimal task to download and publish the Boost C++ libraries can look like this:

from jolt.plugins.conan import Conan

class Boost(Conan):
    packages = ["boost/1.74.0"]

Boost may then be used from Ninja tasks by declaring a requirement:

from jolt.plugins.ninja import CXXExecutable

class BoostApplication(CXXExecutable):
    requires = ["boost"]
    sources = ["main.cpp"]

The task supports using an existing conanfile.txt, but it is not required. Packages are installed into and collected from Jolt build directories. The user’s regular Conan cache will not be affected.

conanfile = None

An existing conanfile.txt file to use.

Instead of generating the conanfile.txt file on-demand, an external file may be used. If this attribute is set, the generators, options and packages attributes must not be set.

See Conan documentation for further details.

generators = []

A list of Conan generators to use.

See Conan documentation for details about supported generators. The json generator is always used.

Example:

generators = ["cmake"]
incremental = True

Keep installed packages in the Conan cache between Jolt invokations.

If incremental build is disabled, the Jolt Conan cache is removed before execution begins.

options = []

A list of Conan package options to apply

The option format is PkgName:Option=Value. See Conan documentation for further details.

Any {keyword} arguments, or macros, found in the strings are automatically expanded to the value of the associated task’s parameters and properties.

Example:

options = [
    "boost:shared=True",
    "zlib:shared=True",
]
packages = []

A list of Conan package references to collect and publish.

The reference format is PkgName/<version>@user/channel. See Conan documentation for further details.

Any {keyword} arguments, or macros, found in the strings are automatically expanded to the value of the associated task’s parameters and properties.

Example:

sdl_version = Parameter("2.0.12")

packages = [
    "boost/1.74.0",
    "sdl2/{sdl_version}@bincrafters/stable",
]
remotes = {}

A dictionary with Conan remotes to use when fetching packages.

The dictionary key is the name of remote and its value is the URL.

Example:

remotes = {
    "bincrafters": "https://api.bintray.com/conan/bincrafters/public-conan"
}

Docker

DockerClient

class jolt.plugins.docker.DockerClient(parameters=None, **kwargs)

Task: Downloads and publishes the Docker command line client.

The task will be automatically made available after importing jolt.plugins.docker.

arch = <jolt.tasks.Parameter object>

Host architecture [x86_64]

host = <jolt.tasks.Parameter object>

Host operating system [autodetected]

name = 'docker/cli'

Name of the task

url = 'https://download.docker.com/{host}/static/stable/{arch}/docker-{version}.tgz'

URL of binaries

version = <jolt.tasks.Parameter object>

Docker version [20.10.13]

DockerContainer

class jolt.plugins.docker.DockerContainer(*args, **kwargs)

Resource: Starts and stops a Docker container.

arguments = []

Container argument list

cap_adds = []

A list of capabilities to add to the container

cap_drops = []

A list of capabilities to remove from the container

entrypoint = None

Container entrypoint

environment = []

Environment variables

image = None

Image tag or Jolt task.

If a Jolt task is specified, its artifact must export a metadata string named tag with the name of the image tag.

labels = []

A list of container metadata labels

ports = []

A list of container ports to publish.

Example:

ports = [
    "80",
    "443:443",
]

Alternatively, assign True to publish all exposed ports to random ports.

privileged = False

Start container with elevated privileges.

user = None

Username or UID.

Defaults to the current user.

volumes = []

A list of volumes to mount.

By default, the cache directory and joltdir are automatically mounted in the container. See volumes_default.

volumes_default = ['{joltdir}:{joltdir}', '{joltcachedir}:{joltcachedir}']

A list of default volumes to mount.

By default, the cache directory and joltdir are automatically mounted in the container. Override to disable.

DockerImage

class jolt.plugins.docker.DockerImage(*args, **kwargs)

Abstract Task: Builds and publishes a Docker image.

Builds the selected Dockerfile and optionally tags and pushes the image to a registry. The image can also be saved to file and published in the task artifact. Compression formats supported are bzip2, gzip and lzma.

By default, base images referenced in the Dockerfile will be pulled during the build. Note that Jolt has no way of knowing beforehand if images have been updated in the registry. Use time-based influence to trigger rebuilds if it’s important that base images are kept up-to-date.

No automatic influence for Dockerfile or context is collected. Make sure to use an appropriate influence decorator.

Optionally add requirements to:

  • docker/cli to provision the Docker client, if none is available on the host.

  • docker/login to automatically login to the Docker registry.

This class must be subclassed.

Example:

# Dockerfile

FROM busybox:latest
CMD ["busybox"]
# build.jolt

from jolt.plugins.docker import DockerImage

class Busybox(DockerImage):
    """ Publishes Busybox image as gzip-compressed tarball """
    compression = "gz"
    requires = ["docker/cli"]
    tags = ["busybox:{identity}"]
autoload = True

Automatically load image file into local registry when the artifact is consumed by another task.

If the built image is saved to a file (i.e. imagefile is set), the image file is automatically loaded into the local Docker registry when the task artifact is consumed by another task. The image is also automatically removed from the registry upon completion of the consumer task.

Default: True.

buildargs = []

List of build arguments and their values (“ARG=VALUE”).

The arguments are passed to Docker using --build-arg.

cleanup = True

Remove image from Docker daemon upon completion [True]

compression = None

Optional image compression “bz2”, “gz”, or “xz”.

context = '.'

Path to build context, relative to joltdir (directory).

dockerfile = 'Dockerfile'

Path to the Dockerfile to build, or the full source code of such a file.

extract = False

Extract image and publish rootfs tree.

This option is useful when building a chroot to be used with jolt.Tools.chroot(). It disables saving of the image to a tarball.

imagefile = '{canonical_name}.tar'

Name of the image tarball published by the task.

If set to None, no image file will be saved and published.

Defaults to the task’s canonical name.

labels = []

A list of image metadata labels

platform = None

Target platform, e.g. linux/arm/v7.

pull = True

Always pull images when building.

Passes –pull to the Docker client.

push = False

Optionally push image to registry [False]

To be able to push images, the current user must login to the Docker Registry. The docker/login Jolt resource can be used for that purpose.

squash = False

Squash image layers

tags = ['{canonical_name}:{identity}']

Optional list of image tags. Defaults to task’s canonical name.

DockerLogin

class jolt.plugins.docker.DockerLogin(*args, **kwargs)

Resource: Logs in and out of a Docker Registry.

If the user and password parameters are unset, credentials are fetched from the environment variables:

  • DOCKER_USER

  • DOCKER_PASSWD

The resource will be automatically made available after importing jolt.plugins.docker.

name = 'docker/login'

Name of the resource

passwd = <jolt.tasks.Parameter object>

Docker Registry password.

If not set, the environment variable DOCKER_PASSWD is read instead.

user = <jolt.tasks.Parameter object>

Docker Registry username.

If not set, the environment variable DOCKER_USER is read instead.

Metadata

The Docker module registers and makes available these artifact metadata attributes:

  • artifact.docker.load - List of image files to be loaded from the artifact into the local Docker registry when the artifact is consumed.

    Example:

    def publish(self, artifact, tools):
        artifact.docker.load.append("image.tar")
    
  • artifact.docker.pull - List of image tags to be pulled from a remote registry to the local Docker registry when the artifact is consumed.

    Example:

    def publish(self, artifact, tools):
        artifact.docker.pull.append("busybox:latest")
    
  • artifact.docker.rmi - List of image tags to remove from the local Docker registry when a consuming task has finished.

    Example:

    def publish(self, artifact, tools):
        artifact.docker.rmi.append("busybox:latest")
    

Ninja

CXXExecutable

class jolt.plugins.ninja.CXXExecutable(*args, **kwargs)

Builds a C/C++ executable.

The task recognizes these source file types: .asm, .c, .cc, .cpp, .cxx, .h, .hh, .hpp, .hxx, .pch, .s, .S

Other file types can be supported through additional rules, see the Rule class.

On Linux, GCC/Binutils is the default toolchain used. The default toolchain can be overridden by setting the environment variables AR, CC, CXX and LD. The prefered method is to assign these variables through the artifact of a special task that you depend on.

On Windows, Visual Studio is the default toolchain and it must be present in the PATH. Run Jolt from a developer command prompt.

Additionally, these environment variables can be used to customize toolchain behavior on any platform:

  • ASFLAGS - compiler flags used for assembly code

  • CFLAGS - compiler flags used for C code

  • CXXFLAGS - compiler flags used for C++ code

  • LDFLAGS - linker flags

asflags = []

A list of compiler flags used when compiling assembler files.

binary = None

Name of the target binary (defaults to canonical task name)

cflags = []

A list of compiler flags used when compiling C files.

cxxflags = []

A list of compiler flags used when compiling C++ files.

incpaths = []

List of preprocessor include paths

incremental = True

Compile incrementally.

If incremental build is disabled, all intermediate files from a previous build will be removed before the execution begins.

ldflags = []

A list of linker flags to use.

libpaths = []

A list of library search paths used when linking.

libraries = []

A list of libraries to link with.

macros = []

List of preprocessor macros to set

publishdir = 'bin/'

The artifact path where the binary is published.

publish(artifact, tools)

Publishes the linked executable.

By default, the executable is collected into a directory as specified by the publishdir class attribute. The relative path of the executable within the artifact is exported as a metadata string. It can be read by consumers by accessing artifact.strings.executable.

The method appends the PATH environment variable with the path to the executable to allow consumers to run it easily.

run(deps, tools)

Generates a Ninja build file and invokes Ninja to build the project.

The build file and all intermediate files are written to a build directory within the workspace. By default, the directory persists between different invokations of Jolt to allow projects to be built incrementally. The behavior can be changed with the incremental class attribute.

selfsustained = True

Consume this executable independently from its requirements.

When self-sustained, all shared libraries listed as requirements are published toghether with the executable.

See Task.selfsustained for general information.

shell(deps, tools)

Invoked to start a debug shell.

The method prepares the environment with attributes exported by task requirement artifacts. The shell is entered by passing the -g flag to the build command.

For Ninja tasks, a special compile command is made available inside the shell. The command can be used to compile individual source files which is useful when troubleshooting compilation errors. Run compile -h for help.

Task execution resumes normally when exiting the shell.

sources = []

A list of sources to compile.

Path names may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

Example

sources = ["src/*.cpp"]
source_influence = True

Let the contents of source files influence the identity of the task artifact.

When True, a source file listed in the sources attribute will cause a rebuild of the task if modified.

Source influence can hurt performance since every files needs to be hashed. It is safe to set this flag to False if all source files reside in a git repository listed as a dependency with the requires attribute or if the task uses the git.influence decorator.

Always use source_influence if you are unsure whether it is needed or not.

strip = True

Remove debug information from binary.

When using the GNU toolchain, debug information is kept in a separate binary which is either published or not depending on the value of this attribute. It’s found in a .debug directory if present.

Only applicable to shared libraries.

CXXLibrary

class jolt.plugins.ninja.CXXLibrary(*args, **kwargs)

Builds a C/C++ library.

The task recognizes these source file types: .asm, .c, .cc, .cpp, .cxx, .h, .hh, .hpp, .hxx, .pch, .s, .S

Other file types can be supported through additional rules, see the Rule class.

On Linux, GCC/Binutils is the default toolchain used. The default toolchain can be overridden by setting the environment variables AR, CC, CXX and LD. The prefered method is to assign these variables through the artifact of a special task that you depend on.

On Windows, Visual Studio is the default toolchain and it must be present in the PATH. Run Jolt from a developer command prompt.

Additionally, these environment variables can be used to customize toolchain behavior on any platform:

  • ASFLAGS - compiler flags used for assembly code

  • CFLAGS - compiler flags used for C code

  • CXXFLAGS - compiler flags used for C++ code

  • LDFLAGS - linker flags

asflags = []

A list of compiler flags used when compiling assembler files.

binary = None

Name of the target binary (defaults to canonical task name)

cflags = []

A list of compiler flags used when compiling C files.

cxxflags = []

A list of compiler flags used when compiling C++ files.

headers = []

List of public headers to be published with the artifact

incpaths = []

List of preprocessor include paths

incremental = True

Compile incrementally.

If incremental build is disabled, all intermediate files from a previous build will be removed before the execution begins.

ldflags = []

A list of linker flags to use.

libpaths = []

A list of library search paths used when linking.

libraries = []

A list of libraries to link with.

macros = []

List of preprocessor macros to set

run(deps, tools)

Generates a Ninja build file and invokes Ninja to build the project.

The build file and all intermediate files are written to a build directory within the workspace. By default, the directory persists between different invokations of Jolt to allow projects to be built incrementally. The behavior can be changed with the incremental class attribute.

publishapi = 'include/'

The artifact path where public headers are published.

publishdir = 'lib/'

The artifact path where the library is published.

publish(artifact, tools)

Publishes the library.

By default, the library is collected into a directory as specified by the publishdir class attribute. Library path metadata for this directory as well as linking metadata is automatically exported. The relative path of the library within the artifact is also exported as a metadata string. It can be read by consumers by accessing artifact.strings.library.

Public headers listed in the headers class attribute are collected into a directory as specified by the publishapi class attribute. Include path metadata for this directory is automatically exported.

selfsustained = False

Consume this library independently from its requirements.

When self-sustained, all static libraries listed as requirements are merged into the final library. Merging can also be achieved by listing libraries as source files.

See Task.selfsustained for general information.

shell(deps, tools)

Invoked to start a debug shell.

The method prepares the environment with attributes exported by task requirement artifacts. The shell is entered by passing the -g flag to the build command.

For Ninja tasks, a special compile command is made available inside the shell. The command can be used to compile individual source files which is useful when troubleshooting compilation errors. Run compile -h for help.

Task execution resumes normally when exiting the shell.

sources = []

A list of sources to compile.

Path names may contain simple shell-style wildcards such as ‘*’ and ‘?’. Note: files starting with a dot are not matched by these wildcards.

Example

sources = ["src/*.cpp"]
source_influence = True

Let the contents of source files influence the identity of the task artifact.

When True, a source file listed in the sources attribute will cause a rebuild of the task if modified.

Source influence can hurt performance since every files needs to be hashed. It is safe to set this flag to False if all source files reside in a git repository listed as a dependency with the requires attribute or if the task uses the git.influence decorator.

Always use source_influence if you are unsure whether it is needed or not.

strip = True

Remove debug information from binary.

When using the GNU toolchain, debug information is kept in a separate binary which is either published or not depending on the value of this attribute. It’s found in a .debug directory if present.

Only applicable to shared libraries.

Decorators

class jolt.plugins.ninja.attributes
static asflags(attrib, prepend=False)

Decorates a task with an alternative asflags attribute.

The new attribute will be concatenated with the regular asflags attribute.

Parameters
  • attrib (str) – Name of alternative attribute. Keywords are expanded.

  • prepend (boolean) – Prepend the value of the alternative attribute. Default: false (append).

static cflags(attrib, prepend=False)

Decorates a task with an alternative cflags attribute.

The new attribute will be concatenated with the regular asflags attribute.

Parameters
  • attrib (str) – Name of alternative attribute. Keywords are expanded.

  • prepend (boolean) – Prepend the value of the alternative attribute. Default: false (append).

static cxxflags(attrib, prepend=False)

Decorates a task with an alternative cxxflags attribute.

The new attribute will be concatenated with the regular cxxflags attribute.

Parameters
  • attrib (str) – Name of alternative attribute. Keywords are expanded.

  • prepend (boolean) – Prepend the value of the alternative attribute. Default: false (append).

static incpaths(attrib, prepend=False)

Decorates a task with an alternative incpaths attribute.

The new attribute will be concatenated with the regular incpaths attribute.

Parameters
  • attrib (str) – Name of alternative attribute. Keywords are expanded.

  • prepend (boolean) – Prepend the value of the alternative attribute. Default: false (append).

static ldflags(attrib, prepend=False)

Decorates a task with an alternative ldflags attribute.

The new attribute will be concatenated with the regular ldflags attribute.

Parameters
  • attrib (str) – Name of alternative attribute. Keywords are expanded.

  • prepend (boolean) – Prepend the value of the alternative attribute. Default: false (append).

static libpaths(attrib, prepend=False)

Decorates a task with an alternative libpaths attribute.

The new attribute will be concatenated with the regular libpaths attribute.

Parameters
  • attrib (str) – Name of alternative attribute. Keywords are expanded.

  • prepend (boolean) – Prepend the value of the alternative attribute. Default: false (append).

static libraries(attrib, prepend=False)

Decorates a task with an alternative libraries attribute.

The new attribute will be concatenated with the regular libraries attribute.

Parameters
  • attrib (str) – Name of alternative attribute. Keywords are expanded.

  • prepend (boolean) – Prepend the value of the alternative attribute. Default: false (append).

static macros(attrib, prepend=False)

Decorates a task with an alternative macros attribute.

The new attribute will be concatenated with the regular macros attribute.

Parameters
  • attrib (str) – Name of alternative attribute. Keywords are expanded.

  • prepend (boolean) – Prepend the value of the alternative attribute. Default: false (append).

static sources(attrib, prepend=False)

Decorates a task with an alternative sources attribute.

The new attribute will be concatenated with the regular sources attribute.

Parameters
  • attrib (str) – Name of alternative attribute. Keywords are expanded.

  • prepend (boolean) – Prepend the value of the alternative attribute. Default: false (append).

Rule

class jolt.plugins.ninja.Rule(command=None, infiles=None, outfiles=None, depfile=None, deps=None, variables=None, implicit=None, order_only=None, aggregate=False, phony=None)

A source transformation rule.

Rules are used to transform files from one type to another. An example is the rule that compiles a C/C++ file to an object file. Ninja tasks can be extended with additional rules beyond those already builtin and the builtin rules may also be overridden.

To define a new rule for a type of file, assign a Rule object to an arbitrary attribute of the compilation task being defined. Below is an example where a rule has been created to generate Qt moc source files from headers.

class MyQtProject(CXXExecutable):
    moc_rule = Rule(
        command="moc -o $out $in",
        infiles=[".h"],
        outfiles=["{outdir}/{in_path}/{in_base}_moc.cpp"])

    sources = ["myqtproject.h", "myqtproject.cpp"]

The moc rule will be run on all .h header files listed as sources, i.e. myqtproject.h. It takes the input header file and generates a corresponding moc source file, myqtproject_moc.cpp. The moc source file will then automatically be fed to the builtin compiler rule from which the output is an object file, myqtproject_moc.o.

__init__(command=None, infiles=None, outfiles=None, depfile=None, deps=None, variables=None, implicit=None, order_only=None, aggregate=False, phony=None)

Creates a new rule.

Parameters
  • command (str, optional) – The command that will be execute by the rule. It can use any of the variables created below.

  • infiles (str, optional) – A list of file extensions that the rule should apply to.

  • outfiles (str, optional) – A list of files created by the rule. Regular keyword expansion is done on the strings but additional keywords are supported, see variables below.

  • variables (str, optional) –

    A dictionary of variables that should be available to Ninja when running the command. By default, only $in and $out will be set, where $in is a single input file and $out is the output file(s). Regular keyword expansion is done on the value strings, see jolt.Tools.expand(). These additional keywords are supported:

    • in_path - the path to the directory where the input file is located

    • in_base - the name of the input file, excluding file extension

    • in_ext - the input file extension

    Example

    Rule(command="echo $extension", variables={"extension": "{in_ext}"}, ...)
    

  • aggregate (boolean, optional) –

    When this attribute is set, the Rule will aggregate all input

    files and transform them with a single command. This is useful, for example, when creating linking and archiving rules. In aggregating rules the $in Ninja variable expands to all matched input files, while the outfiles / $out variable is expanded using the first input in the set, if the in_* keywords are used at all.

    By default, a rule is applied once for each matched input file for improved parallelism.

    Example:

    In this example, the rule concatenates all header files into a single precompiled header.

    pch = Rule(
       command="cat $in > $out",
       infiles=["*.h"],
       outfiles=["{outdir}/all.pch"],
       aggregate=True)
    
    phony (boolean, optional):

    Emit a phony build target depending on all files generated by this rule. The phony build target becomes an implicit dependency in downstream rules that pick up the output from this rule.

    An example use-case is the compilation of protobuffers into C++ headers and source files. The C++ header files must be available before they can be included from other static C++ source files. With the phony target enabled, all protobuffers are compiled before any C++ file is compiled.