Configuration
Client
By default, the Jolt client loads its configuration from
$HOME/.config/jolt/config
on Linux and
%APPDATA%\Roaming\Jolt\config
on Windows.
It uses this format:
[section]
key = value
The configuration can be edited manually using any editor, but
the recommended way to change configuration is to use the Jolt
config
command. For example:
$ jolt config jolt.cachesize 100G
Available sections and their respective keys are detailed below.
The [jolt]
config section contains global configuration:
Config Key
Type
Description
cachedir
String
Filesystem path to a directory where the Jolt artifact cache will reside.Default:
Linux:
~/.cache/jolt
Windows:
%LOCALAPPDATA%/Jolt
cachesize
String
Maximum size of the local artifact cache. When this size is reached, Jolt will start to evict artifacts from the cache until it no longer exceeds the configured limit. Artifacts which are required to execute a task currently present in the dependency tree are never evicted. Therefore, it may not be possible for Jolt to evict enough artifacts to reach the limit. Consider this size advisory. The size is specified in bytes and SI suffixes such as K, M and G are supported.Default:1G
colors
Boolean
Colorize output. When enabled, Jolt uses colors to make it easier to read the console log and to spot errors.Default:true
default
String
When invoked without any arguments, Jolt by default tries to build a task called
default
. The name of the default task can be overridden by setting this configuration key.
download
Boolean
Enable Jolt to download artifacts from remote storage providers when running tasks locally. The option has no effect on distributed executions.Default:true
logcount
Integer
Number of log files to keep.Default:100
logpath
String
Location of Jolt’s log files. | Default:
Linux:
$HOME/.jolt
Windows:
%LOCALAPPDATA%/Jolt
upload
Boolean
Configures if Jolt is allowed to upload artifacts to remote storage providers or not when building locally. The option has no effect on distributed network builds.Default:true
pager
String
The pager to use, e.g. when viewing the logfile. Defaults to the
PAGER
environment variable followed byless
,more
andcat
, in that order.
pluginpath
String
A list of one or more directory names, separated by colon, specifying additional search paths for plugins.
shell
String
The shell to use when entering the interactive task debug shell.
threads
Integer
Used to limit the number of threads used by third party tools such as Ninja. The environment variableJOLT_THREADS
can also be used.The default value is the number of CPUs available.
Alias
This plugin can be used to create user-defined task aliases
through configuration keys. An alias points to one or many
other tasks. For example, to create an alias called deploy
which deploys a fictitious smartphone app to all supported devices,
run:
$ jolt config alias.deploy "deploy/android deploy/iphone"
$ jolt build deploy
Alternatively, edit the configuration manually:
[alias]
deploy = deploy/android deploy/iphone
Aliases cannot be used to override the names of tasks loaded from recipes.
Allure
This plugin generates an Allure test report on the outcome of executed tasks. The report includes:
status of tasks, i.e. successful, failed or skipped
duration of tasks
hostname of executor
logs
The plugin is enabled by adding a [allure]
section in
the Jolt configuration. Its dependencies must also be
installed separately:
$ pip install jolt[allure]
Available configuration keys:
Config Key
Type
Description
loglevel
String
The level of detail to include in task logs:INFO
,VERBOSE
orDEBUG
.Default:INFO
path
String
Path to directory where result files are written.Default:<workspace>/allure-results
Autoweight
The autoweight plugin automatically collects statistics about task execution times. The data is used to assign weights to task, allowing the Jolt scheduler to favor tasks along the critical path. This improves overall execution time in a distributed execution configuration where many tasks are executed in parallel.
The plugin is enabled by adding an [autoweight]
section in
the Jolt configuration.
These configuration keys exist:
Config Key
Type
Description
samples
Integer
The number of execution time samples to store per task in the database. Once the number is exceeded, samples are evicted in FIFO order.Default:10
Cache
The [cache]
section configures a remote artifact cache. The cache
is used to store artifacts that are built by Jolt. When a task is built,
Jolt will first check the cache to see if the artifact is already present.
If it is, the artifact is downloaded and used. If not, the artifact is
built and then uploaded to the cache so that it can be shared with others.
Available configuration keys:
Config Key
Type
Description
uri
String
The URI of the remote artifact cache. The targeted service is expected to implement the default Jolt cache REST API.Default:http://cache
Configuration variables for the cache service itself can be found here: Cache
Dashboard
The dashboard plugin automatically submits required telemetry to the Jolt Dashboard. It should be enabled on both clients and workers.
The plugin is enabled by adding a [dashboard]
section in
the Jolt configuration.
These configuration keys exist:
Config Key
Description
uri
Base URI of the Jolt Dashboard.Default: http://dashboard
Email
The email plugin sends an HTML email report to configured recipients when builds have completed. The email includes a list of interpreted errors in case of failure.
The plugin is enabled by adding a [email]
section in
the Jolt configuration.
These configuration keys exist:
Config Key
Type
Description
server
String
SMTP server used to send emails.
from
String
Sender email address.
to
String
Receiver email address. May also be read from environment, e.g.
{environ[GERRIT_PATCHSET_UPLOADER_EMAIL]}
. Multiple addresses should be separated by a single space.
cc
String
Carbon copy recipients.
bcc
String
Blind carbon copy recipients.
stylesheet
String
An optional custom XSLT stylesheet used to transform the Jolt result manifest into an HTML email.
on_success
Boolean
Send emails when builds are successful.Default:true
on_failure
Boolean
Send emails when builds failed.Default:true
GDB
The GDB plugin enables a new command, gdb
. When invoked, the command
launches GDB with an executable from the specified task’s artifact. It
automatically configures the GDB sysroot based on environment variables
set in the execution environment of the task.
The plugin is enabled by adding a [gdb]
section in
the Jolt configuration. No additional dependencies have to be installed.
HTTP
The HTTP plugin implements an artifact storage provider. When used, artifacts can be automatically uploaded to and downloaded from a configured HTTP server when tasks are executed.
This is useful in many situations, for example:
To support distributed task execution. Task executors must be able to share artifacts between each other. Using a networked storage provider is an easy way to meet that requirement.
To reduce execution time by letting multiple users share the same artifact cache. If one user has already executed a task, its artifact is simply downloaded to others who attempt execution.
To reduce the amount of disk space required locally. Jolt can be configured to evict artifacts more aggressively from the local cache. Artifacts will still be available on the server if needed.
The HTTP plugin is enabled by adding an [http]
section in
the Jolt configuration.
These configuration keys exist:
Config Key
Type
Description
download
Boolean
Allow/disallow artifacts to be downloaded from the HTTP server.Default:true
upload
Boolean
Allow/disallow artifacts to be uploaded to the HTTP server.Default:true
uri
String
URL to the HTTP server.Default:http://cache
keyring.service
String
Keyring service identifier. Currently, only basic authentication is supported. Authentication is disabled if left unset.
keyring.username
String
Username to use when authenticating with the HTTP server.
keyring.password
String
Password to use when authenticating with the HTTP server. Should normally never need to be set in the configuration file. By default, Jolt asks for the password when needed and stores it in a keyring for future use.
Logstash (HTTP)
The logstash plugin is used to collect task logs into a common place. This is useful in distributed execution environments where detailed logs may not always be immediately accessible to ordinary users. Unlike the terminal log output, stashed logs are always unfiltered and include statements from all log levels as well as exception callstacks.
The plugin is enabled by adding a [logstash]
section in
the Jolt configuration.
These configuration keys exist:
Config Key
Type
Description
http.uri
String
An HTTP URL where logs will be stashed. TheHTTP PUT
method is used.Default:http://logstash
failed
Boolean
Stash logs when tasks fail.Default:false
passed
Boolean
Stash logs when tasks pass and finish successfully.Default:false
Network
The [network]
section contains keys applicable when Jolt is started
in network execution mode.
Config Key
Type
Description
config
String
The
config
key contains config file content for Jolt to be used when Jolt is executed on a different machine during distributed execution. The configuration is automatically passed to the remote worker and may contain all subsections and keys detailed in this document. Lines must be properly indented for the key to be considered multiline. Example:[network] config = [jolt] upload = true download = true
Ninja Compilation Database
This plugin enables compilation database generation for Ninja C++
tasks. The database is automatically published in task artifacts.
Note that commands are recorded exactly as invoked by Ninja and
they are therefore not immediately usable because of how Jolt
sandboxes dependencies. A special command, compdb
is made
available to post-process published databases into a database that
is usable with IDEs. The command takes an already built task as
argument:
$ jolt compdb <task>
Upon completion, a path to the resulting database is printed. The database aggregates the databases of the task and all its dependencies.
The plugin is enabled by adding a [ninja-compdb]
section in
the Jolt configuration. Ninja version >= 1.10.0 is required.
Scheduler
The [scheduler]
section configures remote task scheduling.
A remote scheduler accepts task execution requests from the Jolt client
and distributes them to workers. Logs, artifacts and results are collected
from the workers and returned to the client in real-time
Tasks can be assigned a priority. The scheduler will always attempt to execute tasks with the highest priority first, if there is an eligible worker available. If no worker is available, the task is queued until one becomes available. The scheduler will also attempt to execute tasks in the order they were submitted, but this is not guaranteed. In some cases, competing builds with fewer remaining tasks may be prioritized.
Available configuration keys:
Config Key
Type
Description
uri
String
The URI of the scheduler service.Default:tcp://scheduler:9090
Configuration variables for the scheduler service itself can be found here: Scheduler
Selfdeploy
The Selfdeploy plugin automatically deploys the running version of Jolt to all workers in a distrubuted execution environment. This is useful to ensure that the same version of Jolt and its dependencies are used everywhere when tasks are executed.
Before starting execution of a task, a network executor will download Jolt from the configured storage provider and install it into a virtual environment. Multiple versions can co-exist on workers, thus avoiding manual deployment of multiple container images in clusters.
The plugin is enabled by adding a [selfdeploy]
section in
the Jolt configuration. Note that pip
must be installed.
These configuration keys exist:
Config Key
Type
Description
extra
String
Comma separated list of paths to additional python modules to be deployed. The paths should be relative to the workspace root.
Once enabled, the plugin automatically passes two build environment parameters to the scheduler:
Config Key
Type
Description
jolt_url
String
A URL to a compressed tarball with the sources of the running Jolt version.
jolt_identity
String
The identity of the Jolt artifact.
jolt_requires
String
A list of additional Python modules to install on the executor.
Symlinks
The symlink plugin automatically creates symlinks to task artifacts
in the jolt workspace (relative to the topmost .jolt
file). The
symlinks are kept updated and always points to the latest built
artifact.
The plugin is enabled by adding a [symlinks]
section in
the Jolt configuration.
These configuration keys exist:
Config Key
Type
Description
path
String
Path, relative to the workspace root, where symlinks will be created.Default:artifacts
.
Telemetry
The telemtry plugin posts task telemetry to a configured HTTP endpoint. The payload is a JSON object with these fields:
Field
Type
Description
name
String
The name of the task.
identity
String
The identity of the task artifact.
instance
String
A UUID representing the lifecycle of the task. Tasks can be executed multiple times with the same identity, for example if the first execution attempt failed and a subsequent attempt succeeded. The instance ID may be used to distingush between such attempts.
hostname
String
Hostname of the machine from which the telemetry record originated.
role
String
client
orworker
depending on where the record originated.
event
String
queued
,started
,failed
orfinished
.
The plugin is enabled by adding a [telemetry]
section in
the Jolt configuration.
These configuration keys exist:
Config Key
Type
Description
uri
String
Where telemetry records should be posted.
local
Boolean
Submit telemetry for locally executed tasks.Default:true
.
network
Boolean
Submit telemetry for tasks executed by a network worker.Default:true
.
queued
Boolean
Enable queued event.Default:true
.
started
Boolean
Enable started event.Default:true
.
failed
Boolean
Enable failed event.Default:true
.
finished
Boolean
Enable finished event.Default:true
.
Services
All Jolt services can be deployed using container images. The following sections detail how to configure the services using environment variables and/or configuration files.
Cache
The cache service is used to store artifacts that are built by Jolt. The service implements an LRU cache and will evict artifacts when the cache exceeds a configured size. The cache is accessed using a REST API over HTTP(S).
Its container image is available at robrt/jolt-cache
The following volume mount points exist:
Volume Path
Description
/data
The default directory path where artifact files are stored.
The cache service can be configured using environment variables and/or a configuration file at /etc/jolt/cache.yaml
.
Environment Variable
Config File Key
Type
Description
JOLT_CACHE_CERT
cert
String
The path to the server certificate file to use if HTTPS is enabled.
JOLT_CACHE_CERT_KEY
cert_key
String
The path to the server certificate private key file to use if HTTPS is enabled.
JOLT_CACHE_INSECURE
insecure
Boolean
If set totrue
, the cache will not use HTTPS, even if a certificate and key are provided.Default:false
JOLT_CACHE_LISTEN_HTTP
listen_http
String
The address and port on which the cache will listen for HTTP(S) requests.The default is:8080
for HTTP and:8443
for HTTPS.
JOLT_CACHE_MAX_SIZE
max_size
String
The maximum size of the cache in bytes. This is a soft limit and the cache may exceed this size temporarily. The cache will start to evict artifacts when it exceeds this size.Default:10GiB
JOLT_CACHE_PATH
cache_path
String
The path to the cache directory.Default:/data
JOLT_CACHE_VERBOSITY
verbosity
Integer
The verbosity level of the cache. Higher value enables more detailed logs.Default:0
Dashboard
The dashboard service is used to collect and display task telemetry data from the Jolt scheduler.
Its container image is available at robrt/jolt-dashboard. No configuration is currently possible.
Scheduler
The scheduler service is used to distribute tasks from clients to workers. Its container image is available at robrt/jolt-scheduler.
The scheduler can be configured using environment variables and/or a configuration file at /etc/jolt/scheduler.yaml
.
Environment Variable
Config File Key
Description
JOLT_CACHE_URI
cache_uri
The URI of the HTTP cache service from which the scheduler may fetch Jolt clients. Normally, this is not used and the scheduler instead installs the same version of the client from the public Python package index. However, for development purposes it is possible to deploy the source of the running client to the cache and have the scheduler fetch it from there.The format is<scheme>://<host>:<port>
where accepted schemes are:
tcp
for both IPv4 and IPv6 connections
tcp4
for only IPv4 connections
tcp6
for only IPv6 connectionsThe default istcp://cache.
.
JOLT_CACHE_SIZE
cache_size
The maximum size of the local cache in bytes.The default is1000000000
(1 GB).
JOLT_CACHE_PATH
cache_path
The path to the local cache directory.The default is/var/cache/jolt
.
Worker
The worker service is used to execute tasks. Its container image is available at robrt/jolt-worker.
The following volume mount points exist:
Volume Path
Description
/etc/jolt/worker.yaml
The configuration file for the worker.A configuration file may be used instead of environment variables. It uses the same key names as the environment variables, but without theJOLT_
prefix and with lowercase letters.
/data/cache
The directory where the local Jolt artifact cache is kept.The cache may be shared between multiple workers on the same node.
/data/ws
The working directory where tasks are executed.This is where source code and intermediate build files are stored. The working directory is unique to each worker and should not be shared between workers.It is recommended to use a fast SSD for the working directory.
$HOME/.config/jolt/config
The configuration file for the Jolt client that executes tasks on the worker as instructed by the scheduler.See Configuration for details.
The worker can be configured using environment variables and/or a configuration file at /etc/jolt/worker.yaml
.
Environment Variable
Config File Key
Description
JOLT_CACHE_URI
cache_uri
The URI of the HTTP cache service from which the worker may fetch Jolt clients. Normally, this is not used and the worker instead installs the same version of the client from the public Python package index. However, for development purposes it is possible to deploy the source of the running client to the cache and have the worker fetch it from there.The format is<scheme>://<host>:<port>
where accepted schemes are:
tcp
for both IPv4 and IPv6 connections
tcp4
for only IPv4 connections
tcp6
for only IPv6 connectionsThe default istcp://cache.
.
JOLT_PLATFORM
platform
A list of worker properties that tasks may specify in order to run on the worker.The properties are used by the scheduler to select tasks that are compatible with the worker. For example, a task may require a worker with a specific operating system or CPU architecture.The format is<key>=<value>
where the key is the name of the property and the value is its value. Multiple properties can be specified by separating them with a comma or space.A set of default properties are always advertised:
Key
Value
node.os
The name of the operating system, e.g.
linux
,windows
.
node.arch
The name of the CPU architecture, e.g.
amd64
,arm
.
node.cpus
The number of CPUs.
node.id
A unique identifier for the server on which the worker is running.
worker.hostname
The hostname of the worker.
The recommandation is to uselabel
for functional properties, for examplelabel=compilation,label=testing
.
JOLT_TASK_PLATFORM
task_platform
A list of task properties that are required for tasks to run on the worker.For example, the worker may reject tasks that do not have the platform property “label=fast”.The format is<key>=<value>
where the key is the name of the property and the value is its value. Multiple properties can be specified by separating them with a comma or space.The recommandation is to uselabel
for functional properties.
JOLT_SCHEDULER_URI
scheduler_uri
The URIs of the scheduler to which the worker will connect and enlist.SeeJOLT_CACHE_URI
for format. The default istcp://scheduler.:9090
.
Example configuration:
# /etc/jolt/worker.yaml cache_uri: "tcp://cache:80" platform: - "label=compilation" - "label=testing" scheduler_uri: "tcp://scheduler:9090"