Tutorial
Let’s kick off with a few examples, getting more into details as we move along.
First example
Our first example is a classic. Copy/paste the code below into a file called
first_example.jolt
.
from jolt import Task
class HelloWorld(Task):
def run(self, deps, tools):
print("Hello world!")
This task will simply print “Hello world!” on the console when executed.
The task name is automatically derived from the class name, but can
be overridden by setting the name
class attribute.
Now try to execute the task:
$ jolt build helloworld
Once the task has been executed, executing it again won’t have any effect. This is because the task produced an empty artifact which is now stored in the local artifact cache. When Jolt determines if a task should be executed or not, it first calculates a task identity by hashing different attributes that would influence the output of the task, such as:
Source code
Name
Parameters
Dependencies
When the task identity is known, Jolt searches its artifact cache for an
artifact with the same identity. If one is found, no action is taken. You can
try this out by changing the Hello world!
string to something else and
executing the task again. If the string is reverted back to Hello world!
, the
task will regain its first identity and no action will be taken because that
artifact is still present in the cache.
To clean the local cache and remove all artifacts, run:
$ jolt clean
To selectively remove a specific task’s artifacts, run:
$ jolt clean helloworld
Publishing Files
Tasks that don’t produce output are not very useful. Let’s rework our task
to instead produce a file with the Hello world!
message. We also shorten
its name to hello
.
from jolt import *
class HelloWorld(Task):
""" Creates a text file with cheerful message """
name = "hello"
def run(self, deps, tools):
with tools.cwd(tools.builddir()):
tools.write_file("message.txt", "Hello world!")
def publish(self, artifact, tools):
with tools.cwd(tools.builddir()):
artifact.collect("*.txt")
The implementation of the task is now split into two methods,
run
and publish
.
The run
method performs the main work of the task. It creates
a file called message.txt
containing our greeting from the first example.
The file is written into a temporary build directory that will persist for
the duration of the task’s execution. The directory is removed afterwards.
The publish
method collects the output from the work performed by run
.
It does so by instructing the artifact to collect all textfiles from the build
directory.
$ jolt build hello
After executing the task an artifact will be present in the local cache. Let’s investigate its contents, but first we need to know the identity of the task in order to know what artifact to look for. Run:
$ jolt inspect -a hello
The inspect
command displays information about the task, including the
documentation written in its Python class implementation. We’re looking for the
identity:
Identity 50a215905eb28a0911ff83828ac56b542525bce4
With this identity digest at hand, we can dive into the artifact cache.
By default, the cache is located in $HOME/.cache/jolt
. To list the
content of the current hello
artifact, run:
$ ls $HOME/.cache/jolt/hello/50a215905eb28a0911ff83828ac56b542525bce4
You will see the message.txt
file just created.
Parameters
Next, we’re going to use a task parameter to alter the Hello world!
message. Instead of greeting the world, we’ll allow the executor to specify
an alternative recipient. We rename the class to reflect this change and
we also add a parameter class attribute. The run
method is changed to
use the new parameter’s value when writing the message.txt
file.
from jolt import *
class Hello(Task):
""" Creates a text file with a cheerful message """
recipient = Parameter(default="world", help="Name of greeting recipient.")
def run(self, deps, tools):
with tools.cwd(tools.builddir()):
tools.write_file("message.txt", "Hello {recipient}!")
def publish(self, artifact, tools):
with tools.cwd(tools.builddir()):
artifact.collect("*.txt")
By default, the produced message will still read Hello world!
because the
default value of the recipient
parameter is world
. To produce a different
message, try this:
$ jolt build hello:recipient=John
Dependencies
To better illustrate the flexibility of the new parameterized task, let’s add
another task class, Print
, which prints the contents of the message.txt
file to the console. Print
will declare a dependency on Hello
.
from jolt import *
class Hello(Task):
""" Creates a text file with a cheerful message """
recipient = Parameter(default="world", help="Name of greeting recipient.")
def run(self, deps, tools):
with tools.cwd(tools.builddir()):
tools.write_file("message.txt", "Hello {recipient}!")
def publish(self, artifact, tools):
with tools.cwd(tools.builddir()):
artifact.collect("*.txt")
class Print(Task):
""" Prints a cheerful message """
recipient = Parameter(default="world", help="Name of greeting recipient.")
requires = ["hello:recipient={recipient}"]
cacheable = False
def run(self, deps, tools):
hello = deps["hello:recipient={recipient}"]
with tools.cwd(hello.path):
print(tools.read_file("message.txt"))
The output from this task is not cacheable
, forcing the task to be
executed every time. It’s dependency hello
however, will only be
re-executed if its influence changes, for example by passing new values
to the recipient
parameter. Try it out:
$ jolt build print:recipient=John
$ jolt build print:recipient=Lisa
$ jolt build print:recipient=Kelly
Tools
The run
and publish
methods take a tools
argument as their
last parameter. This toolbox provides a large set of tools useful for many
different types of tasks. See the reference documentation for more information.
However, Jolt was originally created with compilation tasks in mind. Below is
a real world example of a task compiling the e2fsprogs
package containing
EXT2/3/4 filesystem utility programs. It uses AutoTools to configure and
build its sources into different binary applications. Luckily, the tools
object
provides utilities for building autotools projects as seen below.
In addition to AutoTools, there is also support for CMake and Meson as well as
generic support for running any third-party build tool.
from jolt import *
from jolt.plugins import git
class E2fsprogs(Task):
""" Ext 2/3/4 filesystem utilities """
requires = ["git:url=git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git"]
def run(self, deps, tools):
ac = tools.autotools()
ac.configure("e2fsprogs")
ac.build()
ac.install()
def publish(self, artifact, tools):
ac = tools.autotools()
ac.publish(artifact)
artifact.environ.PATH.append("bin")
The autotools ac
object automatically creates temporary build and install
(–prefix) directories which are used when configuring, building and installing
the project. All files installed in the installation directory will be published.
Both directories are removed when execution has finished, i.e. the project
will be completely rebuilt if the task’s influence changes.
The task also extends the environment of consumer tasks by adding the artifact’s
bin
directory to the PATH
. That way, any task that depends on
e2fsprogs
will be able to run its published programs directly without
explicitly referencing the artifact where they reside. Use this method to
package tools required by other tasks.
Also, note that the task requires a git
repository hosted at kernel.org
.
This git task, implemented by a builtin plugin, is actually not a
task but a resource. You can read more about resources next.
Resources
Resources are a special kind of task only executed in the context of other tasks. They are invoked to acquire and release a resource before and after the execution of a task. No artifact is produced by a resource.
A common use-case for resources is to allocate and reserve equipment required during the execution of a task. Such equipment could be a build server or a mobile device on which to run tests.
Below is a skeleton example providing mutual exclusion:
from jolt import *
class Exclusivity(Resource):
""" Resource providing mutual exclusion to an object """
to = Parameter(help="Name of shared object")
def acquire(self, artifact, deps, tools):
# TODO: Implement locking
self.info("{to} is now locked")
def release(self, artifact, deps, tools):
# TODO: Implement unlocking
self.info("{to} is now unlocked")
class RebootDevice(Task):
""" Reboots the specified test device """
device = Parameter(help="Name of device to reboot")
requires = ["exclusivity:to={device}"]
cacheable = False
def run(self, deps, tools):
tools.run("ssh {device} reboot")
Tests
After implementing the e2fsprogs
task above, the next logical step is
to write a few test-cases for the utility programs it builds. Luckily, Jolt
has integrated test support.
Test tasks are derived from the Test
base class instead of Task
and
they are implemented like a regular Python unittest.TestCase
. You can use
all assertions and decorators like you normally would. In all other respects,
a Test
task behaves just like a regular Task
.
Below is an example:
from jolt import *
class E2fsTest(Test):
requires = ["e2fsprogs"]
def setup(self, deps, tools):
self.tools = tools
def test_mke2fs(self):
self.assertTrue(self.tools.run("mke2fs"))
def test_badblocks(self):
self.assertTrue(self.tools.run("badblocks"))
def test_tune2fs(self):
self.assertTrue(self.tools.run("tune2fs"))
Influence
It is important that all attributes that define the output of a task are known and registered to avoid false cache hits. For example, in a compilation task all compiled source files should influence the task’s identity and trigger re-execution of the task if changed, otherwise binary compatibility will be lost quickly.
When using an external third-party build tool such as make, Jolt has no way of knowing what source files to monitor. This information must be explicitly provided by the task’s implementor. Luckily, Jolt provides a few builtin class decorators to make it easier.
Let’s revisit the e2fsprogs
task from earlier, but this time we assume that the
repository is already cloned and managed by external tools and not through the
builtin Jolt git
resource. We can no longer rely on the resource to automatically
influence the hash of the task. We instead use the git.influence
decorator:
from jolt import *
from jolt.plugins import git
@git.influence(path="path/to/e2fsprogs")
class E2fsprogs(Task):
def run(self, deps, tools):
ac = tools.autotools()
ac.configure("path/to/e2fsprogs")
ac.build()
ac.install()
The decorator adds the git repository’s tree hash as hash influence.
It will also add the git diff
output as influence to simplify iterative local
development.
There are a number of other useful influence decorators as well:
from jolt import *
from jolt import influence
from jolt.plugins import git
@influence.files("path/to/e2fsprogs/*.c")
@influence.environ("CFLAGS")
@influence.weekly
@influence.attribute("webstatus")
class E2fsprogs(Task):
@property
def webstatus(self):
r = requests.get("http://statusindicator/")
return r.text
def run(self, deps, tools):
ac = tools.autotools()
ac.configure("path/to/e2fsprogs")
ac.build()
ac.install()
self.report()
Above, the git.influence
decorator has been replaced by
influence.files
. The result is virtually the same, the content of all files
matched by the provided pattern will influence the hash of the task. However,
the Git tree hash implementation is more effecient and faster, but it obviously
doesn’t work if sources reside in a different type of repository.
The influence.environ
decorator is used to influence the hash of
the task based on the value of the CFLAGS
environment variable. If the
value of the variable changes the task will be re-executed.
The influence.weekly
decorator adds the week number as hash influence. If nothing else
changes, the task will be re-executed once every week. This can be useful to
verify that external resources, such as files downloaded from the Internet,
are still available. Other time-based decorators include:
influence.yearly
influence.montly
influence.daily
influence.hourly
The influence.attribute
decorator adds the value of an attribute or property as
hash influence. Above, the webstatus
property is registered to influence the task
with data obtained from a web service. The source code of the property itself is
monitored automatically.