Tutorial ========== Let's kick off with a few examples, getting more into details as we move along. First example -------------- Our first example is a classic. Copy/paste the code below into a file called ``first_example.jolt``. .. code-block:: python from jolt import Task class HelloWorld(Task): def run(self, deps, tools): print("Hello world!") This task will simply print "Hello world!" on the console when executed. The task name is automatically derived from the class name, but can be overridden by setting the ``name`` class attribute. Now try to execute the task: .. code-block:: bash $ jolt build helloworld Once the task has been executed, executing it again won't have any effect. This is because the task produced an empty artifact which is now stored in the local artifact cache. When Jolt determines if a task should be executed or not, it first calculates a task identity by hashing different attributes that would influence the output of the task, such as: * Source code * Name * Parameters * Dependencies When the task identity is known, Jolt searches its artifact cache for an artifact with the same identity. If one is found, no action is taken. You can try this out by changing the ``Hello world!`` string to something else and executing the task again. If the string is reverted back to ``Hello world!``, the task will regain its first identity and no action will be taken because that artifact is still present in the cache. To clean the local cache and remove all artifacts, run: .. code-block:: bash $ jolt clean To selectively remove a specific task's artifacts, run: .. code-block:: bash $ jolt clean helloworld Publishing Files ----------------- Tasks that don't produce output are not very useful. Let's rework our task to instead produce a file with the ``Hello world!`` message. We also shorten its name to ``hello``. .. literalinclude:: ../examples/publishing_files/publishing_files.jolt :language: python :caption: examples/publishing_files/publishing_files.jolt The implementation of the task is now split into two methods, ``run`` and ``publish``. The ``run`` method performs the main work of the task. It creates a file called ``message.txt`` containing our greeting from the first example. The file is written into a temporary build directory that will persist for the duration of the task's execution. The directory is removed afterwards. The ``publish`` method collects the output from the work performed by ``run``. It does so by instructing the artifact to collect all textfiles from the build directory. .. code-block:: bash $ jolt build hello After executing the task an artifact will be present in the local cache. Let's investigate its contents, but first we need to know the identity of the task in order to know what artifact to look for. Run: .. code-block:: bash $ jolt inspect -a hello The ``inspect`` command displays information about the task, including the documentation written in its Python class implementation. We're looking for the identity: .. code-block:: bash Identity 50a215905eb28a0911ff83828ac56b542525bce4 With this identity digest at hand, we can dive into the artifact cache. By default, the cache is located in ``$HOME/.cache/jolt``. To list the content of the current ``hello`` artifact, run: .. code-block:: bash $ ls $HOME/.cache/jolt/hello/50a215905eb28a0911ff83828ac56b542525bce4 You will see the ``message.txt`` file just created. Parameters ---------------- Next, we're going to use a task parameter to alter the ``Hello world!`` message. Instead of greeting the world, we'll allow the executor to specify an alternative recipient. We rename the class to reflect this change and we also add a parameter class attribute. The ``run`` method is changed to use the new parameter's value when writing the ``message.txt`` file. .. literalinclude:: ../examples/parameters/parameters.jolt :language: python :caption: examples/parameters/parameters.jolt By default, the produced message will still read ``Hello world!`` because the default value of the ``recipient`` parameter is ``world``. To produce a different message, try this: .. code-block:: bash $ jolt build hello:recipient=John Dependencies ------------ To better illustrate the flexibility of the new parameterized task, let's add another task class, ``Print``, which prints the contents of the ``message.txt`` file to the console. ``Print`` will declare a dependency on ``Hello``. .. literalinclude:: ../examples/dependencies/parameters.jolt :language: python :caption: examples/dependencies/parameters.jolt The output from this task is not ``cacheable``, forcing the task to be executed every time. It's dependency ``hello`` however, will only be re-executed if its influence changes, for example by passing new values to the ``recipient`` parameter. Try it out: .. code-block:: bash $ jolt build print:recipient=John $ jolt build print:recipient=Lisa $ jolt build print:recipient=Kelly Tools ----- The ``run`` and ``publish`` methods take a ``tools`` argument as their last parameter. This toolbox provides a large set of tools useful for many different types of tasks. See the reference documentation for more information. However, Jolt was originally created with compilation tasks in mind. Below is a real world example of a task compiling the ``e2fsprogs`` package containing EXT2/3/4 filesystem utility programs. It uses AutoTools to configure and build its sources into different binary applications. Luckily, the ``tools`` object provides utilities for building autotools projects as seen below. In addition to AutoTools, there is also support for CMake and Meson as well as generic support for running any third-party build tool. .. code-block:: python from jolt import * from jolt.plugins import git class E2fsprogs(Task): """ Ext 2/3/4 filesystem utilities """ requires = ["git:url=git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git"] def run(self, deps, tools): ac = tools.autotools() ac.configure(self.git["e2fsprogs"]) ac.build() ac.install() def publish(self, artifact, tools): ac = tools.autotools() ac.publish(artifact) artifact.environ.PATH.append("bin") The autotools ``ac`` object automatically creates temporary build and install (--prefix) directories which are used when configuring, building and installing the project. All files installed in the installation directory will be published. Both directories are removed when execution has finished, i.e. the project will be completely rebuilt if the task's influence changes. The task also extends the environment of consumer tasks by adding the artifact's ``bin`` directory to the ``PATH``. That way, any task that depends on ``e2fsprogs`` will be able to run its published programs directly without explicitly referencing the artifact where they reside. Use this method to package tools required by other tasks. Also, note that the task requires a ``git`` repository hosted at ``kernel.org``. This git task, implemented by a builtin plugin, is actually not a task but a resource. It clones the repository and makes it available to the ``e2fsprogs`` task through its ``git`` attribute. You can read more about resources next. Resources --------- Resources are a special kind of task only executed in the context of other tasks. They are invoked to acquire and release a resource before and after the execution of a task. No artifact is produced by a resource. A common use-case for resources is to allocate and reserve equipment required during the execution of a task. Such equipment could be a build server or a mobile device on which to run tests. Below is a skeleton example providing mutual exclusion: .. code-block:: python from jolt import * class Exclusivity(Resource): """ Resource providing mutual exclusion to an object """ to = Parameter(help="Name of shared object") def acquire(self, artifact, deps, tools): # TODO: Implement locking self.info("{to} is now locked") def release(self, artifact, deps, tools): # TODO: Implement unlocking self.info("{to} is now unlocked") class RebootDevice(Task): """ Reboots the specified test device """ device = Parameter(help="Name of device to reboot") requires = ["exclusivity:to={device}"] cacheable = False def run(self, deps, tools): tools.run("ssh {device} reboot") Tests ------ After implementing the ``e2fsprogs`` task above, the next logical step is to write a few test-cases for the utility programs it builds. Luckily, Jolt has integrated test support. Test tasks are derived from the ``Test`` base class instead of ``Task`` and they are implemented like a regular Python ``unittest.TestCase``. You can use all assertions and decorators like you normally would. In all other respects, a ``Test`` task behaves just like a regular ``Task``. Below is an example: .. code-block:: python from jolt import * class E2fsTest(Test): requires = ["e2fsprogs"] def setup(self, deps, tools): self.tools = tools def test_mke2fs(self): self.assertTrue(self.tools.run("mke2fs")) def test_badblocks(self): self.assertTrue(self.tools.run("badblocks")) def test_tune2fs(self): self.assertTrue(self.tools.run("tune2fs")) Influence --------- It is important that all attributes that define the output of a task are known and registered to avoid false cache hits. For example, in a compilation task all compiled source files should influence the task's identity and trigger re-execution of the task if changed, otherwise binary compatibility will be lost quickly. When using an external third-party build tool such as make, Jolt has no way of knowing what source files to monitor. This information must be explicitly provided by the task's implementor. Luckily, Jolt provides a few builtin class decorators to make it easier. Let's revisit the ``e2fsprogs`` task from earlier, but this time we assume that the repository is already cloned and managed by external tools and not through the builtin Jolt ``git`` resource. We can no longer rely on the resource to automatically influence the hash of the task. We instead use the ``git.influence`` decorator: .. code-block:: python from jolt import * from jolt.plugins import git @git.influence(path="path/to/e2fsprogs") class E2fsprogs(Task): def run(self, deps, tools): ac = tools.autotools() ac.configure("path/to/e2fsprogs") ac.build() ac.install() The decorator adds the git repository's tree hash as hash influence. It will also add the ``git diff`` output as influence to simplify iterative local development. There are a number of other useful influence decorators as well: .. code-block:: python from jolt import * from jolt import influence from jolt.plugins import git @influence.files("path/to/e2fsprogs/*.c") @influence.environ("CFLAGS") @influence.weekly @influence.attribute("webstatus") class E2fsprogs(Task): @property def webstatus(self): r = requests.get("http://statusindicator/") return r.text def run(self, deps, tools): ac = tools.autotools() ac.configure("path/to/e2fsprogs") ac.build() ac.install() self.report() Above, the ``git.influence`` decorator has been replaced by ``influence.files``. The result is virtually the same, the content of all files matched by the provided pattern will influence the hash of the task. However, the Git tree hash implementation is more effecient and faster, but it obviously doesn't work if sources reside in a different type of repository. The ``influence.environ`` decorator is used to influence the hash of the task based on the value of the ``CFLAGS`` environment variable. If the value of the variable changes the task will be re-executed. The ``influence.weekly`` decorator adds the week number as hash influence. If nothing else changes, the task will be re-executed once every week. This can be useful to verify that external resources, such as files downloaded from the Internet, are still available. Other time-based decorators include: - ``influence.yearly`` - ``influence.montly`` - ``influence.daily`` - ``influence.hourly`` The ``influence.attribute`` decorator adds the value of an attribute or property as hash influence. Above, the ``webstatus`` property is registered to influence the task with data obtained from a web service. The source code of the property itself is monitored automatically.