Distributed Builds

To support distributed and incremental task execution in a build cluster, two things are typically needed:

  • a centralized file server where resulting task artifacts can be stored

  • workers executing tasks and uploading the artifacts to the file server

Task recipies may also have to be adapted to work in a network build environment. In particular, all task dependencies such as tools and source code repositories should be explicitly declared to allow the task to run anywhere. The more explicit the better. Your workers should be capabable of handling any workload. With proper dependency management they can provision the correct environment for any task on demand.

Deploying with Docker Swarm

Docker’s Swarm mode is an easy to use container orchestration tool which can be used to deploy and manage a cluster of Jolt workers. In this example, the Jolt AMQP plugin is used to facilitate communication between Jolt clients and workers. The workers are deployed using the standard Jolt Docker container available in Docker Hub. RabbitMQ has been chosen as the AMQP message broker and Nginx serves as the centralized HTTP artifact cache. The Docker compose file for this setup looks like this:

version: "3.5"
services:
  amqp:
    image: rabbitmq:latest
    ports:
      - "5672:5672"
  cache:
    image: nginx:latest
    deploy:
      replicas: 1
    configs:
      - source: nginx.conf
        target: /etc/nginx/nginx.conf
    ports:
      - "8080:80"
    volumes:
      - cache-http:/usr/share/nginx/html
  dashboard:
    image: robrt/jolt-dashboard:latest
    ports:
      - "80:80"
  worker:
    image: robrt/jolt:latest
    deploy:
      replicas: 2
    command: ["-c", "/opt/jolt/worker.conf", "amqp-worker"]
    configs:
      - source: worker.conf
        target: /opt/jolt/worker.conf
    volumes:
      - cache-node:/root/.cache/jolt
configs:
  worker.conf:
    file: ./worker.conf
  nginx.conf:
    file: ./nginx.conf
volumes:
  cache-node:
  cache-http:

The two Jolt workers are configured through the worker.conf file:

[amqp]
host = amqp
config = /opt/jolt/worker.conf

[http]
uri = http://cache/

[dashboard]

This configuration enables the AMQP and HTTP plugins. The hidden Jolt amqp-worker command enabled by the AMQP plugin will connect to the configured message broker and start processing execution requests. The HTTP plugin will store the resulting artifacts on the configured HTTP server. Jolt clients can then download these artifacts to the local host. In the example, a simple local Docker volume is used as server storage for the artifacts. In a real deployment, you probably want to use something else.

To deploy the system into a swarm, run:

$ docker stack deploy -c jolt.yaml jolt

You can then scale up the the number of workers to a number suitable for your swarm:

$ docker service scale jolt_worker=10

Scaling is possible even with tasks in progress as long as they don’t cause any side effects. If a task is interrupted because the worker is terminated, RabbitMQ will redeliver the execution request to another worker.

The newly deployed swarm of Jolt workers is utilized by configuring the Jolt client as follows:

[jolt]
upload = false

[http]
uri = http://localhost:8080/

[amqp]
host = localhost
port = 5672

[dashboard]
uri = http://localhost

These configuration keys can also be set from command line:

$ jolt config amqp.host localhost
$ jolt config http.uri http://localhost/

If your local machine is not part of the swarm you will need to replace localhost with the IP-address of one of the swarm nodes.

To schedule a task in the swarm, pass the –network flag to the build command:

$ jolt build --network <task>

Alternatively, if you are using a separate configuration file:

$ jolt -c client.conf build --network <task>

Adapting Task Definitions

Task classes may have to be adapted to work in a distributed execution environment. For example, Jolt will by default not transfer any workspace files to a worker. Such dependencies, typically source repositories, must be listed as task requirements.

Another common issue is that workers don’t have the required tools installed. Those tools have to be packaged by Jolt tasks and listed as requirements in order to be automatically provisioned on the workers.