Deploying a Build Cluster

Jolt is designed to be deployed as a set of containers. To deploy a build cluster you typically use a container orchestration environment such as Kubernetes or Docker Swarm. See their respective documentation for installation instructions.

The different components of the build cluster are:

  • The Jolt scheduler, which is responsible for build and task scheduling.

  • The Jolt worker, which executes tasks as instructed by the scheduler.

  • The artifact cache, which is a HTTP server used to cache build artifacts.

  • The Jolt dashboard, which is a web application used to monitor the build cluster.

Each of the components is deployed as a separate container. Information about the images and their configuration environment variables can be found in Container Images

For service metrics, worker-pool utilization and scrape configuration, see Monitoring with Prometheus.

Adapting Task Definitions

Task classes may have to be adapted to work in a distributed execution environment. For example, Jolt will by default not transfer any workspace files to a worker. Such dependencies, typically source repositories, must be listed as task requirements. See the Jolt test suite for examples of how to do this.

Another common issue is that workers don’t have the required tools installed. Those tools should to be packaged by Jolt tasks and listed as requirements in order to be automatically provisioned on the workers. They can also be installed manually in the worker container image, but this is not recommended as it makes administration of the build cluster more difficult, especially when multiple different versions of the same tool are required.

Build Parameters

Build submissions carry the [params] section of the configuration to the scheduler and the workers. The parameters can describe builds, for example the CI job that started them:

$ jolt -c params.origin="$BUILD_URL" -c params.branch=main build -n my-task

They can also be set in the configuration file. Values are expanded on the client before the build is submitted, so they may refer to its environment:

[params]
origin = {environ[BUILD_URL]}

The dashboard shows all parameters of a build, and origin with its tasks, as a link if it is a URL. Other tools that follow the builds of the scheduler see them too. Jolt doesn’t check parameters: whoever submits a build can set them to anything. Who submitted a build is known from its owners instead, the authenticated clients that the scheduler records, see Authentication.

Warning

Anyone who can see the builds of the scheduler can see their parameters. Don’t put passwords or other secrets in [params].

Changing a parameter doesn’t make Jolt rebuild tasks whose artifacts are already in the cache, unless the task asks for it with @influence.config("params", "origin"). The same tasks submitted with different parameters are still listed as separate builds.

Docker Swarm

Docker Swarm is an easy to use container orchestration tool which can be used to deploy and manage the Jolt build cluster. The below Docker stack yaml file will deploy a scheduler and two workers, as well as an artifact cache.

version: "3.5"

# Authentication between the services is commented out below. To enable it:
#
#  1. Create a token for each kind of client with `jolt token <name>`, for
#     example `jolt token workers -p read -p write` for the cache.
#  2. Add their hashes to cache.yaml and scheduler.yaml.
#  3. Give the tokens to the workers and the dashboard below, and to users
#     in client.conf.
#
# Tokens are only sent over TLS. Either give the cache and the scheduler a
# certificate, or allow sending tokens without TLS on a trusted network with
# JOLT_ALLOW_INSECURE_TOKEN. Keep the tokens out of this file in production,
# for example in an env_file that isn't checked in.

services:
  cache:
    image: robrt/jolt-cache:latest
    environment:
      - JOLT_CACHE_INSECURE=true
      - JOLT_CACHE_MAX_SIZE=100GB
      - JOLT_CACHE_VERBOSITY=2
      # TLS, instead of JOLT_CACHE_INSECURE:
      # - JOLT_CACHE_CERT=/run/secrets/cert.pem
      # - JOLT_CACHE_CERT_KEY=/run/secrets/key.pem
    # configs:
    #   - source: cache.yaml
    #     target: /etc/jolt/cache.yaml
    # secrets:
    #   - cert.pem
    #   - key.pem
    ports:
      - "8081:8080"
      - "9091:9090"
    volumes:
      - cache-http:/data
  dashboard:
    image: robrt/jolt-dashboard:latest
    environment:
      - JOLT_SCHEDULER_URI=tcp://scheduler:9090
      # Token with the read permission of the scheduler:
      # - JOLT_SCHEDULER_TOKEN=<dashboard token>
      # - JOLT_ALLOW_INSECURE_TOKEN=true
    ports:
      - "80:80"
  scheduler:
    image: robrt/jolt-scheduler:latest
    # TLS:
    # environment:
    #   - JOLT_CERT=/run/secrets/cert.pem
    #   - JOLT_CERT_KEY=/run/secrets/key.pem
    # secrets:
    #   - cert.pem
    #   - key.pem
    configs:
      - source: scheduler.yaml
        target: /etc/jolt/scheduler.yaml
    ports:
      - "8080:8080"
      - "9090:9090"
  worker:
    environment:
      - "JOLT_CACHE_DIR=/data/cache"
      - "JOLT_CACHE_GRPC_URI=tcp://cache:9090"
      - "JOLT_CACHE_HTTP_URI=http://cache:8080"
      # Tokens with the worker permission of the scheduler, and the read and
      # write permissions of the cache. The worker passes them on to the
      # executors it starts.
      # - "JOLT_SCHEDULER_TOKEN=<workers token>"
      # - "JOLT_CACHE_TOKEN=<workers token>"
      # - "JOLT_ALLOW_INSECURE_TOKEN=true"
      # With TLS, use tls:// and https:// URIs, and give the certificate of
      # a private authority with JOLT_CA_CERT.
    image: robrt/jolt-worker:latest
    deploy:
      replicas: 2
    configs:
      - source: worker.conf
        target: /root/.config/jolt/config
    volumes:
      - cache-node:/data/cache
      - /etc/machine-id:/etc/machine-id
configs:
  # cache.yaml:
  #   file: ./cache.yaml
  scheduler.yaml:
    file: ./scheduler.yaml
  worker.conf:
    file: ./worker.conf
# secrets:
#   cert.pem:
#     file: ./cert.pem
#   key.pem:
#     file: ./key.pem
volumes:
  cache-node:
  cache-http:

The Jolt workers are configured in the worker.conf file:

[jolt]
# The location of the local Jolt artifact cache.
cachedir = /data/cache

# The maximum size of the cache in bytes.
# cachesize = 10G

[cache]
# Location of the Jolt cache inside the build cluster:
grpc_uri = tcp://cache:9090
http_uri = http://cache:8080

[scheduler]
# URI of the Jolt scheduler inside the build cluster:
grpc_uri = tcp://scheduler:9090
http_uri = http://scheduler:8080

The file configures the URIs of the scheduler service and the HTTP cache. In the example, local Docker volumes are used as storage for artifacts. In a real deployment, persistent volumes are recommended. The administrator should also configure the maximum size allowed for the local cache in each node with the jolt.cachesize configuration key. If multiple workers are deployed on the same node, the local cache may be shared between them in the same directory. Fast SSD storage is recommended for the local cache and the worker workspace.

To deploy the system into a swarm, run:

$ docker stack deploy -c jolt.yaml jolt

You can then scale up the the number of workers to a number suitable for your swarm:

$ docker service scale jolt_worker=10

Scaling is possible even with tasks in progress as long as they don’t cause any side effects. If a task is interrupted because the worker is terminated, the scheduler will redeliver the task execution request to another worker.

The newly deployed build cluster is utilized by configuring the Jolt client as follows:

[jolt]
# Disable artifact upload in local builds.
# This is overridden when running a distributed network build.
# It can be overridden on the command line with the --upload flag.
upload = false

[cache]
# Location of the Jolt cache service.
# Replace 'localhost' with the hostname or IP of the cache in your deployment.
grpc_uri = tcp://127.0.0.1:9091
http_uri = http://127.0.0.1:8081
# Token of the cache, if the cache requires authentication.
# token = <users token>

[scheduler]
# Location of the Jolt scheduler.
# Replace 'localhost' with the hostname or IP of the scheduler your deployment.
grpc_uri = tcp://127.0.0.1:9090
http_uri = http://127.0.0.1:8080
# Token of the scheduler, if the scheduler requires authentication.
# token = <users token>

# Users may instead sign in with `jolt login` to an OpenID Connect provider
# that the services accept tokens from:
# [auth]
# issuer = https://keycloak.example.com/realms/jolt

# Tokens are only sent over TLS, with https:// and tls:// URIs above, unless
# this is enabled. ca_cert gives the certificate of a private authority.
# [jolt]
# allow_insecure_token = true
# ca_cert = /etc/ssl/jolt-ca.pem

These configuration keys can also be set from command line:

$ jolt config scheduler.grpc_uri tcp://127.0.0.1:9090
$ jolt config scheduler.http_uri http://127.0.0.1:8080
$ jolt config cache.grpc_uri tcp://127.0.0.1:9091
$ jolt config cache.http_uri http://127.0.0.1:8081

If your local machine is not part of the swarm you will need to replace 127.0.0.1 with the IP-address of one of the nodes in the swarm or, preferably, a load balancing hostname.

To execute a task in the swarm, pass the -n/--network flag to the build command:

$ jolt build -n <task>

Alternatively, if you are using a separate configuration file:

$ jolt -c client.conf build --network <task>

Kubernetes

Kubernetes is a more complex container orchestration tool which can be used to deploy and manage the Jolt build cluster. The below Kubernetes deployment yaml file will deploy a scheduler, two workers, an artifact cache as well as the dashboard. Notice inline ‘’FIXME’’ comments in the yaml file that need to or should be replaced with actual values.

# Authentication between the services is commented out below. To enable it:
#
#  1. Create a token for each kind of client with `jolt token <name>`, for
#     example `jolt token workers -p read -p write` for the cache.
#  2. Add their hashes to cache.yaml and scheduler.yaml in the ConfigMap.
#  3. Add the tokens to the jolt-tokens Secret, which the workers and the
#     dashboard read, and give users theirs in client.conf.
#
# Tokens are only sent over TLS. Either give the cache and the scheduler a
# certificate, or allow sending tokens without TLS inside the cluster with
# JOLT_ALLOW_INSECURE_TOKEN. With authentication, the metrics endpoints also
# require a token with the read permission.
#
# apiVersion: v1
# kind: Secret
# metadata:
#   name: jolt-tokens
# type: Opaque
# stringData:
#   # Worker permission of the scheduler, read and write of the cache.
#   workers: <workers token>
#   # Read permission of the scheduler.
#   dashboard: <dashboard token>
# ---
apiVersion: v1
kind: ConfigMap
metadata:
  name: jolt-config
data:
  # Add "anonymous: [read]" to auth to allow anyone to download artifacts.
  # cache.yaml:
  #   |-
  #     auth:
  #       tokens:
  #         - name: workers
  #           sha256: <hash of the workers token>
  #           permissions: [read, write]
  #         - name: users
  #           sha256: <hash of the users token>
  #           permissions: [read]
  scheduler.yaml:
    |-
      listen_grpc:
       - tcp://:9090
      listen_http:
       - tcp://:8080
      # FIXME: Replace with an address of the scheduler that the browsers
      # of users can reach. The dashboard links to task logs with it.
      public_http:
       - http://jolt-scheduler:8080
      logstash:
        size: 10GB
        storage: disk
        path: /root/logstash
      # auth:
      #   tokens:
      #     - name: workers
      #       sha256: <hash of the workers token>
      #       permissions: [worker]
      #     - name: dashboard
      #       sha256: <hash of the dashboard token>
      #       permissions: [read]
      #     - name: users
      #       sha256: <hash of the users token>
      #       permissions: [build, read]
  worker.conf:
    |-
      # This configuration is used by the Jolt application
      # that is installed and executed by worker containers.

      [jolt]
      # The location of the local Jolt artifact cache.
      cachedir = /data/cache

      # The maximum size of the cache in bytes.
      # FIXME: Replace with an actual value.
      cachesize = 100G

      [cache]
      # Location of the Jolt cache inside the build cluster:
      grpc_uri = tcp://jolt-cache:9090
      http_uri = http://jolt-cache:8080

      [scheduler]
      # URI of the Jolt scheduler inside the build cluster:
      grpc_uri = tcp://jolt-scheduler:9090
      http_uri = http://jolt-scheduler:8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jolt-cache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jolt-cache
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "8080"
      labels:
        app: jolt-cache
    spec:
      containers:
        - name: jolt-cache
          env:
            - name: JOLT_CACHE_INSECURE
              value: "true"
            - name: JOLT_CACHE_MAX_SIZE
              # FIXME: Replace with an actual value.
              value: "100GiB"
            - name: JOLT_CACHE_VERBOSITY
              value: "2"
          image: robrt/jolt-cache:latest
          ports:
            - containerPort: 8080
            - containerPort: 9090
          volumeMounts:
            - name: jolt-cache
              mountPath: /data
            # - name: jolt-config
            #   mountPath: /etc/jolt
      volumes:
      # - name: jolt-config
      #   configMap:
      #     name: jolt-config
      #     items:
      #       - key: cache.yaml
      #         path: cache.yaml
      - name: jolt-cache
        # FIXME: Replace with an actual persistent volume claim.
        #
        # This volume is used to store task artifacts.
        # An emptyDir volume is used as an example, but
        # the volume should typically be backed by a
        # persistent volume claim that survives pod restarts.
        emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jolt-dashboard
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jolt-dashboard
  template:
    metadata:
      labels:
        app: jolt-dashboard
    spec:
      containers:
        - name: jolt-dashboard
          image: robrt/jolt-dashboard:latest
          env:
            # The dashboard subscribes to the scheduler's event stream.
            - name: JOLT_SCHEDULER_URI
              value: tcp://jolt-scheduler:9090
            # - name: JOLT_SCHEDULER_TOKEN
            #   valueFrom:
            #     secretKeyRef:
            #       name: jolt-tokens
            #       key: dashboard
            # - name: JOLT_ALLOW_INSECURE_TOKEN
            #   value: "true"
          ports:
            - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jolt-scheduler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jolt-scheduler
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "8080"
      labels:
        app: jolt-scheduler
    spec:
      containers:
        - name: jolt-scheduler
          image: robrt/jolt-scheduler:latest
          ports:
            - containerPort: 8080
            - containerPort: 9090
          volumeMounts:
            - name: jolt-config
              mountPath: /etc/jolt
      volumes:
      - name: jolt-config
        configMap:
          name: jolt-config
          items:
            - key: scheduler.yaml
              path: scheduler.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jolt-worker
spec:
  replicas: 2
  selector:
    matchLabels:
      app: jolt-worker
  template:
    metadata:
      labels:
        app: jolt-worker
    spec:
      containers:
        - name: jolt-worker
          env:
            - name: JOLT_CACHE_DIR
              value: /data/cache
            - name: JOLT_CACHE_GRPC_URI
              value: "tcp://jolt-cache:9090"
            - name: JOLT_CACHE_HTTP_URI
              value: "http://jolt-cache:8080"
            - name: JOLT_SCHEDULER_GRPC_URI
              value: "tcp://jolt-scheduler:9090"
            # The worker passes the tokens on to the executors it starts.
            # - name: JOLT_SCHEDULER_TOKEN
            #   valueFrom:
            #     secretKeyRef:
            #       name: jolt-tokens
            #       key: workers
            # - name: JOLT_CACHE_TOKEN
            #   valueFrom:
            #     secretKeyRef:
            #       name: jolt-tokens
            #       key: workers
            # - name: JOLT_ALLOW_INSECURE_TOKEN
            #   value: "true"
          image: robrt/jolt-worker:latest
          volumeMounts:
            - name: jolt-cache
              mountPath: /data/cache
            - name: jolt-config
              mountPath: /root/.config/jolt
            - name: jolt-ws
              mountPath: /data/ws
            - name: machine-id
              mountPath: /etc/machine-id
      volumes:
      - name: machine-id
        hostPath:
          path: /etc/machine-id
      - name: jolt-config
        configMap:
          name: jolt-config
          items:
            - key: worker.conf
              path: config
      - name: jolt-cache
        # FIXME: Replace with an actual hostPath volume.
        #
        # This volume is used to store the artifacts locally in the worker.
        # An emptyDir volume is used as an example, but
        # the volume should typically be backed by a hostPath volume
        # with a path that is shared between all workers on the node.
        #
        #   hostPath:
        #     path: /data/cache
        #
        emptyDir: {}
      - name: jolt-ws
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: jolt-cache
spec:
  selector:
    app: jolt-cache
  ports:
    - name: "http"
      port: 8080
      targetPort: 8080
    - name: "grpc"
      port: 9090
      targetPort: 9090
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: jolt-dashboard
spec:
  selector:
    app: jolt-dashboard
  ports:
    - name: "http"
      port: 80
      targetPort: 80
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  name: jolt-scheduler
spec:
  selector:
    app: jolt-scheduler
  ports:
    - name: "http"
      port: 8080
      targetPort: 8080
    - name: "grpc"
      port: 9090
      targetPort: 9090
  type: LoadBalancer
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jolt-cache
spec:
  rules:
    # FIXME:
    # Replace this with the actual domain name of the cache.
    - host: cache.jolt.domain
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: jolt-cache
                port:
                  number: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jolt-dashboard
spec:
  rules:
    # FIXME:
    # Replace this with the actual domain name of the dashboard.
    - host: dashboard.jolt.domain
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: jolt-dashboard
                port:
                  number: 80
---

To deploy the system into a Kubernetes cluster, run:

$ kubectl apply -f jolt.yaml

You can then scale up the the number of workers to a number suitable for your cluster:

$ kubectl scale deployment jolt-worker --replicas=10

Scaling is possible even with tasks in progress as long as they don’t cause any side effects. If a task is interrupted because the worker is terminated, the scheduler will redeliver the task execution request to another worker.

The newly deployed build cluster is utilized by configuring the Jolt client as follows:

[jolt]
# Disable artifact upload in local builds.
# This is overridden when running a distributed network build.
# It can be overridden on the command line with the --upload flag.
upload = false

[cache]
# Location of the Jolt cache service.
# Replace 'localhost' with the hostname or IP of the cache in your deployment.
http_uri = http://<cache-host>:8080
grpc_uri = tcp://<cache-host>:9090
# Token of the cache, if the cache requires authentication.
# token = <users token>

[scheduler]
# Location of the Jolt scheduler.
# Replace 'localhost' with the hostname or IP of the scheduler your deployment.
grpc_uri = tcp://<scheduler-host>:9090
http_uri = http://<scheduler-host>:8080
# Token of the scheduler, if the scheduler requires authentication.
# token = <users token>

# Users may instead sign in with `jolt login` to an OpenID Connect provider
# that the services accept tokens from:
# [auth]
# issuer = https://keycloak.example.com/realms/jolt

# Tokens are only sent over TLS, with https:// and tls:// URIs above, unless
# this is enabled. ca_cert gives the certificate of a private authority.
# [jolt]
# allow_insecure_token = true
# ca_cert = /etc/ssl/jolt-ca.pem

The placeholder hosts should be replaced with the actual hostnames or IPs of the services in the Kubernetes cluster. The services are typically exposed through a load balancer and/or an ingress controller. Both methods are exemplified in the yaml file, but may not work out of the box in all Kubernetes installations. Run the following command to find the ExternalIP addresses of the services:

$ kubectl get services jolt-cache jolt-scheduler

The client configuration keys can also be set from command line:

$ jolt config scheduler.grpc_uri tcp://<scheduler-host>:9090
$ jolt config scheduler.http_uri http://<scheduler-host>:8080
$ jolt config cache.grpc_uri tcp://<cache-host>:9090
$ jolt config cache.http_uri http://<cache-host>:8080

To execute a task in the cluster, pass the -n/--network flag to the build command:

$ jolt build -n <task>

Alternatively, if you are using a separate configuration file:

$ jolt -c client.conf build --network <task>

Authentication

Without configuration, the cache and the scheduler accept requests from anyone who can reach them. Before exposing a build cluster to a network you don’t trust, secure the connections with TLS and authenticate clients, either with static tokens or with the accounts of an OpenID Connect provider. The two can be combined, with tokens for CI and workers and personal logins for users.

TLS

The cache and the scheduler serve TLS on both HTTP and gRPC when they are configured with a certificate and private key (cert and cert_key). Clients then connect with https:// and tls:// URIs:

[cache]
http_uri = https://cache.example.com:8080
grpc_uri = tls://cache.example.com:9090

[scheduler]
grpc_uri = tls://scheduler.example.com:9090
http_uri = https://scheduler.example.com:8080

Clients verify the certificates of services with the authorities of the system. For a private authority, point jolt.ca_cert or JOLT_CA_CERT to a PEM file with its certificate. Jolt, its workers, schedulerctl, fstree and the dashboard all use these settings.

Tokens are only sent over TLS, since they authenticate anyone who sees them. For a trusted network, jolt.allow_insecure_token or JOLT_ALLOW_INSECURE_TOKEN allows sending them without it.

Tokens

A token is a random string that grants permissions. Create one with:

$ jolt token ci --permission read --permission write

The command prints the token and the entry for the auth section of cache.yaml or scheduler.yaml, which is the same for both services except for the permissions. The configuration only contains the SHA-256 hash of the token. Once any token is configured, clients without a token have no permissions, unless auth.anonymous grants them some.

The cache grants read, to download and look up artifacts, and write, to upload them. Example, where anyone may download artifacts but only CI may upload them:

# /etc/jolt/cache.yaml
auth:
  anonymous: [read]
  tokens:
    - name: ci
      sha256: 31f69e6b9c24b00832b59b38a340b5346185c93eb9bb6bb54b2ef985aafdcb8f
      permissions: [read, write]

The scheduler grants these permissions:

Permission

Allows

build

Building, cancelling the builds that the client submitted, and running their tasks. Clients that build also need read to see the logs of their tasks.

worker

Executing tasks, required by workers and the executors they start.

read

Listing builds and workers, following events and reading logs.

admin

Cancelling any build, rescheduling tasks and debugging the scheduler. Administrators also need read to list builds and workers.

Example:

# /etc/jolt/scheduler.yaml
auth:
  tokens:
    - name: developers
      sha256: 9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab
      permissions: [build, read]
    - name: workers
      sha256: 4c1e0fb8de8c8e4c4f5b4ba9c3aa6a9ac5c3d7fb22bb4b8b0f4cbf5e8b0b6c3e
      permissions: [worker]

Clients send their tokens as bearer tokens:

  • Jolt uses cache.token and scheduler.token, or the JOLT_CACHE_TOKEN and JOLT_SCHEDULER_TOKEN environment variables.

  • Workers use scheduler_token and cache_token, and pass them on to the executors they start, which upload artifacts.

  • schedulerctl uses scheduler_token in schedulerctl.yaml or JOLT_SCHEDULER_TOKEN, and the dashboard JOLT_SCHEDULER_TOKEN.

Task logs are opened in a browser, which asks for a user name and password. Enter the token as the password; the user name is ignored.

Single Sign-On

Instead of handing out tokens to users, the services can accept the ID tokens of an OpenID Connect provider, such as Keycloak, Dex or Authentik in front of an LDAP directory, or Microsoft Entra ID. Users sign in once with jolt login, and every service verifies their tokens on its own with the public keys of the provider. The services don’t talk to the directory, and they don’t share any secret.

Register Jolt with the provider as a public client, with the OAuth device authorization grant enabled, and configure the services to accept tokens issued to it. Permissions are granted to all users of the provider and to the members of groups:

# /etc/jolt/cache.yaml
auth:
  oidc:
    issuer: https://keycloak.example.com/realms/jolt
    audience: jolt
    permissions: [read]
    groups:
      - name: jolt-ci
        permissions: [write]

# /etc/jolt/scheduler.yaml
auth:
  oidc:
    issuer: https://keycloak.example.com/realms/jolt
    audience: jolt
    permissions: [build, read]
    groups:
      - name: jolt-admins
        permissions: [admin]

The services contact the provider when they verify the first token, so they start even when it is unavailable. Tokens configured in auth.tokens keep working alongside the provider.

Users configure the provider in the auth section and sign in:

$ jolt config auth.issuer https://keycloak.example.com/realms/jolt
$ jolt login

jolt login asks for the user name and password in the terminal if the provider allows the OAuth password grant for the client. Otherwise it opens the browser, or prints a URL and a code to enter there. The tokens are stored in the system keyring, and Jolt refreshes them when they are about to expire, so users stay signed in until the refresh token expires or they run jolt logout. A token configured for a service takes precedence over the login.

Notes on providers:

  • Keycloak: Groups aren’t included in tokens by default. Add a Group Membership mapper with the claim name groups and Full group path turned off to the client, or use realm roles with groups_claim: realm_access.roles. Turn on OAuth 2.0 Device Authorization Grant for the client, and Direct access grants to sign in with a password in the terminal.

  • Dex: Request the groups scope with auth.scopes.

  • Microsoft Entra ID: Use the issuer of your tenant, https://login.microsoftonline.com/<tenant-id>/v2.0, and the application ID as the audience and auth.client_id. Turn on Allow public client flows for the application. Tokens contain the IDs of groups rather than their names, so prefer app roles assigned to groups, with groups_claim: roles. Users sign in with the browser, which also handles multi-factor authentication.