JobSet: Multi-Job Workloads on Kubernetes

JobSet is a Kubernetes-native API for managing groups of related Jobs as a single unit. It creates one or more Kubernetes Jobs from different or identical templates, controlling their lifecycle together — all succeed or all fail. This makes it ideal for distributed workloads like multi-node training, MPI applications, and HPC simulations where multiple Job groups must coordinate.

This module introduces JobSet concepts, walks you through creating multi-job workloads, and shows how to integrate JobSet with Kueue for admission control.

Why JobSet?

Running a distributed workload on Kubernetes often requires multiple groups of Pods with different roles — workers, parameter servers, drivers. Without JobSet you must:

  • Create and manage each Job independently

  • Coordinate lifecycle across Jobs manually — if one Job fails, you must detect and clean up the others

  • Handle DNS and network discovery between Pods across different Jobs

  • Implement restart logic that restarts all Jobs together, not just the one that failed

JobSet solves these problems by treating a set of Jobs as a single resource with coordinated lifecycle, automatic DNS setup, and unified failure handling.

Core Concepts

ReplicatedJobs

The .spec.replicatedJobs list is the heart of a JobSet. Each entry defines a Job template and a replica count:

  • template — The Job spec (parallelism, completions, Pod template).

  • replicas — How many copies of this Job to create (defaults to 1).

Each Job gets a deterministic name: <jobSetName>-<replicatedJobName>-<jobIndex>, where jobIndex runs from 0 to replicas - 1.

You can define multiple entries in replicatedJobs to create Jobs with different templates — for example, a leader Job and a set of worker Jobs.

Completion Policy

A JobSet is marked as successful when all of its child Jobs complete successfully. If any Job is still running, the JobSet remains active.

Failure Policy

A JobSet failure is counted when any child Job fails. The spec.failurePolicy.maxRestarts field controls how many times JobSet will automatically restart all child Jobs before declaring terminal failure. A restart recreates all Jobs, not just the one that failed — this is critical for distributed workloads where partial state is invalid.

Network and DNS

JobSet automatically creates a headless Service and configures DNS so that Pods can discover each other by hostname. The subdomain defaults to the JobSet name but can be overridden with spec.network.subdomain.

  • Pod hostname: <jobSetName>-<replicatedJobName>-<replicaIndex>-<podIndex>

  • FQDN: <jobSetName>-<replicatedJobName>-<replicaIndex>-<podIndex>.<subdomain>

This deterministic naming is essential for distributed frameworks that need to know peer addresses at startup (e.g., setting MASTER_ADDR in PyTorch distributed training).

Coordinator

JobSet can designate a coordinator Pod using spec.coordinator. The coordinator’s stable network endpoint is automatically injected as a label (jobset.sigs.k8s.io/coordinator) on all Jobs and Pods, so workers can discover the coordinator without hardcoding addresses.

Labels

JobSet applies these labels to all child Jobs and Pods:

Label Value

jobset.sigs.k8s.io/jobset-name

The JobSet’s name

jobset.sigs.k8s.io/replicatedjob-name

The replicatedJob entry name

jobset.sigs.k8s.io/replicatedjob-replicas

The replica count

jobset.sigs.k8s.io/job-index

Ordinal index of the Job (0, 1, 2, …​)

Defaults

JobSet applies sensible defaults for batch workloads:

  • Job completionMode is defaulted to Indexed

  • Pod restartPolicy is defaulted to OnFailure

Exercise 1: Creating a Basic JobSet

In this exercise you will create a JobSet with a single replicated Job group to understand the basic mechanics.

Prerequisites

Ensure the JobSet controller is installed on your cluster.

  1. Verify the JobSet controller is running:

    oc get pods -n jobset-system

    You should see the jobset-controller-manager Pod in Running state.

  2. Create a namespace for the exercises:

    oc new-project jobset-demo --skip-config-update || oc project jobset-demo

Create a simple JobSet

  1. Create a JobSet with 3 worker replicas, each running 2 parallel Pods:

    cat <<EOF | oc apply -f -
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: simple-jobset
  namespace: jobset-demo
spec:
  replicatedJobs:
    - name: workers
      replicas: 3
      template:
        spec:
          parallelism: 2
          completions: 2
          backoffLimit: 0
          template:
            spec:
              containers:
                - name: worker
                  image: registry.access.redhat.com/ubi9/ubi-minimal:latest
                  command:
                    - sh
                    - -c
                    - |
                      echo "JobSet worker started"
                      echo "Hostname: $(hostname)"
                      sleep 30
                      echo "Worker complete"
                  resources:
                    requests:
                      cpu: "250m"
                      memory: "128Mi"
              restartPolicy: OnFailure
EOF

    This creates 3 Jobs (simple-jobset-workers-0, simple-jobset-workers-1, simple-jobset-workers-2), each running 2 Pods.

  2. Watch the Jobs and Pods:

    oc get jobs -n jobset-demo -l jobset.sigs.k8s.io/jobset-name=simple-jobset

  3. View the Pods across all Jobs:

    oc get pods -n jobset-demo -l jobset.sigs.k8s.io/jobset-name=simple-jobset

    You should see 6 Pods total (3 Jobs x 2 Pods each).

  4. Check the JobSet status:

    oc get jobsets -n jobset-demo

Exercise 2: Multi-Role JobSet with Leader and Workers

A common pattern in distributed computing is having distinct roles — a leader (or driver) and workers. JobSet supports this by allowing multiple entries in replicatedJobs with different templates.

  1. Create a JobSet with a leader and workers:

    cat <<EOF | oc apply -f -
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: leader-workers
  namespace: jobset-demo
spec:
  network:
    enableDNSHostnames: true
  coordinator:
    replicatedJob: leader
    jobIndex: 0
    podIndex: 0
  replicatedJobs:
    - name: leader
      replicas: 1
      template:
        spec:
          parallelism: 1
          completions: 1
          backoffLimit: 0
          template:
            spec:
              containers:
                - name: leader
                  image: registry.access.redhat.com/ubi9/ubi-minimal:latest
                  command:
                    - sh
                    - -c
                    - |
                      echo "Leader started at $(hostname)"
                      echo "Waiting for workers to connect..."
                      sleep 45
                      echo "Leader work complete"
                  resources:
                    requests:
                      cpu: "500m"
                      memory: "256Mi"
              restartPolicy: OnFailure
    - name: workers
      replicas: 1
      template:
        spec:
          parallelism: 4
          completions: 4
          backoffLimit: 0
          template:
            spec:
              containers:
                - name: worker
                  image: registry.access.redhat.com/ubi9/ubi-minimal:latest
                  command:
                    - sh
                    - -c
                    - |
                      LEADER=$(cat /etc/hostname | sed 's/-workers-.*/-leader-0-0/')
                      echo "Worker $(hostname) connecting to leader at ${LEADER}"
                      sleep 40
                      echo "Worker $(hostname) complete"
                  resources:
                    requests:
                      cpu: "250m"
                      memory: "128Mi"
              restartPolicy: OnFailure
EOF

    Key elements:

    • The coordinator field designates the leader Pod as the coordinator.

    • enableDNSHostnames: true ensures all Pods get DNS entries for peer discovery.

    • The leader and workers are defined as separate replicatedJobs entries with different resource requirements.

  2. Verify both Job groups are running:

    oc get jobs -n jobset-demo -l jobset.sigs.k8s.io/jobset-name=leader-workers

    You should see two Jobs: leader-workers-leader-0 and leader-workers-workers-0.

  3. Check the Pods and their roles:

    oc get pods -n jobset-demo -l jobset.sigs.k8s.io/jobset-name=leader-workers \
  -o custom-columns=NAME:.metadata.name,ROLE:.metadata.labels.jobset\\.sigs\\.k8s\\.io/replicatedjob-name,STATUS:.status.phase

Exercise 3: Failure Policy and Automatic Restarts

JobSet can automatically restart all Jobs when any Job fails. This is essential for distributed workloads where partial state from a failed run is invalid.

  1. Create a JobSet with a failure policy that allows 2 restarts:

    cat <<EOF | oc apply -f -
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: restart-demo
  namespace: jobset-demo
spec:
  failurePolicy:
    maxRestarts: 2
  replicatedJobs:
    - name: workers
      replicas: 2
      template:
        spec:
          parallelism: 1
          completions: 1
          backoffLimit: 0
          template:
            spec:
              containers:
                - name: worker
                  image: registry.access.redhat.com/ubi9/ubi-minimal:latest
                  command:
                    - sh
                    - -c
                    - |
                      echo "Worker $(hostname) attempt"
                      if [ "$((RANDOM % 3))" -eq "0" ]; then
                        echo "Simulated failure!"
                        exit 1
                      fi
                      sleep 20
                      echo "Worker complete"
                  resources:
                    requests:
                      cpu: "100m"
                      memory: "64Mi"
              restartPolicy: Never
EOF

    If any worker fails, JobSet recreates all Jobs (not just the failed one) up to maxRestarts times.

  2. Watch the JobSet events to see restart behavior:

    oc get events -n jobset-demo --field-selector involvedObject.name=restart-demo --sort-by=.lastTimestamp

  3. Check the JobSet status for restart count:

    oc get jobset restart-demo -n jobset-demo -o yaml | grep -A 5 "status:"

Kueue Integration

JobSet integrates with Kueue for admission control and quota management. When Kueue is installed, JobSets are held in a queue until the ClusterQueue has enough capacity to run all Jobs simultaneously — providing gang-scheduling-like behavior through admission control.

Prerequisites

This section assumes you have Kueue installed and a LocalQueue configured. If you completed the Kueue module, you can reuse the same queue setup.

  1. Set up the Kueue resources (skip if already created):

    cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: default-flavor
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: cluster-queue
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["cpu", "memory"]
      flavors:
        - name: default-flavor
          resources:
            - name: "cpu"
              nominalQuota: 9
            - name: "memory"
              nominalQuota: 36Gi
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: jobset-demo
  name: user-queue
spec:
  clusterQueue: cluster-queue
EOF

Submit a JobSet to a Kueue Queue

The integration point is the kueue.x-k8s.io/queue-name label in the JobSet’s metadata. Kueue calculates the total resource requirements across all replicated Jobs, considering replicas, parallelism, and per-container requests.

  1. Create a JobSet managed by Kueue:

    cat <<EOF | oc apply -f -
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  generateName: queued-jobset-
  namespace: jobset-demo
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  network:
    enableDNSHostnames: false
    subdomain: queued-jobset
  replicatedJobs:
    - name: workers
      replicas: 1
      template:
        spec:
          parallelism: 2
          completions: 2
          backoffLimit: 0
          template:
            spec:
              containers:
                - name: worker
                  image: registry.access.redhat.com/ubi9/ubi-minimal:latest
                  command:
                    - sh
                    - -c
                    - |
                      echo "Kueue-managed worker $(hostname)"
                      sleep 30
                      echo "Done"
                  resources:
                    requests:
                      cpu: "1"
                      memory: "512Mi"
              restartPolicy: Never
    - name: driver
      template:
        spec:
          parallelism: 1
          completions: 1
          backoffLimit: 0
          template:
            spec:
              containers:
                - name: driver
                  image: registry.access.redhat.com/ubi9/ubi-minimal:latest
                  command:
                    - sh
                    - -c
                    - |
                      echo "Kueue-managed driver $(hostname)"
                      sleep 30
                      echo "Done"
                  resources:
                    requests:
                      cpu: "2"
                      memory: "512Mi"
              restartPolicy: Never
EOF

    The kueue.x-k8s.io/queue-name: user-queue label tells Kueue to manage this JobSet. Kueue will calculate the total resource need (2 worker Pods x 1 CPU + 1 driver Pod x 2 CPU = 4 CPUs) and admit the JobSet only when the ClusterQueue has enough quota.

  2. Check the Kueue workload status:

    oc get workloads -n jobset-demo

    The workload should show as ADMITTED if there is enough quota available.

  3. Verify the Jobs are running:

    oc get jobs -n jobset-demo
oc get pods -n jobset-demo

Observe Queuing Behavior

Submit a second JobSet that will exceed the available quota to see Kueue hold it in the queue.

  1. Submit a larger JobSet:

    cat <<EOF | oc apply -f -
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  generateName: large-jobset-
  namespace: jobset-demo
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  replicatedJobs:
    - name: workers
      replicas: 2
      template:
        spec:
          parallelism: 4
          completions: 4
          backoffLimit: 0
          template:
            spec:
              containers:
                - name: worker
                  image: registry.access.redhat.com/ubi9/ubi-minimal:latest
                  command: ["sh", "-c", "sleep 60"]
                  resources:
                    requests:
                      cpu: "1"
                      memory: "512Mi"
              restartPolicy: Never
EOF

  2. Check the workload queue — the larger JobSet should be waiting:

    oc get workloads -n jobset-demo

    The second workload needs 8 CPUs (2 replicas x 4 parallel Pods x 1 CPU), which exceeds the remaining quota. It will be admitted once the first JobSet completes and releases its resources.

Priority with Kueue

You can set workload priority by adding a priorityClassName to the Pod spec in any of the replicatedJobs entries. Kueue uses the first non-empty priorityClassName it finds across the replicated Jobs to determine the workload priority:

spec:
  replicatedJobs:
    - name: workers
      template:
        spec:
          template:
            spec:
              priorityClassName: high-priority
              containers:
                - name: worker
                  ...

JobSet vs Traditional HPC Job Arrays

If you come from an HPC background, JobSet maps to several familiar concepts:

HPC Concept JobSet Equivalent

Job array (--array=0-N)

replicatedJobs with replicas: N

Multiple job steps

Multiple entries in replicatedJobs

All-or-nothing restart

failurePolicy.maxRestarts

Hostfile / peer discovery

Automatic DNS via headless Service

Master/worker pattern

coordinator with separate leader and worker replicatedJobs

Clean Up

  1. Remove all resources created in these exercises:

    oc delete jobsets --all -n jobset-demo
oc delete localqueues user-queue -n jobset-demo
oc delete clusterqueues cluster-queue
oc delete resourceflavors default-flavor

Summary

JobSet provides a Kubernetes-native API for managing groups of related Jobs as a single coordinated unit.

In this module you:

  • Understood why JobSet is needed — Coordinated lifecycle, automatic DNS, and unified failure handling for multi-job workloads.

  • Created a basic JobSet with replicated workers.

  • Built a multi-role JobSet with a leader and workers using the coordinator feature.

  • Explored failure policies and automatic restart behavior.

  • Integrated JobSet with Kueue for admission control and quota-based scheduling.

  • Compared JobSet concepts to traditional HPC job arrays and steps.

In the next section, you will learn about Distributed Training with TrainJob for distributed training workloads.