Kueue: Scheduling and Queuing for Batch Workloads

Kueue is a Kubernetes-native job queue that controls how batch and HPC workloads are admitted to the cluster. This module introduces why a queue layer is needed on top of the Kubernetes scheduler, outlines the core concepts, and walks you through hands-on exercises to configure queues, submit jobs, and work with priority and preemption.

Why Kueue Is Needed

The default Kubernetes scheduler places Pods onto nodes as soon as resources are available. For batch and HPC jobs, that behavior leads to problems:

No admission control — Every submitted job competes immediately for capacity. A large batch job can starve smaller jobs or cause resource fragmentation.
No fair sharing — There is no built-in way to give different teams or projects a share of cluster capacity or to prioritize certain workloads.
No queue semantics — Users expect to submit jobs to a queue and have them start when capacity allows, rather than having jobs fail or sit unscheduled indefinitely.
Mismatch with batch lifecycle — Batch jobs have a clear start and end. The system should hold them until there is enough capacity to run them (often with gang scheduling), then release them as a unit.

Kueue adds a queue layer between job submission and the scheduler. It holds workloads in queues, respects quotas and capacity, and admits them only when the cluster can run them. The Kubernetes scheduler then places the admitted Pods.

Core Concepts

Kueue organizes capacity and workloads around a few key resources. Understanding these gives you the foundation for configuring and using Kueue on OpenShift.

ResourceFlavor

A ResourceFlavor describes what kind of capacity exists in the cluster. It is usually tied to node characteristics:

Hardware — GPU model, CPU architecture (e.g., x86 vs ARM).
Availability — On-demand vs spot or preemptible.
Cost or tier — Different "flavors" for different pricing or SLA.

ClusterQueues define quotas per ResourceFlavor. When a workload requests, for example, "4 GPUs," Kueue matches that to a flavor and checks whether the ClusterQueue has quota for that flavor.

ClusterQueue

A ClusterQueue is a cluster-scoped resource that represents a pool of capacity. It defines:

Resource quotas — How much of each resource (CPU, memory, GPUs, etc.) this queue can use, broken down by ResourceFlavor.
Fair sharing — How capacity is shared among multiple ClusterQueues (via cohorts and borrowing limits).
Admission rules — Which resource flavors can be used together and in what order.

ClusterQueues are created and managed by cluster or batch administrators. Users do not submit jobs directly to a ClusterQueue; they submit to a LocalQueue.

LocalQueue

A LocalQueue is a namespace-scoped resource that represents a queue for a tenant, team, or project. It:

References a ClusterQueue — All workloads in the LocalQueue draw capacity from that ClusterQueue.
Groups related workloads — Users submit jobs to a LocalQueue so that workloads are organized by ownership or purpose.

Workload

In Kueue, a Workload is the unit of work that is queued and admitted. For a Kubernetes Job, Kueue creates a corresponding Workload that represents that job in the queue. The Workload tracks the job’s resource requests, which LocalQueue (and thus ClusterQueue) it belongs to, and its status (queued, admitted, finished).

Admission

Admission is the process by which Kueue decides that a workload is allowed to run. Until a workload is admitted, its Pods are not created or are held in a pending state.

Admission typically involves:

Quota check — Does the ClusterQueue have enough quota (for the requested ResourceFlavors) for this workload?
Admission checks (optional) — Custom checks (e.g., cluster capacity, external approvals) that must pass before the workload can start.

Once admitted, the workload consumes quota from the ClusterQueue until it finishes; then the quota is released for the next workload in the queue.

Exercise 1: Setting Up Kueue Queues

In this exercise you will create the foundational Kueue resources: a ResourceFlavor, a ClusterQueue, and a LocalQueue. This is the equivalent of setting up a batch queue on a traditional HPC cluster.

Create a namespace

Create a namespace for your batch workloads:
```
oc new-project batch-demo
```

Create a ResourceFlavor

A ResourceFlavor with no node labels or taints represents "default" cluster capacity — any available node.

Create a default ResourceFlavor:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: default-flavor
EOF

Create a ClusterQueue

The ClusterQueue defines the capacity pool. Here we allocate 9 CPUs and 36Gi of memory to this queue.

Create the ClusterQueue:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: cluster-queue
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["cpu", "memory"]
      flavors:
        - name: default-flavor
          resources:
            - name: "cpu"
              nominalQuota: 9
            - name: "memory"
              nominalQuota: 36Gi
EOF

Verify the ClusterQueue was created:
```
oc get clusterqueues
```

Create a LocalQueue

The LocalQueue is where users submit jobs. It lives in a namespace and points to a ClusterQueue.

Create a LocalQueue in the batch-demo namespace:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: batch-demo
  name: user-queue
spec:
  clusterQueue: cluster-queue
EOF

Verify the LocalQueue is ready:
```
oc get localqueues -n batch-demo
```
The ADMITTED column should show 0 and the queue should be active.

Exercise 2: Submitting Batch Jobs

Now that the queue infrastructure is in place, you can submit batch jobs. Kueue intercepts the job and holds it until the ClusterQueue has sufficient quota to admit it.

Submit a simple job

Submit a batch job that runs a short computation:

cat <<EOF | oc apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  namespace: batch-demo
  name: sample-job-1
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  parallelism: 3
  completions: 3
  template:
    spec:
      containers:
        - name: worker
          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
          command: ["sh", "-c", "echo 'Processing batch item'; sleep 30; echo 'Done'"]
          resources:
            requests:
              cpu: "1"
              memory: "512Mi"
      restartPolicy: Never
EOF

Notice the label kueue.x-k8s.io/queue-name: user-queue — this is how Kueue knows which LocalQueue the job belongs to.

Watch the job’s admission status:
```
oc get workloads -n batch-demo
```
You should see a Workload object corresponding to your job. The ADMITTED column shows whether Kueue has released the job to the scheduler.
Check the job’s Pods:
```
oc get pods -n batch-demo -l job-name=sample-job-1
```
All 3 Pods should be Running (or Completed if the sleep has finished).

Submit jobs that exceed capacity

To see Kueue’s queuing behavior, submit a second job that would exceed the ClusterQueue’s quota when combined with the first.

Submit a larger job:

cat <<EOF | oc apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  namespace: batch-demo
  name: sample-job-2
  labels:
    kueue.x-k8s.io/queue-name: user-queue
spec:
  parallelism: 8
  completions: 8
  template:
    spec:
      containers:
        - name: worker
          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
          command: ["sh", "-c", "echo 'Processing large batch'; sleep 60; echo 'Done'"]
          resources:
            requests:
              cpu: "1"
              memory: "1Gi"
      restartPolicy: Never
EOF

Check the workload status:
```
oc get workloads -n batch-demo
```
If sample-job-1 is still running, sample-job-2 may be held in the queue (not admitted) because admitting it would exceed the ClusterQueue’s 9-CPU quota. Once sample-job-1 completes and releases its quota, Kueue admits sample-job-2.
Watch the queue drain:
```
watch oc get workloads -n batch-demo
```
Press Ctrl+C when both jobs show as admitted or finished.

Exercise 3: Priority and Preemption

In traditional HPC schedulers, job priority determines which jobs run first when resources are scarce, and preemption allows high-priority jobs to evict lower-priority ones. Kueue provides similar capabilities through WorkloadPriorityClass and ClusterQueue preemption policies.

Create priority classes

Create a low-priority and a high-priority WorkloadPriorityClass:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
  name: low-priority
value: 100
description: "Low priority batch jobs"
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: WorkloadPriorityClass
metadata:
  name: high-priority
value: 1000
description: "High priority batch jobs"
EOF

Enable preemption on the ClusterQueue

Update the ClusterQueue to enable preemption so that higher-priority workloads can evict lower-priority ones:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: cluster-queue
spec:
  namespaceSelector: {}
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  resourceGroups:
    - coveredResources: ["cpu", "memory"]
      flavors:
        - name: default-flavor
          resources:
            - name: "cpu"
              nominalQuota: 9
            - name: "memory"
              nominalQuota: 36Gi
EOF

The key setting is withinClusterQueue: LowerPriority, which tells Kueue it can preempt lower-priority workloads within the same ClusterQueue to make room for higher-priority ones.

Test preemption

First, clean up any previous jobs:
```
oc delete jobs --all -n batch-demo
```

Submit a low-priority job that fills the queue:

cat <<EOF | oc apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  namespace: batch-demo
  name: low-priority-job
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    kueue.x-k8s.io/priority-class: low-priority
spec:
  parallelism: 9
  completions: 9
  template:
    spec:
      containers:
        - name: worker
          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
          command: ["sh", "-c", "echo 'Low priority work'; sleep 120; echo 'Done'"]
          resources:
            requests:
              cpu: "1"
              memory: "1Gi"
      restartPolicy: Never
EOF

Wait for the low-priority job to be admitted:
```
oc get workloads -n batch-demo
```

Now submit a high-priority job:

cat <<EOF | oc apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  namespace: batch-demo
  name: high-priority-job
  labels:
    kueue.x-k8s.io/queue-name: user-queue
    kueue.x-k8s.io/priority-class: high-priority
spec:
  parallelism: 3
  completions: 3
  template:
    spec:
      containers:
        - name: worker
          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
          command: ["sh", "-c", "echo 'HIGH priority work'; sleep 30; echo 'Done'"]
          resources:
            requests:
              cpu: "1"
              memory: "1Gi"
      restartPolicy: Never
EOF

Watch Kueue preempt the low-priority workload to make room:
```
oc get workloads -n batch-demo
oc get pods -n batch-demo
```
You should see some Pods from low-priority-job being evicted as Kueue reclaims quota for the high-priority job. Once high-priority-job finishes, the low-priority workload can be re-admitted.

Kueue vs Traditional HPC Schedulers

If you come from a traditional HPC background (Slurm, PBS, Grid Engine, LSF), it is helpful to understand what Kueue provides today and where it differs from the primitives you may be accustomed to.

What Kueue provides

HPC Scheduler Concept	Kueue Equivalent
Job queue (submit, hold, release)	LocalQueue + ClusterQueue admission
Resource quotas per queue	ClusterQueue resourceGroups with nominalQuota
Fair share scheduling	Cohorts with borrowing and lending limits
Job priority	WorkloadPriorityClass
Preemption	ClusterQueue preemption policies
Multi-cluster job dispatch	Multikueue (see Multikueue)
Resource types/partitions	ResourceFlavor (node selectors, taints)

HPC Scheduler Concept

Kueue Equivalent

Job queue (submit, hold, release)

LocalQueue + ClusterQueue admission

Resource quotas per queue

ClusterQueue resourceGroups with nominalQuota

Fair share scheduling

Cohorts with borrowing and lending limits

Job priority

WorkloadPriorityClass

Preemption

ClusterQueue preemption policies

Multi-cluster job dispatch

Multikueue (see Multikueue)

Resource types/partitions

ResourceFlavor (node selectors, taints)

What Kueue does NOT provide (yet)

Traditional HPC schedulers offer primitives that are not yet available in Kueue or Kubernetes:

Backfill scheduling — HPC schedulers like Slurm can estimate when running jobs will finish and "backfill" smaller jobs into the gaps. Kueue does not estimate job runtimes or perform backfill.
Job arrays with dependencies — Slurm job arrays (--array) and job dependency chains (--dependency=afterok:JOBID) have no direct Kueue equivalent. You can use Kubernetes Job indexed completions and Argo Workflows for similar patterns, but they are not integrated with Kueue’s admission.
Reservations and advance booking — Reserving capacity for a future time window is common in HPC but not available in Kueue.
Detailed accounting and charge-back — HPC schedulers track CPU-hours per user/project for charge-back. Kueue tracks quota usage but does not provide accounting in the traditional sense.
Interactive job sessions — srun or qsub -I style interactive sessions are not a Kueue concept.
Fine-grained resource topology — HPC schedulers understand NUMA topology, socket/core placement, and network fabric. Kueue delegates placement to the Kubernetes scheduler, which has limited topology awareness.

Understanding these gaps helps you evaluate where Kueue fits your workloads today and where you may need complementary tooling or patience as the project matures.

Clean Up

Remove all jobs and resources created in these exercises:

oc delete jobs --all -n batch-demo
oc delete localqueues user-queue -n batch-demo
oc delete clusterqueues cluster-queue
oc delete resourceflavors default-flavor
oc delete workloadpriorityclasses low-priority high-priority

Summary

Kueue provides scheduling and queuing for batch workloads on Kubernetes and OpenShift by introducing a queue layer with quotas, flavors, and admission control. In this module you:

Created a ResourceFlavor, ClusterQueue, and LocalQueue — the building blocks of Kueue’s queue infrastructure.
Submitted batch jobs and observed how Kueue admits them based on available quota.
Configured priority classes and preemption to control which jobs run first when resources are scarce.
Compared Kueue’s capabilities to traditional HPC schedulers to understand what is available today and what gaps remain.

In the next section, you will learn about secondary schedulers and the coscheduling plugin for gang scheduling.