Multikueue: Multi-Cluster Scheduling with ACM

When you have multiple OpenShift clusters — on-premises, in the cloud, or across regions — you want batch jobs to use capacity across the fleet rather than being limited to a single cluster. Multikueue extends Kueue to multi-cluster: a single logical queue can distribute workloads across many clusters.

Together with Red Hat Advanced Cluster Management (ACM), Multikueue provides a full multi-cluster scheduling story. ACM manages the cluster fleet, while Multikueue handles queue-based admission and job dispatch across that fleet.

Why Multi-Cluster Scheduling?

Single-cluster batch scheduling works well until you hit one of these situations:

Capacity limits — Your on-premises cluster cannot handle peak batch load. You want to burst into cloud clusters for overflow.
Data locality — Some jobs need to run near specific data sources. Different clusters may be better positioned for different workloads.
Cost optimization — Spot or preemptible instances in the cloud are cheaper for batch workloads. You want to dispatch to the cheapest available capacity.
Resilience — If one cluster is down for maintenance, batch work continues on other clusters.

Multikueue solves this by keeping the user experience simple: submit a job to a single queue, and the system dispatches it to the best available cluster.

Architecture

Multikueue uses a manager cluster and one or more worker clusters:

Manager Cluster

Runs Kueue and the Multikueue controllers.
Users submit jobs here to LocalQueues backed by ClusterQueues configured for Multikueue.
The manager holds the queue, reserves quota, and dispatches workloads to worker clusters.
It creates and monitors remote Workloads on workers and keeps status in sync.

Worker Clusters

Each worker is a normal Kueue cluster with its own capacity and queues.
The manager copies Workloads (and when one is admitted, the corresponding Job) to the chosen worker.
The first worker to admit a workload "wins"; the manager removes the workload from other workers and tracks the job until completion.

From the user’s perspective: submit a job to the manager as usual. Kueue queues it; when quota is available, Multikueue sends it to one or more workers; one worker admits it and runs the job. No change to the job spec is required.

Dispatching Strategies

Multikueue supports several dispatching strategies for choosing which worker clusters see a workload:

Strategy	Behavior
AllAtOnce (default)	The workload is sent to all worker clusters as soon as it has quota in the manager. The first cluster to admit it runs the job; others are cleaned up. Minimizes admission latency.
Incremental	Worker clusters are tried in rounds. If no cluster admits within a time window, more clusters are added in the next round. Reduces load on worker clusters when many jobs are queued.
External	An external controller sets which worker clusters are nominated for each workload. Multikueue only dispatches to those clusters. This is where ACM integration fits.

Strategy

Behavior

AllAtOnce (default)

The workload is sent to all worker clusters as soon as it has quota in the manager. The first cluster to admit it runs the job; others are cleaned up. Minimizes admission latency.

Incremental

Worker clusters are tried in rounds. If no cluster admits within a time window, more clusters are added in the next round. Reduces load on worker clusters when many jobs are queued.

External

An external controller sets which worker clusters are nominated for each workload. Multikueue only dispatches to those clusters. This is where ACM integration fits.

ACM Integration

Red Hat Advanced Cluster Management for Kubernetes (ACM) provides multi-cluster lifecycle, governance, and observability. Built on Open Cluster Management (OCM), it exposes concepts such as ManagedClusters, Placement, and PlacementDecision.

How ACM and Multikueue Work Together

Cluster discovery and management — ACM discovers and manages your OpenShift clusters as managed clusters. Those clusters are registered as Multikueue worker clusters.
Placement-driven selection — OCM Placement selects clusters based on predicates and priorities (e.g., region, labels, capacity). With Multikueue’s External dispatching mode, a controller reads PlacementDecisions and sets the nominated worker clusters for each workload.
Single control plane — Users submit batch jobs to the manager cluster (typically the ACM hub). Kueue and Multikueue handle queuing and cross-cluster dispatch; ACM handles cluster lifecycle and placement policies.

Exercise 1: Understanding the Multikueue Configuration

This exercise requires a multi-cluster environment with ACM installed on the hub cluster and at least one managed cluster. If your lab environment does not have multiple clusters, follow along with the YAML manifests to understand the configuration.

Manager cluster setup

The manager cluster needs Kueue installed with Multikueue enabled, and it needs connectivity to worker clusters.

Verify Kueue is running on the manager cluster:
```
oc get pods -n kueue-system
```

Create an AdmissionCheck that tells Kueue to use Multikueue for admission:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
  name: multikueue-check
spec:
  controllerName: kueue.x-k8s.io/multikueue
  parameters:
    apiGroup: kueue.x-k8s.io
    kind: MultiKueueConfig
    name: multikueue-config
EOF

Create a MultiKueueConfig that defines the dispatching strategy and worker clusters:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: MultiKueueConfig
metadata:
  name: multikueue-config
spec:
  clusters:
    - name: worker-cluster-1
    - name: worker-cluster-2
EOF

Create MultiKueueCluster resources that define the connection to each worker:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: MultiKueueCluster
metadata:
  name: worker-cluster-1
spec:
  kubeConfig:
    locationType: Secret
    location: worker-cluster-1-kubeconfig
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: MultiKueueCluster
metadata:
  name: worker-cluster-2
spec:
  kubeConfig:
    locationType: Secret
    location: worker-cluster-2-kubeconfig
EOF

Each MultiKueueCluster points to a Secret containing the kubeconfig for that worker cluster.

Create a ClusterQueue with Multikueue

Create a ClusterQueue that uses the Multikueue admission check:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: multikueue-cluster-queue
spec:
  namespaceSelector: {}
  admissionChecks:
    - multikueue-check
  resourceGroups:
    - coveredResources: ["cpu", "memory"]
      flavors:
        - name: default-flavor
          resources:
            - name: "cpu"
              nominalQuota: 100
            - name: "memory"
              nominalQuota: 200Gi
EOF

Create a LocalQueue in your namespace:

cat <<EOF | oc apply -f -
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  namespace: batch-demo
  name: multikueue-queue
spec:
  clusterQueue: multikueue-cluster-queue
EOF

Worker cluster setup

On each worker cluster, you need Kueue running with matching queue names.

The worker clusters need:

Kueue installed
A matching ClusterQueue and LocalQueue (same names as on the manager) so that the manager’s dispatched workloads find a queue
Sufficient capacity to run the batch jobs

# On each worker cluster
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: multikueue-cluster-queue
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources: ["cpu", "memory"]
      flavors:
        - name: default-flavor
          resources:
            - name: "cpu"
              nominalQuota: 50
            - name: "memory"
              nominalQuota: 100Gi

Exercise 2: Submitting a Job to Multikueue

With the manager and worker clusters configured, submitting a job looks identical to single-cluster Kueue. The user does not need to know or specify which cluster the job will run on.

Submit a batch job to the Multikueue-backed queue:

cat <<EOF | oc apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  namespace: batch-demo
  name: multi-cluster-job
  labels:
    kueue.x-k8s.io/queue-name: multikueue-queue
spec:
  parallelism: 5
  completions: 5
  template:
    spec:
      containers:
        - name: worker
          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
          command:
            - sh
            - -c
            - |
              echo "Running on cluster: $(hostname)"
              echo "Started at: $(date)"
              sleep 60
              echo "Completed at: $(date)"
          resources:
            requests:
              cpu: "500m"
              memory: "256Mi"
      restartPolicy: Never
EOF

Check the workload status on the manager:
```
oc get workloads -n batch-demo
```
See which worker cluster admitted the job by checking the workload’s admission status:
```
oc get workload -n batch-demo -o yaml | grep -A5 "admission"
```
The admission status shows which cluster was selected and is running the workload.

Using ACM Placement for Cluster Selection

When using the External dispatching strategy, you can use ACM Placement to control which clusters are eligible for batch workloads.

Example: Region-based placement

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: batch-placement
  namespace: batch-demo
spec:
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchLabels:
            region: us-east
            gpu: "true"
  numberOfClusters: 2

This Placement selects up to 2 clusters in the us-east region that have GPUs. An external controller can read the resulting PlacementDecision and set the nominated clusters on the Multikueue workload, so that only those clusters receive the dispatched job.

Summary

Multikueue extends Kueue to multi-cluster batch scheduling.

In this module you learned:

Why multi-cluster scheduling matters — Capacity bursting, data locality, cost optimization, and resilience.
Manager and worker architecture — The manager queues and dispatches; workers admit and run.
Dispatching strategies — AllAtOnce, Incremental, and External (for ACM integration).
ACM integration — Using Placement and PlacementDecision to control which clusters receive batch workloads.
Configuration — AdmissionCheck, MultiKueueConfig, and MultiKueueCluster resources on the manager; matching queues on workers.

The user experience stays simple: submit a job to a queue, and Multikueue finds the right cluster.

In the next section, you will learn about Distributed Training with TrainJob for distributed training workloads.