Secondary Schedulers and the Coscheduling Plugin

The default Kubernetes scheduler places Pods one at a time, as resources become available. For batch and HPC workloads that require gang scheduling — all Pods of a job scheduled together or none at all — this behavior is insufficient. Secondary schedulers and scheduler plugins extend or work alongside the kube-scheduler to provide gang semantics.

This module introduces why secondary schedulers are needed, how the coscheduling plugin and its PodGroup concept work, walks you through hands-on exercises, and outlines the drawbacks of the approach.

Why Secondary Schedulers Are Needed

The kube-scheduler is designed for independent placement: each Pod is evaluated on its own, and the first Pod that fits gets scheduled. For a distributed job that needs dozens or hundreds of Pods to run as a unit, this leads to:

Resource fragmentation — Some Pods get placed while others remain Pending. The job cannot make progress, but the scheduled Pods still hold resources.
Starvation and deadlock — Competing jobs can each get a subset of their Pods scheduled, so no job ever gets the full set it needs.
Wasted capacity — Partially scheduled gangs consume resources without producing useful work.

Secondary schedulers address this by implementing scheduling logic that the default scheduler does not provide. They can run as a separate process, or they can extend the kube-scheduler via the Kubernetes Scheduling Framework as a plugin.

On OpenShift, you can deploy a secondary scheduler using the Secondary Scheduler Operator.

How the Coscheduling Plugin Works

The coscheduling plugin is a scheduler plugin that provides gang scheduling by grouping Pods into a PodGroup and scheduling the group as a whole.

PodGroup

A PodGroup is a custom resource that represents a gang of Pods that must be scheduled together:

minMember — The minimum number of Pods that must be schedulable before any Pod in the group is scheduled.
Pod association — Pods are linked to the PodGroup via a label (scheduling.x-k8s.io/pod-group: <name>).
Status — The PodGroup status reflects whether the group has enough schedulable Pods.

Scheduling Flow

A controller or user creates a PodGroup with a name and minMember (e.g., 4 for a 4-Pod job).
Pods are created with a label pointing to the PodGroup.
Pods enter the scheduling queue. The coscheduling plugin checks whether at least minMember Pods from that PodGroup can all be placed.
If yes, the plugin allows the scheduler to bind those Pods. If no, the Pods stay pending until the next cycle.
Either the required number of Pods are scheduled together, or none are — gang scheduling.

Exercise 1: Deploying the Secondary Scheduler

On OpenShift, the Secondary Scheduler Operator allows you to run a second scheduler alongside the default kube-scheduler. The coscheduling plugin runs as part of this secondary scheduler.

This exercise assumes the Secondary Scheduler Operator is already installed on your cluster. If it is not, install it from OperatorHub.

Configure the secondary scheduler with the coscheduling plugin

Create a namespace for the secondary scheduler if one does not exist:
```
oc new-project secondary-scheduler
```

Create a KubeSchedulerConfiguration ConfigMap that enables the coscheduling plugin:

cat <<EOF | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: secondary-scheduler-config
  namespace: secondary-scheduler
data:
  config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    leaderElection:
      leaderElect: false
    profiles:
      - schedulerName: secondary-scheduler
        plugins:
          multiPoint:
            enabled:
              - name: Coscheduling
        pluginConfig:
          - name: Coscheduling
            args:
              permitWaitingTimeSeconds: 30
EOF

The permitWaitingTimeSeconds controls how long the scheduler waits for all gang members to become schedulable before giving up on that cycle.

Create a SecondaryScheduler custom resource to deploy the scheduler:

cat <<EOF | oc apply -f -
apiVersion: operator.openshift.io/v1
kind: SecondaryScheduler
metadata:
  name: cluster
  namespace: secondary-scheduler
spec:
  schedulerConfig: secondary-scheduler-config
  schedulerImage: registry.k8s.io/scheduler-plugins/kube-scheduler:v0.28.9
EOF

Verify the secondary scheduler Pod is running:
```
oc get pods -n secondary-scheduler
```
You should see a scheduler Pod in Running state.

Exercise 2: Gang Scheduling with PodGroups

Now that the secondary scheduler with the coscheduling plugin is running, you can create PodGroups and submit jobs that require gang scheduling.

Create a namespace for the exercises

Create or switch to the batch-demo namespace:

oc new-project batch-demo --skip-config-update || oc project batch-demo

Create a PodGroup

Create a PodGroup that requires 4 Pods to be scheduled together:
```
cat <<EOF | oc apply -f -
apiVersion: scheduling.x-k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: gang-job-group
  namespace: batch-demo
spec:
  minMember: 4
  scheduleTimeoutSeconds: 120
EOF
```
The minMember: 4 means all 4 Pods must be schedulable simultaneously, or none will be scheduled. The scheduleTimeoutSeconds sets how long the PodGroup waits before timing out entirely.
Verify the PodGroup:
```
oc get podgroups -n batch-demo
```

Submit a gang-scheduled job

Create a Job with 4 parallel Pods that use the secondary scheduler and reference the PodGroup:

cat <<EOF | oc apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: gang-job
  namespace: batch-demo
spec:
  parallelism: 4
  completions: 4
  template:
    metadata:
      labels:
        scheduling.x-k8s.io/pod-group: gang-job-group
    spec:
      schedulerName: secondary-scheduler
      containers:
        - name: worker
          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
          command:
            - sh
            - -c
            - |
              echo "Gang member $(hostname) started at $(date)"
              sleep 30
              echo "Gang member $(hostname) finished at $(date)"
          resources:
            requests:
              cpu: "500m"
              memory: "256Mi"
      restartPolicy: Never
EOF

Key elements:

schedulerName: secondary-scheduler — Routes these Pods to the coscheduling-enabled scheduler instead of the default.
scheduling.x-k8s.io/pod-group: gang-job-group — Associates each Pod with the PodGroup.

Watch the Pods. They should all start at approximately the same time:
```
oc get pods -n batch-demo -l scheduling.x-k8s.io/pod-group=gang-job-group -w
```
Press Ctrl+C once all Pods are Running or Completed.

Verify that all 4 Pods started together by comparing their start times:

oc get pods -n batch-demo -l scheduling.x-k8s.io/pod-group=gang-job-group \
  -o custom-columns=NAME:.metadata.name,START:.status.startTime,STATUS:.status.phase

The start times should be very close together, confirming gang scheduling behavior.

Check the PodGroup status:
```
oc get podgroup gang-job-group -n batch-demo -o yaml
```
Look at the status section — it shows the scheduling state of the group.

Exercise 3: Observing Gang Scheduling Failure

To see the all-or-nothing behavior, you can create a PodGroup that cannot be fully satisfied.

Create a PodGroup with a high minMember:

cat <<EOF | oc apply -f -
apiVersion: scheduling.x-k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: impossible-group
  namespace: batch-demo
spec:
  minMember: 100
  scheduleTimeoutSeconds: 60
EOF

Submit a job that only creates 3 Pods but requires 100 in the PodGroup:

cat <<EOF | oc apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: impossible-job
  namespace: batch-demo
spec:
  parallelism: 3
  completions: 3
  template:
    metadata:
      labels:
        scheduling.x-k8s.io/pod-group: impossible-group
    spec:
      schedulerName: secondary-scheduler
      containers:
        - name: worker
          image: registry.access.redhat.com/ubi9/ubi-minimal:latest
          command: ["sh", "-c", "echo 'This should never run'; sleep 10"]
          resources:
            requests:
              cpu: "100m"
              memory: "64Mi"
      restartPolicy: Never
EOF

Check the Pods — they should remain in Pending state because the gang condition (100 members) can never be met:
```
oc get pods -n batch-demo -l scheduling.x-k8s.io/pod-group=impossible-group
```

Check the scheduler events to see why:

oc get events -n batch-demo --field-selector reason=Unschedulable | tail -5

Clean up the impossible job:

oc delete job impossible-job -n batch-demo
oc delete podgroup impossible-group -n batch-demo

Drawbacks of Coscheduling

Using the coscheduling plugin for gang scheduling has several drawbacks to be aware of:

Not native to kube-scheduler — Coscheduling is an out-of-tree plugin with its own CRD. You must install and maintain the plugin and upgrade it as the Kubernetes Scheduling Framework evolves.
Coordination and correctness — The plugin must correctly count pending Pods, respect minMember, and avoid races where some Pods are bound and others are not. Edge cases (e.g., PodGroup deleted while Pods are pending, or partial failures) require careful handling.
Starvation and priority — If the cluster never has enough free capacity to schedule a large gang, the entire PodGroup stays pending indefinitely. Without integration with a queue (e.g., Kueue) or priority mechanisms, large jobs can block smaller ones.
Operator and framework dependency — Job frameworks and operators must create PodGroups and label Pods correctly. Misconfiguration leads to Pods that never schedule or that schedule without gang semantics.
Resource efficiency vs fairness — Holding back a whole gang until capacity is free can improve utilization by avoiding partially scheduled jobs, but it can reduce throughput if many small jobs could run in the gaps.

For these reasons, coscheduling is a practical way to get gang scheduling today, but native scheduler support combined with a queue layer (such as Kueue) is the direction the ecosystem is moving.

Clean Up

Remove all resources created in these exercises:

oc delete jobs --all -n batch-demo
oc delete podgroups --all -n batch-demo

Summary

Secondary schedulers and scheduler plugins address the need for gang scheduling on Kubernetes when the default scheduler does not support it natively.

In this module you:

Deployed the Secondary Scheduler Operator with the coscheduling plugin enabled.
Created PodGroups that define which Pods must be co-scheduled.
Submitted jobs that were gang-scheduled — all Pods started together.
Observed the all-or-nothing behavior when a gang condition could not be met.
Reviewed the drawbacks of coscheduling, including maintenance burden, starvation risks, and the need for queue integration.

In the next section, you will learn about Multikueue for multi-cluster batch scheduling with ACM.