Slurm on OpenShift

Slurm (Simple Linux Utility for Resource Management) is the most widely used workload manager in HPC. It provides job scheduling, resource management, and accounting for compute clusters, and is the standard in national labs, universities, and enterprises running HPC workloads.

The Slinky project brings Slurm to Kubernetes and OpenShift, running the Slurm control plane and compute nodes as containers on the platform. This module introduces the architecture, use cases, and practical considerations for running Slurm on OpenShift.

Running Slurm on OpenShift via the Slinky project is not a supported Red Hat product. It is an open-source community project for exploration and evaluation. Production HPC workloads running on Slurm should be evaluated carefully for supportability, and organizations should engage with Red Hat to understand the support boundaries. This workshop section is provided for educational and evaluation purposes.

Why Run Slurm on OpenShift?

Organizations considering Slurm on OpenShift typically fall into one of these categories:

  • HPC modernization — You have existing Slurm-based workflows, scripts, and user expectations. Running Slurm on OpenShift lets you modernize the infrastructure without rewriting job scripts or retraining users.

  • Hybrid workloads — You want both Kubernetes-native workloads (microservices, AI/ML inference) and traditional HPC batch jobs on the same platform, sharing the same hardware.

  • Operational unification — A single platform for your operations team to manage, with consistent security, networking, and monitoring — rather than maintaining separate Kubernetes and HPC clusters.

  • Evaluation — You want to understand what running Slurm on Kubernetes looks like before committing to a migration strategy.

Architecture

The Slinky project deploys Slurm components as Pods on OpenShift:

Slurm Control Plane

  • slurmctld — The central management daemon. Runs as a Pod (often a StatefulSet) with persistent storage for job accounting and state.

  • slurmdbd — The Slurm database daemon for job accounting. Connects to a database (typically MariaDB or MySQL) running as another Pod or an external service.

Compute Nodes

  • slurmd — The compute node daemon. Each slurmd Pod represents a compute node available for running jobs. These Pods register with slurmctld and report their resources (CPU, memory, GPUs).

Supporting Services

  • munge — Authentication service used by Slurm for secure communication between daemons. Runs as a sidecar or init container in each Slurm Pod.

  • Database — MariaDB or MySQL for slurmdbd accounting data.

Networking

Slurm daemons communicate over TCP. On OpenShift, Pod-to-Pod networking and headless Services provide the connectivity. Each slurmd Pod gets a stable DNS name so that slurmctld can reach compute nodes.

How It Differs from Bare-Metal Slurm

Running Slurm on OpenShift is not identical to running it on bare metal. Key differences:

Aspect Bare-Metal Slurm Slurm on OpenShift

Compute nodes

Physical or virtual machines with slurmd installed

slurmd Pods on OpenShift worker nodes

Resource management

Slurm directly manages CPU, memory, GPU on the node

Slurm manages resources within the Pod’s cgroup limits; OpenShift manages the underlying node

User access

SSH to login nodes, then sbatch/srun

Access via OpenShift terminal, port-forward, or a login Pod

Storage

Shared filesystem (NFS, Lustre, GPFS)

PersistentVolumes (NFS, CephFS, or other CSI drivers)

GPU support

Direct device access via gres.conf

GPU access via Kubernetes device plugin and resource requests

Node scaling

Add/remove physical machines

Scale slurmd StatefulSet replicas

Exercise 1: Exploring Slurm on OpenShift

This exercise requires the Slinky project to be deployed on your cluster. See slinky-on-openshift for deployment instructions.

Verify the Slurm deployment

  1. Check the Slurm control plane Pods:

    oc get pods -n slurm -l app=slurmctld
oc get pods -n slurm -l app=slurmdbd

  2. Check the compute node Pods:

    oc get pods -n slurm -l app=slurmd

    You should see multiple slurmd Pods, each representing a compute node.

  3. Access the Slurm login environment. You can exec into the slurmctld Pod or a dedicated login Pod:

    oc exec -it -n slurm deploy/slurmctld -- /bin/bash

Check cluster status

  1. Inside the Slurm environment, check the cluster info:

    sinfo

    You should see the partition name, node list, and state (idle, allocated, etc.).

  2. Check the node details:

    scontrol show nodes

    Each node corresponds to a slurmd Pod. Notice that the resources reported (CPUs, memory) match the Pod’s resource limits.

Exercise 2: Submitting Slurm Jobs

With the Slurm environment accessible, you can submit jobs using the standard Slurm commands that HPC users already know.

  1. Submit a simple batch job:

    sbatch <<'JOBSCRIPT'
#!/bin/bash
#SBATCH --job-name=hello-ocp
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=00:05:00

echo "Hello from Slurm on OpenShift!"
echo "Running on node: $(hostname)"
echo "Date: $(date)"
echo "Job ID: $SLURM_JOB_ID"
JOBSCRIPT

  2. Check the job queue:

    squeue

  3. Wait for the job to complete, then view the output:

    cat slurm-*.out

Submit a parallel job

  1. Submit a multi-node MPI-style job:

    sbatch <<'JOBSCRIPT'
#!/bin/bash
#SBATCH --job-name=parallel-test
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --time=00:05:00

echo "Parallel job starting on $(hostname)"
echo "SLURM_JOB_NODELIST: $SLURM_JOB_NODELIST"
echo "SLURM_NTASKS: $SLURM_NTASKS"

srun hostname
JOBSCRIPT

  2. Monitor the job:

    squeue

  3. View the results:

    cat slurm-*.out

    You should see hostnames from multiple slurmd Pods, confirming that the job ran across multiple compute nodes.

Check job accounting

  1. View completed job information:

    sacct --format=JobID,JobName,State,ExitCode,Elapsed,NodeList

  2. Exit the Slurm environment:

    exit

Exercise 3: Scaling Compute Nodes

One advantage of running Slurm on OpenShift is the ability to dynamically scale compute nodes by adjusting the slurmd StatefulSet.

  1. Check the current number of compute nodes:

    oc get statefulset -n slurm -l app=slurmd

  2. Scale up the compute nodes:

    oc scale statefulset slurmd -n slurm --replicas=4

  3. Watch the new slurmd Pods come up:

    oc get pods -n slurm -l app=slurmd -w

    Press Ctrl+C once all Pods are Running.

  4. Verify Slurm sees the new nodes:

    oc exec -n slurm deploy/slurmctld -- sinfo

    The new nodes should appear in the partition.

  5. Scale back down when done:

    oc scale statefulset slurmd -n slurm --replicas=2

Considerations and Limitations

Running Slurm on OpenShift involves trade-offs:

  • Supportability — Slinky is a community project. Red Hat does not provide commercial support for Slurm itself. The OpenShift platform underneath is supported, but Slurm issues are out of scope for Red Hat support.

  • Performance — Containerized Slurm adds a layer of abstraction. For latency-sensitive HPC workloads (e.g., tightly coupled MPI with high message rates), bare-metal Slurm with RDMA/InfiniBand may be necessary.

  • Storage — Shared filesystems are critical for HPC. You need a CSI driver that provides POSIX-compatible shared storage (NFS, CephFS, Lustre CSI).

  • Security context — Some Slurm operations require elevated privileges (e.g., cgroup management, user switching). OpenShift’s security context constraints (SCCs) may need adjustment.

  • GPU access — GPU scheduling works through the Kubernetes device plugin, which may differ from Slurm’s native gres.conf GPU management.

Summary

Slurm on OpenShift via the Slinky project provides a way to run the most widely used HPC workload manager on a cloud-native platform.

In this module you:

  • Understood the architecture — Slurm control plane and compute nodes running as Pods.

  • Submitted batch and parallel jobs using standard Slurm commands (sbatch, srun, squeue, sacct).

  • Scaled compute nodes dynamically by adjusting StatefulSet replicas.

  • Reviewed the considerations and limitations including supportability, performance, and storage.

This approach is valuable for evaluating how traditional HPC workflows can run on OpenShift and for organizations that want to maintain Slurm compatibility while modernizing their infrastructure.

In the next section, you will learn about the Flux Framework on OpenShift — a next-generation alternative to Slurm with hierarchical scheduling and graph-based resource management.