Batch Computing on OpenShift
Welcome to the Batch Computing on OpenShift workshop.
This workshop teaches you how to run high-performance computing (HPC), batch, and distributed computing workloads on Red Hat OpenShift. You will work hands-on with cloud-native scheduling, queuing, and framework operators that bring traditional HPC capabilities to Kubernetes.
Who is this for?
-
Platform engineers evaluating OpenShift for HPC and batch workloads
-
HPC administrators modernizing scheduling infrastructure
-
Data engineers and scientists running distributed computing frameworks
-
Anyone exploring how Kubernetes-native tools compare to traditional HPC schedulers
What you will learn
This workshop is organized into progressive sections. Each section introduces concepts, explains why the technology matters for batch and HPC, and provides hands-on exercises.
Foundations
-
Introduction to HPC and Batch Computing — How batch and HPC workloads differ from traditional containerized applications, the role of the Kubernetes scheduler, and an overview of the ecosystem.
Scheduling and Queuing
-
Kueue: Scheduling and Queuing — Set up cluster queues, submit batch jobs, configure priority and preemption, and understand how Kueue compares to traditional HPC schedulers.
-
Secondary Schedulers and Coscheduling — Deploy the coscheduling plugin for gang scheduling, create PodGroups, and understand when and why you need a secondary scheduler.
-
Multikueue: Multi-Cluster Scheduling — Extend Kueue across multiple clusters using Red Hat Advanced Cluster Management for multi-cluster batch job dispatch.
Frameworks and Operators
-
Distributed Training with TrainJob — Run distributed training jobs (PyTorch, MPI) using the TrainJob API.
-
Apache Spark on OpenShift — Deploy and run Spark applications using the Spark Operator with Kueue integration.
-
Ray on OpenShift — Deploy Ray clusters and submit RayJobs for distributed computing workloads.
Traditional HPC on OpenShift
-
Slurm on OpenShift — Run Slurm workload manager on OpenShift using the Slinky project, bridging traditional HPC and cloud-native platforms.
-
Flux Framework on OpenShift — Deploy LLNL’s next-generation Flux scheduler on OpenShift via the Flux Operator, with hierarchical scheduling and graph-based resource management.