Flux Framework on OpenShift

The Flux Framework is a next-generation HPC resource management and job scheduling framework developed at Lawrence Livermore National Laboratory (LLNL). Unlike traditional HPC schedulers that view resources as flat lists of nodes, Flux uses hierarchical resource management and graph-based scheduling to make smarter placement decisions across diverse hardware.

The Flux Operator brings Flux to Kubernetes and OpenShift by deploying MiniClusters — self-contained Flux instances that run as Pods on the platform. This is a key project in the converged computing movement, which aims to unite the best of HPC (performance, sophisticated scheduling) with the best of cloud-native platforms (elasticity, portability, manageability).

The Flux Operator is an open-source community project and is not a supported Red Hat product. It is provided here for evaluation and educational purposes. Organizations should assess supportability requirements before using it in production.

Why Flux?

If Slurm is the established standard for HPC scheduling, why consider Flux?

  • Hierarchical scheduling — Flux supports nested instances. A parent Flux instance can allocate resources to a child instance, which then schedules its own sub-jobs. This enables complex, multi-level workflows that are difficult to express in traditional batch schedulers.

  • Graph-based resource model — Instead of treating nodes as interchangeable slots, Flux represents the entire resource hierarchy (racks, nodes, sockets, cores, GPUs, network switches) as a directed graph. The scheduler uses this graph to make topology-aware placement decisions — for example, placing MPI ranks on cores that share an L3 cache or a network switch.

  • High-throughput scheduling — Flux is designed to handle millions of small jobs efficiently, making it suitable for ensemble computing, parameter sweeps, and workflows that submit many short-lived tasks.

  • Framework, not just a scheduler — Flux is extensible. You can plug in custom schedulers, resource types, and workflow services. It is designed to evolve with the hardware and workload landscape.

  • Cloud-native convergence — The Flux Operator brings these HPC capabilities into Kubernetes, so you can run Flux-managed workloads alongside other Kubernetes-native applications on the same OpenShift cluster.

Core Concepts

MiniCluster

A MiniCluster is the Flux Operator’s custom resource. It deploys a Flux instance as a set of Pods on Kubernetes:

  • A broker Pod (index 0) runs the Flux broker, which is the control point for job submission and scheduling.

  • Worker Pods join the broker to form the Flux cluster.

  • By default, the operator maps one Pod per node using affinity rules.

A MiniCluster can be ephemeral (run a single job and clean up) or interactive/persistent (stay running so you can submit multiple jobs).

Flux Commands

Flux uses its own set of commands, analogous to Slurm but with a different design philosophy:

Flux Command Slurm Equivalent Description

flux run

srun

Run a job interactively (blocking)

flux submit

sbatch

Submit a job to the queue (non-blocking)

flux jobs

squeue

List jobs and their status

flux job info

scontrol show job

Show details of a specific job

flux resource list

sinfo

Show available resources

flux batch

sbatch (script)

Submit a batch script

flux cancel

scancel

Cancel a running or pending job

flux queue idle

(wait for drain)

Block until all queued jobs complete

Hierarchical Scheduling

One of Flux’s distinguishing features is the ability to run nested Flux instances. A job running inside Flux can itself be a Flux instance that schedules sub-jobs. This is useful for:

  • Ensemble computing — A parent job allocates resources, then spawns many child jobs that each use a subset.

  • Multi-phase workflows — Phase 1 runs a simulation, phase 2 processes the results, each managed by its own Flux instance.

  • Isolation — Different teams or workflow steps can have their own scheduling policies within their allocated resources.

Exercise 1: Deploying a Flux MiniCluster

Prerequisites

Ensure the Flux Operator is installed on your cluster.

  1. Install the Flux Operator (if not already installed):

    kubectl apply -f https://github.com/flux-framework/flux-operator/releases/latest/download/flux-operator.yaml

  2. Verify the operator is running:

    oc get pods -n operator-system

  3. Create a namespace for Flux workloads:

    oc new-project flux-demo --skip-config-update || oc project flux-demo

Run a simple job

The simplest use of the Flux Operator is an ephemeral MiniCluster that runs a single command and exits.

  1. Create a MiniCluster that runs a hostname command across 4 Pods:

    cat <<EOF | oc apply -f -
apiVersion: flux-framework.org/v1alpha2
kind: MiniCluster
metadata:
  name: flux-hello
  namespace: flux-demo
spec:
  size: 4
  containers:
    - image: ghcr.io/flux-framework/flux-restful-api:latest
      command: flux run -n 4 hostname
EOF

    This creates a 4-Pod MiniCluster. The broker Pod runs flux run -n 4 hostname, which distributes the hostname command across all 4 nodes in the Flux instance.

  2. Watch the Pods start:

    oc get pods -n flux-demo -w

    You should see 4 Pods (flux-hello-0 through flux-hello-3) start up. The broker (index 0) coordinates the Flux instance. Press Ctrl+C once the Pods start running.

  3. View the output from the broker Pod:

    oc logs -n flux-demo flux-hello-0-0-xxxxx 2>/dev/null || \
  oc logs -n flux-demo $(oc get pods -n flux-demo -l job-name=flux-hello -o jsonpath='{.items[0].metadata.name}')

    You should see hostnames from all 4 Pods, confirming that Flux distributed the command across the cluster.

  4. Clean up:

    oc delete minicluster flux-hello -n flux-demo

Exercise 2: Running an HPC Application

The Flux Operator is commonly used to run HPC applications like LAMMPS (molecular dynamics), which is a representative scale-out HPC workload.

  1. Create a MiniCluster that runs LAMMPS:

    cat <<EOF | oc apply -f -
apiVersion: flux-framework.org/v1alpha2
kind: MiniCluster
metadata:
  name: flux-lammps
  namespace: flux-demo
spec:
  size: 2
  containers:
    - image: ghcr.io/rse-ops/lammps:flux-sched-focal-v0.24.0
      workingDir: /home/flux/examples/reaxff/HNS
      command: flux run -n 2 lmp -v x 2 -v y 2 -v z 2 -in in.reaxc.hns -nocite
      resources:
        limits:
          cpu: 4
          memory: 4Gi
        requests:
          cpu: 2
          memory: 2Gi
EOF

    This runs a LAMMPS reactive molecular dynamics simulation across 2 nodes with MPI.

  2. Watch for completion:

    oc get pods -n flux-demo -l job-name=flux-lammps -w

    Press Ctrl+C once the broker Pod completes.

  3. View the simulation output:

    oc logs -n flux-demo $(oc get pods -n flux-demo -l job-name=flux-lammps --sort-by=.metadata.name -o jsonpath='{.items[0].metadata.name}') | tail -30

    You should see LAMMPS output including timesteps, performance metrics, and completion.

  4. Clean up:

    oc delete minicluster flux-lammps -n flux-demo

Exercise 3: Batch Mode — Multiple Jobs

For workloads that need to submit many jobs, use batch mode. This avoids creating a separate MiniCluster (and Kubernetes resources) for each job.

  1. Create a MiniCluster in batch mode that submits several jobs:

    cat <<EOF | oc apply -f -
apiVersion: flux-framework.org/v1alpha2
kind: MiniCluster
metadata:
  name: flux-batch
  namespace: flux-demo
spec:
  size: 2
  containers:
    - image: ghcr.io/flux-framework/flux-restful-api:latest
      batch: |
        #!/bin/bash
        echo "=== Flux Batch Mode ==="
        echo "Submitting multiple jobs to Flux scheduler..."

        flux submit -n 1 echo "Job 1: Hello from Flux"
        flux submit -n 1 echo "Job 2: Computing on OpenShift"
        flux submit -n 1 echo "Job 3: Batch processing"
        flux submit -n 2 hostname
        flux submit -n 1 sleep 5

        echo "All jobs submitted. Waiting for completion..."
        flux queue idle

        echo "=== Job Results ==="
        flux jobs -a
EOF

    The batch field submits the script via flux batch. Inside the script, flux submit queues individual jobs, and flux queue idle waits for all of them to finish.

  2. Wait for the MiniCluster to complete:

    oc get pods -n flux-demo -l job-name=flux-batch -w

    Press Ctrl+C once the broker Pod completes.

  3. View the batch output:

    oc logs -n flux-demo $(oc get pods -n flux-demo -l job-name=flux-batch --sort-by=.metadata.name -o jsonpath='{.items[0].metadata.name}') | tail -20

    You should see the output from all submitted jobs and the final flux jobs -a listing showing their completion status.

  4. Clean up:

    oc delete minicluster flux-batch -n flux-demo

Exercise 4: Interactive MiniCluster

For exploration and development, you can create a persistent MiniCluster that stays running so you can submit jobs interactively.

  1. Create an interactive MiniCluster:

    cat <<EOF | oc apply -f -
apiVersion: flux-framework.org/v1alpha2
kind: MiniCluster
metadata:
  name: flux-interactive
  namespace: flux-demo
spec:
  size: 2
  interactive: true
  containers:
    - image: ghcr.io/flux-framework/flux-restful-api:latest
EOF

    The interactive: true setting starts the Flux broker without a command, keeping it alive for interactive use.

  2. Wait for the Pods to be running:

    oc get pods -n flux-demo -l job-name=flux-interactive -w

    Press Ctrl+C once both Pods are Running.

  3. Exec into the broker Pod:

    oc exec -it -n flux-demo $(oc get pods -n flux-demo -l job-name=flux-interactive --sort-by=.metadata.name -o jsonpath='{.items[0].metadata.name}') -- /bin/bash

  4. Inside the Flux broker, explore the cluster and submit jobs:

    # Check available resources
flux resource list

# Submit a job
flux run -n 2 hostname

# Submit a job to the queue
flux submit echo "Queued job from interactive session"

# Check job status
flux jobs -a

# Run a sleep job in the background and watch it
flux submit -n 1 sleep 10
flux jobs

  5. Exit the interactive session:

    exit

  6. Clean up:

    oc delete minicluster flux-interactive -n flux-demo

Flux vs Slurm on OpenShift

Both Flux and Slurm can run on OpenShift, but they take different approaches:

Aspect Slurm on OpenShift (Slinky) Flux on OpenShift (Flux Operator)

Deployment model

Persistent daemons (slurmctld, slurmd)

Ephemeral or persistent MiniClusters

Resource model

Flat node list with partitions

Hierarchical graph (racks, nodes, sockets, cores)

Job submission

sbatch, srun from login node

flux submit, flux run from broker Pod (or via CRD)

Nested scheduling

Not supported

Native hierarchical instances

High-throughput jobs

Good, but overhead per job

Designed for millions of small jobs

Ecosystem maturity

Decades of production use, vast user base

Newer, rapidly growing in national lab and research communities

Kubernetes integration

Slinky project (community)

Flux Operator with CRD (community)

Choose Slurm if your users already have Slurm workflows and expect Slurm compatibility. Choose Flux if you need hierarchical scheduling, topology-aware placement, or high-throughput job submission for ensemble computing.

Clean Up

  1. Remove all Flux resources:

    oc delete miniclusters --all -n flux-demo

Summary

The Flux Framework brings next-generation HPC scheduling to OpenShift via the Flux Operator.

In this module you:

  • Understood why Flux — hierarchical scheduling, graph-based resource model, high-throughput job submission, and converged computing.

  • Deployed ephemeral MiniClusters that run a single job and clean up.

  • Ran an HPC application (LAMMPS) across multiple Pods using MPI through Flux.

  • Used batch mode to submit multiple jobs within a single MiniCluster.

  • Created an interactive MiniCluster for hands-on exploration of Flux commands.

  • Compared Flux vs Slurm on OpenShift to understand when each fits.