Apache Spark on OpenShift

Apache Spark is a distributed data processing engine widely used for large-scale analytics, ETL pipelines, machine learning, and graph processing. On OpenShift, Spark workloads run via the Spark Operator (from the Kubeflow project), which manages the lifecycle of Spark applications as Kubernetes-native resources.

This module introduces how Spark runs on OpenShift, the role of the Spark Operator, and how it integrates with Kueue for batch scheduling.

Why Spark on OpenShift?

Organizations running Spark on dedicated clusters (YARN, standalone) face familiar operational challenges:

Separate infrastructure — A Spark-specific cluster that must be provisioned, scaled, and maintained independently.
Resource isolation — Spark jobs compete for resources with no integration into the broader platform’s resource management.
Operational overhead — Different tooling, monitoring, and security for the Spark cluster vs the rest of the infrastructure.

Running Spark on OpenShift unifies the platform: Spark workloads share the same cluster, resource quotas, security policies, and monitoring as other applications. The Spark Operator makes Spark a first-class Kubernetes workload.

The Spark Operator is included (or will be included) in Red Hat OpenShift AI (RHOAI), providing a supported path for running Spark alongside AI/ML workloads on the same platform.

How Spark Runs on Kubernetes

Spark on Kubernetes uses a driver/executor pattern:

The driver Pod runs the main Spark application (the SparkContext). It plans the computation and coordinates the executors.
Executor Pods are created dynamically by the driver. Each executor runs a portion of the distributed computation.
When the computation completes, executor Pods are terminated and the driver exits with the final result.

The Spark Operator manages this entire lifecycle through a SparkApplication custom resource.

SparkApplication

A SparkApplication is the custom resource that defines a Spark job on Kubernetes. It specifies:

The Spark application JAR or Python script
Driver and executor resource requests (CPU, memory)
Number of executor instances
Spark configuration properties
Volumes, environment variables, and dependencies

ScheduledSparkApplication

A ScheduledSparkApplication runs SparkApplications on a cron schedule — useful for recurring ETL or reporting jobs.

Exercise 1: Deploying a Spark Application

Prerequisites

Ensure the Spark Operator is installed on your cluster.

Verify the Spark Operator is running:
```
oc get pods -n spark-operator
```

Create a namespace for Spark workloads:

oc new-project spark-demo --skip-config-update || oc project spark-demo

Submit a SparkApplication

Create a SparkApplication that calculates Pi using a Monte Carlo simulation — a classic distributed computing example:

cat <<EOF | oc apply -f -
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: spark-demo
spec:
  type: Scala
  mode: cluster
  image: spark:3.5.3
  imagePullPolicy: IfNotPresent
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.3.jar
  arguments:
    - "1000"
  sparkVersion: "3.5.3"
  driver:
    cores: 1
    memory: "512m"
    serviceAccount: spark-operator-spark
  executor:
    cores: 1
    instances: 3
    memory: "512m"
  restartPolicy:
    type: Never
EOF

This creates a Spark application with 1 driver and 3 executors that collectively compute Pi using 1000 partitions.

Watch the SparkApplication status:
```
oc get sparkapplications -n spark-demo
```
The status progresses through SUBMITTED → RUNNING → COMPLETED.
Watch the Pods:
```
oc get pods -n spark-demo -w
```
You should see the driver Pod start first, then executor Pods spin up. Press Ctrl+C when the driver Pod shows Completed.
View the driver logs to see the computed value of Pi:
```
oc logs -n spark-demo spark-pi-driver | grep "Pi is roughly"
```
You should see output like: Pi is roughly 3.14159…

Exercise 2: PySpark Application

Spark supports Python applications through PySpark. Here is an example of a PySpark job.

Create a ConfigMap with a Python script:

cat <<EOF | oc apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: pyspark-script
  namespace: spark-demo
data:
  wordcount.py: |
    from pyspark.sql import SparkSession

    spark = SparkSession.builder.appName("WordCount").getOrCreate()

    data = ["OpenShift runs Spark workloads",
            "Spark on Kubernetes is cloud native",
            "Batch computing on OpenShift with Spark",
            "HPC and batch jobs run on OpenShift",
            "Distributed computing with Spark and OpenShift"]

    rdd = spark.sparkContext.parallelize(data)
    counts = rdd.flatMap(lambda line: line.split(" ")) \
                .map(lambda word: (word, 1)) \
                .reduceByKey(lambda a, b: a + b) \
                .sortBy(lambda x: -x[1])

    for word, count in counts.collect():
        print(f"{word}: {count}")

    spark.stop()
EOF

Submit the PySpark application:

cat <<EOF | oc apply -f -
apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: pyspark-wordcount
  namespace: spark-demo
spec:
  type: Python
  mode: cluster
  image: spark:3.5.3-python3
  imagePullPolicy: IfNotPresent
  mainApplicationFile: local:///opt/spark/work-dir/wordcount.py
  sparkVersion: "3.5.3"
  driver:
    cores: 1
    memory: "512m"
    serviceAccount: spark-operator-spark
    volumeMounts:
      - name: script
        mountPath: /opt/spark/work-dir
  executor:
    cores: 1
    instances: 2
    memory: "512m"
    volumeMounts:
      - name: script
        mountPath: /opt/spark/work-dir
  volumes:
    - name: script
      configMap:
        name: pyspark-script
  restartPolicy:
    type: Never
EOF

View the results:

oc logs -n spark-demo pyspark-wordcount-driver | tail -20

Kueue Integration

Spark workloads can be integrated with Kueue for admission control. There are two approaches:

Plain Pod Integration

Kueue’s plain pod support can manage Spark driver and executor Pods directly. This works today without changes to the Spark Operator.

Native Integration (In Progress)

Native integration between the Spark Operator and Kueue is being developed upstream. This will allow SparkApplications to be queued as a single workload in Kueue, with the entire application (driver + executors) admitted or held as a unit.

To use Kueue with Spark via plain pod support, add the queue label to the SparkApplication’s Pod templates:

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: queued-spark-job
  namespace: spark-demo
spec:
  type: Scala
  mode: cluster
  image: spark:3.5.3
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.12-3.5.3.jar
  driver:
    cores: 1
    memory: "512m"
    labels:
      kueue.x-k8s.io/queue-name: user-queue
  executor:
    cores: 1
    instances: 3
    memory: "512m"
    labels:
      kueue.x-k8s.io/queue-name: user-queue
  restartPolicy:
    type: Never

Clean Up

Remove all Spark resources:

oc delete sparkapplications --all -n spark-demo
oc delete configmap pyspark-script -n spark-demo

Summary

Apache Spark runs on OpenShift as a first-class workload via the Spark Operator.

In this module you:

Understood why Spark on OpenShift simplifies operations by unifying infrastructure.
Deployed a Scala SparkApplication (Monte Carlo Pi estimation) with driver and executor Pods.
Deployed a PySpark application using a ConfigMap-mounted script.
Explored Kueue integration options for admission control of Spark workloads.

In the next section, you will learn about Ray on OpenShift for distributed computing.