Home
Arenadata Hyperwave
Services
Spark
Integrations
Kubernetes
Arenadata Cloud CLI
Install Spark History Server on Kubernetes

Install Spark History Server on Kubernetes using CLI

Sergei Tikhomirov

Collapse content Expand content

Contents

Prerequisites
Step 1. Install Spark History Server
Step 2. Provide access to Spark History Server web UI
Step 3. Check by submitting a Spark application
Delete instances

Prerequisites

A compatible version of an ADH cluster is installed and running (required only if event log is going to be saved to HDFS or Ozone).
The hub.arenadata.io/adc-enterprise/spark3:<version> image that is unpacked and pushed to your repository.
Spark operator is deployed in Kubernetes according to the instruction. Make sure that Spark History Server’s namespace is included in Spark operator’s payloadNamespaces.

Step 1. Install Spark History Server

Prepare the hadoop_conf.yaml Hadoop configuration file (required only if S3 or the Hadoop services are used):

hadoop_conf.yaml

sites:
  core:
    fs.defaultFS: hdfs://adh
    hadoop.security.authentication: simple
    dfs.client.failover.proxy.provider.adh: org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
    dfs.ha.namenodes.adh: nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3
    dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:8020
    dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:8020
    dfs.nameservices: adh
  hdfs:
    dfs.client.read.shortcircuit: false
  ozone:
    ozone.om.address.adh.om_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:9862
    ozone.om.address.adh.om_tsn-adh-k8s-2: tsn-adh-k8s-2.ru-central1.internal:9862
    ozone.om.address.adh.om_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:9862
    ozone.om.nodes.adh: om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3
    ozone.om.service.ids: adhom
  hive:
    hive.metastore.sasl.enabled: false
    hive.metastore.uris: thrift://tsn-adh-k8s-1.ru-central1.internal:9083
    metastore.use.SSL: false

Initialize Spark History Server:
```
$ ./adc init --spark-history-server --hadoop-file=hadoop_conf.yaml -o spark-history-server.yaml
```
This operation creates the spark-history-server.yaml file with a configuration template.

Edit the configuration file to your needs:

spark-history-server.yaml

apiVersion: adc.arenadata.io/v1alpha1
kind: SparkHistoryServer
metadata:
  name: spark-history-server
  namespace: spark-history-server (1)
spec:
  image: hub.arenadata.io/adc-enterprise/spark3:<tag> (2)

  ## Image pull secret for a private registry.
  ## Set 'externalSecretName' to reference an existing Secret,
  ## or set 'credentials' and optionally 'secretName' to let the CLI create one.
  #imagePullSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-registry-secret
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-registry-secret
  #
  #  #credentials:
  #  #  registry: registry.example.com
  #  #  username: user
  #  #  password: pass

  hadoop: (3)
    core:
      dfs.client.failover.proxy.provider.adh: org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
      dfs.ha.namenodes.adh: nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3
      dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:8020
      dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:8020
      dfs.nameservices: adh
      fs.defaultFS: hdfs://adh
      hadoop.security.authentication: simple
    hdfs:
      dfs.client.read.shortcircuit: "false"
    hive:
      hive.metastore.sasl.enabled: "false"
      hive.metastore.uris: thrift://tsn-adh-k8s-1.ru-central1.internal:9083
      metastore.use.SSL: "false"
    ozone:
      ozone.om.address.adh.om_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-adh-k8s-2: tsn-adh-k8s-2.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:9862
      ozone.om.nodes.adh: om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3
      ozone.om.service.ids: adhom

  ## Kerberos configuration for authentication.
  #kerberos:
  #  realm: EXAMPLE.COM
  #
  #  # Service name in the Kerberos principal. Defaults to the product name.
  #  service: spark-history
  #
  #  # Hostname in the Kerberos principal.
  #  # Required for a fixed service principal; leave it empty only to derive one principal per pod from the cluster domain.
  #  hostname: history.example.com
  #  keytab:
  #    # true - kerberos-operator creates the keytab Secret.
  #    # false (default) - reference an existing keytab Secret with name keytab.secretName.
  #    create: false
  #
  #    # Name of the keytab Secret.
  #    # Optional when create: true - names the generated Secret (default: <name>-keytab).
  #    # Required when create: false - must reference an existing Secret.
  #    secretName: spark-history-keytab
  #
  #    # Label selector for the Pod that generates the keytab.
  #    # Required when create: true; ignored when create: false.
  #    labelSelector:
  #      env: prod
  #    #additionalPrincipals:
  #    #  - HTTP/history-lb.example.com
  #
  #    rotation:
  #      interval: 720h
  #      checkInterval: 1h
  #  spnego: true

  ## Java KeyStore/TrustStore certificate configuration.
  ## Set externalSecretName to reference an existing Secret,
  ## or set files and optional secretName to have ADC create it.
  #ssl:
  #  ## Name of the Secret containing Java keystores.
  #  #secretName: custom-spark-history-ssl
  #  externalSecretName: existing-spark-history-ssl
  #
  #  # Key in the Secret containing the truststore file.
  #  trustStoreKey: truststore.jks
  #
  #  ## Password for the truststore (optional).
  #  #trustStorePassword: bigdata
  #
  #  ## Key in the Secret containing the keystore file (optional).
  #  #keyStoreKey: keystore.jks
  #
  #  ## Password for the keystore (optional).
  #  #keyStorePassword: bigdata
  #
  #  ## Local files 'adc apply' puts into the Secret named by ssl.secretName.
  #  ## Relative paths are resolved against the config file.
  #  #files:
  #  #  trustStorePath: /path/to/truststore.jks
  #  #  #keyStorePath: /path/to/keystore.jks

  ## Use an external Hadoop configs Secret instead of the one rendered by ADC.
  #hadoopConfigsSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-history-hadoop-configs
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-history-hadoop-configs

  ## Use an external Spark History properties Secret instead of the one rendered by ADC.
  #historyPropertiesSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-history-properties
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-history-properties

  eventLogDir: "hdfs:///logs/spark/apps" (4)
  store:
    path: /var/log/spark/history (5)
    emptyDir: {}
    #pvc: (6)
    #  size: 10Gi
    #  storageClassName: my-storage-class
    #  accessModes:
    #    - ReadWriteOnce

  ## Monitoring configuration.
  #monitoring:
  #  # Enables Prometheus metrics export from this product's pods.
  #  exportMetrics: true

  #properties:
  #  spark.history.store.maxDiskUsage: 16g

  server:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: "1"
    #    memory: 1Gi
    #  requests:
    #    cpu: 400m
    #    memory: 256Mi
  #extraMountSecrets:
  #  - mountPath: /etc/spark/extra-secret
  #    options:
  #      containerName: app
  #    secretName: extra-secret

  ## Controls whether Secret/ConfigMap changes restart pods.
  ## Set enabled: false to update referenced Secrets without restarting
  ## the workload; pods keep running the previous configuration until
  ## the policy is re-enabled. Defaults to enabled.
  #configurationRollout:
  #  enabled: false

1	Namespace that Spark History Server will use.
2	URL to the Spark image in your repository.
3	Hadoop settings derived from the hadoop_conf.yaml file.
4	Filesystem URI that the History Server reads (e.g. `hdfs:///var/log/spark/apps`, `ofs://vol.bucket/spark/apps`, `s3a://bucket/spark/apps`). The directory must exist. Spark applications must write to the same path.
5	Event log storage directory.
6	PVC storage request. Either `pvc` or `emptyDir` must be set.

You can check the configuration about to be applied by running the apply command with the --dry-run option:

$ ./adc apply -f spark-history-server.yaml --dry-run > spark-history-server-render.yaml

spark-history-server-render.yaml

---
apiVersion: v1
kind: Secret
metadata:
  name: spark-history-server-configs
  namespace: spark-history-server
stringData:
  core-site.xml: |-
    <configuration>
      <property>
        <name>dfs.client.failover.proxy.provider.adh</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider</value>
      </property>
      <property>
        <name>dfs.client.read.shortcircuit</name>
        <value>false</value>
      </property>
      <property>
        <name>dfs.ha.namenodes.adh</name>
        <value>nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1</name>
        <value>tsn-adh-k8s-1.ru-central1.internal:8020</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3</name>
        <value>tsn-adh-k8s-3.ru-central1.internal:8020</value>
      </property>
      <property>
        <name>dfs.nameservices</name>
        <value>adh</value>
      </property>
      <property>
        <name>fs.defaultFS</name>
        <value>hdfs://adh</value>
      </property>
      <property>
        <name>hadoop.security.authentication</name>
        <value>simple</value>
      </property>
      <property>
        <name>ozone.om.address.adh.om_tsn-adh-k8s-1</name>
        <value>tsn-adh-k8s-1.ru-central1.internal:9862</value>
      </property>
      <property>
        <name>ozone.om.address.adh.om_tsn-adh-k8s-2</name>
        <value>tsn-adh-k8s-2.ru-central1.internal:9862</value>
      </property>
      <property>
        <name>ozone.om.address.adh.om_tsn-adh-k8s-3</name>
        <value>tsn-adh-k8s-3.ru-central1.internal:9862</value>
      </property>
      <property>
        <name>ozone.om.nodes.adh</name>
        <value>om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3</value>
      </property>
      <property>
        <name>ozone.om.service.ids</name>
        <value>adhom</value>
      </property>
    </configuration>
  hive-site.xml: |-
    <configuration>
      <property>
        <name>hive.metastore.sasl.enabled</name>
        <value>false</value>
      </property>
      <property>
        <name>hive.metastore.uris</name>
        <value>thrift://tsn-adh-k8s-1.ru-central1.internal:9083</value>
      </property>
      <property>
        <name>metastore.use.SSL</name>
        <value>false</value>
      </property>
    </configuration>
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: spark-history-server-spark-history-properties
  namespace: spark-history-server
stringData:
  history-server.properties: |
    spark.eventLog.dir hdfs:///logs/spark/apps
    spark.eventLog.enabled true
    spark.history.fs.logDirectory hdfs:///logs/spark/apps
    spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
    spark.history.store.path /var/log/spark/history
    spark.history.ui.port 18080
type: Opaque
---
apiVersion: spark.arenadata.io/v1alpha1
kind: SparkHistoryServer
metadata:
  name: spark-history-server
  namespace: spark-history-server
spec:
  properties:
    secretKey: history-server.properties
    secretName: spark-history-server-spark-history-properties
  server:
    metadata: {}
    replicas: 1
    spec:
      emptyDirs:
      - /var/log/spark/history
      image: hub.arenadata.io/adc-enterprise/spark3:<tag>
      imagePullPolicy: Always
      mountSecrets:
      - mountPath: /etc/hadoop/conf
        options:
          containerName: app
        secretName: spark-history-server-configs
      ports:
      - containerPort: 18080
        name: http
        servicePort: 18080
status: {}

If the manifest is correct, apply the Spark History Server configuration:

$ ./adc apply -f spark-history-server.yaml

The expected output contains a confirmation of success:

time="20260827095515UTC" level="info" msg="cluster spark-history-server applied to namespace spark-history-server"

Verify that the Spark History Server pods are running:

$ kubectl get pods -n spark-history-server

The expected output is:

NAME                                          READY   STATUS    RESTARTS   AGE
spark-history-server-spark-history-server-0   1/1     Running   0          9s

Step 2. Provide access to Spark History Server web UI

To access the Spark History Server web UI, you need to expose the service using one of the supported publication methods, for example, through a load balancer or Ingress controller. All configurations related to exposing a service, including DNS, annotations, Ingress settings, load balancing rules, and other platform-specific settings, must be specified according to your Kubernetes environment.

Get the external IP address of your load balancer or Ingress controller. For example:

NAME                                        TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
spark-lb                                    LoadBalancer   10.85.56.180   10.92.40.88   18080:30930/TCP   119s

Open the Spark History Server web UI in your browser, using the http://10.92.40.88:18080 URL.

Spark History Server web UI

Spark History Server web UI

Step 3. Check by submitting a Spark application

Create a Spark application according to the instruction.
Edit the application’s spec.eventLogDir parameter. This parameter’s value and the Spark History Server’s spec.eventLogDir value should match:
```
eventLogDir: "hdfs:///logs/spark/apps"
```
Submit the Spark application:
```
$ ./adc apply -f spark-application.yaml
```
Inspect the Spark History Server web UI.

Completed job in the Spark History Server web UI

Completed job in the Spark History Server web UI

Delete instances

To delete Spark History Server, run the following command:

$ ./adc delete -f spark-history-server.yaml

Found a mistake? Seleсt text and press Ctrl+Enter to report it