Home
Arenadata Hyperwave
Services
Spark
Integrations
Kubernetes
Arenadata Cloud CLI
Configure Kerberos for Spark on Kubernetes

Configure Kerberos for Spark on Kubernetes using CLI

Sergei Tikhomirov

Collapse content Expand content

Contents

Prerequisites
Step 1. Install Kerberos operator
Step 2. Create the Spark keytab
Step 3. Submit a Spark application
Delete instances

Prerequisites

An ADH cluster (4.2.0 or later) is installed and kerberized.
Spark operator is deployed in Kubernetes according to the instruction.

Step 1. Install Kerberos operator

Unpack and push the Kerberos operator image to your repository.
Initialize Kerberos operator:
```
$ ./adc init --kerberos-operator -o kerberos-operator.yaml
```
This operation creates the kerberos-operator.yaml file with a configuration template.

Edit the configuration file to your needs:

kerberos-operator.yaml

apiVersion: adc.arenadata.io/v1alpha1
kind: KerberosOperator
metadata:
  name: kerberos-operator
  namespace: kerberos-operator (1)
spec:
  image: hub.arenadata.io/adc-enterprise/kerberos-operator:<tag> (2)

  # Number of replicas
  # replicas: 1

  resources:
    limits:
      cpu: 500m
      memory: 256Mi

  # Operator ServiceAccount. create: true (default) also creates the manager and per-payload-namespace Role/RoleBinding bound to it; create: false skips all three - name then refers to a ServiceAccount (and RBAC) managed entirely outside the CLI.
  serviceAccount: (3)
    create: true
    name: "kerberos-operator"

  # Whether the CLI creates the product namespace.
  # The namespace name is set in metadata.namespace.
  namespace:
    create: true

  # Create namespaces to run the payload.
  createPayloadNamespaces: true

  # List of namespaces to run the payload in.
  payloadNamespaces: (4)
    - spark-applications

  ## Image pull secret for a private registry.
  ## Set 'externalSecretName' to reference an existing Secret,
  ## or set 'credentials' and optionally 'secretName' to let the CLI create one.
  #imagePullSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-registry-secret
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-registry-secret
  #
  #  #credentials:
  #  #  registry: registry.example.com
  #  #  username: user
  #  #  password: pass

  ## Operator monitoring configuration. Supports product-specific metrics export and optional vmagent delivery to an external ADM.
  #monitoring:
  #  # Expose the Kerberos operator metrics endpoint on port 8443.
  #  exportMetrics: true
  #
  #  # Renders a namespace-scoped vmagent that sends metrics to an external ADM.
  #  vmagent:
  #    remoteWrite:
  #      url: http://vminsert.example.com/insert/0/prometheus/api/v1/write
  #    scrapeInterval: 15s
  #    image: hub.arenadata.io/adm-enterprise/vmagent:1.136.0-adm-5.0.0-x86_64
  #
  #    ## HTTPS settings used by vmagent when scraping product metrics.
  #    #tls:
  #    #  ## CA certificate source used to verify the metrics endpoint.
  #    #  #ca:
  #    #  #  # Use a Secret managed outside ADC.
  #    #  #  externalSecretName: existing-product-metrics-ca
  #    #  #
  #    #  #  ## Or let ADC create the Secret.
  #    #  #  #secretName: product-metrics-ca
  #    #  #
  #    #  #  # Key containing the CA certificate in the referenced Secret.
  #    #  #  certificateKey: ca.crt
  #    #  #
  #    #  #  ## Local CA certificate read by ADC to create the configured Secret.
  #    #  #  #files:
  #    #  #  #  certificatePath: /path/to/ca.crt
  #    #
  #    #  ## Server name used to verify the metrics endpoint certificate hostname.
  #    #  #serverName: metrics.example.com
  #    #
  #    #  # Skip verification of the metrics endpoint certificate. Do not use together with ca.
  #    #  insecureSkipVerify: true

  ## TLS certificate configuration for metrics endpoint.
  ## Set externalSecretName to reference an existing Secret,
  ## or set files and optional secretName to have ADC create it.
  #metricsTLS:
  #  ## Optional name of the Secret ADC creates from local files.
  #  #secretName: custom-metrics-tls-secret
  #  externalSecretName: existing-metrics-tls-secret
  #
  #  # Key in the Secret containing the TLS certificate.
  #  certificateKey: tls.crt
  #
  #  # Key in the Secret containing the TLS private key.
  #  privateKey: tls.key
  #
  #  ## Local files 'adc apply' puts into the Secret named by metricsTLS.secretName.
  #  ## Relative paths are resolved against the config file.
  #  #files:
  #  #  certificatePath: /path/to/tls.crt
  #  #  privateKeyPath: /path/to/tls.key

  ## Defines the Kerberos realm configuration and krb5.conf-related settings.
  kdc: (5)
    realm: AD.RANGER-TEST
    labelSelector:
      env: prod
    realms:
      AD.RANGER-TEST: |-
        kdc = ad01.adsw.io
        admin_server = ad01.adsw.io
    domainRealm:
      ad.ranger-test: AD.RANGER-TEST
    libdefaults:
  #    debug: "false"
      default_realm: AD.RANGER-TEST
  #    default_tgs_enctypes: aes256-cts-hmac-sha1-96
  #    dns_lookup_kdc: "false"

  ## Defines connection and authentication details for the LDAP server (backend for Kerberos KDC).
  ldapSecret: (6)
    secretName: ldap-credentials
    provider: ad
    address: ldaps://ad01.adsw.io:636
    adminUser: stikhomirov@AD.RANGER-TEST
    adminPassword: <password>
    baseDN: OU=stikhomirov_ou,OU=kerberos,OU=adh,DC=ad,DC=ranger-test
    ca: "" (7)
  #
  #  # Local file paths consumed by the CLI during 'adc apply'.
  #  # Paths are relative to the config file. The CLI reads this file
  #  # and creates the Secret named by ldapSecret.secretName.
  #  files:
  #    certificatePath: /path/to/ca.pem

1	Namespace settings.
2	URL to the Kerberos operator image in your repository.
3	Service account settings.
4	Payload namespace settings. The listed namespaces will be available to the Kerberos operator instance.
5	KDC settings.
6	LDAP settings. If you don’t use SSL, change the protocol to `ldap` and port to `389`.
7	A PEM-encoded CA certificate if LDAP is secured with SSL.

You can check the configuration about to be applied by running the apply command with the --dry-run option:

$ ./adc apply -f kerberos-operator.yaml --dry-run > kerberos-operator-render.yaml

kerberos-operator-render.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: kerberos-operator
spec: {}
status: {}
---
apiVersion: v1
kind: Namespace
metadata:
  name: spark-applications
spec: {}
status: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kerberos-operator
  namespace: kerberos-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kerberos-operator-kerberos-operator-manager
  namespace: kerberos-operator
rules:
- apiGroups:
  - events.k8s.io
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - krb5.arenadata.io
  resources:
  - kdcconfigs
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - krb5.arenadata.io
  resources:
  - kdcconfigs/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - krb5.arenadata.io
  resources:
  - kdcconfigs/finalizers
  verbs:
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kerberos-operator-kerberos-operator-manager
  namespace: kerberos-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kerberos-operator-kerberos-operator-manager
subjects:
- kind: ServiceAccount
  name: kerberos-operator
  namespace: kerberos-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kerberos-operator-kerberos-operator-payload
  namespace: spark-applications
rules:
- apiGroups:
  - events.k8s.io
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - krb5.arenadata.io
  resources:
  - keytabs
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - krb5.arenadata.io
  resources:
  - keytabs/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - krb5.arenadata.io
  resources:
  - keytabs/finalizers
  verbs:
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kerberos-operator-kerberos-operator-payload
  namespace: spark-applications
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kerberos-operator-kerberos-operator-payload
subjects:
- kind: ServiceAccount
  name: kerberos-operator
  namespace: kerberos-operator
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: operator
    app.kubernetes.io/managed-by: adc-cli
    arenadata.io/operator-type: kerberos
  name: kerberos-operator-kerberos-operator
  namespace: kerberos-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: operator
      app.kubernetes.io/managed-by: adc-cli
      app.kubernetes.io/name: kerberos-operator-kerberos-operator
  strategy: {}
  template:
    metadata:
      labels:
        app.kubernetes.io/component: operator
        app.kubernetes.io/managed-by: adc-cli
        app.kubernetes.io/name: kerberos-operator-kerberos-operator
    spec:
      containers:
      - args:
        - -ns=spark-applications
        image: hub.arenadata.io/adc-enterprise/kerberos-operator:<tag>
        imagePullPolicy: Always
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        name: app
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          limits:
            cpu: 500m
            memory: 256Mi
          requests:
            cpu: 500m
            memory: 256Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsGroup: 10001
          runAsNonRoot: true
          runAsUser: 10001
      securityContext:
        fsGroup: 10001
        runAsGroup: 10001
        runAsNonRoot: true
        runAsUser: 10001
      serviceAccountName: kerberos-operator
      terminationGracePeriodSeconds: 10
status: {}
---
apiVersion: krb5.arenadata.io/v1alpha1
kind: KDCConfig
metadata:
  name: kerberos-operator-kdc
  namespace: kerberos-operator
spec:
  domainRealm:
    ad.ranger-test: AD.RANGER-TEST
  labelSelector:
    env: prod
  libdefaults:
    default_realm: AD.RANGER-TEST
  realm: AD.RANGER-TEST
  realms:
    AD.RANGER-TEST: |-
      kdc = ad01.adsw.io
      admin_server = ad01.adsw.io
---
apiVersion: v1
kind: Secret
metadata:
  annotations:
    krb5.arenadata.io/provider: ad
  labels:
    env: prod
  name: ldap-credentials
  namespace: kerberos-operator
stringData:
  addr: ldaps://ad01.adsw.io:636
  adminDN: stikhomirov@AD.RANGER-TEST
  adminPW: <password>
  baseDN: OU=stikhomirov_ou,OU=kerberos,OU=adh,DC=ad,DC=ranger-test
type: krb5.arenadata.io/ldap-credentials

If the manifest is correct, apply the configuration and deploy Kerberos operator:
```
$ ./adc apply -f kerberos-operator.yaml
```

Step 2. Create the Spark keytab

Create a KDC config for the prod environment:

kdc.yaml

apiVersion: krb5.arenadata.io/v1alpha1
kind: KDCConfig
metadata:
  name: ad-kdc
  namespace: kerberos-operator
  labels:
    env: prod
spec:
  realm: AD.RANGER-TEST
  labelSelector:
    env: prod
  realms:
    AD.RANGER-TEST: |
      kdc = ad01.adsw.io
      admin_server = ad01.adsw.io
  domainRealm:
    ad.ranger-test: AD.RANGER-TEST
    .svc.cluster.local: AD.RANGER-TEST
  libdefaults:
    default_realm: AD.RANGER-TEST
    default_tgs_enctypes: aes256-cts-hmac-sha1-96

Apply the KDC config:
```
$ kubectl apply -f kdc.yaml
```

Create the Spark keytab config:

spark-keytab.yaml

apiVersion: krb5.arenadata.io/v1alpha1
kind: Keytab
metadata:
  name: spark-keytab
  namespace: spark-applications
spec:
  items:
    - realm: AD.RANGER-TEST
      labelSelector:
        env: prod
      principals:
        - spark/tsn-adh-k8s-1.ru-central1.internal
  rotation:
    interval: 720h
    checkInterval: 1h

Apply the keytab configuration:
```
$ kubectl apply -f spark-keytab.yaml
```

Check that the keytab and the corresponding secret exist:

$ kubectl get keytabs -n spark-applications
$ kubectl get secrets -n spark-applications

The output should contain the following lines:

NAME           ROTATION            READY             AGE   NEXTROTATION
spark-keytab   RotationScheduled   SecretGenerated   2m   Next rotation at about 2026-09-24T08:41:33Z
NAME                                                 TYPE                       DATA   AGE
spark-keytab                                         krb5.arenadata.io/bundle   2      2m

Step 3. Submit a Spark application

Prepare the hadoop_conf.yaml Hadoop configuration file (required only if the application accesses data managed by these services; for a self-contained JAR the hadoop block can be omitted):

hadoop_conf.yaml

sites:
  core:
    fs.defaultFS: hdfs://adh
    hadoop.security.authentication: kerberos
    dfs.client.failover.proxy.provider.adh: org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
    dfs.ha.namenodes.adh: nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3
    dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:8020
    dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:8020
    dfs.nameservices: adh
    dfs.namenode.kerberos.principal: hdfs-namenode/_HOST@AD.RANGER-TEST
    dfs.journalnode.kerberos.principal: hdfs-journalnode/_HOST@AD.RANGER-TEST
    dfs.datanode.kerberos.principal: hdfs-datanode/_HOST@AD.RANGER-TEST
    hadoop.ssl.enabled: false
  hdfs:
    dfs.client.read.shortcircuit: false
  ozone:
    ozone.om.address.adh.om_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:9862
    ozone.om.address.adh.om_tsn-adh-k8s-2: tsn-adh-k8s-2.ru-central1.internal:9862
    ozone.om.address.adh.om_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:9862
    ozone.om.nodes.adh: om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3
    ozone.om.kerberos.principal: om/_HOST@AD.RANGER-TEST
    ozone.om.service.ids: adhom
  hive:
    hive.metastore.sasl.enabled: true
    hive.metastore.uris: thrift://tsn-adh-k8s-1.ru-central1.internal:9083
    hive.metastore.kerberos.principal: hive/_HOST@AD.RANGER-TEST
    metastore.use.SSL: false

Initialize a Spark application:
```
$ ./adc init --spark-application --hadoop-file hadoop_conf.yaml -o spark-application.yaml
```
This operation creates the spark-application.yaml file with a configuration template.

Edit the configuration file to your needs:

spark-application.yaml

apiVersion: adc.arenadata.io/v1alpha1
kind: SparkApplication
metadata:
  name: spark-application
  namespace: spark-applications (1)
spec:
  image: hub.arenadata.io/adc-enterprise/spark3:<tag> (2)

  ## Image pull secret for a private registry.
  ## Set 'externalSecretName' to reference an existing Secret,
  ## or set 'credentials' and optionally 'secretName' to let the CLI create one.
  #imagePullSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-registry-secret
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-registry-secret
  #
  #  #credentials:
  #  #  registry: registry.example.com
  #  #  username: user
  #  #  password: pass

  hadoop: (3)
    core:
      dfs.client.failover.proxy.provider.adh: org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
      dfs.datanode.kerberos.principal: hdfs-datanode/_HOST@AD.RANGER-TEST
      dfs.ha.namenodes.adh: nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3
      dfs.journalnode.kerberos.principal: hdfs-journalnode/_HOST@AD.RANGER-TEST
      dfs.namenode.kerberos.principal: hdfs-namenode/_HOST@AD.RANGER-TEST
      dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:8020
      dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:8020
      dfs.nameservices: adh
      fs.defaultFS: hdfs://adh
      hadoop.security.authentication: kerberos
      hadoop.ssl.enabled: "false"
    hdfs:
      dfs.client.read.shortcircuit: "false"
    hive:
      hive.metastore.kerberos.principal: hive/_HOST@AD.RANGER-TEST
      hive.metastore.sasl.enabled: "true"
      hive.metastore.uris: thrift://tsn-adh-k8s-1.ru-central1.internal:9083
      metastore.use.SSL: "false"
    ozone:
      ozone.om.address.adh.om_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-adh-k8s-2: tsn-adh-k8s-2.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:9862
      ozone.om.kerberos.principal: om/_HOST@AD.RANGER-TEST
      ozone.om.nodes.adh: om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3
      ozone.om.service.ids: adhom

  ## Kerberos configuration for authentication.
  kerberos: (4)
    principal: spark/tsn-adh-k8s-1.ru-central1.internal@AD.RANGER-TEST
  #
  #  # CLI reads the local files and creates the kerberos-ccache Secret on 'adc apply'.
  #  # Alternative - keytab mode: replace this block with:
       keytab:
         secretName: spark-keytab
  #  ticketCache:
  #    #secretName: custom-ticket-cache
  #    externalSecretName: existing-ticket-cache
  #    #ticketPath: /tmp/krb5cc_1000
  #    #krb5ConfPath: /etc/krb5.conf

  ## Ranger plugin configuration.
  ## Uncomment and fill the lines below. adc apply derives the rest.
  #ranger:
  #  # fill ranger.plugin.spark.policy.rest.url below with Ranger endpoint, e.g. https://adps-adc.ru-central1.internal:6182
  #  # fill ranger.plugin.spark.service.name below with Ranger service name you want to use for product, e.g. adc_spark_id_1
  #  security:
  #    ranger.plugin.spark.policy.rest.url: ""
  #    ranger.plugin.spark.service.name: ""
  #
  #  # fill xasecure.audit.destination.solr.zookeepers below with Zookeepers endpoints to resolve solr service, e.g. adps-adc.ru-central1.internal:2181/Arenadata.Hadoop-2.solr.server
  #  audit:
  #    xasecure.audit.destination.solr.zookeepers: ""
  #
  #  # Local Ranger files 'adc apply' writes into the configs Secret.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    jceksStorePath: /path/to/ranger.jceks

  ## Java KeyStore/TrustStore certificate configuration.
  ## Set externalSecretName to reference an existing Secret,
  ## or set files and optional secretName to have ADC create it.
  #ssl:
  #  ## Name of the Secret containing Java keystores.
  #  #secretName: custom-ssl-secret
  #  externalSecretName: existing-ssl-secret
  #
  #  # Key in the Secret containing the truststore file.
  #  trustStoreKey: truststore.jks
  #
  #  ## Password for the truststore (optional).
  #  #trustStorePassword: bigdata
  #
  #  ## Key in the Secret containing the keystore file (optional).
  #  #keyStoreKey: keystore.jks
  #
  #  ## Password for the keystore (optional).
  #  #keyStorePassword: bigdata
  #
  #  ## Local files 'adc apply' puts into the Secret named by ssl.secretName.
  #  ## Relative paths are resolved against the config file.
  #  #files:
  #  #  trustStorePath: /path/to/truststore.jks
  #  #  #keyStorePath: /path/to/keystore.jks

  ## Use an external Hadoop configs Secret instead of the one rendered by ADC.
  #hadoopConfigsSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-spark-hadoop-configs
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-spark-hadoop-configs

  ## Use an external Ranger configs Secret instead of the one rendered by ADC.
  #rangerConfigsSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-spark-ranger-configs
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-spark-ranger-configs

  # Spark application main resource (e.g. local:///opt/spark/examples/jars/spark-examples.jar).
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples.jar" (5)

  ## HDFS or local directory for the Spark event log.
  ## When set, the CLI adds spark.eventLog.enabled=true, spark.eventLog.dir,
  ## spark.eventLog.rolling.enabled=true and spark.eventLog.rolling.interval=30s to sparkConf.
  #eventLogDir: ""

  ## Fully-qualified main class name. Required for Java/Scala applications.
  mainClass: "org.apache.spark.examples.sql.SparkSQLExample" (6)

  # ServiceAccount used by the Spark driver, also injected into
  # spark.kubernetes.authenticate.driver.serviceAccountName. The CLI creates it, plus a Role
  # and RoleBinding for Spark pods, by default (create: true). Set create: false to skip that
  # and only reference a ServiceAccount managed elsewhere.
  # The Role rules are managed by the CLI and cannot be customized.
  serviceAccount: (7)
    create: true
    name: spark-application
  job: (8)
    ## true (default) deletes the spark-submit Job pod after it finishes; set false to keep it for debugging.
    deleteOnTermination: false

    #resources:
    #  limits:
    #    cpu: "1"
    #    memory: 512Mi
    #  requests:
    #    cpu: 500m
    #    memory: 64Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  executor-memory: 1g
    #  num-executors: "2"

  ## Application arguments appended after mainApplicationFile.
  args:
    - "100"

  # Spark configuration entries (spark.*).
  sparkConf: (9)
    spark.artifactory.dir.path: /tmp/artifacts
    spark.jars.ivy: /tmp/ivy
    spark.local.dir: /tmp/data
    spark.sql.catalog.spark_catalog: org.apache.iceberg.spark.SparkSessionCatalog
    spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
    spark.sql.security.confblacklist: spark.sql.extensions

  ## Seconds after the application finishes (Succeeded, or Failed with no
  ## retries left) before the SparkApplication is deleted. Omit to keep it
  ## until explicit deletion.
  #ttlSecondsAfterFinished: 3600 (10)

  ## Celeborn remote shuffle service for Spark.
  ## tiers mirror the cluster's storage.tiers: local (SSD/HDD) and MEMORY advertise their name only,
  ## remote (S3/HDFS) also set dir (and, for S3, credentials). Ozone is an HDFS tier with an ofs:// dir.
  #celeborn:
  #  # Storage tiers mirroring the cluster's storage.tiers so the client advertises the same layers.
  #  # type: SSD, HDD, S3, HDFS, MEMORY. SSD/HDD and MEMORY are advertise-only here (no dir; worker-only
  #  # fields ignored); remote S3/HDFS set dir (s3a:// for S3; hdfs:// or ofs:// for Ozone on HDFS). Example:
  #  #   - type: MEMORY   # cache tier; pair with a durable tier below, no dir
  #  #   - type: SSD      # advertise-only on the client, no dir needed
  #  #   - dir: s3a://bucket/celeborn
  #  #     s3: { endpoint: https://s3:9878, region: us-east-1 }
  #  #     type: S3
  #  #   - dir: hdfs://nn/celeborn   # or ofs://om/volume/bucket/celeborn for Ozone
  #  #     type: HDFS
  #  tiers:
  #    - dir: s3a://shuffle/my-cluster
  #      s3:
  #        accessKey: <access-key>
  #        endpoint: https://s3.endpoint:443
  #        pathStyleAccess: true
  #        region: <region>
  #        secretKey: <secret-key>
  #      type: S3
  #  masterEndpoint: ""
  #  extraSparkConf:
  #    spark.sql.adaptive.enabled: "true"
  #
  #  # Enable TLS on the client's RPC connection to the Celeborn cluster.
  #  # When true the CLI renders spark.celeborn.ssl.* into the Spark conf
  #  # and requires the ssl section with trustStoreKey.
  #  rpcEncryption: false
  #
  #  # Enable TLS on the client's data module, which carries shuffle push/fetch
  #  # traffic between executors and workers. Requires the ssl section with trustStoreKey.
  #  dataEncryption: false

  ## YuniKorn scheduler configuration for queue selection and Gang scheduling.
  ## Uncomment the block to route the Spark job into a YuniKorn queue; the CLI renders the
  ## scheduler name, queue labels and gang annotations into sparkConf.
  #yunikorn:
  #  queue: root.analytics
  #  taskGroups:
  #    - minMember: 1
  #      minResource:
  #        cpu: "1"
  #        memory: 1433Mi
  #      name: spark-driver
  #    - minMember: 2
  #      minResource:
  #        cpu: "1"
  #        memory: 1433Mi
  #      name: spark-executor

  ## Monitoring configuration.
  #monitoring:
  #  # Enables Prometheus metrics export from this product's pods.
  #  exportMetrics: true

1	Namespace that the Spark application will use.
2	URL to the Spark image in your repository.
3	Hadoop settings derived from the hadoop_conf.yaml file.
4	Kerberos principal to use and the name of the secret holding the corresponding keytab.
5	URL to the application job file (JAR or .py).
6	Main class name for Java/Scala applications.
7	Service account settings.
8	Job settings. Once the job is completed, the application pods are deleted by default. To keep the job and driver pods for debugging after the execution ends (e.g. to inspect logs), set the `deleteOnTermination` parameter to `false`. To keep the executor pods accessible, set the `deleteOnTermination` parameter to `false` inside the `spark.executor` block — it’s not included in the generated minimal configuration, so you need to add it manually.
9	Spark configuration.
10	Amount of time after which the Spark application deployment is deleted regardless of the execution result.

Check that the configuration about to be applied renders the correct manifest by running the apply command with the --dry-run option:

$ ./adc apply -f spark-application.yaml --dry-run > spark-application-render.yaml

spark-application-render.yaml

---
apiVersion: v1
kind: Secret
metadata:
  name: spark-application-configs
  namespace: spark-applications
stringData:
  core-site.xml: |-
    <configuration>
      <property>
        <name>dfs.client.failover.proxy.provider.adh</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
      </property>
      <property>
        <name>dfs.client.read.shortcircuit</name>
        <value>false</value>
      </property>
      <property>
        <name>dfs.datanode.kerberos.principal</name>
        <value>hdfs-datanode/_HOST@AD.RANGER-TEST</value>
      </property>
      <property>
        <name>dfs.ha.namenodes.adh</name>
        <value>nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3</value>
      </property>
      <property>
        <name>dfs.journalnode.kerberos.principal</name>
        <value>hdfs-journalnode/_HOST@AD.RANGER-TEST</value>
      </property>
      <property>
        <name>dfs.namenode.kerberos.principal</name>
        <value>hdfs-namenode/_HOST@AD.RANGER-TEST</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1</name>
        <value>tsn-adh-k8s-1.ru-central1.internal:8020</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3</name>
        <value>tsn-adh-k8s-3.ru-central1.internal:8020</value>
      </property>
      <property>
        <name>dfs.nameservices</name>
        <value>adh</value>
      </property>
      <property>
        <name>fs.defaultFS</name>
        <value>hdfs://adh</value>
      </property>
      <property>
        <name>hadoop.security.authentication</name>
        <value>kerberos</value>
      </property>
      <property>
        <name>hadoop.ssl.enabled</name>
        <value>false</value>
      </property>
    </configuration>
  hive-site.xml: |-
    <configuration>
      <property>
        <name>hive.metastore.kerberos.principal</name>
        <value>hive/_HOST@AD.RANGER-TEST</value>
      </property>
      <property>
        <name>hive.metastore.sasl.enabled</name>
        <value>true</value>
      </property>
      <property>
        <name>hive.metastore.uris</name>
        <value>thrift://tsn-adh-k8s-1.ru-central1.internal:9083</value>
      </property>
      <property>
        <name>metastore.use.SSL</name>
        <value>false</value>
      </property>
    </configuration>
type: Opaque
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: spark-application
  namespace: spark-applications
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: spark-application
  namespace: spark-applications
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - configmaps
  - persistentvolumeclaims
  - services
  - secrets
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
  - deletecollection
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - events.k8s.io
  resources:
  - events
  verbs:
  - create
  - patch
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: spark-application
  namespace: spark-applications
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: spark-application
subjects:
- kind: ServiceAccount
  name: spark-application
  namespace: spark-applications
---
apiVersion: spark.arenadata.io/v1alpha1
kind: SparkApplication
metadata:
  name: spark-application
  namespace: spark-applications
spec:
  args:
  - "100"
  driver:
    metadata: {}
    spec:
      image: hub.arenadata.io/adc-enterprise/spark3:<tag>
      imagePullPolicy: Always
  executor:
    metadata: {}
    spec:
      image: hub.arenadata.io/adc-enterprise/spark3:<tag>
      imagePullPolicy: Always
  hadoopConfigsSecretName: spark-application-configs
  job:
    deleteOnTermination: false
    metadata: {}
    spec:
      image: hub.arenadata.io/adc-enterprise/spark3:<tag>
      imagePullPolicy: Always
  kerberos:
    kerberosSecretName: spark-keytab
    principal: spark/tsn-adh-k8s-1.ru-central1.internal@AD.RANGER-TEST
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples.jar
  mainClass: org.apache.spark.examples.sql.SparkSQLExample
  serviceAccountName: spark-application
  sparkConf:
    spark.artifactory.dir.path: /tmp/artifacts
    spark.jars.ivy: /tmp/ivy
    spark.kerberos.access.hadoopFileSystems: hdfs://adh
    spark.kubernetes.authenticate.driver.serviceAccountName: spark-application
    spark.kubernetes.namespace: spark-applications
    spark.local.dir: /tmp/data
    spark.sql.catalog.spark_catalog: org.apache.iceberg.spark.SparkSessionCatalog
    spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
    spark.sql.security.confblacklist: spark.sql.extensions
status: {}

If the manifest is correct, submit the Spark application:

$ ./adc apply -f spark-application.yaml

The expected output contains a confirmation of success:

time="20260817085516UTC" level="info" msg="cluster spark-application applied to namespace spark-applications"

Verify that the Spark application pods are running:

$ kubectl get pods -n spark-applications

The expected output is:

NAME                                              READY   STATUS    RESTARTS   AGE
spark-application-34a1a4a0387cfba7-driver         1/1     Running   0          13s
spark-application-rrzkl                           1/1     Running   0          16s
spark-sql-basic-example-4791c7a0387d0e72-exec-1   1/1     Running   0          5s
spark-sql-basic-example-4791c7a0387d0e72-exec-2   1/1     Running   0          5s

Once the job finishes, the executor pods are deleted and the status of the application pods changes to Completed:

NAME                                        READY   STATUS      RESTARTS   AGE
spark-application-34a1a4a0387cfba7-driver   0/1     Completed   0          56s
spark-application-rrzkl                     0/1     Completed   0          59s

Inspect the output in the logs of the driver pod:
```
$ kubectl logs spark-application-34a1a4a0387cfba7-driver -n spark-applications
```
The logs should contain lines regarding the job.

Delete instances

IMPORTANT

Delete the operator only after all the resources it manages have been deleted.

To delete a Spark application, run the following command:

$ ./adc delete -f spark-application.yaml

To delete Spark operator, run the following command:

$ ./adc delete -f spark-operator.yaml --purge

To delete Kerberos operator, run the following command:

$ ./adc delete -f kerberos-operator.yaml --purge

Found a mistake? Seleсt text and press Ctrl+Enter to report it