Главная
Arenadata Hyperwave
Сервисы
Spark
Интеграция
Kubernetes
Arenadata Cloud CLI
Настройка Kerberos для Spark в Kubernetes

Настройка Kerberos для Spark в Kubernetes с помощью CLI

Сергей Тихомиров

Свернуть контент Развернуть контент

Содержание

Требования
Шаг 1. Установка оператора Kerberos
Шаг 2. Создание keytab для Spark
Шаг 3. Запуск приложения Spark
Удаление инстансов

Требования

Установленный и керберизованный кластер ADH версии 4.2.0 или более поздней.
Оператор Spark, развернутый в Kubernetes согласно инструкции.

Шаг 1. Установка оператора Kerberos

Извлеките и загрузите образ оператора Kerberos в ваш репозиторий.
Инициализируйте оператор Kerberos:
```
$ ./adc init --kerberos-operator -o kerberos-operator.yaml
```
Данная команда создаст файл kerberos-operator.yaml с шаблоном конфигурации.

Отредактируйте конфигурационный файл:

kerberos-operator.yaml

apiVersion: adc.arenadata.io/v1alpha1
kind: KerberosOperator
metadata:
  name: kerberos-operator
  namespace: kerberos-operator (1)
spec:
  image: hub.arenadata.io/adc-enterprise/kerberos-operator:<tag> (2)

  # Number of replicas
  # replicas: 1

  resources:
    limits:
      cpu: 500m
      memory: 256Mi

  # Operator ServiceAccount. create: true (default) also creates the manager and per-payload-namespace Role/RoleBinding bound to it; create: false skips all three - name then refers to a ServiceAccount (and RBAC) managed entirely outside the CLI.
  serviceAccount: (3)
    create: true
    name: "kerberos-operator"

  # Whether the CLI creates the product namespace.
  # The namespace name is set in metadata.namespace.
  namespace:
    create: true

  # Create namespaces to run the payload.
  createPayloadNamespaces: true

  # List of namespaces to run the payload in.
  payloadNamespaces: (4)
    - spark-applications

  ## Image pull secret for a private registry.
  ## Set 'externalSecretName' to reference an existing Secret,
  ## or set 'credentials' and optionally 'secretName' to let the CLI create one.
  #imagePullSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-registry-secret
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-registry-secret
  #
  #  #credentials:
  #  #  registry: registry.example.com
  #  #  username: user
  #  #  password: pass

  ## Operator monitoring configuration. Supports product-specific metrics export and optional vmagent delivery to an external ADM.
  #monitoring:
  #  # Expose the Kerberos operator metrics endpoint on port 8443.
  #  exportMetrics: true
  #
  #  # Renders a namespace-scoped vmagent that sends metrics to an external ADM.
  #  vmagent:
  #    remoteWrite:
  #      url: http://vminsert.example.com/insert/0/prometheus/api/v1/write
  #    scrapeInterval: 15s
  #    image: hub.arenadata.io/adm-enterprise/vmagent:1.136.0-adm-5.0.0-x86_64
  #
  #    ## HTTPS settings used by vmagent when scraping product metrics.
  #    #tls:
  #    #  ## CA certificate source used to verify the metrics endpoint.
  #    #  #ca:
  #    #  #  # Use a Secret managed outside ADC.
  #    #  #  externalSecretName: existing-product-metrics-ca
  #    #  #
  #    #  #  ## Or let ADC create the Secret.
  #    #  #  #secretName: product-metrics-ca
  #    #  #
  #    #  #  # Key containing the CA certificate in the referenced Secret.
  #    #  #  certificateKey: ca.crt
  #    #  #
  #    #  #  ## Local CA certificate read by ADC to create the configured Secret.
  #    #  #  #files:
  #    #  #  #  certificatePath: /path/to/ca.crt
  #    #
  #    #  ## Server name used to verify the metrics endpoint certificate hostname.
  #    #  #serverName: metrics.example.com
  #    #
  #    #  # Skip verification of the metrics endpoint certificate. Do not use together with ca.
  #    #  insecureSkipVerify: true

  ## TLS certificate configuration for metrics endpoint.
  ## Set externalSecretName to reference an existing Secret,
  ## or set files and optional secretName to have ADC create it.
  #metricsTLS:
  #  ## Optional name of the Secret ADC creates from local files.
  #  #secretName: custom-metrics-tls-secret
  #  externalSecretName: existing-metrics-tls-secret
  #
  #  # Key in the Secret containing the TLS certificate.
  #  certificateKey: tls.crt
  #
  #  # Key in the Secret containing the TLS private key.
  #  privateKey: tls.key
  #
  #  ## Local files 'adc apply' puts into the Secret named by metricsTLS.secretName.
  #  ## Relative paths are resolved against the config file.
  #  #files:
  #  #  certificatePath: /path/to/tls.crt
  #  #  privateKeyPath: /path/to/tls.key

  ## Defines the Kerberos realm configuration and krb5.conf-related settings.
  kdc: (5)
    realm: AD.RANGER-TEST
    labelSelector:
      env: prod
    realms:
      AD.RANGER-TEST: |-
        kdc = ad01.adsw.io
        admin_server = ad01.adsw.io
    domainRealm:
      ad.ranger-test: AD.RANGER-TEST
    libdefaults:
  #    debug: "false"
      default_realm: AD.RANGER-TEST
  #    default_tgs_enctypes: aes256-cts-hmac-sha1-96
  #    dns_lookup_kdc: "false"

  ## Defines connection and authentication details for the LDAP server (backend for Kerberos KDC).
  ldapSecret: (6)
    secretName: ldap-credentials
    provider: ad
    address: ldaps://ad01.adsw.io:636
    adminUser: stikhomirov@AD.RANGER-TEST
    adminPassword: <password>
    baseDN: OU=stikhomirov_ou,OU=kerberos,OU=adh,DC=ad,DC=ranger-test
    ca: "" (7)
  #
  #  # Local file paths consumed by the CLI during 'adc apply'.
  #  # Paths are relative to the config file. The CLI reads this file
  #  # and creates the Secret named by ldapSecret.secretName.
  #  files:
  #    certificatePath: /path/to/ca.pem

1	Настройки пространства имен.
2	URL образа оператора Kerberos в вашем репозитории.
3	Настройки сервисного аккаунта.
4	Список пространств имен, доступных оператору Kerberos.
5	Настройки KDC.
6	Настройки LDAP. Если вы не используете SSL, замените протокол на `ldap` и порт на `389`.
7	CA-сертификат в формате PEM, необходимый, если LDAP защищен SSL.

Вы можете проверить конфигурацию перед ее применением, выполнив команду apply с флагом --dry-run:

$ ./adc apply -f kerberos-operator.yaml --dry-run > kerberos-operator-render.yaml

kerberos-operator-render.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: kerberos-operator
spec: {}
status: {}
---
apiVersion: v1
kind: Namespace
metadata:
  name: spark-applications
spec: {}
status: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kerberos-operator
  namespace: kerberos-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kerberos-operator-kerberos-operator-manager
  namespace: kerberos-operator
rules:
- apiGroups:
  - events.k8s.io
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - krb5.arenadata.io
  resources:
  - kdcconfigs
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - krb5.arenadata.io
  resources:
  - kdcconfigs/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - krb5.arenadata.io
  resources:
  - kdcconfigs/finalizers
  verbs:
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kerberos-operator-kerberos-operator-manager
  namespace: kerberos-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kerberos-operator-kerberos-operator-manager
subjects:
- kind: ServiceAccount
  name: kerberos-operator
  namespace: kerberos-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kerberos-operator-kerberos-operator-payload
  namespace: spark-applications
rules:
- apiGroups:
  - events.k8s.io
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - krb5.arenadata.io
  resources:
  - keytabs
  verbs:
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - krb5.arenadata.io
  resources:
  - keytabs/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - krb5.arenadata.io
  resources:
  - keytabs/finalizers
  verbs:
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kerberos-operator-kerberos-operator-payload
  namespace: spark-applications
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kerberos-operator-kerberos-operator-payload
subjects:
- kind: ServiceAccount
  name: kerberos-operator
  namespace: kerberos-operator
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: operator
    app.kubernetes.io/managed-by: adc-cli
    arenadata.io/operator-type: kerberos
  name: kerberos-operator-kerberos-operator
  namespace: kerberos-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: operator
      app.kubernetes.io/managed-by: adc-cli
      app.kubernetes.io/name: kerberos-operator-kerberos-operator
  strategy: {}
  template:
    metadata:
      labels:
        app.kubernetes.io/component: operator
        app.kubernetes.io/managed-by: adc-cli
        app.kubernetes.io/name: kerberos-operator-kerberos-operator
    spec:
      containers:
      - args:
        - -ns=spark-applications
        image: hub.arenadata.io/adc-enterprise/kerberos-operator:<tag>
        imagePullPolicy: Always
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        name: app
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          limits:
            cpu: 500m
            memory: 256Mi
          requests:
            cpu: 500m
            memory: 256Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsGroup: 10001
          runAsNonRoot: true
          runAsUser: 10001
      securityContext:
        fsGroup: 10001
        runAsGroup: 10001
        runAsNonRoot: true
        runAsUser: 10001
      serviceAccountName: kerberos-operator
      terminationGracePeriodSeconds: 10
status: {}
---
apiVersion: krb5.arenadata.io/v1alpha1
kind: KDCConfig
metadata:
  name: kerberos-operator-kdc
  namespace: kerberos-operator
spec:
  domainRealm:
    ad.ranger-test: AD.RANGER-TEST
  labelSelector:
    env: prod
  libdefaults:
    default_realm: AD.RANGER-TEST
  realm: AD.RANGER-TEST
  realms:
    AD.RANGER-TEST: |-
      kdc = ad01.adsw.io
      admin_server = ad01.adsw.io
---
apiVersion: v1
kind: Secret
metadata:
  annotations:
    krb5.arenadata.io/provider: ad
  labels:
    env: prod
  name: ldap-credentials
  namespace: kerberos-operator
stringData:
  addr: ldaps://ad01.adsw.io:636
  adminDN: stikhomirov@AD.RANGER-TEST
  adminPW: <password>
  baseDN: OU=stikhomirov_ou,OU=kerberos,OU=adh,DC=ad,DC=ranger-test
type: krb5.arenadata.io/ldap-credentials

Если манифест корректный, примените конфигурацию и разверните оператор Kerberos:
```
$ ./adc apply -f kerberos-operator.yaml
```

Шаг 2. Создание keytab для Spark

Создайте конфигурацию KDC для среды prod:

kdc.yaml

apiVersion: krb5.arenadata.io/v1alpha1
kind: KDCConfig
metadata:
  name: ad-kdc
  namespace: kerberos-operator
  labels:
    env: prod
spec:
  realm: AD.RANGER-TEST
  labelSelector:
    env: prod
  realms:
    AD.RANGER-TEST: |
      kdc = ad01.adsw.io
      admin_server = ad01.adsw.io
  domainRealm:
    ad.ranger-test: AD.RANGER-TEST
    .svc.cluster.local: AD.RANGER-TEST
  libdefaults:
    default_realm: AD.RANGER-TEST
    default_tgs_enctypes: aes256-cts-hmac-sha1-96

Примените конфигурацию KDC:
```
$ kubectl apply -f kdc.yaml
```

Создайте конфигурацию для keytab сервиса Spark:

spark-keytab.yaml

apiVersion: krb5.arenadata.io/v1alpha1
kind: Keytab
metadata:
  name: spark-keytab
  namespace: spark-applications
spec:
  items:
    - realm: AD.RANGER-TEST
      labelSelector:
        env: prod
      principals:
        - spark/tsn-adh-k8s-1.ru-central1.internal
  rotation:
    interval: 720h
    checkInterval: 1h

Примените конфигурацию keytab:
```
$ kubectl apply -f spark-keytab.yaml
```

Проверьте, что keytab и соответствующий секрет существуют:

$ kubectl get keytabs -n spark-applications
$ kubectl get secrets -n spark-applications

Вывод должен содержать следующие строки:

NAME           ROTATION            READY             AGE   NEXTROTATION
spark-keytab   RotationScheduled   SecretGenerated   2m   Next rotation at about 2026-09-24T08:41:33Z
NAME                                                 TYPE                       DATA   AGE
spark-keytab                                         krb5.arenadata.io/bundle   2      2m

Шаг 3. Запуск приложения Spark

Подготовьте файл hadoop_conf.yaml с настройками Hadoop (необходимо, только если приложение обращается к данным, управляемым этими сервисами; для самодостаточных JAR-приложений блок hadoop можно опустить):

hadoop_conf.yaml

sites:
  core:
    fs.defaultFS: hdfs://adh
    hadoop.security.authentication: kerberos
    dfs.client.failover.proxy.provider.adh: org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
    dfs.ha.namenodes.adh: nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3
    dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:8020
    dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:8020
    dfs.nameservices: adh
    dfs.namenode.kerberos.principal: hdfs-namenode/_HOST@AD.RANGER-TEST
    dfs.journalnode.kerberos.principal: hdfs-journalnode/_HOST@AD.RANGER-TEST
    dfs.datanode.kerberos.principal: hdfs-datanode/_HOST@AD.RANGER-TEST
    hadoop.ssl.enabled: false
  hdfs:
    dfs.client.read.shortcircuit: false
  ozone:
    ozone.om.address.adh.om_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:9862
    ozone.om.address.adh.om_tsn-adh-k8s-2: tsn-adh-k8s-2.ru-central1.internal:9862
    ozone.om.address.adh.om_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:9862
    ozone.om.nodes.adh: om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3
    ozone.om.kerberos.principal: om/_HOST@AD.RANGER-TEST
    ozone.om.service.ids: adhom
  hive:
    hive.metastore.sasl.enabled: true
    hive.metastore.uris: thrift://tsn-adh-k8s-1.ru-central1.internal:9083
    hive.metastore.kerberos.principal: hive/_HOST@AD.RANGER-TEST
    metastore.use.SSL: false

Инициализируйте приложение Spark:
```
$ ./adc init --spark-application --hadoop-file hadoop_conf.yaml -o spark-application.yaml
```
Данная команда создаст файл spark-application.yaml с шаблоном конфигурации.

Отредактируйте конфигурационный файл:

spark-application.yaml

apiVersion: adc.arenadata.io/v1alpha1
kind: SparkApplication
metadata:
  name: spark-application
  namespace: spark-applications (1)
spec:
  image: hub.arenadata.io/adc-enterprise/spark3:<tag> (2)

  ## Image pull secret for a private registry.
  ## Set 'externalSecretName' to reference an existing Secret,
  ## or set 'credentials' and optionally 'secretName' to let the CLI create one.
  #imagePullSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-registry-secret
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-registry-secret
  #
  #  #credentials:
  #  #  registry: registry.example.com
  #  #  username: user
  #  #  password: pass

  hadoop: (3)
    core:
      dfs.client.failover.proxy.provider.adh: org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
      dfs.datanode.kerberos.principal: hdfs-datanode/_HOST@AD.RANGER-TEST
      dfs.ha.namenodes.adh: nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3
      dfs.journalnode.kerberos.principal: hdfs-journalnode/_HOST@AD.RANGER-TEST
      dfs.namenode.kerberos.principal: hdfs-namenode/_HOST@AD.RANGER-TEST
      dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:8020
      dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:8020
      dfs.nameservices: adh
      fs.defaultFS: hdfs://adh
      hadoop.security.authentication: kerberos
      hadoop.ssl.enabled: "false"
    hdfs:
      dfs.client.read.shortcircuit: "false"
    hive:
      hive.metastore.kerberos.principal: hive/_HOST@AD.RANGER-TEST
      hive.metastore.sasl.enabled: "true"
      hive.metastore.uris: thrift://tsn-adh-k8s-1.ru-central1.internal:9083
      metastore.use.SSL: "false"
    ozone:
      ozone.om.address.adh.om_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-adh-k8s-2: tsn-adh-k8s-2.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:9862
      ozone.om.kerberos.principal: om/_HOST@AD.RANGER-TEST
      ozone.om.nodes.adh: om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3
      ozone.om.service.ids: adhom

  ## Kerberos configuration for authentication.
  kerberos: (4)
    principal: spark/tsn-adh-k8s-1.ru-central1.internal@AD.RANGER-TEST
  #
  #  # CLI reads the local files and creates the kerberos-ccache Secret on 'adc apply'.
  #  # Alternative - keytab mode: replace this block with:
       keytab:
         secretName: spark-keytab
  #  ticketCache:
  #    #secretName: custom-ticket-cache
  #    externalSecretName: existing-ticket-cache
  #    #ticketPath: /tmp/krb5cc_1000
  #    #krb5ConfPath: /etc/krb5.conf

  ## Ranger plugin configuration.
  ## Uncomment and fill the lines below. adc apply derives the rest.
  #ranger:
  #  # fill ranger.plugin.spark.policy.rest.url below with Ranger endpoint, e.g. https://adps-adc.ru-central1.internal:6182
  #  # fill ranger.plugin.spark.service.name below with Ranger service name you want to use for product, e.g. adc_spark_id_1
  #  security:
  #    ranger.plugin.spark.policy.rest.url: ""
  #    ranger.plugin.spark.service.name: ""
  #
  #  # fill xasecure.audit.destination.solr.zookeepers below with Zookeepers endpoints to resolve solr service, e.g. adps-adc.ru-central1.internal:2181/Arenadata.Hadoop-2.solr.server
  #  audit:
  #    xasecure.audit.destination.solr.zookeepers: ""
  #
  #  # Local Ranger files 'adc apply' writes into the configs Secret.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    jceksStorePath: /path/to/ranger.jceks

  ## Java KeyStore/TrustStore certificate configuration.
  ## Set externalSecretName to reference an existing Secret,
  ## or set files and optional secretName to have ADC create it.
  #ssl:
  #  ## Name of the Secret containing Java keystores.
  #  #secretName: custom-ssl-secret
  #  externalSecretName: existing-ssl-secret
  #
  #  # Key in the Secret containing the truststore file.
  #  trustStoreKey: truststore.jks
  #
  #  ## Password for the truststore (optional).
  #  #trustStorePassword: bigdata
  #
  #  ## Key in the Secret containing the keystore file (optional).
  #  #keyStoreKey: keystore.jks
  #
  #  ## Password for the keystore (optional).
  #  #keyStorePassword: bigdata
  #
  #  ## Local files 'adc apply' puts into the Secret named by ssl.secretName.
  #  ## Relative paths are resolved against the config file.
  #  #files:
  #  #  trustStorePath: /path/to/truststore.jks
  #  #  #keyStorePath: /path/to/keystore.jks

  ## Use an external Hadoop configs Secret instead of the one rendered by ADC.
  #hadoopConfigsSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-spark-hadoop-configs
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-spark-hadoop-configs

  ## Use an external Ranger configs Secret instead of the one rendered by ADC.
  #rangerConfigsSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-spark-ranger-configs
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-spark-ranger-configs

  # Spark application main resource (e.g. local:///opt/spark/examples/jars/spark-examples.jar).
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples.jar" (5)

  ## HDFS or local directory for the Spark event log.
  ## When set, the CLI adds spark.eventLog.enabled=true, spark.eventLog.dir,
  ## spark.eventLog.rolling.enabled=true and spark.eventLog.rolling.interval=30s to sparkConf.
  #eventLogDir: ""

  ## Fully-qualified main class name. Required for Java/Scala applications.
  mainClass: "org.apache.spark.examples.sql.SparkSQLExample" (6)

  # ServiceAccount used by the Spark driver, also injected into
  # spark.kubernetes.authenticate.driver.serviceAccountName. The CLI creates it, plus a Role
  # and RoleBinding for Spark pods, by default (create: true). Set create: false to skip that
  # and only reference a ServiceAccount managed elsewhere.
  # The Role rules are managed by the CLI and cannot be customized.
  serviceAccount: (7)
    create: true
    name: spark-application
  job: (8)
    ## true (default) deletes the spark-submit Job pod after it finishes; set false to keep it for debugging.
    deleteOnTermination: false

    #resources:
    #  limits:
    #    cpu: "1"
    #    memory: 512Mi
    #  requests:
    #    cpu: 500m
    #    memory: 64Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  executor-memory: 1g
    #  num-executors: "2"

  ## Application arguments appended after mainApplicationFile.
  args:
    - "100"

  # Spark configuration entries (spark.*).
  sparkConf: (9)
    spark.artifactory.dir.path: /tmp/artifacts
    spark.jars.ivy: /tmp/ivy
    spark.local.dir: /tmp/data
    spark.sql.catalog.spark_catalog: org.apache.iceberg.spark.SparkSessionCatalog
    spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
    spark.sql.security.confblacklist: spark.sql.extensions

  ## Seconds after the application finishes (Succeeded, or Failed with no
  ## retries left) before the SparkApplication is deleted. Omit to keep it
  ## until explicit deletion.
  #ttlSecondsAfterFinished: 3600 (10)

  ## Celeborn remote shuffle service for Spark.
  ## tiers mirror the cluster's storage.tiers: local (SSD/HDD) and MEMORY advertise their name only,
  ## remote (S3/HDFS) also set dir (and, for S3, credentials). Ozone is an HDFS tier with an ofs:// dir.
  #celeborn:
  #  # Storage tiers mirroring the cluster's storage.tiers so the client advertises the same layers.
  #  # type: SSD, HDD, S3, HDFS, MEMORY. SSD/HDD and MEMORY are advertise-only here (no dir; worker-only
  #  # fields ignored); remote S3/HDFS set dir (s3a:// for S3; hdfs:// or ofs:// for Ozone on HDFS). Example:
  #  #   - type: MEMORY   # cache tier; pair with a durable tier below, no dir
  #  #   - type: SSD      # advertise-only on the client, no dir needed
  #  #   - dir: s3a://bucket/celeborn
  #  #     s3: { endpoint: https://s3:9878, region: us-east-1 }
  #  #     type: S3
  #  #   - dir: hdfs://nn/celeborn   # or ofs://om/volume/bucket/celeborn for Ozone
  #  #     type: HDFS
  #  tiers:
  #    - dir: s3a://shuffle/my-cluster
  #      s3:
  #        accessKey: <access-key>
  #        endpoint: https://s3.endpoint:443
  #        pathStyleAccess: true
  #        region: <region>
  #        secretKey: <secret-key>
  #      type: S3
  #  masterEndpoint: ""
  #  extraSparkConf:
  #    spark.sql.adaptive.enabled: "true"
  #
  #  # Enable TLS on the client's RPC connection to the Celeborn cluster.
  #  # When true the CLI renders spark.celeborn.ssl.* into the Spark conf
  #  # and requires the ssl section with trustStoreKey.
  #  rpcEncryption: false
  #
  #  # Enable TLS on the client's data module, which carries shuffle push/fetch
  #  # traffic between executors and workers. Requires the ssl section with trustStoreKey.
  #  dataEncryption: false

  ## YuniKorn scheduler configuration for queue selection and Gang scheduling.
  ## Uncomment the block to route the Spark job into a YuniKorn queue; the CLI renders the
  ## scheduler name, queue labels and gang annotations into sparkConf.
  #yunikorn:
  #  queue: root.analytics
  #  taskGroups:
  #    - minMember: 1
  #      minResource:
  #        cpu: "1"
  #        memory: 1433Mi
  #      name: spark-driver
  #    - minMember: 2
  #      minResource:
  #        cpu: "1"
  #        memory: 1433Mi
  #      name: spark-executor

  ## Monitoring configuration.
  #monitoring:
  #  # Enables Prometheus metrics export from this product's pods.
  #  exportMetrics: true

1	Пространство имен, используемое приложением Spark.
2	URL образа Spark в вашем репозитории.
3	Настройки Hadoop, взятые из ранее созданного файла hadoop_conf.yaml.
4	Используемый принципал Kerberos и имя секрета, содержащего соответствующий keytab.
5	URL к файлу с задачей приложения (JAR или .py).
6	Имя главного класса для приложений на Java/Scala.
7	Настройки сервисного аккаунта.
8	Настройки задачи. После завершения работы по умолчанию поды приложения удаляются автоматически. Чтобы поды задачи и драйвера не удалялись (например, для проверки логов), присвойте параметру `deleteOnTermination` значение `false`. Чтобы оставить executor-поды, присвойте параметру `deleteOnTermination` значение `false` в блоке `spark.executor` — он не включен в сгенерированную минимальную конфигурацию, поэтому его требуется добавить вручную.
9	Настройки Spark.
10	Период в секундах, после которого приложение Spark будет удалено вне зависимости от причины завершения работы.

Вы можете проверить конфигурацию перед ее применением, выполнив команду apply с флагом --dry-run:

$ ./adc apply -f spark-application.yaml --dry-run > spark-application-render.yaml

spark-application-render.yaml

---
apiVersion: v1
kind: Secret
metadata:
  name: spark-application-configs
  namespace: spark-applications
stringData:
  core-site.xml: |-
    <configuration>
      <property>
        <name>dfs.client.failover.proxy.provider.adh</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
      </property>
      <property>
        <name>dfs.client.read.shortcircuit</name>
        <value>false</value>
      </property>
      <property>
        <name>dfs.datanode.kerberos.principal</name>
        <value>hdfs-datanode/_HOST@AD.RANGER-TEST</value>
      </property>
      <property>
        <name>dfs.ha.namenodes.adh</name>
        <value>nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3</value>
      </property>
      <property>
        <name>dfs.journalnode.kerberos.principal</name>
        <value>hdfs-journalnode/_HOST@AD.RANGER-TEST</value>
      </property>
      <property>
        <name>dfs.namenode.kerberos.principal</name>
        <value>hdfs-namenode/_HOST@AD.RANGER-TEST</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1</name>
        <value>tsn-adh-k8s-1.ru-central1.internal:8020</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3</name>
        <value>tsn-adh-k8s-3.ru-central1.internal:8020</value>
      </property>
      <property>
        <name>dfs.nameservices</name>
        <value>adh</value>
      </property>
      <property>
        <name>fs.defaultFS</name>
        <value>hdfs://adh</value>
      </property>
      <property>
        <name>hadoop.security.authentication</name>
        <value>kerberos</value>
      </property>
      <property>
        <name>hadoop.ssl.enabled</name>
        <value>false</value>
      </property>
    </configuration>
  hive-site.xml: |-
    <configuration>
      <property>
        <name>hive.metastore.kerberos.principal</name>
        <value>hive/_HOST@AD.RANGER-TEST</value>
      </property>
      <property>
        <name>hive.metastore.sasl.enabled</name>
        <value>true</value>
      </property>
      <property>
        <name>hive.metastore.uris</name>
        <value>thrift://tsn-adh-k8s-1.ru-central1.internal:9083</value>
      </property>
      <property>
        <name>metastore.use.SSL</name>
        <value>false</value>
      </property>
    </configuration>
type: Opaque
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: spark-application
  namespace: spark-applications
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: spark-application
  namespace: spark-applications
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - configmaps
  - persistentvolumeclaims
  - services
  - secrets
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
  - deletecollection
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - events.k8s.io
  resources:
  - events
  verbs:
  - create
  - patch
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: spark-application
  namespace: spark-applications
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: spark-application
subjects:
- kind: ServiceAccount
  name: spark-application
  namespace: spark-applications
---
apiVersion: spark.arenadata.io/v1alpha1
kind: SparkApplication
metadata:
  name: spark-application
  namespace: spark-applications
spec:
  args:
  - "100"
  driver:
    metadata: {}
    spec:
      image: hub.adsw.io/adh-enterprise/spark3-docker:<tag>
      imagePullPolicy: Always
  executor:
    metadata: {}
    spec:
      image: hub.adsw.io/adh-enterprise/spark3-docker:<tag>
      imagePullPolicy: Always
  hadoopConfigsSecretName: spark-application-configs
  job:
    deleteOnTermination: false
    metadata: {}
    spec:
      image: hub.adsw.io/adh-enterprise/spark3-docker:<tag>
      imagePullPolicy: Always
  kerberos:
    kerberosSecretName: spark-keytab
    principal: spark/tsn-adh-k8s-1.ru-central1.internal@AD.RANGER-TEST
  mainApplicationFile: local:///opt/spark/examples/jars/spark-examples.jar
  mainClass: org.apache.spark.examples.sql.SparkSQLExample
  serviceAccountName: spark-application
  sparkConf:
    spark.artifactory.dir.path: /tmp/artifacts
    spark.jars.ivy: /tmp/ivy
    spark.kerberos.access.hadoopFileSystems: hdfs://adh
    spark.kubernetes.authenticate.driver.serviceAccountName: spark-application
    spark.kubernetes.namespace: spark-applications
    spark.local.dir: /tmp/data
    spark.sql.catalog.spark_catalog: org.apache.iceberg.spark.SparkSessionCatalog
    spark.sql.extensions: org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
    spark.sql.security.confblacklist: spark.sql.extensions
status: {}

Если манифест корректный, примените конфигурацию и запустите приложение Spark:
```
$ ./adc apply -f spark-application.yaml
```
Ожидаемый вывод содержит сообщение с подтверждением успеха:
```
time="20260817085516UTC" level="info" msg="cluster spark-application applied to namespace spark-applications"
```

Проверьте работоспособность подов приложения Spark:

$ kubectl get pods -n spark-applications

Ожидаемый вывод должен быть похож на следующий:

NAME                                              READY   STATUS    RESTARTS   AGE
spark-application-34a1a4a0387cfba7-driver         1/1     Running   0          13s
spark-application-rrzkl                           1/1     Running   0          16s
spark-sql-basic-example-4791c7a0387d0e72-exec-1   1/1     Running   0          5s
spark-sql-basic-example-4791c7a0387d0e72-exec-2   1/1     Running   0          5s

После завершения работы executor-поды удаляются, а статус подов приложения меняется на Completed:

NAME                                        READY   STATUS      RESTARTS   AGE
spark-application-34a1a4a0387cfba7-driver   0/1     Completed   0          56s
spark-application-rrzkl                     0/1     Completed   0          59s

Проверьте вывод в логах driver-пода:
```
$ kubectl logs spark-application-34a1a4a0387cfba7-driver -n spark-applications
```
Логи должны содержать строки, соответствующие задаче.

Удаление инстансов

ВАЖНО

Удалять оператор следует только после удаления всех управляемых им ресурсов.

Чтобы удалить приложение Spark, выполните следующую команду:

$ ./adc delete -f spark-application.yaml

Чтобы удалить оператор Spark, выполните следующую команду:

$ ./adc delete -f spark-operator.yaml --purge

Чтобы удалить оператор Kerberos, выполните следующую команду:

$ ./adc delete -f kerberos-operator.yaml --purge

Нашли ошибку? Выделите текст и нажмите Ctrl+Enter чтобы сообщить о ней