adc init

Definition

Generates a template YAML configuration file.

Usage

$ adc init <object>... [--name <name>] \
                       [--hadoop-file <file>] \
                       [--namespace <namespace>] \
                       [--output <file>] \
                       [--force]
Arguments
Parameter Description

object

Type of the object for which the configuration will be generated. Available values:

  • --trino-operator

  • --trino-cluster

  • --impala-operator

  • --impala-cluster

  • --spark-operator

  • --spark-application

  • --spark-history-server

  • --kerberos-operator

  • --celeborn

  • --yunikorn

--name

Value of metadata.name inside the configuration. The default value is my-cluster

--hadoop-file

Path to a YAML file with Hadoop site configuration for --impala-cluster and --trino-cluster. When the file is provided, the product-specific helpers are injected.

Trino cluster gets the following parameters:

  • hadoop.proxyuser.trino.groups: *

  • hadoop.proxyuser.trino.hosts: *

Impala cluster gets the following parameters if the Hadoop configuration file has metastore.use.SSL: true:

  • hive.metastore.truststore.path: /etc/ssl/truststore.jks

  • hive.metastore.truststore.password: bigdata

-n, --namespace

Value of metadata.namespace inside the configuration. The default value is adh

-o, --output

Path to the YAML output file. By default, the output is not saved and is printed to stdout

--force

Overwrite the existing output file if the filenames match

Examples

Trino operator

Initializing Trino operator with the trino-operator name within the trino-operator namespace, saving the output into the trino-operator.yaml file:

$ adc init --trino-operator \
           --name trino-operator \
           --namespace trino-operator \
           -o trino-operator.yaml

The generated template file is named trino-operator.yaml, metadata.name is set to trino-operator, and metadata.namespace is set to trino-operator:

trino-operator.yaml
apiVersion: adc.arenadata.io/v1alpha1
kind: TrinoOperator
metadata:
  name: trino-operator
  namespace: trino-operator
spec:
  image: hub.adsw.io/ng/trino-operator:1.58.0

  # Number of replicas
  # replicas: 1

  resources:
    limits:
      cpu: 500m
      memory: 256Mi

  # Operator ServiceAccount. create: true (default) also creates the manager and per-payload-namespace Role/RoleBinding bound to it; create: false skips all three - name then refers to a ServiceAccount (and RBAC) managed entirely outside the CLI.
  serviceAccount:
    create: true
    name: ""

  # Whether the CLI creates the product namespace.
  # The namespace name is set in metadata.namespace.
  namespace:
    create: false

  # Create namespaces to run the payload.
  createPayloadNamespaces: true

  # List of namespaces to run the payload in.
  payloadNamespaces:
    - trino-operator

  ## Image pull secret for a private registry.
  ## Either set 'name' to reference an existing Secret,
  ## or set 'credentials' and the CLI will create a dockerconfigjson Secret.
  #imagePullSecret:
  #  name: my-pull-secret
  #  credentials:
  #    registry: registry.example.com
  #    username: user
  #    password: pass

Trino cluster

Initializing Trino cluster with a hadoop_conf.yaml file:

$ adc init --trino-cluster --hadoop-file=hadoop_conf.yaml --name trino -n trino -o trino-cluster.yaml
trino-cluster.yaml
apiVersion: adc.arenadata.io/v1alpha1
kind: TrinoCluster
metadata:
  name: trino
  namespace: trino
spec:
  image: hub.adsw.io/adh-enterprise/trino-docker:476_arenadata2-adh-4.2.0-x86_64

  ## Image pull secret for a private registry.
  ## Either set 'name' to reference an existing Secret,
  ## or set 'credentials' and the CLI will create a dockerconfigjson Secret.
  #imagePullSecret:
  #  name: my-pull-secret
  #  credentials:
  #    registry: registry.example.com
  #    username: user
  #    password: pass
  hadoop:
    core:
      fs.defaultFS: hdfs://adh
      hadoop.proxyuser.trino.groups: '*'
      hadoop.proxyuser.trino.hosts: '*'
      hadoop.security.authentication: simple
    hdfs:
      dfs.client.failover.proxy.provider.adh: org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
      dfs.ha.namenodes.adh: nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3
      dfs.namenode.rpc-address.adh.nn_tsn-k8s-1: tsn-adh-k8s-1.ru-central1.internal:8020
      dfs.namenode.rpc-address.adh.nn_tsn-k8s-3: tsn-adh-k8s-3.ru-central1.internal:8020
      dfs.nameservices: adh
    hive:
      hive.metastore.sasl.enabled: "false"
      hive.metastore.uris: thrift://tsn-adh-k8s-1.ru-central1.internal:9083
      metastore.use.SSL: "false"
    ozone:
      ozone.om.address.adh.om_tsn-k8s-1: tsn-adh-k8s-1.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-k8s-2: tsn-adh-k8s-2.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-k8s-3: tsn-adh-k8s-3.ru-central1.internal:9862
      ozone.om.nodes.adh: om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3
      ozone.om.service.ids: adhom

  ## Kerberos configuration for authentication.
  #kerberos:
  #  realm: EXAMPLE.COM
  #
  #  # Service name in the Kerberos principal. Defaults to the product name.
  #  service: trino
  #
  #  # Hostname in the Kerberos principal.
  #  # Required for a fixed service principal; leave it empty only to derive one principal per pod from the cluster domain.
  #  hostname: kerberos.example.com
  #  keytab:
  #    # true - kerberos-operator creates the keytab Secret.
  #    # false (default) - reference an existing keytab Secret with name keytab.secretName.
  #    create: false
  #
  #    # Name of the keytab Secret.
  #    # Optional when create: true - names the generated Secret (default: <name>-keytab).
  #    # Required when create: false - must reference an existing Secret.
  #    secretName: kerberos-secret
  #
  #    # Label selector for the Pod that generates the keytab.
  #    # Required when create: true; ignored when create: false.
  #    labelSelector:
  #      env: prod
  #    #additionalPrincipals:
  #    #  - HTTP/kerberos.example.com
  #    clusterDomain: cluster.local
  #    rotation:
  #      interval: 24h
  #      checkInterval: 1h

  ## LDAP authentication configuration.
  ## Uncomment and fill url and userBindPattern.
  ## For ldaps:// URLs the ssl: or ca: section must also be configured (depends on product)
  #ldap:
  #  # LDAP service url.
  #  url: ldaps://ldap.example.com:636
  #
  #  # LDAP user Bind pattern.
  #  userBindPattern: uid=${USER},cn=users,dc=example,dc=com

  ## Ranger plugin configuration.
  ## Uncomment and fill the lines below. adc apply derives the rest.
  #ranger:
  #  # fill ranger.plugin.trino.policy.rest.url below with Ranger endpoint, e.g. https://adps-adc.ru-central1.internal:6182
  #  # fill ranger.plugin.trino.service.name below with Ranger service name you want to use for product, e.g. adc_trino_id_1
  #  security:
  #    ranger.plugin.trino.policy.rest.url: ""
  #    ranger.plugin.trino.service.name: ""
  #
  #  # fill xasecure.audit.destination.solr.zookeepers below with Zookeepers endpoints to resolve solr service, e.g. adps-adc.ru-central1.internal:2181/Arenadata.Hadoop-2.solr.server
  #  audit:
  #    xasecure.audit.destination.solr.zookeepers: ""
  #
  #  # Local Ranger files 'adc apply' writes into the configs Secret.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    jceksStorePath: /path/to/ranger.jceks

  ## Java KeyStore/TrustStore certificate configuration.
  ## Either reference an existing Secret via secretName,
  ## or set files: to have the CLI create the Secret from local files.
  #ssl:
  #  # Name of the Secret containing Java keystores.
  #  secretName: ssl-secret
  #
  #  # Key in the Secret containing the truststore file.
  #  trustStoreKey: truststore.jks
  #
  #  ## Password for the truststore (optional).
  #  #trustStorePassword: bigdata
  #
  #  ## Key in the Secret containing the keystore file (optional).
  #  #keyStoreKey: keystore.jks
  #
  #  ## Password for the keystore (optional).
  #  #keyStorePassword: bigdata
  #
  #  # Local files 'adc apply' puts into the Secret named by ssl.secretName.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    trustStorePath: /path/to/truststore.jks
  #    #keyStorePath: /path/to/keystore.jks
  coordinator:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  http-server.http.port: "8080"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata
  worker:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  http-server.http.port: "8080"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata

  ## Trino catalogs (one entry per .properties file).
  ## adc apply derives Kerberos/SSL fields for iceberg catalogs from the surrounding cluster config.
  #catalogs:
  #  iceberg.properties:
  #    connector.name: iceberg
  #    fs.hadoop.enabled: "true"
  #    hive.config.resources: /opt/trino-server/etc/catalog/core-site.xml
  #    hive.hdfs.impersonation.enabled: "true"
  #    hive.metastore.thrift.impersonation.enabled: "true"
  #    hive.metastore.uri: thrift://tsn-adh-k8s-1.ru-central1.internal:9083

  ## HTTPS on the Trino coordinator web endpoint.
  ## Reference a keystore Secret via secretName and keystoreKey,
  ## or set files: to have the CLI create the Secret from a local keystore.
  #webTLS:
  #  secretName: trino-web-tls
  #  keystoreKey: keystore.p12
  #  #keystorePassword: changeit
  #
  #  # Local keystore 'adc apply' puts into the Secret named by webTLS.secretName.
  #  # A relative path is resolved against the config file.
  #  files:
  #    keystorePath: ./keystore.p12

Impala bundle

Initializing both Impala operator and Impala cluster in one configuration file:

$ adc init --impala-operator \
           --impala-cluster \
           --name impala \
           -n adh \
           -o bundle.yaml

The generated file contains configurations for both objects separated by the --- line. Both objects share the same metadata.name and metadata.namespace.

bundle.yaml
apiVersion: adc.arenadata.io/v1alpha1
kind: ImpalaCluster
metadata:
  name: impala
  namespace: adh
spec:
  image: hub.adsw.io/adh-enterprise/impala-docker:4.5.0_arenadata1-adh-4.2.0-x86_64

  ## Image pull secret for a private registry.
  ## Either set 'name' to reference an existing Secret,
  ## or set 'credentials' and the CLI will create a dockerconfigjson Secret.
  #imagePullSecret:
  #  name: my-pull-secret
  #  credentials:
  #    registry: registry.example.com
  #    username: user
  #    password: pass
  hadoop:
    core:
      fs.defaultFS: ""
    hdfs:
      dfs.encrypt.data.transfer.cipher.suites: AES/CTR/NoPadding
    hive:
      hive.metastore.uris: ""

  ## Kerberos configuration for authentication.
  #kerberos:
  #  realm: EXAMPLE.COM
  #
  #  # Service name in the Kerberos principal. Defaults to the product name.
  #  service: impala
  #
  #  # Hostname in the Kerberos principal.
  #  # Required for a fixed service principal; leave it empty only to derive one principal per pod from the cluster domain.
  #  hostname: kerberos.example.com
  #  keytab:
  #    # true - kerberos-operator creates the keytab Secret.
  #    # false (default) - reference an existing keytab Secret with name keytab.secretName.
  #    create: false
  #
  #    # Name of the keytab Secret.
  #    # Optional when create: true - names the generated Secret (default: <name>-keytab).
  #    # Required when create: false - must reference an existing Secret.
  #    secretName: kerberos-secret
  #
  #    # Label selector for the Pod that generates the keytab.
  #    # Required when create: true; ignored when create: false.
  #    labelSelector:
  #      env: prod
  #    #additionalPrincipals:
  #    #  - HTTP/kerberos.example.com
  #    clusterDomain: cluster.local
  #    rotation:
  #      interval: 24h
  #      checkInterval: 1h

  ## LDAP authentication configuration.
  ## Uncomment and fill url and userBindPattern.
  ## For ldaps:// URLs the ssl: or ca: section must also be configured (depends on product)
  #ldap:
  #  # LDAP service url.
  #  url: ldaps://ldap.example.com:636
  #
  #  # LDAP user Bind pattern.
  #  userBindPattern: uid=#UID,cn=users,dc=example,dc=com

  ## Ranger plugin configuration.
  ## Uncomment and fill the lines below. adc apply derives the rest.
  #ranger:
  #  # fill ranger.plugin.impala.policy.rest.url below with Ranger endpoint, e.g. https://adps-adc.ru-central1.internal:6182
  #  # fill ranger.plugin.impala.service.name below with Ranger service name you want to use for product, e.g. adc_impala_id_1
  #  security:
  #    ranger.plugin.impala.policy.rest.url: ""
  #    ranger.plugin.impala.service.name: ""
  #
  #  # fill xasecure.audit.destination.solr.zookeepers below with Zookeepers endpoints to resolve solr service, e.g. adps-adc.ru-central1.internal:2181/Arenadata.Hadoop-2.solr.server
  #  audit:
  #    xasecure.audit.destination.solr.zookeepers: ""
  #
  #  # Local Ranger files 'adc apply' writes into the configs Secret.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    jceksStorePath: /path/to/ranger.jceks

  ## Java KeyStore/TrustStore certificate configuration.
  ## Either reference an existing Secret via secretName,
  ## or set files: to have the CLI create the Secret from local files.
  #ssl:
  #  # Name of the Secret containing Java keystores.
  #  secretName: ssl-secret
  #
  #  # Key in the Secret containing the truststore file.
  #  trustStoreKey: truststore.jks
  #
  #  ## Password for the truststore (optional).
  #  #trustStorePassword: bigdata
  #
  #  ## Key in the Secret containing the keystore file (optional).
  #  #keyStoreKey: keystore.jks
  #
  #  ## Password for the keystore (optional).
  #  #keyStorePassword: bigdata
  #
  #  # Local files 'adc apply' puts into the Secret named by ssl.secretName.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    trustStorePath: /path/to/truststore.jks
  #    #keyStorePath: /path/to/keystore.jks
  catalog:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  redirect_stdout_stderr: "false"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata
  coordinator:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  redirect_stdout_stderr: "false"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata
  executor:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  redirect_stdout_stderr: "false"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata
  statestore:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  redirect_stdout_stderr: "false"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata

  ## TLS certificate configuration.
  ## Either reference an existing Secret via secretName,
  ## or set files: to have the CLI create the Secret from local files.
  #tls:
  #  # Name of the Secret containing TLS certificate and key.
  #  secretName: tls-secret
  #
  #  # Key in the Secret containing the TLS certificate.
  #  certificateKey: tls.crt
  #
  #  # Key in the Secret containing the TLS private key.
  #  privateKey: tls.key
  #
  #  ## Key in the Secret containing the client CA certificate.
  #  #clientCaCertificate: ca.crt
  #
  #  # Local files 'adc apply' puts into the Secret named by tls.secretName.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    certificatePath: /path/to/tls.crt
  #    privateKeyPath: /path/to/tls.key
  #    #clientCaCertificatePath: /path/to/ca.crt

  ## TLS certificate configuration for web UI and HTTP endpoints.
  ## Either reference an existing Secret via secretName,
  ## or set files: to have the CLI create the Secret from local files.
  #webTLS:
  #  # Name of the Secret containing web TLS certificate and key.
  #  secretName: web-tls-secret
  #
  #  # Key in the Secret containing the web TLS certificate.
  #  certificateKey: tls.crt
  #
  #  # Key in the Secret containing the web TLS private key.
  #  privateKey: tls.key
  #
  #  # Local files 'adc apply' puts into the Secret named by webTLS.secretName.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    certificatePath: /path/to/tls.crt
  #    privateKeyPath: /path/to/tls.key

  ## CA certificate configuration for Impala.
  ## Specify a Secret and the key containing the CA certificate.
  #ca:
  #  # Name of the Secret containing the CA certificate.
  #  secretName: ca-secret
  #
  #  # Key in the Secret containing the CA certificate.
  #  certificateKey: ca.crt
  #
  #  # Local files 'adc apply' puts into the Secret named by ca.secretName.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    certificatePath: /path/to/ca.pem
---
apiVersion: adc.arenadata.io/v1alpha1
kind: ImpalaOperator
metadata:
  name: impala
  namespace: adh
spec:
  image: hub.adsw.io/ng/impala-operator:1.73.0

  # Number of replicas
  # replicas: 1

  resources:
    limits:
      cpu: 500m
      memory: 256Mi

  # Operator ServiceAccount. create: true (default) also creates the manager and per-payload-namespace Role/RoleBinding bound to it; create: false skips all three - name then refers to a ServiceAccount (and RBAC) managed entirely outside the CLI.
  serviceAccount:
    create: true
    name: ""

  # Whether the CLI creates the product namespace.
  # The namespace name is set in metadata.namespace.
  namespace:
    create: false

  # Create namespaces to run the payload.
  createPayloadNamespaces: true

  # List of namespaces to run the payload in.
  payloadNamespaces:
    - adh

  ## Image pull secret for a private registry.
  ## Either set 'name' to reference an existing Secret,
  ## or set 'credentials' and the CLI will create a dockerconfigjson Secret.
  #imagePullSecret:
  #  name: my-pull-secret
  #  credentials:
  #    registry: registry.example.com
  #    username: user
  #    password: pass
Found a mistake? Seleсt text and press Ctrl+Enter to report it