Impala resource pool management on Kubernetes via CLI

Sergei Tikhomirov

Prerequisites

  • Impala is deployed in Kubernetes according to the instruction.

Update the Impala cluster configuration

  1. Edit the impala-cluster.yaml configuration file by adding the resource pools configuration block:

    impala-cluster.yaml
    apiVersion: adc.arenadata.io/v1alpha1
kind: ImpalaCluster
metadata:
  name: impala
  namespace: impala
spec:
  image: hub.arenadata.io/adc-enterprise/impala:<tag>

  ## Image pull secret for a private registry.
  ## Set 'externalSecretName' to reference an existing Secret,
  ## or set 'credentials' and optionally 'secretName' to let the CLI create one.
  #imagePullSecret:
  #
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-registry-secret
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-registry-secret
  #  #credentials:
  #  #  registry: registry.example.com
  #  #  username: user
  #  #  password: pass
  hadoop:
    core:
      dfs.client.failover.proxy.provider.adh: org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
      dfs.ha.namenodes.adh: nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3
      dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:8020
      dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:8020
      dfs.nameservices: adh
      fs.defaultFS: hdfs://adh
      hadoop.security.authentication: simple
    hdfs:
      dfs.client.read.shortcircuit: "false"
    hive:
      hive.metastore.sasl.enabled: "false"
      hive.metastore.uris: thrift://tsn-adh-k8s-1.ru-central1.internal:9083
      metastore.use.SSL: "false"
    ozone:
      ozone.om.address.adh.om_tsn-adh-k8s-1: tsn-adh-k8s-1.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-adh-k8s-2: tsn-adh-k8s-2.ru-central1.internal:9862
      ozone.om.address.adh.om_tsn-adh-k8s-3: tsn-adh-k8s-3.ru-central1.internal:9862
      ozone.om.nodes.adh: om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3
      ozone.om.service.ids: adhom

  ## Kerberos configuration for authentication.
  #kerberos:
  #  realm: EXAMPLE.COM
  #
  #  # Service name in the Kerberos principal. Defaults to the product name.
  #  service: impala
  #
  #  # Hostname in the Kerberos principal.
  #  # Required for a fixed service principal; leave it empty only to derive one principal per pod from the cluster domain.
  #  hostname: kerberos.example.com
  #  keytab:
  #    # true - kerberos-operator creates the keytab Secret.
  #    # false (default) - reference an existing keytab Secret with name keytab.secretName.
  #    create: false
  #
  #    # Name of the keytab Secret.
  #    # Optional when create: true - names the generated Secret (default: <name>-keytab).
  #    # Required when create: false - must reference an existing Secret.
  #    secretName: kerberos-secret
  #
  #    # Label selector for the Pod that generates the keytab.
  #    # Required when create: true; ignored when create: false.
  #    labelSelector:
  #      env: prod
  #    #additionalPrincipals:
  #    #  - HTTP/kerberos.example.com
  #    clusterDomain: cluster.local
  #    rotation:
  #      interval: 24h
  #      checkInterval: 1h

  ## LDAP authentication configuration.
  ## Uncomment and fill url and userBindPattern.
  ## For ldaps:// URLs the ssl: or ca: section must also be configured (depends on product)
  #ldap:
  #  # LDAP service url.
  #  url: ldaps://ldap.example.com:636
  #
  #  # LDAP user Bind pattern.
  #  userBindPattern: uid=#UID,cn=users,dc=example,dc=com

  ## Ranger plugin configuration.
  ## Uncomment and fill the lines below. adc apply derives the rest.
  #ranger:
  #  # fill ranger.plugin.impala.policy.rest.url below with Ranger endpoint, e.g. https://adps-adc.ru-central1.internal:6182
  #  # fill ranger.plugin.impala.service.name below with Ranger service name you want to use for product, e.g. adc_impala_id_1
  #  security:
  #    ranger.plugin.impala.policy.rest.url: ""
  #    ranger.plugin.impala.service.name: ""
  #
  #  # fill xasecure.audit.destination.solr.zookeepers below with Zookeepers endpoints to resolve solr service, e.g. adps-adc.ru-central1.internal:2181/Arenadata.Hadoop-2.solr.server
  #  audit:
  #    xasecure.audit.destination.solr.zookeepers: ""
  #
  #  # Local Ranger files 'adc apply' writes into the configs Secret.
  #  # Relative paths are resolved against the config file.
  #  files:
  #    jceksStorePath: /path/to/ranger.jceks

  ## Java KeyStore/TrustStore certificate configuration.
  ## Set externalSecretName to reference an existing Secret,
  ## or set files and optional secretName to have ADC create it.
  #ssl:
  #  ## Name of the Secret containing Java keystores.
  #  #secretName: custom-ssl-secret
  #  externalSecretName: existing-ssl-secret
  #
  #  # Key in the Secret containing the truststore file.
  #  trustStoreKey: truststore.jks
  #
  #  ## Password for the truststore (optional).
  #  #trustStorePassword: bigdata
  #
  #  ## Key in the Secret containing the keystore file (optional).
  #  #keyStoreKey: keystore.jks
  #
  #  ## Password for the keystore (optional).
  #  #keyStorePassword: bigdata
  #
  #  ## Local files 'adc apply' puts into the Secret named by ssl.secretName.
  #  ## Relative paths are resolved against the config file.
  #  #files:
  #  #  trustStorePath: /path/to/truststore.jks
  #  #  #keyStorePath: /path/to/keystore.jks

  ## Use an external complete configs Secret instead of the one rendered by ADC.
  #configsSecret:
  #  # Use a Secret managed outside ADC.
  #  externalSecretName: existing-impala-configs
  #
  #  ## Or let ADC create the Secret.
  #  #secretName: custom-impala-configs
  catalog:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  redirect_stdout_stderr: "false"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata
  coordinator:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  redirect_stdout_stderr: "false"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata
  executor:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  redirect_stdout_stderr: "false"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata
  statestore:
    replicas: 1
    #resources:
    #  limits:
    #    cpu: 500m
    #    memory: 512Mi
    #  requests:
    #    cpu: 250m
    #    memory: 256Mi

    ## Component arguments. Key-value pairs passed to the component configuration.
    #args:
    #  redirect_stdout_stderr: "false"

    ## Environment variables passed to the component container.
    #envs:
    #  - name: JAVA_TOOL_OPTIONS
    #    value: |-
    #      -Djavax.net.ssl.trustStore=/etc/ssl/truststore.jks
    #      -Djavax.net.ssl.trustStorePassword=bigdata

  ## Monitoring configuration.
  #monitoring:
  #  # Enables Prometheus metrics export from this product's pods.
  #  exportMetrics: true
  #
  #  ## Renders a PodMonitor or VMPodScrape for the metrics endpoint.
  #  #scrape:
  #  #  # Type of monitoring operator. PodMonitor for Prometheus, VMPodScrape for VictoriaMetrics.
  #  #  type: PodMonitor
  #  #
  #  #  ## Interval for scrapping metrics.
  #  #  #interval: 30s
  #  #
  #  #  ## Metadata labels added to the PodMonitor or VMPodScrape resource.
  #  #  #labels:
  #  #  #  release: prometheus
  #  #
  #  #  ## HTTPS settings used by PodMonitor, VMPodScrape or vmagent when scraping metrics.
  #  #  #tls:
  #  #  #  ## CA certificate source used to verify the metrics endpoint.
  #  #  #  #ca:
  #  #  #  #
  #  #  #  #  # Use a Secret managed outside ADC.
  #  #  #  #  externalSecretName: existing-product-metrics-ca
  #  #  #  #
  #  #  #  #  ## Or let ADC create the Secret.
  #  #  #  #  #secretName: product-metrics-ca
  #  #  #  #
  #  #  #  #  # Key containing the CA certificate in the referenced Secret.
  #  #  #  #  certificateKey: ca.crt
  #  #  #  #
  #  #  #  #  ## Local CA certificate read by ADC to create the configured Secret.
  #  #  #  #  #files:
  #  #  #  #  #  certificatePath: /path/to/ca.crt
  #  #  #
  #  #  #  # Skip verification of the metrics endpoint certificate. Do not use together with ca.
  #  #  #  insecureSkipVerify: true

  ## Admission-control resource pools. Rendered into a coordinator-only Secret (fair-scheduler.xml and optional llama-site.xml) mounted at /opt/impala/resource-pools.
  resourcePools:
  #  # Fair-scheduler queue tree and placement policy, rendered to fair-scheduler.xml.
    allocations:
  #    # Queue tree, rooted at a single queue.
      queues:
        - aclSubmitApps: ' '
          name: root
          queues:
          - aclSubmitApps: '*'
            maxResources:
              memory: 50000
              vcores: 0
            name: default
            type: leaf
          type: parent
  #
  #    # Rules that route an incoming query to a queue.
      queuePlacementPolicy:
        rules:
          - create: false
            name: specified
          - name: default
  #
  #  # Raw llama-site.xml properties, rendered verbatim in list order.
    llamaProperties:
  #    # List of name/value pairs written to llama-site.xml.
      properties:
        - name: llama.am.throttling.maximum.placed.reservations.root.default
          value: "10"

  ## TLS certificate configuration.
  ## Set externalSecretName to reference an existing Secret,
  ## or set files and optional secretName to have ADC create it.
  #tls:
  #  ## Optional name of the Secret ADC creates from local files.
  #  #secretName: custom-tls-secret
  #  externalSecretName: existing-tls-secret
  #
  #  # Key in the Secret containing the TLS certificate.
  #  certificateKey: tls.crt
  #
  #  # Key in the Secret containing the TLS private key.
  #  privateKey: tls.key
  #
  #  ## Key in the Secret containing the client CA certificate.
  #  #clientCaCertificate: ca.crt
  #
  #  ## Local files 'adc apply' puts into the Secret named by tls.secretName.
  #  ## Relative paths are resolved against the config file.
  #  #files:
  #  #  certificatePath: /path/to/tls.crt
  #  #  privateKeyPath: /path/to/tls.key
  #  #  #clientCaCertificatePath: /path/to/ca.crt

  ## TLS certificate configuration for web UI and HTTP endpoints.
  ## Set externalSecretName to reference an existing Secret,
  ## or set files and optional secretName to have ADC create it.
  #webTLS:
  #  ## Optional name of the Secret ADC creates from local files.
  #  #secretName: custom-web-tls-secret
  #  externalSecretName: existing-web-tls-secret
  #
  #  # Key in the Secret containing the web TLS certificate.
  #  certificateKey: tls.crt
  #
  #  # Key in the Secret containing the web TLS private key.
  #  privateKey: tls.key
  #
  #  ## Local files 'adc apply' puts into the Secret named by webTLS.secretName.
  #  ## Relative paths are resolved against the config file.
  #  #files:
  #  #  certificatePath: /path/to/tls.crt
  #  #  privateKeyPath: /path/to/tls.key

  ## CA certificate configuration for Impala.
  ## Set externalSecretName, or let ADC create a Secret from files.
  #ca:
  #  ## Optional name of the Secret ADC creates from a local file.
  #  #secretName: custom-ca-secret
  #  externalSecretName: existing-ca-secret
  #
  #  # Key in the Secret containing the CA certificate.
  #  certificateKey: ca.crt
  #
  #  ## Local files 'adc apply' puts into the Secret named by ca.secretName.
  #  ## Relative paths are resolved against the config file.
  #  #files:
  #  #  certificatePath: /path/to/ca.pem

  ## Controls whether Secret/ConfigMap changes restart pods.
  ## Set enabled: false to update referenced Secrets without restarting
  ## the workload; pods keep running the previous configuration until
  ## the policy is re-enabled. Defaults to enabled.
  #configurationRollout:
  #  enabled: false

  2. You can check the configuration about to be applied by running the apply command with the --dry-run option:

    $ ./adc apply -f impala-cluster.yaml --dry-run > impala-cluster-render.yaml
    impala-cluster-render.yaml
    ---
apiVersion: v1
kind: Secret
metadata:
  name: impala-configs
  namespace: impala
stringData:
  core-site.xml: |-
    <configuration>
      <property>
        <name>dfs.client.failover.proxy.provider.adh</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider</value>
      </property>
      <property>
        <name>dfs.client.read.shortcircuit</name>
        <value>false</value>
      </property>
      <property>
        <name>dfs.ha.namenodes.adh</name>
        <value>nn_tsn-adh-k8s-1,nn_tsn-adh-k8s-3</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-1</name>
        <value>tsn-adh-k8s-1.ru-central1.internal:8020</value>
      </property>
      <property>
        <name>dfs.namenode.rpc-address.adh.nn_tsn-adh-k8s-3</name>
        <value>tsn-adh-k8s-3.ru-central1.internal:8020</value>
      </property>
      <property>
        <name>dfs.nameservices</name>
        <value>adh</value>
      </property>
      <property>
        <name>fs.defaultFS</name>
        <value>hdfs://adh</value>
      </property>
      <property>
        <name>hadoop.security.authentication</name>
        <value>simple</value>
      </property>
      <property>
        <name>ozone.om.address.adh.om_tsn-adh-k8s-1</name>
        <value>tsn-adh-k8s-1.ru-central1.internal:9862</value>
      </property>
      <property>
        <name>ozone.om.address.adh.om_tsn-adh-k8s-2</name>
        <value>tsn-adh-k8s-2.ru-central1.internal:9862</value>
      </property>
      <property>
        <name>ozone.om.address.adh.om_tsn-adh-k8s-3</name>
        <value>tsn-adh-k8s-3.ru-central1.internal:9862</value>
      </property>
      <property>
        <name>ozone.om.nodes.adh</name>
        <value>om_tsn-adh-k8s-1,om_tsn-adh-k8s-2,om_tsn-adh-k8s-3</value>
      </property>
      <property>
        <name>ozone.om.service.ids</name>
        <value>adhom</value>
      </property>
    </configuration>
  hive-site.xml: |-
    <configuration>
      <property>
        <name>hive.metastore.sasl.enabled</name>
        <value>false</value>
      </property>
      <property>
        <name>hive.metastore.uris</name>
        <value>thrift://tsn-adh-k8s-1.ru-central1.internal:9083</value>
      </property>
      <property>
        <name>metastore.use.SSL</name>
        <value>false</value>
      </property>
    </configuration>
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: impala-scheduler-config
  namespace: impala
stringData:
  fair-scheduler.xml: |-
    <allocations>
      <queue name="root">
        <aclSubmitApps> </aclSubmitApps>
        <queue name="default">
          <maxResources>50000 mb, 0 vcores</maxResources>
          <aclSubmitApps>*</aclSubmitApps>
        </queue>
      </queue>
      <queuePlacementPolicy>
        <rule name="specified" create="false"></rule>
        <rule name="default"></rule>
      </queuePlacementPolicy>
    </allocations>
  llama-site.xml: |-
    <configuration>
      <property>
        <name>llama.am.throttling.maximum.placed.reservations.root.default</name>
        <value>10</value>
      </property>
    </configuration>
type: Opaque
---
apiVersion: impala.arenadata.io/v1alpha1
kind: Cluster
metadata:
  name: impala
  namespace: impala
spec:
  catalog:
    metadata: {}
    replicas: 1
    spec:
      image: hub.arenadata.io/adc-enterprise/impala:<tag>
      imagePullPolicy: Always
  coordinator:
    metadata: {}
    replicas: 1
    spec:
      args:
        fair_scheduler_allocation_path: /opt/impala/resource-pools/fair-scheduler.xml
        llama_site_path: /opt/impala/resource-pools/llama-site.xml
      image: hub.arenadata.io/adc-enterprise/impala:<tag>
      imagePullPolicy: Always
      mountSecrets:
      - mountPath: /opt/impala/resource-pools
        options:
          containerName: app
          readOnly: true
        secretName: impala-scheduler-config
  executor:
    metadata: {}
    replicas: 1
    spec:
      image: hub.arenadata.io/adc-enterprise/impala:<tag>
      imagePullPolicy: Always
  hadoopConfigsSecretName: impala-configs
  statestore:
    metadata: {}
    replicas: 1
    spec:
      image: hub.arenadata.io/adc-enterprise/impala:<tag>
      imagePullPolicy: Always
status: {}

  3. If the manifest is correct, apply the configuration and deploy the Impala cluster:

    $ ./adc apply -f impala-cluster.yaml

    The expected output contains a confirmation of success:

    time="20260518133858UTC" level="info" msg="cluster impala applied to namespace impala"

  4. Verify the Impala cluster pods:

    $ kubectl get pods -n impala

    The expected output is:

    NAME                   READY   STATUS    RESTARTS   AGE
impala-catalog-0       1/1     Running   0          53s
impala-coordinator-0   1/1     Running   0          53s
impala-executor-0      1/1     Running   0          53s
impala-statestore-0    1/1     Running   0          53s

Check the applied restrictions

You can see the resource pool configuration on the /admissions page of the Impala web UI. The pool settings refresh once a query is submitted to the pool.

Also, you can try running a query that requires more memory or cores than you’ve set as a limit. If the configuration is saved correctly, your query will fail to run.

Found a mistake? Seleсt text and press Ctrl+Enter to report it