Configuration parameters

airflow_dir

The Airflow home directory

/srv/airflow/home

db_dir

The location of Metastore DB

/srv/airflow/metastore

db_user

The user to connect to Metadata DB

airflow

db_password

The password to connect to Metadata DB

 — 

db_root_password

The root password to connect to Metadata DB

 — 

db_port

The port to connect to Metadata DB

3307

server_port

The port to run the web server

8080

flower_port

The port that Celery Flower runs on

5555

worker_port

When you start an Airflow Worker, Airflow starts a tiny web server subprocess to serve the Workers local log files to the Airflow main web server, which then builds pages and sends them to users. This defines the port, on which the logs are served. The port must be free and accessible from the main web server to connect to the Workers

8793

redis_port

The port for running Redis

6379

fernet_key

The secret key to save connection passwords in the database

 — 

security

Defines which security module to use. For example, kerberos

 — 

keytab

The path to the keytab file

 — 

reinit_frequency

Sets the ticket renewal frequency

3600

principal

The Kerberos principal

ssl_active

Defines if SSL is active for Airflow

false

web_server_ssl_cert

The path to SSL certificate

/etc/ssl/certs/host_cert.cert

web_server_ssl_key

The path to SSL certificate key

/etc/ssl/host_cert.key

Logging level

Specifies the logging level for Airflow activity

INFO

Logging level for Flask-appbuilder UI

Specifies the logging level for Flask-appbuilder UI

WARNING

cfg_properties_template

The Jinja template to initialize environment variables for Airflow

Database type

The external database type. Possible values: PostgreSQL, MySQL/MariaDB

MySQL/MariaDB

Hostname

The external database host

 — 

Custom port

The external database port

 — 

Airflow database name

The external database name

airflow

jobmanager.rpc.port

The RPC port through which the JobManager is reachable. In the high availability mode, this value is ignored and the port number to connect to JobManager is generated by ZooKeeper

6123

sql-gateway.endpoint.rest.port

A port to connect to the SQL Gateway service

8083

taskmanager.network.bind-policy

The automatic address binding policy used by the TaskManager

name

parallelism.default

The system-wide default parallelism level for all execution environments

1

taskmanager.numberOfTaskSlots

The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline

1

taskmanager.heap.size

The heap size for the TaskManager JVM

1024m

jobmanager.heap.size

The heap size for the JobManager JVM

1024m

security.kerberos.login.use-ticket-cache

Indicates whether to read from the Kerberos ticket cache

false

security.kerberos.login.keytab

The absolute path to the Kerberos keytab file that stores user credentials

 — 

security.kerberos.login.principal

Flink Kerberos principal

 — 

security.kerberos.login.contexts

A comma-separated list of login contexts to provide the Kerberos credentials to

 — 

security.ssl.rest.enabled

Turns on SSL for external communication via REST endpoints

false

security.ssl.rest.keystore

The Java keystore file with SSL key and certificate to be used by Flink’s external REST endpoints

 — 

security.ssl.rest.truststore

The truststore file containing public CA certificates to verify the peer for Flink’s external REST endpoints

 — 

security.ssl.rest.keystore-password

The secret to decrypt the keystore file for Flink external REST endpoints

 — 

security.ssl.rest.truststore-password

The password to decrypt the truststore for Flink’s external REST endpoints

 — 

security.ssl.rest.key-password

The secret to decrypt the key in the keystore for Flink’s external REST endpoints

 — 

Logging level

Defines the logging level for Flink activity

INFO

high-availability

Defines the High Availability (HA) mode used for cluster execution

 — 

high-availability.zookeeper.quorum

The ZooKeeper quorum to use when running Flink in the HA mode with ZooKeeper

 — 

high-availability.storageDir

A file system path (URI) where Flink persists metadata in the HA mode

 — 

high-availability.zookeeper.path.root

The root path for Flink ZNode in Zookeeper

/flink

high-availability.cluster-id

The ID of the Flink cluster used to separate multiple Flink clusters from each other

 — 

sql-gateway.session.check-interval

The check interval to detect idle sessions. A value <= 0 disables the checks

1 min

sql-gateway.session.idle-timeout

The timeout to close a session if no successful connection was made during this interval. A value <= 0 never closes the sessions

10 min

sql-gateway.session.max-num

The maximum number of sessions to run simultaneously

1000000

sql-gateway.worker.keepalive-time

The time to keep an idle worker thread alive. When the worker thread count exceeds sql-gateway.worker.threads.min, excessive threads are killed after this time interval

5 min

sql-gateway.worker.threads.max

The maximum number of worker threads on the SQL Gateway server

500

sql-gateway.worker.threads.min

The minimum number of worker threads. If the current number of worker threads is less than this value, the worker threads are not deleted automatically

500

zookeeper.sasl.disable

Defines the SASL authentication in Zookeeper

false

Custom flink-conf.yaml

 — 

log4j.properties

The contents of the log4j.properties configuration file

log4j-cli.properties

The contents of the log4j-cli.properties configuration file

hbase.balancer.period

The time period to run the Region balancer in Master

300000

hbase.client.pause

General client pause value. Used mostly as value to wait before running a retry of a failed get, region lookup, etc. See hbase.client.retries.number for description of how this pause works with retries

100

hbase.client.max.perregion.tasks

The maximum number of concurrent mutation tasks the Client will maintain to a single Region. That is, if there is already hbase.client.max.perregion.tasks writes in progress for this Region, new puts won’t be sent to this Region, until some writes finishes

1

hbase.client.max.perserver.tasks

The maximum number of concurrent mutation tasks a single HTable instance will send to a single Region Server

2

hbase.client.max.total.tasks

The maximum number of concurrent mutation tasks, a single HTable instance will send to the cluster

100

hbase.client.retries.number

The maximum number of retries. It is used as maximum for all retryable operations, such as: getting a cell value, starting a row update, etc. Retry interval is a rough function based on hbase.client.pause. See the constant RETRY_BACKOFF for how the backup ramps up. Change this setting and hbase.client.pause to suit your workload

15

hbase.client.scanner.timeout.period

The Client scanner lease period in milliseconds

60000

hbase.cluster.distributed

The cluster mode. Possible values are: false — for standalone mode and pseudo-distributed setups with managed ZooKeeper; true — for fully-distributed mode with unmanaged ZooKeeper Quorum. If false, the startup will run all HBase and ZooKeeper daemons together in the one JVM, if true — one JVM instance per daemon

true

hbase.hregion.majorcompaction

The time interval between Major compactions in milliseconds. Set to 0 to disable time-based automatic Major compactions. User-requested and size-based Major compactions will still run. This value is multiplied by hbase.hregion.majorcompaction.jitter to cause compaction to start at a somewhat-random time during a given time frame

604800000

hbase.hregion.max.filesize

The maximum file size. If the total size of some Region HFiles has grown to exceed this value, the Region is split in two. There are two options of how this option works: the first is when any store size exceeds the threshold — then split, and the other is if overall Region size exceeds the threshold — then split. It can be configured by hbase.hregion.split.overallfiles

10737418240

hbase.hstore.blockingStoreFiles

If more than this number of StoreFiles exists in any Store (one StoreFile is written per flush of MemStore), updates are blocked for this Region, until a compaction is completed, or until hbase.hstore.blockingWaitTime is exceeded

16

hbase.hstore.blockingWaitTime

The time for which a Region will block updates after reaching the StoreFile limit, defined by hbase.hstore.blockingStoreFiles. After this time is elapsed, the Region will stop blocking updates, even if a compaction has not been completed

90000

hbase.hstore.compaction.max

The maximum number of StoreFiles that will be selected for a single Minor compaction, regardless of the number of eligible StoreFiles. Effectively, the value of hbase.hstore.compaction.max controls the time it takes for a single compaction to complete. Setting it larger means that more StoreFiles are included in a compaction. For most cases, the default value is appropriate

10

hbase.hstore.compaction.min

The minimum number of StoreFiles that must be eligible for compaction before compaction can run. The goal of tuning hbase.hstore.compaction.min is to avoid a situation with too many tiny StoreFiles to compact. Setting this value to 2 would cause a Minor compaction each time you have two StoreFiles in a Store, and this is probably not appropriate. If you set this value too high, all the other values will need to be adjusted accordingly. For most cases, the default value is appropriate. In the previous versions of HBase, the parameter hbase.hstore.compaction.min was called hbase.hstore.compactionThreshold

3

hbase.hstore.compaction.min.size

A StoreFile, smaller than this size, will always be eligible for Minor compaction. StoreFiles this size or larger are evaluated by hbase.hstore.compaction.ratio to determine, if they are eligible. Because this limit represents the "automatic include" limit for all StoreFiles smaller than this value, this value may need to be reduced in write-heavy environments, where many files in the 1-2 MB range are being flushed, because every StoreFile will be targeted for compaction and the resulting StoreFiles may still be under the minimum size and require further compaction. If this parameter is lowered, the ratio check is triggered more quickly. This addressed some issues seen in earlier versions of HBase, but changing this parameter is no longer necessary in most situations

134217728

hbase.hstore.compaction.ratio

For Minor compaction, this ratio is used to determine, whether a given StoreFile that is larger than hbase.hstore.compaction.min.size, is eligible for compaction. Its effect is to limit compaction of large StoreFile. The value of hbase.hstore.compaction.ratio is expressed as a floating-point decimal

1.2F

hbase.hstore.compaction.ratio.offpeak

The compaction ratio used during off-peak compactions if the off-peak hours are also configured. Expressed as a floating-point decimal. This allows for more aggressive (or less aggressive, if you set it lower than hbase.hstore.compaction.ratio) compaction during a given time period. The value is ignored if off-peak is disabled (default). This works the same as hbase.hstore.compaction.ratio

5.0F

hbase.hstore.compactionThreshold

If more than this number of StoreFiles exists in any Store (one StoreFile is written per flush of MemStore), a compaction is run to rewrite all StoreFiles into a single StoreFile. Larger values delay the compaction, but when compaction does occur, it takes longer to complete

3

hbase.hstore.flusher.count

The number of flush threads. With fewer threads, the MemStore flushes will be queued. With more threads, the flushes will be executed in parallel, increasing the load on HDFS, and potentially causing more compactions

2

hbase.hstore.time.to.purge.deletes

The amount of time to delay purging of delete markers with future timestamps. If unset or set to 0, all the delete markers, including those with future timestamps, are purged during the next Major compaction. Otherwise, a delete marker is kept until the Major compaction that occurs after the marker timestamp plus the value of this setting (in milliseconds)

0

hbase.master.ipc.address

HMaster RPC

0.0.0.0

hbase.normalizer.period

The period at which the Region normalizer runs on Master (in milliseconds)

300000

hbase.regionserver.compaction.enabled

Enables/disables compactions by setting true/false. You can further switch compactions dynamically with the compaction_switch shell command

true

hbase.regionserver.ipc.address

Region Server RPC

0.0.0.0

hbase.regionserver.regionSplitLimit

The limit for the number of Regions, after which no more Region splitting should take place. This is not hard limit for the number of Regions, but acts as a guideline for the Region Server to stop splitting after a certain limit

1000

hbase.rootdir

The directory shared by Region Servers and into which HBase persists. The URL should be fully-qualified to include the filesystem scheme. For example, to specify the HDFS directory /hbase where the HDFS instance NameNode is running at namenode.example.org on port 9000, set this value to: hdfs://namenode.example.org:9000/hbase

 — 

hbase.zookeeper.quorum

A comma-separated list of servers in the ZooKeeper ensemble. For example, host1.mydomain.com,host2.mydomain.com,host3.mydomain.com. By default, this is set to localhost for local and pseudo-distributed modes of operation. For a fully-distributed setup, this should be set to a full list of ZooKeeper ensemble servers. If HBASE_MANAGES_ZK is set in hbase-env.sh, this is the list of servers, which HBase will start/stop ZooKeeper on, as part of cluster start/stop. Client-side, the list of ensemble members is put together with the hbase.zookeeper.property.clientPort config and is passed to the ZooKeeper constructor as the connection string parameter

 — 

zookeeper.session.timeout

The ZooKeeper session timeout in milliseconds. It is used in two different ways. First, this value is processed by the ZooKeeper Client that HBase uses to connect to the ensemble. It is also used by HBase, when it starts a ZooKeeper Server (in that case the timeout is passed as the maxSessionTimeout). See more details in the ZooKeeper documentation. For example, if an HBase Region Server connects to a ZooKeeper ensemble that is also managed by HBase, then the session timeout will be the one specified by this configuration. But a Region Server that connects to an ensemble managed with a different configuration will be subjected to the maxSessionTimeout of that ensemble. So, even though HBase might propose using 90 seconds, the ensemble can have a max timeout, lower than this, and it will take precedence. The current default maxSessionTimeout that ZooKeeper ships with is 40 seconds, which is lower than HBase

90000

zookeeper.znode.parent

The root znode for HBase in ZooKeeper. All of the HBase ZooKeeper files configured with a relative path will go under this node. By default, all of the HBase ZooKeeper file paths are configured with a relative path, so they will all go under this directory unless changed

/hbase

hbase.rest.port

The port used by HBase Rest Servers

60080

hbase.zookeeper.property.authProvider.1

Specifies the ZooKeeper authentication method

hbase.security.authentication

Set the value to true to run HBase RPC with strong authentication

false

hbase.security.authentication.ui

Enables Kerberos authentication to HBase web UI with SPNEGO

 — 

hbase.security.authentication.spnego.kerberos.principal

The Kerberos principal for SPNEGO authentication

 — 

hbase.security.authentication.spnego.kerberos.keytab

The path to the Kerberos keytab file with principals to be used for SPNEGO authentication

 — 

hbase.security.authorization

Set the value to true to run HBase RPC with strong authorization

false

hbase.master.kerberos.principal

The Kerberos principal used to run the HMaster process

 — 

hbase.master.keytab.file

Full path to the Kerberos keytab file to use for logging in the configured HMaster server principal

 — 

hbase.regionserver.kerberos.principal

The Kerberos principal name that should be used to run the HRegionServer process

 — 

hbase.regionserver.keytab.file

Full path to the Kerberos keytab file to use for logging in the configured HRegionServer server principal

 — 

hbase.rest.authentication.type

REST Gateway Kerberos authentication type

 — 

hbase.rest.authentication.kerberos.principal

REST Gateway Kerberos principal

 — 

hbase.rest.authentication.kerberos.keytab

REST Gateway Kerberos principal

 — 

hbase.thrift.keytab.file

Thrift Kerberos keytab

 — 

hbase.rest.keytab.file

HBase REST gateway Kerberos keytab

 — 

hbase.rest.kerberos.principal

HBase REST gateway Kerberos principal

 — 

hbase.thrift.kerberos.principal

Thrift Kerberos principal

 — 

hbase.thrift.security.qop

 — 

phoenix.queryserver.keytab.file

The path to the Kerberos keytab file

 — 

phoenix.queryserver.kerberos.principal

The Kerberos principal to use when authenticating. If phoenix.queryserver.kerberos.http.principal is not defined, this principal specified will be also used to both authenticate SPNEGO connections and to connect to HBase

 — 

phoenix.queryserver.kerberos.keytab

The full path to the Kerberos keytab file to use for logging in the configured HMaster server principal

 — 

phoenix.queryserver.http.keytab.file

The keytab file to use for authenticating SPNEGO connections. This configuration must be specified if phoenix.queryserver.kerberos.http.principal is configured. phoenix.queryserver.keytab.file will be used if this property is undefined

 — 

phoenix.queryserver.http.kerberos.principal

The Kerberos principal to use when authenticating SPNEGO connections. phoenix.queryserver.kerberos.principal will be used if this property is undefined

phoenix.queryserver.kerberos.http.principal

Deprecated, use phoenix.queryserver.http.kerberos.principal instead

 — 

hbase.ssl.enabled

Defines whether SSL is enabled for web UIs

false

hadoop.ssl.enabled

Defines whether SSL is enabled for Hadoop RPC

false

ssl.server.keystore.location

The path to the keystore file

 — 

ssl.server.keystore.password

The password to the keystore

 — 

ssl.server.truststore.location

The path to the truststore to be used

 — 

ssl.server.truststore.password

The password to the truststore

 — 

ssl.server.keystore.keypassword

The password to the key in the keystore

 — 

hbase.rest.ssl.enabled

Defines whether SSL is enabled for HBase REST server

false

hbase.rest.ssl.keystore.store

The path to the keystore used by HBase REST server

 — 

hbase.rest.ssl.keystore.password

The password to the keystore

 — 

hbase.rest.ssl.keystore.keypassword

The password to the key in the keystore

 — 

HBASE Regionserver Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for HBase Region server

-Xms700m -Xmx9G

HBASE Master Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for HBase Master

-Xms700m -Xmx9G

Phoenix Queryserver Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for Phoenix Query server

-Xms700m -Xmx8G

HBASE Thrift2 server Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for HBase Thrift2 server

-Xms700m -Xmx8G

HBASE Rest server Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for HBase Rest server

-Xms200m -Xmx8G

xasecure.audit.destination.solr.batch.filespool.dir

The spool directory path

/srv/ranger/hdfs_plugin/audit_solr_spool

xasecure.audit.destination.solr.urls

Leave this property value empty or set it to NONE when using ZooKeeper to connect to Solr

 — 

xasecure.audit.destination.solr.zookeepers

Specifies the ZooKeeper connection string for the Solr destination

 — 

xasecure.audit.destination.solr.force.use.inmemory.jaas.config

Uses in-memory JAAS configuration file to connect to Solr

 — 

xasecure.audit.is.enabled

Enables Ranger audit

true

xasecure.audit.jaas.Client.loginModuleControlFlag

Specifies whether the success of the module is required, requisite, sufficient, or optional

 — 

xasecure.audit.jaas.Client.loginModuleName

The name of the authenticator class

 — 

xasecure.audit.jaas.Client.option.keyTab

The name of the keytab file to get the principal’s secret key

 — 

xasecure.audit.jaas.Client.option.principal

The name of the principal to be used

 — 

xasecure.audit.jaas.Client.option.serviceName

Represents a user or a service that wants to log in

 — 

xasecure.audit.jaas.Client.option.storeKey

Set this to true if you want the keytab or the principal’s key to be stored in the subject’s private credentials

false

xasecure.audit.jaas.Client.option.useKeyTab

Set this to true if you want the module to get the principal’s key from the keytab

false

ranger.plugin.hbase.policy.rest.url

 — 

ranger.plugin.hbase.service.name

 — 

ranger.plugin.hbase.policy.cache.dir

/srv/ranger/hbase/policycache

ranger.plugin.hbase.policy.pollIntervalMs

30000

ranger.plugin.hbase.policy.rest.client.connection.timeoutMs

The HBase Plugin RangerRestClient connection timeout (in milliseconds)

120000

ranger.plugin.hbase.policy.rest.client.read.timeoutMs

The HBase Plugin RangerRestClient read timeout (in milliseconds)

30000

ranger.plugin.hbase.policy.rest.ssl.config.file

The path to the RangerRestClient SSL config file for HBase plugin

/etc/hbase/conf/ranger-hbase-policymgr-ssl.xml

xasecure.policymgr.clientssl.keystore

The path to the keystore file used by Ranger

 — 

xasecure.policymgr.clientssl.keystore.credential.file

The path to the keystore credentials file

/etc/hbase/conf/ranger-hbase.jceks

xasecure.policymgr.clientssl.truststore.credential.file

The path to the truststore credentials file

/etc/hbase/conf/ranger-hbase.jceks

xasecure.policymgr.clientssl.truststore

The path to the truststore file used by Ranger

 — 

xasecure.policymgr.clientssl.keystore.password

The password to the keystore file

 — 

xasecure.policymgr.clientssl.truststore.password

The password to the truststore file

 — 

Custom hbase-site.xml

 — 

Custom hbase-env.sh

 — 

Ranger plugin enabled

false

Custom ranger-hbase-audit.xml

 — 

Custom ranger-hbase-security.xml

 — 

Custom ranger-hbase-security.xml

 — 

Custom ranger-hbase-policymgr-ssl.xml

 — 

Custom log4j.properties

Custom hadoop-metrics2-hbase.properties

The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The URI scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The URI authority is used to determine the host, port, etc. for a filesystem

 — 

The number of minutes between trash checkpoints. Should be smaller or equal to fs.trash.interval. Every time the checkpointer runs, it creates a new checkpoint out of current and removes checkpoints, created more than fs.trash.interval minutes ago

60

The number of minutes, after which the checkpoint gets deleted. If set to 0, the trash feature is disabled

1440

The base for other temporary directories

/tmp/hadoop-${user.name}

A comma-separated list of pairs <Host>:<Port>. Each corresponds to a ZooKeeper to be used by the Resource Manager for storing Resource Manager state

 — 

The buffer size for sequence files. The size of this buffer should probably be a multiple of hardware page size (4096 on Intel x86), and it determines, how much data is buffered during read and write operations

131072

The script name, that should be invoked to resolve DNS names to NetworkTopology names. Example: the script would take host.foo.bar as an argument, and return /rack1 as the output

 — 

ha.zookeeper.quorum

A list of ZooKeeper Server addresses, separated by commas, that are to be used by the ZKFailoverController in automatic failover

 — 

ipc.client.fallback-to-simple-auth-allowed

When a client is configured to attempt a secure connection, but attempts to connect to an insecure server, that server may instuct the client to switch to SASL SIMPLE (unsecure) authentication. This setting controls whether or not the client will accept this instruction from the server. When set to false (default), the client does not allow the fallback to SIMPLE authentication and will abort the connection

false

hadoop.security.authentication

Defines the authentication type. Possible values: simple — no authentication, kerberos — enables the authentication by Kerberos

simple

hadoop.security.authorization

Enables RPC service-level authorization

false

hadoop.rpc.protection

authentication

hadoop.security.auth_to_local

The value is a string containing new line characters. See Kerberos documentation for more information about the format

 — 

hadoop.http.authentication.type

simple

hadoop.http.authentication.kerberos.principal

Indicates the Kerberos principal to be used for HTTP endpoint when using the kerberos authentication. The principal short name adhere to HTTP per Kerberos HTTP SPNEGO specification

HTTP/localhost@$LOCALHOST

hadoop.http.authentication.kerberos.keytab

The location of the keytab file with the credentials for the Kerberos principal used for the HTTP endpoint

/etc/security/keytabs/HTTP.service.keytab

ha.zookeeper.acl

ACLs for all znodes

 — 

hadoop.http.filter.initializers

Add to this property the org.apache.hadoop.security.AuthenticationFilterInitializer initializer class

 — 

hadoop.http.authentication.signature.secret.file

The signature secret file for signing the authentication tokens. If not set, a random secret is generated during the startup. The same secret should be used for all nodes in the cluster, JobTracker, NameNode, DataNode and TastTracker. This file should be readable only by the Unix user running the daemons

/etc/security/http_secret

hadoop.http.authentication.cookie.domain

The domain to use for the HTTP cookie that stores the authentication token. In order for authentication to work properly across all nodes in the cluster, the domain must be correctly set. There is no default value, the HTTP cookie will not have a domain working only with the hostname issuing the HTTP cookie

 — 

hadoop.ssl.require.client.cert

Defines whether client certificates are required

false

hadoop.ssl.hostname.verifier

The host name verifier to provide for HttpsURLConnections. Valid values are: DEFAULT, STRICT, STRICT_IE6, DEFAULT_AND_LOCALHOST, and ALLOW_ALL

DEFAULT

hadoop.ssl.keystores.factory.class

The KeyStoresFactory implementation to use

org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory

hadoop.ssl.server.conf

A resource file from which the SSL server keystore information will be extracted. This file is looked up in the classpath, typically it should be located in Hadoop conf/ directory

ssl-server.xml

hadoop.ssl.client.conf

A resource file from which the SSL client keystore information will be extracted. This file is looked up in the classpath, typically it should be located in Hadoop conf/ directory

ssl-client.xml

User managed hadoop.security.auth_to_local

Disable automatic generation of hadoop.security.auth_to_local

false

If there is a DataNode/network failure in the write pipeline, DFSClient will try to remove the failed DataNode from the pipeline and then continue writing with the remaining DataNodes. As a result, the number of DataNodes in the pipeline is decreased. The feature is to add new DataNodes to the pipeline. This is a site-wide property to enable/disable the feature. When the cluster size is extremely small, e.g. 3 nodes or less, cluster administrators may want to set the policy to NEVER in the default configuration file or disable this feature. Otherwise, users may experience an unusually high rate of pipeline failures since it is impossible to find new DataNodes for replacement. See also dfs.client.block.write.replace-datanode-on-failure.policy

true

DEFAULT

This property is used only if the value of dfs.client.block.write.replace-datanode-on-failure.enable is true. Best effort means, that the client will try to replace a failed DataNode in write pipeline (provided that the policy is satisfied), however, it continues the write operation in case that the DataNode replacement also fails. Suppose, the DataNode replacement fails: false — an exception should be thrown so that the write will fail; true — the write should be resumed with the remaining DataNodes. Note, that setting this property to true allows writing to a pipeline with a smaller number of DataNodes. As a result, it increases the probability of data loss

false

The minimum number of replications needed not to fail the write pipeline if new DataNodes can not be found to replace failed DataNodes (could be due to network failure) in the write pipeline. If the number of the remaining DataNodes in the write pipeline is greater than or equal to this property value, continue writing to the remaining nodes. Otherwise throw exception. If this is set to 0, an exception will be thrown, when a replacement can not be found. See also dfs.client.block.write.replace-datanode-on-failure.policy

0

The size of the thread pool for the HDFS balancer block mover — dispatchExecutor

200

The time window in milliseconds for the HDFS balancer tracking blocks and its locations

5400000

The thread pool size for executing block moves — moverThreadAllocator

1000

The maximum number of bytes that can be moved by the balancer in a single thread

10737418240

The minimum block threshold size in bytes to ignore, when fetching a source block list

10485760

The total size in bytes of DataNode blocks to get, when fetching a source block list

2147483648

The maximum amount of time for a block to move (in milliseconds). If set greater than 0, the balancer will stop waiting for a block move completion after this time. In typical clusters, a 3-5 minute timeout is reasonable. If the timeout is set for a large proportion of block moves, this needs to be increased. It could also be that too much work is dispatched and many nodes are constantly exceeding the bandwidth limit as a result. In that case, other balancer parameters might need to be adjusted. It is disabled (0) by default

0

If this specified amount of time has elapsed and no blocks have been moved out of a source DataNode, one more attempt will be made to move blocks out of this DataNode in the current Balancer iteration

60000

The maximum amount of time an iteration can be run by the Balancer. After this time the Balancer will stop the iteration, and re-evaluate the work needed to be done to balance the cluster. The default value is 20 minutes

1200000

The default block size for new files (in bytes). You can use the following suffixes to define size units (case insensitive): k (kilo), m (mega), g (giga), t (tera), p (peta), e (exa). For example, 128k, 512m, 1g, etc. You can also specify the block size in bytes (such as 134217728 for 128 MB)

134217728

Turns on short-circuit local reads

true

The maximum number of threads for DataNode balancer pending moves. This value is reconfigurable via the dfsadmin -reconfig command

50

dfs.datanode.data.dir

Determines, where on the local filesystem a DFS data node should store its blocks. If multiple directories are specified, then data will be stored in all named directories, typically on different devices. The directories should be tagged with corresponding storage types (SSD/DISK/ARCHIVE/RAM_DISK) for HDFS storage policies. The default storage type will be DISK if the directory does not have a storage type tagged explicitly. Directories, that do not exist, will be created, if the local filesystem permission allows

/srv/hadoop-hdfs/data:DISK

The maximum disk bandwidth, used by the disk balancer during reads from a source disk. The unit is MB/sec

10

dfs.disk.balancer.block.tolerance.percent

The parameter specifies when a good enough value is reached for any copy step (in percents). For example, if set to to 10 then getting close to 10% of the target value is considered as good enough. In other words, if the move operation is 20GB in size, if 18GB (20 * (1-10%)) can be moved, the entire operation is considered successful

10

During a block move from a source to destination disk, there might be various errors. This parameter defines how many errors to tolerate before declaring a move between 2 disks (or a step) has failed

5

The maximum amount of time a disk balancer plan (a set of configurations that define the data volume to be redistributed between two disks) remains valid. This setting supports multiple time unit suffixes as described in dfs.heartbeat.interval. If no suffix is specified, then milliseconds are assumed

1d

Defines a data storage threshold in percents at which disks start participating in data redistribution or balancing activities

10

The path to a UNIX domain socket that will be used for communication between the DataNode and local HDFS clients. If the string _PORT is present in this path, it will be replaced by the TCP port of the DataNode. The parameter is optional

/var/lib/hadoop-hdfs/dn_socket

dfs.hosts

Names a file that contains a list of hosts allowed to connect to the NameNode. The full pathname of the file must be specified. If the value is empty, all hosts are permitted

/etc/hadoop/conf/dfs.hosts

The minimum time interval for a block to be moved to another location again (in milliseconds)

5400000

Sets the balancer mover thread pool size

1000

The maximum number of retries before the mover considers the move as failed

10

If this specified amount of time has elapsed and no block has been moved out of a source DataNode, one more attempt will be made to move blocks out of this DataNode in the current mover iteration

60000

Determines where on the local filesystem the DFS name node should store the name table (fsimage). If multiple directories are specified, then the name table is replicated in all of the directories, for redundancy

/srv/hadoop-hdfs/name

Determines where on the local filesystem the DFS secondary name node should store the temporary images to merge. If multiple directories are specified, then the image is replicated in all of the directories for redundancy

/srv/hadoop-hdfs/checkpoint

The class that provides access for host files. org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager is used by default that loads files specified by dfs.hosts and dfs.hosts.exclude. If org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager is used, it will load the JSON file defined in dfs.hosts. To change the class name, NameNode restart is required. dfsadmin -refreshNodes only refreshes the configuration files, used by the class

org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager

The actual address, the RPC Server will bind to. If this optional address is set, it overrides only the hostname portion of dfs.namenode.rpc-address. It can also be specified per NameNode or name service for HA/Federation. This is useful for making the NameNode listen on all interfaces by setting it to 0.0.0.0

0.0.0.0

The name of the group of super-users. The value should be a single group name

hadoop

The default block replication. The actual number of replications can be specified, when the file is created. The default is used, if replication is not specified in create time

3

dfs.journalnode.http-address

The HTTP address of the JournalNode web UI

0.0.0.0:8480

dfs.journalnode.https-address

The HTTPS address of the JournalNode web UI

0.0.0.0:8481

dfs.journalnode.rpc-address

The RPC address of the JournalNode web UI

0.0.0.0:8485

dfs.datanode.http.address

The address of the DataNode HTTP server

0.0.0.0:9864

dfs.datanode.https.address

The address of the DataNode HTTPS server

0.0.0.0:9865

dfs.datanode.address

The address of the DataNode for data transfer

0.0.0.0:9866

dfs.datanode.ipc.address

The IPC address of the DataNode

0.0.0.0:9867

dfs.namenode.http-address

The address and the base port to access the dfs NameNode web UI

0.0.0.0:9870

dfs.namenode.https-address

The secure HTTPS address of the NameNode

0.0.0.0:9871

Defines whether automatic failover is enabled

true

A list of scripts or Java classes that will be used to fence the Active NameNode during a failover

shell(/bin/true)

The directory where to store journal edit files

/srv/hadoop-hdfs/journalnode

The directory on shared storage between the multiple NameNodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir. It should be left empty in a non-HA cluster

---

A unique nameservices identifier for a cluster or federation. For a single cluster, specify the name that will be used as an alias. For HDFS federation, specify, separated by commas, all namespaces associated with this cluster. This option allows you to use an alias instead of an IP address or FQDN for some commands, for example: hdfs dfs -ls hdfs://<dfs.internal.nameservices>. The value must be alphanumeric without underscores

 — 

dfs.block.access.token.enable

If set to true, access tokens are used as capabilities for accessing DataNodes. If set to false, no access tokens are checked on accessing DataNodes

false

dfs.namenode.kerberos.principal

The NameNode service principal. This is typically set to nn/_HOST@REALM.TLD. Each NameNode will substitute _HOST with its own fully qualified hostname during the startup. The _HOST placeholder allows using the same configuration setting on both NameNodes in an HA setup

nn/_HOST@REALM

dfs.namenode.keytab.file

The keytab file used by each NameNode daemon to login as its service principal. The principal name is configured with dfs.namenode.kerberos.principal

/etc/security/keytabs/nn.service.keytab

dfs.namenode.kerberos.internal.spnego.principal

HTTP Kerberos principal name for the NameNode

HTTP/_HOST@REALM

dfs.web.authentication.kerberos.principal

Kerberos principal name for the WebHDFS

HTTP/_HOST@REALM

dfs.web.authentication.kerberos.keytab

Kerberos keytab file for WebHDFS

/etc/security/keytabs/HTTP.service.keytab

dfs.journalnode.kerberos.principal

The JournalNode service principal. This is typically set to jn/_HOST@REALM.TLD. Each JournalNode will substitute _HOST with its own fully qualified hostname at startup. The _HOST placeholder allows using the same configuration setting on all JournalNodes

jn/_HOST@REALM

dfs.journalnode.keytab.file

The keytab file used by each JournalNode daemon to login as its service principal. The principal name is configured with dfs.journalnode.kerberos.principal

/etc/security/keytabs/jn.service.keytab

dfs.journalnode.kerberos.internal.spnego.principal

The server principal used by the JournalNode HTTP Server for SPNEGO authentication when Kerberos security is enabled. This is typically set to HTTP/_HOST@REALM.TLD. The SPNEGO server principal begins with the prefix HTTP/ by convention. If the value is *, the web server will attempt to login with every principal specified in the keytab file dfs.web.authentication.kerberos.keytab. For most deployments this can be set to ${dfs.web.authentication.kerberos.principal} that is use the value of dfs.web.authentication.kerberos.principal

HTTP/_HOST@REALM

dfs.datanode.data.dir.perm

Permissions for the directories on the local filesystem where the DFS DataNode stores its blocks. The permissions can either be octal or symbolic

700

dfs.datanode.kerberos.principal

The DataNode service principal. This is typically set to dn/_HOST@REALM.TLD. Each DataNode will substitute _HOST with its own fully qualified host name at startup. The _HOST placeholder allows using the same configuration setting on all DataNodes

dn/_HOST@REALM.TLD

dfs.datanode.keytab.file

The keytab file used by each DataNode daemon to login as its service principal. The principal name is configured with dfs.datanode.kerberos.principal

/etc/security/keytabs/dn.service.keytab

dfs.http.policy

Defines if HTTPS (SSL) is supported on HDFS. This configures the HTTP endpoint for HDFS daemons. The following values are supported: HTTP_ONLY — the service is provided only via http; HTTPS_ONLY — the service is provided only via https; HTTP_AND_HTTPS — the service is provided both via http and https

HTTP_ONLY

dfs.data.transfer.protection

 — 

dfs.encrypt.data.transfer

Defines whether or not actual block data that is read/written from/to HDFS should be encrypted on the wire. This only needs to be set on the NameNodes and DataNodes, clients will deduce this automatically. It is possible to override this setting per connection by specifying custom logic via dfs.trustedchannel.resolver.class

false

dfs.encrypt.data.transfer.algorithm

This value may be set to either 3des or rc4. If nothing is set, then the configured JCE default on the system is used (usually 3DES). It is widely believed that 3DES is more secure, but RC4 is substantially faster. Note that if AES is supported by both the client and server, then this encryption algorithm will only be used to initially transfer keys for AES

3des

dfs.encrypt.data.transfer.cipher.suites

This value can be either undefined or AES/CTR/NoPadding. If defined, then dfs.encrypt.data.transfer uses the specified cipher suite for data encryption. If not defined, then only the algorithm specified in dfs.encrypt.data.transfer.algorithm is used

 — 

dfs.encrypt.data.transfer.cipher.key.bitlength

The key bitlength negotiated by dfsclient and datanode for encryption. This value may be set to either 128, 192, or 256

128

ignore.secure.ports.for.testing

Allows to skip HTTPS requirements in the SASL mode

false

dfs.client.https.need-auth

Whether SSL client certificate authentication is required

false

The ACL for the admins. This configuration is used to control who can access the default servlets for HttpFS server. The value should be a comma-separated list of users and groups. The user list comes first and is separated by a space, followed by the group list, for example: user1,user2 group1,group2. Both users and groups are optional, so you can define only users, or groups, or both of them. Notice that in all these cases you should always use the leading space in the groups list. Using the asterisk grants access to all users and groups

*

The HttpFS temp directory

${hadoop.tmp.dir}/httpfs

Defines whether SSL is enabled. Default is false, that is disabled

false

The location of the Hadoop configuration directory

/etc/hadoop/conf

httpfs.hadoop.authentication.type

Defines the authentication mechanism used by httpfs for its HTTP clients. Valid values are simple and kerberos. If simple is used, clients must specify the username with the user.name query string parameter. If kerberos is used, HTTP clients must use HTTP SPNEGO or delegation tokens

simple

httpfs.hadoop.authentication.kerberos.keytab

The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by httpfs in the HTTP endpoint. httpfs.authentication.kerberos.keytab is deprecated. Instead, use hadoop.http.authentication.kerberos.keytab

/etc/security/keytabs/httpfs.service.keytab

httpfs.hadoop.authentication.kerberos.principal

The HTTP Kerberos principal used by HttpFS in the HTTP endpoint. The HTTP Kerberos principal MUST start with HTTP/ as per Kerberos HTTP SPNEGO specification. httpfs.authentication.kerberos.principal is deprecated. Instead, use hadoop.http.authentication.kerberos.principal

HTTP/${httpfs.hostname}@${kerberos.realm}

xasecure.audit.destination.solr.batch.filespool.dir

The spool directory path

/srv/ranger/hdfs_plugin/audit_solr_spool

xasecure.audit.destination.solr.urls

Leave this property value empty or set it to NONE when using ZooKeeper to connect to Solr

 — 

xasecure.audit.destination.solr.zookeepers

Specifies the ZooKeeper connection string for the Solr destination

 — 

xasecure.audit.destination.solr.force.use.inmemory.jaas.config

Uses in-memory JAAS configuration file to connect to Solr

 — 

xasecure.audit.is.enabled

Enables Ranger audit

true

xasecure.audit.jaas.Client.loginModuleControlFlag

Specifies whether the success of the module is required, requisite, sufficient, or optional

 — 

xasecure.audit.jaas.Client.loginModuleName

The name of the authenticator class

 — 

xasecure.audit.jaas.Client.option.keyTab

The name of the keytab file to get the principal’s secret key

 — 

xasecure.audit.jaas.Client.option.principal

The name of the principal to be used

 — 

xasecure.audit.jaas.Client.option.serviceName

Represents a user or a service that wants to log in

 — 

xasecure.audit.jaas.Client.option.storeKey

Set this to true if you want the keytab or the principal’s key to be stored in the subject’s private credentials

false

xasecure.audit.jaas.Client.option.useKeyTab

Set this to true if you want the module to get the principal’s key from the keytab

false

ranger.plugin.hdfs.policy.rest.url

 — 

ranger.plugin.hdfs.service.name

 — 

ranger.plugin.hdfs.policy.cache.dir

/srv/ranger/hdfs/policycache

ranger.plugin.hdfs.policy.pollIntervalMs

30000

ranger.plugin.hdfs.policy.rest.client.connection.timeoutMs

The HDFS Plugin RangerRestClient connection timeout (in milliseconds)

120000

ranger.plugin.hdfs.policy.rest.client.read.timeoutMs

The HDFS Plugin RangerRestClient read timeout (in milliseconds)

30000

ranger.plugin.hdfs.policy.rest.ssl.config.file

The path to the RangerRestClient SSL config file for the HDFS plugin

/etc/hadoop/conf/ranger-hdfs-policymgr-ssl.xml

HADOOP_CONF_DIR

Hadoop configuration directory

/etc/hadoop/conf

HADOOP_LOG_DIR

Location of the log directory

${HTTPFS_LOG}

HADOOP_PID_DIR

PID file directory location

${HTTPFS_TEMP}

HTTPFS_SSL_ENABLED

Defines if SSL is enabled for httpfs

false

HTTPFS_SSL_KEYSTORE_FILE

The path to the keystore file

admin

HTTPFS_SSL_KEYSTORE_PASS

The password to access the keystore

admin

HDFS_NAMENODE_OPTS

NameNode Heap Memory. Sets initial (-Xms) and maximum (-Xmx) Java heap memory size and environment options for the NameNode

-Xms1G -Xmx8G

HDFS_DATANODE_OPTS

DataNode Heap Memory. Sets initial (-Xms) and maximum (-Xmx) Java heap memory size and environment options for the DataNode

-Xms700m -Xmx8G

HDFS_HTTPFS_OPTS

HttpFS Heap Memory. Sets initial (-Xms) and maximum (-Xmx) Java heap memory size and environment options for the httpfs server

-Xms700m -Xmx8G

HDFS_JOURNALNODE_OPTS

JournalNode Heap Memory. Sets initial (-Xms) and maximum (-Xmx) Java heap memory size and environment options for the JournalNode

-Xms700m -Xmx8G

HDFS_ZKFC_OPTS

ZKFC Heap Memory. Sets initial (-Xms) and maximum (-Xmx) Java heap memory size and environment options for ZKFC

-Xms500m -Xmx8G

ssl.server.truststore.location

The truststore to be used by NameNodes and DataNodes

 — 

ssl.server.truststore.password

The password to the truststore

 — 

ssl.server.truststore.type

The truststore file format

jks

ssl.server.truststore.reload.interval

The truststore reload check interval (in milliseconds)

10000

ssl.server.keystore.location

The path to the keystore file used by NameNodes and DataNodes

 — 

ssl.server.keystore.password

The password to the keystore

 — 

ssl.server.keystore.keypassword

The password to the key in the keystore

 — 

ssl.server.keystore.type

The keystore file format

 — 

ssl.client.truststore.location

The truststore to be used by NameNodes and DataNodes

 — 

ssl.client.truststore.password

The password to the truststore

 — 

ssl.client.truststore.location

The truststore to be used by NameNodes and DataNodes

 — 

ssl.client.truststore.type

The truststore file format

jks

ssl.client.truststore.reload.interval

The truststore reload check interval (in milliseconds)

10000

ssl.client.keystore.location

The path to the keystore file used by NameNodes and DataNodes

 — 

ssl.client.keystore.password

The password to the keystore

 — 

ssl.client.keystore.keypassword

The password to the key in the keystore

 — 

ssl.client.keystore.type

The keystore file format

 — 

DECOMMISSIONED

When an administrator decommissions a DataNode, the DataNode will first be transitioned into DECOMMISSION_INPROGRESS state. After all blocks belonging to that DataNode are fully replicated elsewhere based on each block replication factor, the DataNode will be transitioned to DECOMMISSIONED state. After that, the administrator can shutdown the node to perform long-term repair and maintenance that could take days or weeks. After the machine has been repaired, the machine can be recommissioned back to the cluster

 — 

IN_MAINTENANCE

Sometimes administrators only need to take DataNodes down for minutes/hours to perform short-term repair/maintenance. For such scenarios, the HDFS block replication overhead, incurred by decommission, might not be necessary and a light-weight process is desirable. And that is what maintenance state is used for. When an administrator puts a DataNode in the maintenance state, the DataNode will first be transitioned to ENTERING_MAINTENANCE state. As long as all blocks belonging to that DataNode, are minimally replicated elsewhere, the DataNode will immediately be transitioned to IN_MAINTENANCE state. After the maintenance has completed, the administrator can take the DataNode out of the maintenance state. In addition, maintenance state supports the timeout that allows administrators to configure the maximum duration, in which a DataNode is allowed to stay in the maintenance state. After the timeout, the DataNode will be transitioned out of maintenance state automatically by HDFS without human intervention

 — 

Custom core-site.xml

 — 

Custom hdfs-site.xml

 — 

Custom httpfs-site.xml

 — 

Ranger plugin enabled

 — 

Custom ranger-hdfs-audit.xml

 — 

Custom ranger-hdfs-security.xml

 — 

Custom ranger-hdfs-policymgr-ssl.xml

 — 

Custom httpfs-env.sh

 — 

Custom ssl-server.xml

 — 

Custom ssl-client.xml

 — 

Topology script

The topology script used in HDFS

 — 

Topology data

An otional text file to map host names to the rack number for topology script. Stored to /etc/hadoop/conf/topology.data

 — 

Custom log4j.properties

Custom httpfs-log4j.properties

HADOOP_CLASSPATH

A colon-delimited list of directories, files, or wildcard locations that include all necessary classes

HIVE_HOME

The Hive home directory

/usr/lib/hive

METASTORE_PORT

The Hive Metastore port

9083

HiveServer2 Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for HiveServer2

-Xms256m -Xmx256m

Hive Metastore Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for Hive Metastore

-Xms256m -Xmx256m

hive.cbo.enable

When set to true, enables the cost-based optimizer that uses the Calcite framework

true

hive.compute.query.using.stats

When set to true, Hive will answer a few queries like min, max, and count(1) purely using statistics stored in the Metastore. For basic statistics collection, set the configuration property hive.stats.autogather to true. For more advanced statistics collection, run the ANALYZE TABLE queries

false

hive.execution.engine

Selects the execution engine. Supported values are: mr (Map Reduce, default), tez (Tez execution, for Hadoop 2 only), or spark (Spark execution, for Hive 1.1.0 onward)

Tez

hive.log.explain.output

When enabled, logs the EXPLAIN EXTENDED output for the query at log4j INFO level and in the HiveServer2 web UI (Drilldown → Query Plan). Starting Hive 3.1.0, this configuration property only logs as the log4j INFO. To log the EXPLAIN EXTENDED output in WebUI/Drilldown/Query Plan in Hive 3.1.0 and later, use hive.server2.webui.explain.output

true

hive.metastore.event.db.notification.api.auth

Defines whether the Metastore should perform the authorization against database notification related APIs such as get_next_notification. If set to true, then only the superusers in proxy settings have the permission

false

hive.metastore.uris

The Metastore URI used to access metadata in a remote metastore setup. For a remote metastore, you should specify the Thrift metastore server URI: thrift://<hostname>:<port> where <hostname> is a name or IP address of the Thrift metastore server, <port> is the port, on which the Thrift server is listening

 — 

hive.metastore.warehouse.dir

The absolute HDFS file path of the default database for the warehouse, that is local to the cluster

/apps/hive/warehouse

hive.server2.enable.doAs

Impersonate the connected user

false

hive.stats.fetch.column.stats

Annotation of the operator tree with statistics information requires column statistics. Column statistics are fetched from the Metastore. Fetching column statistics for each needed column can be expensive, when the number of columns is high. This flag can be used to disable fetching of column statistics from the Metastore

 — 

hive.tez.container.size

By default, Tez will spawn containers of the size of a mapper. This parameter can be used to overwrite the default value

 — 

hive.support.concurrency

Defines whether Hive should support concurrency or not. A ZooKeeper instance must be up and running for the default Hive Lock Manager to support read/write locks

false

hive.txn.manager

Set this to org.apache.hadoop.hive.ql.lockmgr.DbTxnManager as part of turning on Hive transactions. The default DummyTxnManager replicates pre-Hive-0.13 behavior and provides no transactions

 — 

javax.jdo.option.ConnectionUserName

The metastore database user name

APP

javax.jdo.option.ConnectionPassword

The password for the metastore user name

 — 

javax.jdo.option.ConnectionURL

jdbc:mysql://{{ groups['mysql.master'][0] | d(omit) }}:3306/hive

javax.jdo.option.ConnectionDriverName

The JDBC driver class name used to access Hive Metastore

com.mysql.jdbc.Driver

hive.server2.transport.mode

Sets the transport mode

tcp

hive.server2.thrift.http.port

The port number for Thrift Server2 to listen on

10001

hive.server2.thrift.http.path

The HTTP endpoint of the Thrift Server2 service

cliservice

hive.server2.authentication.kerberos.principal

Hive server Kerberos principal

hive/_HOST@EXAMPLE.COM

hive.server2.authentication.kerberos.keytab

The path to the Kerberos keytab file containing the Hive server service principal

/etc/security/keytabs/hive.service.keytab

hive.server2.authentication.spnego.principal

The SPNEGO Kerberos principal

HTTP/_HOST@EXAMPLE.COM

hive.server2.webui.spnego.principal

The SPNEGO Kerberos principal to access Web UI

 — 

hive.server2.webui.spnego.keytab

The SPNEGO Kerberos keytab file to access Web UI

 — 

hive.server2.webui.use.spnego

Defines whether to use Kerberos SPNEGO for Web UI access

false

hive.server2.authentication.spnego.keytab

The path to SPNEGO principal

/etc/security/keytabs/HTTP.service.keytab

hive.server2.authentication

Sets the authentication mode

NONE

hive.metastore.sasl.enabled

If true, the Metastore Thrift interface will be secured with SASL. Clients must authenticate with Kerberos

false

hive.metastore.kerberos.principal

The service principal for the metastore Thrift server. The _HOST token will be automatically replaced with the appropriate host name

hive/_HOST@EXAMPLE.COM

hive.metastore.kerberos.keytab.file

The path to the Kerberos keytab file containing the metastore Thrift server’s service principal

/etc/security/keytabs/hive.service.keytab

hive.server2.use.SSL

Defines whether to use SSL for HiveServer2

false

hive.server2.keystore.path

The keystore to be used by Hive

 — 

hive.server2.keystore.password

The password to the Hive keystore

 — 

hive.server2.truststore.path

The truststore to be used by Hive

 — 

hive.server2.webui.use.ssl

Defines whether to use SSL for the Hive web UI

false

hive.server2.webui.keystore.path

The path to the keystore file used to access the Hive web UI

 — 

hive.server2.webui.keystore.password

The password to the keystore file used to access the Hive web UI

 — 

hive.server2.support.dynamic.service.discovery

Defines whether to support dynamic service discovery via ZooKeeper

false

hive.zookeeper.quorum

A comma-separated list of ZooKeeper servers (<host>:<port>) running in the cluster

zookeeper:2181

hive.server2.zookeeper.namespace

Specifies the root namespace on ZooKeeper

hiveserver2

xasecure.audit.destination.solr.batch.filespool.dir

The spool directory path

/srv/ranger/hdfs_plugin/audit_solr_spool

xasecure.audit.destination.solr.urls

Leave this property value empty or set it to NONE when using ZooKeeper to connect to Solr

 — 

xasecure.audit.destination.solr.zookeepers

Specifies the ZooKeeper connection string for the Solr destination

 — 

xasecure.audit.destination.solr.force.use.inmemory.jaas.config

Uses in-memory JAAS configuration file to connect to Solr

 — 

xasecure.audit.is.enabled

Enables Ranger audit

true

xasecure.audit.jaas.Client.loginModuleControlFlag

Specifies whether the success of the module is required, requisite, sufficient, or optional

 — 

xasecure.audit.jaas.Client.loginModuleName

The name of the authenticator class

 — 

xasecure.audit.jaas.Client.option.keyTab

The name of the keytab file to get the principal’s secret key

 — 

xasecure.audit.jaas.Client.option.principal

The name of the principal to be used

 — 

xasecure.audit.jaas.Client.option.serviceName

Represents a user or a service that wants to log in

 — 

xasecure.audit.jaas.Client.option.storeKey

Set this to true if you want the keytab or the principal’s key to be stored in the subject’s private credentials

false

xasecure.audit.jaas.Client.option.useKeyTab

Set this to true if you want the module to get the principal’s key from the keytab

false

ranger.plugin.hive.policy.rest.url

 — 

ranger.plugin.hive.service.name

 — 

ranger.plugin.hive.policy.cache.dir

/srv/ranger/hive/policycache

ranger.plugin.hive.policy.pollIntervalMs

30000

ranger.plugin.hive.policy.rest.client.connection.timeoutMs

The Hive Plugin RangerRestClient connection timeout (in milliseconds)

120000

ranger.plugin.hive.policy.rest.client.read.timeoutMs

The Hive Plugin RangerRestClient read timeout (in milliseconds)

30000

xasecure.hive.update.xapolicies.on.grant.revoke

Controls Hive Ranger policy update from SQL Grant/Revoke commands

true

ranger.plugin.hive.policy.rest.ssl.config.file

The path to the RangerRestClient SSL config file for the Hive plugin

/etc/hive/conf/ranger-hive-policymgr-ssl.xml

xasecure.policymgr.clientssl.keystore

The path to the keystore file used by Ranger

 — 

xasecure.policymgr.clientssl.keystore.credential.file

The path to the keystore credentials file

/etc/hive/conf/ranger-hive.jceks

xasecure.policymgr.clientssl.truststore.credential.file

The path to the truststore credentials file

/etc/hive/conf/ranger-hive.jceks

xasecure.policymgr.clientssl.truststore

The path to the truststore file used by Ranger

 — 

xasecure.policymgr.clientssl.keystore.password

The password to the keystore file

 — 

xasecure.policymgr.clientssl.truststore.password

The password to the truststore file

 — 

tez.am.resource.memory.mb

The amount of memory in MB, that YARN will allocate to the Tez Application Master. The size increases with the size of the DAG

 — 

tez.history.logging.service.class

Enables Tez to use the Timeline Server for History Logging

org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService

tez.lib.uris

HDFS paths containing the Tez JAR files

${fs.defaultFS}/apps/tez/tez-0.9.2.tar.gz

tez.task.resource.memory.mb

The amount of memory used by launched tasks in TEZ containers. Usually this value is set in the DAG

 — 

tez.tez-ui.history-url.base

The URL where the Tez UI is hosted

 — 

tez.use.cluster.hadoop-libs

Specifies, whether Tez will use the cluster Hadoop libraries

true

ssl_certificate

The path to the SSL certificate for NGINX

/etc/ssl/certs/host_cert.cert

ssl_certificate_key

The path to the SSL certificate key for NGINX

/etc/ssl/host_cert.key

ACID Transactions

Defines whether to enable ACID transactions

false

Database type

The type of the external database used for Hive Metastore

mysql

Custom hive-site.xml

 — 

Custom hive-env.sh

 — 

Ranger plugin enabled

false

Custom ranger-hive-audit.xml

 — 

Custom ranger-hive-security.xml

 — 

Custom ranger-hive-policymgr-ssl.xml

 — 

Custom tez-site.xml

 — 

hostname

The hostname to use for the Impala daemon. If Kerberos is enabled, it is also used as a part of the Kerberos principal. If this option is not set, the system default is used

 — 

beeswax_port

The port on which Impala daemons serve Beeswax client requests

21000

fe_port

The frontend port of the Impala daemon

21000

be_port

Internal use only. Impala daemons use this port for Thrift-based communication with each other

22000

krpc_port

Internal use only. Impala daemons use this port for KRPC-based communication with each other

27000

hs2_port

The port on which Impala daemons serve HiveServer2 client requests

21050

hs2_http_port

The port is used by client applications to transmit commands and receive results over HTTP via the HiveServer2 protocol

28000

enable_webserver

Enables or disables the Impala daemon web server. Its Web UI contains information about configuration settings, running and completed queries, and associated resource usage for them. It is primarily used for diagnosing query problems that can be traced to a particular node

True

webserver_require_spnego

Enables the Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Use this option only if Kerberos is enabled for the HDFS service

False

webserver_port

The port where the Impala daemon web server is running

25000

catalog_service_host

The host where the Impala Catalog Service component is running

 — 

catalog_service_port

The port on which the Impala Catalog Service component listens

26000

state_store_host

The host where the Impala Statestore component is running

 — 

state_store_port

The port on which the Impala Statestore component is running

24000

state_store_subscriber_port

The port where StateStoreSubscriberService is running. StateStoreSubscriberService listens on this port for updates from the Statestore daemon

23030

scratch_dirs

The directory where Impala Daemons writes data to free up memory during large sort, join, aggregation, and other operations. The files are removed when the operation finishes. This can potentially be large amounts of data

/srv/impala/

log_dir

The directory where an Impala daemon places its log files

/var/log/impala/impalad/

log_filename

The Prefix of the log filename — the full path is <log_dir>/<log_filename>

impalad

max_log_files

The number of log files that are kept for each severity level (INFO, WARNING, ERROR, and FATAL) before older log files are removed. The number should be greater than 1 to keep at least the current log file to remain open. If set to 0, all log files are retained and log rotation is disabled

10

audit_event_log_dir

The directory in which Impala daemon audit event log files are written if the Impala Audit Event Generation property is enabled

/var/log/impala/impalad/audit

minidump_path

The directory for storing Impala daemon Breakpad dumps

/var/log/impala-minidumps

lineage_event_log_dir

The directory in which the Impala daemon generates its lineage log files if the Impala Lineage Generation property is enabled

/var/log/impala/impalad/lineage

local_library_dir

The local directory into which an Impala daemon copies user-defined function (UDF) libraries from HDFS

/usr/lib/impala/udfs

max_lineage_log_file_size

The maximum size (in entries) of the Impala daemon lineage log file. When the size is exceeded, a new file is created

5000

max_audit_event_log_file_size

The maximum size (in queries) of the Impala Daemon audit event log file. When the size is exceeded, a new file is created

5000

fe_service_threads

The maximum number of concurrent client connections allowed. The parameter determines how many queries can run simultaneously. When more clients try to connect to Impala, the later arriving clients have to wait until previous clients disconnect. Setting the fe_service_threads value too high could negatively impact query latency

64

mem_limit

The memory limit (in bytes) for an Impala daemon enforced by the daemon itself. This limit does not include memory consumed by the daemon’s embedded JVM. The Impala daemon uses up this amount of memory for query processing, cached data, network buffers, background operations, etc. If the limit is exceeded, queries will be killed until the used memory becomes under the limit

1473249280

idle_query_timeout

The time in seconds after which an idle query (no processing work is done and no updates are received from the client) is cancelled. If set to 0, idle queries are never expired

0

idle_session_timeout

The time in seconds after which Impala closes an idle session and cancels all running queries. If set to 0, idle sessions never expire

0

max_result_cache_size

The maximum number of query results a client can request to be cached on a per-query basis to support restarting fetches. This option guards against unreasonably large result caches. Requests exceeding this maximum are rejected

100000

max_cached_file_handles

The maximum number of cached HDFS file handles. Caching HDFS file handles reduces the number of new file handles opened and thus reduces the load on a HDFS NameNode. Each cached file handle consumes a small amount of memory. If set to 0, the file handle caching is disabled

20000

unused_file_handle_timeout_sec

The maximum time in seconds during which an unused HDFS file handle remains in the HDFS file handle cache. When the underlying file for a cached file handle is deleted, the disk space may not be freed until the cached file handle is removed from the cache. This timeout allows the disk space occupied by deleted files to be freed in a predictable period of time. If set to 0, unused cached HDFS file handles are not removed

21600

statestore_subscriber_timeout_seconds

The timeout in seconds for Impala Daemon and Catalog Server connections to Statestore

30

default_query_options

A list of key/value pairs representing additional query options to pass to the Impala Daemon command line, separated by commas

default_file_format=parquet,default_transactional_type=none

load_auth_to_local_rules

If checked (True) and Kerberos is enabled for Impala, Impala uses the auth_to_local option from hadoop.security.auth_to_local rules of the HDFS configuration

True

catalog_topic_mode

The granularity of on-demand metadata fetches between the Impala Daemon coordinator and Impala Catalog Service. See Metadata management

minimal

use_local_catalog

Allows coordinators to cache metadata from Impala Catalog Service. If this is set to True, coordinators pull metadata as needed from catalogd and cache it locally. The cached metadata is automatically removed under memory pressure or after an expiration time. See Metadata management

True

abort_on_failed_audit_event

Specifies whether shutdown Impala if there is a problem with recording an audit event

False

max_minidumps

The maximum number of Breakpad dump files stored by the Impala daemon. A negative value or 0 is interpreted as an unlimited number

9

authorized_proxy_user_config

Specifies the set of authorized proxy users (the users who can impersonate other users during authorization), and users who they are allowed to impersonate. The example of syntax for the option is: authenticated_user1=delegated_user1,delegated_user2;authenticated_user2=*. See Configuring Impala delegation for clients. The list can contain short usernames or * to indicate all users

knox=*;zeppelin=*

queue_wait_timeout_ms

The maximum amount of time (in milliseconds) that a request waits to be admitted before timing out. Must be a positive integer

60000

disk_spill_encryption

Specifies whether to encrypt and verify the integrity of all data spilled to the disk as part of a query

False

abort_on_config_error

Specifies whether to abort Impala startup if there are incorrect configs or Impala is running on unsupported hardware

True

kerberos_reinit_interval

The number of minutes between reestablishing the ticket with the Kerberos server

60

principal

The service Kerberos principal

 — 

keytab_file

The service Kerberos keytab file

 — 

ssl_server_certificate

The path to the TLS/SSL file with the server certificate key used for TLS/SSL. It is used when Impala operates as a TLS/SSL server. The certificate file must be in the PEM format

 — 

ssl_private_key

The path to the TLS/SSL file with the private key used for TLS/SSL. It is used when Impala operates as a TLS/SSL server. The file must be in the PEM format

 — 

ssl_client_ca_certificate

The path to the certificate, in the PEM format, used to confirm the authenticity of SSL/TLS servers that the Impala daemons can connect to. Since the Impala daemons connect to each other, it should also include the CA certificate used to sign all the SSL/TLS certificates. SSL/TLS between Impala daemons cannot be enabled without this parameter

 — 

webserver_certificate_file

The path to the TLS/SSL file with the server certificate key used for TLS/SSL. It is used when the Impala daemon web server operates as a TLS/SSL server. The certificate file must be in the PEM format

 — 

webserver_private_key_file

The path to the TLS/SSL file with the private key used for TLS/SSL. It is used when the Impala daemon web server operates as a TLS/SSL server. The certificate file must be in the PEM format

 — 

ssl_minimum_version

The minimum version of TLS

TLSv1.2

log4j.properties

Apache Log4j utility settings

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the table below

DefaultLimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

DefaultLimitFSIZE

The maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

DefaultLimitDATA

The maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

DefaultLimitSTACK

The maximum stack size allocated to a process, in kilobytes

stack size ( -s)

DefaultLimitCORE

The maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

DefaultLimitRSS

The maximum of resident set size, in kilobytes

max memory size ( -m)

DefaultLimitNOFILE

The maximum number of open file descriptors allowed for the process

open files ( -n)

DefaultLimitAS

The maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

DefaultLimitNPROC

The maximum number of processes

max user processes ( -u)

DefaultLimitMEMLOCK

The maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

DefaultLimitLOCKS

The maximum number of files locked by a process

file locks ( -x)

DefaultLimitSIGPENDING

The maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

DefaultLimitMSGQUEUE

The maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

DefaultLimitNICE

The maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

DefaultLimitRTPRIO

The maximum real-time scheduling priority level

real-time priority ( -r)

DefaultLimitRTTIME

The maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)

hostname

The hostname to use for the Statestore daemon. If Kerberos is enabled, it is also used as a part of the Kerberos principal. If this option is not set, the system default is used

 — 

state_store_host

The host where the Impala Statestore component is running

 — 

state_store_port

The port on which the Impala Statestore component is running

24000

catalog_service_host

The host where the Impala Catalog Service component is running

 — 

catalog_service_port

The port on which the Impala Catalog Service component listens

26000

enable_webserver

Enables or disables the Statestore daemon web server. Its Web UI contains information about memory usage, configuration settings, and ongoing health checks performed by Statestore

True

webserver_require_spnego

Enables the Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Use this option only if Kerberos is enabled for the HDFS service

False

webserver_port

The port on which the Statestore web server is running

25010

log_dir

The directory where the Statestore daemon places its log files

/var/log/impala/statestored/

log_filename

The Prefix of the log filename — the full path is <log_dir>/<log_filename>

statestored

max_log_files

The number of log files that are kept for each severity level (INFO, WARNING, ERROR, and FATAL) before older log files are removed. The number should be greater than 1 to keep at least the current log file to remain open. If set to 0, all log files are retained and log rotation is disabled

10

minidump_path

The directory for storing Statestore daemon Breakpad dumps

/var/log/impala-minidumps

max_minidumps

The maximum number of Breakpad dump files stored by Statestore daemon. A negative value or 0 is interpreted as an unlimited number

9

state_store_num_server_worker_threads

The number of worker threads for the thread manager of the Statestore Thrift server

4

state_store_pending_task_count_max

The maximum number of tasks allowed to be pending by the thread manager of the Statestore Thrift server. The 0 value allows an infinite number of pending tasks

0

kerberos_reinit_interval

The number of minutes between reestablishing the ticket with the Kerberos server

60

principal

The service Kerberos principal

 — 

keytab_file

The service Kerberos keytab file

 — 

ssl_server_certificate

The path to the TLS/SSL file with the server certificate key used for TLS/SSL. It is used when Impala operates as a TLS/SSL server. The certificate file must be in the PEM format

 — 

ssl_private_key

The path to the TLS/SSL file with the private key used for TLS/SSL. It is used when Impala operates as a TLS/SSL server. The file must be in the PEM format

 — 

ssl_client_ca_certificate

The path to the certificate, in the PEM format, used to confirm the authenticity of SSL/TLS servers that the Impala daemons can connect to. Since the Impala daemons connect to each other, it should also include the CA certificate used to sign all the SSL/TLS certificates. SSL/TLS between Impala daemons cannot be enabled without this parameter

 — 

webserver_certificate_file

The path to the TLS/SSL file with the server certificate key used for TLS/SSL. It is used when the Statestore web server operates as a TLS/SSL server. The certificate file must be in the PEM format

 — 

webserver_private_key_file

The path to the TLS/SSL file with the private key used for TLS/SSL. It is used when the Statestore web server operates as a TLS/SSL server. The certificate file must be in the PEM format

 — 

ssl_minimum_version

The minimum version of TLS

TLSv1.2

Custom statestore.conf

 — 

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the table below

DefaultLimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

DefaultLimitFSIZE

The maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

DefaultLimitDATA

The maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

DefaultLimitSTACK

The maximum stack size allocated to a process, in kilobytes

stack size ( -s)

DefaultLimitCORE

The maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

DefaultLimitRSS

The maximum of resident set size, in kilobytes

max memory size ( -m)

DefaultLimitNOFILE

The maximum number of open file descriptors allowed for the process

open files ( -n)

DefaultLimitAS

The maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

DefaultLimitNPROC

The maximum number of processes

max user processes ( -u)

DefaultLimitMEMLOCK

The maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

DefaultLimitLOCKS

The maximum number of files locked by a process

file locks ( -x)

DefaultLimitSIGPENDING

The maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

DefaultLimitMSGQUEUE

The maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

DefaultLimitNICE

The maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

DefaultLimitRTPRIO

The maximum real-time scheduling priority level

real-time priority ( -r)

DefaultLimitRTTIME

The maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)

hostname

The hostname to use for the Catalog Service daemon. If Kerberos is enabled, it is also used as a part of the Kerberos principal. If this option is not set, the system default is used

 — 

state_store_host

The host where the Impala Statestore component is running

 — 

state_store_port

The port on which the Impala Statestore component is running

24000

catalog_service_host

The host where the Impala Catalog Service component is running

 — 

catalog_service_port

The port on which the Impala Catalog Service component listens

26000

enable_webserver

Enables or disables the Catalog Service web server. Its Web UI includes information about the databases, tables, and other objects managed by Impala, in addition to the resource usage and configuration settings of the Catalog Service

True

webserver_require_spnego

Enables the Kerberos authentication for Hadoop HTTP web consoles for all roles of this service using the SPNEGO protocol. Use this option only if Kerberos is enabled for the HDFS service

False

webserver_port

The port on which the Catalog Service web server is running

25020

log_dir

The directory where the Catalog Service daemon places its log files

/var/log/impala/catalogd/

log_filename

The Prefix of the log filename — the full path is <log_dir>/<log_filename>

catalogd

max_log_files

The number of log files that are kept for each severity level (INFO, WARNING, ERROR, and FATAL) before older log files are removed. The number should be greater than 1 to keep at least the current log file to remain open. If set to 0, all log files are retained and log rotation is disabled

10

minidump_path

The directory for storing the Catalog Service daemon Breakpad dumps

/var/log/impala-minidumps

max_minidumps

The maximum number of Breakpad dump files stored by Catalog Service. A negative value or 0 is interpreted as an unlimited number

9

hms_event_polling_interval_s

When this parameter is set to a positive integer, Catalog Service fetches new notifications from Hive Metastore at the specified interval in seconds. If hms_event_polling_interval_s is set to 0, the automatic metadata invalidation and updates are disabled. See Metadata management

2

load_auth_to_local_rules

If checked (True) and Kerberos is enabled for Impala, Impala uses the auth_to_local option from hadoop.security.auth_to_local rules of the HDFS configuration

True

load_catalog_in_background

If it is set to True, the metadata is loaded in the background, even if that metadata is not required for any query. If False, the metadata is loaded when it is referenced for the first time

False

catalog_topic_mode

The granularity of on-demand metadata fetches between the Impala Daemon coordinator and Impala Catalog Service. See Metadata management

minimal

statestore_subscriber_timeout_seconds

The timeout in seconds for Impala Daemon and Catalog Server connections to Statestore

30

state_store_subscriber_port

The port where StateStoreSubscriberService is running. StateStoreSubscriberService listens on this port for updates from the Statestore daemon

23020

kerberos_reinit_interval

The number of minutes between reestablishing the ticket with the Kerberos server

60

principal

The service Kerberos principal

 — 

keytab_file

The service Kerberos keytab file

 — 

ssl_server_certificate

The path to the TLS/SSL file with the server certificate key used for TLS/SSL. It is used when Impala operates as a TLS/SSL server. The certificate file must be in the PEM format

 — 

ssl_private_key

The path to the TLS/SSL file with the private key used for TLS/SSL. It is used when Impala operates as a TLS/SSL server. The file must be in the PEM format

 — 

ssl_client_ca_certificate

The path to the certificate, in the PEM format, used to confirm the authenticity of SSL/TLS servers that the Impala daemons can connect to. Since the Impala daemons connect to each other, it should also include the CA certificate used to sign all the SSL/TLS certificates. SSL/TLS between Impala daemons cannot be enabled without this parameter

 — 

webserver_certificate_file

The path to the TLS/SSL file with the server certificate key used for TLS/SSL. It is used when the Catalog Service web server operates as a TLS/SSL server. The certificate file must be in the PEM format

 — 

webserver_private_key_file

The path to the TLS/SSL file with the private key used for TLS/SSL. It is used when the Catalog Service web server operates as a TLS/SSL server. The certificate file must be in the PEM format

 — 

ssl_minimum_version

The minimum version of TLS

TLSv1.2

Custom catalogstore.conf

 — 

Enable custom ulimits

Switch on the corresponding toggle button to specify resource limits (ulimits) for the current process. If you do not set these values, the default system settings are used. Ulimit settings are described in the table below

DefaultLimitCPU

A limit in seconds on the amount of CPU time that a process can consume

cpu time ( -t)

DefaultLimitFSIZE

The maximum size of files that a process can create, in 512-byte blocks

file size ( -f)

DefaultLimitDATA

The maximum size of a process’s data segment, in kilobytes

data seg size ( -d)

DefaultLimitSTACK

The maximum stack size allocated to a process, in kilobytes

stack size ( -s)

DefaultLimitCORE

The maximum size of a core dump file allowed for a process, in 512-byte blocks

core file size ( -c)

DefaultLimitRSS

The maximum of resident set size, in kilobytes

max memory size ( -m)

DefaultLimitNOFILE

The maximum number of open file descriptors allowed for the process

open files ( -n)

DefaultLimitAS

The maximum size of the process virtual memory (address space), in kilobytes

virtual memory ( -v)

DefaultLimitNPROC

The maximum number of processes

max user processes ( -u)

DefaultLimitMEMLOCK

The maximum memory size that can be locked for the process, in kilobytes. Memory locking ensures the memory is always in RAM and a swap file is not used

max locked memory ( -l)

DefaultLimitLOCKS

The maximum number of files locked by a process

file locks ( -x)

DefaultLimitSIGPENDING

The maximum number of signals that are pending for delivery to the calling thread

pending signals ( -i)

DefaultLimitMSGQUEUE

The maximum number of bytes in POSIX message queues. POSIX message queues allow processes to exchange data in the form of messages

POSIX message queues ( -q)

DefaultLimitNICE

The maximum NICE priority level that can be assigned to a process

scheduling priority ( -e)

DefaultLimitRTPRIO

The maximum real-time scheduling priority level

real-time priority ( -r)

DefaultLimitRTTIME

The maximum pipe buffer size, in 512-byte blocks

pipe size ( -p)

kyuubi.frontend.rest.bind.port

Port on which the REST frontend service runs

10099

kyuubi.frontend.thrift.binary.bind.port

Port on which the Thrift frontend service runs via a binary protocol

10099

kyuubi.frontend.thrift.http.bind.port

Port on which the Thrift frontend service runs via HTTP

10010

kyuubi.frontend.thrift.http.path

The path component of the URL endpoint for the HTTP version of Thrift

cliservice

kyuubi.engine.share.level

An engine share level. Possible values: CONNECTION (one engine per connection), USER (one engine per user), GROUP (one engine per group), SERVER (one engine per server)

USER

kyuubi.engine.type

An engine type supported by Kyuubi. Possible values: SPARK_SQL, FLINK_SQL, TRINO, HIVE_SQL, JDBC

SPARK_SQL

kyuubi.operation.language

Programming language used to interpret inputs. Possible values: SQL, SCALA, PYTHON

SQL

kyuubi.frontend.protocols

A comma-separated list for supported frontend protocols. Possible values: THRIFT_BINARY, THRIFT_HTTP, REST

THRIFT_BINARY

kyuubi.frontend.thrift.binary.ssl.disallowed.protocols

Forbidden SSL versions for Thrift binary frontend

SSLv2,SSLv3,TLSv1.1

kyuubi.frontend.thrift.http.ssl.protocol.blacklist

Forbidden SSL versions for Thrift HTTP frontend

SSLv2,SSLv3,TLSv1.1

kyuubi.ha.addresses

External Kyuubi instance addresses

<hostname_1>:2181, …​, <hostname_N>:2181

kyuubi.ha.namespace

The root directory for the service to deploy its instance URI

kyuubi

kyuubi.metadata.store.jdbc.database.type

A database type for the server metadata store. Possible values: SQLITE, MYSQL, POSTGRESQL

POSTGRESQL

kyuubi.metadata.store.jdbc.url

A JDBC URL for the server metadata store

jdbc:postgresql://{{ groups['adpg.adpg'][0] | d(omit) }}:5432/kyuubi

kyuubi.metadata.store.jdbc.driver

A JDBC driver classname for the server metadata store

org.postgresql.Driver

kyuubi.metadata.store.jdbc.user

A username for the server metadata store

kyuubi

kyuubi.metadata.store.jdbc.password

A password for the server metadata store

 — 

kyuubi.frontend.thrift.binary.ssl.enabled

Indicates whether to use the SSL encryption in the Thrift binary mode

false

kyuubi.frontend.thrift.http.use.SSL

Indicates whether to use the SSL encryption in the Thrift HTTP mode

false

kyuubi.frontend.ssl.keystore.type

Type of the SSL certificate keystore

 — 

kyuubi.frontend.ssl.keystore.path

Path to the SSL certificate keystore

 — 

kyuubi.frontend.ssl.keystore.password

Password for the SSL certificate keystore

 — 

kyuubi.frontend.thrift.http.ssl.keystore.path

Path to the SSL certificate keystore

 — 

kyuubi.frontend.thrift.http.ssl.keystore.password

Password for the SSL certificate keystore

 — 

kyuubi.authentication

Authentication type. Possible values: NONE, KERBEROS

NONE

kyuubi.ha.zookeeper.acl.enabled

Indicates whether the ZooKeeper ensemble is kerberized

false

kyuubi.ha.zookeeper.auth.type

ZooKeeper authentication type. Possible values: NONE, KERBEROS

NONE

kyuubi.ha.zookeeper.auth.principal

Kerberos principal name used for ZooKeeper authentication

 — 

kyuubi.ha.zookeeper.auth.keytab

Path to Kyuubi Server’s keytab used for ZooKeeper authentication

 — 

kyuubi.kinit.principal

Name of the Kerberos principal

 — 

kyuubi.kinit.keytab

Path to Kyuubi Server’s keytab

 — 

kyuubi.spnego.principal

Name of the SPNego service principal. Set only if using SPNego in authentication

 — 

kyuubi.spnego.keytab

Path to the SPNego service keytab. Set only if using SPNego in authentication

 — 

kyuubi.engine.hive.java.options

Extra Java options for the Hive query engine

 — 

KYUUBI_HOME

Kyuubi home directory

/usr/lib/kyuubi

KYUUBI_CONF_DIR

Directory that stores Kyuubi configurations

/etc/kyuubi/conf

KYUUBI_LOG_DIR

Kyuubi server log directory

/var/log/kyuubi

KYUUBI_PID_DIR

Directory that stores the Kyuubi instance .pid-file

/var/run/kyuubi

KYUUBI_ADDITIONAL_CLASSPATH

Path to a directory with additional SSM libraries

/usr/lib/ssm/lib/smart*

HADOOP_HOME

Hadoop home directory

/usr/lib/hadoop

HADOOP_LIB_DIR

Directory that stores Hadoop libraries

${HADOOP_HOME}/lib

KYUUBI_JAVA_OPTS

Java parameters for Kyuubi

-Djava.library.path=${HADOOP_LIB_DIR}/native/ -Djava.io.tmpdir={{ cluster.config.java_tmpdir | d('/tmp') }}

HADOOP_CLASSPATH

A common $HADOOP_CLASSPATH variable value followed by KYUUBI_ADDITIONAL_CLASSPATH

$HADOOP_CLASSPATH:/usr/lib/ssm/lib/smart*

HADOOP_CONF_DIR

Directory that stores Hadoop configurations

/etc/hadoop/conf

SPARK_HOME

Spark home directory

/usr/lib/spark3

SPARK_CONF_DIR

Directory that stores Spark configurations

/etc/spark3/conf

FLINK_HOME

Flink home directory

/usr/lib/flink

FLINK_CONF_DIR

Directory that stores Flink configurations

/etc/flink/conf

FLINK_HADOOP_CLASSPATH

Additional Hadoop .jar files required to use the Kyuubi Flink engine

$(hadoop classpath):/usr/lib/ssm/lib/smart*

HIVE_HOME

Hive home directory

/usr/lib/hive

HIVE_CONF_DIR

Directory that stores Hive configurations

/etc/hive/conf

HIVE_HADOOP_CLASSPATH

Additional Hadoop .jar files required to use the Kyuubi Hive engine

Password

The root password

 — 

SOLR_HOME

The location for index data and configs

/srv/solr/server

SOLR_AUTH_TYPE

Specifies the authentication type for Solr

 — 

SOLR_AUTHENTICATION_OPTS

Solr authentication options

 — 

GC_TUNE

JVM parameters for Solr

-XX:-UseLargePages

SOLR_SSL_KEY_STORE:

The path to the Solr keystore file (.jks)

 — 

SOLR_SSL_KEY_STORE_PASSWORD

The password to the Solr keystore file

 — 

SOLR_SSL_TRUST_STORE

The path to the Solr truststore file (.jks)

 — 

SOLR_SSL_TRUST_STORE_PASSWORD

The password to the Solr truststore file

 — 

SOLR_SSL_NEED_CLIENT_AUTH

Defines if client authentication is enabled

false

SOLR_SSL_WANT_CLIENT_AUTH

Enables clients to authenticate (but not requires)

false

SOLR_SSL_CLIENT_HOSTNAME_VERIFICATION

Defines whether to enable hostname verification

false

SOLR_HOST

Specifies the host name of the Solr server

 — 

ZK_HOST

Comma-separated locations of all servers in the ensemble and the ports on which they communicate. You can put ZooKeeper chroot at the end of your ZK_HOST connection string. For example, host1.mydomain.com:2181,host2.mydomain.com:2181,host3.mydomain.com:2181/solr

 — 

Solr Server Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for Solr Server

-Xms512m -Xmx512m

xasecure.audit.solr.solr_url

A path to a Solr collection to store audit logs

 — 

xasecure.audit.solr.async.max.queue.size

The maximum size of internal queue used for storing audit logs

1

xasecure.audit.solr.async.max.flush.interval.ms

The maximum time interval between flushes to disk (in milliseconds)

100

ranger.plugin.solr.policy.rest.url

 — 

ranger.plugin.solr.service.name

 — 

ranger.plugin.solr.policy.cache.dir

/srv/ranger/yarn/policycache

ranger.plugin.solr.policy.pollIntervalMs

30000

ranger.plugin.solr.policy.rest.client.connection.timeoutMs

The Solr Plugin RangerRestClient connection timeout (in milliseconds)

120000

ranger.plugin.solr.policy.rest.client.read.timeoutMs

The Solr Plugin RangerRestClient read timeout (in milliseconds)

30000

xasecure.policymgr.clientssl.keystore

The path to the keystore file used by Ranger

 — 

xasecure.policymgr.clientssl.keystore.credential.file

The path to the keystore credentials file

/etc/solr/conf/ranger-solr.jceks

xasecure.policymgr.clientssl.truststore.credential.file

The path to the truststore credentials file

/etc/solr/conf/ranger-solr.jceks

xasecure.policymgr.clientssl.truststore

The path to the truststore file used by Ranger

 — 

xasecure.policymgr.clientssl.keystore.password

The password to the keystore file

 — 

xasecure.policymgr.clientssl.truststore.password

The password to the truststore file

 — 

solr.xml

The content of solr.xml

Custom solr-env.sh

 — 

Ranger plugin enabled

Enables the Ranger plugin

false

Dynamic allocation (spark.dynamicAllocation.enabled)

Defines whether to use dynamic resource allocation that scales the number of executors, registered with this application, up and down, based on the workload

false

spark.yarn.archive

The archive containing needed Spark JARs for distribution to the YARN cache. If set, this configuration replaces spark.yarn.jars and the archive is used in all the application containers. The archive should contain JAR files in its root directory. The archive can also be hosted on HDFS to speed up file distribution

hdfs:///apps/spark/spark-yarn-archive.tgz

spark.master

The cluster manager to connect to

yarn

spark.yarn.historyServer.address

Spark History server address

 — 

spark.dynamicAllocation.enabled

Defines whether to use dynamic resource allocation that scales the number of executors, registered with this application, up and down, based on the workload

false

spark.shuffle.service.enabled

Enables the external shuffle service. This service preserves the shuffle files written by executors so that executors can be safely removed, or so that shuffle fetches can continue in the event of executor failure. The external shuffle service must be set up in order to enable it

false

spark.eventLog.enabled

Defines whether to log Spark events, useful for reconstructing the Web UI after the application has finished

true

spark.eventLog.dir

The base directory where Spark events are logged, if spark.eventLog.enabled=true. Within this base directory, Spark creates a sub-directory for each application, and logs the events specific to the application in this directory. You may want to set this to a unified location like an HDFS directory so history files can be read by the History Server

hdfs:///var/log/spark/apps

spark.serializer

The class to use for serializing objects that will be sent over the network or need to be cached in serialized form. The default of Java serialization works with any Serializable Java object but is quite slow, so we recommend using org.apache.spark.serializer.KryoSerializer and configuring Kryo serialization when speed is necessary. Can be any subclass of org.apache.spark.Serializer

org.apache.spark.serializer.KryoSerializer

spark.dynamicAllocation.executorIdleTimeout

If dynamic allocation is enabled and an executor has been idle for more than this duration, the executor will be removed. For more details, see Spark documentation

120s

spark.dynamicAllocation.cachedExecutorIdleTimeout

If dynamic allocation is enabled and an executor which has cached data blocks has been idle for more than this duration, the executor will be removed. For more details, see Spark documentation

600s

spark.history.provider

The name of the class that implements the application history backend. Currently there is only one implementation provided with Spark that looks for application logs stored in the file system

org.apache.spark.deploy.history.FsHistoryProvider

spark.history.fs.cleaner.enabled

Specifies whether the History Server should periodically clean up event logs from storage

true

spark.history.store.path

A local directory where to cache application history data. If set, the History Server will store application data on disk instead of keeping it in memory. The data written to disk will be re-used in case of the History Server restart

/var/log/spark/history

spark.driver.extraClassPath

Extra classpath entries to prepend to the classpath of the driver

/usr/lib/hive/lib/hive-shims-scheduler.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar

spark.history.ui.port

The port number of the History Server web UI

18082

spark.history.fs.logDirectory

The log directory of the History Server

hdfs:///var/log/spark/apps

spark.sql.hive.metastore.jars

The location of the JARs that should be used to instantiate HiveMetastoreClient

/usr/lib/hive/lib/*

spark.sql.hive.metastore.version

The Hive Metastore version

3.0.0

spark.driver.extraLibraryPath:

The path to extra native libraries for driver

/usr/lib/hadoop/lib/native/

spark.yarn.am.extraLibraryPath:

The path to extra native libraries for Application Master

/usr/lib/hadoop/lib/native/

spark.executor.extraLibraryPath

The path to extra native libraries for Executor

/usr/lib/hadoop/lib/native/

spark.yarn.appMasterEnv.HIVE_CONF_DIR

A directory on the Application Master with Hive configs required for running Hive in the cluster mode

/etc/spark/conf

spark.yarn.historyServer.allowTracking

Allows to use Spark History Server for tracking UI even if web UI is disabled for a job

True

spark.ssl.enabled

Defines whether to use SSL for Spark

false

spark.ssl.protocol

TLS protocol to be used. The protocol must be supported by JVM

TLSv1.2

spark.ssl.ui.port

The port where the SSL service will listen on

4040

spark.ssl.historyServer.port

The port to access History Server web UI

18082

spark.ssl.keyPassword

The password to the private key in the key store

 — 

spark.ssl.keyStore

The path to the keystore file

 — 

spark.ssl.keyStoreType

The type of the keystore

JKS

spark.ssl.trustStorePassword

The password to the truststore used by Spark

 — 

spark.ssl.trustStore

The path to the truststore file

 — 

spark.ssl.trustStoreType

The type of the truststore

JKS

spark.history.kerberos.enabled

Indicates whether the History Server should use Kerberos to login. This is required if the History Server is accessing HDFS files on a secure Hadoop cluster

false

spark.acls.enable

Enables Spark ACL

false

spark.modify.acls

Defines who has access to modify a running Spark application

spark,hdfs

spark.modify.acls.groups

A comma-separated list of user groups that have modify access to the Spark application

spark,hdfs

spark.history.ui.acls.enable

Specifies whether ACLs should be checked to authorize users viewing the applications in the History Server. If enabled, access control checks are performed regardless of what the individual applications had set for spark.ui.acls.enable. If disabled, no access control checks are made for any application UIs available through the History Server

false

spark.history.ui.admin.acls

A comma-separated list of users that have view access to all the Spark applications in History Server

spark,hdfs,dr.who

spark.history.ui.admin.acls.groups

A comma-separated list of groups that have view access to all the Spark applications in History Server

spark,hdfs,dr.who

spark.ui.view.acls

A comma-separated list of users that have view access to the Spark application. By default, only the user that started the Spark job has view access. Using * as a value means that any user can have view access to this Spark job

spark,hdfs,dr.who

spark.ui.view.acls.groups

A comma-separated list of groups that have view access to the Spark web UI to view the Spark Job details. This can be used if you have a set of administrators or developers or users who can monitor the Spark job submitted. Using * in the list means any user in any group can view the Spark job details on the Spark web UI. The user groups are obtained from the instance of the groups mapping provider specified by spark.user.groups.mapping

spark,hdfs,dr.who

Spark History Server Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for Spark History Server

1G

Spark Thrift Server Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for Spark Thrift Server

1G

Livy Server Heap Memory

Sets initial (-Xms) and maximum (-Xmx) Java heap size for Livy Server

-Xms300m -Xmx4G

livy.server.host

The host address to start the Livy server. By default, Livy will bind to all network interfaces

0.0.0.0

livy.server.port

The port to run the Livy server

8998

livy.spark.master

The Spark master to use for Livy sessions

yarn-cluster

livy.impersonation.enabled

Defines if Livy should impersonate users when creating a new session

false

livy.server.csrf-protection.enabled

Defines whether to enable the CSRF protection. If enabled, clients should add the X-Requested-By HTTP header for POST/DELETE/PUT/PATCH HTTP methods

true

livy.repl.enable-hive-context

Defines whether to enable HiveContext in the Livy interpreter. If set to true, hive-site.xml and the Livy server classpath will be detected on user request automatically

true

livy.server.recovery.mode

Sets the recovery mode for Livy

recovery

livy.server.recovery.state-store

Defines where Livy should store the state for recovery

filesystem

livy.server.recovery.state-store.url

For the filesystem state store, the path of the state store directory. Do not use a filesystem that does not support atomic rename like S3. For example: file:///tmp/livy or hdfs:///. For ZooKeeper, specify the address to the ZooKeeper servers. For example: host1:port1,host2:port2

/livy-recovery

livy.server.auth.type

Sets the Livy authentication type

 — 

livy.server.access_control.enabled

Defines whether to enable the access control for a Livy server. If set to true, then all the incoming requests will be checked if the requested user has permission

false

livy.server.access_control.users

Users allowed to access Livy. By default, any user is allowed to access Livy. If a user wants to limit the access, the user should list all the permitted users separated by a comma

livy,hdfs,spark

livy.superusers

A list of comma-separated users that have the permissions to change other user’s submitted session, like submitting statements, deleting session, and so on

livy,hdfs,spark

livy.keystore

A path to the keystore file. The path can be absolute or relative to the directory in which the process is started

 — 

livy.keystore.password

The password to access the keystore

 — 

livy.key-password

The password to access the key in the keystore

 — 

livy.server.thrift.ssl.protocol.blacklist

The list of banned TLS protocols

SSLv2,SSLv3,TLSv1,TLSv1.1

Custom spark-defaults.conf

 — 

spark-env.sh

Enter the contents for the spark-env.sh file that is used to initialize environment variables on worker nodes

spark-env.sh

Custom livy.conf

 — 

livy-env.sh

Enter the contents for the livy-env.sh file that is used to prepare the environment for Livy startup

livy-env.sh

thriftserver-env.sh

Enter the contents for the thriftserver-env.sh file that is used to prepare the environment for Thrift server startup

thriftserver-env.sh

spark-history-env.sh

Enter the contents for the spark-history-env.sh file that is used to prepare the environment for History Server startup

spark-history-env.sh

Dynamic allocation (spark.dynamicAllocation.enabled)

Defines whether to use dynamic resource allocation that scales the number of executors, registered with this application, up and down, based on the workload

false

spark.yarn.archive

The archive containing all the required Spark JARs for distribution to the YARN cache. If set, this configuration replaces spark.yarn.jars and the archive is used in all the application containers. The archive should contain JAR files in its root directory. The archive can also be hosted on HDFS to speed up file distribution

hdfs:///apps/spark/spark3-yarn-archive.tgz

spark.yarn.historyServer.address

Spark History server address

 — 

spark.master

The cluster manager to connect to

yarn

spark.dynamicAllocation.enabled

Defines whether to use dynamic resource allocation that scales the number of executors, registered with this application, up and down, based on the workload

false

spark.shuffle.service.enabled

Enables the external shuffle service. This service preserves the shuffle files written by executors so that executors can be safely removed, or so that shuffle fetches can continue in the event of executor failure. The external shuffle service must be set up in order to enable it

false

spark.eventLog.enabled

Defines whether to log Spark events, useful for reconstructing the Web UI after the application has finished

true

spark.eventLog.dir

The base directory where Spark events are logged, if spark.eventLog.enabled=true. Within this base directory, Spark creates a sub-directory for each application, and logs the events specific to the application in this directory. You may want to set this to a unified location like an HDFS directory so history files can be read by the History Server

hdfs:///var/log/spark/apps

spark.dynamicAllocation.executorIdleTimeout

If dynamic allocation is enabled and an executor has been idle for more than this duration, the executor will be removed. For more details, see Spark documentation

120s

spark.dynamicAllocation.cachedExecutorIdleTimeout

If dynamic allocation is enabled and an executor which has cached data blocks has been idle for more than this duration, the executor will be removed. For more details, see Spark documentation

600s

spark.history.provider

The name of the class that implements the application history backend. Currently there is only one implementation provided with Spark that looks for application logs stored in the file system

org.apache.spark.deploy.history.FsHistoryProvider

spark.history.fs.cleaner.enabled

Specifies whether the History Server should periodically clean up event logs from storage

true

spark.history.store.path

A local directory where to cache application history data. If set, the History Server will store application data on disk instead of keeping it in memory. The data written to disk will be re-used in case of the History Server restart

/var/log/spark3/history

spark.serializer

The class used for serializing objects that will be sent over the network or need to be cached in the serialized form. By default, works with any Serializable Java object but it may be quite slow, so we recommend using org.apache.spark.serializer.KryoSerializer and configuring Kryo serialization when speed is necessary. Can be any subclass of org.apache.spark.Serializer

org.apache.spark.serializer.KryoSerializer

spark.driver.extraClassPath

Extra classpath entries to prepend to the classpath of the driver

/usr/lib/hive/lib/hive-shims-scheduler.jar:/usr/lib/hadoop-yarn/hadoop-yarn-server-resourcemanager.jar

spark.history.ui.port

The port number of the History Server web UI

18092

spark.ui.port

The port number of the Thrift Server web UI

4140

spark.history.fs.logDirectory

The log directory of the History Server

hdfs:///var/log/spark/apps

spark.sql.hive.metastore.jars

The location of the JARs that should be used to instantiate HiveMetastoreClient

path

spark.sql.hive.metastore.jars.path

A list of comma-separated paths to JARs used to instantiate HiveMetastoreClient

file:///usr/lib/hive/lib/*.jar

spark.sql.hive.metastore.version

The Hive Metastore version

3.1.2

spark.driver.extraLibraryPath

The path to extra native libraries for driver

/usr/lib/hadoop/lib/native/

spark.yarn.am.extraLibraryPath

The path to extra native libraries for Application Master

/usr/lib/hadoop/lib/native/

spark.executor.extraLibraryPath

The path to extra native libraries for Executor

/usr/lib/hadoop/lib/native/

spark.yarn.appMasterEnv.HIVE_CONF_DIR

A directory on the Application Master with Hive configs required for running Hive in the cluster mode

/etc/spark3/conf

spark.yarn.historyServer.allowTracking

Allows to use Spark History Server for tracking UI even if web UI is disabled for a job

True

spark.connect.grpc.binding.port

The port number to connect to Spark Connect via gRPC

15002

spark.history.kerberos.enabled

Indicates whether the History Server should use Kerberos to login. This is required if the History Server is accessing HDFS files on a secure Hadoop cluster

false

spark.acls.enable

Defines whether Spark ACLs should be enabled. If enabled, checks to see if the user has access permissions to view or modify the job. Note this requires the user to be known, so if the user comes across as null no checks are done. Filters can be used within the UI to authenticate and set the user

false

spark.modify.acls

Defines who has access to modify a running Spark application

spark,hdfs

spark.modify.acls.groups

A comma-separated list of user groups that have modify access to the Spark application

spark,hdfs

spark.history.ui.acls.enable

Specifies whether ACLs should be checked to authorize users viewing the applications in the History Server. If enabled, access control checks are performed regardless of what the individual applications had set for spark.ui.acls.enable. If disabled, no access control checks are made for any application UIs available through the History Server

false

spark.history.ui.admin.acls

A comma-separated list of users that have view access to all the Spark applications in History Server

spark,hdfs,dr.who

spark.history.ui.admin.acls.groups

A comma-separated list of groups that have view access to all the Spark applications in History Server

spark,hdfs,dr.who

spark.ui.view.acls

A comma-separated list of users that have view access to the Spark application. By default, only the user that started the Spark job has view access. Using * as a value means that any user can have view access to this Spark job

spark,hdfs,dr.who

spark.ui.view.acls.groups

A comma-separated list of groups that have view access to the Spark web UI to view the Spark Job details. This can be used if you have a set of administrators or developers or users who can monitor the Spark job submitted. Using * in the list means any user in any group can view the Spark job details on the Spark web UI. The user groups are obtained from the instance of the groups mapping provider specified by spark.user.groups.mapping

spark,hdfs,dr.who

spark.ssl.keyPassword

The password to the private key in the keystore

 — 

spark.ssl.keyStore

Path to the keystore file. The path can be absolute or relative to the directory in which the process is started

 — 

spark.ssl.keyStoreType

The type of keystore used

JKS

spark.ssl.trustStorePassword

The password to the private key in the truststore

 — 

spark.ssl.trustStoreType

The type of the truststore

JKS

spark.ssl.enabled

Defines whether to use SSL for Spark

 — 

spark.ssl.protocol

Defines the TLS protocol to use. The protocol must be supported by JVM

TLSv1.2

spark.ssl.ui.port

The port number used by Spark web UI in case of active SSL

4041

spark.ssl.historyServer.port

The port number used by Spark History Server web UI in case of active SSL

18092

livy.server.host

The host address to start the Livy server. By default, Livy will bind to all network interfaces

0.0.0.0

livy.server.port

The port to run the Livy server

8999

livy.spark.master

The Spark master to use for Livy sessions

yarn

livy.impersonation.enabled

Defines if Livy should impersonate users when creating a new session

true

livy.server.csrf-protection.enabled

Defines whether to enable the CSRF protection. If enabled, clients should add the X-Requested-By HTTP header for POST/DELETE/PUT/PATCH HTTP methods

true

livy.repl.enable-hive-context

Defines whether to enable HiveContext in the Livy interpreter. If set to true, hive-site.xml and the Livy server classpath will be detected on user request automatically

true

livy.server.recovery.mode

Sets the recovery mode for Livy

recovery

livy.server.recovery.state-store

Defines where Livy should store the state for recovery

filesystem

livy.server.recovery.state-store.url

For the filesystem state store, the path of the state store directory. Do not use a filesystem that does not support atomic rename like S3. For example: file:///tmp/livy or hdfs:///. For ZooKeeper, specify the address to the ZooKeeper servers. For example: host1:port1,host2:port2

/livy-recovery

livy.server.auth.type

Sets the Livy authentication type

 — 

livy.server.access_control.enabled

Defines whether to enable the access control for a Livy server. If set to true, then all the incoming requests will be checked if the requested user has permission

false

livy.server.access_control.users

Users allowed to access Livy. By default, any user is allowed to access Livy. If a user wants to limit the access, the user should list all the permitted users separated by a comma

livy,hdfs,spark

livy.superusers

A list of comma-separated users that have the permissions to change other user’s submitted sessions, for example, submitting statements, deleting the session, and so on

livy,hdfs,spark

livy.keystore

A path to the keystore file. The path can be absolute or relative to the directory in which the process is started

 — 

livy.keystore.password

The password to access the keystore

 — 

livy.key-password

The password to access the key in the keystore

 — 

livy.server.thrift.ssl.protocol.blacklist

The list of banned TLS protocols

SSLv2,SSLv3,TLSv1,TLSv1.1

thrift.server.port

The port number used for communication with Spark3 Thrift Server

10116

Spark History Server Heap Memory

Sets the maximum Java heap size for Spark History Server

1G

Custom spark-defaults.conf

 — 

Custom log4j2.properties

The contents of the log4j2.properties file used for logging the Spark3 activity

log4j2.properties

spark-env.sh

The contents of the spark-env.sh file used to initialize environment variables on worker nodes

spark-env.sh

Custom livy.conf

 — 

livy-env.sh

The contents of the livy-env.sh file used to initialize environment variables for the Livy server operation

livy-env.sh

spark-history-env.sh

The contents of the spark-history-env.sh file used to initialize environment variables for the Spark3 History Server operation

spark-history-env.sh

thriftserver-env.sh

The contents of the thriftserver-env.sh file used to initialize environment variables for the Spark3 Thrift Server operation

thriftserver-env.sh

Credential provider path

The path to a keystore file used to encrypt credentials

jceks://file/etc/ssm/conf/ssm.jceks

Custom jceks

Set to true to use a custom JCEKS file. Set to false to use the auto-generated JCEKS keystore

false

Password file name

The name of the file that stores a password to access the keystore

ssm_credstore_pass

smart.hadoop.conf.path

The path to the Hadoop configuration directory

/etc/hadoop/conf

smart.conf.dir

The path to the SSM configuration directory

/etc/ssm/conf

smart.server.rpc.address

The RPC address of the SSM Server

0.0.0.0:7042

smart.server.http.address

The HTTP address (web UI) of the SSM Server

0.0.0.0:7045

smart.agent.master.address

The active SSM server’s address

<hostname>

smart.agent.address

Defines the address of SSM Agent components on each host

0.0.0.0

smart.agent.port

The port number used by SSM agents to communicate with the SSM Server

7048

smart.agent.master.port

The port number used by the SSM Server to communicate with SSM agents

7051

smart.ignore.dirs

A list of comma-separated HDFS directories to ignore. SSM will ignore all files under the given HDFS directories

 — 

smart.cover.dirs

A list of comma-separated HDFS directories where SSM scans for files. By default, all HDFS files are covered

 — 

smart.work.dir

The HDFS directory used by SSM as a working directory to store temporary files. SSM will ignore HDFS inotify events for all files under the working directory. Only one directory can be set

/system/ssm

smart.client.concurrent.report.enabled

Used to enable/disable concurrent reports for Smart Client. If enabled, Smart Client concurrently attempts to connect to multiple configured Smart Servers to find the active Smart Server, which is an optimization. Only the active Smart Server will respond to establish the connection. If the report has been successfully delivered to the active Smart Server, connection attempts to other Smart Servers are canceled

 — 

smart.server.rpc.handler.count

The number of RPC handlers on the server

80

smart.namespace.fetcher.batch

The batch size of the namespace fetcher. SSM fetches namespaces from the NameNode during the startup. Large namespaces may lead to long startup time. A larger batch size can speed up the fetcher efficiency and reduce the startup time

500

smart.namespace.fetcher.producers.num

The number of producers in the namespace fetcher

3

smart.namespace.fetcher.consumers.num

The number of consumers in the namespace fetcher

6

smart.rule.executors

The maximum number of rules that can be executed in parallel

5

smart.cmdlet.executors

The maximum number of cmdlets that can be executed in parallel

10

smart.dispatch.cmdlets.extra.num

The number of extra cmdlets dispatched by Smart Server

10

smart.cmdlet.dispatchers

The maximum number of cmdlet dispatchers that work in parallel

3

smart.cmdlet.mover.max.concurrent.blocks.per.srv.inst

The maximum number of file mover cmdlets that can be executed in parallel per SSM service. The 0 value removes the limit

0

smart.action.move.throttle.mb

The throughput limit (in MB) for the SSM move operation

0

smart.action.copy.throttle.mb

The throughput limit (in MB) for the SSM copy operation

0

smart.action.ec.throttle.mb

The throughput limit (in MB) for the SSM EC operation

0

smart.action.local.execution.disabled

Defines whether the active Smart Server can also execute actions like an agent. If set to true, the active SSM Server will NOT be able to execute actions. This configuration has no impact on a standby Smart Server

false

smart.cmdlet.max.num.pending

The maximum number of pending cmdlets in an SSM Server

20000

smart.cmdlet.hist.max.num.records

The maximum number of historic cmdlet records kept in an SSM server. SSM deletes the oldest cmdlets when this threshold is exceeded

100000

smart.cmdlet.hist.max.record.lifetime

The maximum lifetime of historic cmdlet records kept in an SSM server. The SSM Server deletes cmdlet records after the specified interval. Valid time units are day, hour, min, sec. The minimum update granularity is 5sec

30day

smart.cmdlet.cache.batch

The maximum batch size of the cmdlet batch insert

600

smart.copy.scheduler.base.sync.batch

The maximum batch size of the Copy Scheduler base sync batch insert

500

smart.file.diff.max.num.records

The maximum file diff records with useless state

10000

smart.status.report.period

The status report period for actions in milliseconds

10

smart.status.report.period.multiplier

The report period multiplied by this value defines the largest report interval

50

smart.status.report.ratio

If the finished actions ratio equals or exceeds this value, a status report will be triggered

0.2

smart.top.hot.files.num

The number of top hot files displayed in web UI

200

smart.cmdlet.dispatcher.log.disp.result

Defines whether to log dispatch results for each cmdlet dispatched

false

smart.cmdlet.dispatcher.log.disp.metrics.interval

The time interval in milliseconds to log statistic metrics of the cmdlet dispatcher. If no cmdlets were dispatched within this interval, no output is generated for this interval. The 0 value disables the logger

5000

smart.compression.codec

The default compression codec for SSM compression (Zlib, Lz4, Bzip2, snappy). You can also specify codecs as action arguments, which overrides this setting

Zlib

smart.compression.max.split

The maximum number of chunks split for compression

1000

smart.compact.batch.size

The maximum number of small files to be compacted by the compact action

200

smart.compact.container.file.threshold.mb

The maximum size of a container file in MB

1024

smart.access.count.day.tables.num

The maximum number of tables that can be created in the Metastore database to store the file access count per day

30

smart.access.count.hour.tables.num

The maximum number of tables that can be created in the Metastore database to store the file access count per hour

48

smart.access.count.minute.tables.num

The maximum number of tables that can be created in the Metastore database to store the file access count per minute

120

smart.access.count.second.tables.num

The maximum number of tables that can be created in the Metastore database to store the file access count per second

30

smart.access.event.fetch.interval.ms

The interval in milliseconds between access event fetches

1000

smart.cached.file.fetch.interval.ms

The interval in milliseconds between fetches of cached files from HDFS

5000

smart.namespace.fetch.interval.ms

The interval in milliseconds between namespace fetches from HDFS

1

smart.mover.scheduler.storage.report.fetch.interval.ms

The interval in milliseconds between fetches of storage reports from HDFS DataNodes in the mover scheduler

120000

smart.metastore.small-file.insert.batch.size

The maximum size of the Metastore insert batch with information about small files

200

smart.agent.master.ask.timeout.ms

The maximum time in milliseconds for a Smart Agent to wait for a response from the Smart Server during the submission action

5000

smart.ignore.path.templates

A list of comma-separated regex templates of HDFS paths to be completely ignored by SSM

 — 

smart.internal.path.templates

A list of comma-separated regex templates of internal files to be completely ignored by SSM

.*/\..*,.*/__.*,.*_COPYING_.*

smart.security.enable

Enables Kerberos authentication for SSM

false

smart.server.keytab.file

The path to the SSM Server’s keytab file

 — 

smart.server.kerberos.principal

The SSM Server’s Kerberos principal

 — 

smart.agent.keytab.file

The path to the SSM Agent’s keytab file

 — 

smart.agent.kerberos.principal

The SSM Agent’s Kerberos principal

 — 

db_url

The URL to the Metastore database

jdbc:postgresql://{{ groups['adpg.adpg'][0] | d(omit) }}:5432/ssm

db_user

The user name to connect to the database

ssm

db_password

The user password to connect to the database

 — 

initialSize

The initial number of connections created when the pool is started

10

minIdle

The minimum number of established connections that should be kept in the pool at all times. The connection pool can shrink below this number if validation queries fail

4

maxActive

The maximum number of active connections that can be allocated from this pool at the same time

50

maxWait

The maximum time in milliseconds the pool will wait (when there are no available connections) for a connection to be returned before throwing an exception

60000

timeBetweenEvictionRunsMillis

The time in milliseconds to sleep between the runs of the idle connection validation/cleaner thread. This value should not be set less than 1 second. It specifies how often to check for idle and abandoned connections, and how often to validate idle connections

90000

minEvictableIdleTimeMillis

The minimum amount of time an object may remain idle in the pool before it is eligible for eviction

300000

validationQuery

The SQL query used to validate connections from the pool before returning them to the caller

SELECT 1

testWhileIdle

Indicates whether connection objects are validated by the idle object evictor (if any)

true

testOnBorrow

Indicates whether objects are validated before being borrowed from the pool

false

testOnReturn

Indicates whether objects are validated before being returned to the pool

false

poolPreparedStatements

Enables the prepared statement pooling

true

maxPoolPreparedStatementPerConnectionSize

The maximum number of prepared statements that can be pooled per connection

30

removeAbandoned

A flag to remove abandoned connections if they exceed removeAbandonedTimeout

true

removeAbandonedTimeout

The timeout in seconds before an abandoned (in use) connection can be removed

180

logAbandoned

A flag to log stack traces for application code which abandoned a connection. Logging of abandoned connections adds extra overhead for every borrowed connection

true

filters

Sets the filters that are applied to the data source

stat

LD_LIBRARY_PATH

The path to extra native libraries for SSM

/usr/lib/hadoop/lib/native

HADOOP_HOME

The path to the Hadoop home directory

/usr/lib/hadoop

Enable SmartFileSystem for Hadoop

When enabled, requests from different clients (Spark, HDFS, Hive, etc.) are taken into account when calculating AccessCount for files. Otherwise, the AccessCount value gets incremented only when a file is accessed from SSM

false

log4j.properties

The contents of the log4j.properties configuration file

 — 

zeppelin-site.xml

The contents of the zeppelin-site.xml configuration file. SSM uses a Zeppelin configuration for web UI

 — 

sqoop.metastore.client.autoconnect.url

The connection string to use when connecting to a job-management metastore. If not set, uses ~/.sqoop/

 — 

sqoop.metastore.server.location

The path to the shared metastore database files. If not set, uses ~/.sqoop/

/srv/sqoop/metastore.db