YARN configuration parameters

To configure the service, use the following configuration parameters in ADCM.

NOTE
  • Some of the parameters become visible in the ADCM UI after the Advanced flag has been set.

  • The parameters that are set in the Custom group will overwrite the existing parameters even if they are read-only.

mapred-site.xml
Parameter Description Default value

mapreduce.application.classpath

Classpath for MapReduce applications. A list of files/directories to be added to the classpath. To add more items to the classpath, click Plus icon Plus icon . If mapreduce.application.framework is set, then this parameter must specify the appropriate classpath for that archive, and the name of the archive must be present in the classpath. If mapreduce.app-submission.cross-platform=false, the platform-specific environment variable expansion syntax would be used to construct the default classpath entries. If mapreduce.app-submission.cross-platform=true, the platform-agnostic default classpath for MapReduce applications would be used:

{{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*, {{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/*

Parameter expansion marker will be replaced by NodeManager on container launch, based on the underlying OS accordingly

  • /etc/hadoop/conf/*

  • /usr/lib/hadoop/*

  • /usr/lib/hadoop/lib/*

  • /usr/lib/hadoop-hdfs/*

  • /usr/lib/hadoop-hdfs/lib/*

  • /usr/lib/hadoop-yarn/*

  • /usr/lib/hadoop-yarn/lib/*

  • /usr/lib/hadoop-mapreduce/*

  • /usr/lib/hadoop-mapreduce/lib/*

mapreduce.cluster.local.dir

Local directory where MapReduce stores intermediate data files. May be a comma-separated list of directories on different devices in order to spread disk I/O. Directories that do not exist, are ignored

/srv/hadoop-yarn/mr-local

mapreduce.framework.name

Runtime framework for executing MapReduce jobs. Can be one of local, classic, or yarn

yarn

mapreduce.jobhistory.address

MapReduce JobHistory Server IPC (<host>:<port>)

 — 

mapreduce.jobhistory.bind-host

Setting the value to 0.0.0.0 will cause the MapReduce daemons to listen on all addresses and interfaces of the hosts in the cluster

0.0.0.0

mapreduce.jobhistory.webapp.address

MapReduce JobHistory Server Web UI (<host>:<port>)

 — 

mapreduce.map.env

Environment variables for the map task processes added by a user, specified as a comma separated list. Example: VAR1=value1,VAR2=value2

HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce

mapreduce.reduce.env

Environment variables for the reduce task processes added by a user, specified as a comma separated list. Example: VAR1=value1,VAR2=value2

HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce

yarn.app.mapreduce.am.env

Environment variables for the MapReduce App Master processes added by a user. Examples:

  • A=foo. This sets the environment variable A to foo.

  • B=$B:c. This inherits the tasktracker B environment variable.

HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce

yarn.app.mapreduce.am.staging-dir

The staging directory used while submitting jobs

/user

mapreduce.jobhistory.keytab

The location of the Kerberos keytab file for the MapReduce JobHistory Server

/etc/security/keytabs/mapreduce-historyserver.service.keytab

mapreduce.jobhistory.principal

Kerberos principal name for the MapReduce JobHistory Server

mapreduce-historyserver/_HOST@REALM

mapreduce.jobhistory.http.policy

Configures the HTTP endpoint for JobHistoryServer web UI. The following values are supported:

  • HTTP_ONLY — provides service only via HTTP;

  • HTTPS_ONLY — provides service only via HTTPS.

HTTP_ONLY

mapreduce.jobhistory.webapp.https.address

The HTTPS address where MapReduce JobHistory Server WebApp is running

0.0.0.0:19890

mapreduce.shuffle.ssl.enabled

Defines whether to use SSL for for the Shuffle HTTP endpoints

false

ranger-yarn-audit.xml
Parameter Description Default value

xasecure.audit.destination.solr.batch.filespool.dir

Spool directory path

/srv/ranger/hdfs_plugin/audit_solr_spool

xasecure.audit.destination.solr.urls

A URL of the Solr server to store audit events. Leave this property value empty or set it to NONE when using ZooKeeper to connect to Solr

 — 

xasecure.audit.destination.solr.zookeepers

Specifies the ZooKeeper connection string for the Solr destination

 — 

xasecure.audit.destination.solr.force.use.inmemory.jaas.config

Whether to use in-memory JAAS configuration file to connect to Solr

 — 

xasecure.audit.is.enabled

Enables Ranger audit

true

xasecure.audit.jaas.Client.loginModuleControlFlag

Specifies whether the success of the module is required, requisite, sufficient, or optional

 — 

xasecure.audit.jaas.Client.loginModuleName

Name of the authenticator class

 — 

xasecure.audit.jaas.Client.option.keyTab

Name of the keytab file to get the principal’s secret key

 — 

xasecure.audit.jaas.Client.option.principal

Name of the principal to be used

 — 

xasecure.audit.jaas.Client.option.serviceName

Name of a user or a service that wants to log in

 — 

xasecure.audit.jaas.Client.option.storeKey

Set this to true if you want the keytab or the principal’s key to be stored in the subject’s private credentials

false

xasecure.audit.jaas.Client.option.useKeyTab

Set this to true if you want the module to get the principal’s key from the keytab

false

ranger-yarn-security.xml
Parameter Description Default value

ranger.plugin.yarn.policy.rest.url

The URL to Ranger Admin

 — 

ranger.plugin.yarn.service.name

The name of the Ranger service containing policies for this instance

 — 

ranger.plugin.yarn.policy.cache.dir

The directory where Ranger policies are cached after successful retrieval from the source

/srv/ranger/yarn/policycache

ranger.plugin.yarn.policy.pollIntervalMs

Defines how often to poll for changes in policies

30000

ranger.plugin.yarn.policy.rest.client.connection.timeoutMs

The YARN Plugin RangerRestClient connection timeout (in milliseconds)

120000

ranger.plugin.yarn.policy.rest.client.read.timeoutMs

The YARN Plugin RangerRestClient read timeout (in milliseconds)

30000

ranger.add-yarn-authorization

Set true to use only Ranger ACLs (i.e. ignore YARN ACLs)

false

ranger.plugin.yarn.policy.rest.ssl.config.file

Path to the RangerRestClient SSL config file for the YARN plugin

/etc/yarn/conf/ranger-yarn-policymgr-ssl.xml

yarn-site.xml
Parameter Description Default value

yarn.application.classpath

The classpath for YARN applications. A list of files/directories to be added to the classpath. To add more items to the classpath, click Plus icon Plus icon

  • /etc/hadoop/conf/*

  • /usr/lib/hadoop/*

  • /usr/lib/hadoop/lib/*

  • /usr/lib/hadoop-hdfs/*

  • /usr/lib/hadoop-hdfs/lib/*

  • /usr/lib/hadoop-yarn/*

  • /usr/lib/hadoop-yarn/lib/*

  • /usr/lib/hadoop-mapreduce/*

  • /usr/lib/hadoop-mapreduce/lib/*

  • /usr/lib/hive/lib/*.jar

yarn.cluster.max-application-priority

Defines the maximum application priority in a cluster. Leaf Queue-level priority: each leaf queue provides default priority by the administrator. The queue default priority will be used for any application submitted without a specified priority. $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml is the configuration file for queue-level priority

0

yarn.log.server.url

The URL for log aggregation Server

 — 

yarn.log-aggregation-enable

Whether to enable log aggregation. Log aggregation collects logs from each container and moves these logs onto a file system, for example HDFS, after the application processing completes. Users can configure the yarn.nodemanager.remote-app-log-dir and yarn.nodemanager.remote-app-log-dir-suffix properties to determine, where these logs are moved to. Users can access the logs via the Application Timeline Server

true

yarn.log-aggregation.retain-seconds

Defines how long to keep aggregation logs before deleting them. The value of -1 disables logs saving. Be careful: setting this value too small will spam the NameNode

172800

yarn.nodemanager.local-dirs

The list of directories to store localized. An application localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers work directories, called container_${contid}, will be subdirectories of this

/srv/hadoop-yarn/nm-local

yarn.node-labels.enabled

Enables node labels feature

true

yarn.node-labels.fs-store.root-dir

The URI for NodeLabelManager. The default value is /tmp/hadoop-yarn-${user}/node-labels/ in the local filesystem

hdfs:///system/yarn/node-labels

yarn.timeline-service.bind-host

The actual address the server will bind to. If this optional address is set, the RPC and Webapp servers will bind to this address and the port, specified in yarn.timeline-service.address and yarn.timeline-service.webapp.address, respectively. This is most useful for making the service listen to all interfaces by setting to 0.0.0.0

0.0.0.0

yarn.timeline-service.leveldb-timeline-store.path

Stores file name for leveldb Timeline store

/srv/hadoop-yarn/leveldb-timeline-store

yarn.nodemanager.address

The address of the container manager in the NodeManager

0.0.0.0:8041

yarn.nodemanager.aux-services

A comma-separated list of services, where service name should only contain a-zA-Z0-9_ and cannot start with numbers

mapreduce_shuffle,spark_shuffle

yarn.nodemanager.aux-services.mapreduce_shuffle.class

The auxiliary service class to use

org.apache.hadoop.mapred.ShuffleHandler

yarn.nodemanager.aux-services.spark_shuffle.class

The class name of YarnShuffleService — an external shuffle service for Spark3 on YARN

org.apache.spark.network.yarn.YarnShuffleService

yarn.nodemanager.aux-services.spark_shuffle.classpath

The classpath for external Spark3 shuffle-service in YARN. A list of files/directories to be added to the classpath. To add more items to the classpath, click Plus icon Plus icon

  • /usr/lib/spark3/yarn/lib/*

yarn.nodemanager.recovery.enabled

Enables the NodeManager to recover after starting

true

yarn.nodemanager.recovery.dir

The local filesystem directory, in which the NodeManager will store state, when recovery is enabled

/srv/hadoop-yarn/nm-recovery

yarn.nodemanager.remote-app-log-dir

Defines a directory for logs aggregation

/logs

yarn.nodemanager.resource-plugins

Enables additional discovery/isolation of resources on the NodeManager. By default, this parameters is empty. Acceptable values: yarn.io/gpu, yarn.io/fpga

 — 

yarn.nodemanager.resource-plugins.gpu.path-to-discovery-executables

When yarn.nodemanager.resource.gpu.allowed-gpu-devices=auto, YARN NodeManager needs to run GPU discovery binary (now only support nvidia-smi) to get GPU-related information. When value is empty (default), YARN NodeManager will try to locate discovery executable itself. An example of the config value is: /usr/local/bin/nvidia-smi

/usr/bin/nvidia-smi

yarn.nodemanager.resource.detect-hardware-capabilities

Enables auto-detection of node capabilities such as memory and CPU

true

yarn.nodemanager.vmem-check-enabled

Whether virtual memory limits will be enforced for containers

false

yarn.resource-types

The resource types to be used for scheduling. Use resource-types.xml to specify details about the individual resource types

 — 

yarn.resourcemanager.bind-host

The actual address, the server will bind to. If this optional address is set, the RPC and Webapp servers will bind to this address and the port, specified in yarn.resourcemanager.address and yarn.resourcemanager.webapp.address, respectively. This is most useful for making Resource Manager listen to all interfaces by setting to 0.0.0.0

0.0.0.0

yarn.resourcemanager.cluster-id

Name of the cluster. In the High Availability mode, this parameter is used to ensure that Resource Manager participates in leader election for this cluster and ensures that it does not affect other clusters

 — 

yarn.resource-types.memory-mb.increment-allocation

The FairScheduler grants memory equal to increments of this value. If you submit a task with a resource request which is not a multiple of memory-mb.increment-allocation, the request will be rounded up to the nearest increment

1024

yarn.resource-types.vcores.increment-allocation

The FairScheduler grants vcores in increments of this value. If you submit a task with resource request, that is not a multiple of vcores.increment-allocation, the request will be rounded up to the nearest increment

1

yarn.resourcemanager.ha.enabled

Enables Resource Manager High Availability. When enabled:

  • The Resource Manager starts in the Standby mode by default, and transitions to the Active mode when prompted to.

  • The nodes in the Resource Manager ensemble are listed in yarn.resourcemanager.ha.rm-ids.

  • The id of each Resource Manager either comes from yarn.resourcemanager.ha.id, if yarn.resourcemanager.ha.id is explicitly specified, or can be figured out by matching yarn.resourcemanager.address.{id} with local address.

  • The actual physical addresses come from the configs of the pattern {rpc-config}.{id}.

false

yarn.resourcemanager.ha.rm-ids

The list of Resource Manager nodes in the cluster when the High Availability is enabled. See description of yarn.resourcemanager.ha.enabled for full details on how this is used

 — 

yarn.resourcemanager.hostname

The host name of the Resource Manager

 — 

yarn.resourcemanager.leveldb-state-store.path

The Local path, where the Resource Manager state will be stored, when using org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore as the value for yarn.resourcemanager.store.class

/srv/hadoop-yarn/leveldb-state-store

yarn.resourcemanager.monitor.capacity.queue-management.monitoring-interval

Time between invocations of this QueueManagementDynamicEditPolicy policy (in milliseconds)

1500

yarn.resourcemanager.reservation-system.enable

Enables the ReservationSystem in the ResourceManager

false

yarn.resourcemanager.reservation-system.planfollower.time-step

The frequency of the PlanFollower timer (in milliseconds). A large value is expected

1000

Resource scheduler

The type of a pluggable scheduler for Hadoop. Available values: CapacityScheduler and FairScheduler. CapacityScheduler allows for multiple-tenants to securely share a large cluster such that their applications are allocated resources in a timely manner under constraints of allocated capacities. FairScheduler allows YARN applications to share resources in large clusters fairly

CapacityScheduler

yarn.resourcemanager.scheduler.monitor.enable

Enables a set of periodic monitors (specified in yarn.resourcemanager.scheduler.monitor.policies) that affect the Scheduler

false

yarn.resourcemanager.scheduler.monitor.policies

The list of SchedulingEditPolicy classes that interact with the Scheduler. A particular module may be incompatible with the Scheduler, other policies, or a configuration of either

org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy

yarn.resourcemanager.monitor.capacity.preemption.observe_only

If set to true, run the policy but do not affect the cluster with preemption and kill events

false

yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval

Time between invocations of this ProportionalCapacityPreemptionPolicy policy (in milliseconds)

3000

yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill

Time between requesting a preemption from an application and killing the container (in milliseconds)

15000

yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round

Maximum percentage of resources, preempted in a single round. By controlling this value one can throttle the pace, at which containers are reclaimed from the cluster. After computing the total desired preemption, the policy scales it back within this limit

0.1

yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity

Maximum amount of resources above the target capacity ignored for preemption. This defines a deadzone around the target capacity, that helps to prevent thrashing and oscillations around the computed target balance. High values would slow the time to capacity and (absent natural.completions) it might prevent convergence to guaranteed capacity

0.1

yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor

Given a computed preemption target, account for containers naturally expiring and preempt only this percentage of the delta. This determines the rate of geometric convergence into the deadzone (MAX_IGNORED_OVER_CAPACITY). For example, a termination factor of 0.5 will reclaim almost 95% of resources within 5 * #WAIT_TIME_BEFORE_KILL, even absent natural termination

0.2

yarn.resourcemanager.nodes.exclude-path

Path to the file with nodes to exclude

/etc/hadoop/conf/exclude-path.xml

yarn.resourcemanager.nodes.include-path

Path to the file with nodes to include

/etc/hadoop/conf/include-path

yarn.resourcemanager.recovery.enabled

Enables Resource Manager to recover state after starting. If set to true, then yarn.resourcemanager.store.class must be specified

true

yarn.resourcemanager.store.class

The class to use as the persistent store. If org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore is used, the store is implicitly fenced; meaning a single Resource Manager is able to use the store at any point in time. More details on this implicit fencing, along with setting up appropriate ACLs is discussed under yarn.resourcemanager.zk-state-store.root-node.acl

 — 

yarn.resourcemanager.system-metrics-publisher.enabled

The setting that controls whether YARN system metrics are published on the Timeline Server or not by Resource Manager

true

yarn.scheduler.fair.user-as-default-queue

Defines whether to use the username, associated with the allocation as the default queue name, in the event, that a queue name is not specified. If this is set to false or unset, all jobs have a shared default queue, named default. Defaults to true. If a queue placement policy is given in the allocations file, this property is ignored

true

yarn.scheduler.fair.preemption

Defines whether to use preemption

false

yarn.scheduler.fair.preemption.cluster-utilization-threshold

The utilization threshold after which the preemption kicks in. The utilization is computed as the maximum ratio of usage to capacity among all resources

0.8f

yarn.scheduler.fair.sizebasedweight

Defines whether to assign shares to individual apps based on their size, rather than providing an equal share to all apps regardless of size. When set to true, apps are weighted by the natural logarithm of one plus the app total requested memory, divided by the natural logarithm of 2

false

yarn.scheduler.fair.assignmultiple

Defines whether to allow multiple container assignments in one heartbeat

false

yarn.scheduler.fair.dynamic.max.assign

If assignmultiple is true, this parameter specifies whether to dynamically determine the amount of resources that can be assigned in one heartbeat. When turned on, about half of the non-allocated resources on the node are allocated to containers in a single heartbeat

true

yarn.scheduler.fair.max.assign

If assignmultiple is true, the maximum amount of containers that can be assigned in one heartbeat. Defaults to -1, which sets no limit

-1

yarn.scheduler.fair.locality.threshold.node

For applications that request containers on particular nodes, this parameter defines the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another node. Expressed as a floating number between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. The default value of -1.0 means not to pass up any scheduling opportunities

-1.0

yarn.scheduler.fair.locality.threshold.rack

For applications, that request containers on particular racks, the number of scheduling opportunities since the last container assignment to wait before accepting a placement on another rack. Expressed as a floating point between 0 and 1, which, as a fraction of the cluster size, is the number of scheduling opportunities to pass up. The default value of -1.0 means not to pass up any scheduling opportunities

-1.0

yarn.scheduler.fair.allow-undeclared-pools

If set to true, new queues can be created at application submission time, whether because they are specified as the application queue by the submitter or because they are placed there by the user-as-default-queue property. If set to false, any time an app would be placed in a queue that is not specified in the allocations file, it is placed in the default queue instead. Defaults to true. If a queue placement policy is given in the allocations file, this property is ignored

true

yarn.scheduler.fair.update-interval-ms

Time interval, at which to lock the scheduler and recalculate fair shares, recalculate demand, and check whether anything is due for preemption

500

yarn.scheduler.minimum-allocation-mb

Minimum allocation for every container request at the Resource Manager (in MB). Memory requests, lower than this, will throw InvalidResourceRequestException

1024

yarn.scheduler.maximum-allocation-mb

Maximum allocation for every container request at the Resource Manager (in MB). Memory requests, higher than this, will throw InvalidResourceRequestException

4096

yarn.scheduler.minimum-allocation-vcores

Minimum allocation for every container request at the Resource Manager, in terms of virtual CPU cores. Requests, lower than this, will throw InvalidResourceRequestException

1

yarn.scheduler.maximum-allocation-vcores

Maximum allocation for every container request at the Resource Manager, in terms of virtual CPU cores. Requests, higher than this, will throw InvalidResourceRequestException

2

yarn.timeline-service.enabled

On the server side this parameter indicates, whether Timeline service is enabled or not. And on the client side, this parameter can be used to indicate whether client wants to use Timeline service. If this parameter is set on the client side along with security, then YARN Client tries to fetch the delegation tokens for the Timeline Server

true

yarn.timeline-service.hostname

The hostname of the Timeline service Web application

 — 

yarn.timeline-service.http-cross-origin.enabled

Enables cross-origin support (CORS) for Timeline Server

true

yarn.webapp.ui2.enable

In the Server side it indicates, whether the new YARN UI v2 is enabled or not

true

yarn.resourcemanager.proxy-user-privileges.enabled

If set to true, ResourceManager will have proxy-user privileges. For example: in a secure cluster, YARN requires the user hdfs delegation-tokens to do localization and log-aggregation on behalf of the user. If this is set to true, ResourceManager is able to request new hdfs delegation tokens on behalf of the user. This is needed by long-running-services, because the hdfs tokens will eventually expire and YARN requires new valid tokens to do localization and log-aggregation. Note that to enable this use case, the corresponding HDFS NameNode must have ResourceManager configured as a proxy-user so that ResourceManager can itself ask for new tokens on behalf of the user when tokens are past their max-life-time

false

yarn.resourcemanager.webapp.spnego-principal

The Kerberos principal to be used for SPNEGO filter for the Resource Manager web UI

HTTP/_HOST@REALM

yarn.resourcemanager.webapp.spnego-keytab-file

The Kerberos keytab file to be used for SPNEGO filter for the Resource Manager web UI

/etc/security/keytabs/HTTP.service.keytab

yarn.nodemanager.linux-container-executor.group

The UNIX group that the linux-container-executor should run as

yarn

yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled

A flag to enable override of the default Kerberos authentication filter with the RM authentication filter to allow authentication using delegation tokens (fallback to Kerberos if the tokens are missing). Only applicable when the http authentication type is kerberos

false

yarn.resourcemanager.principal

The Kerberos principal for the Resource Manager

yarn-resourcemanager/_HOST@REALM

yarn.resourcemanager.keytab

The keytab for the Resource Manager

/etc/security/keytabs/yarn-resourcemanager.service.keytab

yarn.resourcemanager.webapp.https.address

The https address of the Resource Manager web application. If only a host is provided as the value, the webapp will be served on a random port

${yarn.resourcemanager.hostname}:8090

yarn.nodemanager.principal

The Kerberos principal for the NodeManager

yarn-nodemanager/_HOST@REALM

yarn.nodemanager.keytab

Keytab for NodeManager

/etc/security/keytabs/yarn-nodemanager.service.keytab

yarn.nodemanager.webapp.spnego-principal

The Kerberos principal to be used for SPNEGO filter for the NodeManager web interface

HTTP/_HOST@REALM

yarn.nodemanager.webapp.spnego-keytab-file

The Kerberos keytab file to be used for SPNEGO filter for the NodeManager web interface

/etc/security/keytabs/HTTP.service.keytab

yarn.nodemanager.webapp.https.address

The HTTPS address of the NodeManager web application

0.0.0.0:8044

yarn.timeline-service.http-authentication.type

Defines the authentication used for the Timeline Server HTTP endpoint. Supported values are: simple, kerberos, #AUTHENTICATION_HANDLER_CLASSNAME#

simple

yarn.timeline-service.http-authentication.simple.anonymous.allowed

Indicates if anonymous requests are allowed by the Timeline Server when using simple authentication

true

yarn.timeline-service.http-authentication.kerberos.keytab

The Kerberos keytab to be used for the Timeline Server (Collector/Reader) HTTP endpoint

/etc/security/keytabs/HTTP.service.keytab

yarn.timeline-service.http-authentication.kerberos.principal

The Kerberos principal to be used for the Timeline Server (Collector/Reader) HTTP endpoint

HTTP/_HOST@REALM

yarn.timeline-service.principal

The Kerberos principal for the timeline reader. NodeManager principal would be used for timeline collector as it runs as an auxiliary service inside NodeManager

yarn/_HOST@REALM

yarn.timeline-service.keytab

The Kerberos keytab for the timeline reader. NodeManager keytab would be used for timeline collector as it runs as an auxiliary service inside NodeManager

/etc/security/keytabs/yarn.service.keytab

yarn.timeline-service.delegation.key.update-interval

The update interval for delegation keys

86400000

yarn.timeline-service.delegation.token.renew-interval

Time to renew delegation tokens

86400000

yarn.timeline-service.delegation.token.max-lifetime

The maxim token lifetime

86400000

yarn.timeline-service.client.best-effort

Defines, whether a failure to obtain a delegation token should be considered as an application failure (false), or the client should attempt to continue to publish information without it (true)

false

yarn.timeline-service.webapp.https.address

The HTTPS address of the Timeline service web application

${yarn.timeline-service.hostname}:8190

yarn.http.policy

This configures the HTTP endpoint for Yarn Daemons. The following values are supported:

  • HTTP_ONLY — provides service only via HTTP;

  • HTTPS_ONLY — provides service only via HTTPS.

HTTP_ONLY

yarn.nodemanager.container-executor.class

Name of the container-executor Java class

org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor

yarn.nodemanager.recovery.supervised

Specifies whether to retain local data if a NodeManager is present in a cluster. If set to true, local data can be restored after a NodeManager restarts

true

container-executor.cfg
CAUTION

In AstraLinux, regular user UIDs can start from 100. For YARN to work correctly on AstraLinux, set the min.user.id parameter value to 100.

Parameter Description Default value

banned.users

A comma-separated list of users who cannot run applications

bin

min.user.id

Prevents other super-users

500

Enable CORS
Parameter Description Default value

yarn.nodemanager.webapp.cross-origin.enabled

Enables cross-origin support for NodeManager web-services

true

yarn.resourcemanager.webapp.cross-origin.enabled

Enables cross-origin support for ResourceManager web-services

true

yarn_site.enable_cors.active

Enables CORS (Cross-Origin Resource Sharing)

true

yarn-env.sh
Parameter Description Default value

YARN_RESOURCEMANAGER_OPTS

YARN ResourceManager heap memory. Sets initial (-Xms) and maximum (-Xmx) Java heap size for ResourceManager

-Xms1G -Xmx8G

YARN_NODEMANAGER_OPTS

YARN NodeManager heap memory. Sets initial (-Xms) and maximum (-Xmx) Java heap size for NodeManager

 — 

YARN_TIMELINESERVER_OPTS

YARN Timeline Server heap memory. Sets initial (-Xms) and maximum (-Xmx) Java heap size for Timeline Server

-Xms700m -Xmx8G

Lists of decommissioned hosts
Parameter Description Default value

DECOMMISSIONED

The list of hosts in the DECOMMISSIONED state

 — 

ranger-yarn-policymgr-ssl.xml
Parameter Description Default value

xasecure.policymgr.clientssl.keystore

Path to the keystore file used by Ranger

 — 

xasecure.policymgr.clientssl.keystore.credential.file

Path to the keystore credentials file

/etc/yarn/conf/ranger-yarn.jceks

xasecure.policymgr.clientssl.truststore.credential.file

Path to the truststore credentials file

/etc/yarn/conf/ranger-yarn.jceks

xasecure.policymgr.clientssl.truststore

Path to the truststore file used by Ranger

 — 

xasecure.policymgr.clientssl.keystore.password

Password to the keystore file

 — 

xasecure.policymgr.clientssl.truststore.password

Password to the truststore file

 — 

mapred-env.sh
Parameter Description Default value

HADOOP_JOB_HISTORYSERVER_OPTS

MapReduce History Server heap memory. Sets initial (-Xms) and maximum (-Xmx) Java heap size for MapReduce History Server

-Xms700m -Xmx8G

Other
Parameter Description Default value

GPU on YARN

Defines, whether to use GPU on YARN

false

capacity-scheduler.xml

The content of capacity-scheduler.xml, which is used by CapacityScheduler

fair-scheduler.xml

The content of fair-scheduler.xml, which is used by FairScheduler

Custom mapred-site.xml

In this section you can define values for custom parameters that are not displayed in ADCM UI, but are allowed in the configuration file mapred-site.xml

 — 

Ranger plugin enabled

Whether or not Ranger plugin is enabled

false

Custom yarn-site.xml

In this section you can define values for custom parameters that are not displayed in ADCM UI, but are allowed in the configuration file yarn-site.xml

 — 

Custom ranger-yarn-audit.xml

In this section you can define values for custom parameters that are not displayed in ADCM UI, but are allowed in the configuration file ranger-yarn-audit.xml

 — 

Custom ranger-yarn-security.xml

In this section you can define values for custom parameters that are not displayed in ADCM UI, but are allowed in the configuration file ranger-yarn-security.xml

 — 

Custom ranger-yarn-policymgr-ssl.xml

In this section you can define values for custom parameters that are not displayed in ADCM UI, but are allowed in the configuration file ranger-yarn-policymgr-ssl.xml

 — 

Custom mapred-env.sh

In this section you can define values for custom parameters that are not displayed in ADCM UI, but are allowed in the configuration file mapred-env.sh

 — 

Custom yarn-env.sh

In this section you can define values for custom parameters that are not displayed in ADCM UI, but are allowed in the configuration file yarn-env.sh

 — 

container-executor.cfg template

The template for the container-executor.cfg configuration file

 — 

YARN NodeManager component
Monitoring
Parameter Description Default value

Java agent path

Path to the JMX Prometheus Java agent

/usr/lib/adh-utils/jmx/jmx_prometheus_javaagent.jar

Prometheus metrics port

Port on which to display YARN NodeManager metrics in the Prometheus format

9205

Mapping config path

Path to the metrics mapping configuration file

/etc/hadoop/conf/jmx_yarn_nodemanager_metric_config.yml

Mapping config

Metrics mapping configuration file

YARN ResourceManager component
Monitoring
Parameter Description Default value

Java agent path

Path to the JMX Prometheus Java agent

/usr/lib/adh-utils/jmx/jmx_prometheus_javaagent.jar

Prometheus metrics port

Port on which to display YARN ResourceManager metrics in the Prometheus format

9204

Mapping config path

Path to the metrics mapping configuration file

/etc/hadoop/conf/jmx_yarn_resourcemanager_metric_config.yml

Mapping config

Metrics mapping configuration file

YARN Timeline Server component
Monitoring
Parameter Description Default value

Java agent path

Path to the JMX Prometheus Java agent

/usr/lib/adh-utils/jmx/jmx_prometheus_javaagent.jar

Prometheus metrics port

Port on which to display YARN Timeline Server metrics in the Prometheus format

9206

Mapping config path

Path to the metrics mapping configuration file

/etc/hadoop/conf/jmx_yarn_timelineserver_metric_config.yml

Mapping config

Metrics mapping configuration file

Found a mistake? Seleсt text and press Ctrl+Enter to report it