FairScheduler

Mikhail Serov

FairScheduler is a pluggable scheduler for Hadoop that enables YARN applications to share resources across large clusters so that all applications get an equal share of resources on average over time.

FairScheduler distributes resources among all running applications so that short-lived applications terminate in a reasonable amount of time and long-lived applications do not starve. It organizes applications into queues and distributes resources fairly among these queues.

By default, scheduling fairness decisions are based on memory as the only primary capacity, and the only queue for applications is default. You can make it take both memory (mb — megabytes) and CPUs (vcores — virtual CPUs) into account. Applications supplied to the scheduler can request creating new queues as described in Setting up queues.

Resource distribution example

Let’s consider two queues:

  • sales

  • finance

At some point, the first job is submitted to the sales queue. Being a sole running job, it will get all the resources.

Then another job is submitted to the finance queue. This will result in the new job gradually getting half of the resources.

So each job in those queues will equally have 50% of the cluster resources. Now another job is submitted to the finance queue. This will result in half of the resources allocated to the finance queue to be allocated to this new job.

Two jobs in the finance queue will now share resources allocated to the finance queue (50% of the total resources) in equal proportions. The sole job in the sales queue will use all resources allocated to this queue, that is, also 50% of the cluster resources.

Support of hierarchical queues

FairScheduler supports hierarchical queues. A queue is hierarchical if it can contain subqueues. All queues descend from the root queue, although in the configuration file, you can omit "root" in the full queue name.Available resources are distributed among the children of the root queue using the fair scheduling concept. Then the children queues distribute the resources assigned to them to their children in the same fashion.

Configuration

To use FairScheduler in YARN, start with assigning the appropriate scheduler class in the yarn-site.xml file:

<property>
  <name>yarn.resourcemanager.scheduler.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>

Set up queues

To set up queues, edit the following XML elements in the fair-scheduler.xml configuration file:

  • <queue> represents a queue. A <queue> element can contain other <queue> elements. Some of the most important properties of this element are listed below:

    • minResources. Specifies the minimum amount of the resources required for the queue (X mb, Y vcores). If this requirement is not met, the scheduler prioritizes this queue over other queues of the same parent when distributing available resources.

    • maxResources. Specifies the maximum amount of the resources that can be allocated to the queue, either in absolute values (X mb, Y vcores) or as a percentage of the cluster resources (X% memory, Y% CPU).

    • weight. Influences the proportion of resources the scheduler will share between the queues. The default weight for a queue is 1. If the queue weight is 2, then this queue should receive approximately twice as many resources as a queue with the default weight.

    • schedulingPolicy. Sets the scheduling policy for the queue. Its value is the name of an object that extends the org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.SchedulingPolicy class. Such objects are: fifo, fair, drf, and other. The default value is fair.

  • <defaultQueueSchedulingPolicy>. Sets the default scheduling policy for queues. The default value is fair. This element is overridden by the schedulingPolicy element in the queue configuration.

  • <queueMaxAppsDefault>. Sets the default upper limit on the number of running applications in a queue. This element is overridden by the maxRunningApps element in the queue configuration.

  • <queuePlacementPolicy>. Contains a list of rules (represented by elements) that tell the scheduler what queues it must use to place the incoming applications. The rules are applied in the order that they are listed. The scheduler follows the first rule it can satisfy. Every rule has the create attribute, which indicates whether the scheduler can create a new queue. The default value of the create attribute is true. If its value is set to false and the rule places applications in a queue that is not configured in the allocation file, the scheduler moves on to the next rule. The possible rules are listed below:

    • specified. The application is placed into the queue that the rule specifies explicitly.

    • user. The application is placed into the queue with the name of the user who submitted it.

    • primaryGroup. The application is placed into the queue with the name of the primary group of the user who submitted it.

    • secondaryGroupExistingQueue. The application is placed into the queue with the name that matches a secondary group of the user who submitted it.

    • nestedUserQueue. The application is placed into the queue with the name of the user under the queue suggested by the nested rule. In other words, this rule must have a nested rule that specifies the parent queue for the queue where the application can be placed to.

    • default. The application is placed into the queue specified in the queue attribute of the default rule. If the queue attribute is not specified, the application is placed into the root.default queue.

    • reject. The application is rejected.

NOTE
For FairScheduler, you can change the scheduler configs (limits, weight, preemption timeout, scheduling policy, etc.) on the fly, without having to run the Reload Scheduler config command. For this, simply edit the allocation file and the scheduler will reload the file with a 10-15 seconds delay after the file has been modified.

Queue configuration example

This example considers two top-level queues (descending directly from root):

  • sales

  • finance

Within the sales queue, there are two subqueues:

  • apac

  • emea

The queues can be set up for a use with FairScheduler as follows:

<allocations>
  <queue name="sales">
      <minResources>10000 mb,0vcores</minResources>
      <maxResources>50000 mb,0vcores</maxResources>
      <weight>2.0</weight>
      <schedulingPolicy>fifo</schedulingPolicy>
      <queue name="emea" />
      <queue name="apac" />
  </queue>
  <queue name="finance">
      <minResources>10000 mb,0vcores</minResources>
      <maxResources>70000 mb,0vcores</maxResources>
      <weight>3.0</weight>
      <schedulingPolicy>fair</schedulingPolicy>
  </queue>
  <queuePlacementPolicy>
      <rule name="specified"   />
      <rule name="primaryGroup" create="false" />
      <rule name="default" queue="finance" />
  </queuePlacementPolicy>
</allocations>
Found a mistake? Seleсt text and press Ctrl+Enter to report it