Rack awareness

Rack awareness in Hadoop is an HDFS feature that provides additional fault tolerance by taking network topology into account when placing data blocks. If turned on, HDFS will place one block replica on a different rack. This insures availability of data in case of a network switch failure or other issues.

To configure rack awareness via ADCM:

  1. On the Clusters page, select the desired cluster.

  2. Go to the Services tab and click at HDFS.

  3. Toggle the Show advanced option and find the Topology script and Topology data fields.

  4. In the Topology script, add a script describing your network topology.

    Example topology script
    #!/bin/bash
    
    # Adjust/Add the property "net.topology.script.file.name"
    # to core-site.xml with the "absolute" path the this
    # file.  ENSURE the file is "executable".
    
    # Supply appropriate rack prefix
    RACK_PREFIX=default
    
    # To test, supply a hostname as script input:
    if [ $# -gt 0 ]; then
    
    CTL_FILE=${CTL_FILE:-"topology.data"}
    
    HADOOP_CONF=${HADOOP_CONF:-"/etc/hadoop/conf"}
    
    if [ ! -f ${HADOOP_CONF}/${CTL_FILE} ]; then
      echo -n "/$RACK_PREFIX/rack "
      exit 0
    fi
    
    while [ $# -gt 0 ] ; do
      nodeArg=$1
      exec< ${HADOOP_CONF}/${CTL_FILE}
      result=""
      while read line ; do
        ar=( $line )
        if [ "${ar[0]}" = "$nodeArg" ] ; then
          result="${ar[1]}"
        fi
      done
      shift
      if [ -z "$result" ] ; then
        echo -n "/$RACK_PREFIX/rack "
      else
        echo -n "/$RACK_PREFIX/rack_$result "
      fi
    done
    
    else
      echo -n "/$RACK_PREFIX/rack "
    fi

    You can find additional script examples in the Rack Awareness article.

  5. In the Topology data field, list racks IPs and their corresponding IDs as follows:

    Example topology data
    # This file should be:
    #  - Placed in the /etc/hadoop/conf directory
    #    - On the Namenode (and backups IE: HA, Failover, etc)
    #    - On the Job Tracker OR Resource Manager (and any Failover JT's/RM's)
    # This file should be placed in the /etc/hadoop/conf directory.
    
    # Add Hostnames to this file. Format <host ip> <rack_location>
    127.0.0.1 01
    127.0.0.2 02
    127.0.0.3 03
  6. Confirm changes to HDFS configuration by clicking Save.

  7. In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to true and click Run.

  8. Restart the YARN service the same way.

Topology script and data fields in ADCM
Topology script and data fields in ADCM

To check if rack awareness has been successfully configured, you can grep the NameNode logs or use the fsck command.

The NameNode log will contain messages similar to these:

2023-10-23 13:09:52,474 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default/rack_03/127.0.0.1:1019
2023-10-23 13:09:52,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default/rack_01/127.0.0.2:1019
2023-10-23 13:09:52,571 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default/rack_02/127.0.0.3:1019

The fsck command output will display the number of racks:

Status: HEALTHY
 Number of data-nodes:  3
 Number of racks:               3
Found a mistake? Seleсt text and press Ctrl+Enter to report it