Rack awareness
Rack awareness in Hadoop is an HDFS feature that provides additional fault tolerance by taking network topology into account when placing data blocks. If turned on, HDFS will place one block replica on a different rack. This insures availability of data in case of a network switch failure or other issues.
To configure rack awareness via ADCM:
-
On the Clusters page, select the desired cluster.
-
Go to the Services tab and click at HDFS.
-
Toggle the Show advanced option and find the Topology script and Topology data fields.
-
In the Topology script, add a script describing your network topology.
Example topology script#!/bin/bash # Adjust/Add the property "net.topology.script.file.name" # to core-site.xml with the "absolute" path the this # file. ENSURE the file is "executable". # Supply appropriate rack prefix RACK_PREFIX=default # To test, supply a hostname as script input: if [ $# -gt 0 ]; then CTL_FILE=${CTL_FILE:-"topology.data"} HADOOP_CONF=${HADOOP_CONF:-"/etc/hadoop/conf"} if [ ! -f ${HADOOP_CONF}/${CTL_FILE} ]; then echo -n "/$RACK_PREFIX/rack " exit 0 fi while [ $# -gt 0 ] ; do nodeArg=$1 exec< ${HADOOP_CONF}/${CTL_FILE} result="" while read line ; do ar=( $line ) if [ "${ar[0]}" = "$nodeArg" ] ; then result="${ar[1]}" fi done shift if [ -z "$result" ] ; then echo -n "/$RACK_PREFIX/rack " else echo -n "/$RACK_PREFIX/rack_$result " fi done else echo -n "/$RACK_PREFIX/rack " fi
You can find additional script examples in the Rack Awareness article.
-
In the Topology data field, list racks IPs and their corresponding IDs as follows:
Example topology data# This file should be: # - Placed in the /etc/hadoop/conf directory # - On the Namenode (and backups IE: HA, Failover, etc) # - On the Job Tracker OR Resource Manager (and any Failover JT's/RM's) # This file should be placed in the /etc/hadoop/conf directory. # Add Hostnames to this file. Format <host ip> <rack_location> 127.0.0.1 01 127.0.0.2 02 127.0.0.3 03
-
Confirm changes to HDFS configuration by clicking Save.
-
In the Actions drop-down menu, select Restart, make sure the Apply configs from ADCM option is set to
true
and click Run. -
Restart the YARN service the same way.
To check if rack awareness has been successfully configured, you can grep the NameNode logs or use the fsck command.
The NameNode log will contain messages similar to these:
2023-10-23 13:09:52,474 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default/rack_03/127.0.0.1:1019 2023-10-23 13:09:52,569 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default/rack_01/127.0.0.2:1019 2023-10-23 13:09:52,571 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default/rack_02/127.0.0.3:1019
The fsck
command output will display the number of racks:
Status: HEALTHY Number of data-nodes: 3 Number of racks: 3