HBase performance tuning

Default: 4096.

This parameter defines the upper limit of files that an HDFS DataNode can service simultaneously

Set the value to 8192.

This parameter has to be added manually in the Custom hdfs-site.xml section of the HDFS service configuration page

Default: 90000 (in milliseconds).

This parameter defines the timeout before the Master server considers a RegionServer inactive. The default value is irrelevant if the ZooKeeper service retains its default value for the tick time (2 seconds). The ZooKeeper service has its own timeouts for inactive services: minimum 2 tick times and maximum 20 tick times. By default, this means that the ZooKeeper service will wait no longer than 40 seconds before removing the inactive RegionServer ethereal znode while the Master server would be still waiting

Lower the HBase timeout parameter or increase the ZooKeeper tick time parameter — whichever is lower takes effect. Mind the following:

The zookeeper.session.timeout parameter is located in the hbase-site.xml section of the HBase service configuration page, the tick time is defined by the tickTime parameter of the zoo.cfg section in the ZooKeeper service configuration page. Both parameters are specified in milliseconds

Default: 30.

This parameter defines the number of open threads for responses to the user tables incoming queries

Each thread consumes CPU time and RAM, so you need to assess the average query payload size before tweaking this parameter. If the queries carry high payload, then this parameter is better kept at a low value. But if the queries are mostly light, then increasing this parameter value might be a good idea: begin with the number of the CPU cores doubled, and then increase it little by little until you reach the peak performance.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 1.

This parameter defines the minimum number of RegionServers that have to be registered with the Master server before it begins to assign regions to them. The default value of 1 means that the Master server will begin assigning regions to the very first RegionServer that registers with it

If your cluster has many regions (e.g. several thousands), it is recommended to increase this parameter value so that the assignment of the regions would be delayed and hence more evenly distributed. Otherwise, the first registered RegionServer may constitute a bottleneck and slow down the cluster startup and even lead to the loss of access to itself.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: -Xms700m -Xmx9G.

This parameter defines the amount of RAM allocated to the Master server

The default value is enough for most cases, as Master server does not require a lot of RAM to work. Nevertheless, if your cluster is really big and has an enormous number of tables and/or regions, you may need to increase the upper memory limit, since the Master server retains the regions statuses in its memory.

This parameter is located in the hbase-env.sh section of the HBase service configuration page

Default: -Xms700m -Xmx9G.

This parameter defines the size of the RegionServer heap memory, which contains the MemStores and the block cache

Generally, the more this parameter value is the better. Use the following considerations:

This parameter is located in the hbase-regionserver-env.sh section of the HBase service configuration page

Default: 0.4.

This parameter defines the share of the block cache in the RegionServer heap memory

The default value is good for a situation when the table data is often heavily read and it is more or less the same data all the time. If the situation is not exactly that (e.g. the requested data is different all the time, or the data requisition is rather light in an average request), you may want to reduce this parameter value down to 0.2 or even 0.1. The general rule is to keep this value no bigger than it really needs to be. Exact value should be picked based on performance, by trial and error. It also must be balanced with the hbase.regionserver.global.memstore.size parameter value.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 0.4.

This parameter defines the share of the MemStore in the RegionServer heap memory

Specify a value so that it combined with the hfile.block.cache.size parameter value would add up to 0.7 or 0.75.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 0.95.

This parameter defines a threshold percentage in MemStores of a region. When the MemStores of a region are filled with new data up to this threshold, updates are blocked and flushes are forced

Default: 134217728 (128 MB).

This parameter defines the size of data in a MemStore when it is flushed and an HFile is created from its contents. When a MemStore is flushed, all the other MemStores in the same region are flushed too. The less this value is, the more often flushes occur, the bigger is the I/O overhead, the larger is the number of HFiles, and the more often the comaction processes are run

It is recommended to double this parameter value (up to 268435456, meaning 256 MB). If your HBase has a heavy write load, you may want to increase the value up to 536870912 (512 MB), but it is not always possible, especially if you have a rather small RegionServer heap memory size and a very big number of regions and column families.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page. Possible values are 134217728, 268435456, and 536870912

Default: 2.

This parameter defines the number of threads for the flush operations. The more this value is, the more effective the flush process is, but at the same time the more the load on the HDFS is, and potentially the bigger is the number of compactions

You can start with increasing this parameter value up to 4 and then try to increase it more and see if that affects the performance.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 4.

This parameter defines the multiplier of possible MemStore size (defined by the hbase.hregion.memstore.flush.size parameter) extension when the update traffic is very high

It is recommended to increase this value up to 8, if the RAM size allows it. When a MemStore size reaches its maximum defined by the multiplication of this parameter value and the hbase.hregion.memstore.flush.size parameter value, the corresponding column family is blocked for all write operations until the MemStore is flushed.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 1000.

This parameter defines the maximum number of regions on a RegionServer after which region splitting can no longer occur. This is not a limit on the number of the regions itself, but rather a stopping point in creating more regions

The default value of 1000 effectively means no limits, since predominantly there are much less regions per region server in HBase clusters. You can calculate this limit based on your particular situation. For example, your cluster has the following settings:

To find the number of regions, you need to multiply the region server heap memory size by the MemStore share quotient in it (the first two parameters), and then consecutively divide the result by the hbase.hregion.memstore.flush.size parameter value and the number of column families. In this case, we have the following formula:

 
Round the result up if it is fractioned and you will have your optimal number of regions per RegionServer.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 10737418240 (10 GB).

This parameter defines the maximum total size of HFiles in a region before it is split

The default value is generally good. You can increase this value based on performance (for example, up to 20 GB), but not too much: much higher values can negatively affect the compaction process.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 3.

This parameter defines the number of HFiles in a region before they undergo a minor compaction

It is recommended to up this value to 5.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 16.

This parameter defines the maximum number of HFiles in a region before the updates to this region are blocked. The block remains until a minor compaction is forcefully run or the timeout specified by the hbase.hstore.blockingWaitTime parameter expires

If your cluster has a heavy write load and flushes are delayed due to too many HFiles, increase this parameter value.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 90000 (in milliseconds).

This parameter defines the timeout for the updates block of a region due to too many HFiles (see the previous parameter). When this time passes, the block is lifted regardless of whether the minor compaction has completed

Adjust this parameter according to the requirements of your cluster.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 3.

This parameter defines the minimum number of HFiles designated for compaction before such compaction can be executed

It is recommended to increase this value to 5.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 10.

This parameter defines the maximum number of HFiles that can participate in a single compaction operation

The default value is good for most cases. If your cluster has a heavy write load, increase this value.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 1.

This parameter defines the number of threads used for the minor compaction process

If your cluster has a heavy write load and minor compactions are executed too often and take too much time, increase this value. However, it should not exceed half the number of disks available for the HBase.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 1.

This parameter defines the number of threads used for the major compaction process

If your cluster has a heavy write load, you can increase this value, but making it higher than the hbase.regionserver.thread.compaction.small parameter value is not recommended.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 134217728 (128 MB).

This parameter defines the maximum HFile size for it to be automatically designated for compaction

If your cluster has a heavy write load, you may have many small HFiles that, when compacted, form an HFile with a size still under this value (especially in case of a low hbase.hstore.compaction.max parameter value) and require a compaction again. Lowering this parameter value may address this problem: specify a value that is a little lower than the multiplication of an average MemStore flush size and the hbase.hstore.compaction.max parameter value.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 1.2F.

This parameter defines a quotient for the previous parameter (hbase.hstore.compaction.min.size). Values of these two parameters multiplied give the upper limit of the HFile size for it to be designated for compaction

The higher this value is, the more often the compaction occurs, but less HFiles need to be read, and vice versa. Lower value is preferrable for clusters with heavy write load, and higher value is preferrable for clusters with heavy read load. Usually the default value is good; if you choose to adjust it — it is recommended to stay within the 1.0F — 1.4F range.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 0.95.

This parameter defines the size of a WAL file relative to the HDFS block size (which is specified by the dfs.blocksize parameter in the hdfs-site.xml section of the HDFS service configuration page). For example, if the HDFS block size is 134217728 (128 MB), then the WAL file size will be 127506842 (approximately 121.6 MB)

It is recommended to leave this parameter as is in all cases.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 32.

This parameter defines the maximum number of WAL files in a region server

You need to calculate the value for this parameter yourself by doing the following:

If your cluster has a heavy write load, when the number of WAL files reaches its limit, region servers might require to flush all MemStores until all WAL files are archived and there is enough space for writing operations. This is called flush storm, and can be recognized by the following line in the log:

If this happens, increase the parameter value.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 2097152 (2 MB).

This parameter defines the size of a chunk in the MemStore Allocation Buffer

Double or quadruple this parameter value if you use large-sized records.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 2097152 (2 MB).

This parameter defines the size of the write buffer. The bigger this value is, the more memory is used on both server and client sides, but the less the number of RPCs is

It is recommended to set this value to 8388608 (8 MB). To estimate the size of the server memory used, multiply this parameter value by the value of the hbase.regionserver.handler.count parameter.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 100.

This parameter defines the duration of the client pause. It is used as a timeout before retrying a failed operation

It is recommended to significantly reduce this value (e.g. down to 3 or 5) in order to avoid data loss.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 15.

This parameter defines the maximum number of retries of a failed operation

It is recommended to reduce this value to 3 or 5 in order to avoid data loss.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 300000 (in milliseconds; 5 minutes).

This parameter defines the timeout for all RPCs that the client sends to the HBase

If your client requests tend to take heavy toll on RAM, increase this parameter value. Avoid increasing it too much, because the client might wait for too long and then fail anyway.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 60000 (in milliseconds; 1 minute).

This parameter defines the timeout for the RPCs related to scanning operations

This parameter value becomes useless if it is set higher than the hbase.rpc.timeout parameter value, because the operation is aborted after whichever timeout expires first.

This parameter is located in the hbase-site.xml section of the HBase service configuration page

Default: 2147483647.

This parameter defines the maximum number of rows that the client scanner can fetch from the HBase in one sitting

If the value is too high, the scanner may become overloaded by fetching and transferring too much data to the client, or even exceed the timeout, failing the job; also this will lead to using too much heap memory even in case of success. A value too low will lead to ineffective use of cluster resources, causing idling. This parameter value must be chosen carefully according to the characteristics of disks, RAM, CPUs, and the average read load.

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page

Default: 2097152 (2 MB).

This parameter defines the maximum number of bytes returned when calling a next scanner method. If a single row size is larger than this value, it is still returned in its entirety

The default value is good for 1G Ethernet networks. If the network capacity in your cluster is better, increase this value, but mind the following:

This parameter has to be added manually in the Custom hbase-site.xml section of the HBase service configuration page