Use medium-sized objects in HBase

Vladimir Adamenkov

Overview

Introduction

Medium-sized object (MOB) is a type of data stored in HBase tables with the order of size from 100 KB to 10 MB or greater. Examples of such objects include images, documents, archives, and other binary files.

Technically, HBase is able to deal with values of any size, but objects larger than 100 KB can and will negatively affect the performance during MemStore flushes: splits and compactions will inevitably lead to write amplification. To avoid that, you can designate certain column families containing large objects in their cells as MOB column families. Table cells supposed to store MOBs only contain the references to the actual files. Those, in turn, are stored in special MOB HFiles which can be located at any specified location (HDFS or not).

Regular column family cells contain data as it is, regardless of size. MOB column family cells discriminate the data by its size according to the specified threshold: smaller data is stored as is, larger data is stored as MOBs.

Architecture

Initially, all data to be written into HBase for the first time is stored in the write-ahead log (WAL) and the MemStore. When a MemStore corresponding to a MOB column family flushes, it creates two HFiles. Cell values smaller than the specified threshold are written into a regular HFile. Cell values greater than (or equal to) the specified threshold are written into a special MOB HFile, and at the same time the references to the locations of those values are written into the corresponding column cells (MOB reference cells) of the regular HFile. After a MemStore is flushed and its corresponding regular HFile is closed, the region server appends the metadata to it. This metadata contains the list of all MOB HFiles referenced by the cells in the regular HFile.

The value of a MOB reference cell consists of two parts: the size of the MOB and the name of the corresponding MOB HFile. It also has two tags: the first one flags the cell as the reference one, and the second one contains the namespace and the table name for the MOB HFile. These tags can help when searching specifically for the MOB cells and to optimize the way the MOB system looks for values in MOB HFiles, especially after several HBase snapshot operations. They cannot be sent over using RPCs by default as they are only available on the HBase servers side.

Upon creation or compaction the MOB HFiles are put into a specific location in the HBase root directory, according to their namespace, table name, MOB logical region, and column family name. An example of path to such location: /hbase/mobdir/data/default/table_with_mobs/463b0c18d2eb1a65b4b940837d6d0a08/mob_column/, where:

  • hbase — HBase root directory;

  • mobdir/data — default subfolder for MOB HFiles;

  • default — namespace title;

  • table_with_mobs — table name;

  • 463b0c18d2eb1a65b4b940837d6d0a08 — logical region ID;

  • mob_column — column family name.

Maintenance

Maintenance of the MOB HFiles is performed by chore routines (chores) on the HBase master server and on the region servers. They are responsible for enforcing the column families TTLs and for compacting the HFiles. When a MOB HFile becomes obsolete after a compaction procedure, a chore moves it to the archive exactly like it does with regular HFiles. The MOB region of a table is independent from the regular regions, so it can share their archive storage location. An example of path to such location: /hbase/archive/data/default/table_with_mobs/463b0c18d2eb1a65b4b940837d6d0a08/mob_column/.

If there exists a snapshot of the table with MOB feature turned on (or a clone of such), the MOB HFiles will remain in the archive as long as those snapshots require them to. The cleaning chores delete the MOB HFiles from the archive when they are no longer needed.

Compaction

During the flush of a MemStore associated with a MOB column family, the HBase writes the values with sizes over the threshold into the MOB HFile. During the region compaction procedure the region server overwrites the data in the regular HFiles, but keeps the references to the MOB HFiles intact. The client can transparently access the original reference data, because the region server internal routines keep track of it and pull the required values from MOB HFiles. This allows you to accumulate a big amount of MOB HFiles without affecting the average speed of obtaining the MOB cells values. Compaction of MOB HFiles is required significatly less often.

If the MOB cells values are freqently deleted or modified, it may cause excessive consumption of space. In order to delete a MOB HFile and release space it is necessary to make sure that no cells have references to it, and its current values must be written into a new MOB HFile. Also, if the filesystem restricts the maximum number of files it can contain (e.g. HDFS), then even if the MOB cells values are not modified or deleted at all, the HFiles will pile up and require to be compacted.

Usage

Configure MOB columns

In HBase shell, two properties define the MOB designation of a column family:

  • IS_MOB — boolean. Defines whether the values in the column can be considered MOBs.

  • MOB_THRESHOLD — integer. Defines the number of bytes starting at which the value is considered a MOB.

The MOB_THRESHOLD property is optional. If left unspecified, it defaults to 102400 (100 KB).

HBase shell example for a command to create a new table with a column family designated to store MOBs:

create 'table_with_mobs', {NAME => 'mob_column', IS_MOB => true, MOB_THRESHOLD => 204800}

In this example the table_with_mobs table is created with the mob_column column family intended to store MOBs. All values larger than 200 KB will be treated as MOBs in this column family.

HBase shell example for a command to add a new column family to this table designated to store MOBs:

alter 'table_with_mobs', {NAME => 'anoter_mob_column', IS_MOB => true, MOB_THRESHOLD => 153600}

In this example the column family anoter_mob_column is added to the table_with_mobs table. It is also intended to store MOBs, and its threshold is set at 150 KB.

Java API example of a code snippet aimed to configure a MOB column family:

HColumnDescriptor mob_column = new HColumnDescriptor("f");
mob_column.setMobEnabled(true);
mob_column.setMobThreshold(102400L);

In this example the mob_column column family is created intended to store MOBs with the default threshold of 100 KB.

Configure a compaction policy

A specific chore on the HBase master server periodically launches a major compaction procedure for both regular and MOB HFiles. By default it starts once a week, but that can be changed:

  1. Go to ADCM UI and select your ADH cluster.

  2. Navigate to Services → HBase → Primary configuration and toggle Show advanced.

  3. Open the Custom hbase-site.xml section and click Add property.

  4. For the field name, enter hbase.mob.compaction.chore.period. For the field value, enter a period length in seconds. For example, day is 86400 and month is 2592000. Click Apply.

  5. Save the configuration by clicking Save → Create and restart the service by clicking Actions → Reconfig and graceful restart.

By default this chore tries to make all regions perform compactions simultaneously and maximize the load on the cluster. You can even the load by setting the limit of concurrently running compaction jobs on different regions:

  1. Go to ADCM UI and select your ADH cluster.

  2. Navigate to Services → HBase → Primary configuration and toggle Show advanced.

  3. Open the Custom hbase-site.xml section and click Add property.

  4. For the field name, enter hbase.mob.major.compaction.region.batch.size. For the field value, enter an integer limit. Zero means the default behavior (all regions working at once). Click Apply.

  5. Save the configuration by clicking Save → Create and restart the service by clicking Actions → Reconfig and graceful restart.

Configure an archiving policy

A specific chore on the HBase master server periodically launches an archivation procedure for MOB HFiles when they are no longer needed. A MOB HFile is sent to the archive in two cases:

  • it is older than its column family’s TTL;

  • no regular HFiles in all regions have references to it.

By default this chore runs daily. This is good for most cases. However, if your MOB column families are updated rather often or have short TTLs, which causes frequent compactions, you may want to adjust the schedule of this chore:

  1. Go to ADCM UI and select your ADH cluster.

  2. Navigate to Services → HBase → Primary configuration and toggle Show advanced.

  3. Open the Custom hbase-site.xml section and click Add property.

  4. For the field name, enter hbase.master.mob.cleaner.period. For the field value, enter the period value in seconds. Click Apply.

  5. Save the configuration by clicking Save → Create and restart the service by clicking Actions → Reconfig and graceful restart.

Optimization

Configure MOB cache

The MOB HFiles are not always kept open because there can be a very large number of them compared to regular HFiles. The reader cache containing MOB HFiles keeps the most recently used ones (LRU cache). You can configure the following parameters:

  • Cache size — limit to the number of opened file handlers to cache. Big cache size value may improve the performance by reducing the freqency of opening and closing the files. Too big cache size value may cause an error.

  • Eviction period — timeout for an unused opened file to be closed and removed from the cache.

  • Eviction to retention ratio — a factor between 0 and 1 determining how many files must be left in the cache when the cache size is exceeded. Upon reaching the cache size limit, the total number of cached file handlers is multiplied by this factor and the resulting number of most recently used remain in the cache; others are evicted.

To add these parameters, do the following:

  1. Go to ADCM UI and select your ADH cluster.

  2. Navigate to Services → HBase → Primary configuration and toggle Show advanced.

  3. Open the Custom hbase-site.xml section and click Add property.

    Use the following values to configure one or more of the required properties:

    • Cache size — for the field name, enter hbase.mob.file.cache.size. For the field value, enter an integer limit. Default is 1000.

    • Eviction period — for the field name, enter hbase.mob.cache.evict.period. For the field value, enter an integer duration in seconds. Default is 3600 (one hour).

    • Eviction to retention ratio — for the field name, enter hbase.mob.cache.evict.remain.ratio. For the field value, enter a float factor between 0.0f and 1.0f. Default is 0.5f.

  4. Click Apply.

  5. Save the configuration by clicking Save → Create and restart the service by clicking Actions → Reconfig and graceful restart.

Manual compaction

If you don’t want to wait for the compaction chore to start on schedule, you can launch the compaction procedure manually. Go to the HBase shell and use one of the commands of the following kind:

major_compact 'table_with_mobs'
major_compact 'table_with_mobs', 'mob_column'

The first command will make all the MOB column families of the table to undergo the compaction process. The second one will only process the specified column family.

Example of the same request in the Java API:

Admin.majorCompact(table_with_mobs, mob_column);
Found a mistake? Seleсt text and press Ctrl+Enter to report it