HBase provider

Elena Kostyuchenko

Overview

The HBase provider enables DAGs to interact with HBase. It provides hooks, operators, and sensors for performing administrative and data operations on HBase tables as well as backup and restore tasks.

The provider can be used to:

  • create and delete tables;

  • insert, update, retrieve, and delete rows;

  • scan tables;

  • execute batch operations;

  • create and restore backups;

  • manage backup sets;

  • monitor table and row availability.

The provider follows the standard Airflow provider architecture, where operators implement DAG tasks, hooks provide low-level communication with external systems, and sensors wait until a specified condition becomes true.

Unlike generic Airflow providers, the HBase provider supports two execution mechanisms:

  • Thrift2 API for online data operations;

  • HBase CLI utilities for backup and restore operations.

Architecture

The provider is organized into several Python modules:

  • Hooks

    This module implements the communication layer. It contains classes that establish connections to HBase and expose a Python interface for performing HBase operations. Two hook implementations are provided:

    • HBaseThriftHook — communicates with the HBase Thrift2 Server component and is responsible for table management, data manipulation, and metadata operations.

    • HBaseCLIHook — executes native HBase client commands and is used exclusively for backup and restore workflows.

  • Operators

    This module contains Airflow tasks that represent individual HBase operations. Each operator performs parameter validation, initializes the appropriate hook, invokes the required hook method, and returns the execution result to Airflow.

  • Sensors

    The module provides Airflow sensors that periodically check the state of HBase resources. Unlike operators, sensors do not modify data. They repeatedly execute lightweight requests until a specified condition is satisfied or the configured timeout expires. Typical use cases include waiting for a table to be created before loading data or waiting for a specific row to appear as a result of another workflow.

  • Utils

    The module contains helper classes shared by multiple provider components. These include enumerations that describe backup types and operator behavior, constants used throughout the provider, retry helpers, and other utility functions that support the implementation without being directly exposed as Airflow tasks.

These modules work as a single execution pipeline. During DAG execution, Airflow creates an operator instance, which selects the appropriate hook according to the requested operation. The hook establishes communication with HBase through either the Thrift2 protocol or the HBase CLI, executes the requested operation, converts the response into Python objects where necessary, and returns the result to the operator. The operator then reports the task status back to the Airflow scheduler.

Requirements

The requirements depend on the functionality used.

If you already have a working ADH cluster with necessary components, you can share its hosts with your ADO cluster by using shared hosts.

Data operations

The following requirements apply to data operators and sensors:

  • HBase Thrift2 Server must be running in the ADH cluster and be accessible from Airflow workers;

  • an HBase connection must be configured in Airflow.

Backup and restore

Backup and restore operations require the following components:

  • HBase Client installed on Airflow hosts;

  • HDFS access configured for Airflow workers, including the required permissions;

  • network connectivity between Airflow workers and HDFS.

Configuration

The HBase provider uses the hbase Airflow connection type to store the parameters required for communicating with an HBase cluster. Every operator and sensor accepts the hbase_conn_id parameter, which references the configured Airflow connection. If no connection identifier is specified explicitly, the provider uses the hbase_thrift2 connection by default.

The provider supports two connection types:

  • hbase is the native connection type implemented by the provider. It is recommended for all HBase workloads because it supports the complete set of provider-specific configuration parameters, including retry policies, connection pooling, SSL/TLS settings, and CLI configuration.

  • generic can be used to connect to arbitrary Thrift servers. This connection type is primarily intended for compatibility scenarios and does not expose the HBase-specific configuration options provided by the native connection type.

Standard configuration

Below is a list of standard Airflow connection fields used by the provider. The remaining configuration is specified in the Extra field as a JSON document.

Field Description

Connection type

Must be set to hbase

Host

Specifies the hostname or IP address of the HBase Thrift2 Server

Port

Specifies the Thrift2 service port. If omitted, the default HBase Thrift2 Server port (9090) is used

Extra

Contains provider-specific configuration parameters

Configuration example for a basic Thrift2 connection with no authentication:

  • Connection Type: hbase

  • Host: hbase-server.example.com

  • Port: 9090

  • Extra:

    {
  "use_http": false
}

Extra parameters

The following parameters control the behavior of the Thrift2 client.

Parameter Description Default value

timeout

Defines the timeout for a Thrift request in milliseconds

30000

namespace

Specifies the default HBase namespace used by operators that do not explicitly define one

default

use_http

Enables HTTP transport instead of the default binary socket transport

false

retry_max_attempts

Defines the maximum number of retry attempts after a connection failure

3

retry_delay

Defines the initial delay between retry attempts in seconds

1.0

retry_backoff_factor

Defines the multiplier used to increase the delay after each unsuccessful retry attempt

2.0

Connection pooling

By default, the provider uses a single Thrift connection for each task execution. This strategy, implemented by Thrift2Strategy, is appropriate for most workloads and minimizes resource consumption.

For production environments that execute large batch operations or multiple concurrent HBase tasks, the provider also supports connection pooling through PooledThrift2Strategy. Instead of creating a new connection for every request, the provider maintains a pool of reusable Thrift connections that can be shared between operations. Reusing established connections reduces connection overhead and significantly improves throughput during intensive workloads.

Connection pooling is disabled by default. To enable it, add the connection_pool section to the connection Extra field.

{
  "connection_pool": {
    "enabled": true,
    "size": 10,
    "timeout": 30
  }
}

The available connection pool parameters are described below.

Parameter Description Default value

enabled

Enables the connection pool

false

size

Specifies the maximum number of Thrift connections maintained in the pool

10

timeout

Defines the maximum time, in seconds, that a task waits for an available connection before reporting an error

30
TIP
Connection pooling is recommended for production deployments that perform batch processing or execute multiple HBase tasks concurrently.

SSL/TLS

The provider supports encrypted communication with HBase Thrift2 Server. SSL/TLS parameters are specified directly in the connection Extra field.

Parameter Description Default value

ca_certs

Specifies the path to the trusted CA certificate set

 — 

validate

Defines whether the server certificate is validated during connection establishment

true

use_http

Must be set to true when HBase Thrift2 Server is configured to use the HTTP with TLS

false

The following example enables SSL connection with server certificate validation:

{
  "ca_certs": "/etc/ssl/hbase_certs.pem",
  "validate": true,
  "use_http": true
}

CLI parameters

Backup and restore operators execute the native HBase Backup CLI on a Airflow worker instead of communicating with HBase through Thrift2. The provider can use the following optional parameters to locate the required runtime environment.

Parameter Description Default value

java_home

Specifies the Java installation used to execute HBase CLI commands

/usr/lib/jvm/java-arenadata-openjdk-8

hbase_home

Specifies the HBase installation directory that contains the client utilities

/usr/lib/hbase

In deployments where the HBase Client component is installed from ADH using shared hosts, these parameters normally do not require modification because they already point to the standard installation directories managed by ADH.

Hooks

Hooks implement the communication layer of the provider. The provider contains two hook implementations because HBase itself exposes different interfaces for different categories of operations.

HBaseThriftHook

HBaseThriftHook is the primary communication interface used by the provider. It establishes a connection to HBase Thrift2 Server and exposes methods for performing table administration, data manipulation, metadata retrieval, and batch processing. All data operators and sensors rely on this hook to communicate with the HBase cluster.

The hook provides methods for the most common HBase operations, including table management, row manipulation, table scanning, and batch processing.

Method Description

table_exists()

Checks whether the specified table exists before performing subsequent operations

create_table()

Creates a new table with the specified column families

delete_table()

Deletes an existing table

put_row()

Creates a new row or updates an existing one

get_row()

Retrieves a single row by its row key

delete_row()

Deletes an entire row or selected columns within the row

scan_table()

Scans a range of rows and returns matching records

batch_put_rows()

Writes multiple rows using configurable batch sizes and optional parallel execution

batch_get_rows()

Retrieves multiple rows within a single batch operation

batch_delete_rows()

Deletes multiple rows in batches

HBaseCLIHook

The hook provides methods that correspond to the HBase Backup CLI operations.

Method Description

create_full_backup()

Creates a full backup of one or more HBase tables

create_incremental_backup()

Creates an incremental backup based on a previous backup image

restore_backup()

Restores data from an existing backup

get_backup_history()

Retrieves the history of completed backup operations

describe_backup()

Displays detailed information about a backup image

create_backup_set()

Creates or modifies a backup set used by backup operations

list_backup_sets()

Returns the configured backup sets

execute_command()

Executes an arbitrary HBase CLI command and returns its output

Operators

Operators implement the executable tasks that can be used directly in Airflow DAGs. Each operator encapsulates a specific HBase operation, validates the supplied parameters, initializes the appropriate hook, executes the requested action, and returns the result to the Airflow task context when applicable.

Table management operators

Table management operators perform administrative operations on HBase tables. The provider includes the table management operators listed below.

Operator Description

HBaseCreateTableOperator

Creates an HBase table and optionally validates whether the table already exists before creation

HBaseDeleteTableOperator

Deletes an existing table and optionally validates that the table exists before execution

Data operators

Data operators perform read and write operations through the HBase Thrift2 Server API. They use HBaseThriftHook, which automatically selects either a dedicated Thrift connection or a pooled connection depending on the configured Airflow connection.

The provider includes the following operators for data manipulation.

Operator Description

HBasePutOperator

Inserts or updates a single row

HBaseBatchPutOperator

Writes multiple rows using configurable batch processing and optional parallel execution

HBaseBatchGetOperator

Retrieves multiple rows in a single operation

HBaseScanOperator

Scans table contents within a specified row range

Backup and restore operators

Backup operators provide access to the HBase Backup subsystem through HBaseCLIHook. The provider includes the backup management operators listed below.

Operator Description

HBaseCreateBackupOperator

Creates full or incremental HBase backups

HBaseBackupHistoryOperator

Retrieves the history of completed backup operations

HBaseRestoreOperator

Restores tables from an existing backup

HBaseBackupSetOperator

Creates and manages HBase backup sets

Sensors

Sensors allow DAGs to wait until a particular condition in HBase becomes true before continuing execution. All sensor parameters that identify HBase resources, such as table names and row keys, are defined as Airflow template fields. This allows sensors to monitor dynamically generated resources whose names are determined during DAG execution.

The provider includes the following sensors.

Sensor Description

HBaseTableSensor

Waits until a specified HBase table becomes available. During every polling cycle, the sensor calls the table_exists() method of HBaseThriftHook to determine whether the table has been created

HBaseRowSensor

Waits until a specific row exists in an HBase table. During each polling cycle, the sensor retrieves the row using HBaseThriftHook.get_row() and considers the condition satisfied when the requested row contains data

Utility classes

The provider defines several enumeration classes that provide strongly typed configuration values for HBase operators. Using these enumerations instead of string literals improves the readability of DAG definitions, validates parameter values before execution, and reduces the likelihood of configuration errors.

The provider defines the following utility classes.

Type Description

BackupType

Specifies whether a backup operation is performed as a full or incremental backup

BackupSetAction

Specifies the action performed by HBaseBackupSetOperator, such as creating or listing backup sets

IfExistsAction

Controls how table creation operators behave when the target table already exists

IfNotExistsAction

Controls how table deletion operators behave when the target table does not exist

Use the provider in a DAG

To use the HBase provider, import the required operators or sensors from the provider package and configure an HBase connection.

The examples in this section demonstrate the most common provider workflows. Complete example DAGs are available in the provider’s repository.

Basic data flow

A typical HBase workflow consists of creating a table, writing data, and reading the stored records.

Use HBaseCreateTableOperator to create an HBase table and define its column families:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseCreateTableOperator

create_table = HBaseCreateTableOperator(
    task_id="create_table",
    table_name="users", (1)
    families={ (2)
        "profile": {},
        "contacts": {},
    },
    hbase_conn_id="hbase_default",
)
1 Name of the HBase table.
2 Column families created together with the table.

Use HBasePutOperator to insert or update a single row:

from airflow.providers.arenadata.hbase.operators.hbase import HBasePutOperator

insert_user = HBasePutOperator(
    task_id="insert_user",
    table_name="users",
    row_key="user_001", (1)
    data={ (2)
        "profile:name": "Alice",
        "profile:age": "30",
        "contacts:email": "alice@example.com",
    },
    hbase_conn_id="hbase_default",
)
1 Unique row identifier.
2 Column names include the column family prefix.

Use HBaseScanOperator to retrieve rows from a table:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseScanOperator

scan_users = HBaseScanOperator(
    task_id="scan_users",
    table_name="users",
    row_start="user_000",
    row_stop="user_999",
    columns=[
        "profile:name",
        "contacts:email",
    ],
    limit=100,
    hbase_conn_id="hbase_default",
)

The operators can be combined into a simple data ingestion workflow.

Batch processing

For large datasets, use the batch operators to reduce the number of Thrift requests and improve throughput.

HBaseBatchPutOperator writes multiple rows using configurable batch sizes.

from airflow.providers.arenadata.hbase.operators.hbase import HBaseBatchPutOperator

batch_insert = HBaseBatchPutOperator(
    task_id="batch_insert",
    table_name="users",
    rows=rows,
    batch_size=200, (1)
    max_workers=4, (2)
    hbase_conn_id="hbase_default",
)
1 Number of rows written in a single request.
2 Number of concurrent workers used for batch processing.

When using multiple workers, enable connection pooling in the HBase connection for improved throughput.

Use HBaseBatchGetOperator to retrieve several rows in a single operation:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseBatchGetOperator

batch_get = HBaseBatchGetOperator(
    task_id="batch_get",
    table_name="users",
    row_keys=[
        "user_001",
        "user_002",
        "user_003",
    ],
    columns=[
        "profile:name",
        "contacts:email",
    ],
    hbase_conn_id="hbase_default",
)

Backup workflow

The provider integrates with the HBase Backup CLI to manage backup sets and perform backup and restore operations.

Before creating backups, prepare the HDFS backup directory:

$ hdfs dfs -mkdir -p /hbase/backup
$ hdfs dfs -chmod 777 /hbase/backup

The chmod 777 access is used only for testing purposes and not suited for production.

Enable backup support in the HBase configuration:

hbase.backup.enable=true

A backup set groups one or more tables into a reusable backup definition.

Create a backup set:

from airflow.providers.arenadata.hbase.operators.hbase import (
    BackupSetAction,
    HBaseBackupSetOperator,
)

create_backup_set = HBaseBackupSetOperator(
    task_id="create_backup_set",
    action=BackupSetAction.ADD,
    backup_set_name="production_tables",
    tables=[
        "users",
        "orders",
    ],
    hbase_conn_id="hbase_default",
)

Use HBaseCreateBackupOperator to create either a full or incremental backup.

Create a backup:

from airflow.providers.arenadata.hbase.operators.hbase import (
    BackupType,
    HBaseCreateBackupOperator,
)

create_backup = HBaseCreateBackupOperator(
    task_id="create_backup",
    backup_type=BackupType.FULL,
    backup_path="hdfs:///hbase/backup", (1)
    backup_set_name="production_tables",
    workers=2,
    hbase_conn_id="hbase_default",
)
1 HDFS location where the backup is stored.

The operator pushes the generated backup identifier to XCom for later restore operations.

Use HBaseBackupHistoryOperator to verify completed backup operations:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseBackupHistoryOperator

backup_history = HBaseBackupHistoryOperator(
    task_id="backup_history",
    backup_set_name="production_tables",
    hbase_conn_id="hbase_default",
)

Restore data using the backup identifier returned by the backup task:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseRestoreOperator

restore_backup = HBaseRestoreOperator(
    task_id="restore_backup",
    backup_path="hdfs:///hbase/backup",
    backup_id="{{ ti.xcom_pull(task_ids='create_backup') }}",
    tables=["users"],
    overwrite=True,
    hbase_conn_id="hbase_default",
)

Monitoring

Use HBaseTableSensor to wait until a table becomes available:

from airflow.providers.arenadata.hbase.sensors.hbase import HBaseTableSensor

wait_for_table = HBaseTableSensor(
    task_id="wait_for_table",
    table_name="users",
    timeout=300,
    poke_interval=30,
    hbase_conn_id="hbase_default",
)

This sensor is typically used when another DAG or external application creates the table.

Use HBaseRowSensor to wait until a specific row has been written:

from airflow.providers.arenadata.hbase.sensors.hbase import HBaseRowSensor

wait_for_row = HBaseRowSensor(
    task_id="wait_for_row",
    table_name="users",
    row_key="user_001",
    timeout=600,
    poke_interval=60,
    hbase_conn_id="hbase_default",
)

This sensor is commonly used to wait for marker rows that indicate data ingestion has completed.

Found a mistake? Seleсt text and press Ctrl+Enter to report it