HBase provider
Overview
The HBase provider enables DAGs to interact with HBase. It provides hooks, operators, and sensors for performing administrative and data operations on HBase tables as well as backup and restore tasks.
The provider can be used to:
-
create and delete tables;
-
insert, update, retrieve, and delete rows;
-
scan tables;
-
execute batch operations;
-
create and restore backups;
-
manage backup sets;
-
monitor table and row availability;
-
trigger DAGs based on changes to HBase tables using Airflow datasets (assets in Airflow 3).
The provider follows the standard Airflow provider architecture, where operators implement DAG tasks, hooks provide low-level communication with external systems, and sensors wait until a specified condition becomes true.
The provider supports Airflow 3 and maintains backward compatibility with Airflow 2.
Unlike generic Airflow providers, the HBase provider supports two execution mechanisms:
-
Thrift2 API for online data operations;
-
HBase CLI utilities for backup and restore operations.
Architecture
The provider is organized into several Python modules:
-
Hooks
This module implements the communication layer. It contains classes that establish connections to HBase and expose a Python interface for performing HBase operations. Two hook implementations are provided:
-
HBaseThriftHook— communicates with the HBase Thrift2 Server component and is responsible for table management, data manipulation, and metadata operations. -
HBaseCLIHook— executes native HBase client commands and is used for backup and restore workflows.
-
-
Operators
This module contains Airflow tasks that represent individual HBase operations. Each operator performs parameter validation, initializes the appropriate hook, invokes the required hook method, and returns the execution result to Airflow.
-
Sensors
The module provides Airflow sensors that periodically check the state of HBase resources. Unlike operators, sensors do not modify data. They repeatedly execute lightweight requests until a specified condition is satisfied or the configured timeout expires. Typical use cases include waiting for a table to be created before loading data or waiting for a specific row to appear as a result of another workflow.
-
Datasets
This module provides the
hbase_table_dataset()function for creating Airflow datasets that represent HBase tables. In Airflow 3, the function creates an Asset instead. Datasets (assets) can be used to establish dependencies between DAGs and trigger downstream workflows when data in an HBase table is updated. -
Utils
The module contains helper classes and functions shared by multiple provider components. These include enumerations that describe backup types and operator behavior, constants used throughout the provider, retry helpers, and data conversion utilities.
These modules work as a single execution pipeline. During DAG execution, Airflow creates an operator instance, which selects the appropriate hook according to the requested operation. The hook establishes communication with HBase through either the Thrift2 protocol or the HBase CLI, executes the requested operation, converts the response into Python objects where necessary, and returns the result to the operator. The operator then reports the task status back to the Airflow scheduler.
The provider uses a compatibility module to support the different Airflow APIs. In Airflow 3, it uses classes and execution context from the Airflow SDK. In Airflow 2, it uses the corresponding classes from the legacy Airflow modules.
Requirements
The requirements depend on the functionality used.
If you already have a working ADH cluster with the necessary components, you can share its hosts with your ADO cluster by using shared hosts.
Data operations
The following requirements apply to data operators and sensors:
-
HBase Thrift2 Server must be running in the ADH cluster and be accessible from Airflow workers;
-
an HBase connection must be configured in Airflow.
Airflow versions compatibility
The HBase provider supports Airflow 3 and Airflow 2. The provider selects the appropriate Airflow classes according to the installed Airflow version.
In Airflow 3, the provider uses the Airflow SDK for the following components:
-
DAGdefinitions; -
base operator and sensor classes;
-
connection and variable access;
-
execution context.
In Airflow 2, the provider continues to use the corresponding classes from the legacy Airflow modules.
The example DAGs support both Airflow 2 and Airflow 3. When creating DAGs for Airflow 3, use the new import paths provided by Airflow.
For example, import DAG from airflow.sdk:
from airflow.sdk import DAG
with DAG(
dag_id="example_hbase",
schedule=None,
catchup=False,
):
...
If a DAG also uses PythonOperator, import it from airflow.providers.standard.operators.python in Airflow 3.
Configuration
The HBase provider uses the hbase Airflow connection type to store the parameters required for communicating with an HBase cluster.
Every operator and sensor accepts the hbase_conn_id parameter, which references the configured Airflow connection. If no connection identifier is specified explicitly, the provider uses the hbase_default connection by default.
The HBase connection supports both Thrift2 and CLI-based communication. The same connection identifier can be used by HBaseThriftHook and HBaseCLIHook.
Standard configuration
Below is a list of standard Airflow connection fields used by the provider. The remaining configuration is specified in the Extra field as a JSON document.
| Field | Description |
|---|---|
Connection type |
Must be set to |
Host |
Specifies the hostname or IP address of the HBase Thrift2 Server |
Port |
Specifies the Thrift2 service port. If omitted, the default HBase Thrift2 Server port ( |
Extra |
Contains provider-specific configuration parameters |
Configuration example for a basic Thrift2 connection with no authentication:
-
Connection ID: hbase_default -
Connection Type: hbase -
Host: hbase-server.example.com -
Port: 9090 -
Extra:{ "timeout": 30000, "namespace": "default" }
Extra parameters
The following parameters control the behavior of the Thrift2 client.
| Parameter | Description | Default value |
|---|---|---|
timeout |
Defines the timeout for a Thrift request in milliseconds |
30000 |
namespace |
Specifies the default HBase namespace used by operators that do not explicitly define one |
default |
use_http |
Enables HTTP transport instead of the default binary socket transport |
false |
retry_max_attempts |
Defines the maximum number of retry attempts after a connection failure |
3 |
retry_delay |
Defines the initial delay between retry attempts in seconds |
1.0 |
retry_backoff_factor |
Defines the multiplier used to increase the delay after each unsuccessful retry attempt |
2.0 |
Kerberos authentication
The provider supports Kerberos authentication through the GSSAPI mechanism.
Configure the parameters in the connection Extra field described in the table below.
| Parameter | Description | Default value |
|---|---|---|
auth_method |
Authentication method. Set to |
— |
kerberos_service_name |
Specifies the Kerberos service principal name |
hbase |
kerberos_principal |
Specifies the Kerberos principal used by the Airflow connection |
— |
kerberos_keytab |
Specifies the path to the Kerberos keytab file |
— |
Example:
{
"auth_method": "GSSAPI",
"kerberos_service_name": "hbase",
"kerberos_principal": "airflow@YOUR.REALM",
"kerberos_keytab": "/etc/security/keytabs/airflow.keytab",
"timeout": 30000
}
A valid Kerberos configuration and the required authentication environment must be available on Airflow workers.
For CLI-based backup and restore operations, the provider also supports Kerberos authentication using the HBase service keytab. The relevant CLI configuration is described in the CLI parameters section.
Connection pooling
By default, the provider uses a single Thrift connection for each task execution. This strategy, implemented by Thrift2Strategy, is appropriate for most workloads and minimizes resource consumption.
For production environments that execute large batch operations or multiple concurrent HBase tasks, the provider also supports connection pooling through PooledThrift2Strategy. Instead of creating a new connection for every request, the provider maintains a pool of reusable Thrift connections that can be shared between operations. Reusing established connections reduces connection overhead and improves throughput during intensive workloads.
Connection pooling is disabled by default. To enable it, add the connection_pool section to the connection Extra field.
{
"connection_pool": {
"enabled": true,
"size": 10,
"timeout": 30
}
}
The available connection pool parameters are described below.
| Parameter | Description | Default value |
|---|---|---|
enabled |
Enables the connection pool |
false |
size |
Specifies the maximum number of Thrift connections maintained in the pool |
10 |
timeout |
Defines the maximum time, in seconds, that a task waits for an available connection before reporting an error |
30 |
|
TIP
Connection pooling is recommended for production deployments that perform batch processing or execute multiple HBase tasks concurrently. When using multiple workers for batch operations, configure a pool large enough to accommodate the expected concurrency.
|
SSL/TLS
The provider supports encrypted communication with HBase Thrift2 Server. SSL/TLS parameters can be specified directly in the connection Extra field.
The provider supports both flat and nested SSL configuration formats:
-
Flat — SSL parameters are specified directly in Extra.
-
Nested — SSL parameters are specified inside the
ssl_optionsobject.
| Parameter | Description | Default value |
|---|---|---|
ca_certs |
Specifies the path to the trusted CA certificate file |
— |
cert_file |
Specifies the path to the client certificate file |
— |
key_file |
Specifies the path to the client private key file |
— |
validate |
Defines whether the server certificate is validated during connection establishment |
true |
use_http |
Must be set to |
false |
The following example enables SSL connection with server certificate validation:
{
"use_http": true,
"ssl_options": {
"ca_certs": "/etc/ssl/certs/ca-bundle.pem",
"cert_file": "/etc/ssl/certs/client.pem",
"key_file": "/etc/ssl/private/client.key",
"validate": true
}
}
The same SSL parameters can be specified directly in Extra without the ssl_options object.
CLI parameters
Backup and restore operators execute the native HBase Backup CLI on an Airflow worker instead of communicating with HBase through Thrift2.
Below are the optional parameters for configuring the runtime environment.
| Parameter | Description | Default value |
|---|---|---|
hbase_cmd |
Specifies the HBase command name |
hbase |
java_home |
Specifies the Java installation used to execute HBase CLI commands |
|
hbase_home |
Specifies the HBase installation directory that contains the client utilities |
|
These parameters are constructor arguments of HBaseCLIHook. They can also be supplied through the HBASE_CMD, JAVA_HOME, and HBASE_HOME environment variables, respectively.
In deployments where the HBase Client component is installed from ADH using shared hosts, the default values normally do not require modification because they point to the standard installation directories managed by ADH.
For Kerberos-enabled CLI operations, the hook also supports the connection Extra parameters described in the table below.
| Parameter | Description | Default value |
|---|---|---|
kerberos_keytab |
Enables Kerberos-based CLI execution when a keytab is specified |
— |
hbase_service_keytab |
Specifies the HBase service keytab used to execute commands as the |
/etc/security/keytabs/hbase.service.keytab |
hbase_service_principal |
Specifies the HBase service principal |
Generated using the worker hostname and |
kerberos_realm |
Specifies the Kerberos realm used to construct the HBase service principal when |
KRB5-TEST |
For Kerberos-based CLI execution, the Airflow worker must have passwordless sudo access to run kinit and HBase CLI commands as the hbase user.
Hooks
Hooks implement the communication layer of the provider. The provider contains two hook implementations because HBase itself exposes different interfaces for different categories of operations.
HBaseThriftHook
HBaseThriftHook is the primary communication interface used by the provider. It establishes a connection to HBase Thrift2 Server and exposes methods for performing table administration, data manipulation, metadata retrieval, and batch processing. All data operators and sensors rely on this hook to communicate with the HBase cluster.
The hook provides methods for the most common HBase operations, including table management, row manipulation, table scanning, and batch processing.
| Method | Description |
|---|---|
table_exists() |
Checks whether the specified table exists before performing subsequent operations |
create_table() |
Creates a new table with the specified column families |
delete_table() |
Deletes an existing table |
put_row() |
Creates a new row or updates an existing one |
get_row() |
Retrieves a single row by its row key |
delete_row() |
Deletes an entire row or selected columns within the row |
scan_table() |
Scans a range of rows and returns matching records |
batch_put_rows() |
Writes multiple rows using configurable batch sizes and optional parallel execution |
batch_get_rows() |
Retrieves multiple rows within a single batch operation |
batch_delete_rows() |
Deletes multiple rows in batches |
The hook also supports OpenLineage integration. When the OpenLineage provider is available, it can return HBase-specific database information for lineage reporting.
HBaseCLIHook
HBaseCLIHook executes native HBase CLI commands on the Airflow worker. It is used for administrative operations that require the HBase Backup subsystem.
The hook provides methods that correspond to the HBase Backup CLI operations.
| Method | Description |
|---|---|
create_full_backup() |
Creates a full backup of one or more HBase tables |
create_incremental_backup() |
Creates an incremental backup based on a previous backup image |
restore_backup() |
Restores data from an existing backup |
get_backup_history() |
Retrieves the history of completed backup operations |
describe_backup() |
Displays detailed information about a backup image |
create_backup_set() |
Creates or modifies a backup set used by backup operations |
list_backup_sets() |
Returns the configured backup sets |
execute_command() |
Executes an arbitrary HBase CLI command and returns its output |
Operators
Operators implement the executable tasks that can be used directly in Airflow DAGs. Each operator encapsulates a specific HBase operation, validates the supplied parameters, initializes the appropriate hook, executes the requested action, and returns the result to the Airflow task context when applicable.
The provider includes 10 operators organized into three groups: table management, data manipulation, and backup and restore.
Table management operators
Table management operators perform administrative operations on HBase tables. The provider includes the table management operators listed below.
| Operator | Description |
|---|---|
HBaseCreateTableOperator |
Creates an HBase table with the specified column families. The |
HBaseDeleteTableOperator |
Deletes an existing table. The |
Data operators
Data operators perform read and write operations through the HBase Thrift2 Server API. They use HBaseThriftHook, which automatically selects either a dedicated Thrift connection or a pooled connection depending on the configured Airflow connection.
The provider includes the following operators for data manipulation.
| Operator | Description |
|---|---|
HBasePutOperator |
Inserts or updates a single row |
HBaseBatchPutOperator |
Writes multiple rows using configurable batch processing and parallel execution |
HBaseBatchGetOperator |
Retrieves multiple rows in a single batch operation |
HBaseScanOperator |
Scans table contents within a specified row range |
The batch operators support configurable batch sizes. HBaseBatchPutOperator also supports parallel execution through the max_workers parameter.
Backup and restore operators
Backup operators provide access to the HBase Backup subsystem through HBaseCLIHook. The provider includes the backup management operators listed below.
| Operator | Description |
|---|---|
HBaseCreateBackupOperator |
Creates full or incremental HBase backups |
HBaseBackupHistoryOperator |
Retrieves the history of completed backup operations |
HBaseRestoreOperator |
Restores tables from an existing backup |
HBaseBackupSetOperator |
Creates or lists HBase backup sets |
The HBaseCreateBackupOperator returns the generated backup identifier. This identifier can be passed to HBaseRestoreOperator through XCom.
Sensors
Sensors allow DAGs to wait until a particular condition in HBase becomes true before continuing execution. All sensor parameters that identify HBase resources, such as table names and row keys, are defined as Airflow template fields. This allows sensors to monitor dynamically generated resources whose names are determined during DAG execution.
The provider includes the following sensors.
| Sensor | Description |
|---|---|
HBaseTableSensor |
Waits until a specified HBase table becomes available. During every polling cycle, the sensor calls the |
HBaseRowSensor |
Waits until a specific row exists in an HBase table. During each polling cycle, the sensor retrieves the row using |
Both sensors inherit the standard Airflow sensor parameters, including timeout, poke_interval, and mode.
Datasets and event-based scheduling
The HBase provider includes the hbase_table_dataset() function, which creates an Airflow Dataset representing an HBase table.
In Airflow 3, Airflow datasets are implemented as assets. The provider automatically uses the appropriate object depending on the installed Airflow version.
The function accepts the following parameters:
-
host— hostname of the HBase Thrift2 Server; -
port— port of the HBase Thrift2 Server (the default value is9090); -
table_name— name of the HBase table (this parameter is required).
Example:
from airflow.providers.arenadata.hbase.datasets.hbase import hbase_table_dataset
users_dataset = hbase_table_dataset(
host="hbase-server.example.com",
port=9090,
table_name="users",
)
The resulting asset can be used to establish dependencies between DAGs.
For example, a producer DAG can declare an HBase table as an outlet:
from airflow.providers.arenadata.hbase.datasets.hbase import hbase_table_dataset
from airflow.providers.arenadata.hbase.operators.hbase import HBasePutOperator
users_dataset = hbase_table_dataset(
host="hbase-server.example.com",
table_name="users",
)
write_users = HBasePutOperator(
task_id="write_users",
table_name="users",
row_key="user_001",
data={"profile:name": "Alice"},
hbase_conn_id="hbase_default",
outlets=[users_dataset],
)
A downstream DAG can use the same asset as its schedule. Airflow triggers that DAG when the producer task successfully updates the declared outlet.
This mechanism is used in example DAGs to trigger backup workflows based on HBase table updates.
Utility classes
The provider defines several enumeration classes that provide strongly typed configuration values for HBase operators. Using these enumerations instead of string literals improves the readability of DAG definitions, validates parameter values before execution, and reduces the likelihood of configuration errors.
The provider defines the following utility classes.
| Type | Description |
|---|---|
BackupType |
Specifies whether a backup operation is performed as a full or incremental backup |
BackupSetAction |
Specifies the action performed by |
IfExistsAction |
Controls how table creation operators behave when the target table already exists |
IfNotExistsAction |
Controls how table deletion operators behave when the target table does not exist |
Use the provider in a DAG
To use the HBase provider, import the required operators or sensors from the provider package and configure an HBase connection.
The examples in this section demonstrate the most common provider workflows. The provider package also includes example DAGs for basic operations, Kerberos authentication, batch processing, connection pooling, sensors, backup, and restore.
Basic data flow
A typical HBase workflow consists of creating a table, writing data, and reading the stored records.
Use HBaseCreateTableOperator to create an HBase table and define its column families:
from airflow.providers.arenadata.hbase.operators.hbase import HBaseCreateTableOperator
create_table = HBaseCreateTableOperator(
task_id="create_table",
table_name="users", (1)
families={ (2)
"profile": {},
"contacts": {},
},
hbase_conn_id="hbase_default",
)
| 1 | Name of the HBase table. |
| 2 | Column families created together with the table. |
Use HBasePutOperator to insert or update a single row:
from airflow.providers.arenadata.hbase.operators.hbase import HBasePutOperator
insert_user = HBasePutOperator(
task_id="insert_user",
table_name="users",
row_key="user_001", (1)
data={ (2)
"profile:name": "Alice",
"profile:age": "30",
"contacts:email": "alice@example.com",
},
hbase_conn_id="hbase_default",
)
| 1 | Unique row identifier. |
| 2 | Column names include the column family prefix. |
Use HBaseScanOperator to retrieve rows from a table:
from airflow.providers.arenadata.hbase.operators.hbase import HBaseScanOperator
scan_users = HBaseScanOperator(
task_id="scan_users",
table_name="users",
row_start="user_000",
row_stop="user_999",
columns=[
"profile:name",
"contacts:email",
],
limit=100,
hbase_conn_id="hbase_default",
)
In Airflow 3, define the DAG using the new SDK import:
from datetime import datetime
from airflow.sdk import DAG
from airflow.providers.arenadata.hbase.operators.hbase import (
HBaseCreateTableOperator,
HBasePutOperator,
HBaseScanOperator,
)
with DAG(
dag_id="hbase_basic_workflow",
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
) as dag:
create_table = HBaseCreateTableOperator(
task_id="create_table",
table_name="users",
families={"profile": {}, "contacts": {}},
hbase_conn_id="hbase_default",
)
insert_user = HBasePutOperator(
task_id="insert_user",
table_name="users",
row_key="user_001",
data={
"profile:name": "Alice",
"profile:age": "30",
"contacts:email": "alice@example.com",
},
hbase_conn_id="hbase_default",
)
scan_users = HBaseScanOperator(
task_id="scan_users",
table_name="users",
row_start="user_000",
row_stop="user_999",
limit=100,
hbase_conn_id="hbase_default",
)
create_table >> insert_user >> scan_users
Batch processing
For large datasets, use the batch operators to reduce the number of Thrift requests and improve throughput.
HBaseBatchPutOperator writes multiple rows using configurable batch sizes.
from airflow.providers.arenadata.hbase.operators.hbase import HBaseBatchPutOperator
batch_insert = HBaseBatchPutOperator(
task_id="batch_insert",
table_name="users",
rows=rows,
batch_size=200, (1)
max_workers=4, (2)
hbase_conn_id="hbase_default",
)
| 1 | Number of rows written in a single batch. |
| 2 | Number of concurrent workers used for batch processing. |
When using multiple workers, enable connection pooling in the HBase connection for improved throughput.
Use HBaseBatchGetOperator to retrieve several rows in a single operation:
from airflow.providers.arenadata.hbase.operators.hbase import HBaseBatchGetOperator
batch_get = HBaseBatchGetOperator(
task_id="batch_get",
table_name="users",
row_keys=[
"user_001",
"user_002",
"user_003",
],
columns=[
"profile:name",
"contacts:email",
],
hbase_conn_id="hbase_default",
)
Backup workflow
The provider integrates with the HBase Backup CLI to manage backup sets and perform backup and restore operations.
Before creating backups, prepare the HDFS backup directory:
$ hdfs dfs -mkdir -p /hbase/backup
$ hdfs dfs -chmod 777 /hbase/backup
The chmod 777 access is used only for testing purposes and is not suitable for production.
Enable backup support in the HBase configuration:
hbase.backup.enable=true
A backup set groups one or more tables into a reusable backup definition.
Create a backup set:
from airflow.providers.arenadata.hbase.operators.hbase import (
BackupSetAction,
HBaseBackupSetOperator,
)
create_backup_set = HBaseBackupSetOperator(
task_id="create_backup_set",
action=BackupSetAction.ADD,
backup_set_name="production_tables",
tables=[
"users",
"orders",
],
hbase_conn_id="hbase_default",
)
Use HBaseCreateBackupOperator to create either a full or incremental backup.
Create a backup:
from airflow.providers.arenadata.hbase.operators.hbase import (
BackupType,
HBaseCreateBackupOperator,
)
create_backup = HBaseCreateBackupOperator(
task_id="create_backup",
backup_type=BackupType.FULL,
backup_path="hdfs:///hbase/backup", (1)
backup_set_name="production_tables",
workers=2,
hbase_conn_id="hbase_default",
)
| 1 | HDFS location where the backup is stored. |
The operator pushes the generated backup identifier to XCom for later restore operations.
Use HBaseBackupHistoryOperator to verify completed backup operations:
from airflow.providers.arenadata.hbase.operators.hbase import HBaseBackupHistoryOperator
backup_history = HBaseBackupHistoryOperator(
task_id="backup_history",
backup_set_name="production_tables",
hbase_conn_id="hbase_default",
)
Restore data using the backup identifier returned by the backup task:
from airflow.providers.arenadata.hbase.operators.hbase import HBaseRestoreOperator
restore_backup = HBaseRestoreOperator(
task_id="restore_backup",
backup_path="hdfs:///hbase/backup",
backup_id="{{ ti.xcom_pull(task_ids='create_backup') }}",
tables=["users"],
overwrite=True,
hbase_conn_id="hbase_default",
)
Monitoring
Use HBaseTableSensor to wait until a table becomes available:
from airflow.providers.arenadata.hbase.sensors.hbase import HBaseTableSensor
wait_for_table = HBaseTableSensor(
task_id="wait_for_table",
table_name="users",
timeout=300,
poke_interval=30,
hbase_conn_id="hbase_default",
)
This sensor is typically used when another DAG or external application creates the table.
Use HBaseRowSensor to wait until a specific row has been written:
from airflow.providers.arenadata.hbase.sensors.hbase import HBaseRowSensor
wait_for_row = HBaseRowSensor(
task_id="wait_for_row",
table_name="users",
row_key="user_001",
timeout=600,
poke_interval=60,
hbase_conn_id="hbase_default",
)
This sensor is commonly used to wait for marker rows that indicate data ingestion has completed.