HBase provider

Overview

The HBase provider enables DAGs to interact with HBase. It provides hooks, operators, and sensors for performing administrative and data operations on HBase tables as well as backup and restore tasks.

The provider can be used to:

  • create and delete tables;

  • insert, update, retrieve, and delete rows;

  • scan tables;

  • execute batch operations;

  • create and restore backups;

  • manage backup sets;

  • monitor table and row availability;

  • trigger DAGs based on changes to HBase tables using Airflow datasets (assets in Airflow 3).

The provider follows the standard Airflow provider architecture, where operators implement DAG tasks, hooks provide low-level communication with external systems, and sensors wait until a specified condition becomes true.

The provider supports Airflow 3 and maintains backward compatibility with Airflow 2.

Unlike generic Airflow providers, the HBase provider supports two execution mechanisms:

  • Thrift2 API for online data operations;

  • HBase CLI utilities for backup and restore operations.

Architecture

The provider is organized into several Python modules:

  • Hooks

    This module implements the communication layer. It contains classes that establish connections to HBase and expose a Python interface for performing HBase operations. Two hook implementations are provided:

    • HBaseThriftHook — communicates with the HBase Thrift2 Server component and is responsible for table management, data manipulation, and metadata operations.

    • HBaseCLIHook — executes native HBase client commands and is used for backup and restore workflows.

  • Operators

    This module contains Airflow tasks that represent individual HBase operations. Each operator performs parameter validation, initializes the appropriate hook, invokes the required hook method, and returns the execution result to Airflow.

  • Sensors

    The module provides Airflow sensors that periodically check the state of HBase resources. Unlike operators, sensors do not modify data. They repeatedly execute lightweight requests until a specified condition is satisfied or the configured timeout expires. Typical use cases include waiting for a table to be created before loading data or waiting for a specific row to appear as a result of another workflow.

  • Datasets

    This module provides the hbase_table_dataset() function for creating Airflow datasets that represent HBase tables. In Airflow 3, the function creates an Asset instead. Datasets (assets) can be used to establish dependencies between DAGs and trigger downstream workflows when data in an HBase table is updated.

  • Utils

    The module contains helper classes and functions shared by multiple provider components. These include enumerations that describe backup types and operator behavior, constants used throughout the provider, retry helpers, and data conversion utilities.

These modules work as a single execution pipeline. During DAG execution, Airflow creates an operator instance, which selects the appropriate hook according to the requested operation. The hook establishes communication with HBase through either the Thrift2 protocol or the HBase CLI, executes the requested operation, converts the response into Python objects where necessary, and returns the result to the operator. The operator then reports the task status back to the Airflow scheduler.

The provider uses a compatibility module to support the different Airflow APIs. In Airflow 3, it uses classes and execution context from the Airflow SDK. In Airflow 2, it uses the corresponding classes from the legacy Airflow modules.

Requirements

The requirements depend on the functionality used.

If you already have a working ADH cluster with the necessary components, you can share its hosts with your ADO cluster by using shared hosts.

Data operations

The following requirements apply to data operators and sensors:

  • HBase Thrift2 Server must be running in the ADH cluster and be accessible from Airflow workers;

  • an HBase connection must be configured in Airflow.

Backup and restore

Backup and restore operations require the following components:

  • HBase Client installed on Airflow hosts;

  • HDFS access configured for Airflow workers, including the required permissions;

  • network connectivity between Airflow workers and HDFS.

Airflow versions compatibility

The HBase provider supports Airflow 3 and Airflow 2. The provider selects the appropriate Airflow classes according to the installed Airflow version.

In Airflow 3, the provider uses the Airflow SDK for the following components:

  • DAG definitions;

  • base operator and sensor classes;

  • connection and variable access;

  • execution context.

In Airflow 2, the provider continues to use the corresponding classes from the legacy Airflow modules.

The example DAGs support both Airflow 2 and Airflow 3. When creating DAGs for Airflow 3, use the new import paths provided by Airflow.

For example, import DAG from airflow.sdk:

from airflow.sdk import DAG

with DAG(
    dag_id="example_hbase",
    schedule=None,
    catchup=False,
):
    ...

If a DAG also uses PythonOperator, import it from airflow.providers.standard.operators.python in Airflow 3.

Configuration

The HBase provider uses the hbase Airflow connection type to store the parameters required for communicating with an HBase cluster.

Every operator and sensor accepts the hbase_conn_id parameter, which references the configured Airflow connection. If no connection identifier is specified explicitly, the provider uses the hbase_default connection by default.

The HBase connection supports both Thrift2 and CLI-based communication. The same connection identifier can be used by HBaseThriftHook and HBaseCLIHook.

Standard configuration

Below is a list of standard Airflow connection fields used by the provider. The remaining configuration is specified in the Extra field as a JSON document.

Field Description

Connection type

Must be set to hbase

Host

Specifies the hostname or IP address of the HBase Thrift2 Server

Port

Specifies the Thrift2 service port. If omitted, the default HBase Thrift2 Server port (9090) is used

Extra

Contains provider-specific configuration parameters

Configuration example for a basic Thrift2 connection with no authentication:

  • Connection ID: hbase_default

  • Connection Type: hbase

  • Host: hbase-server.example.com

  • Port: 9090

  • Extra:

    {
      "timeout": 30000,
      "namespace": "default"
    }

Extra parameters

The following parameters control the behavior of the Thrift2 client.

Parameter Description Default value

timeout

Defines the timeout for a Thrift request in milliseconds

30000

namespace

Specifies the default HBase namespace used by operators that do not explicitly define one

default

use_http

Enables HTTP transport instead of the default binary socket transport

false

retry_max_attempts

Defines the maximum number of retry attempts after a connection failure

3

retry_delay

Defines the initial delay between retry attempts in seconds

1.0

retry_backoff_factor

Defines the multiplier used to increase the delay after each unsuccessful retry attempt

2.0

Kerberos authentication

The provider supports Kerberos authentication through the GSSAPI mechanism.

Configure the parameters in the connection Extra field described in the table below.

Parameter Description Default value

auth_method

Authentication method. Set to GSSAPI to enable Kerberos authentication

 — 

kerberos_service_name

Specifies the Kerberos service principal name

hbase

kerberos_principal

Specifies the Kerberos principal used by the Airflow connection

 — 

kerberos_keytab

Specifies the path to the Kerberos keytab file

 — 

Example:

{
  "auth_method": "GSSAPI",
  "kerberos_service_name": "hbase",
  "kerberos_principal": "airflow@YOUR.REALM",
  "kerberos_keytab": "/etc/security/keytabs/airflow.keytab",
  "timeout": 30000
}

A valid Kerberos configuration and the required authentication environment must be available on Airflow workers.

For CLI-based backup and restore operations, the provider also supports Kerberos authentication using the HBase service keytab. The relevant CLI configuration is described in the CLI parameters section.

Connection pooling

By default, the provider uses a single Thrift connection for each task execution. This strategy, implemented by Thrift2Strategy, is appropriate for most workloads and minimizes resource consumption.

For production environments that execute large batch operations or multiple concurrent HBase tasks, the provider also supports connection pooling through PooledThrift2Strategy. Instead of creating a new connection for every request, the provider maintains a pool of reusable Thrift connections that can be shared between operations. Reusing established connections reduces connection overhead and improves throughput during intensive workloads.

Connection pooling is disabled by default. To enable it, add the connection_pool section to the connection Extra field.

{
  "connection_pool": {
    "enabled": true,
    "size": 10,
    "timeout": 30
  }
}

The available connection pool parameters are described below.

Parameter Description Default value

enabled

Enables the connection pool

false

size

Specifies the maximum number of Thrift connections maintained in the pool

10

timeout

Defines the maximum time, in seconds, that a task waits for an available connection before reporting an error

30

TIP
Connection pooling is recommended for production deployments that perform batch processing or execute multiple HBase tasks concurrently. When using multiple workers for batch operations, configure a pool large enough to accommodate the expected concurrency.

SSL/TLS

The provider supports encrypted communication with HBase Thrift2 Server. SSL/TLS parameters can be specified directly in the connection Extra field.

The provider supports both flat and nested SSL configuration formats:

  • Flat — SSL parameters are specified directly in Extra.

  • Nested — SSL parameters are specified inside the ssl_options object.

Parameter Description Default value

ca_certs

Specifies the path to the trusted CA certificate file

 — 

cert_file

Specifies the path to the client certificate file

 — 

key_file

Specifies the path to the client private key file

 — 

validate

Defines whether the server certificate is validated during connection establishment

true

use_http

Must be set to true when HBase Thrift2 Server is configured to use HTTP transport

false

The following example enables SSL connection with server certificate validation:

{
  "use_http": true,
  "ssl_options": {
    "ca_certs": "/etc/ssl/certs/ca-bundle.pem",
    "cert_file": "/etc/ssl/certs/client.pem",
    "key_file": "/etc/ssl/private/client.key",
    "validate": true
  }
}

The same SSL parameters can be specified directly in Extra without the ssl_options object.

CLI parameters

Backup and restore operators execute the native HBase Backup CLI on an Airflow worker instead of communicating with HBase through Thrift2.

Below are the optional parameters for configuring the runtime environment.

Parameter Description Default value

hbase_cmd

Specifies the HBase command name

hbase

java_home

Specifies the Java installation used to execute HBase CLI commands

JAVA_HOME environment variable or /usr/lib/jvm/java-arenadata-openjdk-8

hbase_home

Specifies the HBase installation directory that contains the client utilities

HBASE_HOME environment variable or /usr/lib/hbase

These parameters are constructor arguments of HBaseCLIHook. They can also be supplied through the HBASE_CMD, JAVA_HOME, and HBASE_HOME environment variables, respectively.

In deployments where the HBase Client component is installed from ADH using shared hosts, the default values normally do not require modification because they point to the standard installation directories managed by ADH.

For Kerberos-enabled CLI operations, the hook also supports the connection Extra parameters described in the table below.

Parameter Description Default value

kerberos_keytab

Enables Kerberos-based CLI execution when a keytab is specified

 — 

hbase_service_keytab

Specifies the HBase service keytab used to execute commands as the hbase user

/etc/security/keytabs/hbase.service.keytab

hbase_service_principal

Specifies the HBase service principal

Generated using the worker hostname and kerberos_realm

kerberos_realm

Specifies the Kerberos realm used to construct the HBase service principal when hbase_service_principal is not specified

KRB5-TEST

For Kerberos-based CLI execution, the Airflow worker must have passwordless sudo access to run kinit and HBase CLI commands as the hbase user.

Hooks

Hooks implement the communication layer of the provider. The provider contains two hook implementations because HBase itself exposes different interfaces for different categories of operations.

HBaseThriftHook

HBaseThriftHook is the primary communication interface used by the provider. It establishes a connection to HBase Thrift2 Server and exposes methods for performing table administration, data manipulation, metadata retrieval, and batch processing. All data operators and sensors rely on this hook to communicate with the HBase cluster.

The hook provides methods for the most common HBase operations, including table management, row manipulation, table scanning, and batch processing.

Method Description

table_exists()

Checks whether the specified table exists before performing subsequent operations

create_table()

Creates a new table with the specified column families

delete_table()

Deletes an existing table

put_row()

Creates a new row or updates an existing one

get_row()

Retrieves a single row by its row key

delete_row()

Deletes an entire row or selected columns within the row

scan_table()

Scans a range of rows and returns matching records

batch_put_rows()

Writes multiple rows using configurable batch sizes and optional parallel execution

batch_get_rows()

Retrieves multiple rows within a single batch operation

batch_delete_rows()

Deletes multiple rows in batches

The hook also supports OpenLineage integration. When the OpenLineage provider is available, it can return HBase-specific database information for lineage reporting.

HBaseCLIHook

HBaseCLIHook executes native HBase CLI commands on the Airflow worker. It is used for administrative operations that require the HBase Backup subsystem.

The hook provides methods that correspond to the HBase Backup CLI operations.

Method Description

create_full_backup()

Creates a full backup of one or more HBase tables

create_incremental_backup()

Creates an incremental backup based on a previous backup image

restore_backup()

Restores data from an existing backup

get_backup_history()

Retrieves the history of completed backup operations

describe_backup()

Displays detailed information about a backup image

create_backup_set()

Creates or modifies a backup set used by backup operations

list_backup_sets()

Returns the configured backup sets

execute_command()

Executes an arbitrary HBase CLI command and returns its output

Operators

Operators implement the executable tasks that can be used directly in Airflow DAGs. Each operator encapsulates a specific HBase operation, validates the supplied parameters, initializes the appropriate hook, executes the requested action, and returns the result to the Airflow task context when applicable.

The provider includes 10 operators organized into three groups: table management, data manipulation, and backup and restore.

Table management operators

Table management operators perform administrative operations on HBase tables. The provider includes the table management operators listed below.

Operator Description

HBaseCreateTableOperator

Creates an HBase table with the specified column families. The if_exists parameter controls the behavior when the table already exists

HBaseDeleteTableOperator

Deletes an existing table. The disable parameter controls whether the table is disabled before deletion. The if_not_exists parameter controls the behavior when the table does not exist

Data operators

Data operators perform read and write operations through the HBase Thrift2 Server API. They use HBaseThriftHook, which automatically selects either a dedicated Thrift connection or a pooled connection depending on the configured Airflow connection.

The provider includes the following operators for data manipulation.

Operator Description

HBasePutOperator

Inserts or updates a single row

HBaseBatchPutOperator

Writes multiple rows using configurable batch processing and parallel execution

HBaseBatchGetOperator

Retrieves multiple rows in a single batch operation

HBaseScanOperator

Scans table contents within a specified row range

The batch operators support configurable batch sizes. HBaseBatchPutOperator also supports parallel execution through the max_workers parameter.

Backup and restore operators

Backup operators provide access to the HBase Backup subsystem through HBaseCLIHook. The provider includes the backup management operators listed below.

Operator Description

HBaseCreateBackupOperator

Creates full or incremental HBase backups

HBaseBackupHistoryOperator

Retrieves the history of completed backup operations

HBaseRestoreOperator

Restores tables from an existing backup

HBaseBackupSetOperator

Creates or lists HBase backup sets

The HBaseCreateBackupOperator returns the generated backup identifier. This identifier can be passed to HBaseRestoreOperator through XCom.

Sensors

Sensors allow DAGs to wait until a particular condition in HBase becomes true before continuing execution. All sensor parameters that identify HBase resources, such as table names and row keys, are defined as Airflow template fields. This allows sensors to monitor dynamically generated resources whose names are determined during DAG execution.

The provider includes the following sensors.

Sensor Description

HBaseTableSensor

Waits until a specified HBase table becomes available. During every polling cycle, the sensor calls the table_exists() method of HBaseThriftHook to determine whether the table has been created

HBaseRowSensor

Waits until a specific row exists in an HBase table. During each polling cycle, the sensor retrieves the row using HBaseThriftHook.get_row() and considers the condition satisfied when the requested row contains data

Both sensors inherit the standard Airflow sensor parameters, including timeout, poke_interval, and mode.

Datasets and event-based scheduling

The HBase provider includes the hbase_table_dataset() function, which creates an Airflow Dataset representing an HBase table.

In Airflow 3, Airflow datasets are implemented as assets. The provider automatically uses the appropriate object depending on the installed Airflow version.

The function accepts the following parameters:

  • host — hostname of the HBase Thrift2 Server;

  • port — port of the HBase Thrift2 Server (the default value is 9090);

  • table_name — name of the HBase table (this parameter is required).

Example:

from airflow.providers.arenadata.hbase.datasets.hbase import hbase_table_dataset

users_dataset = hbase_table_dataset(
    host="hbase-server.example.com",
    port=9090,
    table_name="users",
)

The resulting asset can be used to establish dependencies between DAGs.

For example, a producer DAG can declare an HBase table as an outlet:

from airflow.providers.arenadata.hbase.datasets.hbase import hbase_table_dataset
from airflow.providers.arenadata.hbase.operators.hbase import HBasePutOperator

users_dataset = hbase_table_dataset(
    host="hbase-server.example.com",
    table_name="users",
)

write_users = HBasePutOperator(
    task_id="write_users",
    table_name="users",
    row_key="user_001",
    data={"profile:name": "Alice"},
    hbase_conn_id="hbase_default",
    outlets=[users_dataset],
)

A downstream DAG can use the same asset as its schedule. Airflow triggers that DAG when the producer task successfully updates the declared outlet.

This mechanism is used in example DAGs to trigger backup workflows based on HBase table updates.

Utility classes

The provider defines several enumeration classes that provide strongly typed configuration values for HBase operators. Using these enumerations instead of string literals improves the readability of DAG definitions, validates parameter values before execution, and reduces the likelihood of configuration errors.

The provider defines the following utility classes.

Type Description

BackupType

Specifies whether a backup operation is performed as a full or incremental backup

BackupSetAction

Specifies the action performed by HBaseBackupSetOperator, such as creating or listing backup sets

IfExistsAction

Controls how table creation operators behave when the target table already exists

IfNotExistsAction

Controls how table deletion operators behave when the target table does not exist

Use the provider in a DAG

To use the HBase provider, import the required operators or sensors from the provider package and configure an HBase connection.

The examples in this section demonstrate the most common provider workflows. The provider package also includes example DAGs for basic operations, Kerberos authentication, batch processing, connection pooling, sensors, backup, and restore.

Basic data flow

A typical HBase workflow consists of creating a table, writing data, and reading the stored records.

Use HBaseCreateTableOperator to create an HBase table and define its column families:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseCreateTableOperator

create_table = HBaseCreateTableOperator(
    task_id="create_table",
    table_name="users", (1)
    families={ (2)
        "profile": {},
        "contacts": {},
    },
    hbase_conn_id="hbase_default",
)
1 Name of the HBase table.
2 Column families created together with the table.

Use HBasePutOperator to insert or update a single row:

from airflow.providers.arenadata.hbase.operators.hbase import HBasePutOperator

insert_user = HBasePutOperator(
    task_id="insert_user",
    table_name="users",
    row_key="user_001", (1)
    data={ (2)
        "profile:name": "Alice",
        "profile:age": "30",
        "contacts:email": "alice@example.com",
    },
    hbase_conn_id="hbase_default",
)
1 Unique row identifier.
2 Column names include the column family prefix.

Use HBaseScanOperator to retrieve rows from a table:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseScanOperator

scan_users = HBaseScanOperator(
    task_id="scan_users",
    table_name="users",
    row_start="user_000",
    row_stop="user_999",
    columns=[
        "profile:name",
        "contacts:email",
    ],
    limit=100,
    hbase_conn_id="hbase_default",
)

In Airflow 3, define the DAG using the new SDK import:

from datetime import datetime

from airflow.sdk import DAG
from airflow.providers.arenadata.hbase.operators.hbase import (
    HBaseCreateTableOperator,
    HBasePutOperator,
    HBaseScanOperator,
)

with DAG(
    dag_id="hbase_basic_workflow",
    start_date=datetime(2024, 1, 1),
    schedule=None,
    catchup=False,
) as dag:
    create_table = HBaseCreateTableOperator(
        task_id="create_table",
        table_name="users",
        families={"profile": {}, "contacts": {}},
        hbase_conn_id="hbase_default",
    )

    insert_user = HBasePutOperator(
        task_id="insert_user",
        table_name="users",
        row_key="user_001",
        data={
            "profile:name": "Alice",
            "profile:age": "30",
            "contacts:email": "alice@example.com",
        },
        hbase_conn_id="hbase_default",
    )

    scan_users = HBaseScanOperator(
        task_id="scan_users",
        table_name="users",
        row_start="user_000",
        row_stop="user_999",
        limit=100,
        hbase_conn_id="hbase_default",
    )

    create_table >> insert_user >> scan_users

Batch processing

For large datasets, use the batch operators to reduce the number of Thrift requests and improve throughput.

HBaseBatchPutOperator writes multiple rows using configurable batch sizes.

from airflow.providers.arenadata.hbase.operators.hbase import HBaseBatchPutOperator

batch_insert = HBaseBatchPutOperator(
    task_id="batch_insert",
    table_name="users",
    rows=rows,
    batch_size=200, (1)
    max_workers=4, (2)
    hbase_conn_id="hbase_default",
)
1 Number of rows written in a single batch.
2 Number of concurrent workers used for batch processing.

When using multiple workers, enable connection pooling in the HBase connection for improved throughput.

Use HBaseBatchGetOperator to retrieve several rows in a single operation:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseBatchGetOperator

batch_get = HBaseBatchGetOperator(
    task_id="batch_get",
    table_name="users",
    row_keys=[
        "user_001",
        "user_002",
        "user_003",
    ],
    columns=[
        "profile:name",
        "contacts:email",
    ],
    hbase_conn_id="hbase_default",
)

Backup workflow

The provider integrates with the HBase Backup CLI to manage backup sets and perform backup and restore operations.

Before creating backups, prepare the HDFS backup directory:

$ hdfs dfs -mkdir -p /hbase/backup
$ hdfs dfs -chmod 777 /hbase/backup

The chmod 777 access is used only for testing purposes and is not suitable for production.

Enable backup support in the HBase configuration:

hbase.backup.enable=true

A backup set groups one or more tables into a reusable backup definition.

Create a backup set:

from airflow.providers.arenadata.hbase.operators.hbase import (
    BackupSetAction,
    HBaseBackupSetOperator,
)

create_backup_set = HBaseBackupSetOperator(
    task_id="create_backup_set",
    action=BackupSetAction.ADD,
    backup_set_name="production_tables",
    tables=[
        "users",
        "orders",
    ],
    hbase_conn_id="hbase_default",
)

Use HBaseCreateBackupOperator to create either a full or incremental backup.

Create a backup:

from airflow.providers.arenadata.hbase.operators.hbase import (
    BackupType,
    HBaseCreateBackupOperator,
)

create_backup = HBaseCreateBackupOperator(
    task_id="create_backup",
    backup_type=BackupType.FULL,
    backup_path="hdfs:///hbase/backup", (1)
    backup_set_name="production_tables",
    workers=2,
    hbase_conn_id="hbase_default",
)
1 HDFS location where the backup is stored.

The operator pushes the generated backup identifier to XCom for later restore operations.

Use HBaseBackupHistoryOperator to verify completed backup operations:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseBackupHistoryOperator

backup_history = HBaseBackupHistoryOperator(
    task_id="backup_history",
    backup_set_name="production_tables",
    hbase_conn_id="hbase_default",
)

Restore data using the backup identifier returned by the backup task:

from airflow.providers.arenadata.hbase.operators.hbase import HBaseRestoreOperator

restore_backup = HBaseRestoreOperator(
    task_id="restore_backup",
    backup_path="hdfs:///hbase/backup",
    backup_id="{{ ti.xcom_pull(task_ids='create_backup') }}",
    tables=["users"],
    overwrite=True,
    hbase_conn_id="hbase_default",
)

Monitoring

Use HBaseTableSensor to wait until a table becomes available:

from airflow.providers.arenadata.hbase.sensors.hbase import HBaseTableSensor

wait_for_table = HBaseTableSensor(
    task_id="wait_for_table",
    table_name="users",
    timeout=300,
    poke_interval=30,
    hbase_conn_id="hbase_default",
)

This sensor is typically used when another DAG or external application creates the table.

Use HBaseRowSensor to wait until a specific row has been written:

from airflow.providers.arenadata.hbase.sensors.hbase import HBaseRowSensor

wait_for_row = HBaseRowSensor(
    task_id="wait_for_row",
    table_name="users",
    row_key="user_001",
    timeout=600,
    poke_interval=60,
    hbase_conn_id="hbase_default",
)

This sensor is commonly used to wait for marker rows that indicate data ingestion has completed.

Found a mistake? Seleсt text and press Ctrl+Enter to report it