Overview of Trino connectors
Overview
Trino connectors are Java-based plugins through which Trino can interact with various data sources, such as Hadoop data lakes, Hive warehouses, Apache Iceberg tables, PostgreSQL databases, and many others. Trino accesses data in a data source by using a connector, which is configured in a catalog. Connectors are responsible for managing the communication between Trino and the underlying data source. They also translate ANSI SQL statements into operations understood by the underlying data source. One connector can be used to access one or more data sources of the same type.
At a high level, the connectors' role in Trino interaction is presented in the following diagram.
Supported connectors
The full list of connectors supported by Trino can be found in Trino documentation.
Configure connectors
In Trino, connectors can be configured using catalog configuration files.
The connector to be used for accessing the catalog’s data source is specified in the <catalog_name>.properties configuration file using the connector.name property.
For example:
connector.name=hive
...|
NOTE
The same connector can be used by several catalogs.
For example, if you have two Hive warehouses with separate Metastore databases, you can configure two catalogs that use the same connector for Hive (connector.name=hive) but provide different connection properties for each catalog.
|
Catalog configuration files may also include auxiliary connector-specific properties for fine-tuning of the interaction with the underlying data source. For example:
hive.metastore.thrift.impersonation.enabled=True
iceberg.catalog.type=hive_metastore
iceberg.compression-codec=SNAPPY
iceberg.file-format=PARQUET
...For more information on connector-specific properties, see the corresponding connector section of the Trino documentation.
Create a custom connector
Apart from the connectors list supported by the Trino service out-of-the-box, you can develop your own connectors to access almost any data source. Even if your data source does not utilize concepts like tables, columns, rows, partitions, etc., once you adapt your data source to the API expected by Trino, you can write queries in ANSI SQL and fetch results through the connector.
Technically, new Trino connectors are added as plugins that integrate with the Trino framework to connect to a new data source. Any connector must implement the Trino’s service provider interface (SPI), which allows Trino to interact with a data source. Every connector must implement the following parts of the API:
-
Operations on retrieving metadata about tables, views, and schemas.
-
Operations to produce logical units of data partitioning to allow Trino to parallelize read and write statements.
-
Data sources and sinks that convert the source data to/from the in-memory objects expected by the Trino query engine.
After implementing the SPI for a connector, Trino can use standard operations internally to connect to any data source and perform operations on any data source. The particular connector’s implementation takes care of the details relevant to the specific features of the underlying data source.
For example, a basic Trino connector that supports reading data from a data source must implement a set of methods, including listTables().
The Trino engine invokes this method for all connectors to get a list of available tables in a schema/database.
The Trino engine is not aware of how each particular connector gets the tables list.
Some connectors may retrieve the list from an information schema, others may query a metastore, and some have to request the list via the API of the data source.
It is only the connector that knows how to get the list for the particular data source.
For details and examples on creating custom connectors, see the following sections in the Trino documentation:
Also, you can explore existing Trino connectors source code in GitHub.