ADQM Spark 3 Connector options
Parameters for connecting to an ADQM database and read/write parameters are set via key/value pairs.
The following options are available for ADQM Spark 3 Connector.
| Key | Description | Operation type | Required | Default value |
|---|---|---|---|---|
spark.adqm.url |
Database connection string via a driver that uses the TCP protocol |
Read/write |
Yes |
— |
spark.adqm.dbschema |
Name of the database schema to which the table belongs |
Read/write |
Yes |
default |
spark.adqm.dbtable |
Name of the database table |
Read/write |
Yes |
— |
spark.adqm.user |
ADQM user/role |
Read/write |
Yes |
— |
spark.adqm.password |
ADQM user password |
Read/write |
No |
— |
spark.adqm.cluster.name |
ADQM cluster name |
Read/write |
No |
— |
spark.adqm.jdbc.pool.maxsize |
Maximum number of connections in the connection pool |
Read/write |
No |
4 |
spark.adqm.jdbc.pool.timeout |
Time in milliseconds after which an inactive connection is considered idle |
Read/write |
No |
10000 |
spark.adqm.jdbc.pool.minidle |
Minimum number of idle connections supported in the connection pool |
Read/write |
No |
0 |
spark.adqm.partition.column |
Name of the table column used for partitioning in Spark 3. This column should be of the integer or date/time type |
Read |
No |
— |
spark.adqm.partition.count |
Number of partitions in Spark 3. Can only be specified in conjunction with |
Read |
No |
— |
spark.adqm.insert.batchsize |
Batch size when inserting into a target table |
Write |
No |
10000 |
spark.adqm.insert_distributed_sync |
Insert mode in ADQM: |
Write |
No |
0 |
spark.adqm.table.truncate |
Used for the |
Write |
No |
false |
For the overwrite mode with the spark.adqm.table.truncate option set to true and for the errorIfExists mode, there are additional options described in the table below. In other modes, these options are ignored.
| Key | Description | Required |
|---|---|---|
spark.adqm.create.table.engine |
Target table engine |
Yes |
spark.adqm.create.table.storage_policy |
Target table storage policy |
No |
spark.adqm.create.table.distributed.sharding_key |
Sharding key for a Distributed target table |
No |
spark.adqm.create.table.merge_tree.sorting_key |
Sorting key for a target table of the MergeTree family. If |
No |
spark.adqm.create.table.merge_tree.sampling_key |
Sampling key for a target table of the MergeTree family |
No |
spark.adqm.create.table.merge_tree.primary_key |
Primary key for a target table of the MergeTree family |
No |
spark.adqm.create.table.merge_tree.partition_key |
Partitioning key for a target table of the MergeTree family |
No |
spark.adqm.create.table.summing_merge_tree.columns |
Columns of a (Replicated)SummingMergeTree target table |
No |
spark.adqm.create.table.replacing_merge_tree.version |
Column with version of data records for a (Replicated)ReplacingMergeTree target table |
No |
spark.adqm.create.table.collapsing_merge_tree.sign |
Sign of a (Replicated)CollapsingMergeTree/(Replicated)VersionedCollapsingMergeTree table |
No |
spark.adqm.create.table.collapsing_merge_tree.version |
Version of a (Replicated)CollapsingMergeTree/(Replicated)VersionedCollapsingMergeTree target table |
No |