ADQM Spark 3 Connector options

Parameters for connecting to an ADQM database and read/write parameters are set via key/value pairs.

The following options are available for ADQM Spark 3 Connector.

ADQM Spark 3 Connector options
Key Description Operation type Required Default value

spark.adqm.url

Database connection string via a driver that uses the TCP protocol

Read/write

Yes

 — 

spark.adqm.dbschema

Name of the database schema to which the table belongs

Read/write

Yes

default

spark.adqm.dbtable

Name of the database table

Read/write

Yes

 — 

spark.adqm.user

ADQM user/role

Read/write

Yes

 — 

spark.adqm.password

ADQM user password

Read/write

No

 — 

spark.adqm.cluster.name

ADQM cluster name

Read/write

No

 — 

spark.adqm.jdbc.pool.maxsize

Maximum number of connections in the connection pool

Read/write

No

4

spark.adqm.jdbc.pool.timeout

Time in milliseconds after which an inactive connection is considered idle

Read/write

No

10000

spark.adqm.jdbc.pool.minidle

Minimum number of idle connections supported in the connection pool

Read/write

No

0

spark.adqm.partition.column

Name of the table column used for partitioning in Spark 3. This column should be of the integer or date/time type

Read

No

 — 

spark.adqm.partition.count

Number of partitions in Spark 3. Can only be specified in conjunction with spark.adqm.partition.column

Read

No

 — 

spark.adqm.insert.batchsize

Batch size when inserting into a target table

Write

No

10000

spark.adqm.insert_distributed_sync

Insert mode in ADQM: 0 — insert in the asynchronous mode; 1 — insert in the synchronous mode

Write

No

0

spark.adqm.table.truncate

Used for the overwrite writing mode. Performs the truncate table operation, if the value is true, otherwise — drops the table

Write

No

false

For the overwrite mode with the spark.adqm.table.truncate option set to true and for the errorIfExists mode, there are additional options described in the table below. In other modes, these options are ignored.

ADQM Spark 3 Connector additional options
Key Description Required

spark.adqm.create.table.engine

Target table engine

Yes

spark.adqm.create.table.storage_policy

Target table storage policy

No

spark.adqm.create.table.distributed.sharding_key

Sharding key for a Distributed target table

No

spark.adqm.create.table.merge_tree.sorting_key

Sorting key for a target table of the MergeTree family. If spark.adqm.create.table.merge_tree.primary_key is specified, the value is primaryKey, otherwise — tuple()

No

spark.adqm.create.table.merge_tree.sampling_key

Sampling key for a target table of the MergeTree family

No

spark.adqm.create.table.merge_tree.primary_key

Primary key for a target table of the MergeTree family

No

spark.adqm.create.table.merge_tree.partition_key

Partitioning key for a target table of the MergeTree family

No

spark.adqm.create.table.summing_merge_tree.columns

Columns of a (Replicated)SummingMergeTree target table

No

spark.adqm.create.table.replacing_merge_tree.version

Column with version of data records for a (Replicated)ReplacingMergeTree target table

No

spark.adqm.create.table.collapsing_merge_tree.sign

Sign of a (Replicated)CollapsingMergeTree/(Replicated)VersionedCollapsingMergeTree table

No

spark.adqm.create.table.collapsing_merge_tree.version

Version of a (Replicated)CollapsingMergeTree/(Replicated)VersionedCollapsingMergeTree target table

No

Found a mistake? Seleсt text and press Ctrl+Enter to report it