ADB Spark 3 Connector options

spark.adb.url

Database connection string

Read/write

Yes

 — 

spark.adb.dbschema

Name of the database schema to which the table belongs

Read/write

Yes

public

spark.adb.dbtable

Name of the database table

Read/write

Yes

 — 

spark.adb.driver

The full path to the JDBC driver when using a custom driver

Read/write

No

org.postgresql.Driver

spark.adb.user

User/ADB role

Read/write

Yes

 — 

spark.adb.password

User password in ADB

Read/write

No

 — 

spark.adb.server.usehostname

Allows to use the Spark 3 executor node name as a gpfdist server address

Read/write

No

false

spark.adb.server.env.name

Name of the environment variable whose value determines the name of the Spark 3 executor node or the IP address on which the gpfdist server process runs

Read/write

No

 — 

spark.adb.server.port

A port number or a port range of the gpfdist server process on the Spark 3 executor node

Read/write

No

 — 

spark.adb.pool.maxsize

Maximum number of connections in the connection pool

Read/write

No

4

spark.adb.pool.timeoutms

Time in milliseconds after which an inactive connection is considered idle

Read/write

No

10000

spark.adb.pool.minidle

Minimum number of idle connections supported in the connection pool

Read/write

No

0

spark.adb.debugmode

Enables event logging mode in the adb_spark_debug_query_log ADB table

Read/write

No

false

spark.adb.partition.column

Name of the table column used for partitioning in Spark 3. This column should be of the integer or date/time data types

Read

No

 — 

spark.adb.partition.count

Number of partitions in Spark 3. Can be specified either independently or simulataneously with spark.add.partition.column or spark.adb.partition.hash

Read

No

 — 

spark.adb.partition.hash

Expression used as partitioning key when reading data in Spark 3. Specified simultaneously with spark.adb.partition.count. This expression should return a value of the integer data type

Read

No

 — 

spark.adb.batch.enable

Enables batch mode. It is assumed, that when reading data from ADB to Spark 3, the ColumnarBatch structure is used. This structure is optimized for work with scan-type operations. This feature is experimental

Read

No

false

spark.adb.batch.memorymode

Type of data structure that is used to organizing the ColumnarBatch. Acceptable values: OFF_HEAP, ON_HEAP. Specified simultaneously with spark.adb.batch.enable

Read

No

 — 

spark.adb.table.truncate

Used when writing in the Overwrite mode. Performs the truncate table operation, if the value is true, otherwise drops the table

Write

No

false

spark.adb.create.table.with

Used when writing in the Overwrite and errorIfExists modes. Stores parameters when creating a table via the WITH expression

Write

No

 — 

spark.adb.create.table.distributedby

Used when writing in the Overwrite and errorIfExists modes. Works as distribution key when creating the target table via the DISTRIBUTED BY expression

Write

No

RANDOMLY