Spark connector

Spark connector provides the possibility of high-speed, parallel data exchange between Spark and Arenadata Database. You can find full product description on the ADB Spark connector page.

Using the Spark connector

The package contains a license text file and a jar file. The package file has the following name pattern: adb-spark-connector_<scala_version>-<adb-connector-version>.tar.gz. For example, adb-spark-connector_2.11-1.0.tar.gz.

There is no need to set up Spark connector additionally for ADH. The jar file of the Spark connector should be placed in the libraries directory of Spark itself, which is /usr/lib/spark/jars. You need to put the adb-spark-connector_2.11-1.0.tar.gz file into this directory.

When you want to use this jar with your project, you need to specify it in the dependencies of your project.

  • Maven

  • Ivy

  • Gradle

  • Sbt

<dependency org="io.arenadata.spark" name="adb-spark-connector_2.11" rev="1.0.4-spark-2.3.x">
    <artifact name="adb-spark-connector_2.11" ext="jar"/>
compile(group: 'io.arenadata.spark', name: 'adb-spark-connector_2.11', version: '1.0.4-spark-2.3.x')
libraryDependencies += "io.arenadata.spark" % "adb-spark-connector_2.11" % "1.0.4-spark-2.3.x"

After you have specified all dependencies, you can run your Spark application as usual.

Found a mistake? Seleсt text and press Ctrl+Enter to report it