ADB to Kafka Connector installation

Daria Barysheva

To install ADB to Kafka Connector, follow the steps:

  1. Make sure that PXF and ADB to Kafka services are installed. For information on how to add services to ADB clusters, see Add services.

  2. Add the new custom profile kafka to the PXF configuration file /var/lib/pxf/conf/pxf-profiles.xml on Master. To edit the file, you can use the vi or vim command:

    $ sudo vi /var/lib/pxf/conf/pxf-profiles.xml

    The content of the changed file can look as follows. In our case, the file does not contain other profiles:

    <profiles>
    <profile>
        <name>kafka</name>
        <description>A profile for export data into Apache Kafka</description>
        <plugins>
            <accessor>org.greenplum.pxf.plugins.kafka.KafkaAccessor</accessor>
            <resolver>org.greenplum.pxf.plugins.kafka.KafkaResolver</resolver>
        </plugins>
        <optionMappings>
            <!-- Kafka properties -->
            <mapping option="BOOTSTRAP_SERVERS" property="kafka.bootstrap.servers"/>
            <mapping option="SECURITY_PROTOCOL" property="kafka.property.security.protocol" />
            <mapping option="SASL_MECHANISM" property="kafka.property.sasl.mechanism" />
            <mapping option="TRUSTSTORE_LOCATION" property="kafka.property.ssl.truststore.location" />
            <mapping option="TRUSTSTORE_PASSWORD" property="kafka.property.ssl.truststore.password" />
            <mapping option="KERBEROS_SERVICE_NAME" property="kafka.property.sasl.kerberos.service.name" />

            <!-- Kafka JAAS properties -->
            <mapping option="KERBEROS_PRINCIPAL" property="kafka.jaas.property.sasl.kerberos.principal" />
            <mapping option="KERBEROS_KEYTAB" property="kafka.jaas.property.sasl.kerberos.keytab" />
            <mapping option="KERBEROS_USE_KEYTAB" property="kafka.jaas.property.sasl.kerberos.useKeyTab" />
            <mapping option="KERBEROS_STORE_KEY" property="kafka.jaas.property.sasl.kerberos.storeKey" />
            <mapping option="KERBEROS_USE_TICKET_CACHE" property="kafka.jaas.property.sasl.kerberos.useTicketCache" />
            <mapping option="KERBEROS_DEBUG" property="kafka.jaas.property.sasl.kerberos.debug" />
            <mapping option="SASL_USER" property="kafka.jaas.property.sasl.user" />
            <mapping option="SASL_USER_PASSWORD" property="kafka.jaas.property.sasl.user.password" />
            <mapping option="JAAS_CONFIG_FILE" property="kafka.jaas.property.config.file" />

            <!-- General properties -->
            <mapping option="BATCH_SIZE" property="kafka.batch.size"/>
            <mapping option="KAFKA_ADMIN_CONNECTION_TIMEOUT" property="kafka.admin.close.connection.timeout"/>
            <mapping option="TOPIC_AUTO_CREATE_FLAG" property="kafka.topic.auto.create"/>
            <mapping option="AVRO_DEFAULT_DECIMAL_PRECISION" property="avro.decimal.default.precision" />
            <mapping option="AVRO_DEFAULT_DECIMAL_SCALE" property="avro.decimal.default.scale" />
        </optionMappings>
    </profile>
</profiles>
    NOTE

    The directory, where PXF configuration files are located, is defined by the $PXF_BASE environment variable. The default value is /var/lib/pxf. The path may differ in your environment.

  3. Login under the default user name gpadmin. All the commands listed in the subsequent steps should be performed on Master:

    $ sudo su - gpadmin

  4. Synchronize the PXF configuration on all hosts in the ADB cluster:

    $ pxf cluster sync

    The result:

    Syncing PXF configuration files from master host to standby master host and 2 segment hosts...
PXF configs synced successfully on 3 out of 3 hosts

  5. Restart PXF:

    $ pxf cluster restart

    The result:

    Restarting PXF on master host, standby master host, and 2 segment hosts...
PXF restarted successfully on 4 out of 4 hosts
Found a mistake? Seleсt text and press Ctrl+Enter to report it