Kafka REST Proxy overview
Kafka REST Proxy features
Kafka REST Proxy is a special proxy server created by Confluent. Kafka REST Proxy provides a RESTful interface to interact with Kafka clusters, that allows users to write and read messages via HTTP/HTTPS requests.
Interaction between external systems and Kafka brokers is done by default via a specific set of API and CLI tools provided by the Java client library. Also client libraries are supported for some programming languages.
Kafka REST Proxy helps you overcome the limitations associated with the narrow choice of ways to interact with Kafka. Any application, including not based on the JVM, can be configured to work with a Kafka cluster, regardless of programming language, as long as the application has the ability to send HTTP/HTTPS requests.
The main functionality of Kafka REST Proxy is listed below:
-
The HTTP RESTful interface allows clients to interact with Kafka using standard HTTP methods, thereby ensuring compatibility with various programming languages.
-
Interaction with Kafka API via a proxy server is provided:
-
creating and sending messages to Kafka topics in various formats;
-
subscription to topics and reading messages in real time;
-
obtaining data about the cluster and topics.
-
-
Integration with Schema Registry for creating and developing data schemas is supported.
-
Authentication and authorization mechanisms are supported to protect access to Kafka clusters.
Kafka REST Proxy architecture
The figure below shows the architecture of Kafka REST Proxy.
HTTP requests to Kafka REST Proxy
Each Kafka REST Proxy instance is a separate proxy server, which is an additional layer between the client and the Kafka broker.
The client makes an HTTP request to the Kafka REST Proxy server’s REST API using the IP address of the Kafka REST Proxy server, not the Kafka broker.
A write request is a POST method that specifies metadata (the location where the message will be stored) and a request body with the contents of the message.
A request to retrieve data from Kafka is a GET method specifying metadata (the location from which the message should be read).
|
NOTE
For examples of requests to the Kafka REST Proxy server, you can refer to the Kafka REST Proxy usage example article.
|
The Kafka REST Proxy server converts received requests into Kafka API application instances (Producer API, Consumer API, or Admin API) and returns the response generated by the application’s execution to the user.
Serialization and deserialization
To perform data serialization and deserialization, the Kafka REST Proxy server connects to the Schema Registry server’s REST API and writes a new one or retrieves an existing schema.
For this purpose, special headers are used in user requests to the Kafka REST Proxy server:
-
Content-Type— specifies the type of content that should be written to Kafka. -
Accept— specifies the type of content that the client can understand.
These headers indicate content type in the format used by the Schema Registry API.
For example, below is an example query that writes data to a Kafka topic and also registers a schema in the Schema Registry:
$ curl -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" \
-H "Accept: application/vnd.kafka.v2+json" \
--data '{"value_schema": "{\"type\": \"record\", \"name\": \"User\", \"fields\": [{\"name\": \"name\", \"type\": \"string\"}]}", "records": [{"value": {"name": "testUser"}}]}' \
"http://localhost:8082/topics/avrotest"Security
On the Kafka REST Proxy server side, authentication is configured for two connections:
-
Connection between client and Kafka REST Proxy server.
This connection is configured with basic HTTP authentication using a pair
<username>:<password>. User authentication data is passed to the Kafka REST Proxy in the HTTP request using theAuthorizationheader.To enable mutual SSL/TLS authentication, configure:
-
support HTTPS for the Kafka REST Proxy server;
-
Kafka REST Proxy truststore for validating incoming client certificates.
-
-
Connection between Kafka REST Proxy and Kafka brokers.
If Kafka REST Proxy is used to communicate with a Kafka broker that is already configured with SASL authentication, for each Kafka REST Proxy server, you need to configure the appropriate parameters to authenticate the Kafka REST Proxy user in Kafka.
If SSL/TLS authentication is configured for the Kafka broker, for each Kafka REST Proxy server, you need to configure the SSL parameters to authenticate the Kafka REST Proxy user in Kafka.
Kafka REST Proxy limitations
When using Kafka REST Proxy in data processing systems, you should remember the following limitations:
-
Requests to write messages to several topics simultaneously are not supported — each HTTP request from the manufacturer can only access one topic or partition.
-
Most producer and consumer configuration overrides are not available using HTTP requests.
-
It is not possible to use different serializers for a key and a value in a message.
-
Requests to Kafka REST Proxy are processed using a synchronous request-response protocol. At the same time, Kafka can connect up to tens of parallel instances of Kafka REST Proxy, which is enough for different use cases.
Kafka REST Proxy in ADS
Connection
After addition and installation of the Kafka REST Proxy service as part of an ADS cluster, the request to the Kafka REST Proxy server must specify the host where the Kafka REST Proxy component is located, and the port specified as the rest_listener_port parameter (default 8082) in the Main group on the configuration page of the Kafka REST Proxy service page.
Kafka
In ADS, the Kafka REST Proxy service can be installed only after installing the Kafka service. After installing Kafka REST Proxy, the bootstrap.servers parameter in the /etc/kafka-rest/kafka-rest.properties configuration file is automatically set to communicate with the Kafka broker.
Schema Registry
The Schema Registry service, installed in ADS simultaneously with the Kafka REST Proxy service, allows you to serialize and deserialize data when working with Kafka topics via the RESTful interface. After installing both services, the schema.registry.url parameter in the /etc/kafka-rest/kafka-rest.properties configuration file is automatically set.
Security
Most security options for Kafka REST Proxy can be configured using the ADCM interface:
-
In the ADS cluster, basic HTTP authentication for Kafka REST Proxy can be configured by enabling Basic authentication.
-
In the ADS cluster, SASL authentication is enabled using the Manage Kerberos action or by running the SASL PLAINTEXT mechanism.
-
In the ADS cluster, the kafka-rest-security special module provides the ability to authenticate in Kafka with different principals when connecting to Kafka via Kafka REST Proxy after enabling SASL authentication.
-
In an ADS cluster, SSL authentication is enabled using the Manage SSL action.
Configure Kafka REST Proxy
Configuring Kafka REST Proxy parameters in the ADCM interface is performed on the configuration page of the Kafka REST Proxy service.
To configure the parameters of the /etc/kafka-rest/kafka-rest.properties configuration files, set the Show advanced switch to active, expand the kafka-rest.properties node, and enter new values for the parameters. To change Kafka REST Proxy parameters that are not available in the ADCM interface, use the Add key,value field. Select Add property and enter the name of the parameter and its value.
After changing the parameters using the ADCM interface, restart the Kafka REST Proxy service. To do this, apply the Restart action by clicking 
in the Actions column.