1. Overview

Apache Kafka is a powerful platform for building real-time data pipelines and streaming applications. When running a Kafka cluster in production, we must secure the cluster by applying the security features in Apache Kafka. To access the Kafka server requiring authentication, we must configure the clients to pass the relevant credentials.

In this article, we’ll explore Kafka authentication schemes and demonstrate configuring the kafka-topics.sh script to authenticate with the Kafka server.

2. Security in Kafka

Apache Kafka offers several features that improve the security of a Kafka cluster. Primarily, the server listeners can configure one of the supported security protocols to secure the point of connection. These protocols affect how the clients authenticate themselves with the server.

2.1. Security Protocols

Apache Kafka server supports four security protocols, PLAINTEXTSSLSASL_PLAINTEXT, and SASL_SSL. The protocols mainly differ in the authentication mechanism and the presence of a secure communication channel over the network.

Firstly, the PLAINTEXT protocol offers no authentication and no encryption for data in transmission. In other words, the PLAINTEXT protocol is the option that disables all the security measures on the Kafka broker. Then, the SSL security protocol provides encryption for data in transmission by mandating the establishment of a secure communication channel between client and server using the Secure Socket Channel (SSL) protocol.

Then, the SASL_PLAINTEXT protocol enforces client authentication using the Simple Authentication and Security Layer (SASL) framework. However, the SASL_PLAINTEXT doesn’t encrypt data in transmission using SSL. Finally, the SASL_SSL combines the SASL_PLAINTEXT and SSL protocols by authenticating clients using the SASL mechanism and communicating over a secure channel using the SSL protocol.

In short, here’s how each security protocol differs from each other in terms of authentication and data encryption:

Security Protocol

Has Authentication?

Is Data Encrypted on Transmission?

PLAINTEXT

No

No

SSL

Yes, through SSL’s mTLS mechanism

Yes

SASL_PLAINTEXT

Yes, through the SASL framework

No

SASL_SSL

Yes, through the SASL framework

Yes

2.2. SASL Mechanism

When we choose the SASL framework for authentication, as in the case of SASL_PLAINTEXT and SASL_SSL, we must choose the underlying authentication mechanism. Apache Kafka supports several SASL mechanisms for authenticating the client, such as PLAINGSSAPISCRAM-SHA512, and OAUTHBEARER. These mechanisms differ in how they authenticate clients to ensure they’re who they claim to be.

For example, the PLAIN mechanism is a simple username and password authentication mechanism for verifying the client’s identity. The server maintains a list of users and their passwords and matches the credentials against this list.

*Typically, the authentication mechanism is specified as SASL/, where the MECHANISM can be any supported mechanism*. For example, we can refer to the authentication mechanism that uses the SASL framework and the PLAIN mechanism as SASL/PLAIN.

In the remainder of the tutorial, we’ll use the SASL/PLAIN authentication in our explanation and demonstration as it’s the easiest to set up. However, the same pattern can be applied regardless of the authentication mechanism used.

3. Authenticating Kafka Clients Using SASL/PLAIN

Clients connecting to any Kafka servers that enabled the SASL authentication will receive a challenge that it must respond to. In the SASL/PLAIN case, the server challenges the clients to provide a username and password to authenticate themselves. The client must send a valid credentials pair to the server in response before the server takes in the request.

Java-based clients mostly use the Java Authentication and Authorization Service (JAAS) to handle the SASL process. For example, the clients in the Apache Kafka installation’s bin directory such as the kafka-topics.sh and kafka-console-producer.sh use the JAAS for authentication purposes. Concretely, we can configure the client’s JAAS through the sasl.jaas.config property key.

Here’s how we would configure our client for authenticating with servers that enabled the SASL/PLAIN security protocol:

$ cat client-config.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="bob" password="bobpassword";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

Firstly, the PlainLoginModule is an authentication module that authenticates using the provided username and password in the same line. In the example, the client will attempt to authenticate itself as bob with the server using the password bobpassword.

Then, we run the Kafka client scripts with the –command-config options to point to the configuration file that contains the JAAS configuration. For example:

$ ./kafka-topics.sh --list --bootstrap-server localhost:9092 --command-config client-config.properties

On start-up, the client code will configure the client code using the properties in the file as specified by the –command-config option.

4. Demonstration

We’ll demonstrate the idea by setting up a Kafka server:

Kafka sasl plaintext listener

Firstly, we’ll configure a listener for our Kafka server that listens for traffic at port 9094. Then, we enable the SASL_PLAINTEXT protocol for the listener. Finally, the client, kafka-topics.sh will connect to the server through the listener.

4.1. Setting up a Kafka Server

We’ll set up a Kafka server with a listener on port 9094. Furthermore, we’ll configure the listener to use the SASL_PLAINTEXT security protocol. This setup allows us to see how to authenticate our clients against the server using the method outlined in the previous section.

Firstly, we spin up the bitnami/kafka Docker container, passing the required configurations through the environment variables:

$ cat docker-compose.yaml
version: "3"
services:
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9094:9094'
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=SECURE://:9094,CONTROLLER://:9093
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=SECURE:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT
      - KAFKA_CFG_ADVERTISED_LISTENERS=SECURE://127.0.0.1:9094
      - KAFKA_NODE_ID=1
      - KAFKA_CLIENT_USERS=bob
      - KAFKA_CLIENT_PASSWORDS=bobpassword
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@localhost:9093

We define a Docker Compose YAML file that creates a single Kafka service. Then, we configure one listener, SECURE, to listen to port 9094 using the KAFKA_CFG_LISTENERS environment variable. Additionally, we use the KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP to map our SECURE listener to the SASL_PLAINTEXT security protocol.

Finally, we hardcode the credentials using the KAFKA_CLIENT_USERS and KAFKA_CLIENT_PASSWORDS environment variables. Ultimately, we’re setting up our server such that only clients that can produce the username bob and the bobpassword can access the Kafka server through port 9094. To understand the rest of the configuration, we can refer to the official documentation.

Let’s create the service by running docker-compose up -d:

$ docker-compose up -d
[+] Running 2/2
 ✔ Network kafka-authentication_default    Created                                                                                            0.0s
 ✔ Container kafka-authentication-kafka-1  Started                                                                                            0.4s

Finally, we check that the server is properly started by checking the logs from the kafka-authentication-kafka-1 container:

$ docker logs -n 5 kafka-authentication-kafka-1
...
[2024-06-22 10:11:16,353] INFO Kafka startTimeMs: 1719051076353 (org.apache.kafka.common.utils.AppInfoParser)
[2024-06-22 10:11:16,355] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)

If we see the message Kafka Server started, we can proceed to the next step.

4.2. Connecting to Kafka Without Authentication

To test the server setup, let’s first connect to the Kafka server without JAAS configuration. Specifically, we’ll use the kafka-topics.sh script to create the Messages topic on the Kafka server:

$ ./kafka-topics.sh --create --topic Messages --bootstrap-server localhost:9094
Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2024-06-22 10:16:35,530] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
 (org.apache.kafka.tools.TopicCommand)

As expected, we’re getting an error executing the client script. Although the error message is not obvious, the time-out is caused by the client failing to authenticate with Kafka servers.

4.3. Configuring Authentication on Kafka Clients

Let’s create a file, client-config.properties to store the required authentication information:

$ cat client-config.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="bob" password="bobpassword"; 
security.protocol=SASL_PLAINTEXT 
sasl.mechanism=PLAIN

We configure our client to use PlainLoginModule to handle the authentication request. Additionally, we pass the bob and bobpassword as the username and password values to match the user and password we’ve set on the server.

We now run the same command with kafka-topics.sh again, adding the –command-config option to supply the additional configuration:

$ ./kafka-topics.sh --create --topic Messages --bootstrap-server localhost:9094 --command-config
 client-config.properties
Created topic Messages.
$ ./kafka-topics.sh --list --bootstrap-server localhost:9094 --command-config
Messages

As we can see, our client successfully created the topic Messages on the Kafka cluster. This is only possible with the correct authentication information passed to the server.

5. Conclusion

In this tutorial, we’ve first learned about the security mechanism in Apache Kafka. Specifically, we’ve seen that Kafka supports the SASL_PLAINTEXT protocol that requires clients to authenticate themselves to connect to the server. Then, we learned that the Kafka clients such as kafka-topics.sh take in the authentication information through the property key sasl.jaas.config.

Subsequently, we’ve set up a simple Kafka server that uses the SASL_PLAINTEXT.  Finally, we’ve demonstrated passing the authentication information using the sasl.jaas.config to the kafka-topics.sh script.