1. Overview
Apache Kafka is a powerful platform for building real-time data pipelines and streaming applications. When running a Kafka cluster in production, we must secure the cluster by applying the security features in Apache Kafka. To access the Kafka server requiring authentication, we must configure the clients to pass the relevant credentials.
In this article, we’ll explore Kafka authentication schemes and demonstrate configuring the kafka-topics.sh script to authenticate with the Kafka server.
2. Security in Kafka
Apache Kafka offers several features that improve the security of a Kafka cluster. Primarily, the server listeners can configure one of the supported security protocols to secure the point of connection. These protocols affect how the clients authenticate themselves with the server.
2.1. Security Protocols
Apache Kafka server supports four security protocols, PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL. The protocols mainly differ in the authentication mechanism and the presence of a secure communication channel over the network.
Firstly, the PLAINTEXT protocol offers no authentication and no encryption for data in transmission. In other words, the PLAINTEXT protocol is the option that disables all the security measures on the Kafka broker. Then, the SSL security protocol provides encryption for data in transmission by mandating the establishment of a secure communication channel between client and server using the Secure Socket Channel (SSL) protocol.
Then, the SASL_PLAINTEXT protocol enforces client authentication using the Simple Authentication and Security Layer (SASL) framework. However, the SASL_PLAINTEXT doesn’t encrypt data in transmission using SSL. Finally, the SASL_SSL combines the SASL_PLAINTEXT and SSL protocols by authenticating clients using the SASL mechanism and communicating over a secure channel using the SSL protocol.
In short, here’s how each security protocol differs from each other in terms of authentication and data encryption:
Security Protocol
Has Authentication?
Is Data Encrypted on Transmission?
PLAINTEXT
No
No
SSL
Yes, through SSL’s mTLS mechanism
Yes
SASL_PLAINTEXT
Yes, through the SASL framework
No
SASL_SSL
Yes, through the SASL framework
Yes
2.2. SASL Mechanism
When we choose the SASL framework for authentication, as in the case of SASL_PLAINTEXT and SASL_SSL, we must choose the underlying authentication mechanism. Apache Kafka supports several SASL mechanisms for authenticating the client, such as PLAIN, GSSAPI, SCRAM-SHA512, and OAUTHBEARER. These mechanisms differ in how they authenticate clients to ensure they’re who they claim to be.
For example, the PLAIN mechanism is a simple username and password authentication mechanism for verifying the client’s identity. The server maintains a list of users and their passwords and matches the credentials against this list.
*Typically, the authentication mechanism is specified as SASL/
In the remainder of the tutorial, we’ll use the SASL/PLAIN authentication in our explanation and demonstration as it’s the easiest to set up. However, the same pattern can be applied regardless of the authentication mechanism used.
3. Authenticating Kafka Clients Using SASL/PLAIN
Clients connecting to any Kafka servers that enabled the SASL authentication will receive a challenge that it must respond to. In the SASL/PLAIN case, the server challenges the clients to provide a username and password to authenticate themselves. The client must send a valid credentials pair to the server in response before the server takes in the request.
Java-based clients mostly use the Java Authentication and Authorization Service (JAAS) to handle the SASL process. For example, the clients in the Apache Kafka installation’s bin directory such as the kafka-topics.sh and kafka-console-producer.sh use the JAAS for authentication purposes. Concretely, we can configure the client’s JAAS through the sasl.jaas.config property key.
Here’s how we would configure our client for authenticating with servers that enabled the SASL/PLAIN security protocol:
$ cat client-config.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="bob" password="bobpassword";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
Firstly, the PlainLoginModule is an authentication module that authenticates using the provided username and password in the same line. In the example, the client will attempt to authenticate itself as bob with the server using the password bobpassword.
Then, we run the Kafka client scripts with the –command-config options to point to the configuration file that contains the JAAS configuration. For example:
$ ./kafka-topics.sh --list --bootstrap-server localhost:9092 --command-config client-config.properties
On start-up, the client code will configure the client code using the properties in the file as specified by the –command-config option.
4. Demonstration
We’ll demonstrate the idea by setting up a Kafka server:
Firstly, we’ll configure a listener for our Kafka server that listens for traffic at port 9094. Then, we enable the SASL_PLAINTEXT protocol for the listener. Finally, the client, kafka-topics.sh will connect to the server through the listener.
4.1. Setting up a Kafka Server
We’ll set up a Kafka server with a listener on port 9094. Furthermore, we’ll configure the listener to use the SASL_PLAINTEXT security protocol. This setup allows us to see how to authenticate our clients against the server using the method outlined in the previous section.
Firstly, we spin up the bitnami/kafka Docker container, passing the required configurations through the environment variables:
$ cat docker-compose.yaml
version: "3"
services:
kafka:
image: 'bitnami/kafka:latest'
ports:
- '9094:9094'
environment:
- KAFKA_ENABLE_KRAFT=yes
- KAFKA_CFG_PROCESS_ROLES=broker,controller
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_CFG_LISTENERS=SECURE://:9094,CONTROLLER://:9093
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=SECURE:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT
- KAFKA_CFG_ADVERTISED_LISTENERS=SECURE://127.0.0.1:9094
- KAFKA_NODE_ID=1
- KAFKA_CLIENT_USERS=bob
- KAFKA_CLIENT_PASSWORDS=bobpassword
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@localhost:9093
We define a Docker Compose YAML file that creates a single Kafka service. Then, we configure one listener, SECURE, to listen to port 9094 using the KAFKA_CFG_LISTENERS environment variable. Additionally, we use the KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP to map our SECURE listener to the SASL_PLAINTEXT security protocol.
Finally, we hardcode the credentials using the KAFKA_CLIENT_USERS and KAFKA_CLIENT_PASSWORDS environment variables. Ultimately, we’re setting up our server such that only clients that can produce the username bob and the bobpassword can access the Kafka server through port 9094. To understand the rest of the configuration, we can refer to the official documentation.
Let’s create the service by running docker-compose up -d:
$ docker-compose up -d
[+] Running 2/2
✔ Network kafka-authentication_default Created 0.0s
✔ Container kafka-authentication-kafka-1 Started 0.4s
Finally, we check that the server is properly started by checking the logs from the kafka-authentication-kafka-1 container:
$ docker logs -n 5 kafka-authentication-kafka-1
...
[2024-06-22 10:11:16,353] INFO Kafka startTimeMs: 1719051076353 (org.apache.kafka.common.utils.AppInfoParser)
[2024-06-22 10:11:16,355] INFO [KafkaRaftServer nodeId=1] Kafka Server started (kafka.server.KafkaRaftServer)
If we see the message Kafka Server started, we can proceed to the next step.
4.2. Connecting to Kafka Without Authentication
To test the server setup, let’s first connect to the Kafka server without JAAS configuration. Specifically, we’ll use the kafka-topics.sh script to create the Messages topic on the Kafka server:
$ ./kafka-topics.sh --create --topic Messages --bootstrap-server localhost:9094
Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2024-06-22 10:16:35,530] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listTopics
(org.apache.kafka.tools.TopicCommand)
As expected, we’re getting an error executing the client script. Although the error message is not obvious, the time-out is caused by the client failing to authenticate with Kafka servers.
4.3. Configuring Authentication on Kafka Clients
Let’s create a file, client-config.properties to store the required authentication information:
$ cat client-config.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="bob" password="bobpassword";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
We configure our client to use PlainLoginModule to handle the authentication request. Additionally, we pass the bob and bobpassword as the username and password values to match the user and password we’ve set on the server.
We now run the same command with kafka-topics.sh again, adding the –command-config option to supply the additional configuration:
$ ./kafka-topics.sh --create --topic Messages --bootstrap-server localhost:9094 --command-config
client-config.properties
Created topic Messages.
$ ./kafka-topics.sh --list --bootstrap-server localhost:9094 --command-config
Messages
As we can see, our client successfully created the topic Messages on the Kafka cluster. This is only possible with the correct authentication information passed to the server.
5. Conclusion
In this tutorial, we’ve first learned about the security mechanism in Apache Kafka. Specifically, we’ve seen that Kafka supports the SASL_PLAINTEXT protocol that requires clients to authenticate themselves to connect to the server. Then, we learned that the Kafka clients such as kafka-topics.sh take in the authentication information through the property key sasl.jaas.config.
Subsequently, we’ve set up a simple Kafka server that uses the SASL_PLAINTEXT. Finally, we’ve demonstrated passing the authentication information using the sasl.jaas.config to the kafka-topics.sh script.