Home>Articles>Confluent Kafka Security: Security & SSL Setup in Confluent Kafka
Confluent Kafka SSL configuration
Articles Confluent Kafka Kafka Kafka Security

Confluent Kafka Security: Security & SSL Setup in Confluent Kafka

Confluent Kafka security configuration

Secure Socket Layer (SSL) is a security protocol for the transport layer. In SSL Protocol data is divided into fragments. The fragments are compressed and encrypted Message Authentication Code (MAC) generated by algorithms like Secure Hash Protocol(SHA) and MD5(Message Digest) is appended. SSL is the predecessor of Transport Layer Security(TLS) .After encryption of data, finally the SSL header is appended to the data.

By default, Confluent Kafka communicates with an unsecured plaintext protocol over 9092 port.

Confluent kafka security supports SSL security protocol in intra broker and client communications. Each broker authenticates other brokers and the clients. Brokers and the clients both authenticate each other (2 way authentication). One can also use SSL with SASL security, hit the reference section for Confluent kafka sasl.

Note: Enabling SSL (TLS) in Confluent Kafka security would override the zero-copy optimization in Kafka consumers. In the Zero-copy mechanism, data is copied into pagecache only once and reused on each consumption instead of being stored in memory and copied out to user-space every time it is read. This allows messages to be consumed at a rate that approaches the limit of the network connection.

For a successful handshake:

  • Each broker should have its own private-key/certificate pair, and the client uses the certificate to authenticate the broker.
  • Each client should have a private-key/certificate pair if client authentication is enabled*, and the broker uses the certificate to authenticate the client.

*Client authentication is optional but recommended. The tradeoff of having it is, performance implications but not having it can allow non-authenticated brokers (impersonation) to communicate to clients.

 

Steps to enable SSL protocol in Kafka Brokers to enable Confluent Kafka Security:

Note: Below steps do not include steps for creating SSL certificates & keys, refer to this link.

  • Update the server.properties with below corresponding configurations.

 

ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks

ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks

ssl.truststore.password=secret123

ssl.keystore.password=secret123

ssl.key.password=secret123

 

  • Set the below config in server.properties for enabling inter-broker SSL communication.

security.inter.broker.protocol=SSL

 

  • Now update the listeners and advertised listeners (If different) in the server.properties. Generally we use 9093 port for SSL as 9092 is default for plaintext.

listeners=SSL://kafkabroker1:9093

advertised.listeners=SSL://localhost:9093

 

For continuing plaintext protocol as well along with SSL use below.

listeners=PLAINTEXT://kafkabroker1:9092,SSL://kafkabroker1:9093

advertised.listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093

 

Once the SSL port works fine, we can remove/disable the 9092 plaintext port.

 

  • ¬†For enabling client authentication (2 way authentication) the below configuration must be enabled.

ssl.client.auth=required

 

  • On the client side, we are required to have both certificates and keys. Below configuration is to be mentioned in all clients communicating with SSL enabled Kafka cluster.

These configs can be put in a file and reused with regular client application files.

bootstrap.servers=kafkabroker1:9093

security.protocol=SSL

ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks

ssl.truststore.password=secret123

ssl.keystore.location=/var/private/ssl/kafka.client.keystore.jks

ssl.keystore.password=secret123

ssl.key.password=secret123

Note: ssl.truststore.password  value is optional in both broker and client configuration but is strongly recommended for integrity checking.

Examples of Kafka clients with SSL meta configuration in Confluent Kafka Security are below.

  • ${home}/bin/kafka-console-producer –broker-list kafkabroker1:9093 –topic test_topic –producer.config client-ssl.properties
  • ${home}/bin/kafka-console-consumer –bootstrap-server kafkabroker1:9093 –topic test_topic –consumer.config client-ssl.properties –from-beginning
  • For JVM based producers & consumers below export can be included in start scripts, in command line or set as environment variable for dedicated machines.

export KAFKA_OPTS=”-Djavax.net.ssl.trustStore=/path/to/truststore.jks -Djavax.net.ssl.trustStoreType=jks -Djavax.net.ssl.trustStorePassword=<password> -Djavax.net.ssl.keyStore=/path/to/keystore.jks -Djavax.net.ssl.keyStoreType=jks -Djavax.net.ssl.keyStorePassword=<password>”

Some additional references:

Kafka with SSL

Encryption with SSL | Confluent Security

SSL with SASL authentication

Leave a Reply

Your email address will not be published. Required fields are marked *