Kafka Connect Framework is a robust abstraction of Kafka producers and Consumers with high availability and distributed operation. That means, this piece of cake is what organisations need. These are special JVM applications which manage the underlying server resources. Developers and architects need not worry about writing too much custom code in this case to integrate Kafka to external systems in most cases.
A quick byte about Kafka Connect Security
For getting started with Connecters hit the Confluent Hub link. There are lot of stuff they have got. (like a super store for connectors!!)
Now bringing in data from external systems or inter-department systems in an organisation, requires to comply with some standard security protocol. Or even if it’s not required still, you wouldn’t want your data flowing through the wire in plaintext waiting to be intercepted and accessed by unauthorized entities.
To make Kafka Connect Security enterprise ready, it supports multiple security standards. You don’t have to squeeze in extra efforts to remain compliant. Here we will be discussing basic Kafka Connect security to begin with.
Secure Socket Layer (SSL) is a security protocol for the transport layer. In SSL Protocol data is divided into fragments. The fragments are compressed and encrypted Message Authentication Code (MAC) generated by algorithms like Secure Hash Protocol(SHA) and MD5(Message Digest) is appended. SSL is the predecessor of Transport Layer Security(TLS) .After encryption of data, finally the SSL header is appended to the data. Read all about confluent kafka security configuration.
Kafka connect comprise of the following components:
- Worker – Basically a JVM process, abstracts server resources.
- Connector – Simply a configuration to communicate and establish a channel to copy data to and from external systems.
- Tasks – Multi initialization of connectors to handle transfers parallely.
Now these workers can host both sink and source connectors therefore, respective configuration would need entity prefixed. For configuring Kafka connect SSL, below are some kafka connect ssl examples.
Kafka Connect configuration for Meta-data topics
For Sink connectors
Kafka connect workers expose functionality via REST APIs, these are used to register, delete, pause, resume, validate etc. (Almost everything and that’s awesome !!) Therefore, there might be a need to secure these with HTTPS SSL.
Additionally, the connect framework’s rest APIs can be secured with SSL. (But this severely affects administration efforts)
SASL Plain authentication
SASL Plain is the base authentication, easy to implement and provides bare minimum enterprise-grade security to both Clients and services. It is a simple username/password based authentication used with SSL (Earlier called TLS). Both combined provide encryption and authentication. (isn’t it cool !!) Plain should not be confused with plaintext or no-SSL. It is highly recommended to use SSL with SASL PLAIN.
Below are the steps for enabling clients to communicate via SASL SSL PLAIN mechanism.
producer.sasl.mechanism=PLAIN (In case of producer client)
consumer.sasl.mechanism=PLAIN (In case of consumer client)
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
Create a JAAS file (Used for automated authentication) and include below lines.
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
There are other SASL protocols available for securing Kafka