Connect to Secure Kafka Cluster

To manage event stream processing tasks, Streaming Data Framework for MATLAB^® Production Server™ requires configuration information. For example, to connect to a secure Kafka^® cluster, the framework must know the security protocol and the SSL certificate to use. You provide this information by setting provider properties when creating the stream connector object. After creating the object, configuration properties are read-only. These properties are used during desktop development and then collected for deployment into production.

You can provide configuration information using two types of properties of the stream and stream processing objects:

• Named object properties — Properties required to create the MATLAB objects that interact with the stream, such as the ConnectionTimeout property of the KafkaStream object.

• Third-party provider properties — Properties that are not properties of the MATLAB objects in the streaming data framework, such as the retention.ms Kafka property or properties such as security.protocol and ssl.truststore.type that are required to connect to a secure Kafka cluster.

Kafka Provider Properties

When you create a KafkaStream object to connect to a Kafka host, specify Kafka provider properties and their corresponding values using one or more propname,propval input argument pairs. Use single-quotes or double-quotes around propname. You can specify several properties and their values in any order as propname1,propval1,...,propnameN,propvalN. For example, kafkaStream(host,port,topic,"sasl.mechanism","SCRAM-SHA-512") sets the Kafka property sasl.mechanism to SCRAM-SHA-512. For a complete list of Kafka properties, see Kafka Configuration in the Kafka documentation. The streaming framework provides a pass-through mechanism for these properties, where they are passed directly to the Kafka configuration mechanism without any validation.

Connect to Secure Kafka Cluster

When creating an object to connect to a secure Kafka cluster, the Kafka properties that you specify differ based on these factors:

• Whether the Kafka cluster is secured using TLS or SASL

• Whether you use the object to read from the stream or write to the stream

• Whether when using the object to read, you set the Order property of a KafkaStream object to "EventTime" or "IngestTime".

Read Events from SSL-Secured Kafka Cluster

Specify the following Kafka properties when creating an object to read from the Kafka stream.

• security.protocol — Set the security protocol to SSL.

• ssl.truststore.type — Set the file format of the truststore file to SSL or JKS.

• ssl.truststore.location — If your server certificate is not present in your system truststore, set the location of the truststore file.

For example, the following syntax creates an object to read events from a recamanSum_data topic on a Kafka host located at network address kafka.host.com:9093 in an SSL-secured cluster.

ks_read = kafkaStream("kafka.host.com",9093,"recamanSum_data", ... 
      "security.protocol","SSL","ssl.truststore.type","PEM", ...
      "ssl.truststore.location","mps-kafka.pem")

Write Events to SSL-Secured Kafka Cluster

Specify the following Kafka properties when creating an object to write to the stream or to read from the stream when Order="IngestTime".

• security.protocol — Set the security protocol to SSL.

• ssl.ca.location — Set the location of the certificate authority (CA) root certificate.

For example, the following syntax creates an object to write events to a recamanSum_results topic on a Kafka host located at the network address kafka.host.com:9093 in an SSL-secured cluster.

outKS = kafkaStream("kafka.host.com",9093,"recamanSum_results", ... 
      "security.protocol","SSL", ...
      "ssl.ca.location","my-ssl-cert.pem");

Read Events from SASL-Secured Kafka Cluster

To create an object to read from a SASL-secured Kafka cluster, setting the sasl.jaas.config Kafka property is required. The value of the sasl.jaas.config property is long, structured, and difficult to type. To make it easier to provide the sasl.jaas.config value, the framework provides two properties, sasl.user and sasl.password, that you can set instead. The framework synthesizes the value for the sasl.jaas.config property using the values of sasl.user, sasl.password, security.protocol, and sasl.mechanism.

Specify the following Kafka properties when creating an object to read from the stream.

• security.protocol — Set the security protocol to SASL.

• ssl.truststore.type — Set the file format of the truststore file to SSL or JKS.

• ssl.truststore.location — If your server certificate is not present in your system truststore, set the location of the truststore file.

• sasl.mechanism — Set the SASL mechanism used for client connections.

• sasl.user — Set the SASL-authorized username.

• sasl.password — Set the SASL password for sasl.user.

For example, the following syntax creates an object to read events from the recamanSum_data topic on a Kafka host located at the network address kafka.host.com:9093 in a SASL-secured cluster.

inKS_sasl = kafkaStream("kafka.host.com",9093,"recamanSum_data", ...
      "security.protocol","SASL_SSL", 
      "ssl.truststore.type","PEM",...
      "ssl.truststore.location","my-ssl-cert.pem", ...
      "sasl.mechanism","SCRAM-SHA-512", ...
      "sasl.user","sasl-consumer", ...
      "sasl.password","apachekafka")

Write Events to SASL-Secured Kafka Cluster

Specify the following Kafka properties when creating an object to write to the stream or to read from the stream when Order="IngestTime".

• security.protocol — Set the security protocol to SASL.

• ssl.ca.location — Set the location of the CA root certificate.

• sasl.mechanism — Set the SASL mechanism used for client connections.

• sasl.user — Set the SASL-authorized username.

• sasl.password — Set the SASL password for sasl.user.

For example, the following syntax creates an object to write events to the recamanSum_results topic on a Kafka host located at the network address kafka.host.com:9093 in a SASL-secured cluster.

outKS_sasl = kafkaStream("kafka.host.com",9093,"recamanSum_results", ...
        "security.protocol","SASL_SSL", ...
        "ssl.ca.location","my-ssl-cert.pem", ...
        "sasl.mechanism","SCRAM-SHA-512", ...
        "sasl.user","sasl-producer", ...
        "sasl.password","apachekafka")

Client-Side Authentication

To enable client-side authentication, you must set the ssl.keystore.location property to the location of your client certificate, the certificate the client must send to the server. If your server or client certificates are password protected, you might also need to set the ssl.truststore.password property and the ssl.keystore.password property.