Kafka configuration instance - properties file - error connecting to external Kafka

General
, , , ,
1

Hello all,

I am getting an error when trying to create a Kafka instance to connect to an external Kafka broker.

Pega 8.7.3

Need to use username/password.

I create new Kafka instance, enter Host/Port, upload Configuration properties file, click Test connectivity.

After 60 seconds of wait cursor - I get “Connection failed Error connection to Kafka..” (out of a java step in Data-Admin-Kafka.pzCheckConnection).

I’ve tried different versions of the properties file - here is one:

sasl.mechanism=PLAIN security.protocol=SASL_PLAINTEXT sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username=“ggggg” \ password=“ggggg”;

Also tried:
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=“gggg” password=“gggg”;
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

No matter what I pass in username/password - it spins one minute and shows that error (it even says Timeout expired while fetching topic metadata).

Am I using incorrect format or properties in that configuration file?

Thank you in advance,

Bogdan

2

@BogdanD16582931 I can see that you logged support ticket INC-270302 for this.

The investigation showed you were introducing a Data-Admin-Kafka rule which used SASL authentication, but while testing the connectivity a generic message was happening suggesting a timeout scenario or error while collecting the topics metadata, which indicated that either authentication was failing or some parameters that the external brokers were expecting was not being passed as part of the client properties on Pega side.

Pega GCS provided a standalone Kafka application that has maximum level of debugging enabled on all Apache Kafka classes. Using this approach we were able to identify that the connection was happening but it was failing at the very first stage of SASL authentication challenges (SEND_APIVERSIONS_REQUEST) and then it was getting immediately disconnected.

The issue was resolved by introducing additional Kafka client properties which were expected on external Kafka side (SSL encryption with truststore certificate) and modifying the security protocol (from SASL_PLAINTEXT to SASL_SSL).

You responded that the connect issue was resolved - solution was to use SASL_SSL security protocol and specify a truststore with valid certificate.

I will mark this issue with the above Accepted Solution.

3

@MarijeSchillern Can you share the property files that you used at the end?

4

@AleMax225

You can refer to below DSS or prconfig defined at Pegasystems Documentation

Pega-Engine • prconfig/services/stream/encryption/sasl/jaas/config/default RS:Pega-ProcessCommander

Pega-Engine • prconfig/services/stream/name/pattern/default RS:Pega-ProcessCommander

Sample Value: topicname-env-{stream.name}
Dev: topicname-dev-{stream.name}
QA: topicname-qa-{stream.name}
Prod: topicname-prod-{stream.name}

Pega-Engine • prconfig/services/stream/broker/url/default RS:Pega-ProcessCommander

Pega-Engine • prconfig/services/stream/provider/url/default RS:Pega-ProcessCommander

Pega-Engine • prconfig/services/stream/encryption/sasl/mechanism/default RS:Pega-ProcessCommander

Pega-Engine • prconfig/services/stream/encryption/security/protocol/default

Pega-Engine • prconfig/services/stream/external/replication/factor/default
Default Value: 3

Pega-Engine • prconfig/services/stream/provider/default
Default Value: ExternalKafka

5

@SUMAN_GUMUDAVELLY we have already configured the external Kafka, here the problem is with a new Kafka Instance: Records > SysAdmin > Kafka, the related documentation can be seen here: Creating a Kafka configuration instance (pega.com). I am interested in having the client properties configuration file used by @BogdanD16582931 to solve the issue above.

6

@AleMax225

So you do not want External Stream and Looking for Kafka data instance connection.

For confluent, please use below content for your property file.

sasl.mechanism=PLAIN
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required
username=“UserName”
password=“password”;

If you are connecting Hadoop based kafka, then use following properties,

security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka
sasl.jaas.config=com.ibm.security.auth.module.Krb5LoginModule required
credsType=both
useDefaultCcache=false
useKeytab=“location of .keytab file”
principal=“haddop principle”;