Error by switching from internal to external Kafka service

General
, , ,
1

Hello,

I facing an issue by switching from an internal to external managed service.

When I configure the external kafka the pods for backend and frontend throws some exceptions see attachments.

Are there special configurations for the kafka services which pega needs?

The connection is configured in the helmchart, I’ve only use the following options for the connecion

stream:

enabled: true

bootstrapServer: “<domain_to_managed_kafka>”

Attached you find the logs of the two above mentioned pods.

Thank you

webpod_error_logs.txt (164 KB)

batchpod error logs.txt (44.9 KB)

2

@JulianB70 `can you tell us what research you’ve carried out so far?

Which documentation have you followed so far?

Externalization of services in your deployment

Externalization for a Kafka service

Were any of the previous questions on the forum useful?

Configure External Kafka as a stream service

Externalizing Kafka possible on VMs?

Kafka configuration instance - properties file - error connecting to external Kafka

3

@MarijeSchillern

I followed this documentation: pega-helm-charts/charts/pega/MigrationToExternalStream.md at master · pegasystems/pega-helm-charts · GitHub

The kafka has no User/password configured and Kafka SASL_SSL is not configured.

Domain to managed kafka service is reachable from the kubernetes cluster.

All used components running in a cloud.

JulianB70

4

@JulianB70 the error includes:

Caused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient

Caused by: org.apache.kafka.common.config.ConfigException: Invalid url in bootstrap.servers: kafka.extelastic1.bpmaas.de

I guess searching on those errors would be a good starting point.

Failing further responses on the forum I would suggest that you log a support incident on the MSP. Please let us know the INC reference if you chose to go down that route.

5

@JulianB70

Setting should be set to false.

Input the external kafka configuration as part of prconfig.

stream:

enabled: false

6

@MarijeSchillern

Thank you for your reply.
I’ve a question, in which format did pega expect the URL for the bootstrap servers?

Here some examples:

  1. bootstrapServer: “kafka.mydomain.de”
  2. bootstrapServer: “kafka.mydomain.de:9092”
  3. bootstrapServer: “http://kafka.mydomain.de/”
  4. bootstrapServer: “http://kafka.mydomain.de:9092/”
  5. bootstrapServer: “”
  6. bootstrapServer: “:9092”

In the Helmchart is no example for a valid configuration of the “bootstrapServer:” entry.
I will open a ticket soon in MSP.

Thank you

Julian

7

@SriHarsha Anika @MarijeSchillern

Problem is solved, the main problem was a missing port number at the end of the domain for the bootstrap server.

Changeing it to:

bootstrapServer: “kafka.mydomain.de:9092”

helps to get the Kafka connected.

Thank you

Julian

8

@JulianB70
Hello!, can you specify where you set the bootstrap servers?
I set the HOST Url on the Kafka Rule, and Schema Registry URLs on the AVRO Schema… but I can’t find where to set the bootstrap servers.

If I add them to the .properties file, I always get an error about “disallowed Kafka client configuration properties”.

Thank you!