Pre-requisite for setting up external Kafka

General
, , , , ,
1

We are planning to switch embedded kafka to external kafka, would like to understand any pre-requisite to handle below items.

  1. How would kafka topics be created for the existing queue processors in the external kafka broker?

  2. Does all the events in the embedded kafka topics moves automatically into external kafka topics?

  3. Does number of partition in the topic created in external broker would still follow ‘20’ partitions? I assume it can be overridden.

  4. Would topic in external kafka, continue with same offset in the embedded kafka topics? If not , how do we reset the offset at the consumer?

2

@SARATH_VISWA

  1. Given sufficient administrative privilege to the Pega account which is interacting with kafka - Topics will be auto created

  2. Existing events may not be migrated unless you leverage migration tools

  3. partitions will as per the default settings provided during the kafka configuration

  4. Offset will not be the same, it will be different as topics are created brand new with new infrastructure zookeeper starts freshly.

3

@SriHarsha Anika

Thanks for the explanation. We are also in the process to move towards externalizing the embedded services starting with Kafka and these answers are really helpful. However I had a follow-up question on this.

When we externalize kafka and connect to an external cluster ( confluent) , I believe the current file system ( kafka-data) will be created fresh in the external cluster. Will Pega continue to maintain the metadata, topic information in the stream DB tables? Or will these be handled in the external kafka clusters?

As in the current processing scenario, if the queuing fails for some reason the information is stored in the Pega DB and when the node is available it will pick up the queued messages from the DB. How will this scenario work when the topics are in teh external clusters?