Lets say i am having 2 or more stream nodes in the pega cluster. If i create a dedicated queue processor in pega, will pega creates the kafka topic and same number of default partitions in all the stream nodes?
In that case, how dataflow running on the background processing node picks the message from which stream node?
Could you please help me on this query.
@P.Siva.Praveen When you create a dedicated queue processor in Pega the system creates a Kafka topic that corresponds to that queue processor. Based on the number of partitions mentioned in the server.properties file the system creates the same number of folders in the tomcat Kafka data folder. At least one stream node is necessary for the system to queue messages to the Kafka server. If you do not define a stream node in a cluster the system queues items to the database and then processes these items when a stream node is available. The data flow running on the background processing node picks the message from the Kafka topic which is managed by the stream nodes. The stream nodes handle the distribution and replication of messages ensuring that the data flow can retrieve and process the messages efficiently.
This is a GenAI-powered tool. All generated answers require validation against the provided references.
Queue processors and job schedulers
@nairv1 can you offer any input?