How does a Queue processor works?

General
1

How does a Queue processor works?Like with Kafka and Data Set/Data Flows being involved what is the entire mechanism that takes place in the backend?

2

In Pega, a Queue Processor is an internal background process used for high-performance, asynchronous message processing. It was introduced in Pega 8 to replace standard agents, leveraging Apache Kafka for queuing to provide better scalability and multi-threading.

How It Works (The Core Mechanism)

The Queue Processor functions through a producer-consumer model:

  1. Publishing (Producer): Your application pushes a message into a queue using the Queue-For-Processing activity method or the Run in Background smart shape in a flow.

  2. Streaming: The message is sent to a dedicated Kafka topic. This requires at least one Stream node to be active in the cluster.

  3. Processing (Consumer): An internal Data Flow work object, created automatically for each queue processor, “subscribes” to the Kafka topic. It picks up messages and executes a specified activity to process them.

  4. Parallelism: Each topic can have multiple partitions (default is often 20). This allows up to 20 separate threads to process messages simultaneously without conflict, greatly increasing throughput

4 Likes
3

thanks @Anoop_Krishna for such a detailed explaination.