How pega interacts with kafka for queueing items and how does kafka knows that the item needs to be processed.

General
,
1

How pega interacts with kafka for queueing items and how does kafka knows that the item needs to be processed.

How an item is queued in kafka using queue processor?

2

Hi @TanyaS58,

Queue Processor uses Kafka to execute the queue.

When the time reached, it publishes to the Kafka and Queue Processor process the data.

Let’s understand how it integrates with Kafka on high level.

In an insurance company, Pega manages claims processing and sends claim details to a Kafka topic named claims-submissions. This message includes the claim ID, customer details, and claim details. Kafka stores this message, making it available for subscribed services. This integration enables efficient and reliable claims processing.

  • Integration via REST API: Pega can use REST connectors to send messages to Kafka. This involves setting up a REST connector in Pega that points to the Kafka REST Proxy.
  • Data Flows: Pega’s Data Flow feature can be configured to produce messages to Kafka topics. This involves setting up data flow rules and configuring Kafka as a destination.
  • Also uses few Java rules to integrate with Kafka.

We can see Kafka topics:

PRPCPersonalEdition(DirectoryName)\tomcat\kafka-data

It also uses data base transactions for following aspects.

  1. If stream node is down, it saves as a delayed item.
    1. DelayedItem a System-Message-QueueProcessor-DelayedItem à pr_sys_delayed_queue
  2. The broken items are also persisted in the data base table.
    1. BrokenItem a System-Message-QueueProcessor-BrokenItem à data.pr_sys_msg_qp_brokenitems
  3. The delayed items (When to process configuration in queue processor rule) are persisted.
    1. DelayedItem a System-Message-QueueProcessor-DelayedItem à pr_sys_delayed_queue

I hope this clarifies doubt.

Thanks,

Ravoof.

3

@Mohammad Ravoof so when we are using a queue processor toqueue items do we need to integrate with Kafka everytime?so that it can queue the topic within itself?

4

@TanyaS58

The Queue Processor is designed to manage the queuing and re-queuing of items from Kafka seamlessly. Think of it as similar to a Standard agent, built-in automatic queue management functionalities. Tasks such as exception handling and locking mechanisms are handled by the Queue Processor itself, simplifying the process. All that’s required from our end is to add entries for our queue items to the queue table.

For each Queue Processor, a data flow will be created. These instances can be seen under Pega-DM-DDF-Work. So, these details can be stored under data-pr_data_decision_df_part.

When tracing the Queue Processor through data flow, you gain insight into its operation at a granular level. You can track various elements such as events, partitions created, and other detailed information specific to each item being processed. This level of visibility allows for thorough monitoring and analysis, ensuring smooth and efficient handling of tasks within the queue.