Does each queue processor have 20 threads, or do all queue processors share 20 threads?

General
, ,
1 
I referred to the following 2 URLs

Queues are multi-threading and shared across all nodes. Each queue processor can process messages across six partitions so that queue processor rules can support up to 20 separate processing threads simultaneously without conflict. Leveraging multiple queue processors on separate nodes to process the items in a queue can also improve throughput.

Pegasystems Documentation?

When I create a queue processor, what happens if I enter a large number of threads per node?
The maximum number of Kafka partitions is 6 for one Pega Platform cluster, which means that the maximum number of queue processor threads that can process messages across all nodes in a cluster is 6. This is not the global limit, but it is the limit for each queue processor.

2

Hi @wangd1

The number of partitions that we configure in Queue Processor is the number of partitions per node for that Queue Processor itself.

For example, if we configure 7 partitions in a Queue Processor and it is configured to run in Background Processing Nodes, then considering that there are 3 Background Processing nodes, there will be 20 partitions as max count is 20.

Please note that these partitions will be created only for that Queue Processor. If you create another one, separate partitions will be created.

Thanks,

Megha

3

Hi @wangd1,

Till 8.6 version, Pega made 20 partitions available for dedicated Queue Processor(QP) that got created. After 8.6. This number of partitions recommended is 6 for each dedicated Queue Processor.

Whether it is 20 or 6(w.r.t Pega version), we still have the possibility to extend this number by considering the system configuration and keeping Performance impact in mind.

In addition to that, 20 or 6 is maximum collective number which means each QP will have that partitions to process messages. QP can process messages in parallel to the partitions available. Partitions in turn called as Threads.

Thank You.

4

@wangd1

You can go through the below URL and try to get the information from the Queue Processor Product Team.

and always getting confused, Up to 8.6, it will create 20 partitions, and from 8.7 onwards, it will create only 6 partitions.

If you want to increase or decrease the number of participants from Pega product 8.7, you can run the below activity.

Activity Name: pxAlterStreamPartitions (This activity available in Pega 8.7 and above versions)

Parameters:

stream: Pass in Your QP name All in Capital Letters

Not mentioned in the document, maximum number.

Kindly get the information from the queue processor product team, and I am also waiting for the response from the queue processor product team for this question.

Hi @MarissaRogers and @MarijeSchillern Kindly add LSA queue processor product team member for this thread.

Thanks,

Ashok

5

To gain clarification on the section ‘How do I increase the performance of my queue processor rules?’ in our latest documentation, could I ask that you submit a documentation feedback by clicking on the ‘Feedback’ (speechbubble) icon on the top right of the Queue processor FAQ document?

I can see no difference in the description between the Pega 8.7 Queue processor FAQ and the Pega '23 Queue processor FAQ .

However, the difference is mentioned in our Pega Academy documentation:

Queue processors and job schedulers (from the Advanced background processing v4 module for Platform 8.8)

88.jpg
88.jpg1261×694 202 KB

The reference to ‘20 separate processing threads per cluster’ is omitted in later mission:

Queue processors and job schedulers (from the Advanced background processing v5 module for Platform '23)

23.jpg
23.jpg1317×674 188 KB

@Bhumireddy starting from 8.7 Pega platform defaults the QP partition count to 6 (earlier 20) and provided an extension to alter appropriate QPs.

On Pega 8.6.4+ versions, clients have the option to alter the partitions only for specific topics through the activity pxAlterStreamPartitions - in case this approach is used, the corresponding topic needs to be manually updated to match with the new partition count (Kafka only supports increasing partitions for an existing topic - decreasing the number of partitions will require re-creation of the topic).

Users can use the OOTB pxAlterStreamPartitions activity to increase the partition count for specific QP’s that require high throughput and parallel processing.

Sizing queue processors and Stream data sets individually by changing the number of partitions of a topic

Sizing queue processors and stream Data Sets individually by changing the number of partitions of a topic

Also note that pxAlterStreamPartitions reset the stream partitions count to default 6 during the case if Kafka data directory is also cleared or removed during restarts. In such cases, we have to redo the step.

To answer your question about activity 'pxAlterStreamPartitions’ : You need to pass the QP name and number of partitions are 6. Then verify the QP dataflow if the partitions are reduced to 6. If not, please stop the QP from admin studio and run ‘pxRestartByID’ by passing the QP name. Please check the checkbox ‘Allow dataloss’ while running this activity.

----------------------------------- Details ----------------------------

When using external Stream, you actually need to pass the full Kafka topic name for the Queue Processor in question which you’re attempting to alter the partitions count.

With the addition of external stream support, it was included a new DSS/Prconfig to customize the prefix and/or suffix for all Pega related topics, so you have the option to point multiple Pega instances to the same external Kafka cluster using unique topic names. This customization possibility was added to avoid multiple Pega instances attempting to use the same topic names. Following is the DSS/Prconfig in question:

  • Dynamic System Setting:

Owning Ruleset: Pega-Engine
Setting Purpose: prconfig/services/stream/name/pattern/default
Value: prefix-{stream.name}-suffix

  • Prconfig:

Please note that the variable {stream.name} should be always present in the pxAlterStreamPartitions activity value as it will get replaced by the kafka topic name during runtime.

  1. Older Pega versions:

@Anoop Krishna clarified in this thread Number of Threads to Number of Partition relation in Queue Processor that in the older Pega versions it was 20 partitions. :

Maximum number of threads across all nodes in a cluster for a QP is 20. That’s the limit for a QP. Please check 8th point in the below url.

https://docs-previous.pega.com/system-administration/87/queue-p

wfhfg.jpg
wfhfg.jpg1508×145 60.5 KB

And if you increase beyond 20, only 20 will process messages from a QP. And the threads should be increased on the CPU and heap. We need to ensure if Host server can handle the load before increasing the threads to 20. But maximum number of threads limit is 20 for a QP.

https://support.pega.com/question/increasing-thread-count-stream-node-and-custom-node

  1. Latest Pega versions:

@Bhumireddy specified that now in recent versions pega supports 6 partitions in a cluster. (see 6th question below):

https://docs.pega.com/bundle/platform/page/platform/background-processing/queue-processor-faq.html

fnjjkkkk.jpg
fnjjkkkk.jpg1197×155 59.2 KB

Clarification requested in this PSC post:

@nairv1 @REDDR3 can someone comment about the above statements?

6

Hi @MarijeSchillern,

Yes I can see from the 6th Point

When I create a queue processor, what happens if I enter a large number of threads per node?

The maximum number of Kafka partitions is 6 for one Pega Platform cluster, which means that the maximum number of queue processor threads that can process messages across all nodes in a cluster is 6. This is not the global limit, but it is the limit for each queue processor.

Pegasystems Documentation

But the thing is that we gave the option to increase the queue processor partitions from 8.7 onwards by using this activity: AlterStreamPartitions ( This Activity Written by : Natalia Varzari )

No ware mentioned that, Maximum global limit. (E.g 20 like that from 8.7 Onwards, Before we don’t have option to increase and fixed number 20 for one queue processor (Upto 8.6))

If you increase each partition, it will occupy 20 MB of memory for one single partition. (I have increased to 100 + partitions on Pega Community Edition and Personal Edition 8.7)

Sizing queue processors and Stream data sets individually by changing the number of partitions of a topic

Managing data flow runs

There is only a difference before 8.6; from 8.7, there is no limit; that’s what I am saying.

"I can see no difference in the description between the Pega 8.7 Queue processor FAQ and the Pega '23 Queue processor FAQ "-- Yes Ture

Special thanks to @MarijeSchillern for bringing the product team into this thread.

We will be waiting for valuable points from the product team too.

Thanks,

Ashok

7

@Bhumireddy

  1. When we create a Queue Processor( QP) – 6 partitions are created by default. Partitions is a logical separation on the underlying Kafka topic and messages in a Partition can be independently consumed and processed by a thread.
  2. Number of available threads can be configured from QP rule form. If the node type has 3 nodes, 3 threads will be available
  3. For optimal performance and resource allocation – the number of threads should be equal to number of partitions. If available threads are lesser than number of partitions – threads will process messages from partitions in a round robin manner
8

Hi @nairv1

I totally understand, from Pega 8.7 On wards, 6 partitions are created by default for One Queue Processor.

What is the maximum number, I mean Upto 8.6 we have 20 partitions, From 8.7 I can’t see the maximum number of partitions.

We have a activity to increase the number of partitions.(Activity Name: AlterStreamPartitions).

And also, this tooltip message is still showing the 20 numbers in Enterprise Edition 8.7.6 ( and Personal Edition 8.7) if the 6 partitions are created by default for the One Queue Processor, if we give 20 number, how will it process?

Note: It will process only 6, because 6 partitions are created by default.

QP Image.jpg
QP Image.jpg772×319 54.4 KB

I can see It showing the 6 partitions in Community Edition, Nevertheless, It should be always dynamic right I mean it should be fetched value from the Data Flow rule right. (If we increased number of participants, then It shows the different value in Data flow rule and tooltip showing the different value)

QP.jpg
QP.jpg712×325 59.5 KB

Could you please address this? I am getting confused. And what is the maximum number of partitions can you recommend?

Let’s suppose we have 100 million and more records there, and we have to process them.

Thanks,

Ashok

9

@Bhumireddy We have not made any change in the recommendations for the the maximum numnber of partitions . However increasing the partitions to very high numbers can impact the performance of Kafka. So, this will have to be a decision taken carefully in consultaion with Kafka administrators. At anypoint , the number of threads configured on the data flow has to be less than the number of partitions, but as close to the partition count as possible , to support maximum parallel processing