Queue processor on data objects

General
, , , ,
1

Hi,

I have a requirement to get data class records like 1000 and iterate them, call a service on each record and save each data instance into another data table with the response from service. As we need to process 20,000/- records per day, that means 20,000 service calls I am thinking to utilize queue processors so they can share the work load and do not impact the system performance. But my question is can we call a queue processor on a data class page and queue the data objects? As, queue processors are same like Standard agents which we use on work objects I am not sure whether we can utilize queue processors for data objects.

Right now we are using Job scheduler for this requirement but each hour getting 1000 records and processing them for 20 hours per day may impact the system performance so, thinking about an alternative. Could someone please suggest me a better approach?

Thanks in advance.

2

@DurgaPrasadBali

I think both the queue processor and standard agent can handle any class instance, be it a data class or work class that is saved to a DB table (Queue processor can even handle page snapshot).

Using a Queue processor seems to be the best option in your scenario, IMO.

Couple of suggestions -

  1. Remember to adjust the ‘Number of threads per node’ settings on your Queue processor for optimized performance. Normally, it’s calculated in this way : 20 / No. of Nodes where this queue processor will be enabled.
  2. Given that for each instance you queue, an external system/service will be contacted via Connector, keep the network bandwidth requirement in mind.
  3. Queue processor can fork multiple parallel threads on the same node to process multiple queued instances in parallel. So, check with the external service if they can handle at max 20 (in worst case scenario) parallel connection to their service (The same limitation we face when we setup a service at PEGA end i.e. service requestor pooling).

Hope it helps.

3

@Nirmalya.SenSharma

Hi, Thank you very much for your valuable suggestions which helps me a lot. I will try this solution and follow the suggestions and hope it works :slight_smile: