We need to fetch millions of record via service rest,can it be done?

General
1

We need to fetch millions of record via service rest,can it be done?

2

Hi @TanyaS58

It is technically possible, but is not recommended.

Fetching millions of records in a single REST service call is against Pega best practices and can cause memory issues, timeouts, and performance degradation, since REST services are synchronous and memory‑bound.

Best Practices:

  • Use pagination for REST APIs (small batches like 500–5000 records per call)

  • For millions of records, use asynchronous batch processing (Queue Processor / Job Scheduler)

  • Prefer file‑based export (CSV/JSON) for large data volumes

  • Avoid returning huge payloads in a single REST response

Conclusion: For large datasets, REST should be paginated or used only to trigger background processing—not to return millions of records in one response. Let me know what is the exact use case you are trying to achieve.

Thanks,

Roshan Raj Ak

3

Hi @RoshanRajAK1 actually this was a scenario based question and i was asked if this is possible via pooling or something which even i am not aware,can you share some insights on pooling of Service rest?

4

Hi @TanyaS58 - In Pega, pooling or Requestor Pooling is generally not the solution for fetching millions of records. Pooling is designed to manage requestor reuse for high-frequency, small-payload requests, not for handling massive data volumes in a single execution.

If you try to fetch millions of records through a standard Service REST, you will likely hit memory limits (causing OutOfMemory errors) or transaction timeouts before the data can be serialized into JSON and sent.

Why “Pooling” isn’t the answer for high volume

  • Requestor Pooling Purpose: It manages concurrency. For example, if 100 people call your API at once, pooling ensures Pega has requestors ready to handle those 100 separate small tasks without the delay of creating a new session each time.

  • The Volume Problem: If one of those requests asks for 2 million records, pooling won’t help. The requestor (even if pulled from a pool) still has to load all those records into its own memory (Clipboard), which will crash the node

Other than this, Even I am not sure if we can use the concept of pooling to fetch a large volume of records using Service-REST.

Let me know if this helps

Thanks,

Roshan Raj A K

5

This is NOT recommended because of the following reasons,

  • Timeouts
  • Memory issues on Pega node (Potential Out of Memory issues)
  • Large clipboard size
  • Slow response times
  • Network bottlenecks
  • JSON parsing overhead

The Recommendation is to use to Pagination, Narrow down the search critieria to retrieve the limited results. Leverage Real time BIX , near time BIX or Data Flow for any requirements to send the records to their data warehouse.

6

Thanks @RoshanRajAK1

7

Hi @RoshanRajAK1 how can we use pagination though?

8

Hi @TanyaS58 - Service‑REST does not provide pagination out of the box. Pagination must be implemented manually in the processing logic by controlling how records are retrieved and returned.

Approach 1: Using Report Definition (preferred):
Call an RD from the invoking Activity and Configure the RD for pagination

  • Enable paging on the RD

  • Use dynamic page size

  • RD fetches only required rows, improving performance

Approach 2: Using Obj‑Browse:
Limit records by setting MaxRecords = pageSize and handle offsets in the Activity. This works but is less efficient for large datasets.

Recommendation: Use Report Definition with paging for better performance and scalability in Service‑REST APIs.

Please let me know if you have any other questions

Thanks,

Roshan Raj A k

9

we can use query param in Rest APIs for granular control of results.

In REST APIs, pagination is typically controlled using query parameters like

  • page

pageSize

offset

limit