We were exploring Pega Case Archiving feature as explained here (Pegasystems Documentation) and have following questions when we use File system as a repository:
When cases are deleted using the expunger job, restoring the archived file back to the file system does not help because Pega removes all associated metadata from the Elasticsearch index. This behaviour is also documented in Pega’s official guidance.
Is there a way to retain the Elasticsearch index data while cleaning secondary storage through the expunger job?
The reason for this request is that, in exceptional scenarios, we may need to restore data from cold storage (via our backup team) to secondary storage if a case is deleted by mistake and the business requires recovery. If the expunger job deletes everything, it seems there is no way to view the case data in Pega even after restoring the file
Out of the box, we only see a checkbox that purges all case data in secondary storage. What if we want to delete only specific data, for e.g., cases older than 3 years, while retaining the rest?
Is there an alternative approach within Pega to achieve this?
If not, the only workaround appears to be deleting archived files manually or via a non-Pega/cron job. Ideally, we would like each application team to manage housekeeping for their case data in secondary storage, and doing this through Pega jobs would make the process much easier.
In repository setting where we provide Resource path only accepts the hardcoded path name, is there way to generate this path dynamically depending on the application as each application may want to define their own repository path. Currently all case types are stored under one single folder
You can’t keep Elasticsearch index data once the expunger job deletes cases, because it is designed to remove all related search metadata, so restoring only the archive files from the file system will not make those cases visible again in Pega; to recover deleted cases you’d need a full platform/database and index restore or a reindex from a backup environment. There is no standard setting to tell the expunger to “only clear S3/file data but keep the index,” so for exceptional restore scenarios you should adjust your archiving policy (for example, longer retention before expunge, or a separate “legal hold” class) and rely on backup/DR processes for true recovery. Out of the box, filtering is mainly by age and status of cases; if you need finer control (for example, only delete files older than 3 years while keeping others) you typically implement a custom job/activity or use an external script/cron against the repository with your own rules. For housekeeping, a common pattern is: let Pega handle move-to-archive and high-level retention, but let each application team manage deeper cold storage clean-up in the repository itself with tooling outside Pega. The repository resource path is static by design, so if each application needs its own folder, you either define multiple repository rules with different base paths or add an extra application-specific folder level inside your archiving strategy; Pega does not dynamically build the base path per application out of the box.
@Sairohith Thanks for your response! Regarding multiple repository when you say “you either define multiple repository rules with different base paths or add an extra application-specific folder level inside your archiving strategy” - could you advise how we can do it for multiple application sharing the same pega instance? as far as i know, one pega instance can only support one file repository defined by a single DSS - dataarchival/repositoryForStoringArchivedData, as documented here: https://docs.pega.com/bundle/platform/page/platform/system-administration/case-archival-repository-configuration.html.
@SairohithFound a solution/answer to the original problem.
Running the pyArchival_ReIndexer can restore the indexes which were removed by Expunger job so i put the file back to archive location and run this job, i can view the case data online again.
In 25 will support multiple data retention policy for purge process as well
There is now way currently to support repository per application sharing the same pega instance
You can also reach out to Pega directly. If the cases have been expunged, they are typically retained for up to six months I think. Retrieval can be difficult, but it’s not impossible. You’ll need to share the case creation date and any other relevant details so Pega can review it. This will require coordinating with Pega on their side.