Kafka Externalization Using AWS MSK with DR Setup(Active/Passive)

General
1

Hi All,

We are currently in the process of externalizing the Pega Stream Service (Kafka) in Pega Platform 23 using AWS MSK as the Kafka provider. We have successfully configured and validated connectivity in our non-production environment.

As part of our production architecture, we’re planning an Active/Passive disaster recovery setup using multi-region Amazon MSK clusters with MSK Replicator for cross-region replication.

From my understanding, MSK Replicator synchronizes Kafka topics, consumer groups, and related metadata between clusters. In a typical Active/Passive scenario, replication would run from the primary region to the secondary region. However, after a disaster failover to the secondary region and subsequent failback to the primary region, reverse replication would be required to keep data synchronized.

While reviewing AWS documentation, I found the following statement:

“MSK Replicator replication is asynchronous, so duplicates can occur during failover/failback events. AWS recommends consumer-side deduplication or idempotent downstream processing.”

Reference: Build multi-Region resilient Apache Kafka applications with identical topic names using Amazon MSK and Amazon MSK Replicator | AWS Big Data Blog

I have a few questions regarding Pega’s behavior in this type of architecture:

  1. Consumer Offset Management

    • I noticed that Pega maintains consumer partition tracking information in the pr_data_qp_run_partition table.
    • Does Pega primarily rely on this table for offset management rather than Kafka consumer group offsets maintained within the Kafka cluster?

  2. Duplicate Message Handling

    • Since MSK Replicator is asynchronous and duplicate messages may occur during failover/failback scenarios, does Pega provide any built-in protection against duplicate processing?
    • Are Stream consumers in Pega designed to be idempotent, or is duplicate handling expected to be implemented at the application level?

  3. Need for Kafka Data Replication

    • For an Active/Passive DR topology with externalized Kafka (AWS MSK), is topic/data replication between regions a recommended and supported approach from a Pega perspective?
    • Has anyone implemented a similar architecture successfully?

  4. Offset Mismatch During Failover/Failback

    • If consumer offsets become out of sync between the primary and secondary regions during failover or failback, how does Pega handle the situation?
    • Is there guidance on recovering or realigning offsets to avoid message loss or duplicate processing?

If anyone has experience running Pega 23 with externalized Kafka/MSK across multiple regions, I’d appreciate hearing about your architecture, lessons learned, or any recommendations from Pega Support/Product Engineering.

Thanks,
Madhav