I am looking for some clarification regarding the transaction lifecycle when using BIX Real-Time Data Extraction in Pega.
As I understand it, when Real-Time Data Extraction is enabled, a declare trigger is created and configured to execute on committed save. However, the trigger activity would execute before the database commit has fully completed, as it runs within the transaction scope.
In this scenario, the data might get pushed before the database transaction is permanently committed. If the database commit subsequently fails and the transaction is rolled back, the external system may have already received the event, resulting in a potential “ghost” message. For example, a Kafka message may be published even though the corresponding database transaction was not successfully committed.
Could you help clarify how this scenario is handled? Specifically:
How does Pega ensure consistency between the database transaction and the Real-Time Data Extraction event?
Is Real time data extract guaranteed to publish events only after a successful database commit, or are there edge cases where the event can be sent before the commit is finalized?
What mechanisms or best practices are available to prevent or handle such ghost messages?
Real-Time Data Extraction generates events from committed save operations and I believe it internally uses asynchronous Queue Processor-based architecture.
The events are generated for committed save operations, which suggests publication occurs after a successful commit.
My understanding with what I have observed during testing and in the Pega Tracer, is that the Declare Trigger associated with a committed save is initiated as part of the database transaction, but the trigger activity itself executes before the database commit is fully completed. Based on what I have seen, the sequence appears to be:
Because of this, I would like to better understand how potential ghost event scenarios are handled. If the event payload is generated or queued while the transaction is still in progress, there appears to be a possibility that the originating transaction could subsequently fail or be rolled back after the trigger activity has executed.
In that scenario, what prevents the corresponding event from being published to the downstream system even though the underlying database change was never committed?
I also reviewed the Queue Processor configuration used by Real-Time Data Extraction and noticed that the Immediate option is enabled. This made me curious about whether the Real-Time Data Extraction event can be processed /published before the originating database transaction has been successfully committed.
I’m mainly trying to understand the architectural guarantee here. Is a Real-Time Data Extraction event guaranteed to represent only a successfully committed transaction? If so, what mechanism ensures that guarantee and prevents events from being published for transactions that are ultimately rolled back?
Hello Sridhar, very valid point. Remember exploring this in 24.2 but never did a real production implementation.
Ideally activity run inside a declare trigger should not using commit method unless you use the option - run in background!!
I believe, Pega by default would have created a declare trigger rule for your class name with status available and run immedicately (I also see it from tracer)
I would easily ask to have your own declare trigger activity with same set of configuration except queue immediately, so everything goes in a single transaction, but using OOTB can be much better! Please also pass this to the BIX product support team as well via support ticket to get the understanding about why queue immediately!!
Hey Prem, good to connect with you after a long time!
Yes, I agree. Having a separate Declarative Trigger with the same Queue Processor activity, but without the Immediate option, could be one potential option to address the ghost event concern.
There is also a separate aspect around event ordering that comes to mind. Queue Processors leverage Kafka partitions for parallel processing. At least till v24.1/24.2, the default routing mechanism distributes messages across partitions without a business key based routing option. As a result, events related to the same business object could land on different partitions, and processing order across partitions isn’t guaranteed. I remember there were discussions/feedback raised around providing custom or key-based routing options for Queue Processors. I’m not sure if this has since been addressed or if any enhancement has been introduced around this in the latest versions.
Of course, ordering may not be a requirement for every customer/use case, so this may not always be a concern. But where sequencing of events for the same case is important, it would be another aspect to consider independently of the Immediate option.
Coming back to the original transaction consistency concern, I would also be interested to understand if there is any other supported/recommended pattern for handling this rather than overriding the OOTB Real time extract behavior at the application layer.
Let’s see if other fellow community members or someone from the Platform/BIX product team can provide some insight into the intended architecture and the guarantees provided by the OOTB implementation. If not, I will raise SR to get further clarification and keep this forum updated.
