While the GenAI Agent and Coach rules provide Auditing and Monitoring out of the box as described in Murali Divakaran’s post and Cedric Vallette’ post describes how you can enable reporting against the Pega-Autopilot-Conversation class for Agent and Coach rule audit reporting, I have not found a comparable approach with the GenAI Connect rule. To date the only option I have found, is to create a Summarization and Audit field associated with the output object that I create for my Connect rule (ex: .DriversLicense.Audit). Which I then populate by providing additional instructions in my user prompt to write both the user prompt and system prompt, model information, and document reference to the Audit field. I also create a summarization field (ex: .DriversLicense.Summary) thru the user prompt. As this is using a fixed field, rerunning this step in the flow overwrites the “audit” information, which is not ideal, but it is better than the other options that I have found.
What are the other approaches that have been used?
If the expectation is to add Audit information after GenAI Connect rule, then you can use the Data transform and invoke AddHistory OOTB activity to add the Audit information on the case. Rerunning the Gen AI connect rule could trigger the Data transform using the Form refresh settings and add the Audit logs to the Case.
The fixed audit field approach works, but it is not ideal because each rerun overwrites the previous audit entry.
Adding to @RameshSangili 's answer- Alternatively if you need a complete execution history for every rerun, you can store the GenAI metadata in an append-only audit data type or page list of audit records, so you preserve the full history of prompts, model details, source references, and outputs. That gives you a proper audit trail and is easier to report on than a single field on the output object
@cantg We have built a wrapper component around the GenAI smart shape. OOTB smartshape works in a way that if we pass ResponsePageName param it returns whole metrics. So as Ravi & Ramesh suggested, post GenAI execution these metrics can be used for Auditing and then can be removed.
The Connect GenAI rule does not provide out-of-the-box tracing or auditing capabilities. We had a very similar requirement in our project.
To address this, we created a generic Summarization/Execute Connect GenAI case type that executes the rule. The Pega-Autopilot-Conversation class is also associated with a case type, so our approach is very similar to the product’s implementation for agent and coach.
This setup allows you to easily report on and audit all GenAI interactions. Although I have not tested it myself, you should also be able to pass the prompt dynamically by mapping a property to the field value, providing additional flexibility in how prompts are managed and tracked.
Thank you for sharing the solution! I believe you’re calling the Generic Case Type and in turn the Case workflow calls an Agent rule to perform the Summary/Extraction. Is my understanding correct?
Case is calling connect gen ai rule. But we are logging everything which we are sending or receiveing. We also get some number like eecution time etc. If you need more, you can get from connect gen ai clipboard page and store it on your case.
I was also trying this approach to get all execution. It is also working. we were calculating some metrics based on our case type. This approach looks more suitable for generic approach. thank you for sharing.
