In QA testing cycles, how do we prove by simulating the various happy, exception path combination(s) of AI-powered image analysis and document extraction inputs / outputs analysis to ensure AI Quality is accurate.
Appreciate any guidance or best practices to follow and/or any tools available to use to hear from enterprise AI grade production rollout perspective.
This is a short question with potentially a very long set of answers
Let me kick off by naming a couple of mechanism we use in the field today:
Golden Truth set for testing >> A set of (in your case) documents or images for which you already know what the right extraction / interpretation is. Run these through your GenAI Connect mechanism, and compare the real extraction to the expected transaction. >> This should be your first step, if you do not have a representative set to Golden Truths yet, then work with your business stakeholders to get this created.
Make sure this Golden Truth set is well stratified: if we’re talking images, include low quality images, images of other things, unsupported file types, etc., so edg cases also get coverage.
Automated golden truth testing >> Two elements:
Running your test set in bulk through your extraction mechanism so you do not manually have to run each test record one by one.
Running the outcomes through an evaluation mechanism automatically so it gets scored.
You can use another Pega GenAI Connect call to achieve an LLM-As-A-Judge setup.
Or you can connect with a service / library outside of Pega that dos this evaluation for you. Think of options like DeepEval / Prometheus.
These eval models should look at things like Completeness, Content adherence, Hallucination rate, etc.
Running in production in Shado Mode first >> Have the image extraction run in the background for production input, but do not show the extracted data to the user. Have the user go through the original, manual extraction and data entry steps. Then compare the hidden GenAI extraction with the human data entry. Only move to GenAI first mode if Shadow Mode achieves high quality.
Production metrics >> Assuming you have a Human in the Loop step, track in your Pega case if each extraction by your GenAI Connect / Agent is accepted by the user as-is, or if it was changed. This is a key metric to know if your mechanism is providing high quality results in a real usage setting.
User feedback process >> Provide capabilities for your users to leave feedback, including notes about what they did not like. Set up a BAU process to triage these feedbacks and finetune your mechanism.
Consider regular Expert reviews >> Regularly take production samples and have second-line experts review and assess if they agree with the users outcomes including GenAI generated output.
Synthetic data: Consider extending the Golden Truth set with synthetic data to massively increase the test coverage. Again, if we’re talking Image input, think of using Generative AI to artificially create test cases. (e.g., taking a standard invoice and programmatically adding noise, blurring it, or omitting fields).
The above approaches are mostly applicable to non-image/document extraction as well, e.g. evaluation Knowledge Buddy or Step Agent outputs.
Thanks Tim for your detailed response and considerations.
I would like to check Deep Eval tool can be supported to perform AI quality testing / regression testing on Gen AI connect rules utilized for image analysis, document extraction purposes in the case journey lifecycle.
Are there any specific examples or guidance from the field would be greatly help.
Here’s the below AI generated extract based on the further digging - pls have a look and shared your thoughts in this regard.
The golden dataset would contain:
The input document or conversation
Expected extracted values or expected response
Expected tool invocation
Expected case type and workflow outcome
DeepEval would replay that scenario and determine whether the latest GenAI Connect output still meets the selected thresholds.
An important distinction:
Use Pega GenAI Connect for a focused AI operation inside a workflow—summarise, extract, classify, recommend, or generate.
Use Pega GenAI Agent with Tool Rules when you need a conversational agent that decides which actions to perform.
Use DeepEval Pega to regression-test the resulting agent conversation.
Use Pega Agent Inspector to inspect and manually test the agent.
The current repository does not directly inspect or invoke a standalone Gen AI Connect Rule. It tests it indirectly when the Pega agent or case workflow invokes that rule and returns the result. If you want Gen AI Connect to appear as an agent capability, normally you would expose the appropriate Pega workflow/automation through an Agent Tool Rule.
