Pega GenAI Agent Evaluation

AI
1

Hi :)!

I would like ask what the prescribed pattern is to evaluate the quality of Pega GenAI Agent outputs? It is a conversational agent helping with email generation.

I’m looking for a way to collect and assess data about my Agent’s performance: dataset experiments, deterministic checks, LLM-as-a-Judge etc.

Are there any Pega internal tools for that? Or ways to connect it to technologies like Langfuse for example?

Some input on this matter would be much appreciated - thank you!

2

We have couple of Gen AI Agent Insights built in Infinity 26 version. I’ll keep you posted once I have the information.

3

Thank you Ramesh, much appreciated!

Does this mean that there is nothing of that sort in Infinity 25?

Would also be very interested in how other clients evaluate the results of their Agents. I mean, without that, how do you, for example, know if changing to a different model really leads to desired outcomes etc.?

4

Hi Franz,

Evals are super important in agentic AI, and quite often, more of an afterthought. So it is good to see it seems you are thinking about this proactively.

Indeed, if you only have some ad hoc tests it will be quite hard to make decisions when designing the agent, and it will be hard to monitor quality during live usage.

The obvious issue that people think of is a lack of accuracy, but frankly another consideration could be cost: using very expensive models where simple models would have done the trick, as different use cases require models of different strength.

Also, lacking proper evals could negatively impact time-to-market. Without formal evaluation, you will see that PoCs will have a hard time to go live, and pilots will have hard time to scale, as users or stakeholders will get very nervous about very specific corner cases where performance is sub-optimal. Whereas we all know AI is a numbers game, you need to look across hundreds of cases or more, or worse, solving a corner case may negatively affect other cases.

Ok I realize I might be preaching to the choir here ;). In terms of patterns I wouldnt fixate on the availability of specific Pega versions (with the exception to making use of what is there OOTB) as there are plenty of open-source frameworks out there that could be combined with Pega agents, and the concepts and principles are more important. Here are two of them, looking for the community to chime in with further suggestions:

  1. Only manual tests is no test. As mentioned, there is little value to single case / manual testing only, you really need automated testing at scale. Assuming your use cases start with some key inputs and ends in an email single shot, you could start with reference-based testing. Based on suite of test cases consisting of reference inputs and ‘good’ answers, generate new outputs based on the inputs and compare. This could be simple keyword/edit distance based (using metrics like ROUGE, METEOR etc) or a semantic check allowing for different ways to say the same (BERTscore, or a LLM as a judge to compare against the reference).
  2. Only manual reference data will provide poor coverage. Once you get started with 1) you will soon find out that your test set of manual reference (inputs, emails) will be too small to provide good coverage. A common approach then is to move towards using synthetic data to expand the reference set. At first it is advisable to do this at design-time: with a very strong model generating reference pairs and double check them (or based on your use case, you may have a large sample of real input), and keep the good ones. Always make sure to store the lineage as well (that it was synthetically generated and with what config). If the process is reliable enough you may even generate reference data at run time during testing (as frameworks like RAGAS do in certain cases for RAGs).

An example framework could be DeepEval, I havent played with this repo but have a look: infinity-ai-agent-demonstrations/apps/deepeval-pega/README.md at master · pegasystems/infinity-ai-agent-demonstrations · GitHub

Another example a bit outside of your scope is what we have used in the AI Lab for evaluating various Knowledge Buddies/Agents using RAGAS plus synthetic data, see [2508.11758] Can we Evaluate RAGs with Synthetic Data?

Looking forward to more ideas from the community!

5

Hi Peter,
thank you so much for your reply!
And thank you for highlighting how important this topic in general really is. So more insights from the community would indeed be amazing.
I’m gonna investigate the repo you shared, looks quite interesting! Much appreciated.
A lot to digest now :)!