Implementing Evals with Pega and DeepEval

AI
1

A simple step-by-step approach to implement Pega Agent Evals with DeepEval

Step 1: Set Up the Evaluation Framework

Start by downloading the open-source DeepEval Pega framework from GitHub. Install Python 3.11+, Node.js, and the required dependencies. Configure connectivity to your Pega environment using OAuth 2.0 credentials, the Pega agent name, and an LLM evaluation provider such as OpenAI, Gemini, or AWS Bedrock.

Step 2: Create Golden Evaluation Datasets

Golden datasets contain predefined test scenarios with expected agent behaviors and outcomes. These can be created by capturing existing Pega conversations using PXCONV-* identifiers or importing structured agent outputs. Include scenarios covering successful workflows, incorrect tool selection, hallucinations, missing information, and authorization failures.

Step 3: Execute Automated Evaluations

Initialize the evaluation database using python create_db.py and launch the Reflex evaluation dashboard using reflex run.

The framework supports automated testing through pytest:

python -m pytest test_golden_session.py -v -s

DeepEval evaluates multiple quality dimensions, including:

  • Hallucination and response accuracy
  • Knowledge retention and role adherence
  • MCP/tool invocation correctness
  • Pega case lifecycle and business workflow compliance
  • Contextual precision, recall, and execution latency

Step 4: Analyze Results and Integrate CI/CD

Evaluation results help identify failed scenarios, incorrect agent decisions, and performance regressions. Combine LLM-based evaluation with deterministic validation of case states, authorization, and required approvals.

Integrating these tests into CI/CD pipelines enables automated regression testing whenever agent instructions, tools, or business workflows change.

GitHub Repository and Implementation Resources

Pega has made an open-source implementation available on GitHub, providing source code, configuration examples, golden dataset management, automated regression tests, and evaluation dashboards.

Access the implementation here:

Pega DeepEval — GitHub Repository

Kudos to the Pega team for developing and sharing this open-source repository!

2

Evals are super important in AI and agentic systems.

In the regular systems development lifecycle, the evaluation/QA is often introduced (too) late in the process.

In many AI systems, think by instance of pre-generative AI machine learning model development, the whole process actually starts with gathering training- and test data.

Agentic AI is a bit in between, but if anything, evals will ‘shift left’ more and more as they will be used to design and hyperoptimize the system, or even drive recursive self-improvement at run time.