# Implementing Evals with Pega and DeepEval

**URL:** <https://forums.pega.com/t/implementing-evals-with-pega-and-deepeval/13916>\
**Category:** AI\
**Created:** [November 1, 2026, 3:02am UTC](https://forums.pega.com/t/implementing-evals-with-pega-and-deepeval/13916 "2026-11-01T03:02:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sairohith](https://yyz2.discourse-cdn.com/pega/user_avatar/forums.pega.com/sairohith/32/23362_2.png) [@Sairohith](https://forums.pega.com/u/Sairohith)\
**Post date:** [November 1, 2026, 3:02am UTC](https://forums.pega.com/t/implementing-evals-with-pega-and-deepeval/13916/1 "2026-11-01T03:02:36Z")

</div>

**A simple step-by-step approach to implement Pega Agent Evals with DeepEval**

**Step 1: Set Up the Evaluation Framework**

Start by downloading the open-source DeepEval Pega framework from GitHub. Install Python 3.11+, Node.js, and the required dependencies. Configure connectivity to your Pega environment using OAuth 2.0 credentials, the Pega agent name, and an LLM evaluation provider such as OpenAI, Gemini, or AWS Bedrock.

**Step 2: Create Golden Evaluation Datasets**

Golden datasets contain predefined test scenarios with expected agent behaviors and outcomes. These can be created by capturing existing Pega conversations using PXCONV-\* identifiers or importing structured agent outputs. Include scenarios covering successful workflows, incorrect tool selection, hallucinations, missing information, and authorization failures.

**Step 3: Execute Automated Evaluations**

Initialize the evaluation database using python create\_db.py and launch the Reflex evaluation dashboard using reflex run.

The framework supports automated testing through pytest:

python -m pytest test\_golden\_session.py -v -s

DeepEval evaluates multiple quality dimensions, including:

- Hallucination and response accuracy
- Knowledge retention and role adherence
- MCP/tool invocation correctness
- Pega case lifecycle and business workflow compliance
- Contextual precision, recall, and execution latency

**Step 4: Analyze Results and Integrate CI/CD**

Evaluation results help identify failed scenarios, incorrect agent decisions, and performance regressions. Combine LLM-based evaluation with deterministic validation of case states, authorization, and required approvals.

Integrating these tests into CI/CD pipelines enables automated regression testing whenever agent instructions, tools, or business workflows change.

**GitHub Repository and Implementation Resources**

Pega has made an open-source implementation available on GitHub, providing source code, configuration examples, golden dataset management, automated regression tests, and evaluation dashboards.

**Access the implementation here:**

**[Pega DeepEval — GitHub Repository](https://github.com/pegasystems/infinity-ai-agent-demonstrations/tree/master/apps/deepeval-pega)**

Kudos to the Pega team for developing and sharing this open-source repository!

---

<div class="post-metadata">

**Author:** ![Peter\_van\_der\_Putten](https://yyz2.discourse-cdn.com/pega/user_avatar/forums.pega.com/peter_van_der_putten/32/7759_2.png) [@Peter\_van\_der\_Putten](https://forums.pega.com/u/Peter_van_der_Putten)\
**Post date:** [November 2, 2026, 1:30pm UTC](https://forums.pega.com/t/implementing-evals-with-pega-and-deepeval/13916/2 "2026-11-02T13:30:10Z")

</div>

Evals are super important in AI and agentic systems.

In the regular systems development lifecycle, the evaluation/QA is often introduced (too) late in the process.

In many AI systems, think by instance of pre-generative AI machine learning model development, the whole process actually _starts_ with gathering training- and test data.

Agentic AI is a bit in between, but if anything, evals will ‘shift left’ more and more as they will be used to design and hyperoptimize the system, or even drive recursive self-improvement at run time.
