Pega Knowledge Buddy allows users to specify GenAI for extraction during ingestion. GenAI extraction preserves tables and other formatting, which can lead to better answers. You configure GenAI extraction on the data source as shown below.
However, the Large Language Models (LLMs) used for GenAI for extraction have size limits. If you are extracting text from large documents, it may yield inconsistent or incorrect results. For example, if a document contains a large table, it may also result in an ingestion failed message. Or the table content (rows and columns) may not be retained.
To resolve this issue, Pega Knowledge Buddy version 25.1.3 includes Hybrid Data Chunking. Hybrid data chunking restructures content into focused, meaningful sections to improve the accuracy of AI-powered search and retrieval. Hybrid Chunking was specifically designed for complex business documents such as contracts, manuals, presentations, and scanned files. Hybrid Data Chunking prepares documents for generative AI extraction.
You enable the Hybrid data chunking on the data collection, not the data source. Choose HYBRID under the content processing options.
LLMs deliver the best results with small chunks of content rather than large ones. Hybrid data chunking creates the smaller chunks needed for GenAI extraction.