This article provides a knowledge-sharing overview of how Pega Knowledge Buddy leverages the Knowledge Loader component to ingest SharePoint documents through the Microsoft Graph API.
The content and guidance presented in this article are based on the following documentation.
Extend the Pega Knowledge Loader application
Knowledge Loader has own application stack, so your application stack will be similar like below.
This picture represents only content loader development cycle. You can also add Content loader your Knowledge Buddy stack easily. (Creating a Pega Knowledge Loader implementation layer)
Security :
Your SharePoint credentials must have at least the Site.Selected API permission with read access to access and retrieve content. If the target SharePoint site, library, or folder requires additional permissions, you must grant the necessary API permissions and ensure they are assigned to the corresponding application or user credentials in Azure.
Important Tool: Microsoft Graph Explorer is available to everyone and allows you to sign in using your own credentials. It provides an easy way to test Microsoft Graph API endpoints, explore requests and responses, and validate integrations directly from your browser. (Graph Explorer | Try Microsoft Graph APIs - Microsoft Graph)
Rule Changes :
- Creating confoguration case type : As a best practice, we decided to create another case type on the top of the Sharepoint case type. No customization required for the case type unless you have additional requirements.
-
GetRepositoryFiles_Extn : This is the first extension point to get the list of documents from sharepoint. You should check your repository to run right rules like below.
You must set some mandatory properties about the documents. This extension has a list of the properties which are mandatory. Inside of the GetFilesForMsGraph data transform. It is calling the MSGraph API to get the list of the documents from a specific folder.
You can easily review other product configurations, such as SharePoint or Confluence, to understand the required fields and setup.
The connection data pages will be explained in a later section.
- GetfileStream_Extn : This extension point to get the file content from Sharepoint.
- GetRepositoryMetaData_Extn: Easily set to the document metadata. Setting the minimum fields to run the logic.
MSGraph API details :
Knowledge Loader repository configuration stores the base URL and folder path of the sharepoint folder/sites. MSGraph API is based on IDs. So you need to convert text to IDs.
Sample Site : https://myDummyCo.sharepoint.com/sites/DummySite
Each MSGraph API is requesting site ID, so you should calculate or store the value somewhere.
- Retrieve Site ID :
API URL : https://graph.microsoft.com/v1.0/sites/{SiteDomain}:/{SitePath}
Sample Site URL : https://graph.microsoft.com/v1.0/sites/myDummyCo.sharepoint.com/sites/DummySite
Sample Response :
You can find the site id with the help of this API. Sample ID look like below.
myDummyCo.sharepoint.com,60c9361b-2303-49ba-95d5-e1494506eef9,xxxxxx-0a35-4e53-xxxx-ca9470f1a27f
Just create your connection to place related point on the extension points. It will need all API calls.
-
Get Site drive list : Each SharePoint site contains one or more drives. This API is used to resolve a drive name into its corresponding Drive ID, similar to how a site name is mapped to a Site ID. In the Knowledge Loader repository configuration, the Base URL and folder path are provided as inputs. Since Microsoft Graph APIs support filtering, this API relies on the filter functionality to identify and retrieve the correct Drive ID for the specified drive.
Api URL : https://graph.microsoft.com/v1.0/sites/{SiteID}/drives
Site ID: myDummyCo.sharepoint.com,60c9361b-xxxx-yyyy-95d5-e1494506eef9,xxxxxxxx-0a35-4e53-af1f-ca9470f1a27f
WebUrl : Filtering specific drive by Weburl. WebUrl is the combination of the BaseURL and Folder Paths which are in the MSGraph Configuration case type.Regenerate WebUrl during runtime in the response data transform to use.
FolderPath: This is the SharePoint folder path. (Ex: “/sites/DummySite”)
Your connection data page should filter the drive based on its Web URL, as shown below. First, retrieve the complete list of drives for the site, and then apply the filtering logic on the Web URL in the response data transform.
-
Get Documents List with Folder Path: The Knowledge Loader repository configuration can contain one or more folder paths. This API retrieves a list of documents available within a specified folder.
Api URL : https://graph.microsoft.com/v1.0/sites/{SiteID}/drives/{DriveID}/root:/{FolderPath}/children
SiteID is the id of the site like “myDummyCo.sharepoint.com,60c9361b-xxxx-yyyy-95d5-e1494506eef9,xxxxxxxx-0a35-4e53-af1f-ca9470f1a27f” . Details are in the previous page.
DriveID: SharePoint is working with Drives like OneDrive. Look at the next page to the details about DriveID. (Ex: b!GzbJYAMxxxxxxxQbu-Qbayyyyyy_KlHDxonch8zzzzzcDKzBGQjb) (Drive Name Dummy Site docs)
FolderPath: This is the SharePoint folder path.(Ex: “/sites/DummySite”)
This API retrieves all document metadata. Therefore, the GetRepositoryMetaData_Extn extension point should leverage this integration to populate and manage the metadata attributes for the corresponding document.
-
Get Document Content: This API returns the content of the document.
Api URL: https://graph.microsoft.com/v1.0/sites/{SiteID}/drives/{DriveID}/items/{ItemID}/Content
SiteID and DriveID values are very specific for your documents and folders which you already found the values.
ItemID is the Sharepoint id ofthe document from previous API. (Ex: 01XRZUOQXXXOJBUA2BJEKYYYYD4DETHXW)
The Content command is used to retrieve only the document content from the API response. It returns the actual contents of the document. However, the Pega API Wizard may not be able to generate the response structure correctly due to the large content size or because the response format is not recognized as valid by the wizard. In such cases, you may need to configure the connection manually.
Do not attempt to perform any operations on the content, such as decryption. The Knowledge Loader handles such processing automatically.
Microsoft Graph API provides capabilities to access SharePoint sites, making it possible to extend the ingestion process to SharePoint web pages as well. While some validation and additional development efforts may be required, our research indicates that this can be achieved in two ways:
-
Ingest ASPX Pages Directly:
Configure the repository to point to the folder containing the SharePoint .aspx pages and ingest those files as content sources.
-
Use SharePoint Pages APIs
Leverage Microsoft Graph APIs to retrieve and process SharePoint site pages.
Retrieve the list of pages available within a SharePoint site and obtain the corresponding Page IDs:
https://graph.microsoft.com/v1.0/sites/{SiteID}pagesFor each Page ID, retrieve the associated web parts to access the page content and structure:
https://graph.microsoft.com/v1.0/sites/{SiteID}/pages/{PageID}/microsoft.graph.sitePage/webParts
This approach enables the extraction of page content directly from SharePoint sites and can be integrated into the Knowledge Loader ingestion framework with the necessary extensions and validations.