Savable Data Pages: Creating a new instance [LSA Data Excellence]

General
, , , ,
1

Can I use Savable Data Pages to create a new instance?

We had a lot of questions about this - it can be done.

Firstly, remember that the first thing any Data Page (including Savable Data Pages) does is to attempt to load from its Data Source, likely using keys captured via the application UI used to lookup the instance from the System of Record. If the instance doesn’t yet exist, depending on the data source this will yield either an empty result set, or an error.

If your application need is to capture details for creating a new instance when none exists, ensure the Response Data Transform or Post Load Processing Activity clears any “record not found” error from the page (for it is not an error in your scenario) and stages the Data Page to provide any default values for creation of a new instance.
Depending on the User Experience you provide for creating a new instance, the loading of the Data Page may be triggered before you have captured (all of) your keys. Consider having the parameters optional and have conditional Data Sources that run an inexpensive, ‘stub’ Data Transform (such as @baseclass.empty) if the keys are blank or incomplete. Once all key values have been captured, these are a distinct parameter combination resulting in a new Data Page instance being loaded, this time attempting a look-up from the genuine Data Source. Any existing instance for the keys entered can then be retrieved and shown in the User Experience, rather than capturing a second item for the same set of keys.

If your Data Page is used multiple times to capture multiple new instances in a list (for example, adding many drivers to a car insurance policy), also include an additional optional “index” parameter to the Data Page that holds a different value for each item in the list. This is purely to ensure that multiple, potentially-unkeyed Data Pages remain resident as distinct pages on your clipboard to capture each new instance separately. Remember if the application seeks a Data Page with the same parameter values as one that already exists on the clipboard, it will use that one and not load a new one.

Once the Data Page is loaded with default values for a new instance, it can become the context page for part of your Flow Action’s User Interface and be directly populated with content entered in the application. Assuming Flow Action completion is the right time to save this Savable Data Page, its Save Plan can navigate the different approaches needed.

  • For a Database Save, like with Obj-Save, the record will be created if a record doesn’t exist with the values entered for the keys; or will be updated otherwise. Saves are deferred by default.
  • For other Save types, conditional logic in the Save Plan can test a property staged on the Data Page to determine whether it should do a Create or Update operation, and use the appropriate save path. For example, Connect-REST rules typically use POST for Create operations but PUT for Update Operations, with some slight variance in the Request payload. This complexity could be pushed into an Activity if needed. Consider this as a last resort, but feel empowered to do so.

However, don’t ever use Commit in your Save Plan activity as this will commit deferred saves that are unrelated to your Savable Data Page. Investigate the WriteNow support for the Save Data Page options if your Save Plan database changes need to be immediate, but ideally treat these as any other deferred saves.

Finally! Pega 8.4 adds better support for referencing Savable Data Pages from Flow Actions, Flows and Activities via Auto-populate properties. This allows you to encapsulate the relationship between your case data model and the Data Pages that supply your case with persistent data within the Property configuration, and reduces the number of places that you need to both explicitly reference the Data Page names and wire in the correct parameters they require.

The create/update use case is a good example of where the “Copy data from data page” Auto-populate option could be used. As above, once you have the keys, you likely want to retrieve any existing instance (or stage a new, ‘default’ instance if none exists), allow the user to start capturing modifications, but allow those in-flight modifications to be saved with the case data if they don’t complete the action before the end of their day. So long as the keys don’t change, the Auto-populate property that uses “Copy data from data page” will retain the in-flight modifications and not reload the Data Page from the System of Record, allowing the user to complete their action later. On completion of the action, if the Auto-populate property is included on the Flow Action’s “Save Data Pages” list, then the changes are posted to the System of Record by the Save Plan on the Savable Data Page.

Discussion on this topic was sought from the LSA Data Excellence (Pega 8.4) webinar conducted in July 2020. The webinar and its full set of discussions that arose from it are available at LSA Data Excellence: Webinar, Questions & Answers.

2

@BraamCLSA Thanks for this post! Is the ability to create new instance using Savable Data Pages introduced only in Pega 8.x or also available to Pega 7.x?

Thanks,

Udhay

3

@UdhayachanderG7215

I do not believe there is anything in the pattern that is unique to Pega 8.x, so you should be able to adopt this in any version of Pega that has the Savable Data Pages feature. This is essentially

  1. Checking - in the Data Page data source (which still runs first) - that you don’t yet have keys to look up an existing instance
  2. Ensuring the Data Page data source prepares a new instance for data entry when there are no keys
  3. For Data Page save plans that are not “Database save”, determining whether different execution is needed for create vs update. For example, Connect-REST would typically use POST for create, and PUT for update.

For save plans that are “Database save”, you don’t typically need a distinct save plan for create and update, as a Pega database save will figure this out based on whether the key properties are set or not on the page you are saving.

To be clear, all of the above is what you would have had to do (in an Activity) previously. The idea is to pull as much of this as possbile into a Data Page ruleform, reducing the usage of Activities, surfacing more of Pega’s error detection, and encouraging its usage as part of the Pega engine’s default transaction management provided by Flow Actions.

4

@BraamCLSA Thanks Braam!

5

@BraamSmithCLSA

Thank you for your post! We have implemented a Savable Data Page, with 2 Data Sources, to insert and update records into a Data Type stored locally. It seems to work well. But in our Test environment, some users complained, and one of the symptoms we found was that this record was not inserted sometimes.

When I investigated in Test environment, I found we get a warning when we save the record with the message “Record not found”, but is it just a warning we can ignore, or should we not get this warning at all, and maybe this is why our record is not saved?

The screenshot of our Savable Data Page and the screenshot of our Activity with Save-DataPage method are attached.

Savable Data Page.png
Savable Data Page.png1687×957 45.4 KB

Activity with Save-DataPage.png
Activity with Save-DataPage.png1364×757 45.1 KB

6

@Filipe Lopes

Hi Filipe, the message “Record not found” seems to indicate that you are trying to perform a save on an existing record that is not there. What is the situation when this error occurs? Is it a save for a new record or is it a save of what is an existing record? In that last case, did you check if the record actually exists in the database?

I would certainly trace this situation to get a better sense of the actual context; is there an existing record or not, if its a new record, did the data transform run and are all required fields available, check the query and response, etc. And see if the logic gets run in a transaction so that is does get a commit by pega.

7

@DionLammers Hi Dion, thanks for your quick answer!

For now, we are only using this Data Page to insert new records, so it’s definitely an Insert. When I traced it, System shows this Warning with the message “Record not found”. As described in the screenshots, we have 2 sources, a Data Transform, and a Lookup. On the tracer, we see the Data Transform is correctly being run. In the Data Page and the Data Transform, we have parameters with the properties defined as keys, and the Data Transform receives those values, and sets them in the Data Page correctly. Then, when we run the Save Data Page in the activity, that’s when we see this warning in the tracer.

But I found out something interesting about the error I was trying to investigate: the issue was, when I was running this under a non-developer user, the record was not being inserted, and that user / access group lacked the access to write in that class. But the fact we are using a Savable Data Page would not show any error to the end user, only in Tracer. So I guess, in the Activity where we use Save-DataPage, we should explicitly use a Jump condition e.g. StepStatusFail to catch that error, and trigger a message to the end user. When we use a regular page-new / obj-save, that error would be caught, and user would know something wrong had happened, but not with Savable Data Page.

So, I guess the warning was not the problem, I think it may just be “normal”, when we use a Savable Data Page to insert a new record.

8

@Filipe Lopes happy to hear you found the issue and it was due to insufficient access rights by the users.

In activities -especially with OBJ type of actions- error handling is crucial and often forgotten. Not handling the error may result in a loss of data integrity, ungraceful errors, and uncertain application logic. Therefore we should always leverage an activity step method transition with the when rule checking for StepStatusFail to handle such errors.

That isn’t different for data pages; we typically catch the error in the data page itself. At a minimum, create a single generic Data Transform that does error detection and handling, and include this in all Response data transforms. Then in your activity use the hasMessages when rule to this data page and with that you can gracefully handle the error. More information to be found here: Pegasystems Documentation

9

@DionLammers thanks for stepping in and helping Filipe out on this one.

@Filipe Lopes when reading through this thread, I did wonder why it would have been fine in your Dev environment, only to find issues in the Test environment. I recommend verifying all runtime behaviour - in Dev - using end-user access groups as part of your Definition of Done, so as to catch issues like this before they leave Dev.

Seconding Dion’s most recent comments: any Activity operation that has a reasonable risk of failure - such as Obj-Save and Save-DataPage - should have post-processing (Jump) configuration to test the success of that operation. Depending on the failure scenarios, you may need to use hasMessages, StepStatusFail or both to detect failure. Have test cases that exercise the likely failure scenarios to verify that your error-checking configuration is catching the failures.

Can you set up your workflow step (Flow Action) so that the Savable Data Page is ready to save when the user submits the assignment? If so, you can configure the Flow Action to nominate which Savable Data Pages (or, ideally, auto-populate properties from your case data model that reference the Savable Data Pages to save) should be saved as part of assignment completion. The Platform engine will check these Savable Data Pages for failure messages after each Save operation, and put these messages on the Flow Action’s primary page. If you configure nothing further, the existence of messages on the primary page prevent the assignment from completing, re-show the UI including the messages collected from all Savable Data Pages that failed to save. This would have been a more correct outcome for the scenarios your test users were observing, and is zero activity logic.

If your scenario is not a workflow step, then fine: you have to use an activity to orchestrate the Save-DataPage action. With that comes the added task of implementing the appropriate error handling that is appropriate for whatever context that activity executes in.