Caused by: com.pega.pegarules.pub.database.BadTableMappingException: Cannot save instance of class Data-Decision-IH-Fact; it belo.

General
, , , , , ,
1

Hello dears,

We’ve added new column to PR_DATA_IH_FACT table and property to Data-Decision-IH-Fact class and other respective changes (saved Data-Decision-IH-Fact Database Table instance in development environment) have been completed. DB add column operation have been completed on upper environmets first and product rule includes property and related data transform rules populating the FACTEXTENSIONID property value and database table instance has been deployed on upper environments.

We have obtained logs below one of the upper environment. Unusual thing here is that we have not faced with the same issue on another upper environment after the same package has been imported.

Caused by: com.pega.pegarules.pub.database.BadTableMappingException: Cannot save instance of class Data-Decision-IH-Fact; it belongs to fully exposed table PEGA_DATA_PROD_SAT.PR_DATA_IH_FACT in database PegaDATA, but the instance has properties which do not correspond to columns: FACTEXTENSIONID 
	at com.pega.pegarules.data.internal.access.PageDatabaseMapperImpl.fillInDataStoreSaveStatementWithoutStream(PageDatabaseMapperImpl.java:1704) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.PageDatabaseMapperImpl.fillInSaveStatementWithoutStream(PageDatabaseMapperImpl.java:1518) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.BatchProcessor.fillInStmt(BatchProcessor.java:311) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.BatchProcessor.save(BatchProcessor.java:159) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.DatabaseImpl.saveDeferred(DatabaseImpl.java:2823) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.DatabaseImpl.saveDeferred(DatabaseImpl.java:2814) ~[prprivate-data.jar:?] 
	at com.pega.pegarules.data.internal.access.DatabaseImpl.saveDeferred(DatabaseImpl.java:2807) ~[prprivate-data.jar:?] 
	at com.pegarules.generated.pzStoreFactRecords_080101__6610569284587962636.pzStoreFactRecords08_01_01(pzStoreFactRecords_080101__6610569284587962636.java:135) ~[?:?] 
	at com.pegarules.generated.pzStoreFactRecords_080101__6610569284587962636.invoke(pzStoreFactRecords_080101__6610569284587962636.java:77) ~[?:?] 
	at com.pega.pegarules.generation.internal.library.LibraryRuntime.resolveAndinvokeFunctionViaReflection(LibraryRuntime.java:238) ~[prprivate-generation.jar:?] 
	at com.pega.pegarules.generation.internal.library.LibraryRuntime.invokeLibraryRuntime(LibraryRuntime.java:125) ~[prprivate-generation.jar:?] 
	at com.pega.pegarules.session.internal.mgmt.Executable.invokeLibraryRuntime(Executable.java:9829) ~[prprivate-session.jar:?] 
	at com.pega.pegarules.priv.generator.LibrarySupport.resolveAndInvokeFunctionViaReflection(LibrarySupport.java:260) ~[prpublic.jar:?] 
	at com.pegarules.generated.pega_decisionengine_interactionhistory.pzStoreFactRecords(pega_decisionengine_interactionhistory.java:613) ~[?:?] 
	at com.pegarules.generated.pzStoreInInteractionHistory_080201__3072507141625005151.pzStoreInInteractionHistory08_02_01(pzStoreInInteractionHistory_080201__3072507141625005151.java:208) ~[?:?] 
	at com.pegarules.generated.pzStoreInInteractionHistory_080201__3072507141625005151.invoke(pzStoreInInteractionHistory_080201__3072507141625005151.java:77) ~[?:?] 
	at com.pega.pegarules.generation.internal.library.LibraryRuntime.resolveAndinvokeFunctionViaReflection(LibraryRuntime.java:238) ~[prprivate-generation.jar:?] 
	at com.pega.pegarules.generation.internal.library.LibraryRuntime.invokeLibraryRuntime(LibraryRuntime.java:125) ~[prprivate-generation.jar:?] 
	at com.pega.pegarules.session.internal.mgmt.Executable.invokeLibraryRuntime(Executable.java:9829) ~[prprivate-session.jar:?] 
	at com.pega.pegarules.priv.generator.LibrarySupport.resolveAndInvokeFunctionViaReflectionWithException(LibrarySupport.java:277) ~[prpublic.jar:?] 
	at com.pegarules.generated.dataset.ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65$2$1.handleAndResetBatch(ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65.java:306) ~[?:?] 
	at com.pegarules.generated.dataset.ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65$2$1.storeInIH(ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65.java:288) ~[?:?] 
	at com.pegarules.generated.dataset.ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65$2$1.emit(ra_action_pxinteractionhistory_624f284cfd1b2c283c9d1ee687491a65.java:235) ~[?:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl$3.emit(DataObservableImpl.java:176) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.dataset.ih.stream.IHSavingInstance.saveToIH(IHSavingInstance.java:139) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.dataset.ih.stream.IHSavingInstance.saveFilteredRecords(IHSavingInstance.java:110) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.dataset.ih.stream.IHSavingInstance.save(IHSavingInstance.java:63) ~[d-node.jar:?] 
	at com.pega.dsm.dnode.impl.dataset.ih.stream.IHCombinedSaveOperation.lambda$save$0(IHCombinedSaveOperation.java:52) ~[d-node.jar:?]

What did we try to solve this issue?

1. rule/clear api has been executed for all the respective nodes in the cluster.

1.png
1.png1450×460 15.4 KB

2. Revalidate and save operation has been executed for the type of Rule-Obj-Property in respective ruleset version includes FACTEXTENSIONID property rule.

4.png
4.png476×331 11.3 KB

3. One of Pega cache cleaner Agent’s activity has been manually executed.

4. All nodes in the cluster has been restarted along with deleting the cache files in pega_temp folder.

Unfortunately, after taking all the actions above, same issue was still there.

What else can we do? What could be causing this issue?

Regards.

Mert.

2

@m.caldag please log a support ticket via the MSP and provide the INC id here.

3

@MarijeSchillern

INC-B46574

4

@m.caldag I can see that INC-B46574 has been closed with the following summary:

The primary reason for BadTableMappingException is due to the lost synchronization between the Data-Admin-DB-Table instance and the physical changes(DDL being applied) on the underlying table.

In this case as you are adding new properties to the IH Fact table, moving the Data-Admin-DB-Table instance is important to have the required synchronization.

You have created properties in Upper case, it is not a necessary step, as properties can be created in mixed case as you have done originally.

As the next step you will do a cleanup of the table by dropping the already added column.

However, please keep in mind that you already have data in those column on Prod.

As a general cleanup you are going to validate these step between Dev and Staging ( However, please plan for backup of the data as this involves coloumn drop)

TO-DO
0. Re ADD FACTEXTENSIONID Column to PR_DATA_IH_FACT Table.

  1. Save DB Table Instance of Data-Decision-IH-Fact in dev environment.
    Prepare Product rule will include withdrawn properties (FACTEXTENSIONID, FactExtensionId)
    and DB Table Instance of Data-Decision-IH-Fact in dev environment.
  2. Deploy it to upper environment (One another testing environment apart from [removed by mod])
  3. Drop DB Table Column FACTEXTENSIONID

From the rule or data instance synchronization perspective, the Data-Admin-DB-Table instance is the one that should be visible across the entire cluster as this is the linkage between the underlying table and the class definition.
As a part of the SOP, as we discussed, always include the resaved version of the database table instance (data-Admin-DB-Table instance).

As Data-Decision-IH-Fact is an OOTB class you cannot update its external mapping, so the database table instance has to be resaved to avoid any class mapping issue. The reason for resaving is to correctly build the mapping, as the underlying physical table changes.

Perform the table operation in off hours when there is much less traffic.

5

Hi @MarijeSchillern thank you. I am adding the additional notes from the ticket to here as well.

Hello Abhijit,

Thank you for sharing your notes here and providing details during the call. I also want to add my comments on top of yours.

As we talked in the call, we have solved this issue with identifying new property as FACTEXTENSIONKEY and the column name as FACTEXTENSIONKEY and saving DB table instance on development environment and add them in product rule and deploy to production on 13.11.2024 before having our call but also in parallel we decided to consult you in order to find some logical solution if any instead of introducing new column and property.

I am listing the the outcomes and what we’ve learned

We come to the conclusion that this issue happening due to lack of rule synchronization among the nodes and it would be better to check Pega 0090 alerts from the PDC before deploying such critical change (Adding DB Column to OOTB IH tables, property and the DB table instance)

There is no manual triggering feature for rule synchronization so if something wrong with rule synchronization whilst deploying product through Deployment Manager and on top of post deployment if we recognize the BadTableMappingException exceptions while monitoring, only action we can take on production through the loadbalancer link of the production is resaving respective DB Table instance even if it is not providing the guarantee solution as we tried before and issue was still there. However, this is only action we have to take and monitor further.

As we discussed that we know that identifying property name with same as column name is not a mandatory thing, we will still stick with generating DB table column name and Pega property name same as all the characters upper case (both literally same). Because, we cannot save the Data-Decision-IH-Fact class with adding External mapping like FactExtensionId for property and FACTEXTENSIONID for column as I mentioned in the call. In addition, I have an experienced that for instance if we add AGE property MCCM-Data-Customer and the AGE column to its DB table, even if we are not setting the External mapping, pega is mapping them automatically after some time. I do believe that if we introduce the property all lower cases in this case we have to map both column and property from the external mapping configuration of the respective class.

Regarding dropping the column (FACTEXTENSIONID) which belongs to previous implementation causing issues on production, my colleague has taken the actions related TO-DOs we determined in the call and whenever we click test connection, we see the information below.

Class Data-Decision-IH-Fact is mapped to external data table PEGA_DATA_7_3_DEV.PR_DATA_IH_FACT in database PegaDATA, but the column FACTEXTENSIONID does not have the corresponding Rule-Obj-Property FACTEXTENSIONID

Alternatively, we are planning to drop the FACTEXTENSIONID on development database again and then save the DB Table instance of Data-Decision-IH-Fact class then prepare product includes the DB table instance. We may cut traffic a couple minutes whenever we drop the column from the production DB and deploy the product includes the DB Table instance belongs the Data-Decision-IH-Fact class and enable the traffic again. What do you think?

Thank you for your collaboration and support once again.
Regards.
Mert.

–

Hello Mert,

Thanks for the detailed write up.

From the rule or data instance synchronization perspective, the Data-Admin-DB-Table instance is the one that should be visible across the entire cluster as this is the linkage between the underlying table and the class definition.
As a part of the SOP, as we discussed, always include the resaved version of the database table instance (data-Admin-DB-Table instance). As Data-Decision-IH-Fact is an OOTB class you cannot update its external mapping, so the database table instance has to be resaved to avoid any class mapping issue. The reason for resaving is to correctly build the mapping, as the underlying physical table changes.
Perform the table operation in off hours when there is much less traffic.
Please let me know if you have any additional queries.
Also let me know if it is good to close this ticket.

Regards
Abhijit