Upgrade to 8.5.3 - Restart app server always stuck on loading cache

General
,
1

We have upgraded our application from Pega 7.1.7 to Pega 8.5.3. When we truncate all the cache tables, Pega is up and running cleanly, but if we restart the application server after deploying any war file app server is stuck at below.

2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    - JVM Information:
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -     Vendor: Oracle Corporation, JVM version: 1.8.0_291
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -     VM Name: Java HotSpot(TM) 64-Bit Server VM(build 1.8.0_291-b10)
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    - Host OS Information:
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -     OS: Windows Server 2019, version: 10.0
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -     Architecture: amd64
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    - PegaRULES Enterprise Tier 8.5.3
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    - PRPC-8.5.3-551: 2021-03-10 14.22 EST
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    - Deployed in web tier only
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -                        Servlet container: Tomcat
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -                               Web server: Apache Tomcat/8.5.65
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -                   Web server environment: Tomcat
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -        Web-tier default Engine bean name: EngineBMT
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    -     Web to Enterprise tier delegate type: pojo
2021-05-14 10:01:32,952 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      etier.impl.EngineStartup) INFO    - Now starting the PegaRULES Engine...
2021-05-14 10:01:34,339 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      internal.mgmt.PRNodeImpl) WARN    - tier name was not provided. setting it to default
2021-05-14 10:01:36,085 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (ary.ClassMapDatabasePageReader) INFO    - Read 8485 class definitions
2021-05-14 10:01:36,883 [ocalhost-startStop-1] [          ] [                    ] [                    ] (      internal.mgmt.PRNodeImpl) INFO    - Using codeset version and timestamp map: {pega-dsmcode=06-02-20@2012-06-13 13:28:00.0, pdfgen=01-01-01@2015-03-24 17:06:00.0, pega-enginecode=08-05-04@2021-05-12 16:44:00.0, geoprojection=06-01-01@2015-10-29 10:16:00.0, customer=06-01-01@1970-01-01 10:00:00.0}
2021-05-14 10:01:37,727 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (ternal.dictionary.ClassMapImpl) INFO    - ClassMapData is preloaded
2021-05-14 10:01:38,541 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (ternal.dictionary.ClassMapImpl) INFO    - ClassMap initialization completed in 7753 ms
2021-05-14 10:01:38,545 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (  internal.access.DatabaseImpl) INFO    - Database product: Microsoft SQL Server
2021-05-14 10:01:38,545 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (  internal.access.DatabaseImpl) INFO    - Database version: 11.00.7001
2021-05-14 10:01:38,545 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (  internal.access.DatabaseImpl) INFO    - Driver: Microsoft JDBC Driver 6.0 for SQL Server
2021-05-14 10:01:38,545 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (  internal.access.DatabaseImpl) INFO    - Driver version: 6.0.8112.200
2021-05-14 10:01:38,545 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (  internal.access.DatabaseImpl) INFO    - Version of JDBC supported by driver: 4.2
2021-05-14 10:01:39,278 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (  internal.access.DatabaseImpl) INFO    - This system does not support batch ID generation, using legacy ID generation
2021-05-14 10:01:39,934 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (        internal.access.Lister) INFO    - Fetch size for certain Postgres RDB queries is enabled
2021-05-14 10:01:43,555 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (rnal.access.CacheContainerImpl) INFO    - Using the Dynamic Cache Probation Policy - cache after 2 accesses AND with a blob increment of 102400.
2021-05-14 10:01:44,854 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (rnal.access.CacheContainerImpl) INFO    - Using the Dynamic Cache Probation Policy - cache after 2 accesses AND with a blob increment of 102400.
2021-05-14 10:01:44,854 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (rnal.access.CacheContainerImpl) INFO    - Using the Dynamic Cache Probation Policy - cache after 2 accesses AND with a blob increment of 102400.
2021-05-14 10:01:44,854 [StartUp-TaskExecutor] [          ] [                    ] [                    ] (rnal.access.CacheContainerImpl) INFO    - Using the Dynamic Cache Probation Policy - cache after 2 accesses AND with a blob increment of 102400.

All the information related to software’s are in the above log file.

2

@kandi_ak Could you please share full Pegarules and alert logs generated during restart.

3

@ravik2 Attached is the PegaRULES log file. Nothing generated in Alerts log file.

PegaRULES.txt (13.5 KB)

4

@kandi_ak We are also facing same issue, did you got any solution, could you please share the same

5

@ravik2 We are also getting same issue , post upgrade tried to start the server but after some time it got stuck and don’t see any progress.

last msg in log : Using the Dynamic Cache Probation Policy - cache after 2 accesses AND with a blob increment of 102400.

6

@kandi_ak Can you confirm if you followed the steps provided in installation guide?

This issue has been known to happen when needed Datasource connection pool configurations and PegatempDir locations are missing

Suggestion:

Apply the following param and connection pool to the Pegarules/Datasource Connection pooling configuration –

factory=“org.apache.tomcat.jdbc.pool.DataSourceFactory”
maxTotal=“100”
maxIdle=“30”
maxWaitMillis=“10000”

i.e.

<Resource name=“jdbc/PegaRULES”
auth=“Container”

type=“javax.sql.DataSource”

factory=“org.apache.tomcat.jdbc.pool.DataSourceFactory”

maxActive=“100”

minIdle=“10”

maxWait=“60000”

driverClassName=“org.postgresql.Driver”

url=“hidden”

username=“hidden”

password=“”

connectionProperties=“”

validationQuery=“SELECT 1”

validationInterval=“30000”

testWhileIdle=“true”

timeBetweenEvictionRunsMillis=“10000”

/>

NEXT STEPS:
If the above does not help you may need to log a support ticket in order to have an engineer look at it in detail. Please provide the INC ticket id here so we can track it.

In the INC support ticket please provide the team with the below information:

    • What version of Postgres are you running?

  2. When you checked the logs did you see any errors?

  3. How many nodes do you have in this environment?

4 Are you using any framework/application on top of the base Pega version as well?

5- Please provide a copy of your prconfig , setupdatabase.properties file and context.xml file for your Tomcat configuration.

7

@yarlp Changing DNodeType didn’t work but update to connection pool settings worked. I was trying to load 8.7.5 with PostgreSQL 13.10, Tomcat 9.0.71 with OpenJDK 1.8.0.345 (RedHat).

8

@DharanitharanR5430 Did you got any solution for this issue .

Have tried to truncate all the cache tables in database and cleared temp file then again re started the server but still getting this issue.

9

@MarijeSchillern While I was setting-up Pega 8.8 in my local, I ran into the same issue and the suggested solution worked for me.

Earlier also, I installed 8.8 version but didn’t face this issue. The only difference I could remember is earlier, I set

-DNodeType=Stream,BackgroundProcessing,WebUser,Search but now I set it as -DNodeType=Universal.

Not sure whether it makes any difference in the installation process.