Could not initialize services of type [StreamServer]

kaspk · March 15, 2021, 2:26pm

Hi All

We are trying to perform an out-of-place upgrade from 6.3 to 8.5.1

We followed below steps:

Install clean 8.5.1
Copy pega_rules and pega_data schema from 6.3 and run upgrade.sh
Point websphere to upgraded schemas in step 2.

Along with above major steps, there are other minor steps like cleanup, adding privileges, etc.

We are facing below error on startup. We have also updated jdk config to not inlcude ‘Stream’ node for -DNodeType. Start up is still failing with NullPointerException. Does anyone know how to solve this?

2021-03-15 11:45:11,701 [  wokvl0-sam-upgrade] [  STANDARD] [                    ] [                    ] (      internal.mgmt.PRNodeImpl) INFO    - Created provisional node: "wokvl0-sam-upgrade pega 2021-03-15 11:45:00.885 GMT", []

2021-03-15 11:45:12,149 [  wokvl0-sam-upgrade] [  STANDARD] [                    ] [                    ] (   prpc.service.ServiceManager) ERROR   - Could not initialize services of type [StreamServer]

2021-03-15 11:45:12,191 [  wokvl0-sam-upgrade] [  STANDARD] [                    ] [                    ] (   internal.mgmt.PREnvironment) ERROR   - java.lang.NullPointerException

2021-03-15 11:45:12,192 [  wokvl0-sam-upgrade] [  STANDARD] [                    ] [                    ] (      etier.impl.EngineStartup) ERROR   - PegaRULES initialization failed. Server: wokvl0-sam-upgrade

com.pega.pegarules.pub.context.InitializationFailedError: PRNodeImpl init failed

        at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:427) ~[prprivate-session.jar:?]

        at com.pega.pegarules.session.internal.PRSessionProviderImpl.getThreadAndInitialize(PRSessionProviderImpl.java:2176) ~[prprivate-session.jar:?]

        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineStartup.initEngine(EngineStartup.java:683) ~[prprivate-session.jar:?]

        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._initEngine_privact(EngineImpl.java:180) ~[prprivate-session.jar:?]

        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.doStartup(EngineImpl.java:152) ~[prprivate-session.jar:?]

        at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.doStartup(EngineBean.java:127) ~[prprivate-session.jar:?]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]

        at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]

        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:388) ~[prbootstrap.jar:8.5.1-208]

        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:430) ~[prbootstrap.jar:8.5.1-208]

        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:225) ~[prbootstrap-api.jar:8.5.1-208]

        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:247) ~[prbootstrap-api.jar:8.5.1-208]

        at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.doStartup(EngineBeanBoot.java:135) ~[prbeans.jar:8.5.1-208]

        at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.doStartup(Unknown Source) ~[prbeans.jar:8.5.1-208]

        at com.pega.pegarules.web.servlet.WebAppLifeCycleListener._contextInitialized_privact(WebAppLifeCycleListener.java:210) ~[prwebj2ee.jar:?]

        at com.pega.pegarules.web.servlet.AbstractLifeCycleListener._contextInitialized_privact(AbstractLifeCycleListener.java:145) ~[prwebj2ee.jar:?]

        at com.pega.pegarules.web.servlet.AbstractLifeCycleListener.contextInitialized(AbstractLifeCycleListener.java:76) ~[prwebj2ee.jar:?]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[?:1.8.0]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[?:1.8.0]

        at java.lang.reflect.Method.invoke(Method.java:508) ~[?:1.8.0]

        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:388) ~[prbootstrap.jar:8.5.1-208]

        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:430) ~[prbootstrap.jar:8.5.1-208]

        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:225) ~[prbootstrap-api.jar:8.5.1-208]

        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:274) ~[prbootstrap-api.jar:8.5.1-208]

        at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:92) ~[prbootstrap-api.jar:8.5.1-208]

        at com.ibm.ws.webcontainer.webapp.WebApp.notifyServletContextCreated(WebApp.java:1837) ~[com.ibm.ws.webcontainer.jar:?]

        at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:443) ~[com.ibm.ws.webcontainer.jar:?]

        at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88) ~[com.ibm.ws.webcontainer.jar:?]

        at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:171) ~[com.ibm.ws.webcontainer.jar:?]

        at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:904) ~[com.ibm.ws.webcontainer.jar:?]

        at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:789) ~[com.ibm.ws.webcontainer.jar:?]

        at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:427) ~[com.ibm.ws.webcontainer.jar:?]

        at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:719) ~[com.ibm.ws.webcontainer.jar:?]

        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1249) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1590) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:707) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:1161) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:801) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2325) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5446) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5662) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2330) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:436) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:379) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:127) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:985) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:524) ~[com.ibm.ws.runtime.jar:?]

        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1909) ~[com.ibm.ws.runtime.jar:?]

Caused by: java.lang.NullPointerException

        at com.pega.dsm.dnode.api.server.StreamServiceInitializationTask.validateStreamAPI(StreamServiceInitializationTask.java:95) ~[d-node.jar:?]

        at com.pega.dsm.dnode.api.server.StreamServiceInitializationTask.access$100(StreamServiceInitializationTask.java:32) ~[d-node.jar:?]

        at com.pega.dsm.dnode.api.server.StreamServiceInitializationTask$1.run(StreamServiceInitializationTask.java:82) ~[d-node.jar:?]

        at com.pega.dsm.dnode.api.server.StreamServiceInitializationTask$1.run(StreamServiceInitializationTask.java:77) ~[d-node.jar:?]

        at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper$2.run(ServiceHelper.java:291) ~[d-node.jar:?]

        at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1388) ~[prprivate-session.jar:?]

        at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1131) ~[prprivate-session.jar:?]

        at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:938) ~[prprivate-session.jar:?]

        at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:284) ~[d-node.jar:?]

        at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:146) ~[d-node.jar:?]

        at com.pega.dsm.dnode.api.server.StreamServiceInitializationTask.initStreamServerServiceAndValidate(StreamServiceInitializationTask.java:77) ~[d-node.jar:?]

        at com.pega.dsm.dnode.api.server.StreamServiceInitializationTask.runTask(StreamServiceInitializationTask.java:51) ~[d-node.jar:?]

        at com.pega.pegarules.session.internal.async.startup.StartupTaskUtil.runTask(StartupTaskUtil.java:89) ~[prprivate-session.jar:?]

        at com.pega.pegarules.session.internal.mgmt.PRNodeImpl.initializeSystem(PRNodeImpl.java:1034) ~[prprivate-session.jar:?]

        at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:407) ~[prprivate-session.jar:?]

        ... 52 more

PrakashKaragi · March 15, 2021, 2:43pm

@kaspk

Please try the workaround solutions suggested as part of below articles.

Let me know if you have tried already

kaspk · March 15, 2021, 3:01pm

@PrakashKaragi We do not have these DSS values in the pr_data_admin table.

Also, for us, the error seems about Stream node

Could not initialize services of type [StreamServer]

As mentioned we have also removed it from the -DNodeType in jdk config. However, this issue is still there.

PrakashKaragi · March 15, 2021, 3:15pm

@kaspk

Does the below queries returns 0 records ?

select * from .pr_data where pxobjclass like ‘Data-Decision-Service%’ and pxinsname=‘DEFAULT’;
select * from .pr_data where pxobjclass like ‘Data-Decision-Service%’;

kaspk · March 15, 2021, 3:21pm

@PrakashKaragi No, these queries are returning one single record for

DATA-DECISION-SERVICE-DDS DEFAULT

Do I need to get rid of this record?

PrakashKaragi · March 15, 2021, 3:29pm

@kaspk

I am suspecting the Data instances related to 'Data-Decision-Service’ are deleted from the PR_DATA table.

Perform the following local-change:

Export the data instances identified by the below query from a working environment and import the missing data instances in the environment having the issue:
select * from .pr_data where pxobjclass like ‘Data-Decision-Service%’;
Restart the Pega nodes.

If this does not help then please report the support request as this needs detailed investigations

kaspk · March 15, 2021, 5:57pm

@PrakashKaragi Thank you Prakash

After importing these entries, we could restart our server.

Conversation		Replies	Views
On-premise '23 Pega platform does not start with Tomcat General pega-platform , lead-system-architect , system-administration , decision-management , installation-and-deployment , consumer-services	11	2249	November 17, 2023
Background node does not start on On-premise Pega '23 General pega-platform , installation-and-deployment , java-and-activities , dx-api	9	1060	January 26, 2024
Pega Platform 8.23.0 General system-administration , updates , 23-1-1	2	528	April 25, 2024
ERROR - Cannot start service [StreamServer.Default]. Will retry in 180 seconds. Remaining attempts: 2 com.pega.dsm.dnode.api.StreamServiceException: Unable to start Kafka broker. Last state was: NotConnected [2024-07-19 19:11:38,908] ERROR Fatal error d General pega-platform , 8-8-3	1	436	August 6, 2024
INITIALIZE_STREAM_SERVICE failing on pod startup General pega-platform , system-administration , decision-management , installation-and-deployment , performance , 24-1-2	2	261	January 15, 2025

Could not initialize services of type [StreamServer]

Related topics