ADM nodes are restarted with NullPointerException!

General
, , , ,
1

Hi,

Greetings. Hope you are doing well! We are on containarised deployments and recently observed that ADM nodes are getting restarted with NullPointerException when communicating with HazelCast Clustering Service, please see below the exception.

Couldn’t connect to the service, [1] retrying in 1 seconds…

heartbeat",“@version”:1,“tenantid”:“”,“logger_name”:“com.pega.platform.serviceregistry.internal.database.tasks.TopologyChangedDetector”}
{“exception”:{“stacktrace”:“java.lang.NullPointerException\n\tat com.pega.decision.adm.service.ADMService.handleMessage(ADMService.java:684)\n\tat com.pega.dsm.dnode.api.prpc.service.AbstractDsmService.handleMessageInternal(AbstractDsmService.java:175)\n\tat com.pega.dsm.dnode.impl.prpc.service.PostMessageJob$PostMessageJobData.handleMessage(PostMessageJob.java:87)\n\tat com.pega.dsm.dnode.impl.prpc.service.RegistryBasedDeployment.lambda$new$0(RegistryBasedDeployment.java:116)\n\tat com.pega.platform.messaging.internal.hazelcastv5.HazelcastTopic.lambda$null$1(HazelcastTopic.java:58)\n\tat com.pega.hazelcast.v5.client.impl.proxy.ClientTopicProxy$TopicItemHandler.handleTopicEvent(ClientTopicProxy.java:132)\n\tat com.pega.hazelcast.v5.client.impl.protocol.codec.TopicAddMessageListenerCodec$AbstractEventHandler.handle(TopicAddMessageListenerCodec.java:139)\n\tat com.pega.hazelcast.v5.client.impl.proxy.ClientTopicProxy$TopicItemHandler.handle(ClientTopicProxy.java:120)\n\tat com.pega.hazelcast.v5.client.impl.spi.impl.listener.ClientListenerServiceImpl.handleEventMessageOnCallingThread(ClientListenerServiceImpl.java:190)\n\tat com.pega.hazelcast.v5.client.impl.spi.impl.listener.ClientListenerServiceImpl$ClientEventProcessor.run(ClientListenerServiceImpl.java:361)\n\tat com.pega.hazelcast.v5.internal.util.executor.StripedExecutor$Worker.process(StripedExecutor.java:245)\n\tat com.pega.hazelcast.v5.internal.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:228)\n”,“exception_class”:“java.lang.NullPointerException”},“app”:“”,“stack”:“”,“source_host”:“umb2cprdpega8-pega-adm-5dc8475d7d-hwpwf”,“level”:“FATAL”,“appender_ref”:“CLUSTER”,“message”:"hz.client_1 [prpchz] [5.1.5] hz.client_1.event-1 caught an exception while processing:com.pega.hazelcast.v5.client.impl.spi.impl.listener.ClientListenerServiceImpl$ClientEventProcessor@

Any ideas on what is causing this issue?

Regards,

Bharat

2

@KOMARINA please can you log this as a support incident via the MSP? Please provide the INC id here so we can help track it.

3

@MarijeSchillern Hi, here is the INC-A24720, this incident was resolved as we have not been able to replicate this on-demand and after a cluster restart, this did not happen again.

Regards,

Bharat

4

@KOMARINA

Thanks Bharat! I have updated your post with the Incident ID and marked your reply as the resolution.