Hazlecast: Member has left the cluster

General
, ,
1

Hi All,

We are running the NBAD scheduler campaign, and it has failed with the error:

2022-12-27 13:48:40,850 [ PegaRULES-Batch-15] [ STANDARD] [ ] [ BCARL:01.02.01] (PegaMKT_Work_ProgramRun.Action) ERROR BE2N8FRDZLWVNVBASIN39LX6RSDVPJ9XRA - Failure in ExecuteProgramRun: PRRuntimeError:PR-70014

and we found the following error has been observed in the log:

2022-12-27 12:05:01,632 [d0e90dc72a30.event-3] [ ] [ ] [ ] (til.HazelcastMembershipManager) INFO - Member has left the cluster: Member: [name=eb06fdafab8102eb1131ab3b508317aa, address=RHU3PEGADTFL01/10.43.3.153:5701, uuid=d787d1b5-e60a-462e-a72a-37672dd0cdba, member version=3.12.10, mode=SERVER]

Could someone please provide insights about this? Thank you.

PFA’s for the logs

PegaRULES-ALERT-Batch,RealTime.zip (122 KB)

PegaRULES-Batch,RealTime.zip (7.62 KB)

2

@AnggunMD

Member left the cluster issue could be for multiple reasons.

  1. In-sufficient Requestor Pool Thread

  2. Hung threads

  3. If Stream, the member is trying to refer the old kafka partitions which requires cleaning of System cache Tables etc..

If your application is running in On-Prem, You can idenitfy the node type and perform the rollover restart to monitor if it’s joining the cluster back or not. Sometimes, it’s required to perform the whole system maintenance if Stream node left the cluster or any issues with Cluster on Joining back.

It’s advisable to raise an INC in MSP portal for further alaysis on your application.

3

Hi @KishoreSanagapalli

Thank you for your response. Our application is running on-premises, and for this issue, we performed the restart and monitored the cluster joining back, and our campaign is running properly again.
Thank you.