Error "Cannot add new node. Make sure all listed nodes are available or reattach the volume from one of these nodes to this node"

General
, , , ,
1

Any suggestion on this issue…

2024-09-24 14:58:49,283 [StreamServer.Default] [ STANDARD] (rvice.operation.StartOperation) ERROR - Cannot
start service [StreamServer.Default]. Will retry in 180 seconds. Remaining attempts: 2
com.pega.dsm.dnode.api.StreamServiceException: Cannot add new node. Make sure all listed nodes are available or reattach the volume from one of these nodes t
o this node.
[Node=pegabatchserver101, broker=1005]
at com.pega.dsm.dnode.api.server.StreamServerService$StreamServiceStartOperation$2.emit(StreamServerService.java:675) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:117) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:106) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation$1.execute(StartOperation.java:167) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.OperationWithLock$LockingOperation.couldAcquireLock(OperationWithLock.java:190) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.OperationWithLock$LockingOperation.performLockOperation(OperationWithLock.java:157) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.OperationWithLock$LockingOperation.access$200(OperationWithLock.java:102) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:99) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.OperationWithLock.doWithLock(OperationWithLock.java:95) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeWithLockInternal(ServiceHelper.java:273) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeWithLock(ServiceHelper.java:221) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.doActualServerStart(StartOperation.java:164) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.performStartupWithRetries(StartOperation.java:137) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.initializeServerMode(StartOperation.java:117) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.prpc.service.operation.StartOperation.lambda$bootstrap$0(StartOperation.java:85) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:117) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.await(DataObservableImpl.java:106) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.ServiceDefinition.startService(ServiceDefinition.java:81) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.ServiceDefinition.start(ServiceDefinition.java:66) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.prpc.service.ServiceManager$4.run(ServiceManager.java:429) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:77) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContextInternal(ServiceHelper.java:305) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.service.ServiceHelper.executeInPrpcContext(ServiceHelper.java:150) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.prpc.service.ServiceManager.startServiceDefinition(ServiceManager.java:426) ~[d-node.jar:?]
at com.pega.dsm.dnode.api.prpc.service.ServiceManager.lambda$bootstrap$3(ServiceManager.java:388) ~[d-node.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_351]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125
) ~[guava-25.1-jre.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) ~[guava-25.1-jre.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) ~[guava-25.1-jre.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_351]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_351]
at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:69) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:66) ~[d-node.jar:?]
at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:77) ~[d-node.jar:?]
at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:164) ~[d-node.jar:?]

2024-09-24 15:07:52,892 [StreamServer.Default] [ STANDARD] (prpc.service.ServiceDefinition) ERROR - Failed
to start service: [StreamServer.Default]
com.pega.dsm.dnode.api.StreamServiceException: Cannot add new node. Make sure all listed nodes are available or reattach the volume from one of these nodes t
o this node.

Application is not accessible due to this issue

2

@NATESHG1

At first sight, your initially referenced stream node is not there anymore.

If you wait for long, your application will finally come up and you can go to Decisioning > Infrastructure > Stream and you’ll see old stream node ref that you can decom and the new one that you can start.

You can also consider clearing all below tables to set back all stream data/infra from scratch.

pr_data_stream_nodes;
pr_data_stream_sessions;
pr_data_stream_node_updates;

These last 2 holds the references of the stream nodes registered.
pr_sys_serviceregistry;
pr_sys_serviceregistry_kvs;

Regards

Anthony