Invalid configuration. Undefined stream provider end point.

General
, ,
1

Hello!

I have a Pega 8.4.1 application and it is throwing this exception:

(w.listener.RetryLoggerListener) ERROR - [pyBIBatchIndexClassesProcessor] Preparing retry 1091. Next attempt in 2m on data flow [pyBIBatchIndexClassesProcessor] on stage [InputStage] with affected partitions: [0, 1, 10, 11]
java.util.concurrent.CompletionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:284) ~[?:1.8.0]

The stream node has a status of LEAVING.

The only way I know to fix this is to restart the application.

Is there anything I can do to prevent the system from getting into this state?

Thanks!

Fred.

Spinning icon issue while open the popups in our application
2

Good question but unless we know the root cause of your unavailable stream service, hard to answer. Can you share the kafka logs covering the incidence period, especially period before this error happens?

3

Hi,

I have the exact same issue. After some days of running fine I get this exception, then my server is not responding properly, and it is restarting (hosted on kubernetes so restarting automaticaly with probes not repsonding)

Looking at the log just few seconds before for stream error I have this log :

@timestamp
2020-10-20T08:25:56.059Z
@version
1
app
level
ERROR
logger_name
com.pega.dsm.dnode.api.server.StreamServerService
message
Unexpected broker state NORMAL, current env StreamServiceCluster[
MemberInfo{address=10.240.0.XXX, prpcId=pega-pegademo-xxxxxxxx-7vltj, state=NORMAL, sessionId=xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxd92eb4e}

]
nodeType
Stream,WebUser,BackgroundProcessing,Search,Batch,RealTime,Custom1,Custom2,Custom3,Custom4,Custom5,BIX,ADM,RTDG
pegathread
STANDARD
source_host
pega-pegademo-xxxxxxxxxx-7vltj
stack
tenantid
thread_name
PegaRULES-HeatlthMonitorTaskExecutor

(format is from json export of container logs)

then this exception, full content here for more details :

@timestamp
2020-10-20T08:26:04.978Z
@version
1
CorrelationId
pyProcessCaseEvents
app
exception
{“stacktrace”:“java.util.concurrent.CompletionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point.\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)\n\tat java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)\n\tat java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)\n\tat java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)\n\tat com.pega.fnx.stream.spi.impl.metric.StreamSPIMetricsProxy.execute(StreamSPIMetricsProxy.java:41)\n\tat com.pega.fnx.stream.spi.impl.StreamSP.execute(StreamSP.java:68)\n\tat com.pega.fnx.stream.spi.loader.SPIProxy.execute(SPIProxy.java:48)\n\tat com.pega.platform.stream.StreamSPIStateValidator.execute(StreamSPIStateValidator.java:67)\n\tat com.pega.platform.stream.StreamSPIRequestValidator.execute(StreamSPIRequestValidator.java:57)\n\tat com.pega.fnx.stream.api.PollingSubscriber.poll(PollingSubscriber.java:80)\n\tat com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation.processRecords(StreamBrowseOperation.java:138)\n\tat com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation.access$000(StreamBrowseOperation.java:61)\n\tat com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation$1.emit(StreamBrowseOperation.java:107)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55)\n\tat com.pega.dsm.dnode.impl.dataflow.resilience.retry.RetryableBrowse$RetryableEmitter.produceRecords(RetryableBrowse.java:145)\n\tat com.pega.dsm.dnode.impl.dataflow.resilience.retry.RetryableBrowse$RetryableEmitter.emit(RetryableBrowse.java:111)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl$3.emit(DataObservableImpl.java:176)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353)\n\tat com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55)\n\tat com.pega.dsm.dnode.api.dataflow.DataFlow$3.run(DataFlow.java:432)\n\tat com.pega.dsm.dnode.api.dataflow.DataFlow$3.run(DataFlow.java:425)\n\tat com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67)\n\tat com.pega.dsm.dnode.impl.dataflow.DataFlowThreadContext.lambda$submitInput$3(DataFlowThreadContext.java:228)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)\n\tat com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)\n\tat com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:59)\n\tat com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:56)\n\tat com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67)\n\tat com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:124)\nCaused by: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point.\n\tat com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getBootstrapServers(KafkaSettingsProvider.java:157)\n\tat com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.defaultConsumerProperties(KafkaSettingsProvider.java:80)\n\tat com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getConsumerProperties(KafkaSettingsProvider.java:62)\n\tat com.pega.fnx.stream.spi.impl.kafka.KafkaServerImpl.getConsumer(KafkaServerImpl.java:49)\n\tat com.pega.fnx.stream.spi.impl.processor.SubscriberRequestProcessor.getConsumer(SubscriberRequestProcessor.java:41)\n\tat com.pega.fnx.stream.spi.impl.processor.GetMessageRequestProcessor.execute(GetMessageRequestProcessor.java:75)\n\tat com.pega.fnx.stream.spi.impl.processor.GetMessageRequestProcessor.execute(GetMessageRequestProcessor.java:46)\n\tat com.pega.fnx.stream.spi.impl.StreamSPNative.execute(StreamSPNative.java:61)\n\t… 32 more\n”,“exception_class”:“java.util.concurrent.CompletionException”,“exception_message”:“com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point.”}
exception_class
java.util.concurrent.CompletionException
exception_message
com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point.
stacktrace
java.util.concurrent.CompletionException: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point.
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)
at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
at com.pega.fnx.stream.spi.impl.metric.StreamSPIMetricsProxy.execute(StreamSPIMetricsProxy.java:41)
at com.pega.fnx.stream.spi.impl.StreamSP.execute(StreamSP.java:68)
at com.pega.fnx.stream.spi.loader.SPIProxy.execute(SPIProxy.java:48)
at com.pega.platform.stream.StreamSPIStateValidator.execute(StreamSPIStateValidator.java:67)
at com.pega.platform.stream.StreamSPIRequestValidator.execute(StreamSPIRequestValidator.java:57)
at com.pega.fnx.stream.api.PollingSubscriber.poll(PollingSubscriber.java:80)
at com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation.processRecords(StreamBrowseOperation.java:138)
at com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation.access$000(StreamBrowseOperation.java:61)
at com.pega.dsm.dnode.impl.dataset.stream.StreamBrowseOperation$1.emit(StreamBrowseOperation.java:107)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55)
at com.pega.dsm.dnode.impl.dataflow.resilience.retry.RetryableBrowse$RetryableEmitter.produceRecords(RetryableBrowse.java:145)
at com.pega.dsm.dnode.impl.dataflow.resilience.retry.RetryableBrowse$RetryableEmitter.emit(RetryableBrowse.java:111)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$3.emit(DataObservableImpl.java:176)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl$SafeDataSubscriber.subscribe(DataObservableImpl.java:353)
at com.pega.dsm.dnode.impl.stream.DataObservableImpl.subscribe(DataObservableImpl.java:55)
at com.pega.dsm.dnode.api.dataflow.DataFlow$3.run(DataFlow.java:432)
at com.pega.dsm.dnode.api.dataflow.DataFlow$3.run(DataFlow.java:425)
at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67)
at com.pega.dsm.dnode.impl.dataflow.DataFlowThreadContext.lambda$submitInput$3(DataFlowThreadContext.java:228)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:59)
at com.pega.dsm.dnode.util.PrpcRunnable$1.run(PrpcRunnable.java:56)
at com.pega.dsm.dnode.util.PrpcRunnable.execute(PrpcRunnable.java:67)
at com.pega.dsm.dnode.impl.prpc.PrpcThreadFactory$PrpcThread.run(PrpcThreadFactory.java:124)
Caused by: com.pega.fnx.stream.spi.StreamServiceException: Invalid configuration. Undefined stream provider end point.
at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getBootstrapServers(KafkaSettingsProvider.java:157)
at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.defaultConsumerProperties(KafkaSettingsProvider.java:80)
at com.pega.fnx.stream.spi.impl.kafka.KafkaSettingsProvider.getConsumerProperties(KafkaSettingsProvider.java:62)
at com.pega.fnx.stream.spi.impl.kafka.KafkaServerImpl.getConsumer(KafkaServerImpl.java:49)
at com.pega.fnx.stream.spi.impl.processor.SubscriberRequestProcessor.getConsumer(SubscriberRequestProcessor.java:41)
at com.pega.fnx.stream.spi.impl.processor.GetMessageRequestProcessor.execute(GetMessageRequestProcessor.java:75)
at com.pega.fnx.stream.spi.impl.processor.GetMessageRequestProcessor.execute(GetMessageRequestProcessor.java:46)
at com.pega.fnx.stream.spi.impl.StreamSPNative.execute(StreamSPNative.java:61)
… 32 more
level
ERROR
logger_name
com.pega.dsm.dnode.impl.dataflow.listener.RetryLoggerListener
message
[pyProcessCaseEvents] Preparing retry 1. Next attempt in 1s on data flow [pyProcessCaseEvents] on stage [InputStage] with affected partitions: [0, 1, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 2, 3, 4, 5, 6, 7, 8, 9]
nodeType
Stream,WebUser,BackgroundProcessing,Search,Batch,RealTime,Custom1,Custom2,Custom3,Custom4,Custom5,BIX,ADM,RTDG
pegathread
STANDARD
source_host
pega-pegademo-xxxxxxxxxxxx-7vltj
stack
tenantid
thread_name
DataFlow-Service-PickingupRun-pyProcessCaseEvents_Input:100, Access group: [PRPC:AsyncProcessor], Partitions=[0,1,10,11,12,13,14,15,16,17,18,19,2,3,4,5,6,7,8,9]

4

can’t get kafka logs as they are deleted when container is restarted.

Would be great if next Pega image could publish kafka logs as container logs !

5

Well, got lucky and was able to download diagnostic laogs from Pega portal when I got the error logs again.See attached zip file generated by pega when the issue reproduced.

I am not a Kafka expert but not able to see anything wrong in those logs.

This time issue started on Pega logs at 2:27:19 PM

kafkaDiagnostics_1603204315032.zip (120 KB)

6

Hi,

Created a support ticket for this issue INC-151877.

Fixed it by changing the helm char to use stateless instead of deployments, and applying a hotfix that fix an issue where long queries kill the process.

HFIX-65676

I still have the issue but much less.

Hope this help other people.

Regards,

Marc-Antoine

7

@FredG487 I also see this error in logs and I think that the root cause may be due to a Kafka failure. I see this error earlier in the logs:-

ERROR - Failed to start Kafka on 1 attempt, kafka log ERROR StatusLogger Unable to access file:/F:/Program%20Files/Apache%20Software%20Foundation/Tomcat%209.0/kafka-1.1.0.4/bin/file:F:/Program%20Files/Apache%20Software%20Foundation/Tomcat%209.0/kafka-1.1.0.4/config/log4j2.xml java.io.FileNotFoundException: F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-1.1.0.4\bin\file:F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-1.1.0.4\config\log4j2.xml (The filename, directory name, or volume label syntax is incorrect) at java.base/java.io.FileInputStream.open0(Native Method) at

…

kafka.utils.Logging$class.$init$(Logging.scala:47) at kafka.Kafka$.(Kafka.scala:30) at kafka.Kafka$.(Kafka.scala) at kafka.Kafka.main(Kafka.scala)ERROR StatusLogger Reconfiguration failed: No configuration found for ‘2aae9190’ at ‘null’ in 'null’12:00:18.879 [kafka-log-cleaner-thread-0] ERROR kafka.server.LogDirFailureChannel - Failed to clean up log for __consumer_offsets-0 in dir F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-data due to IOExceptionjava.nio.file.FileSystemException: F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-data__consumer_offsets-0\00000000000000000000.log.cleaned → F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-data__consumer_offsets-0\00000000000000000000.log.swap: The process cannot access the file because it is being used by another process. at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) ~[?:?]

…

1.1.0.4.jar:?] at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:291) [kafka_2.11-1.1.0.4.jar:?] at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82) [kafka_2.11-1.1.0.4.jar:?]12:05:22.088 [LogDirFailureHandler] ERROR kafka.log.LogManager - Shutdown broker because all log dirs in F:\Program Files\Apache Software Foundation\Tomcat 9.0\kafka-data have failed

The only solution I have at the moment is to delete the Kaka folders under Tomcat and restart.

8

@armid : After upgrading from 8.3 to 8.6 Pega we are getting same error, is there any fix provided for this ?

Please let us know.

9

Then I get an update on servers topology

Services topology has changed. Current topology:
This node ==> ADM.Default pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 e8430c81-8f6f-4d2e-8370-d554da9c78a5
This node ==> Aggregation.Default pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 589ca581-f5d7-4ac3-b577-71e354dacfd2
This node ==> DataFlow.BackgroundProcessing pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 0a2e8192-5328-42c7-8041-7c84cd1911fd
This node ==> DataFlow.Batch pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 1b5e520e-9645-43df-8d54-6e41285d070c
This node ==> DataFlow.PegaMKTEmail pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 0cd435e7-c4d7-4b04-9bcd-b8d53b2f97ec
This node ==> DataFlow.PegaMKTSMS pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 d8d16b84-3a15-4c3d-b987-9a19b95384d5
This node ==> DataFlow.RealTime pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 20498716-0723-4aea-bf21-d047eb61bb9a
This node ==> DataFlow.Search pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 8047d3f8-be58-4a6f-b5cd-b1697b3b6743
This node ==> StreamServer.Default pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx STOPPED (was NORMAL) 8.4.2 87006bb9-4899-4b05-9399-72b1ad92eb4e
This node ==> VBD.Default pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx NORMAL 8.4.2 fc334fcf-315b-42e0-8b0a-1e92b7e2deaf
This node ==> pega pega-pegademo-xxxxxxxxxx-7vltj 10.240.0.xxx running 8.4.2 39486aed-64f8-4801-ab13-35d1f9f187c7