Stream Service is stopped

General
, , , ,
1

The Stream service is stopping regularly , when timestamps are verified between servers then the following exceptions are found in Kafka server logs file . Attached the complete log file.

Seems to be trying to clean up the log file but unable to do it and then the stream service is getting stopped.

Once stream node is stopped then getting many exceptions in all nodes .

I found Pega article related to same problem but it Applies to Pega Platform™ version 8.5 through Pega Platform version 8.7 . we are on Pega 8.8.1

Kafka.jpg
Kafka.jpg1859×683 367 KB

Kafka_Log.txt (20.5 KB)

2

@Brahmesh@ I have checked internally and was able to find another Pega 8.8 client who experienced the same errors.

If the stream node is running on windows server then you may be experiencing the issue described in Support Document Kafka fails to clean up compacted topics (Windows Server 2016)

Note that this is not Pega issue but kafka limitation on Windows.

The changes shown in the article should prevent this issue. Also note that for any reason if kafka-data folder is deleted from sesrver, you have to perform above changes.

If the above resolves your issue I will contact the author of the Support Document in order to ask that Pega 8.8 version be added to the ‘applies to’ list.

3

@Brahmesh@ Based on the log you have shared and the error messages, it appears the Stream Services are not stopped gracefully.

When Steam Services are not externalized and had a improper shutdown of the Steam Services, it wll start behaving differently, usually kafka try to create a topic but that topic already exist or running from other stream node , since it was an ungraceful shutdown, no control communication between the members of the Kafka cluster.

Perhaps, review following steps to troubleshoot the problem.

– Stream Nodes Issue —

Shutdown Stream Nodes [Make sure both Kafka and Pega JVM Stopped]

Truncate Table dbo.pr_data_stream_node_updates;
Truncate Table dbo.pr_data_stream_nodes;
Truncate Table dbo.pr_data_stream_sessions;
Truncate Table dbo.pr_data_qp_run_partition;

Delete Kafka-Data

===

Stream Nodes Should be Stopped One at a Time.
Stop Sequence is exact Opposite from Start Sequence
Always One Stream Node should be up and running for Pega before any node type node can come up
Web User Nodes always can be started last[Just best Practice]. Stopping doesnt matter.
Search Nodes always one at a time .. Typically Primary Search Node followed by Secondary Search Node should start first. While stopping same Order can be followed.

4

@SUMAN_GUMUDAVELLY I tried the steps that suggested but still getting the below exception in kafka server log.

we have only one stream node in the cluster but unable to figure out with this exception.

Cluster step

Cluster.jpg
Cluster.jpg421×272 22.1 KB

[2023-07-06 14:02:05,856] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PZSTANDARDPROCESSOR with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.TopicExistsException: Topic ‘PZSTANDARDPROCESSOR’ already exists.
[2023-07-06 14:02:05,919] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PZATTACHMENTMIGRATIONPROCESSOR with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.TopicExistsException: Topic ‘PZATTACHMENTMIGRATIONPROCESSOR’ already exists.
[2023-07-06 14:02:05,981] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PYPROCESSNOTIFICATION with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.TopicExistsException: Topic ‘PYPROCESSNOTIFICATION’ already exists.
[2023-07-06 14:02:06,041] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PYBIBATCHINDEXPROCESSOR with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.TopicExistsException: Topic ‘PYBIBATCHINDEXPROCESSOR’ already exists.
[2023-07-06 14:02:06,105] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic ASYNCPROCESSING with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.TopicExistsException: Topic ‘ASYNCPROCESSING’ already exists.
[2023-07-06 14:02:06,169] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic ASYNCPROCESSINGDELAYED with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.TopicExistsException: Topic ‘ASYNCPROCESSINGDELAYED’ already exists.
[2023-07-06 14:02:06,227] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PXINTERACTIONAGGREGATOR with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.TopicExistsException: Topic ‘PXINTERACTIONAGGREGATOR’ already exists.
[2023-07-06 14:02:06,280] INFO [Admin Manager on Broker 1001]: Error processing create topic request for topic PYSENDEMAIL with arguments (numPartitions=6, replicationFactor=1, replicasAssignments={}, configs={retention.ms=216000000}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.TopicExistsException: Topic ‘PYSENDEMAIL’ already exists.

5

@Brahmesh@ This Error is Indicative of corrupted data in the Folder: Kafka-Data, which needs to e cleared. Even after clearing the Kafka-Data and restart the JVM’s, that requires additional debugging.

6

@MarijeSchillern we had raised the ticket INC-A17719 for it ..it got solved after executing the script sfor clean up policy that mentioned in the article .