Pega Constantly Reconnecting to External Kafka Topic decision_monitoring

Pega-as-a-Service
, ,
1

I have an external Kafka set up for my Pega servers. Whenever I point a Pega cluster to it, I get logs on my Kafka server of a consumer connecting and disconnecting to the decision_monitoring topic. The batch of logs below occurs every few minutes. Is this a misconfiguration of the external Kafka?

I noticed this is the only topic with a single partition instead of the default 6. Is this topic designed specifically to monitor the health of the Kafka server? And is this constant connecting and disconnecting how Pega does that monitoring?

[2024-02-09 13:40:41,069] INFO [GroupCoordinator 0]: Dynamic member with unknown member id joins group pega-decision_monitoring in Empty state. Created a new member id consumer-pega-decision_monitoring-1535-ecbe57da-3ec1-4459-93a2-bbdb7ef05f12 and request the member to rejoin with this id. (kafka.coordinator.group.GroupCoordinator)                                                                                                           
[2024-02-09 13:40:41,070] INFO [GroupCoordinator 0]: Preparing to rebalance group pega-decision_monitoring in state PreparingRebalance with old generation 9233 (__consumer_offsets-18) (reason: Adding new member consumer-pega-decision_monitoring-1535-ecbe57da-3ec1-4459-93a2-bbdb7ef05f12 with group instance id None; client reason: not provided) (kafka.coordinator.group.GroupCoordinator)
[2024-02-09 13:41:11,071] INFO [GroupCoordinator 0]: Stabilized group pega-decision_monitoring generation 9234 (__consumer_offsets-18) with 1 members (kafka.coordinator.group.GroupCoordinator)
[2024-02-09 13:41:11,072] INFO [GroupCoordinator 0]: Assignment received from leader consumer-pega-decision_monitoring-1535-ecbe57da-3ec1-4459-93a2-bbdb7ef05f12 for group pega-decision_monitoring for generation 9234. The group has 1 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator)
[2024-02-09 13:42:41,361] INFO [GroupCoordinator 0]: Preparing to rebalance group pega-decision_monitoring in state PreparingRebalance with old generation 9234 (__consumer_offsets-18) (reason: Removing member consumer-pega-decision_monitoring-1535-ecbe57da-3ec1-4459-93a2-bbdb7ef05f12 on LeaveGroup; client reason: not provided) (kafka.coordinator.group.GroupCoordinator)
[2024-02-09 13:42:41,361] INFO [GroupCoordinator 0]: Group pega-decision_monitoring with generation 9235 is now empty (__consumer_offsets-18) (kafka.coordinator.group.GroupCoordinator)
[2024-02-09 13:42:41,361] INFO [GroupCoordinator 0]: Member MemberMetadata(memberId=consumer-pega-decision_monitoring-1535-ecbe57da-3ec1-4459-93a2-bbdb7ef05f12, groupInstanceId=None, clientId=consumer-pega-decision_monitoring-1535, clientHost=/10.0.1.25, sessionTimeoutMs=30000, rebalanceTimeoutMs=300000, supportedProtocols=List(range, cooperative-sticky)) has left group pega-decision_monitoring through explicit `LeaveGroup`; client reason: not provided (kafka.coordinator.group.GroupCoordinator)
2

@RyanD662 Can you check once if the stream nodes are working properly, and getting normal status.

and what is your pega version, if its 8.4 or below pega provides internal kafka make sure its not conflicting and it has a hard dependecy with the stream node configuration and startup for which pega moved to externalized kafka in later versions.

Streamhealth.PNG
Streamhealth.PNG1857×794 75.1 KB

3

@Rinsler1107 My stream nodes are reporting as NORMAL. This is also specifically happening with External Kafka in Infinity 23. We’re not running any internal Stream nodes. Good idea to check if they were conflicting though.

4

@RyanD662 can you confirm that you have now logged support ticket INC-B6395 (Pega Constantly Reconnecting to External Kafka) for this?

I believe our support team are now waiting for you to provide the pr config you are using to configure the external kafka.

5

@RyanD662 I checked INC-B6395 and can see the issue was resolved.

The posted solution in the notes:

Issue primary reason description:

the log messages with our engineering team and may confirm that they are normal and not any indication of a problem.

Decision_monitoring consumer group is used for ADM models input/output monitoring in Predictions component. The consumers (clients) which Platform spawns to consume using the group are anonymous, so they don’t have a ID assigned to each consumer client, this is as the current product design and doesn’t represent any issues

Explanation description:

If you wish to possibly reduce these messages related to Decision_monitoring consumer group, you may confirm if Predictions/ADM Models aren’t being used for this instance and disable the monitoring feature from Prediction Studio > Settings > Monitor & notification settings > disable “Monitor model input and output data”.