ERROR - Unable to calculate Kafka lag for icmp.documentnotify.communitybank, environment statenull org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 5000ms
You might be experiencing a timeout issue due to network latency or Kafka being overloaded.
You can try adjusting the Kafka client properties. Specifically, you can increase the number of retries and the time between retries. For example, you can set retries=20 and retry.backoff.ms=200. This will allow the client to retry the operation in case of a timeout, potentially avoiding the error.
Also check stream nodes in the cluster are in the same region as other nodes in the cluster.
Please log a support issue via the MSP and provide the INC id here so that we can help track it.