We have notice randomly throw time out exception in KAFKA connection while pushing data to KAFKA. There is no specific pattern or scenarios, however there are lots of time out exceptions in production log .
org.apache.kafka.common.errors.TimeoutException: Topic XXXXXXX not present in metadata after 10000 ms.
@KasunJ check the authentication that your KAFKA server is expecting from Pega and also see the reliability of the KAFKA server from the KAFKA control centre.
@RaghuramJ16633460 Is there way to check from Pega side that the message pass correctly along with Kerberos authentication?
we push around 12K records to KAFKA topic however around 2K records are missing from KAFKA topic daily due to timeout exception.
@KasunJ hi whats the latest on this
@KasunJ following property settings updated and uploaded to kafka client properties.
retries=20
retry.backoff.ms=200
we don’t see time out exceptions now.
@KasunJ you have updated this in client property file?
@KasunJ I think the time out was internal configuration of KAFKA server, Talk to your KAFKA Admin , what was the availability of KAFKA server and at wha frequency is it setup to rebound/ refresh.
Analyse the failed cases from the logs and see if there is any pattern on the data for the failure ones.