Issue stating PE 851 due to kafka server issues

General
, , ,
1

I quite often encounter issues when starting my personal edition 8.5.1 due to kafka related issues as per tomcat console. Attaching the latest log from kafka folder for any suggestions on this behavior and its resolution.

During such occurrences as the instance doesn’t come up I connect to the database and try to clear the tables-pr_data_stream_nodes,pr_data_stream_sessions and the instance comes up. However any suggestions to avoid this behaviour would be helpful as this has been a recurring issue.

Any comments or suggestions are appreciated.

Thanks,

Satish

server-02-14-2022-1.log_.zip (97.6 KB)

2

@SatishKumarL5551 could you confirm your version of Kafka?

Your error extract

ERROR Failed to clean up log for __consumer_offsets-0 in dir C:\PRPC851\PRPCPersonalEdition\tomcat\kafka-data due to IOException (kafka.server.LogDirFailureChannel)

java.nio.file.FileSystemException: C:\PRPC851\PRPCPersonalEdition\tomcat\kafka-data__consumer_offsets-0\00000000000000000000.log.cleaned: The process cannot access the file because it is being used by another process.

at java.nio.file.Files.deleteIfExists(Files.java:1165) ~[?:1.8.0_121]
at kafka.log.Cleaner.deleteCleanedFileIfExists$1(LogCleaner.scala:488) ~[kafka_2.11-1.1.0.4.jar:?]

----------------------------

Kafka has known issues on windows environment.

Pega did addressed this issue by update the kafka version.

The kafka version kafka_2.11-1.1.0.5, which is default in Pega 8.5.4 have taken care of the issues.

There is a local change. However my recommendation will be to wait and apply if the issue is frequent.

Until 8.6 Pega has uses two compacted topics: __consumer_offsets and df_run_metrics. Covert these two topics to the not compacted topics:

$./<kafka-1.1.0.5>/bin/kafka-topics.sh --zookeeper localhost:2181 --topic __consumer_offsets --alter --config cleanup.policy=delete
$./<kafka-1.1.0.5>/bin/kafka-topics.sh --zookeeper localhost:2181 --topic df_run_metrics --alter --config cleanup.policy=delete

3

@MarijeSchillern Thanks for the prompt response. Based on kafka folder name(kafka-1.1.0.4) in the PE installation, its version as well is 1.1.0.4

Please advise accordingly.

4

@SatishKumarL5551 the issue caused by the Stream service failing because of an existing bug in Kafka ([KAFKA-1194] The kafka broker cannot delete the old log files after the configured time - ASF Jira) is a recognised problem.

The solution is to shutdown the Pega server and remove the Kafka data folder located at tomcat\kafka-data.

When you restart the server this directory will be recreated.

I will check internally to see if there are any plans to change the PE kafka provision.

5

@SatishKumarL5551 You stated that you were using PE 8.5.1 then it won’t contain the fixes as it’s just a download of the Pega software pushed out by release engineering at GA time. .

The latest version of Kafka shipped with the known Kafka-on-Windows fixes is Kafka-1.1.0.5, a version that is available in the following patch releases:

Platform version ‘Kafka-1.1.0.5’ availability
8.5.3+ yes
8.6+ yes

I am unable to comment when the PE version will be updated from 8.5.1 to 8.5.3

Please can you test by installing the latest version from here?

6

@SatishKumarL5551

The below workaround was something I found which may be used to stabilize the affected environment, but cannot be considered as a permanent fix (and I don’t have access to the jar files listed).

  • When the platform starts it unpacks Kafka to the kafka-1.1.0.X folder, and then spins the Kafka process using the libraries from that folder.
  • For the following procedure the kafka-1.1.0.X folder should already exist on the disk.
    • Move out from the kafka-1.1.0.X/lib folder two jars: kafka_2.11-1.1.0.3.jar and kafka-clients-1.1.0.3.jar
    • Copy into the kafka-1.1.0.X/lib folder two patched jars kafka_2.11-1.1.0.5.jar and kafka-clients-1.1.0.5.jar from the kafka-1.1.0.5 distribution.
  • Restart Stream service or the node.
7

@MarijeSchillern Thanks for the details. Let me check how I can replace current version and install Kafka-1.1.0.5 as current version came along with PE instance. Please do share in case of any instructions around it.

8

@MarijeSchillern I was able to get the suggested jars and follow the steps suggested above which worked like a charm.

Appreciate your prompt responses in resolving my issue.