Warning messages in pega log while Posting messages to external Kafka topic

General
, , , ,
1

Hi All,

We are getting lot of warning messages in pega logs when we write messages to external Kafka topic(avro message) . we analysed and found its due to field’s definition in avro schema as it has data type as “null and string”(sample given below). but Kafka team clarified, its the standard to define non-mandatory fields. but in pega we are getting warning messages for those non mandatory fields and its flooding pega log with warnings.

to avoid writing these warning messages from pega logs we have created the log category with “Error” for below class com.pega.dsm.kafka.api.serde.avro.AvroSerdeException to write error and above on the pega logs for this class but it didnt help. we are still seeing the messages in pega logs.

Has anyone faced this issue. please help

Pega Version : 8.7.0

sample definition for non mandatory field

{
“name”: “Version”,
“type”: [
“null”,
{
“type”: “string”,
“avro.java.string”: “String”
}

sample definition for mandatory field

“name”: “toDate”,
“type”: {
“type”: “string”,
“avro.java.string”: “String”
},

Warning message in Pega logs

Property couldn’t be matched with the potential schema.
com.pega.dsm.kafka.api.serde.avro.AvroSerdeException: Unsupported data type! NULL
at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.resolveObjectByFieldType(ClipboardPageToGenericRecordConverter.java:97) ~[d-node.jar:?]
at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.handleUnionType(ClipboardPageToGenericRecordConverter.java:108) ~[d-node.jar:?]
at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.resolveObjectByFieldType(ClipboardPageToGenericRecordConverter.java:70) ~[d-node.jar:?]
at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.convertClipboardPageToGenericRecord(ClipboardPageToGenericRecordConverter.java:51) ~[d-node.jar:?]
at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.resolveObjectByFieldType(ClipboardPageToGenericRecordConverter.java:67) ~[d-node.jar:?]
at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.convertClipboardPageToGenericRecord(ClipboardPageToGenericRecordConverter.java:51) ~[d-node.jar:?]
at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.resolveObjectByFieldType(ClipboardPageToGenericRecordConverter.java:67) ~[d-node.jar:?]
at com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter.convertClipboardPageToGenericRecord(ClipboardPageToGenericRecordConverter.java:51) ~[d-node.jar:?]
at com.pega.dsm.kafka.api.serde.avro.AvroValuesSchemaRegistrySerde.serialize(AvroValuesSchemaRegistrySerde.java:99) ~[d-node.jar:?]
at com.pega.dsm.kafka.api.serde.LazyExceptionHandlingSerDe.serialize(LazyExceptionHandlingSerDe.java:66) ~[d-node.jar:?]

2

@Prabhu Subramani any update on this issue. Since i am also facing the same.

3

@Prabhu Subramani I can see that a support incident was logged for this and our support team are looking into this. So far the analysis done is that the supported schema types are “RECORD, UNION, ARRAY, MAP , LONG, ENUM , STRING , INT , FLOAT , DOUBLE, BOOLEAN”, for anything else the unsupported warning along with AvroSerdeException is logged. Our support team will need to discuss with the Product team regarding the reason for not supporting NULL type.

@AshrafK4 I suggest you log an incident ticket via the MSP in order for your issue to be investigated seperately.

When you log the support ticket:

  1. Can you try setting the log level to Error for the below class and check if the warnings are suppressed?

com.pega.dsm.kafka.api.serde.avro.ClipboardPageToGenericRecordConverter

  1. In your support ticket you’ll need to share the below information,
  • AVRO schema document
  • Apart from the log files getting filled up with messages, do you have any other business impact ?
4

@MarijeSchillern

since logs are getting full we are seeing no space left on device. due to which kafka instance which is running on stream node is down and as well as web instance is not working properly. In all our system is going down frequently. i have to clear FS and restart trice a day, which is effecting out BAT.

5

@AshrafK4 please log a support ticket, as already suggested.