KAFKA data via AVRO Schema

General
, , ,
1

Pega need to pass data to KAFKA via AVRO schema format.

the connectivity in Kafka connection and Data set is successfully.

however, In runtime, Data set it throws below exceptions.

  1. javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alert.createSSLException

  2. Failed to save page to data set due an exception][STACK][com.pega.dsm.kafka.api.serde.avro.AvroSerdeException: org.apache.kafka.common.errors.SerializationException: Error serializing Avro message at com.pega.dsm.kafka.api.serde.avro.AvroValuesSchemaRegistrySerde.serialize(AvroValuesSchemaRegistrySerde.java:103) at com.pega.dsm.kafka.api.serde.LazyExceptionHandlingSerDe.serialize(LazyExceptionHandlingSerDe.java:66)

2

Hi @KasunJ,

I get the same error “unable to find valid certification path to requested target”. Could you please let me know where you set the certificate for schema registry server please?

3

@JoelB270 The Certificate needs to upload to app server truststore JKS file

4

@KasunJ

After setting the certificate, we are seeing new error message.

[MSG][Conversion of ClipboardPage to GenericRecord has failed.][STACK][org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: XXX Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Incompatible Avro schema; error code: 40901 at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:209) at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:235)

5

@KasunJ Hi Kasun, are you using a Pega Cloud or On-Premise PEGA instance ?

6

@KasunJ any updates to my question ?

7

@KasunJ

Can you please elaborate how to upload certificate for truststore jks file..?

we have cert file and truststore jks file. But not able to figure out, how to upload certificate to app server truststore jks file.

8

@KasunJ Validate your AVRO schema and upload a right format. Also, check if your organization allow updates to Schema registry once published. or you may need to create a new Event Type and Upload a new version of Schema.

9

@ArnaudW On Permise pega

10

We configured the AVRO schema in Pega to read a Kafka message from the kafa topic. We added the JKS file and encountered no certificate issues. However, Pega shows the following error when consuming the Kafka topic.
Error: Error deserializing stream while reading from Kafka topic.

@KasunJ

11

This is likely related to the AVRO payload rather than connectivity. Please verify:

  • The Kafka producer is publishing messages using the expected AVRO format.
  • The schema registered in Schema Registry matches the schema configured in Pega.
  • The consumer and producer are using compatible schema versions.
  • The topic does not contain non AVRO messages or older messages serialized with a different schema.
12

Thanks for quick response.