How can I parameterize the Kafka topic name in dataset?
We have a situation where our Kafka cluster is shared across QA and UAT environments. To prevent QA data from landing in UAT (and vice versa), planning to append the environment name as a suffix to each Kafka topic name. Using the same topic name across environments would result in data corruption.
For example, our Kafka topic names follow this pattern:
@Sudheer Nimmagadda To parameterize the Kafka topic name in your dataset and avoid hardcoding environment-specific values, you can use environment variables or configuration parameters within your data pipeline or ingestion framework (e.g., Spark, NiFi, or a custom app). Define a parameter like env.name=QA1 or env.name=UAT2 in your configuration file or environment setup. Then, construct the topic name dynamically by using string interpolation, such as Insurance-Health-Medical-${env.name}. This way, the dataset will always send or receive messages to the correct environment-specific Kafka topic. This approach prevents cross-environment data mixing and makes deployments cleaner across QA and UAT environments.
Please let me know if you need any more info. Thanks
