We are processing a CSV file using a Data Set (CSV file type) and observed that our Data Flow fails on a particular record. Upon investigation, we found that one of the cells (Column G) contains a multi-line value.
When previewing the file in the Data Set rule, the second line of the multi line value is being treated as a new row, causing subsequent values to shift into incorrect columns and resulting in processing errors.
Previously, we used a File Listener to read the same file and persist the data to the database. In that implementation, the entire multi-line content in Column G was correctly treated as a single value and saved successfully without errors.
Could you please help us understand:
Whether CSV Data Sets/Data Flows support multi-line values within a field.
If any configuration is available to correctly parse such records.
Recommended alternative approaches or best practices for handling files containing multi-line field values when using Data Sets.
This is a known limitation of the File Data Set CSV parser rather than a configuration mistake on your side.
This is mentioned explicitly in the documentation: “Field values cannot span multiple lines; newline characters within column values are not supported.” and is consistent across the versions, see 25 for example.
Practically speaking I can see only two alternative options:
Stay with the File Listener, as you confirmed it worked before. What is the reason for change in the first place?
Pre-process the file to remove the in-field line breaks before the Data Set reads it. Maybe even outside pega, closer to the source system.
Thanks for the clarification, suggestion and for pointing to the documentation.
The reason we’re exploring alternatives to the File Listener is related to our current Azure-based file storage setup.
In our implementation, files are placed in an Azure storage location and are read from there. While processing larger files (typically around 10 MB or more), we are intermittently encountering timeout exceptions when reading the file from Azure. Based on this challenge, some of our senior team members suggested evaluating the File Data Set approach as a potential alternative.
However, we’ve now identified this limitation regarding multiline field values in CSV files, which is impacting our use case.
Do you happen to have any suggestions or recommendations for handling large CSV files from Azure storage while avoiding these timeout issues?
Any guidance or experience you can share would be greatly appreciated.
Which deployment are you on, and how is Azure being read? Are you on Pega Cloud (Pega-as-a-Service), and is the file being accessed via a Repository (Azure Blob) or a mounted/SFTP path? I ask because 10 MB really isn’t a large file — a timeout at that size usually points to the connection/read path rather than the volume of data.
File Listener — line-by-line reading. The File Listener can be configured to read/process the file line by line rather than loading it entirely, which generally improves throughput and memory usage and may relieve the timeout you’re seeing. One caveat: reading line by line can reintroduce the same multiline-cell splitting, but worth trying.
Generally I can see two options for solving this:
Preprocess at the source (my preferred option). Replace the in-field newline characters before the file lands in Azure (or as it’s written out). That removes the parsing problem entirely and leaves you free to use whichever reader performs best — Data Set or Listener.
Chunk the files into smaller pieces. Split the source into files comfortably under 10 MB — whatever threshold is practical for your data — so each read stays well inside the timeout window.
We also faced the same issue. We had a discussion with our client and came up with a solution. Ask your client to generate files with encoded data. And then when you process the file data, you decode the data and then do what you are doing. When you encode and decode the data, multi lines will be removed.
