Progressive Character Corruption ("Mojibake") After UI Selections in Pega Cloud: Root Cause and Remediation

General
, ,
1

Introduction

Teams migrating applications from on-premises Pega deployments to Pega Cloud sometimes encounter a subtle but frustrating defect: accented or special characters (e.g., “Salário”, “INCLUSÃO”) render correctly on the initial page load, but progressively become corrupted, showing up as “Salário”, “INCLUS�O”, or “INCLUS�?O”, after specific user interactions such as AutoComplete selections, grid add/remove row actions, or automatic section refreshes.

This article documents a real-world investigation of this issue on a Pega Constellation application running on Pega Cloud, the root cause identified through Tracer analysis, the layered remediation strategy implemented, and recommendations for preventing recurrence. The goal is to give other teams a reference document so they don’t have to rediscover this from scratch.

1. Problem Description

During the use of a file layout configuration feature (mapping uploaded file columns to application fields), users observed inconsistent rendering of accented characters after specific actions, including:

  • Selecting items from AutoComplete/combobox controls;
  • Adding new rows to a grid (`addRow`);
  • Automatic UI refresh of a section or table.

Example:

  • Expected: `DATA INCLUSÃO`
  • Observed: `DATA INCLUS�O` / `DATA INCLUS�?O`

The corruption was progressive and non-deterministic: some scenarios rendered correctly on initial load but broke after subsequent interactions, while others broke immediately. This behavior strongly suggested the root cause was not in the data source (CSV file or backend), but somewhere in the UI refresh and clipboard/PageList reconstruction cycle.

2. Technical Analysis

Instrumentation with the Pega Tracer tool showed that:

  • Data was loaded correctly as UTF-8 at the source.
  • The corruption occurred during refresh cycles (`pzTransformAndRun`), commonly triggered by:
    • Section reloads;
    • Grid actions (`addRow`, `deleteRow`, filter, paginate);
    • Reconstruction of `pxResults` PageLists through Pega’s out-of-the-box (OOTB) UI pipeline.

Specifically, PageLists bound to the affected grids were being reconstructed from payloads that had already been corrupted, and even after applying corrective Data Transforms, the platform re-applied broken values coming from:

  • Parameter Pages;
  • Snapshots of the grid’s previous state.

In short: the defect was not isolated to a single point, but stemmed from the absence of a consistent normalization step throughout the entire refresh cycle.

Why the issue did not manifest on-premises

In the on-premises environment, the same functionality did not exhibit the problem, due to architectural characteristics rather than the absence of the underlying defect:

  • Use of session affinity (sticky sessions);
  • The clipboard remaining in memory within the same JVM across interactions;
  • Fewer serialization/deserialization cycles overall.

Why the issue manifests on Pega Cloud

Pega Cloud uses a stateless architecture:

  • Each UI interaction is treated as an independent request;
  • There is no guarantee that the same node or in-memory clipboard will be reused;
  • The clipboard state is serialized and rehydrated on every action.

As a result, string values containing special characters pass through multiple encoding interpretation cycles and can be:

  • Reinterpreted with the wrong charset (UTF-8 read as ISO-8859-1 or CP1252);
  • Partially corrupted (characters replaced with `�` or `?`);
  • Re-propagated into the model, producing a cumulative “mojibake” effect.

This is consistent with Pega’s own documentation on Pega Cloud’s stateless request handling and JSON Data Transform serialization/deserialization behavior, where Dynamic System Settings control how clipboard pages, including PageLists, are generated when state is rehydrated between requests.

3. Remediation Strategy

Given a legacy application context where a full architectural refactor (e.g., separating the internal value from the display value on combobox controls) was not feasible in the short term, the team adopted an incremental, low-risk, three-pillar strategy:

3.1 A centralized encoding-correction utility function

A dedicated utility function (referred to here generically as `FixMojibakeUTF8`) was created to:

  • Detect typical mojibake patterns (e.g., `Ã`, `Â`, `�` sequences);
  • Re-interpret the string correctly as UTF-8;
  • Strip invisible characters (NBSP, zero-width spaces);
  • Guarantee idempotency, so it can be safely applied multiple times without side effects.

This function became the single centralized point of encoding correction, replacing scattered, ad hoc fixes.

Simplified excerpt of the pattern used:

if (s.indexOf('\\uFFFD') >= 0) {
    s = s.replace("INCLUS\\uFFFDO", "INCLUSÃO")
         .replace("INCLUS\\uFFFD?O", "INCLUSÃO")
         .replace("\\uFFFDO", "ÃO")
         .replace("\\uFFFDA", "Á")
         .replace("\\uFFFDE", "É");
         // additional known patterns...
}

3.2 Systematic normalization inside Data Transforms

The correction function was applied consistently within the functional Data Transforms responsible for:

  • Applying the selected file layout;
  • Applying field mappings;
  • Validating duplicate entries;
  • Updating column status.

Normalization was applied at three critical points:

  1. On parameter input (Param.*);
  2. During intermediate capture inside loops (For Each Page);
  3. On final write to the PageLists consumed by the UI.

This eliminated corruption for individual selection scenarios.

3.3 Correcting the add/remove list flow via execution order

The most complex scenario involved adding or removing items from a list: even with already-sanitized PageLists, the platform’s OOTB grid pipeline re-applied corrupted values during the refresh that followed the action.

Tracer analysis showed that the fix here was not additional code, but control of execution order. The remediation was to reorganize the Action Sets on the Section and on the Add/Remove buttons so that:

  • The sanitization Data Transforms always executed before the list was rendered;
  • No grid refresh occurred after cleanup without the fix being reapplied.

With this reordering, the UI always displayed the already-sanitized state, even when Pega internally reconstructed the PageLists behind the scenes.

4. Results

After implementing the three pillars above:

  • :white_check_mark: Accented characters rendered correctly in 100% of tested scenarios: initial load, layout selection, field selection, row addition/removal, and automatic section refresh;
  • :white_check_mark: No functional regression was introduced;
  • :white_check_mark: The solution was compatible with the existing legacy design, without requiring customization of critical OOTB rules;
  • :white_check_mark: A clear, reusable pattern (FixMojibakeUTF8) is now available for future similar cases.

5. Known Limitations of This Fix

This remediation is effective but should be understood as a mitigation, not a structural elimination of the root cause:

  • If a character has already been replaced with ?, the original information is irreversibly lost, the function can only repair recoverable corruption patterns (e.g., �), not data that was already truncated;
  • The function relies on known corruption patterns; new patterns may require extending the replacement map;
  • Behavior may still vary depending on the type of user interaction, the sequence of actions (add/remove/refresh), and the origin of the data.

6. Recommended Long-Term Solution

The durable fix is not correcting corrupted data after the fact, but standardizing data encoding at the point of entry:

  1. Standardize input files: provide users with CSV/Excel templates and ensure all files are generated in UTF-8 without ambiguity.
  2. Guide end users: avoid manual file generation in tools that default to “ANSI” or locale-dependent encodings; prioritize tools/settings that allow explicitly selecting UTF-8 (for example, using Excel’s Data > Get Data > From Text/CSV import wizard with File Origin set to “65001: Unicode (UTF-8)”, rather than double-clicking a CSV to open it).
  3. Add data governance controls: validate file encoding at upload time, and reject or normalize files that fall outside the expected standard.

7. Complementary Official Pega Guidance

While there is no single official Pega article covering this exact end-to-end scenario, the following official resources are directly relevant and worth reviewing alongside this article:

  • Pega Platform and the PegaRULES database support UTF-8 and UTF-16; the encoding must be chosen at installation time, as a UTF-8 system cannot later be switched to UTF-16, see “Internationalization and localization” in the Pega Platform documentation.
  • Dynamic System Settings control how Clipboard pages, including PageLists, are generated during JSON Data Transform serialization/deserialization, see the Pega Platform “Enhancements and resolved issues” release notes.
  • Pega Cloud requires UTF-8 at the system locale level, and PostgreSQL-related migrations should confirm UTF-8 compatibility for any custom tables, see “Remediations for Pega Cloud Compliance.”
  • For CSV import/export, Pega documentation recommends never opening exported CSVs by double-clicking in Excel, and instead using Excel’s Text/CSV import wizard with the file origin explicitly set to UTF-8, see “Working with CSV Files in Pega Platform” (Pega Support).
  • In some Pega Cloud 3 migrations, locale/encoding-related formatting issues have required the JVM parameter -Djava.locale.providers=COMPAT, applied via a Cloud Change request, see “Currency formatting issues after updating Pega Cloud 2 to 3” (Pega Support). While originally documented for currency formatting, the same JVM locale behavior can be a contributing factor worth ruling out in broader character-rendering investigations.

Conclusion

This was not a one-off bug, but a systemic issue rooted in how Pega Cloud’s stateless architecture serializes and rehydrates clipboard state across UI interactions. On-premises deployments simply masked the defect through session affinity and in-memory clipboard persistence, they didn’t avoid it.

The fix implemented here (a centralized correction utility, systematic normalization inside Data Transforms, and controlled Action Set execution order) fully resolved the symptoms without requiring a broader refactor, but the most robust long-term prevention remains standardizing all data ingestion on UTF-8 from the source.

Credits: Root cause analysis and solution design by Fernando Koarata.

Disclaimer: Client-specific ruleset names, environment identifiers, and application-specific naming have been generalized in this article for public sharing. The technical mechanics described are unchanged from the original investigation.

2

@koarf
The Credits are for you! You did the Root cause analysis and solution design.