Update on Operator ID is not reflected on other app nodes

Hi,

Customer reported to me that they changed operator ID’s password but the operator couldn’t log in to the system with a new password. Since this was happening sporadically, we conducted an investigation and figured that update on operator ID on one node is not reflected on other nodes. In this post, I am sharing the issue, and how to fix it. This issue is specific to Pega 8.6.x. It won’t happen in the prior version.

  • Steps to reproduce the issue
  1. Log in to one of the app nodes, and update an arbitrary operator ID’s information. In this example I have changed operator ID’s first name from “Kensho” to “Kensho2”, but attribute could be anything.

  1. Log in to a different app node than 1, and check the updated operator ID. You may find changes are not reflected. Even if you check it after hours, days, or rebooting the system with cache deletion, the result may be the same.

Note: This issue is not always replicatable, but I would say it’s a pretty high possibility. I’ve verified that this can occur in both Pega Cloud and On-Premise.

  • Root cause

Starting in Pega 8.6, operator pages started to be cached for improving performance, but System Pulse for ‘Data-Admin-Operator-ID’ was not whitelisted.

  • Resolution

The issue will be fixed in Pega 8.6.3, and you are advised to apply the patch release. If you are already facing this issue and need immediate action, raise an SR to obtain a hotfix (HFIX-81660) for 8.6.0.

Hope this helps.

Thanks,

@KenshoTsuchihashi

I’m experiencing this exact issue in our 8.6.0 environments.

I’ve searched everywhere until somehow I found this!

Thanks so much for posting this.

@KenshoTsuchihashi - We are seeing similar issues in 8.6.2 version. Can some one please provide the hot fix number for 8.6.2 version?

@SrikanthS9349 @leej3

This bug is already fixed in 8.6.3, and I would recommend that you apply patch release. If you still need to install Hofix, please see below for each version.

  • 8.6.0: HFIX-81660
  • 8.6.1: HFIX-81740
  • 8.6.2: HFIX-81741

Thanks,

@KenshoTsuchihashi what do you mean that system pulse was not whitelisted? System pulse is not some different url it is just an agent.

@topra

Commit is not required, but restart is required. I’ve also done a quick PoC on my local Pega 8.6.0 environment with Hotfix and verified that the issue still happens sporadically until I restart system. After all app nodes restart, the issue stopped occurring.

Thanks,

@KenshoTsuchihashi

Yes, definitely needs a restart.

We can now see the immediate changes on the operators.

Thanks for the prompt reply :slight_smile:

@KenshoTsuchihashi

Hi Kensho,

Thank you for posting this, it was very helpful.

At the moment, we are experiencing the exact problem you described above and we are planning to install the hotfix.

However, before installing it, we would like to know if there is anything we can do to synchronize the operator ID data momentarily. We tried restarting the app server in the past with positive results, but we are unsure whether that would work again.

@campr

No, there is no easy way to fix this issue as this is an engine level defect. We’ve observed that restarting Tomcat with cache deletion did not help. Please apply a Patch Release. Or if you need to take immediate action, install Hotfix.

Thanks,

@KenshoTsuchihashi thank you! this information helped..

@SurajK47

It is just an internal Pega’s term that means Operator update pulse message was ignored.

@KenshoTsuchihashi

It is not mentioned explicitly and the changes seems no need for restart but we couldn’t make these operators updated.

The fix is not committed.

Do we need a restart ?