Hello Everyone,
We recently did an in-place upgrade from 8.2.1 to 8.6.2 and we see that the Queue Processors are not working. All the items are there in “Scheduled” status but not getting picked up by the processor.
As per below article, we have to enable one of the job scheduler pzDelayedQueueProcessorSchedule. We did that still it is not working. Only pega OOTB queue processors are working after this scheduler was enabled.
Side Question - If job scheduler pzDelayedQueueProcessorSchedule is responsible for working on delayed queue processor, why is it disabled by default?
@AroraPiyush did you also follow-up on the additional suggestions in the post you referenced?
ie
"Make sure Stream node is up and in normal state , for queue processor to pick the messages from topic .
To start the stream if other than normal state , you can go to actions and select Start as an action as attached here.
Else you can try for restarting the server notifying your codevelopers about restart."
Can you tell us if the Queue processor rules are running? If they are not then queues like FTSIncrementalIndexer will be sitting in a scheduled state without moving.
Your logs should identify the reason for no Queue processors running. One example could be if the Stream service is down - errors for this would be:
In the Decisioning Services > Stream tab:‘RuntimeException: Unable to unpack Apache Kafka distribution’
In the Decisioning Services > Decision Data Store tab: ‘RuntimeExeption: Unable to unpack Apache Cassandra distribution’
Do you see any errors in the logs?
We may need to ask you to log this as a Support ticket for further troubleshooting.
@MarijeSchillern
We have 2 nodes (1 stream and 1 background) and both of them are in running status. Interestingly, FTSIncrementalIndexer is working.
@AroraPiyush I suggest that you log a support ticket and there please share details of the non-OOTB schedulers experiencing the issue. Pease include screenshots of the Admin studio QP depicting scheduled, broken.. information . Please also have the logs to hand and details of the upgrade logs showing successful update status.
@MarijeSchillern Thanks. I have created INC-209829 for this issue.
@AroraPiyush Do you have an outcome of the ticket and a solution for the issue? Seems like we’re experiencing similar behaviour.
@AroraPiyush @NielsHeinis
I have checked the ticket and I believe the issue was resolved by updating the BATCH requestor type to use the default AccessGroup to PRPC:Agents
Root Cause:
The DelayedItemsDataflowService couldn’t start due to an access issue.
The DF code(RunDbPersistence) could not invoke createWorkPage OOTB activity and failed with an Access issue - PRPC:AsyncProcessor was missing permissions.
The access/permission was due to the change of BATCH requestor type at client side to PRPC:AsyncProcessor AG in place of PRPC:Agents.
(You can replicate the exact issue by changing systemName!BATCH requestor type access group from PRPC:Agents to PRPC:Async and delete the runID for delayedDF and restart the JVM ).
Both OOTB AccessGroups need privilege " pzSystemOperationsAdministrator".
Solution:
Use OOTB PRPC:Agents Access Group in place of PRPC:AsyncProcessor. The latter is not valid for BATCH requestor type. I believe the documentation will reflect this requirement in the future.