Up to Pega 24.2, for PoC purposes, I was able to create a single Docker container (NODE_TYPE=WebUser,BackgroundProcessing,Search) Pega node while externalizing Kafka and SRS. I used the same “recipe” for Pega 25, the node came up successfully. I am able to log in, navigate Dev / App studios etc. However, there’s one issue.
When I used Admin Studio to list the nodes, I got ZERO nodes running. Admin studio did not list the node that I have logged into.
2025-12-11 04:54:35,011 [ AsyncServices-5] [ STANDARD] [ shared] [ Verify:01.01.01] (rationsDistributedExecutorImpl) ERROR author@verify - Caught exception while getting result from future object java.util.concurrent.ExecutionException: com.pega.platform.remoteexecution.management.UnresponsiveNodeException: Remote execution request 9fe00bbd-02bb-4533-b6dc-e429dafea629 was not acknowledged by node 6100dd74-1fdf-4caa-814b-fa88cb372cbe within the timeout at com.pega.platform.remoteexecution.ResponseMessage.getResponse(ResponseMessage.java:60) at com.pega.platform.remoteexecution.internal.ResponseFuture.get(ResponseFuture.java:84) at com.pega.platform.remoteexecution.internal.ResponseFuture.get(ResponseFuture.java:23) at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:75) at com.pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.createResultFromOneFutureObject(SystemOperationsDistributedExecutorImpl.java:249) at com.pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.collectDoneResults(SystemOperationsDistributedExecutorImpl.java:237) at com.pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.getResultsFromFutureObjects(SystemOperationsDistributedExecutorImpl.java:197) at java.base/java.util.Optional.map(Optional.java:260) at com.pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.executeOnAllNodesInCluster(SystemOperationsDistributedExecutorImpl.java:102) at com.pega.pegarules.management.internal.system.operations.AbstractManagementAPI.executeOnAllNodes(AbstractManagementAPI.java:195) at com.pega.pegarules.management.internal.system.operations.ClusterManagementAPI.getNodesInCluster(ClusterManagementAPI.java:714) at com.pegarules.generated.activity.ra_action_pzgetnodesincluster_fea19cc7f900010c93054d4b0f950a2e.step1_circum0(ra_action_pzgetnodesincluster_fea19cc7f900010c93054d4b0f950a2e.java:165) at com.pegarules.generated.activity.ra_action_pzgetnodesincluster_fea19cc7f900010c93054d4b0f950a2e.perform(ra_action_pzgetnodesincluster_fea19cc7f900010c93054d4b0f950a2e.java:76)
Upon more investigation, I have isolated the line of Java code that gives the exception is as follows:
It’s my understanding that Pega 25 no longer makes use of Hazelcast. My question is, what additional environment variables are required to pass in a Pega 25+ container in order to resolve the problem that I outlined above?
@TerenceY0215 In Pega 25, Admin Studio discovers nodes through internal cluster messaging, and if that messaging is not running you can log in fine but Admin Studio will still show 0 nodes and you’ll see remote execution timeouts like the one in your log. The most common miss in a single-container setup is not running a Stream node type. Update your container to include Stream (or just use Universal), for example NODE_TYPE=WebUser,BackgroundProcessing,Stream,Search, then restart the container. The Stream service is Kafka-based and Pega expects at least one Stream node for it to function, even when you point it to an external Kafka broker. After restart, confirm the Stream landing page shows NORMAL for external Kafka and then re-check Admin Studio nodes. If it still shows 0, double-check your external Kafka connectivity and that the container hostname/IP is stable and resolvable (so node acknowledgements don’t get lost).
Currently, the NODE_TYPE is WebUser,BackgroundProcessing,Stream. The container is configured to use External Kafka and SRS. I also have Kafka and SRS in separate containers. And the containers communicate through docker networking.
The Stream landing page shows NORMAL for External Kafka. I have the same set up with Pega 24.2, and it was fine.
With Pega 25, I see exceptions like the following which I didn’t see in Pega 24.2:
2025-12-13 17:32:46,816 [ pool-29-thread-1] [ ] [ ] [ ] (ssor.StreamPartitionsInfoCache) ERROR - Exception occurred in updateTopicsInCache
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at com.pega.fnx.stream.spi.impl.processor.StreamPartitionsInfoCache.updateTopicsInCache(StreamPartitionsInfoCache.java:221)
at com.pega.fnx.stream.spi.impl.processor.StreamPartitionsInfoCache.updateAndGetPartitionoffsetsFromCache(StreamPartitionsInfoCache.java:190)
at com.pega.fnx.stream.spi.impl.processor.StreamPartitionsInfoCache.getStreamPartitionOffsets(StreamPartitionsInfoCache.java:178)
at com.pega.fnx.stream.spi.impl.processor.GetStreamPartitionsRequestProcessor.getDetails(GetStreamPartitionsRequestProcessor.java:60)
at com.pega.fnx.stream.spi.impl.processor.GetStreamPartitionsRequestProcessor.lambda$execute$0(GetStreamPartitionsRequestProcessor.java:51)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
I wonder with Pega 25, with the removal of Hazelcast, Pega is leaning more into the use of Kafka for cluster communication.
The exceptions that I noted earlier go away after I stop and restart the Pega node. With that, I presume they show up when the topics have yet to be created in Kafka. Once they have been created, those exceptions no longer show up.
However, the same errors continue to occur. And it complains that no nodes are provided for Background processing.
2025-12-14 14:02:15,718 [e-registry-heartbeat] [ STANDARD] [ ] [ ] (.tasks.TopologyChangedDetector) INFO - Services topology has changed. Current topology:
This node ==> DataFlow.BackgroundProcessing pega-web-tomcat-0 172.18.0.9 NORMAL 25.1.1 0ebafb79-b9a8-49c0-bb98-3fdee5357f30
This node ==> DataFlow.Search pega-web-tomcat-0 172.18.0.9 NORMAL 25.1.1 a9df680d-0ff5-4259-888f-8bbd99e2873a
This node ==> pega pega-web-tomcat-0 172.18.0.9 running (was startingUp) 25.1.1 ac2ad558-b731-458a-b913-9c8ffff4c7ae
2025-12-14 14:02:15,887 [g-topology-handler-0] [ STANDARD] [ ] [ ] (ckgroundprocess.TierObservable) INFO - The version: 25.1 is consistent: true
2025-12-14 14:02:27,472 [stream-ops-trigger:9] [ STANDARD] [ ] [ ] ( api.server.StreamOpsManager) INFO - This node is selected as stream operations nodes.
2025-12-14 14:03:17,341 [OBSCHEDULER_THREAD_9] [ STANDARD] [ ] [ ] (rationsDistributedExecutorImpl) ERROR B5QP0JF6T3GW89I53AI15BDF9FZ9LWHQ7A - Caught exception while getting result from future object
java.util.concurrent.ExecutionException: com.pega.platform.remoteexecution.management.UnresponsiveNodeException: Remote execution request c32ee7c2-3b01-4c5e-bbdb-2208621d317d was not acknowledged by node ac2ad558-b731-458a-b913-9c8ffff4c7ae within the timeout
at com.pega.platform.remoteexecution.ResponseMessage.getResponse(ResponseMessage.java:60)
at com.pega.platform.remoteexecution.internal.ResponseFuture.get(ResponseFuture.java:84)
at com.pega.platform.remoteexecution.internal.ResponseFuture.get(ResponseFuture.java:23)
at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:75)
at com.pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.createResultFromOneFutureObject(SystemOperationsDistributedExecutorImpl.java:249)
at com.pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.collectDoneResults(SystemOperationsDistributedExecutorImpl.java:237)
at com.pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.getResultsFromFutureObjects(SystemOperationsDistributedExecutorImpl.java:197)
at java.base/java.util.Optional.map(Optional.java:260)
at com.pega.pegarules.management.internal.system.operations.SystemOperationsDistributedExecutorImpl.executeOnAllNodesInCluster(SystemOperationsDistributedExecutorImpl.java:102)
at com.pega.pegarules.management.internal.system.operations.AbstractManagementAPI.executeOnAllNodes(AbstractManagementAPI.java:195)
at com.pega.pegarules.management.internal.system.operations.ClusterManagementAPI.getNodesInCluster(ClusterManagementAPI.java:714)
at com.pega.pegarules.cluster.internal.NodeClassificationImpl.getNodesTypes(NodeClassificationImpl.java:83)
at com.pega.pegarules.cluster.internal.NodeClassificationImpl.getNodesOfType(NodeClassificationImpl.java:55)
at com.pega.pegarules.management.internal.system.operations.MemoryUsageMetricMonitor.getNodesOfType(MemoryUsageMetricMonitor.java:99)
at com.pega.pegarules.management.internal.system.operations.MemoryUsageMetricMonitor.getNodesOfType(MemoryUsageMetricMonitor.java:89)
at com.pega.pegarules.management.internal.system.operations.MemoryUsageMetricMonitor.isMetricHealthy(MemoryUsageMetricMonitor.java:49)
at com.pega.pegarules.management.internal.system.operations.QueueProcessorManagementAPI.isMetricHealthy(QueueProcessorManagementAPI.java:276)
at com.pegarules.generated.activity.ra_action_pyautotuneqpthreads_191ca6b47c8d715fe84a9445035c583a.step1_circum0(ra_action_pyautotuneqpthreads_191ca6b47c8d715fe84a9445035c583a.java:175)
at com.pegarules.generated.activity.ra_action_pyautotuneqpthreads_191ca6b47c8d715fe84a9445035c583a.perform(ra_action_pyautotuneqpthreads_191ca6b47c8d715fe84a9445035c583a.java:76)
at com.pega.pegarules.session.internal.mgmt.Executable.doActivity(Executable.java:2940)
at com.pega.platform.executor.jobscheduler.internal.ActivityExecutor.runActivity(ActivityExecutor.java:61)
at com.pega.platform.executor.jobscheduler.internal.ActivityExecutor.executeActivity(ActivityExecutor.java:53)
at com.pega.platform.executor.jobscheduler.internal.ActivityProcessor.executeActivity(ActivityProcessor.java:73)
at com.pega.platform.executor.jobscheduler.internal.ActivityProcessor.execute(ActivityProcessor.java:59)
at com.pega.platform.executor.jobscheduler.internal.ActivityProcessor.run(ActivityProcessor.java:110)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1379)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1122)
at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1003)
at com.pega.pegarules.session.internal.PRSessionProviderImplForModules.doWithRequestorLocked(PRSessionProviderImplForModules.java:83)
at com.pega.platform.executor.jobscheduler.internal.ActivityProcessor.run(ActivityProcessor.java:101)
at com.pega.platform.executor.jobscheduler.internal.JobSchedulerProcessor.execute(JobSchedulerProcessor.java:151)
at com.pega.platform.executor.jobscheduler.internal.JobSchedulerProcessor.lambda$execute$1(JobSchedulerProcessor.java:133)
at com.pega.platform.executor.internal.LogContextDecorator.runInDecoratedScope(LogContextDecorator.java:38)
at com.pega.platform.executor.jobscheduler.internal.JobSchedulerProcessor.execute(JobSchedulerProcessor.java:131)
at com.pega.platform.executor.jobscheduler.scheduler.internal.JobRunTimeImpl.execute(JobRunTimeImpl.java:104)
at com.pega.platform.executor.jobscheduler.scheduler.internal.JobRunTimeDecorator.execute(JobRunTimeDecorator.java:57)
at com.pega.platform.executor.jobscheduler.scheduler.internal.persistence.PersistentEligibleJobExecution.execute(PersistentEligibleJobExecution.java:51)
at com.pega.platform.executor.jobscheduler.scheduler.internal.ClusterJobDecorator.run(ClusterJobDecorator.java:89)
at com.pega.platform.executor.jobscheduler.scheduler.internal.service.SelfMonitoringRunnable.run(SelfMonitoringRunnable.java:74)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.pega.platform.remoteexecution.management.UnresponsiveNodeException: Remote execution request c32ee7c2-3b01-4c5e-bbdb-2208621d317d was not acknowledged by node ac2ad558-b731-458a-b913-9c8ffff4c7ae within the timeout
at com.pega.platform.remoteexecution.internal.ExecutionManager.failJob(ExecutionManager.java:89)
at com.pega.platform.remoteexecution.internal.management.AcknowledgeRetryController.retryJob(AcknowledgeRetryController.java:92)
at com.pega.platform.remoteexecution.internal.management.RetryTask.run(RetryTask.java:42)
... 3 more
2025-12-14 14:03:17,344 [OBSCHEDULER_THREAD_9] [ STANDARD] [ ] [ ] (perations.ClusterManagementAPI) ERROR B5QP0JF6T3GW89I53AI15BDF9FZ9LWHQ7A - exception while fetching high level node info for the node pega-web-tomcat-0
2025-12-14 14:03:17,352 [OBSCHEDULER_THREAD_9] [ STANDARD] [ ] [ ] (.cluster.ClusterRemoteExecutor) ERROR B5QP0JF6T3GW89I53AI15BDF9FZ9LWHQ7A - Skip task PRPCTask{tenant-name=shared, class=com.pega.pegarules.management.internal.system.operations.jobs.MemoryUsageMetricJob, data=com.pega.pegarules.management.internal.system.operations.jobinput.MemoryUsageMetricJobInput@4bda9aac} execution. None of members [Member: [name=pega-web-tomcat-0, address=pega-web-tomcat-0/172.18.0.9:0, serviceVersion=25.1.1, uuid=ac2ad558-b731-458a-b913-9c8ffff4c7ae, mode=SERVER]] match filtering criteria MembersByNamePredicate{members=[]}.
2025-12-14 14:03:17,352 [OBSCHEDULER_THREAD_9] [ STANDARD] [ ] [ ] (tions.MemoryUsageMetricMonitor) ERROR B5QP0JF6T3GW89I53AI15BDF9FZ9LWHQ7A - No nodes were found for the provided node types: [BackgroundProcessing]
2025-12-14 14:03:17,352 [OBSCHEDULER_THREAD_9] [ STANDARD] [ ] [ ] ( QPAutoTuneUpdate) ERROR B5QP0JF6T3GW89I53AI15BDF9FZ9LWHQ7A - Nodes are reporting unhealthy for a threshold of 90.0
It turned out to be a setting in my single node Kafka container. Adding the following env variable solved the problem:
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
But it’s still interesting that without the above set, my Kafka container was “good enough” as a regular stream node, but “not good enough” for the purpose of cluster management.
