Unable to fill pool: exception

General
, , , , , ,
1

We are not able to connect to the pega (7.3) application as we are getting the below highlighted error.

Recently we moved the database from stack 7 to stack 8 and with the new user we are able to connect to oracle client.

However Jboss connection is getting failed with below error for one of our environment.

Previously we have done the same change for other environment and its working fine.

Please provide suggestions.

2022-06-07 00:04:22,635 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (JCA PoolFiller) IJ000610: Unable to fill pool: java:/jdbc/AdminPegaRULES: javax.resource.ResourceException: IJ031084: Unable to create connection
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:345)
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:352)
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1322)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.fillTo(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1142)
at org.jboss.jca.core.connectionmanager.pool.mcp.PoolFiller.run(PoolFiller.java:97)
at java.lang.Thread.run(Thread.java:748)

Caused by: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:774)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:688)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:39)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:691)
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321)
… 6 more

Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection

2022-06-07 00:08:37,099 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (JCA PoolFiller) IJ000610: Unable to fill pool: java:/jdbc/PegaRULES: javax.resource.ResourceException: IJ031084: Unable to create connection
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:345)
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:352)
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1322)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.fillTo(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1142)
at org.jboss.jca.core.connectionmanager.pool.mcp.PoolFiller.run(PoolFiller.java:97)
at java.lang.Thread.run(Thread.java:748)

Caused by: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

2

@LivyaJ02 can you give us more details of the exact method used for your change involved with "..Recently we moved the database from stack 7 to stack 8 "?

What is the current Oracle version deployed?

The error shows that Pega successfully calls Data Source Manager but database connection is not returned from connection pool. This appears to be an app server issue rather than a Pega error. Please debug further what reason is of exception

OnePool] (JCA PoolFiller) IJ000610: Unable to fill pool: java:/jdbc/AdminPegaRULES: javax.resource.ResourceException: IJ031084: Unable to create connection

The resource from Jboss, might be helpful -

https://stackoverflow.com/questions/2579999/jboss-jdbc-warning-unable-to-fill-pool

"java.net.SocketException: Connection reset" while connecting to Oracle DB - Red Hat Customer Portal

https://developer.jboss.org/thread/158347

https://community.oracle.com/tech/developers/discussion/1331914/unable-to-fill-pool

Please note that connection test validates direct connection. It does not get connection from a pool. Your issue appears to in how DB is handling stale connections

I have found an open support ticket - can you confirm the INC number that you have already logged with the support department?

3

@LivyaJ02

The support ticket was resolved when it was found that you were getting a connection from DB is getting timed out.

Logs were showing that while connection to the DB the request got timedout.

Our support staff suggested to check with your network team on why it is taking long time to connect to the below highlighted address.

Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:523)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:521)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:660)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:286)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1438)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:518)
… 10 more
Caused by: java.io.IOException: Connection timed out, socket connect lapse 127232 ms. no8dt210_vip.os.amadeus.net/xx.xx.xx.xxx xxxxx 0 1 true
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:209)
at oracle.net.nt.ConnOption.connect(ConnOption.java:161)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470)
… 15 more

Caused by: java.net.ConnectException: Connection timed out
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:482)
at sun.nio.ch.Net.connect(Net.java:474)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
at oracle.net.nt.TimeoutSocketChannel.(TimeoutSocketChannel.java:81)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:169)
… 17 more

I will close this thread based on the understanding that you can check further steps with your network team and DB team