[pool] Possible deadlock or user error (me) commons-pool 2.5.0

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[pool] Possible deadlock or user error (me) commons-pool 2.5.0

Bruce Milner
Hello,

I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
for not using out-of-the-box is that the existing code relies on
changing catalogs at runtime reusing an existing connection. The
original design was to use multiple databases using the same connection
and this cannot be changed.

I recently replaced a lot of hand crafted code with the commons-pool2
implementation.

The issue I have is that one server I manage went into a state where
there are plenty of connections, but none are being returned to the
pool. They are all stuck on a lock inside of
GenericKeyedObjectPool.returnObject.

The config is basically
         GenericKeyedObjectPoolConfig config = new
GenericKeyedObjectPoolConfig();
         config.setBlockWhenExhausted(true);
         config.setMaxTotal(120);
         config.setMaxTotalPerKey(60);
         config.setTestOnBorrow(true);
         config.setTimeBetweenEvictionRunsMillis(60000);
         config.setMinEvictableIdleTimeMillis(0); // don't starve
connections because of catalog switches.
         /**
          * For database connections, use FIFO so that we get rid of
older connections first before newer ones.
          */
         config.setLifo(false);
         return new GenericKeyedObjectPool<String, PooledConnection>(new
PooledConnectionFactory(), config);

There are 150 of these threads waiting on a lock to release connections
    java.lang.Thread.State: WAITING (parking)
     at sun.misc.Unsafe.park(Native Method)
     - parking to wait for  <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
     at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
     at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
     at
org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
     at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
     at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
     at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
     at
com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
     at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)

and 158 of these threads waiting to open connections.
  java.lang.Thread.State: WAITING (parking)
     at sun.misc.Unsafe.park(Native Method)
     - parking to wait for  <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
     at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
     at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
     at
org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
     at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
     at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
     at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
     at com.ilrn.util.sql.Database.getConnection(Database.java:1273)

For the "- parking to wait for  <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
Though I see no other messages with that object in the thread dump.

Has anyone run into this? It seems like some sort of deadlock.

--
Bruce Milner
Senior Software Developer (Emberex)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

garydgregory
Hi,

A side question: Have you tried Apache Commons DBCP (which is based on
Commons Pool)?

https://commons.apache.org/proper/commons-dbcp/

Gary

On Fri, Sep 7, 2018 at 5:24 PM Bruce Milner <[hidden email]>
wrote:

> Hello,
>
> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
> for not using out-of-the-box is that the existing code relies on
> changing catalogs at runtime reusing an existing connection. The
> original design was to use multiple databases using the same connection
> and this cannot be changed.
>
> I recently replaced a lot of hand crafted code with the commons-pool2
> implementation.
>
> The issue I have is that one server I manage went into a state where
> there are plenty of connections, but none are being returned to the
> pool. They are all stuck on a lock inside of
> GenericKeyedObjectPool.returnObject.
>
> The config is basically
>          GenericKeyedObjectPoolConfig config = new
> GenericKeyedObjectPoolConfig();
>          config.setBlockWhenExhausted(true);
>          config.setMaxTotal(120);
>          config.setMaxTotalPerKey(60);
>          config.setTestOnBorrow(true);
>          config.setTimeBetweenEvictionRunsMillis(60000);
>          config.setMinEvictableIdleTimeMillis(0); // don't starve
> connections because of catalog switches.
>          /**
>           * For database connections, use FIFO so that we get rid of
> older connections first before newer ones.
>           */
>          config.setLifo(false);
>          return new GenericKeyedObjectPool<String, PooledConnection>(new
> PooledConnectionFactory(), config);
>
> There are 150 of these threads waiting on a lock to release connections
>     java.lang.Thread.State: WAITING (parking)
>      at sun.misc.Unsafe.park(Native Method)
>      - parking to wait for  <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>      at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>      at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>      at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>      at
>
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>      at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>      at
>
> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
>      at
>
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
>      at
>
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
>      at
>
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
>      at
>
> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
>      at
>
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>
> and 158 of these threads waiting to open connections.
>   java.lang.Thread.State: WAITING (parking)
>      at sun.misc.Unsafe.park(Native Method)
>      - parking to wait for  <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>      at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>      at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>      at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>      at
>
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>      at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>      at
>
> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
>      at
>
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
>      at
>
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
>      at
>
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
>      at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>
> For the "- parking to wait for  <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
> Though I see no other messages with that object in the thread dump.
>
> Has anyone run into this? It seems like some sort of deadlock.
>
> --
> Bruce Milner
> Senior Software Developer (Emberex)
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

Bruce Milner
Hello,

I did a while back, but my understanding of DBCP is that it has one pool
per database and we have thousands.

With the number of nodes serving the application multiplied by the
number of databases, it could easily exceed maximum number of
connections to existing SQL database server. The individual databases
are mostly shared by one database server each have individual schemas.
We keep track of the database URL and switch the connection via
connector.setCatalog(). We also have some that co-exist, so the
connection pool has the smarts to decide if it needs a catalog change.

The commons pool has been working great so far, and this is the only
case I have see where we ended up in this state. We haven't seen this
with load tests, but this once in production.

I was hoping if this exposed a bug, could get fixed in the pool code. I
don't have a reproduction case at this time. I forgot to mention that
the environment is java 8 141 with Connector/J 5.1.45

--bruce

On 9/7/2018 5:05 PM, Gary Gregory wrote:

> Hi,
>
> A side question: Have you tried Apache Commons DBCP (which is based on
> Commons Pool)?
>
> https://commons.apache.org/proper/commons-dbcp/
>
> Gary
>
> On Fri, Sep 7, 2018 at 5:24 PM Bruce Milner <[hidden email]>
> wrote:
>
>> Hello,
>>
>> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
>> for not using out-of-the-box is that the existing code relies on
>> changing catalogs at runtime reusing an existing connection. The
>> original design was to use multiple databases using the same connection
>> and this cannot be changed.
>>
>> I recently replaced a lot of hand crafted code with the commons-pool2
>> implementation.
>>
>> The issue I have is that one server I manage went into a state where
>> there are plenty of connections, but none are being returned to the
>> pool. They are all stuck on a lock inside of
>> GenericKeyedObjectPool.returnObject.
>>
>> The config is basically
>>           GenericKeyedObjectPoolConfig config = new
>> GenericKeyedObjectPoolConfig();
>>           config.setBlockWhenExhausted(true);
>>           config.setMaxTotal(120);
>>           config.setMaxTotalPerKey(60);
>>           config.setTestOnBorrow(true);
>>           config.setTimeBetweenEvictionRunsMillis(60000);
>>           config.setMinEvictableIdleTimeMillis(0); // don't starve
>> connections because of catalog switches.
>>           /**
>>            * For database connections, use FIFO so that we get rid of
>> older connections first before newer ones.
>>            */
>>           config.setLifo(false);
>>           return new GenericKeyedObjectPool<String, PooledConnection>(new
>> PooledConnectionFactory(), config);
>>
>> There are 150 of these threads waiting on a lock to release connections
>>      java.lang.Thread.State: WAITING (parking)
>>       at sun.misc.Unsafe.park(Native Method)
>>       - parking to wait for  <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>       at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>       at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>>       at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>>       at
>>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>>       at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>>       at
>>
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
>>       at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
>>       at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
>>       at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
>>       at
>>
>> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
>>       at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>>
>> and 158 of these threads waiting to open connections.
>>    java.lang.Thread.State: WAITING (parking)
>>       at sun.misc.Unsafe.park(Native Method)
>>       - parking to wait for  <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>       at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>       at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>>       at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>>       at
>>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>>       at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>>       at
>>
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
>>       at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
>>       at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
>>       at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
>>       at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>>
>> For the "- parking to wait for  <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>> Though I see no other messages with that object in the thread dump.
>>
>> Has anyone run into this? It seems like some sort of deadlock.
>>
>> --
>> Bruce Milner
>> Senior Software Developer (Emberex)
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

--
Bruce Milner
Senior Software Developer (Emberex)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

Mark Thomas
In reply to this post by Bruce Milner
On 07/09/18 22:56, Bruce Milner wrote:

> Hello,
>
> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
> for not using out-of-the-box is that the existing code relies on
> changing catalogs at runtime reusing an existing connection. The
> original design was to use multiple databases using the same connection
> and this cannot be changed.
>
> I recently replaced a lot of hand crafted code with the commons-pool2
> implementation.
>
> The issue I have is that one server I manage went into a state where
> there are plenty of connections, but none are being returned to the
> pool. They are all stuck on a lock inside of
> GenericKeyedObjectPool.returnObject.
>
> The config is basically
>         GenericKeyedObjectPoolConfig config = new
> GenericKeyedObjectPoolConfig();
>         config.setBlockWhenExhausted(true);
>         config.setMaxTotal(120);
>         config.setMaxTotalPerKey(60);
>         config.setTestOnBorrow(true);
>         config.setTimeBetweenEvictionRunsMillis(60000);
>         config.setMinEvictableIdleTimeMillis(0); // don't starve
> connections because of catalog switches.
>         /**
>          * For database connections, use FIFO so that we get rid of
> older connections first before newer ones.
>          */
>         config.setLifo(false);
>         return new GenericKeyedObjectPool<String, PooledConnection>(new
> PooledConnectionFactory(), config);
>
> There are 150 of these threads waiting on a lock to release connections
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>
>     at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>
>     at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>     at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
>
>     at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
>
>     at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
>
>     at
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
>
>     at
> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
>
>     at
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>
>
> and 158 of these threads waiting to open connections.
>  java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>
>     at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>
>     at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>     at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
>
>     at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
>
>     at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
>
>     at
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
>
>     at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>
> For the "- parking to wait for  <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
> Though I see no other messages with that object in the thread dump.
>
> Has anyone run into this? It seems like some sort of deadlock.

Do you still have the full thread dump? Can you post it somewhere (where
we can look at it)?

Mark



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

Bernd Eckenfels
In reply to this post by Bruce Milner
Hello Bruce,

This sounds a bit like a discussion we had about missing wakeups. I think it’s was related to depleted pools. Didn’t find the discussion, hopefully somebody else recalls the conditions? I think it was not fixed.

Gruss
Bernd
--
http://bernd.eckenfels.net

________________________________
Von: Bruce Milner <[hidden email]>
Gesendet: Samstag, September 8, 2018 8:19 PM
An: [hidden email]
Betreff: Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

Hello,

I did a while back, but my understanding of DBCP is that it has one pool
per database and we have thousands.

With the number of nodes serving the application multiplied by the
number of databases, it could easily exceed maximum number of
connections to existing SQL database server. The individual databases
are mostly shared by one database server each have individual schemas.
We keep track of the database URL and switch the connection via
connector.setCatalog(). We also have some that co-exist, so the
connection pool has the smarts to decide if it needs a catalog change.

The commons pool has been working great so far, and this is the only
case I have see where we ended up in this state. We haven't seen this
with load tests, but this once in production.

I was hoping if this exposed a bug, could get fixed in the pool code. I
don't have a reproduction case at this time. I forgot to mention that
the environment is java 8 141 with Connector/J 5.1.45

--bruce

On 9/7/2018 5:05 PM, Gary Gregory wrote:

> Hi,
>
> A side question: Have you tried Apache Commons DBCP (which is based on
> Commons Pool)?
>
> https://commons.apache.org/proper/commons-dbcp/
>
> Gary
>
> On Fri, Sep 7, 2018 at 5:24 PM Bruce Milner <[hidden email]>
> wrote:
>
>> Hello,
>>
>> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
>> for not using out-of-the-box is that the existing code relies on
>> changing catalogs at runtime reusing an existing connection. The
>> original design was to use multiple databases using the same connection
>> and this cannot be changed.
>>
>> I recently replaced a lot of hand crafted code with the commons-pool2
>> implementation.
>>
>> The issue I have is that one server I manage went into a state where
>> there are plenty of connections, but none are being returned to the
>> pool. They are all stuck on a lock inside of
>> GenericKeyedObjectPool.returnObject.
>>
>> The config is basically
>> GenericKeyedObjectPoolConfig config = new
>> GenericKeyedObjectPoolConfig();
>> config.setBlockWhenExhausted(true);
>> config.setMaxTotal(120);
>> config.setMaxTotalPerKey(60);
>> config.setTestOnBorrow(true);
>> config.setTimeBetweenEvictionRunsMillis(60000);
>> config.setMinEvictableIdleTimeMillis(0); // don't starve
>> connections because of catalog switches.
>> /**
>> * For database connections, use FIFO so that we get rid of
>> older connections first before newer ones.
>> */
>> config.setLifo(false);
>> return new GenericKeyedObjectPool<String, PooledConnection>(new
>> PooledConnectionFactory(), config);
>>
>> There are 150 of these threads waiting on a lock to release connections
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>> at
>>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>> at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>> at
>>
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>>
>> and 158 of these threads waiting to open connections.
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>> at
>>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>> at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>> at
>>
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
>> at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>>
>> For the "- parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>> Though I see no other messages with that object in the thread dump.
>>
>> Has anyone run into this? It seems like some sort of deadlock.
>>
>> --
>> Bruce Milner
>> Senior Software Developer (Emberex)
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

--
Bruce Milner
Senior Software Developer (Emberex)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

Bruce Milner
In reply to this post by Mark Thomas
Hello Mark,

I sent the thread dump directly to you.

On 9/8/2018 11:36 AM, Mark Thomas wrote:

> On 07/09/18 22:56, Bruce Milner wrote:
>> Hello,
>>
>> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
>> for not using out-of-the-box is that the existing code relies on
>> changing catalogs at runtime reusing an existing connection. The
>> original design was to use multiple databases using the same connection
>> and this cannot be changed.
>>
>> I recently replaced a lot of hand crafted code with the commons-pool2
>> implementation.
>>
>> The issue I have is that one server I manage went into a state where
>> there are plenty of connections, but none are being returned to the
>> pool. They are all stuck on a lock inside of
>> GenericKeyedObjectPool.returnObject.
>>
>> The config is basically
>>          GenericKeyedObjectPoolConfig config = new
>> GenericKeyedObjectPoolConfig();
>>          config.setBlockWhenExhausted(true);
>>          config.setMaxTotal(120);
>>          config.setMaxTotalPerKey(60);
>>          config.setTestOnBorrow(true);
>>          config.setTimeBetweenEvictionRunsMillis(60000);
>>          config.setMinEvictableIdleTimeMillis(0); // don't starve
>> connections because of catalog switches.
>>          /**
>>           * For database connections, use FIFO so that we get rid of
>> older connections first before newer ones.
>>           */
>>          config.setLifo(false);
>>          return new GenericKeyedObjectPool<String, PooledConnection>(new
>> PooledConnectionFactory(), config);
>>
>> There are 150 of these threads waiting on a lock to release connections
>>     java.lang.Thread.State: WAITING (parking)
>>      at sun.misc.Unsafe.park(Native Method)
>>      - parking to wait for  <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>      at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>
>>      at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>>
>>      at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>>
>>      at
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>>
>>      at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>>      at
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
>>
>>      at
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
>>
>>      at
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
>>
>>      at
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
>>
>>      at
>> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
>>
>>      at
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>>
>>
>> and 158 of these threads waiting to open connections.
>>   java.lang.Thread.State: WAITING (parking)
>>      at sun.misc.Unsafe.park(Native Method)
>>      - parking to wait for  <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>      at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>
>>      at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>>
>>      at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>>
>>      at
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>>
>>      at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>>      at
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
>>
>>      at
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
>>
>>      at
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
>>
>>      at
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
>>
>>      at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>>
>> For the "- parking to wait for  <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>> Though I see no other messages with that object in the thread dump.
>>
>> Has anyone run into this? It seems like some sort of deadlock.
> Do you still have the full thread dump? Can you post it somewhere (where
> we can look at it)?
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>

--
Bruce Milner
Senior Software Developer (Emberex)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

Mark Thomas
In reply to this post by Mark Thomas
On 08/09/18 19:36, Mark Thomas wrote:

<snip/>

>> For the "- parking to wait for  <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>> Though I see no other messages with that object in the thread dump.
>>
>> Has anyone run into this? It seems like some sort of deadlock.
>
> Do you still have the full thread dump? Can you post it somewhere (where
> we can look at it)?

Thanks. Lots of threads waiting for the lock. None holding it. That is
very strange.

Some variation of this?

https://confluence.atlassian.com/jirakb/jira-applications-stall-due-to-stackoverflowerror-exception-941601100.html

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

Phil Steitz
On 9/9/18 12:16 AM, Mark Thomas wrote:

> On 08/09/18 19:36, Mark Thomas wrote:
>
> <snip/>
>
>>> For the "- parking to wait for  <0x00000006471cd7d8> (a
>>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>>> Though I see no other messages with that object in the thread dump.
>>>
>>> Has anyone run into this? It seems like some sort of deadlock.
>> Do you still have the full thread dump? Can you post it somewhere (where
>> we can look at it)?
> Thanks. Lots of threads waiting for the lock. None holding it. That is
> very strange.
>
> Some variation of this?
>
> https://confluence.atlassian.com/jirakb/jira-applications-stall-due-to-stackoverflowerror-exception-941601100.html
>
Here is one thing to look at.  Note that the take request from
borrowObject comes from
p = objectDeque.getIdleObjects().pollFirst();

This LinkedBlockingDeque method does not register the thread as
waiting on the notEmpty condition in LinkedBlockingDeque the way
that say
  p = objectDeque.getIdleObjects().pollFirst(
                                     borrowMaxWaitMillis,
TimeUnit.MILLISECONDS);
does.

Phil

> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0

Bruce Milner
In reply to this post by Mark Thomas
On 9/9/2018 12:16 AM, Mark Thomas wrote:

> On 08/09/18 19:36, Mark Thomas wrote:
>
> <snip/>
>
>>> For the "- parking to wait for  <0x00000006471cd7d8> (a
>>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>>> Though I see no other messages with that object in the thread dump.
>>>
>>> Has anyone run into this? It seems like some sort of deadlock.
>> Do you still have the full thread dump? Can you post it somewhere (where
>> we can look at it)?
> Thanks. Lots of threads waiting for the lock. None holding it. That is
> very strange.
>
> Some variation of this?
>
> https://confluence.atlassian.com/jirakb/jira-applications-stall-due-to-stackoverflowerror-exception-941601100.html
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
Yeah, I looked closer at thread dump and no thread holding lock. Also
looked in past several months of logs and no StackOverflowError's .
My guess now is JVM bug.

Thanks for looking!

--bruce

--
Bruce Milner
Senior Software Developer (Emberex)


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]