SPECvirt APP workload errors regarding max-pool-size

[bcarson]

[#|2015-05-06T11:52:34.867-0700|WARNING|glassfish3.1.2|javax.enterprise.system.container.ejb.entity.finder|_ThreadID=205;_ThreadName=Thread-2;|JDO74004: Bean 'SComponentEnt' method ejbFindByPrimaryKey:
com.sun.jdo.api.persistence.support.JDODataStoreException: JDO77006: SQL exception: state = null, error code = 0.
NestedException: java.sql.SQLException: Error in allocating a connection. Cause: In-use connections equal max-pool-size and expired max-wait-time. Cannot allocate more connections.
   at com.sun.jdo.spi.persistence.support.sqlstore.impl.TransactionImpl.getConnectionInternal(TransactionImpl.java:1469)
   at com.sun.jdo.spi.persistence.support.sqlstore.impl.TransactionImpl.getConnection(TransactionImpl.java:1362)
   at com.sun.jdo.spi.persistence.support.sqlstore.SQLStoreManager.executeQuery(SQLStoreManager.java:451)
   at com.sun.jdo.spi.persistence.support.sqlstore.SQLStoreManager.retrieve(SQLStoreManager.java:380)
   at com.sun.jdo.spi.persistence.support.sqlstore.SQLStateManager.retrieve(SQLStateManager.java:2063)
   at com.sun.jdo.spi.persistence.support.sqlstore.SQLStateManager.reload(SQLStateManager.java:1201)
   at com.sun.jdo.spi.persistence.support.sqlstore.SQLStateManager.reload(SQLStateManager.java:1157)
   at com.sun.jdo.spi.persistence.support.sqlstore.impl.PersistenceManagerImpl.getObjectById(PersistenceManagerImpl.java:662)
   at com.sun.jdo.spi.persistence.support.sqlstore.impl.PersistenceManagerWrapper.getObjectById(PersistenceManagerWrapper.java:280)
   at org.spec.jappserver.supplier.scomponentent.ejb.SComponentCmp20EJB_1116577083_ConcreteImpl.ejbFindByPrimaryKey(SComponentCmp20EJB_1116577083_ConcreteImpl.java:249)
   at sun.reflect.GeneratedMethodAccessor215.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
   at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
   at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:4180)
   at com.sun.ejb.containers.EntityContainer.invokeFindByPrimaryKey(EntityContainer.java:813)
   at com.sun.ejb.containers.EJBLocalHomeInvocationHandler.invoke(EJBLocalHomeInvocationHandler.java:241)
   at com.sun.proxy.$Proxy233.findByPrimaryKey(Unknown Source)
   at org.spec.jappserver.supplier.buyermdb.ejb.BuyerMDB.add(BuyerMDB.java:110)
   at org.spec.jappserver.supplier.buyermdb.ejb.BuyerMDB.onMessage(BuyerMDB.java:90)
   at sun.reflect.GeneratedMethodAccessor224.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:606)
   at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
   at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
   at com.sun.ejb.containers.BaseContainer.invokeTargetBeanMethod(BaseContainer.java:4180)
   at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5368)
   at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
   at com.sun.ejb.containers.MessageBeanContainer.deliverMessage(MessageBeanContainer.java:1099)
   at com.sun.ejb.containers.MessageBeanListenerImpl.deliverMessage(MessageBeanListenerImpl.java:81)
   at com.sun.enterprise.connectors.inbound.MessageEndpointInvocationHandler.invoke(MessageEndpointInvocationHandler.java:171)
   at com.sun.proxy.$Proxy261.onMessage(Unknown Source)
   at com.sun.messaging.jms.ra.OnMessageRunner.run(OnMessageRunner.java:260)
   at com.sun.enterprise.connectors.work.OneWork.doWork(OneWork.java:114)
   at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
   at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
Caused by: java.sql.SQLException: Error in allocating a connection. Cause: In-use connections equal max-pool-size and expired max-wait-time. Cannot allocate more connections.
   at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:120)
   at com.sun.jdo.spi.persistence.support.sqlstore.ejb.TransactionHelperImpl.getConnection(TransactionHelperImpl.java:216)
   at com.sun.jdo.spi.persistence.support.sqlstore.ejb.EJBHelper.getConnection(EJBHelper.java:201)
   at com.sun.jdo.spi.persistence.support.sqlstore.impl.TransactionImpl.getConnectionInternal(TransactionImpl.java:1451)
   ... 36 more

[AnoopGupta]

Hi Bruce,

Thanks for posting this question on this forum. Have you tried tuning GlassFish, like the max-pool-size and expired max-wait-time for the datasource you have configured (see GlassFish domain.xml)?

For tuning, please refer to the tgz archive for a SPECvirt_sc2013 publication using similar software stack as yours.

You may also want to check your DB performance, especially for txn_log_table.

[klindgren]

Hello all,

I tired tuning based upon examples that I saw from others domain.xml for glassfish however I am still running into errors.  We don't use SpecjApp or Glassfish at all so we used the setup scripts in the forum posting to get everything installed.  We also don't use postgress at all so I am bit like a fish out of water here (also used the setup scripts to setup the db server).

We keep seeing max-pool-size errors after the appserver test has been running for a while 30+ minutes and typically it starts when the injection rate goes above 150.

The domain.xml appserver and postgress errors can be found here: https://gist.github.com/krislindgren/bcfdae12e52ebc5f289e

[AnoopGupta]

The jdbc-connection-pool tuning in the domain.xml does not look correct. For example, the following are not properties, but attributes:
 <property name="idle-timeout-in-seconds" value="600"></property>
      <property name="max-pool-size" value="2000"></property>
      <property name="pool-resize-quantity" value="16"></property>
      <property name="connection-validation-method" value="auto-commit"></property>
      <property name="steady-pool-size" value="256"></property>

These <attribute>=<value> need to be along with the datasource-classname and res-type within the jdbc-connection-pool tag.

Please see others' domain.xml.

Thanks,
Anoop