CDV ❯ Interrupted Exception icould cause the server to stall.
-
Bug
-
Status: Closed
-
1 Critical
-
Resolution: Fixed
-
-
-
rsingh
-
Reporter: beallman
-
September 17, 2009
-
1
-
Watchers: 3
-
July 27, 2012
-
December 08, 2009
Description
The TPS of the server fell to zero, the server log doesnt give much. But the console logs have this exception.
java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1062) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:307) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:240) at com.tc.util.concurrent.TCLinkedBlockingQueue.put(TCLinkedBlockingQueue.java:34) at com.tc.async.impl.StageQueueImpl.add(StageQueueImpl.java:90) at com.tc.net.protocol.delivery.StateMachineRunner.scheduleIfNeeded(StateMachineRunner.java:74) at com.tc.net.protocol.delivery.StateMachineRunner.addEvent(StateMachineRunner.java:68) at com.tc.net.protocol.delivery.GuaranteedDeliveryProtocol.send(GuaranteedDeliveryProtocol.java:32) at com.tc.net.protocol.delivery.OnceAndOnlyOnceProtocolNetworkLayerImpl.send(OnceAndOnlyOnceProtocolNetworkLayerImpl.java:97) at com.tc.net.protocol.tcm.AbstractMessageChannel.send(AbstractMessageChannel.java:165) at com.tc.net.protocol.tcm.TCMessageImpl.basicSend(TCMessageImpl.java:325) at com.tc.net.protocol.tcm.TCMessageImpl.send(TCMessageImpl.java:320) at com.tc.management.remote.protocol.terracotta.TunnelingMessageConnection.writeMessage(TunnelingMessageConnection.java:82) at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.sendWithReturn(ClientSynchroMessageConnectionImpl.java:231) at javax.management.remote.generic.ClientIntermediary.mBeanServerRequest(ClientIntermediary.java:975) at javax.management.remote.generic.ClientIntermediary.mBeanServerRequest(ClientIntermediary.java:957) at javax.management.remote.generic.ClientIntermediary.getAttribute(ClientIntermediary.java:369) at javax.management.remote.generic.GenericConnector$RemoteMBeanServerConnection.getAttribute(GenericConnector.java:490) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:175) at $Proxy0.getSupportedStatistics(Unknown Source) at com.tc.statistics.agent.StatisticsAgentConnection.getSupportedStatistics(StatisticsAgentConnection.java:70) at com.tc.statistics.beans.impl.StatisticsGatewayMBeanImpl.getSupportedStatistics(StatisticsGatewayMBeanImpl.java:121) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at com.sun.jmx.mbeanserver.StandardMetaDataImpl.getAttribute(StandardMetaDataImpl.java:637) at com.sun.jmx.mbeanserver.StandardMetaDataImpl.getAttribute(StandardMetaDataImpl.java:265) at javax.management.StandardMBean.getAttribute(StandardMBean.java:278) at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.getAttribute(DynamicMetaDataImpl.java:96) at com.sun.jmx.mbeanserver.MetaDataImpl.getAttribute(MetaDataImpl.java:181) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:638) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:659) at javax.management.remote.generic.ServerIntermediary.handleRequest(ServerIntermediary.java:222) at javax.management.remote.generic.ServerIntermediary$PrivilegedRequestJob.run(ServerIntermediary.java:951) at java.security.AccessController.doPrivileged(Native Method) at javax.management.remote.generic.ServerIntermediary$RequestHandler.handleMBSReqMessage(ServerIntermediary.java:727) at javax.management.remote.generic.ServerIntermediary$RequestHandler.execute(ServerIntermediary.java:629) at com.sun.jmx.remote.generic.ServerSynchroMessageConnectionImpl$RemoteJob.run(ServerSynchroMessageConnectionImpl.java:266) at com.sun.jmx.remote.opt.util.ThreadService$ThreadServiceJob.run(ThreadService.java:208) at com.sun.jmx.remote.opt.util.JobExecutor.run(JobExecutor.java:59)
Comments
Russell Beall 2009-09-17
Russell Beall 2009-09-21
This issue appears related to the console or the generic use of the JMX service to obtain usage statistics. Can someone confirm that this error will be avoided by refraining from using the console and (perhaps) other JMX monitoring services?
I was unable to edit many of the fields when I cloned this issue, so several of the notes above, such as which version this applies to, are incorrect as related to this latest issue. Someone with admin access should edit these to indicate it applies to 3.0.1 at least, and remove the items which indicate it was fixed already.
Saravanan Subbiah 2009-09-22
Look at this commit by Tim.
move JMX processing into seperate threads to avoid interrupt() issues (DEV-1955)
Manoj Govindassamy 2009-09-24
CDV-1389 console exception is same as DEV-1955 .
DEV-1955 fix is available in 2.6, 2.6.3, 2.7 and trunk. But, CDV-1389 is reported in 3.0.1 line right ?
Russell Beall 2009-09-24
Yes. At USC we were staging in a 3.0.1 cluster. I had a developer console attached and monitoring. I also had the hyperic sigar libraries for solaris 64 installed (so theoretically that should have helped it work better perhaps…).
The stacktrace above (the one I included in my above comment) was taken from the server log in the 3.0.1 cluster.
Manoj Govindassamy 2009-09-24
just verified 3.0 and 3.1 lines. They have the fix merged.
Saravanan Subbiah 2009-09-24
The exception happened while adding to OOO, it might have messed up the state.
Saravanan Subbiah 2009-09-24
Is pool-1-thread-20 a JMX thread ? What is it trying to do ? May be the exception has nothing to do with the stall ? Not sure though.
Tim Eck 2009-10-20
It is possible that the thread isolation provided by the remote JMX stuff only applies to jmx “operations” and not attribute requests. This looks like an attribute request and thus jmx might interrupt a thread while it is in the middle of adding to OOO
Saravanan Subbiah 2009-11-26
To answer the question that Russell asked, this shouldnt happen if developer console is not connected to the server.
Raghvendra Singh 2009-12-08
merged in 3.2 with rev 14149 and in 3.1 with rev 14156
The above cloned issue seems to have reappeared. We implemented a 3.0.1 cluster and gradually staged it into production. This same error above stalled the entire cluster. Here is the exception I had in my server log:
2009-09-17 10:44:02,658 [pool-1-thread-20] ERROR com.tc.async.api.Sink: ooo_net_send_stage - Exception thrown java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1135) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:241) at com.tc.util.concurrent.TCLinkedBlockingQueue.put(TCLinkedBlockingQueue.java:34) at com.tc.async.impl.StageQueueImpl.add(StageQueueImpl.java:90) at com.tc.net.protocol.delivery.StateMachineRunner.scheduleIfNeeded(StateMachineRunner.java:72) at com.tc.net.protocol.delivery.StateMachineRunner.addEvent(StateMachineRunner.java:66) at com.tc.net.protocol.delivery.GuaranteedDeliveryProtocol.send(GuaranteedDeliveryProtocol.java:33) at com.tc.net.protocol.delivery.OnceAndOnlyOnceProtocolNetworkLayerImpl.send(OnceAndOnlyOnceProtocolNetworkLayerImpl.java:98) at com.tc.net.protocol.tcm.AbstractMessageChannel.send(AbstractMessageChannel.java:197) at com.tc.net.protocol.tcm.TCMessageImpl.basicSend(TCMessageImpl.java:350) at com.tc.net.protocol.tcm.TCMessageImpl.send(TCMessageImpl.java:345) at com.tc.management.remote.protocol.terracotta.TunnelingMessageConnection.writeMessage(TunnelingMessageConnection.java:82) at com.sun.jmx.remote.generic.ClientSynchroMessageConnectionImpl.sendWithReturn(ClientSynchroMessageConnectionImpl.java:231) at javax.management.remote.generic.ClientIntermediary.mBeanServerRequest(ClientIntermediary.java:975) at javax.management.remote.generic.ClientIntermediary.mBeanServerRequest(ClientIntermediary.java:957) at javax.management.remote.generic.ClientIntermediary.getAttribute(ClientIntermediary.java:369) at javax.management.remote.generic.GenericConnector$RemoteMBeanServerConnection.getAttribute(GenericConnector.java:490) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:263) at $Proxy13.getCpuUsage(Unknown Source) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:65) at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:216) at com.sun.jmx.mbeanserver.MBeanSupport.getAttributes(MBeanSupport.java:223) at javax.management.StandardMBean.getAttributes(StandardMBean.java:376) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:726) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:665) at com.tc.stats.DSO$AttributeListTask.call(DSO.java:493) at com.tc.stats.DSO$AttributeListTask.call(DSO.java:481) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)