Task #5498
Determine the cause of the Jun2014 cn-orc-1 out of memory issue
0%
Description
We experienced a Java OutOfMemory exception for Tomcat on cn-orc-1 on June 14, 2014, and had to restart Tomcat. We need to look back in the logs to determine the cause and potentially prevent it in the future since it's not normal behavior. In /var/log/tomcat6/catalina.out, we saw:
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "FileWatchdog" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "FileWatchdog" Exception in thread "Thread-6" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "hz.dataonePortalInstance.ServiceThread" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-8080-Acceptor-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "hz.dataonePortalInstance.scheduled.thread-2" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-376" java.lang.NullPointerException
Jun 14, 2014 8:43:59 AM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception (java.lang.OutOfMemoryError: GC overhead limit exceeded) executing org.apache.jk.common.ChannelSocket$SocketConnection@5a423982, terminating thread
Exception in thread "hz.dataonePortalInstance.scheduled.thread-1" Exception in thread "hz._hzInstance_1_DataONE.cached.thread-288" at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "hz.dataonePortalInstance.IO.thread-1" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.nio.NodeIOService.onOutOfMemory(NodeIOService.java:56)
at com.hazelcast.nio.InOutSelector.run(InOutSelector.java:155)
at java.lang.Thread.run(Thread.java:662)
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "hz.dataonePortalInstance.scheduled.thread-3" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-379" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
Exception in thread "hz._hzInstance_1_DataONE.ServiceThread" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.cluster.ClusterService.run(ClusterService.java:214)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-385" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-381" java.lang.NullPointerException
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-380" at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:884)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-347" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Jun 14, 2014 10:14:19 AM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "hz._hzInstance_1_DataONE.scheduled.thread-2" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-351" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-250" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Jun 14, 2014 12:11:31 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-342" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-291" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-296" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-301" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-294" java.lang.NullPointerException
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2475)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2266)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:976)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1451)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-300" java.lang.NullPointerException
CN Dispatching: /d1/cn/object
HTTP Verb: GET
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-352" java.lang.NullPointerException
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-360" java.lang.NullPointerException
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-289" java.lang.NullPointerException
Jun 14, 2014 5:11:09 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception (java.lang.OutOfMemoryError: GC overhead limit exceeded) executing org.apache.jk.common.ChannelSocket$SocketConnection@4c5aadd4, terminating thread
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-292" java.lang.NullPointerException
Jun 14, 2014 5:39:40 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-252" java.lang.NullPointerException
Jun 14, 2014 6:44:50 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet jsp threw exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
Jun 14, 2014 6:48:50 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet jsp threw exception
java.lang.NullPointerException
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-359" java.lang.NullPointerException
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-243" java.lang.NullPointerException
org.dataone.service.exceptions.ServiceFailure: class javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:352)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:390)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:149)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:562)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
at org.dataone.client.RestClient.doRequest(RestClient.java:326)
at org.dataone.client.RestClient.doRequestNoBody(RestClient.java:266)
at org.dataone.client.RestClient.doGetRequest(RestClient.java:181)
at org.dataone.client.D1RestClient.doGetRequest(D1RestClient.java:234)
at org.dataone.client.D1RestClient.doGetRequest(D1RestClient.java:223)
at org.dataone.client.CNode.listNodes(CNode.java:505)
at gov.ornl.mercury3.web.util.MemberNodeSourcesMapCache.refreshMNSourceMap(MemberNodeSourcesMapCache.java:85)
at gov.ornl.mercury3.web.util.MemberNodeSourcesMapCache.getMNSourceMap(MemberNodeSourcesMapCache.java:75)
at gov.ornl.mercury3.web.jsptag.MNSourceOptionsTag.doStartTag(MNSourceOptionsTag.java:40)
at org.apache.jsp.index_jsp._jspx_meth_m3_005fmnSourceOptions_005f0(index_jsp.java:786)
at org.apache.jsp.index_jsp._jspService(index_jsp.java:622)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.webdroid.catnip.filter.BadInputFilter.doFilter(BadInputFilter.java:398)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
... 46 more
Exception in thread "hz.dataonePortalInstance.scheduled.thread-4" java.lang.NullPointerException
Jun 14, 2014 7:39:37 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request processing
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "hz.dataonePortalInstance.scheduled.thread-5" java.lang.NullPointerException
20140614-20:12:24: [WARN]: javax.naming.CommunicationException: localhost:389 connection closed [org.dataone.cn.ldap.D1UnsolicitedNotificationListener]
Exception in thread "Thread-3" java.lang.OutOfMemoryError: GC overhead limit exceeded
javax.naming.CommunicationException: localhost:389 connection closed
at com.sun.jndi.ldap.LdapClient.processConnectionClosure(LdapClient.java:478)
at com.sun.jndi.ldap.Connection.cleanup(Connection.java:676)
at com.sun.jndi.ldap.Connection.run(Connection.java:975)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-344" java.lang.NullPointerException
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-244" java.lang.NullPointerException
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-249" java.lang.NullPointerException
History
#1 Updated by David Doyle over 10 years ago
No idea if this has anything to do with anything, but adding a link to https://redmine.dataone.org/issues/3717 here to remind myself that one of these exceptions is mentioned elsewhere. (Possibly more - need to keep looking)
#2 Updated by David Doyle over 10 years ago
More notes:
First GC warning occurred at 8:09 AM Eastern, consisting of:
[ WARN] 2014-06-14 08:09:23,996 (?:?:?) [160.36.13.150]:5701 [DataONE] GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.AbstractStringBuilder.(AbstractStringBuilder.java:45) at java.lang.StringBuilder.(StringBuilder.java:68) at com.hazelcast.impl.BaseManager$ResponseQueueCall.waitAndGetResult(BaseManager.java:613) at com.hazelcast.impl.BaseManager$ResponseQueueCall.getRedoAwareResult(BaseManager.java:641) at com.hazelcast.impl.BaseManager$ResponseQueueCall.getResult(BaseManager.java:636) at com.hazelcast.impl.BaseManager$RequestBasedCall.getResultAsBoolean(BaseManager.java:447) at com.hazelcast.impl.BaseManager$ResponseQueueCall.getResultAsBoolean(BaseManager.java:555) at com.hazelcast.impl.ConcurrentMapManager$MPut.mergeOne(ConcurrentMapManager.java:1752) at com.hazelcast.impl.ConcurrentMapManager$MPut.merge(ConcurrentMapManager.java:1741) at com.hazelcast.impl.LifecycleServiceImpl$1.run(LifecycleServiceImpl.java:143) at com.hazelcast.impl.executor.ParallelExecutorService$ParallelExecutorImpl$ExecutionSegment.run(ParallelExecutorService.java:212) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
Followed by
[ WARN] 2014-06-14 08:29:36,444 (Thread:run:662) [160.36.13.150]:5701 [DataONE] Selector exception at hz._hzInstance_1_DataONE.IO.thread-0, cause= java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded
And then ~500+ instances of this error (last one included here to note end time)
[ WARN] 2014-06-14 22:50:58,820 (?:?:?) [160.36.13.150]:5701 [DataONE] GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded
MEANWHILE, this warning occurred at 3:55 AM Eastern:
[ WARN] 2014-06-14 07:55:36,723 (?:?:?) [160.36.13.150]:5701 [DataONE] Caller -> RedoLog{name=c:hzSystemMetadata, redoType=REDO_PARTITION_MIGRATING, operation=CONCURRENT_MAP_MERGE, target=Address[128.111.54.80]:5701 / connected=true, redoCount=22, migrating=null partition=Partition [1]{ 0:Address[128.111.54.80]:5701 } }
Followed immediately by 4000+ instances of
[ WARN] 2014-06-14 07:55:35,165 (?:?:?) [160.36.13.150]:5701 [DataONE] Caller -> RedoLog{name=c:hzObjectPath, redoType=REDO_TARGET_UNKNOWN, operation=CONCURRENT_MAP_MERGE, target=null / connected=false, redoCount=21, migrating=null partition=Partition [75]{ } }
and
[ WARN] 2014-06-14 07:55:32,657 (?:?:?) [160.36.13.150]:5701 [DataONE] Caller -> RedoLog{name=c:hzSystemMetadata, redoType=REDO_TARGET_UNKNOWN, operation=CONCURRENT_MAP_MERGE, target=null / connected=false, redoCount=16, migrating=null partition=Partition [255]{ } }
with some like this in there as well
[ WARN] 2014-06-14 07:55:21,605 (?:?:?) [160.36.13.150]:5701 [DataONE] Caller -> RedoLog{name=c:hzSystemMetadata, redoType=REDO_MEMBER_UNKNOWN, operation=CONCURRENT_MAP_MERGE, target=Address[128.111.54.80]:5701 / connected=true, redoCount=51, migrating=null partition=Partition [162]{ 0:Address[128.111.54.80]:5701 1:Address[160.36.13.150]:5701 } }
Five connection close errors occurred during this time:
[DEBUG] 2014-06-15 01:07:43,433 (Connection:close:220) [160.36.13.150]:5701 [DataONE] An error occurred on connection to Address[64.106.40.6]:5701 Cause => java.io.IOException {Connection timed out}, Error-Count: 1
[DEBUG] 2014-06-14 07:54:52,403 (Connection:close:220) [160.36.13.150]:5701 [DataONE] An error occurred on connection to Address[128.111.54.80]:5701 Cause => java.io.EOFException {null}, Error-Count: 1
[DEBUG] 2014-06-14 07:39:45,936 (Connection:close:220) [160.36.13.150]:5701 [DataONE] An error occurred on connection to Address[128.111.54.80]:5701 Cause => java.io.EOFException {null}, Error-Count: 1
[DEBUG] 2014-06-14 07:32:15,425 (Connection:close:220) [160.36.13.150]:5701 [DataONE] An error occurred on connection to Address[128.111.54.80]:5701 Cause => java.io.EOFException {null}, Error-Count: 1
[DEBUG] 2014-06-14 07:13:13,279 (Connection:close:220) [160.36.13.150]:5701 [DataONE] An error occurred on connection to Address[64.106.40.6]:5701 Cause => java.io.EOFException {null}, Error-Count: 1
Will begin researching these various warnings and errors next.
#3 Updated by David Doyle about 10 years ago
- Assignee changed from David Doyle to Michael Campfield
Assigning to Michael Campfield for future investigation.
See http://stackoverflow.com/questions/4371505/gc-overhead-limit-exceeded for GC overhead limit exceeded error explanation, and http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html for explanation of GC tuning. Will need to find root cause, compare tuning recommendations w/current implementation at DataONE, etc.
