Project

General

Profile

Bug #8195

Tomcat out of memory error

Added by Dave Vieglais about 4 years ago. Updated almost 4 years ago.

Status:
In Progress
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
2017-10-17
Due date:
% Done:

30%

Milestone:
None
Product Version:
*
Story Points:
Sprint:

Description

Tomcat crashed on cn-orc-1 with an out of memory error:

INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:35336
Oct 17, 2017 8:44:38 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [/0:0:0:0:0:0:0:1:35336] lost. Reason: java.io.EOFException[null]
Oct 17, 2017 8:46:01 PM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [76063] operation=GET_MEMBERS in 5 seconds.
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" java.lang.OutOfMemoryError: Java heap space
20171017-20:57:05: [WARN]: [160.36.13.150]:5702 [hzProcess] Master node has timed out its heartbeat and will be removed [com.hazelcast.cluster.ClusterManager]
Oct 17, 2017 8:45:47 PM com.hazelcast.client.ConnectionManager
WARNING: Server didn't respond to client's ping call within 10 seconds!
Oct 17, 2017 8:51:41 PM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [76063] operation=GET_MEMBERS in 10 seconds.
Oct 17, 2017 8:57:09 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [Address[128.111.54.80]:5703] lost. Reason: java.io.EOFException[null]
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /object
original requestURI: /metacat/d1/cn/v1/object
stripping /metacat/d1/cn/v1 from requestURI
new pathinfo: /object
Exception in thread "Thread-77686" java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ajp-bio-8009-exec-49"
Exception in thread "Timer-3" java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "FileWatchdog"
Exception in thread "hz.dataonePortalInstance.ServiceThread" java.lang.OutOfMemoryError: Java heap space
Exception in thread "ajp-bio-8009-exec-5" java.lang.OutOfMemoryError: Java heap space
Oct 17, 2017 9:03:27 PM com.hazelcast.nio.InOutSelector
WARNING: [160.36.13.150]:5703 [DataONE] Selector exception at  hz.dataonePortalInstance.IO.thread-2, cause= java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space

Exception in thread "ajp-bio-8009-exec-307" java.lang.OutOfMemoryError: Java heap space
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6293" java.lang.NullPointerException
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6289" Oct 17, 2017 9:08:42 PM com.hazelcast.nio.InOutSelector
WARNING: [160.36.13.150]:5703 [DataONE] Selector exception at  hz.dataonePortalInstance.IO.thread-1, cause= java.lang.NullPointerException
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.nio.NodeIOService.onOutOfMemory(NodeIOService.java:56)
    at com.hazelcast.nio.AbstractSelectionHandler.handleSocketException(AbstractSelectionHandler.java:58)
    at com.hazelcast.nio.ReadHandler.handle(ReadHandler.java:92)
    at com.hazelcast.nio.InOutSelector.run(InOutSelector.java:142)
    at java.lang.Thread.run(Thread.java:748)

Exception in thread "hz.dataonePortalInstance.cached.thread-5"  at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
    at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
    at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
    at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
Oct 17, 2017 9:09:03 PM com.hazelcast.nio.InOutSelector
WARNING: [160.36.13.150]:5703 [DataONE] unhandled exception in hz.dataonePortalInstance.IO.thread-2
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.nio.NodeIOService.onOutOfMemory(NodeIOService.java:56)
    at com.hazelcast.nio.InOutSelector.handleSelectorException(InOutSelector.java:171)
    at com.hazelcast.nio.InOutSelector.run(InOutSelector.java:150)
    at java.lang.Thread.run(Thread.java:748)

java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
Oct 17, 2017 9:09:21 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [Address[64.106.40.6]:5703] lost. Reason: java.lang.IllegalArgumentException[null]
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6292" Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6285" Oct 17, 2017 9:09:28 PM com.hazelcast.nio.InOutSelector
WARNING: [160.36.13.150]:5703 [DataONE] Selector exception at  hz.dataonePortalInstance.IO.thread-1, cause= java.lang.NullPointerException
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.nio.NodeIOService.disconnectExistingCalls(NodeIOService.java:180)
    at com.hazelcast.nio.Connection.close(Connection.java:218)
    at com.hazelcast.nio.AbstractSelectionHandler.handleSocketException(AbstractSelectionHandler.java:63)
    at com.hazelcast.nio.ReadHandler.handle(ReadHandler.java:92)
    at com.hazelcast.nio.InOutSelector.run(InOutSelector.java:142)
    at java.lang.Thread.run(Thread.java:748)

    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6288" Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6290" Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6284" Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6286" at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
    at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
    at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
    at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.OutOfMemoryError: Java heap space
Exception in thread "hz._hzInstance_1_DataONE.cached.thread-6282" java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:40)
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
    at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
    at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
Exception in thread "hz._hzInstance_1_DataONE.IO.thread-Acceptor"   at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
    at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
    at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
    at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
    at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
    at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
    at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
    at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
    at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
    at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
    at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
    at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
    at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.impl.ThreadContext.get(ThreadContext.java:67)
    at com.hazelcast.impl.ExecutorManager.threadPoolBeforeExecute(ExecutorManager.java:396)
    at com.hazelcast.impl.ExecutorManager.access$000(ExecutorManager.java:44)
    at com.hazelcast.impl.ExecutorManager$1.beforeExecute(ExecutorManager.java:87)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:38)
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher$DefaultOutOfMemoryHandler.onOutOfMemory(OutOfMemoryErrorDispatcher.java:79)
    at com.hazelcast.impl.OutOfMemoryErrorDispatcher.onOutOfMemory(OutOfMemoryErrorDispatcher.java:70)
    at com.hazelcast.nio.SocketAcceptor.run(SocketAcceptor.java:66)
    at java.lang.Thread.run(Thread.java:748)
Oct 17, 2017 9:11:29 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46066
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
Oct 17, 2017 9:11:33 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46066
Oct 17, 2017 9:11:49 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46311
Oct 17, 2017 9:11:49 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46311
Oct 17, 2017 9:11:49 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [/0:0:0:0:0:0:0:1:46311] lost. Reason: java.io.EOFException[null]
20171017-21:12:10: [WARN]: [160.36.13.150]:5702 [hzProcess] Master node has timed out its heartbeat and will be removed [com.hazelcast.cluster.ClusterManager]
Oct 17, 2017 9:12:10 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46487
Oct 17, 2017 9:12:10 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46487
Oct 17, 2017 9:12:10 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46489
Oct 17, 2017 9:12:10 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46489
Oct 17, 2017 9:12:13 PM com.hazelcast.nio.SocketAcceptor
INFO: [160.36.13.150]:5703 [DataONE] 5703 is accepting socket connection from /0:0:0:0:0:0:0:1:46488
Oct 17, 2017 9:12:13 PM com.hazelcast.nio.ConnectionManager
INFO: [160.36.13.150]:5703 [DataONE] 5703 accepted socket connection from /0:0:0:0:0:0:0:1:46488
20171017-21:12:22: [WARN]: [160.36.13.150]:5702 [hzProcess] ExecutorService is rejecting an execution. java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@732a0f33 [com.hazelcast.impl.ExecutorManager]
Oct 17, 2017 9:12:26 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [/0:0:0:0:0:0:0:1:46488] lost. Reason: java.io.EOFException[null]
20171017-21:12:26: [WARN]: [160.36.13.150]:5702 [hzProcess] ExecutorService is rejecting an execution. com.hazelcast.impl.SplitBrainHandler$1@1af304f [com.hazelcast.impl.ExecutorManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state!  [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state!  [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state!  [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state!  [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state!  [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state!  [com.hazelcast.impl.PartitionManager]
20171017-21:12:29: [WARN]: [160.36.13.150]:5702 [hzProcess] This is the master node and received a PartitionRuntimeState from Address[128.111.54.80]:5702. Ignoring incoming state!  [com.hazelcast.impl.PartitionManager]
Oct 17, 2017 9:12:36 PM com.hazelcast.nio.Connection
INFO: [160.36.13.150]:5703 [DataONE] Connection [/0:0:0:0:0:0:0:1:46487] lost. Reason: java.io.EOFException[null]
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
20171017-21:13:09: [WARN]: [160.36.13.150]:5702 [hzProcess] hz.hzProcessInstance.IO.thread-2 Closing socket to endpoint Address[128.111.54.80]:5702, Cause:java.lang.OutOfMemoryError: Java heap space [com.hazelcast.nio.ReadHandler]
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at com.hazelcast.nio.DataHolder.<init>(DataHolder.java:28)
    at com.hazelcast.nio.Packet.read(Packet.java:379)
    at com.hazelcast.nio.SocketPacketReader$DefaultPacketReader.readPacket(SocketPacketReader.java:88)
    at com.hazelcast.nio.SocketPacketReader.read(SocketPacketReader.java:62)
    at com.hazelcast.nio.ReadHandler.handle(ReadHandler.java:85)
    at com.hazelcast.nio.InOutSelector.run(InOutSelector.java:142)
    at java.lang.Thread.run(Thread.java:748)
20171017-21:13:09: [WARN]: [160.36.13.150]:5702 [hzProcess] hz.hzProcessInstance.IO.thread-1 Closing socket to endpoint Address[64.106.40.6]:5702, Cause:java.lang.OutOfMemoryError: Java heap space [com.hazelcast.nio.ReadHandler]
java.lang.OutOfMemoryError: Java heap space
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
In D1URLFilter.
HTTP Verb: GET
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /object
original requestURI: /metacat/d1/cn/v1/object
stripping /metacat/d1/cn/v1 from requestURI
new pathinfo: /object
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /log
original requestURI: /metacat/d1/cn/v2/log
stripping /metacat/d1/cn/v2 from requestURI
new pathinfo: /log
java.lang.NullPointerException
    at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:3124)
    at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2901)
    at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1209)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1689)
    at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1570)
    at edu.ucsb.nceas.metacat.replication.ReplicationService.getURLStream(ReplicationService.java:2341)
    at edu.ucsb.nceas.metacat.replication.ReplicationService.getURLContent(ReplicationService.java:2299)
    at edu.ucsb.nceas.metacat.replication.ReplicationService.handleForceReplicateDataFileRequest(ReplicationService.java:936)
    at edu.ucsb.nceas.metacat.replication.ReplicationServlet.handleGetOrPost(ReplicationServlet.java:157)
    at edu.ucsb.nceas.metacat.replication.ReplicationServlet.doGet(ReplicationServlet.java:87)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
java.lang.NullPointerException
In D1URLFilter.
HTTP Verb: GET
original pathInfo: /object
original requestURI: /metacat/d1/cn/v1/object
stripping /metacat/d1/cn/v1 from requestURI
new pathinfo: /object

Related issues

Related to Infrastructure - Story #8199: Increase memory allocation to CNs Closed 2017-10-24

History

#1 Updated by Dave Vieglais about 4 years ago

  • Related to Story #8199: Increase memory allocation to CNs added

#2 Updated by Dave Vieglais about 4 years ago

As an emergency stopgap, the memory allocation on CNs was manually adjusted from 8GB to 12GB to avoid OOM issues with Tomcat.

Additional RAM needs to be added to the VMs before taking these settings much higher.

/etc/default/tomcat7

JAVA_OPTS="-Djava.awt.headless=true -Xmx12288M -XX:+UseParallelGC -Xms1024M -XX:MaxPermSize=512M"

#3 Updated by Dave Vieglais about 4 years ago

  • Status changed from New to In Progress
  • Assignee set to Dave Vieglais
  • % Done changed from 0 to 30

#4 Updated by Dave Vieglais almost 4 years ago

  • Description updated (diff)

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 14.8 MB)