Story #8734
Updated by Rob Nahf about 6 years ago
I came across Could this while trying to create an object on the MN. The error leaves Metacat up, but unable to access Hazelcast.
mn-demo-6 is running on 8Gb memory be a root cause?
~~~
root@mn-demo-6:/var/log/tomcat7# grep -B15 -A50 Hazelcast catalina.out | more
original requestURI: /knb/d1/mn/v2/checksum/testV1CreateV2Query_201528602433267
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /checksum/testV1CreateV2Query_201528602433267
After decoded: testV1CreateV2Query_201528602433267
Oct 18, 2018 12:16:57 AM com.hazelcast.client.ConnectionManager
WARNING: Server didn't respond to client's requests for 30 seconds. Assuming it is dead, closing the connection!
original pathInfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/dirtySystemMetadata
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/checksum/testV2CreateV1ListObjects_20152951117775
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /checksum/testV2CreateV1ListObjects_20152951117775
After decoded: testV2CreateV1ListObjects_20152951117775
Oct 18, 2018 12:16:59 AM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_LOST
Oct 18, 2018 12:16:59 AM com.hazelcast.nio.Connection
INFO: [127.0.0.1]:5701 [knb] Connection [Address[127.0.0.1]:43181] lost. Reason: java.io.EOFException[null]
Oct 18, 2018 12:16:59 AM com.hazelcast.client.ConnectionManager
WARNING: Server didn't respond to client's requests for 30 seconds. Assuming it is dead, closing the connection!
Exception in thread "Timer-3" java.lang.NullPointerException
at com.hazelcast.client.ConnectionManager$1.run(ConnectionManager.java:94)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
metacat 20181018-00:17:14: [ERROR]: Error while looking up the local identifier: An I/O error occured while sending to the backend. [edu.ucsb.nceas.metacat.IdentifierManager:getLocalId:1823]
Oct 18, 2018 12:17:03 AM com.hazelcast.nio.Connection
INFO: [127.0.0.1]:5701 [knb] Connection [Address[127.0.0.1]:46297] lost. Reason: java.io.EOFException[null]
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ContainerBackgroundProcessor[S
tandardEngine[Catalina]]"
Oct 18, 2018 12:17:03 AM com.hazelcast.client.ConnectionManager
WARNING: Connection to Connection [0] [localhost/127.0.0.1:5701 -> 127.0.0.1:5701] is lost
metacat 20181018-00:17:35: [ERROR]: NamingException when getting referral contex. Skipping this referral. ldap.ecoinform
atics.org:389 [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:17:49: [ERROR]: NamingException when getting referral contex. Skipping this referral. ldap.ecoinform
atics.org:389 [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
Oct 18, 2018 12:17:01 AM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [622795] operation=CONCURRENT_MAP_REMOVE in 10 seconds.
Oct 18, 2018 12:17:36 AM com.hazelcast.client.ConnectionManager
WARNING: got an exception on getConnection:Java heap space
java.lang.OutOfMemoryError: Java heap space
original pathInfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/dirtySystemMetadata
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /dirtySystemMetadata
original pathInfo: /checksum/testV2CreateV1GetSysMeta_201710915250987
original requestURI: /knb/d1/mn/v2/checksum/testV2CreateV1GetSysMeta_201710915250987
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /checksum/testV2CreateV1GetSysMeta_201710915250987
After decoded: testV2CreateV1GetSysMeta_201710915250987
original pathInfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/dirtySystemMetadata
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /dirtySystemMetadata
original pathInfo: /checksum/testV2CreateV1GetSysMeta_201636402417180
original requestURI: /knb/d1/mn/v2/checksum/testV2CreateV1GetSysMeta_201636402417180
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /checksum/testV2CreateV1GetSysMeta_201636402417180
After decoded: testV2CreateV1GetSysMeta_201636402417180
original pathInfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/dirtySystemMetadata
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /dirtySystemMetadata
metacat 20181018-00:17:54: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:18:00: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:18:00: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:17:59: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:17:59: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
~~~
mn-demo-6 is running on 8Gb memory be a root cause?
~~~
root@mn-demo-6:/var/log/tomcat7# grep -B15 -A50 Hazelcast catalina.out | more
original requestURI: /knb/d1/mn/v2/checksum/testV1CreateV2Query_201528602433267
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /checksum/testV1CreateV2Query_201528602433267
After decoded: testV1CreateV2Query_201528602433267
Oct 18, 2018 12:16:57 AM com.hazelcast.client.ConnectionManager
WARNING: Server didn't respond to client's requests for 30 seconds. Assuming it is dead, closing the connection!
original pathInfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/dirtySystemMetadata
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/checksum/testV2CreateV1ListObjects_20152951117775
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /checksum/testV2CreateV1ListObjects_20152951117775
After decoded: testV2CreateV1ListObjects_20152951117775
Oct 18, 2018 12:16:59 AM com.hazelcast.client.LifecycleServiceClientImpl
INFO: HazelcastClient is CLIENT_CONNECTION_LOST
Oct 18, 2018 12:16:59 AM com.hazelcast.nio.Connection
INFO: [127.0.0.1]:5701 [knb] Connection [Address[127.0.0.1]:43181] lost. Reason: java.io.EOFException[null]
Oct 18, 2018 12:16:59 AM com.hazelcast.client.ConnectionManager
WARNING: Server didn't respond to client's requests for 30 seconds. Assuming it is dead, closing the connection!
Exception in thread "Timer-3" java.lang.NullPointerException
at com.hazelcast.client.ConnectionManager$1.run(ConnectionManager.java:94)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
metacat 20181018-00:17:14: [ERROR]: Error while looking up the local identifier: An I/O error occured while sending to the backend. [edu.ucsb.nceas.metacat.IdentifierManager:getLocalId:1823]
Oct 18, 2018 12:17:03 AM com.hazelcast.nio.Connection
INFO: [127.0.0.1]:5701 [knb] Connection [Address[127.0.0.1]:46297] lost. Reason: java.io.EOFException[null]
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ContainerBackgroundProcessor[S
tandardEngine[Catalina]]"
Oct 18, 2018 12:17:03 AM com.hazelcast.client.ConnectionManager
WARNING: Connection to Connection [0] [localhost/127.0.0.1:5701 -> 127.0.0.1:5701] is lost
metacat 20181018-00:17:35: [ERROR]: NamingException when getting referral contex. Skipping this referral. ldap.ecoinform
atics.org:389 [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:17:49: [ERROR]: NamingException when getting referral contex. Skipping this referral. ldap.ecoinform
atics.org:389 [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
Oct 18, 2018 12:17:01 AM com.hazelcast.client.ProxyHelper
INFO: There is no response for Call [622795] operation=CONCURRENT_MAP_REMOVE in 10 seconds.
Oct 18, 2018 12:17:36 AM com.hazelcast.client.ConnectionManager
WARNING: got an exception on getConnection:Java heap space
java.lang.OutOfMemoryError: Java heap space
original pathInfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/dirtySystemMetadata
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /dirtySystemMetadata
original pathInfo: /checksum/testV2CreateV1GetSysMeta_201710915250987
original requestURI: /knb/d1/mn/v2/checksum/testV2CreateV1GetSysMeta_201710915250987
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /checksum/testV2CreateV1GetSysMeta_201710915250987
After decoded: testV2CreateV1GetSysMeta_201710915250987
original pathInfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/dirtySystemMetadata
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /dirtySystemMetadata
original pathInfo: /checksum/testV2CreateV1GetSysMeta_201636402417180
original requestURI: /knb/d1/mn/v2/checksum/testV2CreateV1GetSysMeta_201636402417180
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /checksum/testV2CreateV1GetSysMeta_201636402417180
After decoded: testV2CreateV1GetSysMeta_201636402417180
original pathInfo: /dirtySystemMetadata
original requestURI: /knb/d1/mn/v2/dirtySystemMetadata
stripping /knb/d1/mn/v2 from requestURI
new pathinfo: /dirtySystemMetadata
metacat 20181018-00:17:54: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:18:00: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:18:00: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:17:59: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
metacat 20181018-00:17:59: [ERROR]: NamingException when getting referral contex. Skipping this referral. LDAP response
read timed out, timeout used:5000ms. [edu.ucsb.nceas.metacat.AuthLdap:getGroups:1019]
~~~
